Evaluation of Automatic Text Summarizations based on Human Summaries

The goal of this paper is to compare summaries generated by different automatic text summarization methods and those generated by human beings. To achieve this end, we did two series of experiments: in the first one, we employed automatically produced extractive summaries; in the second one, manually-produced summaries obtained by several English teachers were used. Our automatic summaries were obtained using Fuzzy method and Vector approach. Using Rouge evaluation system, we compared the manually-produced summaries and the automatically-produced ones. Rouge evaluation of generated summaries indicated the superiority of summaries produced by humans over the automatically produced summaries. On the other hand, the comparison between the generated summaries showed that summaries produced by Fuzzy method were much more acceptable and understandable compared to summaries produced by Vector approach. This can provide support for the replacement of manually generated summaries by summaries produced using Fuzzy method in certain cases where real time summaries are needed.