1 Discourse Mode Identification in EssaysWei Song Capital Normal University Cooperating with Dong Wang, Ruiji Fu, Lizhen Liu, Ting Liu, Guoping Hu IFLYTEK Research and Harbin Institute of Technology

2

3 Outline Discourse Modes Data Annotation Discourse Mode IdentificationEssay Scoring with Discourse Modes Conclusion

4 Outline Discourse Modes Data Annotation Discourse Mode IdentificationEssay Scoring with Discourse Modes Conclusion

5 Discourse Modes Discourse modes, also known as rhetorical modes, describe the purpose and conventions of the main kinds of language based communication Several taxonomies of discourse moods in the literature

6 Taxonomies of Discourse ModesDiscourse modes by C. Smith, studying discourse passages from a linguistic view of point Narration Description Argument Information Report

7 Taxonomies of Discourse ModesDiscourse modes in rhetoric Narration Description Argumentation Exposition

8 Taxonomies of Discourse ModesDiscourse modes in Chinese composition Narration Description Argument Exposition Emotion Expressing

9 Functions of Discourse Modes in a textVarious discourse modes stand for unity of a text Discourse modes can reflect the organization and progression of a text Indicating the intention of writing a passage Discourse modes have rhetorical significance Preferring different expressive styles Flexible use of multiple discourse modes

10 Research Questions Discourse mode identification is a fundamental but less studied problem in NLP Can we annotate a corpus with acceptable agreement? Can discourse modes be identified automatically? Can discourse mode identification help downstream NLP tasks

11 Outline Discourse Modes Data Annotation Discourse Mode IdentificationEssay Scoring with Discourse Modes Conclusion

12 Discourse Modes in this workWe follow the Chinese convention Narration is to introduce an event or series of events Exposition is to explain or instruct or provide background information in narrative context Description is to re-creates, invents, or vividly show what things are like Argument is to make a point of view and prove its validity towards a topic Emotion Expressing is to presents the writer’s motions, usually in a subjective, personal and lyrical way

13 Data Collect 415 narrative essays written by high school students in native Chinese language 32 sentences and 670 words in average Two annotators were asked to label discourse modes for each sentence Each sentence can have more than one discourse mode, but a dominant mode should be informed

14 Inter-Annotator Agreement on the dominant mode50 essays were annotated independently by two annotators Measured by PRF and Kappa Example: “父亲的爱是灯塔，引导我一生前进的路！”

15 Inter-Annotator Agreement on the dominant mode50 essays were annotated independently by two annotators Measured by PRF and Kappa

16 Distribution of Discourse ModesDistribution is imbalanced

17 Co-Occurrence 22% sentences have more than one discourse modesDescription tends to co-occur with narration and emotion Providing details of events Evoking emotions Emotion co-occurs with argument Proper emotional appeals can enhance the strength of argument 海上生明月，天涯共此时。

18 Transitions Most modes tend to transit to themselvesContextual information should be helpful

19 Summary Annotators can achieve an acceptable agreement after trainingAbout 22% sentences have more than one discourse mode Distribution of discourse modes is imbalanced Discourse modes have local transition patterns

20 Outline Discourse Modes Data Annotation Discourse Mode IdentificationEssay Scoring with Discourse Modes Conclusion

21 Discourse Mode IdentificationWe view it as a multi-label sequence labeling problem Pre-trained Embeddings

22 Discourse Mode IdentificationDeal with multiple-Label outputs

23 Discourse Mode IdentificationConsidering paragraph boundaries

24 Evaluation Comparisons SVM with unigram and bigram featuresCNN (Kim et al. 2014) GRU GRU-GRU (GG): Our hierarchical model GRU-GRU-SEG (GG-SEG): Consider paragraph boundaries on the top of GG

25 Evaluation F1-score is reportedNeural models outperform bag-of-words method RNN is slightly better than CNN Sequence information is useful Minority modes are more sensitive to positions Overall average F1 is 0.7 Average F1 on three main modes is above 0.76

26 Outline Discourse Modes Data Annotation Discourse Mode IdentificationEssay Scoring with Discourse Modes Conclusion

27 Automatic Essay Scoring (AES)AES is the task of building a computer-aided scoring system, in order to reduce the involvement of human raters. AES as a regression problem Support Vector Regression Bayesian linear ridge regression

28 Feature Sets Discourse mode featuresBasic features (Phandi et al. 2015) Length features Prompt features Content features Selected unigrams and bigrams The number of Chinese idioms The number of words in Chinese Proficiency Test 6 Dictionary Discourse mode features Discourse mode ratio #sentence with the discourse mode / #sentences Unigrams and bigrams of discourse mode sequences

29 Data and Settings Three promptsNarrative essays written by junior school students in local tests 5-folds cross-validation Evaluated with Quadratic Weighted Kappa (QWK)

30 Evaluation Overall performance BLRR performs betterDiscourse mode features are useful

31 Evaluation Pearson correlation coefficient between discourse mode ratio and scores Narration has a negative correlation Description is most relevant Emotion expressing has a weak correlation

32 Evaluation Performance on essays with different lengthWhen the effect of length becomes weaker, AES becomes harder In hard cases, the role of discourse mode features becomes more important

33 Outline Discourse Modes Data Annotation Discourse Mode IdentificationEssay Scoring with Discourse Modes Conclusion

34 Conclusion We have studied a fundamental but less studied problem in NLP Both manual and automatic discourse mode identification is feasible Discourse mode features are shown useful for automatic essay scoring Discourse mode identification can support other downstream NLP applications potentially

35 Thank you

36 Main References Carlota S Smith Modes of discourse: The local structure of texts, volume 103. Cambridge University Press. Cleanth Brooks and Robert Penn Warren Modern rhetoric. Harcourt, Brace. Yoon Kim Convolutional neural networks for sentence classification. In Proceedings of EMNLP pages 1746–1751. Peter Phandi, Kian Ming A. Chai, and Hwee Tou Ng Flexible domain adaptation for automated essay scoring using correlated linear regression. In Proceedings of EMNLP pages 431–439.