Sentiment Analysis and Stance Detection on German YouTube Comments on Gender Diversity
Keywords:stance detection, sentiment analysis, BERT, neural networks, annotation, YouTube comments, gender diversity
This paper explores different options of detecting the stance of German YouTube comments regarding the topic of gender diversity and compares the respective results with those of sentiment analysis, showing that these are two very different NLP tasks focusing on distinct characteristics of the discourse. While an already existing model was used to analyze the comments’ sentiment (BERT), the comments’ stance was first annotated and then used to train different models – SVM with TF-IDF, DistilBERT, LSTM and CNN – for predicting the stance of unseen comments. The best results were achieved by the CNN, reaching 78.3% accuracy (92% after dataset normalization) on the test set. Whereas the most common stance identified in the comments is a neutral one (neither completely in favor nor completely against gender diversity), the overall sentiment of the discourse turns out to be negative. This shows that the discourse revolving around the topic of gender diversity in YouTube comments is filled with strong opinions, on the one hand, but also opens up a space for anonymously inquiring and learning about the topic and its implications, on the other. Our research thereby (1) contributes to the understanding and application of different NLP tasks used to predict the sentiment and stance of unstructured textual data, and (2) provides relevant insights into society’s attitudes towards a changing system of values and beliefs.
ALDayel, Abeer, and Walid Magdy. 2021. "Stance detection on social media: State of the art and trends." Information Processing and Management 58: 1-22. https://doi.org/10.1016/j.ipm.2021.102597
Augenstein, Isabelle, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016. "Stance Detection with Bidirectional Conditional Encoding." In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, November 01-05. Association for Computational Linguistics. 876-885. https://doi.org/10.18653/v1/D16-1084
Biber, Douglas, and Edward Finegan. 1988. "Adverbial stance types in English." Discourse Processes 11(1): 1-34. https://doi.org/10.1080/01638538809544689
Birjali, Marouane, Mohammed Kasri, and Abderrahim Beni-Hssane. 2021. "A comprehensive survey on sentiment analysis: Approaches, challenges and trends." Knowledge-Based Systems 226(107134). https://doi.org/10.1016/j.knosys.2021.107134
Brownlee, Jason. 2017. Long Short-Term Memory Networks With Python: Develop Sequence Prediction Models With Deep Learning. Machine Learning Mastery. Retrieved from https://machinelearningmastery.com/lstms-with-python/
Chopra, Sahil, Saachi Jain, and John Merriman Sholar. 2017. "Towards Automatic Identification of Fake News: Headline-Article Stance Detection with LSTM Attention Models." CS224N project report, Stanford University. https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1174/reports/2761028.pdf
Cieliebak, Mark, Jan Milaln Deriu, Dominic Egger, and Fatih Uzdilli. 2017. A Twitter Corpus and Benchmark Resources for German Sentiment Analysis. Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-1106
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, June 02-07. Association for Computational Linguistics. 4171-4186. https://doi.org/10.18653/v1/N19-1423
Dey, Kuntal, Ritvik Shrivastava, and Saroj Kaushik. 2018. "Topical Stance Detection for Twitter: A Two-Phase LSTM Model Using Attention." Paper presented at 40th European Conference on IR Research 2018, Grenoble, France, March 26-29. doi:10.48550/arXiv.1801.03032
Du Bois, John W. 2007. "The stance triangle." In Stancetaking in Discourse. Subjectivity, evaluation, interaction, edited by R. Englebretson, 139-182. Amsterdam/Philadelphia: John Benjamins Publishing Company. https://doi.org/10.1075/pbns.164.07du
Ezen-Can, Aysu. 2020. "A Comparison of LSTM and BERT for Small Corpus." Available online: https://arxiv.org/abs/2009.05451.
Go, Alec, Richa Bhayani, and Lei Huang. 2009. "Twitter Sentiment Classification using Distant Supervision." CS224N project report, Stanford University. https://www-cs.stanford.edu/people/alecmgo/papers/TwitterDistantSupervision09.pdf
Göhring, Anne, Manfred Klenner, and Sophia Conrad. 2021. "DeInStance: Creating and Evaluating a German Corpus for Fine-Grained Inferred Stance Detection." In Proceedings of the 17th Conference on Natural Language Processing, Düsseldorf, Germany, September 06-09. KONVENS 2021 Organizers. 213-217. http://aclanthology.org/2021.konvens-1.20/
Goldberg, Yoav. 2016. "A Primer on Neural Network Models for Natural Language Processing." Journal of Artificial Intelligence Research 57(1): 35l-420. https://doi.org/10.1613/jair.4992
Goldhahn, Dirk, Thomas Eckart, and Uwe Quasthoff. 2012. Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages. European Language Resources Association (ELRA).
Gonçalves, Pollyanna, Matheus Araújo, Fabrício Benevenuto, and Meeyoung Cha. 2014. "Comparing and Combining Sentiment Analysis Methods." In Proceedings of the first ACM conference on Online social networks, Boston, Massachusetts, October 07-08. New York: Association for Computing Machinery. 27-38. https://doi.org/10.1145/2512938.2512951
Guhr, Oliver, Anne-Kathrin Schumann, Frank Bahrmann, and Hans-Joachim Böhme. 2020. "Training a Broad-Coverage German Sentiment Classification Model for Dialog Systems." In Proceedings of the 12th Conference on Language Resources and Evaluation, Marseille, France, May 11-16. European Language Resources Association. 1627-1632. http://aclanthology.org/2020.lrec-1.202/.
He, Haibo, and Edwardo A. Garcia. 2009. "Learning from Imbalanced Data." IEEE Transactions on Knowledge and Data Engineering 21(9): 1263 l-1284. https://doi.org/10.1109/TKDE.2008.239
IBM a. "What are Recurrent Neural Networks?" September 14, 2020. https://www.ibm.com/cloud/learn/recurrent-neural-networks.
IBM b. "Convolutional Neural Networks." October 20, 2020. https://www.ibm.com/cloud/learn/convolutional-neural-networks.
Kim, Yoon. 2014. "Convolutional Neural Networks for Sentence Classification." In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, October 25-29. Association for Computational Linguistics. 1746-1751. https://doi.org/10.3115/v1/D14-1181
Kingma, Diederik P., and Jimmy Lei Ba. 2015. "Adam: A method for stochastic optimization." Paper presented at the 3rd International Conference for Learning Representations, San Diego, California, May 07-09. http://arxiv.org/pdf/1412.6980.pdf.
Krejzl, Peter, Barbora Hourová, and Josef Steinberger. 2017. "Stance detection in online discussions." Work-in-progress paper. doi:10.48550/arXiv.1701.00504.
Landis, J. Richard, and Gary G. Koch. 1977. "The Measurement of Observer Agreement for Categorical Data." Biometrics 33(1): 159-174. https://doi.org/10.2307/2529310
Medhat, Walaa, Ahmed Hassan, and Hoda Korashy. 2014. "Sentiment analysis algorithms and applications: A survey." Ain Shams Engineering Journal 5: 1093-1113. https://doi.org/10.1016/j.asej.2014.04.011
Munaro, Ana Cristina, Renato Hübner Barcelos, Eliane Cristine Francisco Maffezzolli, João Pedro Santos Rodrigues, and Emerson Cabrera Paraiso. 2021. "To engage or not engage? The features of video content on YouTube affecting digital consumer engagement." Journal of Consumer Behaviour 20(5): 1336-1352. https://doi.org/10.1002/cb.1939
Poddar, Lahari, Wynne Hsu, Mong Li Lee, and Shruti Subramaniyam. 2018. "Predicting Stances in Twitter Conversations for Detecting Veracity of Rumors: a Neural Approach." In Proceedings of the 2018 IEEE 30th International Conference on Tools with Artificial Intelligence, Volos, Greece, November 05-07. The Institute of Electrical and Electronics Engineers, Inc. 65-72. https://doi.org/10.1109/ICTAI.2018.00021
Prati, Ronaldo C., Gustavo E.A.P.A. Batista, and Maria C. Monard. 2004. "Class imbalances versus class overlapping: an analysis of a learning system behavior." In MICAI 2004: Advances in Artificial Intelligence, Third Mexian International Conference on Artificial Intelligence, Mexico City, Mexico, April 26-30, 2004, 312-321. Berlin/Heidelberg: Springer. https://doi.org/10.1007/978-3-540-24694-7_32
Saif, Hassan, Yulan He, and Harith Alani. 2012. "Semantic Sentiment Analysis of Twitter." In The Semantic Web - ISWC 2012. Proceedings, Part I, Boston, Massachusetts, USA, November 11-15. Berlin/Heidelberg: Springer. 508-524. https://doi.org/10.1007/978-3-642-35176-1_32
Sänger, Mario, Ulf Leser, Steffen Kemmerer, Peter Adolphs, and Roman Klinger. 2016. SCARE - The Sentiment Corpus of App Reviews with Fine-grained Annotations in German. European Language Resources Association (ELRA).
Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." Paper presented at The 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, Co-located with the 33rd Conference on Neural Information Processing Systems 2019, Vancouver, British Columbia, December 13. doi:10.48550/arXiv.1910.01108
Sarlan, Aliza, Chayanit Nadam, and Shuib Basri. 2014. Twitter Sentiment Analysis. In 2014 International Conference on Information Technology and Multimedia, Putrajaya, Malaysia, November 18-20. IEEE. 212-216. https://doi.org/10.1109/ICIMU.2014.7066632
Sidarenka, Uladzimir. 2016. PotTS: The Potsdam Twitter Sentiment Corpus. European Language Resources Association (ELRA).
Suthaharan, Shan. 2016. Machine Learning Models and Algorithms for Big Data Classification. New York: Springer. https://doi.org/10.1007/978-1-4899-7641-3
Taher, SM Abu, Kazi Afsana Akhter, and K.M. Azharul Hasan. 2018. "N-gram Based Sentiment Mining for Bangla Text Using Support Vector Machine." In 2018 International Conference on Bangla Speech and Language Processing, Sylhet, Bangladesh, September 21-22. IEEE. 70-75. https://doi.org/10.1109/ICBSLP.2018.8554716
Wojatzki, Michael, Eugen Ruppert, Sarah Holschneider, Torsten Zesch, and Chris Biemann. 2017. "GermEval 2017: Shared Task on Aspect-based Sentiment in Social Media Customer Feedback." In Proceedings of the GermEval 2017, Berlin, Germany, September 12. GSCL. 1-12. doi:10.17185/duepublico/72074
Yusof, Nor Nadiah, Azlinah Mohamed, and Shuzlina Abdul-Rahman. 2015. "Reviewing Classification Approaches in Sentiment Analysis." In Soft Computing in Data Science. First Interntational Conference 2015. Proceedings, Putrajaya, Malaysia, September 02-03. Singapore: Springer. 43-53. https://doi.org/10.1007/978-981-287-936-3_5
Copyright (c) 2022 Journal of Computer-Assisted Linguistic Research
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
This journal is licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License