Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification

This study presents a comparison of four different machine learning algorithms for sentiment analysis on a small subset of the AR-P (Amazon Reviews - Polarity) dataset. The algorithms evaluated are multilayer perceptron (MLP), Naive Bayes, Decision Tree, and Transformer architectures. The results show that the Transformer-based DistilBERT model performed the best with an accuracy rate of 96.10%, while MLP had a better performance than the other remaining methods. Confusion matrices and ROC curves are provided to illustrate the results, and a comparison with previous studies is presented. The study concludes that the results can serve as a basis for future work, such as using larger datasets or comparing the performance of algorithms on different tasks. Overall, this study provides insights into the use of traditional machine learning and modern deep learning methods for sentiment analysis and their potential applications in real-world scenarios.

Keywords:

Deep Learning, Distilbert, Sentiment Analysis Text Classification, Transformer,

Tam Metin

___

Abdi, A., Shamsuddin, S. M., Hasan, S., & Piran, J. (2019). Deep learning-based sentiment classification of evaluative text based on Multi-feature fusion. Information Processing & Management, 56(4), 1245-1259.
Ain, Q. T., Ali, M., Riaz, A., Noureen, A., Kamran, M., Hayat, B., & Rehman, A. (2017). Sentiment analysis using deep learning techniques: a review. International Journal of Advanced Computer Science and Applications, 8(6).
Alexandridis, G., Varlamis, I., Korovesis, K., Caridakis, G., & Tsantilas, P. (2021). A survey on sentiment analysis and opinion mining in greek social media. Information, 12(8), 331.
Al-Garadi, M. A., Yang, Y. C., Cai, H., Ruan, Y., O’Connor, K., Graciela, G. H., ... & Sarker, A. (2021). Text classification models for the automatic detection of nonmedical prescription medication use from social media. BMC Medical Informatics and Decision Making, 21(1), 1-13. DOI: 10.1186/s12911-021-01488-1
Balahur, A., Turchi, M., & Steinberger, R. (2013). Multilingual sentiment analysis using machine translation–based techniques. ACM Transactions on Intelligent Systems and Technology (TIST), 4(1), 1-26. DOI: 10.1145/2444776.2444777
Bayat, S., & Işık, G. (2022). Recognition of Aras Bird Species From Their Voices With Deep Learning Methods. Journal of the Institute of Science and Technology, 12(3), 1250-1263.
Bollen, J., Mao, H., &. Zeng, X. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2(1), 1-8. https://doi.org/10.1016/j.jocs.2010.12.007
Boyko, N., & Boksho, K. (2020, November). Application of the Naive Bayesian Classifier in Work on Sentimental Analysis of Medical Data. In Proceedings of the International Conference on Intelligent Data and Digital Medicine (IDDM) (pp. 230-239).
Cambria, E., & White, B. (2014). Jumping NLP curves: A review of natural language processing research. IEEE Journal of Computational Intelligence, 9(2), 48-57. https://doi.org/10.1109/MCI.2014.2307227
Chen, L. C., Lee, C. M., and Chen, M. Y. (2020) published a study in Soft Computing, in which they explored social media for sentiment analysis using deep learning techniques.
Delobelle, P., Winters, T., & Berendt, B. (2020). Robbert: a dutch roberta-based language model. arXiv preprint arXiv:2001.06286.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
Dogra, V., Verma, S., Kavita, C., Chatterjee, P., Shafi, J., Choi, J., & Ijaz, M. F. (2022). A Complete Process of Text Classification System Using State-of-the-Art NLP Models. Computational Intelligence and Neuroscience, 2022, 1883698. https://doi.org/10.1155/2022/1883698
Gao, J., &. Wong, K.-F. (2014). A review of sentiment analysis research in Chinese language. Informatics, 1(3), 191-208. https://doi.org/10.3390/informatics1030191
Ghulam, H., Zeng, F., Li, W., & Xiao, Y. (2019). Deep learning-based sentiment analysis for roman urdu text. Procedia computer science, 147, 131-135.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
Gupta, R., Sameer, S., Muppavarapu, H., Enduri, M. K., & Anamalamudi, S. (2021, September). Sentiment analysis on Zomato reviews. In 2021 13th International Conference on Computational Intelligence and Communication Networks (CICN) (pp. 34-38). IEEE.
Gündüz, M. Ş., & Işık, G. (2023). A new YOLO-based method for social distancing from real-time videos. Neural Computing and Applications, 1-11.
Han, J., & Kamber, M. (2011). Data mining: concepts and techniques. Elsevier.
Hugging Face. (2023b). AutoModelForSequenceClassification. https://huggingface.co/transformers/model_doc/auto.html#automodelforsequenceclassification
Hutto, C. J., & Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. Eighth International AAAI Conference on Weblogs and Social Media. https://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/view/8109/8122
Joshi, A., Bhattacharyya, P., & Carman, M. J. (2020). A thorough examination of the DistilBERT model for sentence classification. arXiv preprint arXiv:2010.16061.
Kim, S. (2020). Sentiment analysis: A comprehensive guide to detecting emotions, opinions, and sentiments.
Kumar, V. (2022). A Review of Decision Tree Algorithms for Classification in Machine Learning. International Journal of Computer Applications, 182(40), 10-16.
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint arXiv:1909.11942.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
M. Al-Qurishi, T. Khalid and R. Souissi, "Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues," in IEEE Access, vol. 9, pp. 126917-126951, 2021, doi: 10.1109/ACCESS.2021.3110912.
Mohammad, S. M., & Bravo-Marquez, F. (2017). WASSA-2017 shared task on emotion intensity. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 1-10. https://doi.org/10.18653/v1/W17-5201
Nandwani, P., & Verma, R. (2021). A review on sentiment analysis and emotion detection from text. Social Network Analysis and Mining, 11(1), 81.
Othan, D., Kilimci, Z. H., & Uysal, M. (2019, December). Financial sentiment analysis for predicting direction of stocks using bidirectional encoder representations from transformers (BERT) and deep learning models. In Proc. Int. Conf. Innov. Intell. Technol (pp. 30-35).
Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2), 1-135. https://doi.org/10.1561/1500000011
Poria, S., Cambria, E., &. Bajpai, R. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98-125. https://doi.org/10.1016/j.inffus.2017.02.003
Raschka, S., & Mirjalili, V. (2021). Naive Bayes and Text Classification. In Python Machine Learning, Third Edition (pp. 373-394). Packt Publishing.
Ray, P., & Chakrabarti, A. (2022). A mixed approach of deep learning method and rule-based method to improve aspect level sentiment analysis. Applied Computing and Informatics, 18(1/2), 163-178.
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Sudhir, P., & Suresh, V. D. (2021). Comparative study of various approaches, applications and classifiers for sentiment analysis. Global Transitions Proceedings, 2(2), 205-211.
Sun, C., Li, L., Wang, W., &. Jiang, B. (2021). Multi-task learning for sentiment analysis using transformer- Khalid based models. Neural Networks, 137, 181-190. https://doi.org/10.1016/j.neunet.2020.11.010
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008.
Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731-5780.

___

Bibtex	@araştırma makalesi { jist1292050, journal = {Journal of the Institute of Science and Technology}, eissn = {2536-4618}, address = {}, publisher = {Iğdır Üniversitesi}, year = {2023}, volume = {13}, number = {3}, pages = {1496 - 1510}, doi = {10.21597/jist.1292050}, title = {Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification}, key = {cite}, author = {Bayat, Seda and Işık, Gültekin} }
APA	Bayat, S. & Işık, G. (2023). Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification . Journal of the Institute of Science and Technology , 13 (3) , 1496-1510 . DOI: 10.21597/jist.1292050
MLA	Bayat, S. , Işık, G. "Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification" . Journal of the Institute of Science and Technology 13 (2023 ): 1496-1510 <
Chicago	Bayat, S. , Işık, G. "Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification". Journal of the Institute of Science and Technology 13 (2023 ): 1496-1510
RIS	TY - JOUR T1 - Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification AU - SedaBayat, GültekinIşık Y1 - 2023 PY - 2023 N1 - doi: 10.21597/jist.1292050 DO - 10.21597/jist.1292050 T2 - Journal of the Institute of Science and Technology JF - Journal JO - JOR SP - 1496 EP - 1510 VL - 13 IS - 3 SN - -2536-4618 M3 - doi: 10.21597/jist.1292050 UR - Y2 - 2023 ER -
EndNote	%0 Iğdır Üniversitesi Fen Bilimleri Enstitüsü Dergisi Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification %A Seda Bayat , Gültekin Işık %T Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification %D 2023 %J Journal of the Institute of Science and Technology %P -2536-4618 %V 13 %N 3 %R doi: 10.21597/jist.1292050 %U 10.21597/jist.1292050
ISNAD	Bayat, Seda , Işık, Gültekin . "Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification". Journal of the Institute of Science and Technology 13 / 3 (Eylül 2023): 1496-1510 .
AMA	Bayat S. , Işık G. Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification. Iğdır Üniv. Fen Bil Enst. Der.. 2023; 13(3): 1496-1510.
Vancouver	Bayat S. , Işık G. Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification. Journal of the Institute of Science and Technology. 2023; 13(3): 1496-1510.
IEEE	S. Bayat ve G. Işık , "Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification", , c. 13, sayı. 3, ss. 1496-1510, Eyl. 2023, doi:10.21597/jist.1292050