DANA App Sentiment Analysis: Comparison of XGBoost, SVM, and Extra Trees
(1) Department of Informatics, Faculty of Engineering, University of Muhammadiyah Malang, Jawa Timur, Indonesia
(2) Department of Informatics, Faculty of Engineering, University of Muhammadiyah Malang, Jawa Timur, Indonesia
(*) Corresponding Author
Abstract
This research aims to analyze sentiment on DANA application reviews to find out user perceptions by comparing Extreme Gradient Boosting (XGBoost), Support Vector Machine (SVM), and Extra Trees Classifier classification methods. DANA application review data is obtained from the Kaggle site which consists of 50,000 Indonesian-language reviews labeled with positive and negative sentiments. The research stages include data preprocessing to clean and prepare the review text, applying word weighting using Word2Vec to give weight to words based on their context, balancing sentiment classes using SMOTE to address the imbalance of positive and negative review classes. It should be noted that the initial proportion of data before applying SMOTE may affect the results. The data is then divided into training and testing sets, then the models are trained and evaluated using Confusion Matrix and K-Fold Cross-Validation. The results of the three classification methods are measured by the accuracy matrix and F1-Score to assess model performance, the SVM and XGBoost methods obtained an accuracy of 93% and the ETC method achieved an F1-Score value of 96% at K=6, the three models proved to be very accurate in predicting the sentiment of DANA application reviews both positive and negative. The practical implications of this research can identify areas for application improvement, develop popular features, personalize services based on user preferences, and manage application reputation.
Keywords
Full Text:
PDFReferences
M. Najib and F. Fahma, “Investigating the adoption of digital payment system through an extended technology acceptance model: An insight from the Indonesian small and medium enterprises,” Int. J. Adv. Sci. Eng. Inf. Technol., vol. 10, no. 4, pp. 1702–1708, 2020, doi: 10.18517/ijaseit.10.4.11616.
Y. Dinda Oktaviani Waruwu, “Sistem Pendukung Keputusan Pemilihan E – Wallet Terbaik Dengan Menggunakan Metode Analytical Hierracy Process (AHP),” J. Ilm. Sain dan Teknol., vol. 2, no. 2, pp. 101–116, 2024.
H. Ammar Faris, dkk, “DeLone and McLean Model Analysis of Success Factors of SIDEMANG Application in Palembang City,” Sisfokom., vol. 13, no. 2, p. 160, Juny 2024, doi: 10.32736/sisfokom.v13i2.1894.
P. A. Permatasari, L. Linawati, and L. Jasa, “Survei Tentang Analisis Sentimen Pada Media Sosial,” Maj. Ilm. Teknol. Elektro, vol. 20, no. 2, p. 177, Dec. 2021, doi: 10.24843/MITE.2021.v20i02.P01.
K. Kusnawi, M. Rahardi, and V. D. Pandiangan, “Sentiment Analysis of Neobank Digital Banking using Support Vector Machine Algorithm in Indonesia,” JOIV Int. J. Informatics Vis., vol. 7, no. 2, p. 377, May 2023, doi: 10.30630/joiv.7.2.1652.
D. A. Putri, D. A. Kristiyanti, E. Indrayuni, A. Nurhadi, and D. R. Hadinata, “Comparison of Naive Bayes Algorithm and Support Vector Machine using PSO Feature Selection for Sentiment Analysis on E-Wallet Review,” J. Phys. Conf. Ser., vol. 1641, no. 1, p. 012085, Nov. 2020, doi: 10.1088/1742-6596/1641/1/012085.
B. Andrian, T. Simanungkalit, I. Budi, and A. F. Wicaksono, “Sentiment Analysis on Customer Satisfaction of Digital Banking in Indonesia,” Int. J. Adv. Comput. Sci. Appl., vol. 13, no. 3, pp. 466–473, 2022, doi: 10.14569/IJACSA.2022.0130356.
B. Harnadi and A. D. Widiantoro, “Evaluating the Performance and Accuracy of Supervised Learning Models on Sentiment Analysis of E-Wallet,” in 2023 7th International Conference on Information Technology (InCIT), IEEE, Nov. 2023, pp. 175–180. doi: 10.1109/InCIT60207.2023.10413111.
H. Wisnu, M. Afif, and Y. Ruldevyani, “Sentiment analysis on customer satisfaction of digital payment in Indonesia: A comparative study using KNN and Naïve Bayes,” J. Phys. Conf. Ser., vol. 1444, no. 1, p. 012034, Jan. 2020, doi: 10.1088/1742-6596/1444/1/012034.
F. R. Ananda Dimas Sanjaya, Tacbir Hendro Pudjiantoro, Ade Kania Ningsih, “Sentiment Analysis Of E-Wallets on Twitter social media With Naïve Bayes and Lexicon-Based Methods,” in Proceedings of the International Conference on Industrial Engineering and Operations Management, Michigan, USA: IEOM Society International, 2022, pp. 1002–1010. doi: 10.46254/AP03.20220200.
M. Omarkhan, G. Kissymova, and I. Akhmetov, “Handling data imbalance using CNN and LSTM in financial news sentiment analysis,” Proc. - 2021 16th Int. Conf. Electron. Comput. Comput. ICECCO 2021, pp. 1–8, 2021, doi: 10.1109/ICECCO53203.2021.9663802.
D. A. Kristiyanti, D. A. Putri, E. Indrayuni, A. Nurhadi, and A. H. Umam, “E-Wallet Sentiment Analysis Using Naïve Bayes and Support Vector Machine Algorithm,” J. Phys. Conf. Ser., vol. 1641, no. 1, 2020, doi: 10.1088/1742-6596/1641/1/012079.
K. Afifah, I. N. Yulita, and I. Sarathan, “Sentiment Analysis on Telemedicine App Reviews using XGBoost Classifier,” 2021 Int. Conf. Artif. Intell. Big Data Anal. ICAIBDA 2021, pp. 22–27, 2021, doi: 10.1109/ICAIBDA53487.2021.9689762.
I. R. Hendrawan, E. Utami, and A. D. Hartanto, “Comparison of Word2vec and Doc2vec Methods for Text Classification of Product Reviews,” Proceeding - 6th Int. Conf. Inf. Technol. Inf. Syst. Electr. Eng. Appl. Data Sci. Artif. Intell. Technol. Environ. Sustain. ICITISEE 2022, pp. 530–534, 2022, doi: 10.1109/ICITISEE57756.2022.10057702.
A. N. Azhar, M. L. Khodra, and A. P. Sutiono, “Multi-label Aspect Categorization with Convolutional Neural Networks and Extreme Gradient Boosting,” Proc. Int. Conf. Electr. Eng. Informatics, vol. 2019-July, no. July, pp. 35–40, 2019, doi: 10.1109/ICEEI47359.2019.8988898.
M. Anita, R. Mannava, M. L. Deep, and M. V. R. Durga Prasad, “Using Machine Learning to Analyze Twitter’s Sentiment,” Proc. 2nd IEEE Int. Conf. Adv. Comput. Commun. Appl. Informatics, ACCAI 2023, pp. 1–8, 2023, doi: 10.1109/ACCAI58221.2023.10199230.
M. Syarifuddinn, “Analisis Sentimen Opini Publik Terhadap Efek Psbb Pada Twitter Dengan Algoritma Decision Tree,Knn, Dan Naïve Bayes,” INTI Nusa Mandiri, vol. 15, no. 1, pp. 87–94, 2020, doi: 10.33480/inti.v15i1.1433.
N. D. Kusumawati, S. Al Faraby, and M. Dwifebri, “Analisis Sentimen Komentar Beracun pada Media Sosial Menggunakan Word2Vec dan Support Vectore Machine ( SVM ),” e-Proceeding Eng., vol. 8, no. 5, pp. 10038–10050, 2021.
V. K. S. Que, A. Iriani, and H. D. Purnomo, “Analisis Sentimen Transportasi Online Menggunakan Support Vector Machine Berbasis Particle Swarm Optimization,” J. Nas. Tek. Elektro dan Teknol. Inf., vol. 9, no. 2, pp. 162–170, 2020, doi: 10.22146/jnteti.v9i2.102.
A. Fahmi Sabani, Adiwijaya, and W. Astuti, “Analisis Sentimen Review Film pada Website Rotten Tomatoes Menggunakan Metode SVM Dengan Mengimplementasikan Fitur Extraction Word2Vec,” e-Proceeding Eng., vol. 9, no. 3, p. 1800, 2022.
D. I. Af’idah, D. Dairoh, S. F. Handayani, and R. W. Pratiwi, “Pengaruh Parameter Word2Vec terhadap Performa Deep Learning pada Klasifikasi Sentimen,” J. Inform. J. Pengemb. IT, vol. 6, no. 3, pp. 156–161, 2021, doi: 10.30591/jpit.v6i3.3016.
G. Yang and L. Qicheng, “An Over Sampling Method of Unbalanced Data Based on Ant Colony Clustering,” IEEE Access, vol. 9, pp. 130990–130996, 2021, doi: 10.1109/ACCESS.2021.3114443.
E. D. N. Sari and I. Irhamah, “Analisis Sentimen Nasabah pada Layanan Perbankan Menggunakan Metode Regresi Logistik Biner, Naïve Bayes Classifier (NBC), dan Support Vector Machine (SVM),” J. Sains dan Seni ITS, vol. 8, no. 2, Feb. 2020, doi: 10.12962/j23373520.v8i2.44565.
W. Fangyu, Z. Jianhui, B. Youjun, and C. Bo, “Research on imbalanced data set preprocessing based on deep learning,” in 2021 Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS), IEEE, Jan. 2021, pp. 75–79. doi: 10.1109/ACCTCS52002.2021.00023.
K. R. Kavitha, A. Gopinath, and M. Gopi, “Applying improved svm classifier for leukemia cancer classification using FCBF,” in 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), IEEE, Sep. 2017, pp. 61–66. doi: 10.1109/ICACCI.2017.8125817.
S. Zahoor and R. Rohilla, “Twitter Sentiment Analysis Using Machine Learning Algorithms: A Case Study,” in 2020 International Conference on Advances in Computing, Communication & Materials (ICACCM), IEEE, Aug. 2020, pp. 194–199. doi: 10.1109/ICACCM50413.2020.9213011.
R. M. Simranjot Kaur, “Sentiment Analysis on twitter data using Machine Learning,” J. Xidian Univ., vol. 14, no. 12, Dec. 2020, doi: 10.37896/jxu14.12/039.
M. T. Akter, M. Begum, and R. Mustafa, “Bengali Sentiment Analysis of E-commerce Product Reviews using K-Nearest Neighbors,” in 2021 International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD), IEEE, Feb. 2021, pp. 40–44. doi: 10.1109/ICICT4SD50815.2021.9396910.
M. Syukron, R. Santoso, and T. Widiharih, “PERBANDINGAN METODE SMOTE RANDOM FOREST DAN SMOTE XGBOOST UNTUK KLASIFIKASI TINGKAT PENYAKIT HEPATITIS C PADA IMBALANCE CLASS DATA,” J. Gaussian, vol. 9, no. 3, pp. 227–236, Aug. 2020, doi: 10.14710/j.gauss.v9i3.28915.
A. S. Aribowo, H. T. Jaya, U. Pembangunan, and N. Veteran, “An Evaluation of Preprocessing Steps and Tree-based Ensemble Machine Learning for Analysing Sentiment on Indonesian YouTube Comments,” Int. J. Adv. Trends Comput. Sci. Eng., vol. 9, no. 5, 2020, doi: 10.30534/ijatcse/2020/29952020.
DOI: https://doi.org/10.32736/sisfokom.v13i3.2239
Refbacks
- There are currently no refbacks.
Indexed By:
