DANA App Sentiment Analysis: Comparison of XGBoost, SVM, and Extra Trees
DOI:
https://doi.org/10.32736/sisfokom.v13i3.2239Keywords:
Sentiment Analysis, XGBoost, Support Vector Machine, Extra Trees Classifier, Word2Vec, SMOTE,Abstract
This research aims to analyze sentiment on DANA application reviews to find out user perceptions by comparing Extreme Gradient Boosting (XGBoost), Support Vector Machine (SVM), and Extra Trees Classifier classification methods. DANA application review data is obtained from the Kaggle site which consists of 50,000 Indonesian-language reviews labeled with positive and negative sentiments. The research stages include data preprocessing to clean and prepare the review text, applying word weighting using Word2Vec to give weight to words based on their context, balancing sentiment classes using SMOTE to address the imbalance of positive and negative review classes. It should be noted that the initial proportion of data before applying SMOTE may affect the results. The data is then divided into training and testing sets, then the models are trained and evaluated using Confusion Matrix and K-Fold Cross-Validation. The results of the three classification methods are measured by the accuracy matrix and F1-Score to assess model performance, the SVM and XGBoost methods obtained an accuracy of 93% and the ETC method achieved an F1-Score value of 96% at K=6, the three models proved to be very accurate in predicting the sentiment of DANA application reviews both positive and negative. The practical implications of this research can identify areas for application improvement, develop popular features, personalize services based on user preferences, and manage application reputation.References
M. Najib and F. Fahma, “Investigating the adoption of digital payment system through an extended technology acceptance model: An insight from the Indonesian small and medium enterprises,” Int. J. Adv. Sci. Eng. Inf. Technol., vol. 10, no. 4, pp. 1702–1708, 2020, doi: 10.18517/ijaseit.10.4.11616.
Y. Dinda Oktaviani Waruwu, “Sistem Pendukung Keputusan Pemilihan E – Wallet Terbaik Dengan Menggunakan Metode Analytical Hierracy Process (AHP),” J. Ilm. Sain dan Teknol., vol. 2, no. 2, pp. 101–116, 2024.
H. Ammar Faris, dkk, “DeLone and McLean Model Analysis of Success Factors of SIDEMANG Application in Palembang City,” Sisfokom., vol. 13, no. 2, p. 160, Juny 2024, doi: 10.32736/sisfokom.v13i2.1894.
P. A. Permatasari, L. Linawati, and L. Jasa, “Survei Tentang Analisis Sentimen Pada Media Sosial,” Maj. Ilm. Teknol. Elektro, vol. 20, no. 2, p. 177, Dec. 2021, doi: 10.24843/MITE.2021.v20i02.P01.
K. Kusnawi, M. Rahardi, and V. D. Pandiangan, “Sentiment Analysis of Neobank Digital Banking using Support Vector Machine Algorithm in Indonesia,” JOIV Int. J. Informatics Vis., vol. 7, no. 2, p. 377, May 2023, doi: 10.30630/joiv.7.2.1652.
D. A. Putri, D. A. Kristiyanti, E. Indrayuni, A. Nurhadi, and D. R. Hadinata, “Comparison of Naive Bayes Algorithm and Support Vector Machine using PSO Feature Selection for Sentiment Analysis on E-Wallet Review,” J. Phys. Conf. Ser., vol. 1641, no. 1, p. 012085, Nov. 2020, doi: 10.1088/1742-6596/1641/1/012085.
B. Andrian, T. Simanungkalit, I. Budi, and A. F. Wicaksono, “Sentiment Analysis on Customer Satisfaction of Digital Banking in Indonesia,” Int. J. Adv. Comput. Sci. Appl., vol. 13, no. 3, pp. 466–473, 2022, doi: 10.14569/IJACSA.2022.0130356.
B. Harnadi and A. D. Widiantoro, “Evaluating the Performance and Accuracy of Supervised Learning Models on Sentiment Analysis of E-Wallet,” in 2023 7th International Conference on Information Technology (InCIT), IEEE, Nov. 2023, pp. 175–180. doi: 10.1109/InCIT60207.2023.10413111.
H. Wisnu, M. Afif, and Y. Ruldevyani, “Sentiment analysis on customer satisfaction of digital payment in Indonesia: A comparative study using KNN and Naïve Bayes,” J. Phys. Conf. Ser., vol. 1444, no. 1, p. 012034, Jan. 2020, doi: 10.1088/1742-6596/1444/1/012034.
F. R. Ananda Dimas Sanjaya, Tacbir Hendro Pudjiantoro, Ade Kania Ningsih, “Sentiment Analysis Of E-Wallets on Twitter social media With Naïve Bayes and Lexicon-Based Methods,” in Proceedings of the International Conference on Industrial Engineering and Operations Management, Michigan, USA: IEOM Society International, 2022, pp. 1002–1010. doi: 10.46254/AP03.20220200.
M. Omarkhan, G. Kissymova, and I. Akhmetov, “Handling data imbalance using CNN and LSTM in financial news sentiment analysis,” Proc. - 2021 16th Int. Conf. Electron. Comput. Comput. ICECCO 2021, pp. 1–8, 2021, doi: 10.1109/ICECCO53203.2021.9663802.
D. A. Kristiyanti, D. A. Putri, E. Indrayuni, A. Nurhadi, and A. H. Umam, “E-Wallet Sentiment Analysis Using Naïve Bayes and Support Vector Machine Algorithm,” J. Phys. Conf. Ser., vol. 1641, no. 1, 2020, doi: 10.1088/1742-6596/1641/1/012079.
K. Afifah, I. N. Yulita, and I. Sarathan, “Sentiment Analysis on Telemedicine App Reviews using XGBoost Classifier,” 2021 Int. Conf. Artif. Intell. Big Data Anal. ICAIBDA 2021, pp. 22–27, 2021, doi: 10.1109/ICAIBDA53487.2021.9689762.
I. R. Hendrawan, E. Utami, and A. D. Hartanto, “Comparison of Word2vec and Doc2vec Methods for Text Classification of Product Reviews,” Proceeding - 6th Int. Conf. Inf. Technol. Inf. Syst. Electr. Eng. Appl. Data Sci. Artif. Intell. Technol. Environ. Sustain. ICITISEE 2022, pp. 530–534, 2022, doi: 10.1109/ICITISEE57756.2022.10057702.
A. N. Azhar, M. L. Khodra, and A. P. Sutiono, “Multi-label Aspect Categorization with Convolutional Neural Networks and Extreme Gradient Boosting,” Proc. Int. Conf. Electr. Eng. Informatics, vol. 2019-July, no. July, pp. 35–40, 2019, doi: 10.1109/ICEEI47359.2019.8988898.
M. Anita, R. Mannava, M. L. Deep, and M. V. R. Durga Prasad, “Using Machine Learning to Analyze Twitter’s Sentiment,” Proc. 2nd IEEE Int. Conf. Adv. Comput. Commun. Appl. Informatics, ACCAI 2023, pp. 1–8, 2023, doi: 10.1109/ACCAI58221.2023.10199230.
M. Syarifuddinn, “Analisis Sentimen Opini Publik Terhadap Efek Psbb Pada Twitter Dengan Algoritma Decision Tree,Knn, Dan Naïve Bayes,” INTI Nusa Mandiri, vol. 15, no. 1, pp. 87–94, 2020, doi: 10.33480/inti.v15i1.1433.
N. D. Kusumawati, S. Al Faraby, and M. Dwifebri, “Analisis Sentimen Komentar Beracun pada Media Sosial Menggunakan Word2Vec dan Support Vectore Machine ( SVM ),” e-Proceeding Eng., vol. 8, no. 5, pp. 10038–10050, 2021.
V. K. S. Que, A. Iriani, and H. D. Purnomo, “Analisis Sentimen Transportasi Online Menggunakan Support Vector Machine Berbasis Particle Swarm Optimization,” J. Nas. Tek. Elektro dan Teknol. Inf., vol. 9, no. 2, pp. 162–170, 2020, doi: 10.22146/jnteti.v9i2.102.
A. Fahmi Sabani, Adiwijaya, and W. Astuti, “Analisis Sentimen Review Film pada Website Rotten Tomatoes Menggunakan Metode SVM Dengan Mengimplementasikan Fitur Extraction Word2Vec,” e-Proceeding Eng., vol. 9, no. 3, p. 1800, 2022.
D. I. Af’idah, D. Dairoh, S. F. Handayani, and R. W. Pratiwi, “Pengaruh Parameter Word2Vec terhadap Performa Deep Learning pada Klasifikasi Sentimen,” J. Inform. J. Pengemb. IT, vol. 6, no. 3, pp. 156–161, 2021, doi: 10.30591/jpit.v6i3.3016.
G. Yang and L. Qicheng, “An Over Sampling Method of Unbalanced Data Based on Ant Colony Clustering,” IEEE Access, vol. 9, pp. 130990–130996, 2021, doi: 10.1109/ACCESS.2021.3114443.
E. D. N. Sari and I. Irhamah, “Analisis Sentimen Nasabah pada Layanan Perbankan Menggunakan Metode Regresi Logistik Biner, Naïve Bayes Classifier (NBC), dan Support Vector Machine (SVM),” J. Sains dan Seni ITS, vol. 8, no. 2, Feb. 2020, doi: 10.12962/j23373520.v8i2.44565.
W. Fangyu, Z. Jianhui, B. Youjun, and C. Bo, “Research on imbalanced data set preprocessing based on deep learning,” in 2021 Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS), IEEE, Jan. 2021, pp. 75–79. doi: 10.1109/ACCTCS52002.2021.00023.
K. R. Kavitha, A. Gopinath, and M. Gopi, “Applying improved svm classifier for leukemia cancer classification using FCBF,” in 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), IEEE, Sep. 2017, pp. 61–66. doi: 10.1109/ICACCI.2017.8125817.
S. Zahoor and R. Rohilla, “Twitter Sentiment Analysis Using Machine Learning Algorithms: A Case Study,” in 2020 International Conference on Advances in Computing, Communication & Materials (ICACCM), IEEE, Aug. 2020, pp. 194–199. doi: 10.1109/ICACCM50413.2020.9213011.
R. M. Simranjot Kaur, “Sentiment Analysis on twitter data using Machine Learning,” J. Xidian Univ., vol. 14, no. 12, Dec. 2020, doi: 10.37896/jxu14.12/039.
M. T. Akter, M. Begum, and R. Mustafa, “Bengali Sentiment Analysis of E-commerce Product Reviews using K-Nearest Neighbors,” in 2021 International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD), IEEE, Feb. 2021, pp. 40–44. doi: 10.1109/ICICT4SD50815.2021.9396910.
M. Syukron, R. Santoso, and T. Widiharih, “PERBANDINGAN METODE SMOTE RANDOM FOREST DAN SMOTE XGBOOST UNTUK KLASIFIKASI TINGKAT PENYAKIT HEPATITIS C PADA IMBALANCE CLASS DATA,” J. Gaussian, vol. 9, no. 3, pp. 227–236, Aug. 2020, doi: 10.14710/j.gauss.v9i3.28915.
A. S. Aribowo, H. T. Jaya, U. Pembangunan, and N. Veteran, “An Evaluation of Preprocessing Steps and Tree-based Ensemble Machine Learning for Analysing Sentiment on Indonesian YouTube Comments,” Int. J. Adv. Trends Comput. Sci. Eng., vol. 9, no. 5, 2020, doi: 10.30534/ijatcse/2020/29952020.
Downloads
Published
Issue
Section
License
The copyright of the article that accepted for publication shall be assigned to Jurnal Sisfokom (Sistem Informasi dan Komputer) and LPPM ISB Atma Luhur as the publisher of the journal. Copyright includes the right to reproduce and deliver the article in all form and media, including reprints, photographs, microfilms, and any other similar reproductions, as well as translations.
Jurnal Sisfokom (Sistem Informasi dan Komputer), LPPM ISB Atma Luhur, and the Editors make every effort to ensure that no wrong or misleading data, opinions or statements be published in the journal. In any way, the contents of the articles and advertisements published in Jurnal Sisfokom (Sistem Informasi dan Komputer) are the sole and exclusive responsibility of their respective authors.
Jurnal Sisfokom (Sistem Informasi dan Komputer) has full publishing rights to the published articles. Authors are allowed to distribute articles that have been published by sharing the link or DOI of the article. Authors are allowed to use their articles for legal purposes deemed necessary without the written permission of the journal with the initial publication notification from the Jurnal Sisfokom (Sistem Informasi dan Komputer).
The Copyright Transfer Form can be downloaded [Copyright Transfer Form Jurnal Sisfokom (Sistem Informasi dan Komputer).
This agreement is to be signed by at least one of the authors who have obtained the assent of the co-author(s). After submission of this agreement signed by the corresponding author, changes of authorship or in the order of the authors listed will not be accepted. The copyright form should be signed originally, and send it to the Editorial in the form of scanned document to sisfokom@atmaluhur.ac.id.
