Enhancing BERT for Fake News Classification Considering News Body: Segment-based Feature Extraction with Custom Task-Specific Layer
Keywords:
BERT, Transformer, Long document processing, fake news , machine learning(ML), deep learning(DL), fake news body , natural language processing(NLP) ,LSTM.Abstract
Fake news dissemination on social media is a major concern today. Social media fake news is mostly multimodal in nature, where text forms the major component. People tend to believe whatever they read. Such news is a deception, impressively written with dishonest intents, and creates an impact on its readers, compelling them to like and further share it. Existing works mainly use the title of the news to extract features such as linguistic cues and psychological features, text semantics. These features can be extracted via different machine learning (ML) and deep learning models (DL). The most popular ones are language models in NLP. This models are trained on a large corpus like Wikipedia, which makes them capable of learning the complexities of language and capturing rich context information. For news titles, such models provide good performance for a given dataset but fail to generalize, as news titles are short providing less information to learn. Hence, processing news bodies is equally important to get state of art results. However, because of structural limitations, this architecture cannot be applied to lengthy news bodies. For example, a Language model like BERT is capable of processing limited text i.e. 512 sub-words.
In Natural Language Processing (NLP) using language models for feature extraction is a commonly adopted procedure. In this work, we try to address the problem of long document processing with respect to news bodies. We first divide the lengthy news body into overlapping segments with a maximum size of 200 words. A pre-trained BERT is used to learn local semantic contextual embeddings of the segment. All the segments of a news body are then passed to a custom-designed task specific layer to capture global contextual embeddings of the news body. We validate the efficiency of the proposed architecture on two real-world datasets. A comparative analysis in terms of various performance metrics is presented.
Downloads
References
J. Xue, X. Tang and L. Zheng, "A Hierarchical BERT-Based Transfer Learning Approach for Multi-Dimensional Essay Scoring," in IEEE Access, vol. 9, pp. 125403-125415, 2021, doi: 10.1109/ACCESS.2021.3110683.
Lu, J., Henchion, M., Bacher, I., Namee, B.M. (2021). A Sentence-Level Hierarchical BERT Model for Document Classification with Limited Labelled Data. In: Soares, C., Torgo, L. (eds) Discovery Science. DS 2021. Lecture Notes in Computer Science(), vol 12986. Springer, Cham. https://doi.org/10.1007/978-3-030-88942-5_18
Nishant Rai, Deepika Kumar, Naman Kaushik, Chandan Raj, Ahad Ali, “Fake News Classification using transformer based enhanced LSTM and BERT”, International Journal of Cognitive Computing in Engineering, Volume 3, 2022, Pages 98-105, ISSN 2666-3074, https://doi.org/10.1016/j.ijcce.2022.03.003.
Farokhian, M., Rafe, V., & Veisi, H. (2022). Fake news detection using parallel BERT deep neural networks. ArXiv, abs/2204.04793.
Pappagari, R., Żelasko, P., Villalba, J., Carmiel, Y., & Dehak, N. (2019). Hierarchical Transformers for Long Document Classification. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 838-844.
S. Singh and A. Mahmood, "The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures," in IEEE Access, vol. 9, pp. 68675-68702, 2021, doi: 10.1109/ACCESS.2021.3077350.
Bharadwaj, Pranav Shao, Zongru Darren, Seth. (2019). FAKE NEWS DETECTION WITH SEMANTIC FEATURES AND TEXT MINING. Journal of Natural Language Processing. 8. 17-22.
S. Girgis, E. Amer and M. Gadallah, ”Deep Learning Algorithms for Detecting Fake News in Online Text,” 2018 13th International Conference on Computer Engineering and Systems (ICCES), 2018, pp. 93-97, doi: 10.1109/IC17 CES.2018.8639198.
O. Ajao, D. Bhowmik, and S. Zargari. 2018. Fake News Identification on Twitter with Hybrid CNN and RNN Models. In Proceedings of the 9th International Conference on Social Media and Society (SMSociety ’18). Association for Computing Machinery, New York, NY, USA, 226–230. DOI:https://doi.org/10.1145/3217804.3217917
M. Dong, L. Yao, X. Wang, B. Benatallah, X. Zhang and Q. Z. Sheng, ”Dual-stream Self-Attentive Random Forest for False Information Detection,” 2019 International Joint Conference on Neural Networks (IJCNN), 2019, pp. 1-8, doi: 10.1109/IJCNN.2019.8851765
R. Dziseviˇc and D. Seˇsok, ”Text Classification using Different Feature Extraction Approaches,” 2019 Open Conference of Electrical, Electronic and Information Sciences (eStream), 2019, pp. 1-4, doi: 10.1109/eStream.2019.8732167.
Pinjarkar V., Jain A., “Analysis of Classification Rules Optimization Using Genetic Algorithm”, Journal of Design Engineering, volume 2021, issue 9, pp: 4096-4107, 2021.
BERT:How to Handle Long Documents [Online]. Available: https://www.saltdatalabs.com/blog/bert-how-to-handle-long-documents Accessed on: September 15,2023.
Romano, A. (2018). Mark zuckerberg lays out facebooks 3-pronged approach to fake news.
Fake News: Categories. [Online]. Available: https://library-nd.libguides.com/fakenews/categories Accessed on: September 29,2023.
Learning After Overfitting, 06 April, 2022. [Online]. Available: https://www.deeplearning.ai/the-batch/learning-after-overfitting/ Accessed on: October 10,2023.
Fake News Dataset, [Online]. Available: https://www.kaggle.com/search?q=fake+news+dataset Accessed on: August 25,2023.
ISOT Fake News Dataset [Online]. Available: https://www.kaggle.com/datasets/emineyetm/fake-news-detection-datasets Accessed on: August 30, 2023.
Assam Lynching case, 18 June,2018 [Online]. Available: https://timesofindia.indiatimes.com/india/assam-lynching-case-16-arrested-search-on-for-remaining-accused/articleshow/64533326.cms Accessed on 21 July ,2023.
Dr. Niraj Kumar, Deep Learning, [Online]. Available: https://ai-leader.com/deep-learning/ Accessed on: 10, July,2023.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
