Performance analysis of MT tools through Hindi to English web query translation
Keywords:
machine translation, web query translation, information retrieval, online translators, translation qualityAbstract
Query translation plays crucial and important role in cross-lingual information retrieval (CLIR) systems, where retrieval efficiency is closely tied with translation accuracy. Indian languages require an effective and efficient machine translation (MT) tool to effectively transform query intent into other language. As machine translation becomes increasingly prevalent in the translation industry, understanding its quality is gaining greater importance. However, the focus on the acceptance of MT output based on performance, and more importantly, how acceptable it is to human translators, has been relatively limited. The complexity of MT arises from factors such as words with multiple meanings, sentences with various interpretations, and differing grammatical structures across languages. This complexity is further intensified by the lack of structural constraints and the presence of ambiguity, particularly in the case of web queries. The goal of this work is to evaluate the accuracy of free online MT tools in translating Hindi web queries. The accuracy has been measured using several metrics, including METEOR, BLEU, NIST, hLEPOR, CHRF, and GLEU. Our results show that translation accuracy is higher for longer queries compared to shorter ones. Among the translators tested, Google Translate performed the best, while Systran performed the worst, with a performance gap of more than 42% between the two. The present research work assesses the performance and effectiveness of the popular MT tools for Hindi to English query translation
Downloads
References
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318.
S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72.
M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul, “A study of translation edit rate with targeted human annotation,” in Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, 2006, pp. 223–231.
E. M. Visser and M. Fuji, “Using sentence connectors for evaluating MT output,” arXiv preprint cmp-lg/9608019, 1996.
C. Callison-Burch, “Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk,” in Proceedings of the 2009 conference on empirical methods in natural language processing, 2009, pp. 286–295.
C. Callison-Burch, P. Koehn, C. Monz, K. Peterson, M. Przybocki, and O. Zaidan, “Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation,” in Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, 2010, pp. 17–53.
P. Koehn et al., “Moses: Open source toolkit for statistical machine translation,” in Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, 2007, pp. 177–180.
D. Chiang, S. DeNeefe, Y. S. Chan, and H. T. Ng, “Decomposability of translation metrics for improved evaluation and efficient algorithms,” in Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 2008, pp. 610–619.
R. Rei, C. Stewart, A. C. Farinha, and A. Lavie, “COMET: A neural framework for MT evaluation,” arXiv preprint arXiv:2009.09025, 2020.
E. Fonseca, L. Yankovskaya, A. F. T. Martins, M. Fishel, and C. Federmann, “Findings of the WMT 2019 shared tasks on quality estimation,” in Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), 2019, pp. 1–10.
S. Chauhan, S. Saxena, and P. Daniel, “Monolingual and parallel corpora for Kangri low resource language,” arXiv preprint arXiv:2103.11596, 2021.
S. Chauhan, P. Daniel, A. Mishra, and A. Kumar, “Adableu: A modified bleu score for morphologically rich languages,” IETE J Res, vol. 69, no. 8, pp. 5112–5123, 2023.
G. Doddington, “Automatic evaluation of machine translation quality using n-gram co-occurrence statistics,” in Proceedings of the second international conference on Human Language Technology Research, 2002, pp. 138–145.
A. L. F. Han, D. F. Wong, and L. S. Chao, “LEPOR: A robust evaluation metric for machine translation with augmented factors,” in Proceedings of COLING 2012: Posters, 2012, pp. 441–450.
Y. Wu, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016.
P. A. Watters and M. Patel, “Semantic processing performance of Internet machine translation systems,” Internet Research, vol. 9, no. 2, pp. 153–160, 1999.
P. Boonkwan and A. Kawtrakul, “Plaesarn: machine-aided translation tool for English-to-Thai,” in COLING-02: Machine Translation in Asia, 2002.
O. Bezhanova, M. Byezhanova, and O. Landry, “Comparative analysis of the translation quality produced by three MT systems,” McGill University, Montreal, Canada, 2005.
M. Aiken and Z. Wong, “Spanish-to-English translation using the Web,” Southwest Decision Sciences Institute, 2006.
M. Aiken, K. Ghosh, J. Wee, and M. Vanjani, “An evaluation of online Spanish and German translation accuracy,” Communications of the IIMA, vol. 9, no. 4, pp. 67–84, 2009.
S. Hampshire and C. P. Salvia, “Translation and the Internet: evaluating the quality of free online machine translators,” Quaderns: revista de traducció, pp. 197–209, 2010.
A. Guerberof Arenas, “Exploring Machine Translation on the Web,” revistatradumática, no. 8, pp. 1–6, 2010.
R. de Oliveira and D. Anastasiou, “Comparison of SYSTRAN and Google Translate for English→ Portuguese,” Tradumática, no. 9, pp. 118–136, 2011.
H. Li, A. C. Graesser, and Z. Cai, “Comparison of Google translation with human translation,” in the twenty-seventh international flairs conference, 2014.
S. Chand, “Empirical survey of machine translation tools,” in 2016 Second International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN), 2016, pp. 181–185.
I. Sutskever, “Sequence to Sequence Learning with Neural Networks,” arXiv preprint arXiv:1409.3215, 2014.
C. Chandler, P. W. Foltz, and B. Elvevåg, “Improving the applicability of AI for psychiatric applications through human-in-the-loop methodologies,” Schizophr Bull, vol. 48, no. 5, pp. 949–957, 2022.
A. S. Utami, N. Yusliani, M. D. Marieska, and A. Abdiansah, “Query Reformulation for Indonesian Question Answering System Using Word Embedding of Word2Vec,” Computer Engineering and Applications Journal, vol. 11, no. 1, pp. 1–14, 2022.
Y. Mao, C. Wan, Y. Jiang, and X. Gu, “Self-supervised query reformulation for code search,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 363–374.
S. Agrawal, S. Merugu, and V. Sembium, “Enhancing e-commerce product search through reinforcement learning-powered query reformulation,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 4488–4494.
C. Callison-Burch, P. Koehn, C. Monz, and O. Zaidan, “Findings of the 2011 workshop on statistical machine translation,” in Proceedings of the sixth workshop on statistical machine translation, 2011, pp. 22–64.
A. Agarwal and A. Lavie, “Meteor, m-bleu and m-ter: Evaluation metrics for high-correlation with human rankings of machine translation output,” in Proceedings of the Third Workshop on Statistical Machine Translation, 2008, pp. 115–118.
D. Zhou, M. Truran, T. Brailsford, V. Wade, and H. Ashman, “Translation techniques in cross-language information retrieval,” ACM Computing Surveys (CSUR), vol. 45, no. 1, pp. 1–44, 2012.
E. Hovy, M. King, and A. Popescu-Belis, “Principles of context-based machine translation evaluation,” Machine Translation, vol. 17, pp. 43–75, 2002.
Amit Asthana, Ganesh Chandra, Sanjay K. Dwivedi, “Approach to Handle Compound Out of Vocabulary Words in Hindi Web Queries”, Journal of Information Systems Engineering and Management, Vol. 10No.13s (2025), https://doi.org/10.52783/jisem.v10i13s.2158
Asthana Amit, and Sanjay K. Dwivedi. "Exploring snippets as a dataset to overcome challenges in CLIR." ITM web of conferences. Vol. 54. EDP Sciences, 2023.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Amit Asthana, Sanjay K. Dwivedi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
