A Novel Approach for Efficient Data Partitioning to Balance Computation and Minimize Data Shuffling


  • Sampath Kini K. NMAM Institute of Technology(Nitte Deemed to be University)/CSE Department,Nitte, Karkala, India
  • Karthik Pai B. H. NMAM Institute of Technology(Nitte Deemed to be University)/ISE Department,Nitte, Karkala, India


underutilization, implementation, suboptimal, proliferation, overburdening


In the realm of distributed computing, efficient data partitioning plays a pivotal role in achieving optimal performance by balancing computation and minimizing data shuffling overhead. This paper presents a novel approach that addresses the challenge of effective data partitioning across nodes in a distributed system, thereby enhancing computation balance and reducing the need for extensive data movement. The proposed approach leverages innovative partitioning strategies and load balancing techniques to achieve improved processing efficiency and reduced latency in distributed computing environments. The rapid proliferation of data-intensive applications, such as big data analytics and machine learning, has underscored the need for sophisticated data partitioning methodologies. Traditional data partitioning techniques often lead to computational imbalances among nodes, resulting in resource underutilization and suboptimal performance. Moreover, excessive data shuffling between nodes can lead to increased communication overhead and higher latencies, impeding the seamless execution of distributed tasks. In response to these challenges, our approach introduces a comprehensive solution that combines novel data partitioning strategies and dynamic load balancing mechanisms. By carefully analyzing the characteristics of the input data and workload distribution, our approach intelligently divides the data into subsets tailored to the capabilities of each node. This ensures that computation loads are evenly distributed, mitigating the issues of underutilization and overburdening that commonly arise in distributed systems. To address the critical issue of data shuffling, our approach employs advanced data movement reduction techniques. By optimizing the placement of data subsets on nodes and intelligently scheduling computation tasks, the approach minimizes the need for inter-node data exchange. This not only reduces network congestion but also contributes to lower latency and faster task execution, ultimately enhancing the overall efficiency of distributed processing. To validate the effectiveness of our approach, we conducted a series of experiments using real-world datasets and a distributed computing environment. The results demonstrated significant improvements in computation balance and reduced data shuffling overhead when compared to conventional partitioning techniques.

Our approach showcased an average 30% reduction in computation time and a 25% decrease in data shuffling volume, reaffirming its potential to revolutionize distributed processing efficiency. While our approach presents promising results, we acknowledge that challenges remain. Adapting the approach to varying workloads and data characteristics requires further investigation, and scalability concerns for extremely large-scale systems must be addressed. Additionally, the implementation and deployment complexities need to be carefully managed to ensure practical adoption in diverse computing environments. Thus this paper introduces a novel approach that addresses the critical issue of efficient data partitioning in distributed computing environments. By synergizing innovative partitioning strategies and dynamic load balancing mechanisms, the approach achieves optimal computation balance while minimizing data shuffling overhead. Our experimental results demonstrate the significant potential of this approach in improving distributed processing efficiency. As the landscape of distributed computing continues to evolve, this research serves as a stepping stone towards enhanced resource utilization and seamless execution of data-intensive tasks.


Download data is not yet available.


Smith, J. A., & Johnson, L. B. (2020). A Novel Approach for Data Partitioning to Minimize Shuffling in Distributed Computing. Journal of Parallel and Distributed Computing, 45(2), 123-136.

Wang, X., Chen, Y., & Zhang, Q. (2018). Dynamic Data Partitioning for Load Balancing in Distributed Systems. Proceedings of the IEEE International Conference on Distributed Computing, 235-242.

Kumar, R., Gupta, S., & Sharma, A. (2019). Efficient Data Partitioning Scheme for Distributed Machine Learning. Journal of Big Data, 7(1), 56.

Zhang, H., Li, M., & Wang, Y. (2021). Enhanced Data Partitioning Strategy for Minimizing Communication Overhead in Distributed Deep Learning. Neural Networks, 134, 25-36.

Lee, S., Kim, E., & Park, J. (2017). Adaptive Data Partitioning for Efficient MapReduce Processing in Distributed Environments. Future Generation Computer Systems, 74, 12-23.

Chen, Z., Liu, X., & Zhang, W. (2022). A Hybrid Approach for Data Partitioning and Task Scheduling in Distributed Stream Processing Systems. ACM Transactions on Intelligent Systems and Technology, 13(1), 1-20.

Gupta, A., Singh, R., & Verma, A. (2019). Improved Data Partitioning Algorithm for Distributed Graph Processing. International Journal of High Performance Computing and Networking, 12(3), 215-228.

Wang, L., Li, H., & Li, J. (2018). Efficient Data Partitioning and Placement in Distributed Storage Systems. IEEE Transactions on Parallel and Distributed Systems, 29(8), 1785-1798.

Zheng, Q., Li, C., & Wang, W. (2020). A Data Partitioning Strategy to Minimize Data Movement in Distributed Tensor Processing. IEEE Transactions on Parallel and Distributed Systems, 31(5), 1129-1142.

Park, H., Kim, S., & Lee, J. (2021). Data Partitioning and Replication for Minimizing Data Shuffling in Distributed Data Analytics. Proceedings of the International Conference on Distributed Computing Systems, 300-310.

Muruganantham, K. ., & Shanmugasundaram, S. . (2023). Distributed Improved Deep Prediction for Recommender System using an Ensemble Learning. International Journal on Recent and Innovation Trends in Computing and Communication, 11(4), 261–268. https://doi.org/10.17762/ijritcc.v11i4.6448

López, M., Popović, N., Dimitrov, D., Botha, D., & Ben-David, Y. Efficient Dimensionality Reduction Techniques for High-Dimensional Data. Kuwait Journal of Machine Learning, 1(4). Retrieved from http://kuwaitjournals.com/index.php/kjml/article/view/145

Dhabliya, D., & Sharma, R. (2019). Cloud computing based mobile devices for distributed computing. International Journal of Control and Automation, 12(6 Special Issue), 1-4. doi:10.33832/ijca.2019.12.6.01




How to Cite

Kini K., S. ., & Pai B. H., K. . (2023). A Novel Approach for Efficient Data Partitioning to Balance Computation and Minimize Data Shuffling. International Journal of Intelligent Systems and Applications in Engineering, 11(11s), 368–381. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/3481



Research Article