Scalable Query Processing in Big Data Environments With Hadoop Distributed File System and Particle Swarm Optimization

Abstract

Big data are enormous, intricate sets of information that are too large, too quick, and too difficult to handle using conventional methods. They consist of both structured and unstructured data. Due to the alarmingly high rates of data generation, there is pressure to adopt simple and expensive procedures for data recovery and storage. Conventional database systems cannot handle the processing power of big data. In order to evaluate and query this enormous amount of data, new technologies have recently been created. In this work, a novel hybrid approach for efficient big data query processing through a two-phase approach is introduced. In the first phase, the Hadoop Distributed File System (HDFS), Map Reduce are utilized. Second phase involves optimization using Particle Swarm Optimization (PSO). The proposed algorithm aims to enhance search speed by strategically optimizing various processes. This two-step methodology combines the parallel processing capabilities of HDFS Map-Reduce with the optimization power of PSO, contributing to an overall improvement in query processing efficiency for large-scale datasets. Experimental results demonstrate significant outperformance of the model in expediting query processing.