Research on Efficiency Optimization of Data Mining Algorithms in Big Data Environments
Abstract
This paper investigates the efficiency optimization of data mining algorithms in big data environments.We validated our optimization scheme for two algorithms (K-means and FP-Growth) through experiments.Our experiments show that on the large dataset of scale tens of millions, the optimized scheme has 3~5 times faster than original one, saves 40% memory space,and shows nearly linear speedup with an increasing number of nodes – i.e., it is scalable.The results indicate that data mining algorithms in Big Data environment can be used for application in engineering area.
Keywords
Big data; Data mining; Algorithm optimization; Parallel computing; Spark; K-means; FP-Growth
Full Text:
PDFReferences
[1] Wu L. Research on frequent pattern mining algorithms in big data environments [D]. Guangdong University of Technology, 2019. DOI:10.27029/d.cnki.ggdgu.2019.000405.
[2] Liu W. Research on Network Performance Analysis and Optimization Algorithms in the Big Data Environment [J]. Science and Technology Economic Market, 2023, (9): 49-51.
[3] Cao T X, Zhu J S, Xiao H J. Research on Data Mining Recommendation Algorithms Based on User Online Behaviors in the Big Data Environment [C]//Proceedings of the 2019 Yunnan Power Technology Forum, 2019: 2-2.
DOI: http://dx.doi.org/10.70711/cle.v2i10.9811
Refbacks
- There are currently no refbacks.
