Optimization of data segments and number of cores for defining popularity of kazakh words using apache spark

Authors

  • М. Meraliyev Suleyman Demirel University
  • K. Orynbekova Suleyman Demirel University
  • А. Talasbek Suleyman Demirel University
  • D. Kariboz Suleyman Demirel University
  • A. Issabek Suleyman Demirel University

DOI:

https://doi.org/10.51301/vest.su.2021.i3.06

Keywords:

Apache Spark, RDD, data partitions, NLP, MapReduce paradigm.

Abstract

Kazakh is an agglutinative language which has complex structure. In this work Apache Spark was used to specify the popularity of Kazakh words in 3 popular kazakh compositions. The main goal was to find the optimal number of data segments for a specific number of cores in order to find the best computational speed. To do so, the data was divided into several segments and ran on a cluster with a different number of cores each time. Results show that the amount of data segments directly affects the computing speed.

Downloads

Published

2021-06-30

How to Cite

Meraliyev М. . . . . ., Orynbekova, K. . . . . . ., Talasbek А. . . . . . . . . . . . . ., Kariboz, D. . . ., & Issabek, A. . . . . . . . . . (2021). Optimization of data segments and number of cores for defining popularity of kazakh words using apache spark. Engineering Journal of Satbayev University, 143(3), 39–42. https://doi.org/10.51301/vest.su.2021.i3.06

Issue

Vol. 143 No. 3 (2021): Vestnik KazNRTU

Section

Physics and Mathematics

License

Copyright (c) 2021 VESTNIK KAZNRTU

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

<div class="pkpfooter-son">
<a rel="license" href="http://creativecommons.org/licenses/by-nc/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc/4.0/80x15.png"></a><br>This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 International License</a>.
</div>