Optimal Resource Allocation for Machine Learning Tasks in Distributed Computing Environments

Abstract

This paper considers resource allocation problems in distributed computing environments for machine learning tasks based on mathematical programming and greedy algorithms. We implement a distributed computing platform based on Docker, Kubernetes, and Rancher, mainly for machine learning applications. The simulation results show how the prediction quality of the computation time of machine learning tasks affects scheduling. We also verify our approach by evaluating real machine learning tasks for predicting the backbone structure of proteins.