Cascaded Code Distributed Computing With Low Complexity and Improved Flexibility
share
Coded distributed computing, proposed by Li et al., offers significant potential for reducing the communication load in MapReduce computing systems. In the setting of the cascaded coded distributed computing that consisting of K nodes, N input files, and Q output functions, the objective is to compute each output function through s≥ 1 nodes with a computation load r≥ 1, enabling the application of coding techniques during the Shuffle phase to achieve minimum communication load. However, for most existing coded distributed computing schemes, a major limitation lies in their demand for splitting the original data into an exponentially growing number of input files in terms of N/Kr∈ℕ and requiring an exponentially large number of output functions Q/Ks∈ℕ, which imposes stringent requirements for implementation and results in significant coding complexity when K is large. In this paper, we focus on the cascaded case of K/s∈ℕ, deliberately designing the strategy of input files store and output functions assignment based on a grouping method, such that a low-complexity two-round Shuffle phase is available. The main advantages of our proposed scheme contains: 1) the communication load is quilt close to or surprisingly better than the optimal state-of-the-art scheme proposed by Li et al.; 2) our scheme requires significantly less number of input files and output functions; 3) all the operations are implemented over the minimum binary field 𝔽_2.
READ FULL TEXT