CN111444148A - Data transmission method and device based on MapReduce - Google Patents

Abstract

The embodiments of the present application disclose a MapReduce-based data transmission method and device. A specific implementation of the method includes: executing a Map task to generate a calculation result file, wherein the calculation result file includes the same number of partitions as the Reduce side and their corresponding data; uploading the calculation result file to provide redundancy The stored target file system, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system, wherein the target file system names the calculation result file according to a predetermined naming rule, and according to a predetermined directory structure The calculation result file is stored. This embodiment avoids computing resource consumption and time consumption caused by recalculation, improves the stability of the Shuffle process, and has better universality.

Description

MapReduce-based data transmission method and device

technical field

The embodiments of the present application relate to the field of computer technologies, and in particular, to a MapReduce-based data transmission method and apparatus.

Background technique

With the rapid development of computer technology, MapReduce distributed computing framework has been widely used. Between the Map (mapping) and Reduce (reduction) processes, Shuffle (shuffling) is required to realize the transmission of data from the output of the Map task (Task) to the input of the Reduce task. Since the Shuffle operation is an indispensable bridge between the Map process and the Reduce process, and is often accompanied by a large number of network transfers and disk reads and writes, the performance of Shuffle often directly affects the performance and throughput of the entire MapReduce process.

In practice, node failures, network delays, and high cluster load may cause data transmission to time out during the shuffle process. As a result, the Reducer cannot obtain the required data from the Mapper, and the shuffle process fails. A related method is usually to use the fault-tolerant mechanism provided by MapReduce itself to perform recalculation on tasks that fail Shuffle (that is, select some input data to re-execute the Map process and then continue the previously interrupted Reduce process).

SUMMARY OF THE INVENTION

The embodiments of the present application propose a MapReduce-based data transmission method and device.

In a first aspect, an embodiment of the present application provides a MapReduce-based data transmission method, which is applied to the Map side. The method includes: executing a Map task to generate a calculation result file, wherein the calculation result file includes the same number as the Reduce side. The partition and its corresponding data; upload the calculation result file to the target file system that provides redundant storage, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system, wherein the target file system is named according to the predetermined name The rules name the calculation result files, and store the calculation result files according to a predetermined directory structure.

In some embodiments, the above-mentioned calculation result file includes an index file and a data file, the above-mentioned data file is used to record data, and the above-mentioned index file is used to identify the starting position and ending position of the data of each partition in the data file; and the method The method also includes: sending the metadata information of the calculation result file to the task scheduler (MapReduce task scheduler), wherein the metadata information includes the corresponding relationship between the calculation result file and the Map side.

In some embodiments, the above-mentioned predetermined naming rule includes that the name of the calculation result file includes a corresponding map terminal identifier, and distinguishes the data file and the index file of the calculation result file according to the suffix.

In some embodiments, the predetermined directory structure includes a tree structure, and the tree structure includes, from top to bottom, an identifier of an application, an identifier of a MapReduce process belonging to an application, an identifier of a Map task belonging to a MapReduce process, and a calculation belonging to a Map task. The ID of the result file.

In a second aspect, an embodiment of the present application provides a MapReduce-based data transmission method, which is applied to the Reduce side. The method includes: in response to determining that the acquisition of data from the Map side corresponding to the Reduce side fails, from the target providing redundant storage The file system obtains the data of the partition corresponding to the Reduce side, wherein the target file system stores the calculation result file according to a predetermined directory structure, and the calculation result file includes the same number of partitions as the Reduce side and their corresponding data; using the obtained data, Execute a Reduce task to generate the final result of the MapReduce process.

In some embodiments, before acquiring the data of the partition corresponding to the Reduce side from the file system, the method further includes: acquiring the address of at least one Map side corresponding to the Reduce side from the task scheduling side, wherein the task scheduling side stores a Metadata information of the calculation result file, the metadata information includes the corresponding relationship between the calculation result file and the Map side; according to the address of the at least one Map side, a data acquisition request for acquiring the data of the partition corresponding to the Reduce side is sent to the at least one Map side.

In a third aspect, an embodiment of the present application provides a MapReduce-based data transmission device, which is applied to the Map side. The device includes: a first generating unit configured to execute a Map task to generate a calculation result file, wherein the calculation result The file includes the same number of partitions as the Reduce side and its corresponding data; the uploading unit is configured to upload the calculation result file to the target file system that provides redundant storage, so that the corresponding Reduce side can obtain the calculation result through the target file system The data in the file, wherein the target file system names the calculation result file according to a predetermined naming rule, and stores the calculation result file according to a predetermined directory structure.

In some embodiments, the above-mentioned calculation result file includes an index file and a data file, the above-mentioned data file is used to record data, and the above-mentioned index file is used to identify the starting position and ending position of the data of each partition in the data file; and the device It also includes: a sending unit configured to send the metadata information of the calculation result file to the task scheduling terminal, wherein the metadata information includes the corresponding relationship between the calculation result file and the Map terminal.

In some embodiments, the above-mentioned predetermined naming rule includes that the name of the calculation result file includes a corresponding map terminal identifier, and distinguishes the data file and the index file of the calculation result file according to the suffix.

In some embodiments, the predetermined directory structure includes a tree structure, and the tree structure includes, from top to bottom, an identifier of an application, an identifier of a MapReduce process belonging to an application, an identifier of a Map task belonging to a MapReduce process, and a calculation belonging to a Map task. The ID of the result file.

In a fourth aspect, an embodiment of the present application provides a MapReduce-based data transmission device, which is applied to the Reduce side. The device includes: a first acquisition unit, configured to respond to determining that acquisition of data from the Map side corresponding to the Reduce side fails. , obtain the data of the partition corresponding to the Reduce side from the target file system that provides redundant storage, wherein the target file system stores the calculation result file according to a predetermined directory structure, and the calculation result file includes the same number of partitions as the Reduce side and their corresponding partitions. data; a second generating unit configured to utilize the acquired data to execute a Reduce task to generate the final result of the MapReduce process.

In some embodiments, the apparatus further includes: an address obtaining unit, configured to obtain the address of at least one Map end corresponding to the Reduce end from the task scheduling end, wherein the task scheduling end stores metadata information of the calculation result file, The data information includes the corresponding relationship between the calculation result file and the Map side; the second acquisition unit is configured to send a data acquisition request for acquiring the data of the partition corresponding to the Reduce side to the at least one Map side according to the address of the at least one Map side.

In a fifth aspect, an embodiment of the present application provides a MapReduce-based data transmission system, the system includes: a Map end, configured to execute the method described in any of the implementation manners in the first aspect; and a Reduce end, configured to Execution implements the method described in any implementation manner of the second aspect; the target file system is configured to, in response to receiving the calculation result file, name the calculation result file according to a predetermined naming rule, and name the calculation result file according to a predetermined directory structure file is stored.

In some embodiments, the target file system described above is further configured to store multiple copies of the calculation result file in the data nodes of the target file system.

In a sixth aspect, an embodiment of the present application provides an electronic device, the electronic device includes: one or more processors; a storage device on which one or more programs are stored; when one or more programs are stored by one or more The multiple processors execute such that the one or more processors implement a method as described in any one of the implementations of the first aspect.

In a seventh aspect, an embodiment of the present application provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, implements the method described in any implementation manner of the first aspect.

In the MapReduce-based data transmission method and device provided by the embodiments of the present application, a Map task is first executed to generate a calculation result file. The above calculation result file includes the same number of partitions and their corresponding data as the Reduce side. Then, the calculation result file is uploaded to the target file system that provides redundant storage, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system. The target file system names the calculation result file according to a predetermined naming rule, and stores the calculation result file according to a predetermined directory structure. In this way, the backup of the calculation result file is realized through the target file system, and an alternate data source is provided for the Reduce side to obtain the shuffle failure. This avoids the consumption of computing resources and time costs caused by recalculation, and improves the stability of the Shuffle process. Moreover, since the entire MapReduce model has not been changed, it is less intrusive to the computing framework and has good universality.

Description of drawings

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 is an exemplary system architecture diagram to which an embodiment of the present application may be applied;

2 is a flowchart of an embodiment of a MapReduce-based data transmission method according to the present application;

3 is a schematic diagram of a directory structure of a MapReduce-based data transmission method according to an embodiment of the present application;

FIG. 4 is a flowchart of another embodiment of a MapReduce-based data transmission method according to the present application;

5 is a schematic structural diagram of an embodiment of a MapReduce-based data transmission device according to the present application;

6 is a schematic structural diagram of an embodiment of a MapReduce-based data transmission device according to the present application;

7a is a sequence diagram of interaction between various devices in an embodiment of MapReduce-based data transmission according to the present application;

7b is a schematic diagram of interaction between various devices in an embodiment of MapReduce-based data transmission according to the present application;

FIG. 8 is a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application.

Detailed ways

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

FIG. 1 shows an exemplary architecture 100 to which the MapReduce-based data transmission method or the MapReduce-based data transmission apparatus of the present application may be applied.

As shown in FIG. 1 , the system architecture 100 may include terminal devices 101 , 102 , and 103 , a network 104 , and servers 1051 , 1052 , 1053 , 1054 , 1055 , and 1056 . The network 104 is a medium used to provide a communication link between the terminal devices 101 , 102 , 103 and the server 1051 . The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The terminal devices 101, 102, and 103 interact with the server 1051 through the network 104 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 101 , 102 and 103 , such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that have a display screen and support the display of calculation results, including but not limited to smart phones, tablet computers, e-book readers, laptop computers and desktops computer, etc. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. It can be implemented as a plurality of software or software modules (eg, software or software modules for providing distributed services), or can be implemented as a single software or software module. There is no specific limitation here.

The server 1051 may be a server that provides various services, such as a background server that provides support for applications running on the terminal devices 101 , 102 , and 103 . The background server can decompose the received computing task and send it to the computing node for computing to generate the computing result, and feed the computing result back to the terminal device. Specifically, the server 1051 (eg, the MRAppMaster node) can assign the Map task to the Map nodes 1052 and 1053 for execution. The Map node may upload the intermediate result file generated by executing the Map task to the file server 1054 . The Reduce nodes 1055 and 1056 can obtain intermediate result files from the Map nodes 1052 and 1053 or the file server 1054 to perform the Reduce tasks and generate calculation results.

It should be noted that the server may be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or can be implemented as a single server. When the server is software, it can be implemented as a plurality of software or software modules (for example, software or software modules for providing distributed services), or can be implemented as a single software or software module. There is no specific limitation here.

It should be noted that the MapReduce-based data transmission method provided by the embodiments of the present application is generally executed by the servers 1052, 1053 or 1055, 1056, and correspondingly, the MapReduce-based data transmission device is generally set on the servers 1052, 1053 or 1055, 1056 middle.

It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Continuing to refer to FIG. 2 , a flow 200 of an embodiment of the MapReduce-based data transmission method according to the present application is shown. The MapReduce-based data transmission method can be applied to the Map side, which can include the following steps:

Step 201: Execute a Map task to generate a calculation result file.

In this embodiment, the execution body of the MapReduce-based data transmission method (the server 1052 or 1053 shown in FIG. 1 ) may execute the Map task according to the received data, thereby generating a calculation result file. The above calculation result file may include the same number of partitions and their corresponding data as the Reduce side.

In this embodiment, the above-mentioned execution subject may acquire the Map task in a wired connection manner or a wireless connection manner. For example, the above-mentioned execution body may acquire the Map task from an electronic device (for example, the server 1051 shown in FIG. 1 ) connected to it in communication. The above-mentioned Map task may include data processing logic sent by the user through the terminal device. And, the above-mentioned execution body can read the input data set from the specified position, and process the above-mentioned input data set according to the above-mentioned data processing logic to obtain an intermediate result. Then, the above-mentioned execution body can write the obtained intermediate result into a memory buffer (eg, a ring memory buffer). And, partition sorting (sort) is performed in the process of the above-mentioned writing. In response to determining that the amount of written data reaches a preset threshold (eg, 80%), the execution subject may start an overflow thread to spill the data in the memory buffer to the local disk, thereby generating a temporary file. When the intermediate results corresponding to the above-mentioned Map tasks have been written, the above-mentioned execution body can determine whether there is an overflowed temporary file. In response to determining that it does not exist, the execution body may directly write the data in the memory buffer to the local disk, thereby generating the calculation result file. In response to the determination of existence, the above-mentioned execution body may merge (merge) all the above-mentioned temporary files and data in the memory buffer according to the above-mentioned partitions, that is, merge the data whose partitions are consistent. Then, the execution body may sort the merged data again according to the partition, and write the sorted data to the local disk, thereby generating the calculation result file.

Optionally, the above-mentioned sorting according to the partition may specifically include sorting according to the partition to which the hash value (hash) of the key of the intermediate result belongs. Optionally, the above-mentioned execution body may also combine the above-mentioned key value with the value of the same key.

It should be noted that, in the MapReduce framework, the above-mentioned Map-side reading the input data set can usually be performed by multiple Map-sides in parallel.

In some optional implementations of this embodiment, the above calculation result file may include an index file and a data file. Wherein, the above-mentioned data file can be used for recording data. The above-mentioned index file may be used to identify the start position and end position of the data of each partition in the above-mentioned data file.

Based on the foregoing optional implementation manner, the foregoing index file and data file are generated on the local disk of the foregoing execution body. Therefore, each reducer in MapReduce can more quickly determine the data location of its corresponding partition according to the above index file, thereby improving the efficiency of data query and acquisition.

In some optional implementations of this embodiment, based on the index file and the data file included in the calculation result file, the execution subject may also send metadata information of the calculation result file to the task scheduling end. The above metadata information may include the corresponding relationship between the above calculation result file and the Map terminal. Generally, the above-mentioned task scheduling terminal may be responsible for maintaining the metadata information uploaded by each Map terminal in the above-mentioned MapReduce.

Based on the foregoing optional implementation manner, each Reduce end in the foregoing MapReduce may determine the Map node where the data required by each is located according to the foregoing metadata information.

Step 202: Upload the calculation result file to the target file system that provides redundant storage, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system.

In this embodiment, the above-mentioned execution body may upload the calculation result file generated in the above-mentioned step 201 to the target file system that provides redundant storage through a wired connection or a wireless connection. The above-mentioned target file system for providing redundant storage may name the above-mentioned calculation result file according to a predetermined naming rule, and store the above-mentioned calculation result file according to a predetermined directory structure. Wherein, the above naming rules and directory structures may be any pre-specified rules and target structures. Generally, the above-mentioned target file system providing redundant storage can organize files through the above-mentioned naming rules and directory structure.

In this embodiment, the above-mentioned target file system for providing redundant storage may be any file system that is pre-specified according to actual application requirements and is different from the local disk on the map side. The above-mentioned target file system for providing redundant storage may also be a file system determined according to rules, for example, a file system capable of providing multiple copies, such as HDFS (Hadoop Distributed File System, Hadoop Distributed File System).

In some optional implementations of this embodiment, the above-mentioned predetermined naming rule may include that the name of the above-mentioned calculation result file includes a corresponding map terminal identifier, and the data file and the index file of the above-mentioned calculation result file are distinguished according to the suffix.

As an example, the Map side with the identifiers "0" and "1" uploads the calculation result file to the target file system after writing the calculation result file to the local disk, and the file names correspond to the unique identifier of the Map side. For example, the index file and the data file uploaded by the Map terminal with the identifier "0" stored in the above-mentioned target file system are named "0.index" and "0.data" respectively. The index file and data file uploaded by the Map terminal with the identifier "1" stored in the above target file system are named "1.index" and "1.data" respectively.

Based on the above-mentioned optional implementation manner, the additional overhead caused by maintaining file name information and the mapping relationship between the Map side and the file can be avoided, thereby facilitating the Reduce side to directly access the required files and data.

In some optional implementations of this embodiment, the above-mentioned predetermined directory structure may include a tree structure. Wherein, the above-mentioned tree structure from top to bottom may include the identifier of the application, the identifier of the MapReduce process belonging to the application, the identifier of the Map task belonging to the MapReduce process, and the identifier of the calculation result file belonging to the Map task.

As an example, as shown in FIG. 3 , the first level directory of the above tree structure 300 may be a unique identifier (applicationID) of an application, such as “application_0”, to distinguish data belonging to different applications. The second-level directory of the above-mentioned tree structure 300 may be a unique identifier (shuffleID) of Shuffle in a certain MapReduce process. Since there may be multiple MapReduce processes in a single application, there are multiple Shuffled data. In order to distinguish data belonging to different shuffles, the unique identifiers of the two shuffle processes of "apply_0", such as "shuffle_0" and "shuffle_1", can be used as the second-level directory. The third-level directory of the above-mentioned tree structure 300 may be a unique identifier (MapID) of a certain Map task, such as "map_0" and "map_1. The fourth level of the above-mentioned tree structure 300 is the stored calculation result file. IDs, such as "0.index" and "0.data". Therefore, the data files and index files belonging to the same Map task will be stored under the unique ID of the Map task.

Based on the above-mentioned optional implementation manner, the above-mentioned execution body can avoid additional overhead caused by maintaining metadata information such as file storage paths, thereby improving the query efficiency of the above-mentioned calculation result file in the above-mentioned target file system. Moreover, the Reduce side can directly obtain its storage path according to the required calculation result file, and then directly access the above-mentioned target file system, without first obtaining the file path and other information through other systems.

In some optional implementations of this embodiment, the data nodes of the target file system may also store multiple copies of each calculation result file in a distributed manner, thereby implementing data backup.

At present, one of the existing technologies is usually that only the Map side writes the calculation result file generated by the Map task to the local disk, so that the required data cannot be sent to the corresponding Reduce side in the case of Map node failure, network delay, etc. Causes a recalculation of the MapReduce model. However, in the method provided by the above-mentioned embodiments of the present application, the calculation result file generated by the Map task is written into the local disk by the Map side and then uploaded to the target file system providing redundant storage, so that the target file system providing redundant storage can be used to perform data processing. Backup of the calculation result file to provide an alternate data source for the Reduce side to obtain when Shuffle fails. This avoids the consumption of computing resources and time costs caused by recalculation, and improves the stability of the Shuffle process. Moreover, since the entire MapReduce model has not been changed, it is less intrusive to the computing framework and has good universality.

Continuing to refer to FIG. 4 , FIG. 4 is a flowchart 400 of an embodiment of a MapReduce-based data transmission method according to an embodiment of the present application. The MapReduce-based data transmission method can be applied to the Reduce side, which can include the following steps:

Step 401 , in response to determining that the data acquisition from the Map side corresponding to the Reduce side fails, acquire the data of the partition corresponding to the Reduce side from the target file system providing redundant storage.

In this embodiment, the execution body of the MapReduce-based data transmission method (the server 1055 or 1056 shown in FIG. 1 ) may first determine through various methods that it fails to acquire data from the Map side corresponding to the Reduce side. For example, it can be that the communication between the Reduce side and the Map side times out or the returned content received from the Map side is abnormal. Then, the above-mentioned execution body can obtain the data of the partition corresponding to the Reduce side from the target file system that provides redundant storage. Wherein, the above-mentioned target file system providing redundant storage may store the calculation result file according to a predetermined directory structure. The above calculation result file may include the same number of partitions and their corresponding data as the Reduce side.

It should be noted that, the above-mentioned target file system and calculation result file for providing redundant storage may be consistent with the descriptions in the foregoing embodiments, which will not be repeated here.

Specifically, in response to determining that the acquisition of the data fails, the execution body may acquire the storage path of the calculation result file generated by the Map side according to the above-mentioned generation rule of the directory structure of the target file system providing redundant storage. Then, the above-mentioned execution body may acquire the data of the corresponding partition according to the above-mentioned storage path.

Optionally, based on the index file and data file included in the above-mentioned calculation result file, the above-mentioned execution body can parse the index file in the above-mentioned calculation result file according to the above-mentioned storage path, thereby obtaining the data of the corresponding partition in the data file corresponding to the above-mentioned index file. start and end positions in . Then, the above-mentioned execution body can read the above-mentioned data from the above-mentioned corresponding data file.

In some optional implementation manners of this embodiment, before the foregoing step 401, the foregoing execution subject may further perform the following steps:

The first step is to obtain the address of at least one Map side corresponding to the Reduce side from the task scheduling side.

In these implementation manners, the above-mentioned execution subject may obtain the addresses of each Map terminal in the above-mentioned MapReduce from the task scheduling terminal. The above-mentioned task scheduling terminal may store the metadata information of the above-mentioned calculation result file. The above-mentioned metadata information may include the corresponding relationship between the above-mentioned calculation result file and the Map terminal.

It should be noted that, the above task scheduling terminal may be consistent with the description in the foregoing embodiment, and details are not repeated here.

In the second step, according to the address of the at least one Map side, a data acquisition request for acquiring the data of the partition corresponding to the Reduce side is sent to the at least one Map side.

In these implementation manners, the above-mentioned execution subject may send a data acquisition request to each Map terminal in the above-mentioned MapReduce to acquire the data of the partition corresponding to the Reduce terminal.

It should be noted that, in the MapReduce framework, the above-mentioned Reduce side reads the data belonging to the respective partitions from the Map side or the above-mentioned target file system, which can usually be performed by multiple Reduce sides in parallel.

Step 402, using the acquired data, execute a Reduce task to generate the final result of the MapReduce process.

In this embodiment, the above-mentioned execution body may use the data obtained in step 401 to execute the Reduce task to generate the final result of the MapReduce process. Specifically, the above executive body may first sort and combine the acquired data. Then, the calculation is performed according to the data processing logic indicated by the Reduce task, so as to obtain the above final result.

At present, one of the existing technologies usually recalculates the Map process by the MapReduce model when the Reduce side cannot obtain the required data from the Map side due to the failure of the Map node, network delay, heavy load, etc., resulting in computing resources. waste and time consuming. However, in the method provided by the above-mentioned embodiments of the present application, when the required data cannot be obtained from the Map side, the target file system that provides redundant storage for the calculation result file is accessed instead, thereby avoiding the computational resources caused by recalculation. Consumption and time cost to improve the stability of the Shuffle process. Moreover, since the entire MapReduce model has not been changed, it is less intrusive to the computing framework and has good universality.

With further reference to FIG. 5 , as an implementation of the methods shown in the above figures, the present application provides an embodiment of a MapReduce-based data transmission apparatus, which corresponds to the method embodiment shown in FIG. 2 . Can be used in various electronic devices.

As shown in FIG. 5 , the MapReduce-based data transmission apparatus 500 provided in this embodiment includes a first generating unit 501 and an uploading unit 502 . Wherein, the first generating unit 501 is configured to execute a Map task to generate a calculation result file, wherein the calculation result file includes the same number of partitions as the Reduce side and their corresponding data; the uploading unit 502 is configured to calculate the The result file is uploaded to the target file system that provides redundant storage, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system. A predetermined directory structure stores calculation result files.

In this embodiment, in the data transmission device 500 based on MapReduce: the specific processing of the first generating unit 501 and the uploading unit 502 and the technical effects brought by them may refer to steps 201 and 202 in the corresponding embodiment of FIG. 2 respectively. The related descriptions are not repeated here.

In some optional implementations of this embodiment, the above calculation result file may include an index file and a data file. The above data files can be used to record data. The above-mentioned index file may be used to identify the start position and end position of the data of each partition in the above-mentioned data file. The above-mentioned MapReduce-based data transmission apparatus 500 may further include: a sending unit (not shown in the figure) configured to send the metadata information of the calculation result file to the task scheduling end. The above metadata information may include the corresponding relationship between the calculation result file and the Map terminal.

In some optional implementations of this embodiment, the above-mentioned predetermined naming rules may include that the name of the calculation result file includes a corresponding map identifier, and the data file and the index file of the calculation result file are distinguished according to the suffix.

In some optional implementations of this embodiment, the predetermined directory structure may include a tree structure, and the tree structure includes, from top to bottom, an identifier of an application, an identifier of a MapReduce process belonging to the application, and a Map task belonging to the MapReduce process. , the identifier of the calculation result file belonging to the Map task.

In the apparatus provided by the above embodiments of the present application, the first generating unit 501 executes the Map task to generate a calculation result file. Among them, the calculation result file includes the same number of partitions as the Reduce side and their corresponding data. Then, the uploading unit 502 uploads the calculation result file to the target file system that provides redundant storage, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system. The target file system names the calculation result file according to a predetermined naming rule, and stores the calculation result file according to a predetermined directory structure. In this way, the backup of the calculation result file is realized through the target file system, and an alternate data source is provided for the Reduce side to obtain the shuffle failure. Moreover, it avoids computing resource consumption and time cost caused by recalculation, and improves the stability of the Shuffle process. Moreover, since the entire MapReduce model has not been changed, it is less intrusive to the computing framework and has good universality.

With further reference to FIG. 6 , as an implementation of the methods shown in the above figures, the present application provides an embodiment of a MapReduce-based data transmission apparatus. The apparatus embodiment corresponds to the method embodiment shown in FIG. 4 . Can be used in various electronic devices.

As shown in FIG. 6 , the MapReduce-based data transmission apparatus 600 provided in this embodiment includes a first obtaining unit 601 and a second generating unit 602 . The first obtaining unit 601 is configured to, in response to determining that obtaining data from the Map end corresponding to the Reduce end fails, obtain data of the partition corresponding to the Reduce end from the target file system providing redundant storage, wherein the target file system The calculation result file is stored according to a predetermined directory structure, and the calculation result file includes the same number of partitions as the Reduce side and their corresponding data; the second generation unit 602 is configured to use the acquired data to execute a Reduce task to generate a MapReduce process the final result.

In this embodiment, in the MapReduce-based data transmission device 600: the specific processing of the first obtaining unit 601 and the second generating unit 602 and the technical effects brought by them may refer to steps 401 and 401 in the corresponding embodiment of FIG. 4, respectively. The relevant description of step 402 is not repeated here.

In some optional implementations of this embodiment, the above-mentioned MapReduce-based data transmission apparatus 600 may further include an address obtaining unit (not shown in the figure) and a second obtaining unit (not shown in the figure). The above address obtaining unit may be configured to obtain the address of at least one Map end corresponding to the Reduce end from the task scheduling end. The above task scheduling terminal may store metadata information of the calculation result file. The above-mentioned metadata information may include the corresponding relationship between the calculation result file and the Map terminal. The above-mentioned second obtaining unit may be configured to send a data obtaining request for obtaining the data of the partition corresponding to the Reduce end to the at least one Map end according to the address of the at least one Map end.

In the apparatus provided by the above embodiments of the present application, first, the first obtaining unit 601, in response to determining that obtaining data from the Map end corresponding to the Reduce end fails, obtains the data of the partition corresponding to the Reduce end from the target file system that provides redundant storage . The target file system stores the calculation result file according to a predetermined directory structure. The calculation result file includes the same number of partitions as the Reduce side and their corresponding data. Then, the second generating unit 602 uses the acquired data to execute the Reduce task to generate the final result of the MapReduce process. This avoids the consumption of computing resources and time costs caused by recalculation, and improves the stability of the Shuffle process. Moreover, since the entire MapReduce model has not been changed, it is less intrusive to the computing framework and has good universality.

With further reference to FIG. 7a, a sequence 700 of interactions between various devices in one embodiment of the MapReduce-based data transmission method is shown. The MapReduce-based data transmission system may include: a Map side (eg, servers 1052 and 1053 shown in FIG. 1 ), a Reduce side (eg, servers 1055 and 1056 shown in FIG. 1 ), and a target file system (eg, the servers 1055 and 1056 shown in FIG. 1 ) server 1054). The above-mentioned Map terminal may be configured to implement the MapReduce-based data transmission method described in the embodiment shown in FIG. 2 . The above-mentioned Reduce side may be configured to implement the MapReduce-based data transmission method described in the embodiment shown in FIG. 4 . The above-mentioned target file system may be configured to, in response to receiving the calculation result file, name the above-mentioned calculation result file according to a predetermined naming rule, and store the above-mentioned calculation result file according to a predetermined directory structure.

In some optional implementations of this embodiment, the above-mentioned target file system is further configured to store multiple copies of the above-mentioned calculation result file in the data nodes of the above-mentioned target file system.

As shown in Fig. 7a, in step 701, the Map side executes the Map task to generate a calculation result file.

In step 702, the Map side uploads the calculation result file to the target file system that provides redundant storage.

In some optional implementations of this embodiment, the Map terminal (not shown in FIG. 1 ) sends the metadata information of the calculation result file to the task scheduling terminal.

In step 703, in response to receiving the calculation result file, the target file system names the calculation result file according to a predetermined naming rule, and stores the calculation result file according to a predetermined directory structure.

In this embodiment, the above-mentioned target file system may name the calculation result file according to a predetermined naming rule, and store the calculation result file according to a predetermined directory structure. Therefore, the above-mentioned target file system can uniformly manage the calculation result files uploaded by each Map terminal in the above-mentioned MapReduce.

It should be noted that, the above-mentioned predetermined naming rules and predetermined directory structures may be consistent with the descriptions of the foregoing embodiments, which will not be repeated here.

In some optional implementation manners of this embodiment, the above-mentioned target file system providing redundant storage may further store multiple copies of the above-mentioned calculation result file in a data node of the above-mentioned target file system in a distributed manner. Thus, the backup of the calculation result files uploaded by each Map terminal in the above MapReduce is completed.

In step 704, the Reduce side obtains the address of at least one Map side corresponding to the Reduce side from the task scheduling side.

In step 705, according to the address of the at least one Map side, the Reduce side sends a data acquisition request for acquiring the data of the partition corresponding to the Reduce side to the at least one Map side.

In this embodiment, as an example, as shown in Fig. 7b. The Reduce side_0 with the identifier "0" can send a data acquisition request to the Map side_0 with the identifier "0" and the Map side_1 with the identifier "1" respectively.

In step 706, the Reduce side receives the response information of the data request sent by the corresponding Map side.

In this embodiment, after receiving the data acquisition request sent by the Reduce side, the Map side in the above-mentioned MapReduce usually takes the data of the partition of the Reduce side corresponding to the above-mentioned data acquisition request in the local calculation result file of the Map side as the response information , which is returned to the Reduce side that sent the above data acquisition request. Therefore, the above-mentioned Reduce side can receive the response information of the data request sent by the corresponding Map side.

As an example, continue to refer to Figure 7b, Map side_0 may respond to the data acquisition request of Reduce side_0, and return the data of the 0th partition in the local data file "0.data" as response information to Reduce side_0. Therefore, the above-mentioned Reduce side_0 can obtain the data belonging to the partition corresponding to the Reduce side_0 from the Map side_0.

In practice, the Reduce side usually sends data acquisition requests to all the Map sides in the above MapReduce. Due to possible factors such as Map node failure, high load or network delay, in some implementations, the Reduce side often cannot receive all the data response information returned by the Map side. For the Map side corresponding to the unreceived data response information, the Reduce side can determine that it fails to obtain data from the Map side.

As an example, continue to refer to Figure 7b, after Reduce side_0 sends a data acquisition request, Map side_1 may not be able to respond to the above data acquisition request in time due to factors such as node failure, network delay, and high node load, so that Reduce side_0 cannot respond to the above data acquisition request in time. Get the required data of the 0th area from the Map side_1, and the data acquisition fails.

In step 707, in response to determining that the acquisition of data from the Map side corresponding to the Reduce side fails, the Reduce side acquires data of the partition corresponding to the Reduce side from the target file system providing redundant storage.

Optionally, continuing to refer to FIG. 7b, in response to determining that the response of Map side_1 to the above data acquisition request times out, the above-mentioned Reduce side_0 can access the target file system to obtain the 0th partition in the calculation result file of Map side_1. The data. Map side_1 can first obtain the index file "1.index" uploaded by Map side_1 stored in the target file system, and obtain the data of the 0th partition in the calculation result file uploaded by Map side_1 in the data file "1.data" ” start and end positions. The above-mentioned Reduce side_0 can read the required data from the above-mentioned data file "1.data" according to the instruction of the above-mentioned index file "1.index".

Based on the above-mentioned optional implementation manner, since the above-mentioned target file system can provide a multi-copy function, the above-mentioned target file system is responsible for the fault tolerance of Shuffle data (that is, the calculation result file uploaded by each Map terminal), so the required data will not be stored in the above-mentioned An inaccessible problem in the target file system.

In step 708, the Reduce side uses the acquired data to execute the Reduce task to generate the final result of the MapReduce process.

The above steps 701 and 702 are respectively consistent with the steps 201 and 202 in the foregoing embodiment and their optional implementation manners, and the foregoing steps 704, 705, 707 and 708 are respectively the same as the steps 401 and 402 in the foregoing embodiment. Its optional implementation is consistent. The above descriptions of steps 201, 202 and their optional implementations and steps 401, 402 and their optional implementations also apply to steps 701 and 702, steps 704, 705, 707 and 708. It is not repeated here.

In the MapReduce-based data transmission system provided by the above embodiments of the present application, first, the Map side executes the Map task to generate a calculation result file. Then, the Map side uploads the calculation result file to the target file system that provides redundant storage. In response to receiving the calculation result file, the target file system names the calculation result file according to a predetermined naming rule, and stores the calculation result file according to a predetermined directory structure. Next, the Reduce side obtains the address of at least one Map side corresponding to the Reduce side from the task scheduling side. According to the address of the at least one Map side, the Reduce side sends a data acquisition request for acquiring the data of the partition corresponding to the Reduce side to the at least one Map side. Then, in response to determining that the acquisition of data from the Map side corresponding to the Reduce side fails, the Reduce side acquires the data of the partition corresponding to the Reduce side from the target file system providing redundant storage. Finally, the Reduce side uses the acquired data to execute the Reduce task to generate the final result of the MapReduce process.

This avoids recomputing tasks caused by shuffle failures caused by Map node failure, high load, or network delay, improves the stability of the shuffle process in the MapReduce model, and improves the running speed of tasks. Moreover, the present application can use the existing file system that supports multiple copies of data to back up Shuffle files, and only adds a layer of pluggable computing framework and the connection layer of the file system, and does not make any changes to the entire MapReduce model. Computational frameworks are minimally intrusive. Therefore, the solution of the present application for the Shuffle process of the MapReduce model is applicable to any big data computing framework based on the MapReduce model.

Referring next to FIG. 8 , it shows a schematic structural diagram of an electronic device (eg, the server 1052 or 1055 in FIG. 1 ) 800 suitable for implementing the embodiments of the present application. The server shown in FIG. 8 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 8 , an electronic device 800 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 801 that may be loaded into random access according to a program stored in a read only memory (ROM) 802 or from a storage device 808 Various appropriate actions and processes are executed by the programs in the memory (RAM) 803 . In the RAM 803, various programs and data necessary for the operation of the electronic device 800 are also stored. The processing device 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

In general, the following devices can be connected to the I/O interface 805: input devices 806 including, for example, a touch screen, touchpad, keyboard, mouse, etc.; output devices including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, etc. 807; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication means 809 may allow electronic device 800 to communicate wirelessly or by wire with other devices to exchange data. While FIG. 8 shows an electronic device 800 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in FIG. 8 can represent one device, and can also represent multiple devices as required.

In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 809 , or from the storage device 808 , or from the ROM 802 . When the computer program is executed by the processing device 801, the above-mentioned functions defined in the methods of the embodiments of the present application are executed.

It should be noted that the computer-readable medium described in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the embodiments of the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. While in embodiments of the present application, a computer-readable signal medium may include a data signal in baseband or propagated as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: electric wire, optical cable, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is made to: execute a Map task to generate a calculation result file, wherein the calculation result file includes and The number of partitions and their corresponding data on the reduce side is the same; upload the calculation result file to the target file system that provides redundant storage, so that the corresponding reduce side can obtain the data in the calculation result file through the target file system, where the target file system The calculation result file is named according to a predetermined naming rule, and the calculation result file is stored according to a predetermined directory structure.

Computer program code for performing the operations of the embodiments of the present application may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units involved in the embodiments of the present application may be implemented in a software manner, and may also be implemented in a hardware manner. The described unit may also be provided in the processor, for example, it may be described as: a processor including a first generating unit and an uploading unit. The names of these units do not constitute a limitation on the unit itself in some cases. For example, the first generation unit may also be described as "a unit that executes a Map task to generate a calculation result file, where the calculation result file is It includes the same number of partitions as the Reduce side and its corresponding data".

The above description is only a preferred embodiment of the present application and an illustration of the applied technical principles. It should be understood by those skilled in the art that the scope of the invention involved in the embodiments of the present application is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover, without departing from the above inventive concept, the above Other technical solutions formed by any combination of technical features or their equivalent features. For example, a technical solution is formed by replacing the above features with the technical features disclosed in the embodiments of the present application (but not limited to) with similar functions.

Claims (12)

1. A data transmission method based on MapReduce, applied to the Map side, comprising: Execute the Map task to generate a calculation result file, wherein the calculation result file includes the same number of partitions as the Reduce side and their corresponding data; Upload the calculation result file to a target file system that provides redundant storage, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system, wherein the target file system is named according to a predetermined name The rules name the calculation result files, and store the calculation result files according to a predetermined directory structure. 2. The method according to claim 1, wherein the calculation result file comprises an index file and a data file, the data file is used to record data, and the index file is used to identify the data of each partition in the data file. the start and end positions in ; and The method also includes: The metadata information of the calculation result file is sent to the task scheduling terminal, wherein the metadata information includes the corresponding relationship between the calculation result file and the Map terminal. 3. The method according to claim 2, wherein, the predetermined naming rule comprises the identification of the corresponding Map end in the name of the calculation result file, and distinguishes the data file and the index file of the calculation result file according to the suffix . 4. The method according to any one of claims 1-3, wherein the predetermined directory structure comprises a tree structure, and the tree structure comprises, from top to bottom, an identification of an application, an identification of a MapReduce process belonging to an application, an identification of a MapReduce process belonging to an application, a The ID of the Map task in the MapReduce process, and the ID of the calculation result file belonging to the Map task. 5. A data transmission method based on MapReduce, applied to the Reduce side, comprising: In response to determining that the acquisition of data from the Map side corresponding to the Reduce side fails, acquire data of the partition corresponding to the Reduce side from a target file system that provides redundant storage, wherein the target file system stores data according to a predetermined directory structure There is a calculation result file, and the calculation result file includes the same number of partitions as the Reduce side and their corresponding data; Using the acquired data, a Reduce task is executed to generate the final result of the MapReduce process. 6. The method according to claim 5, wherein before the acquiring data of the partition corresponding to the Reduce side from the file system, the method further comprises: Acquire the address of at least one Map terminal corresponding to the Reduce terminal from the task scheduling terminal, wherein the task scheduling terminal stores the metadata information of the calculation result file, and the metadata information includes the calculation result file and the Map terminal. Correspondence between the ends; According to the address of the at least one Map side, a data acquisition request for acquiring the data of the partition corresponding to the Reduce side is sent to the at least one Map side. 7. A data transmission device based on MapReduce, applied to the Map side, comprising: a first generating unit, configured to execute a Map task to generate a calculation result file, wherein the calculation result file includes the same number of partitions as the Reduce side and their corresponding data; The uploading unit is configured to upload the calculation result file to the target file system, so that the corresponding Reduce side obtains the data in the calculation result file through the target file system, wherein the target file system according to a predetermined The naming rule names the calculation result file, and stores the calculation result file according to a predetermined directory structure. 8. A data transmission device based on MapReduce, applied to the Reduce side, comprising: The first obtaining unit is configured to obtain data of the partition corresponding to the Reduce end from the target file system in response to determining that the data obtaining request to the Map end corresponding to the Reduce end fails, wherein the target file system is configured according to the The predetermined directory structure stores a calculation result file, and the calculation result file includes the partitions with the same number as the Reduce side and their corresponding data; The second generating unit is configured to execute a Reduce task using the acquired data to generate the final result of the MapReduce process. 9. A data transmission system based on MapReduce, comprising: The Map side is configured to implement the method according to any one of claims 1-4; The Reduce side is configured to implement the method according to any one of claims 5-6; The target file system is configured to, in response to receiving the calculation result file, name the calculation result file according to a predetermined naming rule, and store the calculation result file according to a predetermined directory structure. 10. The system of claim 9, wherein the target file system is further configured to store multiple copies of the computation result file in data nodes of the target file system. 11. An electronic device comprising: one or more processors; a storage device on which one or more programs are stored; The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6. 12. A computer-readable medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the method of any one of claims 1-6.

Images