WO2017206591A1

WO2017206591A1 - Data processing system and data processing method

Info

Publication number: WO2017206591A1
Application number: PCT/CN2017/079761
Authority: WO
Inventors: 张延松; 张宇; 李翠平; 孙东旺
Original assignee: 华为技术有限公司
Priority date: 2016-06-01
Filing date: 2017-04-07
Publication date: 2017-12-07
Also published as: CN107451090B; CN107451090A

Abstract

A data processing system (100) and a data processing method. The data processing system (100) comprises a main processor (110), a co-processor (120) and a control module (130), wherein the main processor (110) is used for sending data to be processed to the control module (130); the control module (130) is used for receiving the data to be processed sent by the main processor (110), and sending the data to be processed to the co-processor (120); the co-processor (120) is used for receiving the data to be processed sent by the control module (130), processing the data to be processed to obtain a processing result of the data to be processed, and sending the processing result of the data to be processed to the main processor (110) via the control module (130); and the main processor (110) is used for receiving the processing result of the data to be processed sent by the co-processor (120) via the control module (130). The data processing system (100) and the data processing method can simplify the design of the system, and alleviate the workload of the main processor (110).

Description

Data processing system and data processing method

The present application claims priority to Chinese Patent Application No. 201610387562.5, entitled "Data Processing System and Data Processing Method", which is incorporated herein by reference. .

Technical field

The present application relates to the field of databases, and more particularly to data processing systems and data processing methods.

Background technique

The hybrid platform architecture that the main processor and coprocessor assist in processing data typically consists of 1-2 central processing units (CPUs) and 1-8 integrated general core (MIC) coprocessors. Coprocessor processors are often used as online analytical processing (OLAP) query accelerators, primarily for computationally intensive tasks in OLAP queries. From the perspective of computational features, coprocessors are especially useful for stand-alone operations such as connection processing or aggregation processing of data that are time-consuming and suitable for parallel execution.

In the process of processing data between the existing CPU and MIC hybrid architecture, it is necessary to allocate space on the MIC end, and copy the two required to be processed from the CPU side through the peripheral component interconnect express (PCIe) channel to the MIC end. Data, then execute the real-time operating system (kernel) program on the MIC side to process the two data, and copy the processing result from the MIC end to the CPU end through the PCIe channel, and finally the MIC end releases the allocated space.

However, when large data volume data processing is required, the data cannot be stored all at once on the MIC end. In this case, data partitioning of the two data is performed on the CPU side, and then the above data processing process is performed on each partition at the MIC end. In this way, all data processing, data transfer and memory access management work is performed by the main processor, resulting in a heavy workload of the main processor, and integrated data transmission, data processing and memory in the system code design. Access management features, the code is more complicated.

Summary of the invention

The present application provides a system and method for data processing that simplifies the design of the system and reduces the workload of the main processor.

In a first aspect, a data processing system is provided, including a main processor, a coprocessor, and a control module;

The main processor is configured to send the to-be-processed data to the control module, and the control module is configured to receive the to-be-processed data sent by the main processor, and send the to-be-processed data to the coprocessor;

The coprocessor is configured to receive the to-be-processed data sent by the control module, process the to-be-processed data, obtain a processing result of the to-be-processed data, and send the processing result of the to-be-processed data to the main processor;

The main processor is configured to receive a processing result of the to-be-processed data sent by the coprocessor through the control module.

The data processing system of the embodiment of the present application reduces the workload of the main processor by adding a control module for controlling transmission data between the main processor and the coprocessor, and simplifies the integration of data processing and data transmission. And system code for memory management.

In conjunction with the first aspect, in a first possible implementation manner of the first aspect, the control module is configured to: Processing data is sent to the coprocessor at one time, or the data to be processed is sent to the coprocessor in batches, wherein the storage attribute of the data to be processed is carried when the data to be processed is sent to the coprocessor, The storage attribute includes a resident attribute or a flow attribute indicating that the pending data can be accessed multiple times, the flow attribute indicating that the pending data can only be accessed once.

In conjunction with the first possible implementation of the first aspect, in a second possible implementation manner of the first aspect, the control module is specifically configured to: according to a processing context of the to-be-processed data, a data volume of the to-be-processed data And the memory size available to the coprocessor to determine the storage attributes of the pending data.

In conjunction with the first or second possible implementation of the first aspect, in a third possible implementation of the first aspect, the control module is further configured to: send the to-be-processed data to the coprocessor The memory size of the resident area in the memory of the coprocessor and the memory size of the streaming area are determined according to the data amount of the data to be processed, the storage attribute of the data to be processed, and the memory size available to the coprocessor. The data stored in the resident area can be accessed multiple times, and the data in the streaming area can only be accessed once and deleted or overwritten after being accessed.

It should be understood that the storage attribute of the data to be processed includes a resident attribute or a flow attribute, wherein the resident attribute indicates that the pending data can be accessed multiple times, and the streaming attribute indicates that the pending data can only be accessed once.

It should also be understood that the amount of memory available to the coprocessor is the amount of memory in the coprocessor other than the memory occupied by the coprocessor system.

Optionally, the control module is configured to carry the storage attribute of the to-be-processed data in the to-be-processed data, and send the to-be-processed data block carrying the storage attribute to the coprocessor, or may be in the coprocessor The indication information for indicating the storage attribute of the data to be processed is not limited in this embodiment of the present application.

Optionally, the control module is further configured to: before sending the to-be-processed data to the coprocessor, determine a memory size of the resident area in the co-processing and a memory size of the streaming area, where the resident area is stored The data can be accessed multiple times, the data stored in the streaming area can only be accessed once, and will be deleted or overwritten after the access.

In the data processing system of the embodiment of the present application, the control module sets the resident area and the flow area in the coprocessor, so that the data needs to be stored in the resident area for multiple times during the calculation, and the data used once is batched. The transfer area is transferred and processed separately with the data in the camp, which reduces the amount of data transmission each time through the PCIe channel and improves the efficiency of data processing.

In conjunction with the third possible implementation of the first aspect, in a fourth possible implementation manner of the first aspect, the flow area of the coprocessor includes a plurality of consecutive sub-flow regions;

The control module is specifically configured to: send the data to be processed in batches to each of the plurality of consecutive sub-flow regions;

The coprocessor is specifically configured to: sequentially read data in each of the plurality of sub-flow regions, and process the data in each of the sub-flow regions, in the data in each of the sub-flow regions After the processing is completed, each sub-flow area is identified as idle; after the data processing in the last sub-flow area of the consecutive plurality of sub-flow areas is completed, and the first sub-flow area of the consecutive plurality of sub-flow areas When the flag is idle, the data is read from the first sub-streaming area of the consecutive plurality of sub-streaming areas.

Optionally, the number of the plurality of sub-flow areas = the memory size of the flow area / the memory size of the sub-flow area, wherein the size of the sub-flow area is set such that data in the sub-flow area is transmitted to the coprocessor The time is the same or approximately the same as the time at which the core of the coprocessor computes the data in the subflow region.

Optionally, the memory size of each of the plurality of sub-flow regions may be the same.

The data processing system of the embodiment of the present application can improve the data transmission efficiency of the PCIe channel and the processing efficiency of the data processing system by setting a plurality of sub-streaming areas and considering the memory size of the coprocessor and the transmission performance of the PCIe channel.

With reference to any one of the possible implementations of the first to fourth possible implementations of the first aspect, in a fifth possible implementation of the first aspect, the control module is specifically configured to:

If the data to be processed includes the first data block and the second data block, the data amount of the first data block is smaller than the memory size available to the coprocessor, and the data amount of the second data block is greater than that available to the coprocessor a memory size, determining a storage attribute of the first data block as a resident attribute, and sending the first data block to a resident area of the coprocessor, determining that a storage attribute of the second data block is a flow attribute, and The second data block is divided into a plurality of second sub-blocks that are sequentially sent in batches to a flow area of the coprocessor.

In conjunction with the fifth possible implementation of the first aspect, in a sixth possible implementation manner of the first aspect, the coprocessor is specifically configured to: the first data block and the flow in the resident area Processing each of the plurality of second sub-blocks in the region to obtain a processing result of the first data block and each of the second sub-blocks; according to the first data block and the Processing result of each second sub-block of the plurality of second sub-blocks, and processing results of the first block and the second block are obtained.

In the data processing system of the embodiment of the present application, the process of processing the data to be processed by the coprocessor and the process of sending the data to be processed by the control module to the coprocessor can be performed simultaneously, thereby reducing the amount of data transmission each time through the PCIe channel, thereby further improving Data transmission efficiency.

With reference to any one of the possible implementations of the first to the sixth possible implementations of the first aspect, in a seventh possible implementation manner of the first aspect, the control module is further configured to: Processing data includes a first data block and a second data block, the data amount of the second data block being greater than a memory size available to the coprocessor, and the data amount of the first data block after being hashed is greater than the coprocessing The available memory size, determining the storage attribute of the first data block as a flow attribute, and dividing the first data block into a plurality of first sub-blocks sent to the flow area of the coprocessor in batches, determining the first The storage attribute of the two data blocks is a flow attribute, and the second data block is divided into a plurality of second sub-blocks and sent to the flow area of the coprocessor in batches.

In conjunction with the seventh possible implementation of the first aspect, in an eighth possible implementation of the first aspect, the coprocessor is further configured to: in the multiple first sub-blocks in the forwarding area Processing each of the first sub-data block and each of the plurality of second sub-data blocks to obtain a processing result of each of the first sub-data blocks and each of the second sub-blocks; Obtaining the first data block and the second data block according to a processing result of each of the first sub-data blocks and each of the plurality of second sub-data blocks Processing results.

With reference to any one of the first to the eighth possible implementation manners of the first aspect, in a ninth possible implementation manner of the first aspect, the main processor includes a first buffer area and a second buffer area; the control module is configured to store the to-be-processed data in the first buffer area, and send the data in the first buffer area to the co-processor after the first buffer area is full. The remaining pending data is further stored to the second buffer area, and after the second buffer area is full, the data in the second cache is sent to the coprocessor.

In conjunction with the ninth possible implementation of the first aspect, in a tenth possible implementation manner of the first aspect, the control module is further configured to send data in the first buffer area to the coprocessor At the same time, the data to be processed is continuously stored in the second buffer area, and after the second buffer area is full, the data in the second buffer area is sent to the coprocessor.

Optionally, the memory size of the first buffer area and the second buffer area in the main processor are the same as the memory size of the sub-streaming area in the coprocessor, which can improve memory utilization during data processing. Rate and improve data processing efficiency.

In the data processing system of the embodiment of the present application, the control module can improve the parallelism of the processed data and improve the transmission efficiency of the PCIe channel by alternately transmitting data between the first buffer area and the second buffer area in the main processor.

In a second aspect, a data processing method is provided, the method being applied to a data processing system in a first aspect, the data processing system comprising a main processor, a coprocessor, and a control module, wherein the main processor, the coordinating The processor and the control module are configured to perform the method corresponding to the main processor, the coprocessor, and the control module in any of the possible implementations of the first aspect or the first aspect described above.

In a third aspect, a computer readable medium is provided for storing a computer program comprising instructions for performing the method of any of the second aspect or any of the possible implementations of the second aspect.

DRAWINGS

FIG. 1 is an architectural example of a data processing system of an embodiment of the present application.

2 is a schematic block diagram of a data processing system of an embodiment of the present application.

FIG. 3 is a schematic flowchart of a data processing method according to an embodiment of the present application.

FIG. 4 is a schematic flowchart of another data processing method according to an embodiment of the present application.

FIG. 5 is a schematic flowchart of still another data processing method according to an embodiment of the present application.

FIG. 6 is a schematic flowchart of still another data processing method according to an embodiment of the present application.

FIG. 7 is a schematic diagram of a scenario of a data processing method according to an embodiment of the present application.

FIG. 8 is a schematic diagram of a scenario of another data processing method according to an embodiment of the present application.

FIG. 9 is a schematic diagram of a scenario of still another data processing method according to an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

1 shows an architectural example of a data processing system 100 of an embodiment of the present application. As shown in FIG. 1, the data processing system includes: a main processor 110, a coprocessor 120, and a control module 130, and the main processor 110 and Data transmission between the coprocessors 120 through the PIC channel, wherein the main processor 110 typically includes 1-2 CPUs, and the coprocessor 120 typically includes 1-8 MIC processors, the control module 130 is configured to control the data. Data processing and data transfer between the main processor 110 and the coprocessor 120.

Specifically, the main processor 110 is configured to receive a data processing request of the user end, and determine data to be processed according to the data processing request of the user end, where the data processing request may be, for example, a connection processing request, a query processing request, or an aggregation processing request, etc. The application embodiment does not limit this.

Specifically, the control module 130 is configured to receive the to-be-processed data sent by the main processor 110, determine a storage attribute of the to-be-processed data in the coprocessor, and send the to-be-processed data to the association by using two different transmission modes. Processor 120.

It should also be understood that the control module may send the data to be processed to the coprocessor at one time, or may send the data to be processed in batches to the coprocessor in sequence.

Optionally, the control module block may be implemented by software, for example, may be a program code in the system, or may be implemented by hardware, for example, may be a device integrated in other control devices, which is not limited by the embodiment of the present application.

Specifically, the coprocessor 120 is configured to receive, by the control module 130, the to-be-processed data sent by the main processor 110, process the data to be processed, obtain a processing result, and finally return the processing result to the main processor 110 through the control module 130. .

Optionally, the coprocessor 120 can be used to perform hash connection processing or OLAP query on the data, and the operations such as the computationally intensive data task or the aggregation processing are time-consuming and suitable for the parallel operation. .

The data processing system of the embodiment of the present application encapsulates a program and code that the main processor needs to call in the data processing process in the control module by adding a control module for controlling data transmission between the main processor and the coprocessor. In the process of data processing, the main processor does not need to call all the code that needs to be used every time, and only needs to simply call the control module to control the data processing, data transmission and memory management, and reduce the control. The workload of the main processor and simplifying the code design of the system.

FIG. 2 shows a schematic block diagram of a system 200 for data processing in accordance with an embodiment of the present application. The system 200 includes a main processor 210, a coprocessor 220, and a control module 230.

The main processor 210 is configured to send data to be processed to the control module 230.

The control module 230 is configured to receive the to-be-processed data sent by the main processor 210, and send the to-be-processed data to the coprocessor 220.

The coprocessor 220 is configured to receive the to-be-processed data sent by the control module 230, process the to-be-processed data, obtain a processing result of the to-be-processed data, and send the processing result of the to-be-processed data to the control module. To the main processor 210;

The main processor 210 is configured to receive a processing result of the to-be-processed data sent by the coprocessor 220 through the control module 230.

The data processing system of the embodiment of the present application encapsulates a program and code that the main processor needs to call in the data processing process in the control module by adding a control module for controlling data transmission between the main processor and the coprocessor. In the process of data processing, the main processor does not need to call all the code that needs to be used multiple times, and only needs to simply call the control module, thereby realizing control of data processing, data transmission, and memory management, reducing the main The workload of the processor and simplifies the code design of the system.

Specifically, the main processor is configured to determine data to be processed, and send the to-be-processed data to the control module.

Specifically, the control module is specifically configured to send the to-be-processed data to the coprocessor or send the to-be-processed data to the coprocessor in batches, and send the to-be-processed data to the control module. The coprocessor carries the storage attribute of the data to be processed.

Optionally, the control module is specifically configured to: before the control module sends the to-be-processed data to the coprocessor, according to a processing context of the to-be-processed data, an amount of data of the to-be-processed data, and an available The memory size determines the storage attribute of the data to be processed, but the embodiment of the present application is not limited thereto.

Optionally, the control module may determine, according to whether the to-be-processed data needs to be used multiple times, and whether the to-be-processed data can be all stored to the coprocessor, and determine a storage attribute of the to-be-processed data.

Optionally, the control module may determine, according to the data volume of the data to be processed, the storage attribute of the data to be processed, and the memory size of the coprocessor, the memory size and the flow area of the resident area in the coprocessor. memory size.

It should be understood that the control module may preferentially consider the data to be processed of the resident attribute, determine the memory size of the resident area in the available memory of the coprocessor according to the data amount of the data to be processed of the resident attribute, and then The remaining available memory of the coprocessor is determined as the memory size of the streaming area.

Optionally, the control module may divide the flow area into a plurality of sub-flow areas, where the number of the multiple sub-flow areas may be a memory size of the flow area divided by a memory size of each of the sub-flow areas, where each of the sub-flow areas The memory size of the flow area can be pre-tested according to the platform of the data processing system, that is, the memory size of each sub-flow area can be set such that the data in each sub-flow area is copied to the coprocessor and the association. The core of the processor calculates the time of the data in each of the sub-streaming areas to be the same or similar, and the transmission performance of the data of each sub-streaming area in the PCIe channel transmission may be considered, which is not limited in this embodiment of the present application.

Specifically, the control module is specifically configured to establish a queue of the plurality of sub-flow areas, form a plurality of consecutive sub-flow areas, and sequentially send the to-be-processed data to each of the plurality of sub-flow areas in batches.

Specifically, the coprocessor is specifically configured to sequentially read data in each of the plurality of sub-flow regions, and process the data in the sub-flow region, and in each of the self-flow regions After the data is processed, the each self-flowing rotor region is identified as idle, after the data processing in the last sub-flow region of the consecutive plurality of sub-flow regions is completed, and the continuous plurality of sub-flow regions are detected. When the first sub-flow area is identified as idle, data is continuously read from the first sub-flow area of the plurality of consecutive sub-flow areas, and processed.

In the data processing system of the embodiment of the present application, the coprocessor reads and processes data through a plurality of cyclic sub-flow regions, thereby improving memory utilization of the coprocessor and parallelism of the data processing.

Optionally, the main processor may include a first buffer area and a second buffer area, where the control module is configured to The data is stored in the first buffer area, and after the first buffer area is full, the data in the first buffer area is sent to the coprocessor.

Optionally, the control module is configured to continue to store the data to be processed in the second buffer area while the data in the first buffer area is sent to the coprocessor, and save the data in the second buffer area. After the full, the data in the second buffer area is sent to the coprocessor.

The data processing system of the embodiment of the present application improves the parallelism of the system and the data processing efficiency by setting two buffer areas in the main processor to alternately transmit and store data, and can set the size of the buffer area to It is the same size as each of the reflow areas in the coprocessor, which improves coprocessor memory utilization and improves data processing efficiency.

As an optional embodiment, it is assumed that the to-be-processed data includes a first data block and a second data block, wherein a data amount of the first data block is smaller than a memory size available to the coprocessor, and the second data block The amount of data is greater than the amount of memory available to the coprocessor.

Specifically, the control module is configured to receive the first data block and the second data block sent by the main processor, according to the data amount of the first data block, the data amount of the second data block, and the coprocessor The available memory size and the processing context, determining that the first data block is a resident attribute, the second data block is a streaming attribute, and sending the first data block to a resident area of the coprocessor, the second The data block is divided into a plurality of second sub-blocks, and the plurality of second sub-blocks are sent in batches to a flow area of the coprocessor.

Optionally, the first data block may be sent to the resident area of the coprocessor at one time, and the first data block may be a hashed first data block, which is not limited in this embodiment of the present application.

Specifically, the coprocessor is configured to process the first data block in the resident area and each of the plurality of second sub data blocks in the flow area, the first data block Processing result corresponding to each second sub-block, and obtaining the first data block and the first data block according to a processing result of the first data block and each second sub-data block of the plurality of second sub-data blocks Processing result of the two data blocks, and transmitting the processing result of the first data block and the second data block to the main processor through the control module.

As another optional embodiment, it is assumed that the to-be-processor data includes a first data block and a second data block, wherein a data amount of the second data block is greater than a memory size available to the coprocessor, and is hashed. The processed data volume of the first data block is also larger than the memory size available to the coprocessor.

The control module is configured to receive the first data block and the second data block sent by the main processor, according to the data amount of the first data block, the data amount of the second data block, and the available memory of the coprocessor Determining the size and the processing context, determining that the first data block and the second data block are both flow attributes, and dividing the first data block into a plurality of first sub-blocks sent to the flow area of the coprocessor in batches, The second data block is divided into a plurality of second sub-blocks and sent to the streaming area of the co-processor in batches.

Optionally, the main processor may include a first buffer area and a second buffer area, where the first cache has the same memory size as the second cache, and the control module may use the first buffer area and the second buffer area to A plurality of first sub-blocks and a plurality of second sub-blocks are sent to the coprocessor, which is not limited in this embodiment of the present application.

Specifically, the control module is configured to divide the first data block and the second data block into blocks according to a memory size of the first buffer area or the second buffer area, to obtain a plurality of first sub-blocks and a second sub-block corresponding to each of the plurality of first sub-blocks, and such that each of the first sub-blocks and the second sub-block corresponding to each of the first sub-blocks The sum of the data amounts is equal to the first buffer area or the second buffer area, and the control module is configured to rotate each of the plurality of first sub-blocks by the first buffer area and the second buffer area. Subdata The block and each of the plurality of second sub-blocks are sent to the coprocessor.

Specifically, the coprocessor is configured to process each of the first one of the plurality of first subblocks and the second of the plurality of second subblocks in the streaming area Obtaining a processing result of each of the first sub-blocks and each of the second sub-blocks, according to each of the plurality of first sub-blocks and the plurality of second sub-blocks Processing result of each second sub-block, obtaining a processing result of the first data block and the second data block, and sending the processing result of the first data block and the second data block to the main processor.

The system for data processing of the present application has been described above with reference to Figs. 1 and 2, and a method of performing data processing in the above data processing system will be described below with reference to Figs.

FIG. 3 is a schematic flowchart of a method 300 for data processing according to an embodiment of the present application. The method 300 can be applied to the system shown in FIG. 2, and can implement various functions implemented by the main processor, the coprocessor, and the control module in the foregoing data processing system. To avoid repetition, details are not described herein again.

S301. The main processor sends the to-be-processed data to the control module.

S302. The control module receives the to-be-processed data sent by the main processor, and sends the to-be-processed data to the coprocessor.

S303, the coprocessor receives the to-be-processed data sent by the control module, processes the to-be-processed data, obtains a processing result of the to-be-processed data, and sends the processing result of the to-be-processed data to the Main processor.

S304. The main processor receives a processing result of the to-be-processed data sent by the coprocessor through the control module.

Optionally, the control module sends the to-be-processed data to the coprocessor, including:

Sending the to-be-processed data to the coprocessor once, or sending the to-be-processed data to the coprocessor in batches, and carrying the to-be-processed data when the to-be-processed data is sent to the coprocessor An attribute comprising a resident attribute or a flow attribute, the resident attribute indicating that the pending data can be accessed multiple times, the flow attribute indicating that the pending data can only be accessed once.

Optionally, the control module determines, according to a processing context of the to-be-processed data, an amount of data of the to-be-processed data, and a memory size available to the coprocessor, a storage attribute of the to-be-processed data.

Optionally, before the sending the to-be-processed data to the coprocessor, the method further includes:

The control module determines the memory size of the resident area in the memory of the coprocessor and the memory size of the streaming area according to the data amount of the data to be processed, the storage attribute of the data to be processed, and the available memory size of the coprocessor. Where the data stored in the resident area can be accessed multiple times, the data in the streaming area can only be accessed once and deleted or overwritten after being accessed.

Optionally, the flow area of the coprocessor includes a plurality of consecutive sub-flow areas; the control module sends the data to be processed to the coprocessor in batches, and the control module sends the data to be processed in batches in sequence. Up to each of the plurality of sub-flow regions; the coprocessor processing the data to be processed, and obtaining the processing result of the data to be processed, including:

The coprocessor sequentially reads data in each of the plurality of sub-flow regions, and processes the data in each of the sub-flow regions. After the data processing in each of the sub-flow regions is completed, Identifying each sub-flow area as idle; the coprocessor is after the data processing in the last sub-flow area of the consecutive plurality of sub-flow areas is completed, and the first sub-flow area of the consecutive plurality of sub-flow areas When the flag is idle, the data is continuously read from the first sub-streaming area of the plurality of consecutive sub-streaming areas.

Optionally, the control module sends the to-be-processed data to the coprocessor in batches, including:

If the data to be processed includes the first data block and the second data block, the data amount of the first data block is smaller than the memory size available to the coprocessor, and the data amount of the second data block is greater than that available to the coprocessor a memory size, the control module determines that a storage attribute of the first data block is a resident attribute, and sends the first data block to a resident area of the coprocessor, and the control module determines a storage attribute of the second data block For the flow attribute, the second data block is divided into a plurality of second sub-blocks and sent to the flow area of the co-processor in batches.

Optionally, the coprocessor processes the to-be-processed data to obtain a processing result of the to-be-processed data, including:

The coprocessor processes the first data block in the resident area and each of the plurality of second sub data blocks in the flow area to obtain the first data block and the a processing result of each second sub-block; the coprocessor obtains the first data block according to the processing result of the first data block and each second sub-block of the plurality of second sub-blocks The processing result of the second data block.

Optionally, the control module sends the to-be-processed data to the coprocessor, including: if the to-be-processed data includes the first data block and the second data block, the data volume of the second data block is greater than the coprocessor The available memory size, and the data amount of the hashed first data block is greater than the available memory size of the coprocessor, the control module determines that the storage attribute of the first data block is a streaming attribute, and the first a data block is divided into a plurality of first sub-blocks that are sent in batches to a flow area of the coprocessor, and the control module determines that a storage attribute of the second data block is a flow attribute, and divides the second data block into multiple The second sub-blocks are sent in batches to the flow area of the coprocessor.

The coprocessor processes each of the plurality of first sub-blocks and the second of the plurality of second sub-blocks in the flow region to obtain each of the plurality of second sub-blocks Processing results of the first sub-block and the each second sub-block; the co-processor according to each of the plurality of first sub-blocks and the plurality of second sub-blocks The processing result of each second sub-block obtains the processing result of the first data block and the second data block.

Optionally, the main processor includes a first buffer area and a second buffer area; the control module sends the to-be-processed data to the coprocessor, and the control module stores the to-be-processed data to the first cache. The area, until the first buffer area is full, sends the data in the first buffer area to the coprocessor.

While the data in the first buffer area is sent to the coprocessor, the control module continues to store the to-be-processed data to the second buffer area, and after the second buffer area is full, the first The data in the second buffer is sent to the coprocessor.

The data processing method of the embodiment of the present application completes data transmission and data processing between the main processor and the coprocessor through a control module between the main processor and the coprocessor, thereby reducing the workload of the main processor and simplifying System code that originally integrates data processing, data transfer, and memory management.

The data processing method of the embodiment of the present application is described in detail with reference to FIG. 3, and the data processing method of the embodiment of the present application will be described in detail below with reference to specific embodiments.

FIG. 4 is a schematic flowchart of another data processing method 400 of the embodiment of the present application. The method 400 can be applied to the system shown in FIG. 2, and the data processing system is described in detail in conjunction with FIG. 7. A method flow when the table and the S table perform a hash join process, wherein the R table is associated with the S table.

It should be understood that the embodiment of the present application can be applied to a relational database. In a relational database, each data table has several attributes. If one of the attributes can uniquely identify the data table, the attribute group is A primary key of the form, for example, the R table may be a student form, the student form includes a student number, a name, and a class, and each student's student number is unique, and the student number is a primary key, and the foreign key is mainly Another data table is associated. For example, the S table can be a transcript. The transcript includes a student number, a course number, and a grade. The course number and the student number can be used together to determine the grade, so the primary key of the transcript is the student number. And the course number, and the student number in the transcript corresponds to the student number in the student table. Therefore, the student number in the transcript is a foreign key of the student table, and the student table and the transcript are associated by the student number.

S401. The control module receives an R table and an S table sent by the main processor, where the R table and the S table are associated with each other, and the amount of data in the R table is smaller than a memory size available to the coprocessor, and the data in the S table. The amount is greater than the memory size that the coprocessor can.

S402. The control module identifies the R-type resident attribute according to the data amount of the R table, the data amount of the S-table, the available memory size of the coprocessor, and the processing context, and identifies the S-table as a streaming attribute.

Specifically, the control module may send the R table to the camping area of the coprocessor at one time, and send the S table to the streaming area of the coprocessor in batches.

Optionally, before the S402, the control module may: according to the data amount of the R table, the data amount of the S table, the resident attribute of the R table, the streaming attribute of the S table, and the available memory of the coprocessor Size, determines the memory size of the resident area in the coprocessor and the memory size of the flow area.

S403. The control module performs a hash operation on the R table to obtain an R table hash table.

S404. The control module sends the R table hash table to a resident area of the coprocessor.

S405. The control module divides the S table into a plurality of sub-S tables. S406. The control module sends the jth sub-S table of the plurality of sub-S tables to the i-th free sub-streaming area of the coprocessor.

Optionally, the main processor may include a first buffer area and a second buffer area, and the control module may send, by using the first buffer area and the second buffer area, each sub-S table of the plurality of sub-S tables to the The flow area of the coprocessor.

S407. The coprocessor performs a hash join operation on the R table hash table and the jth child S table in the i th substreaming area in the resident area, to obtain a processing result corresponding to the jth child S table.

S408. The coprocessor identifies the i-th sub-streaming area as idle.

S409. The control module sends the j+1th sub-S table in the plurality of sub-S tables to the i+1th idle sub-streaming area of the coprocessor.

S410, the coprocessor performs a hash join operation on the R table hash table and the j+1th sub S table in the i+1th substreaming area in the resident area, to obtain the j+1th child S The processing result corresponding to the table.

S411. The coprocessor identifies the i+1th sub-streaming area as idle.

Optionally, the coprocessor may include a plurality of consecutive sub-flow regions, and the coprocessor may sequentially read the sub-S tables in each of the consecutive plurality of sub-flow regions, and each of the sub-flow regions The child S table in the process is processed with the R table in the resident area.

S412. The coprocessor obtains the R table and the S table according to the hash join result of the R table hash table in the dwell area and the sub S table in each of the plurality of subflow regions. Hash the result of the connection.

Specifically, the coprocessor can obtain the data connection processing result corresponding to each sub-S table in the plurality of sub-S tables according to the method described in S406-S411, and further obtain the hash connection of the R table and the S table. result.

S413. The coprocessor sends the hash connection result to the main processor through the control module.

FIG. 5 is a schematic flowchart of still another data processing method 500 of the embodiment of the present application. The method 500 can be applied to the system as shown in FIG. 2, and another method flow of the data processing system in performing data connection processing on the R table and the S table is described in detail in conjunction with FIG. 8.

S501. The control module receives an R table and an S table sent by the main processor, where the R table is associated with the S table, and the data amount of the S table is greater than a memory size available to the coprocessor, and the hash operation is performed. The amount of data in the R table is also greater than the amount of memory available to the coprocessor.

S502. The control module identifies the R table and the S table as a flow attribute according to the data amount of the R table, the data amount of the S table, the available memory size of the coprocessor, and the processing context.

Optionally, before the S502, the control module may: according to the data amount of the R table, the data amount of the S table, the resident attribute of the R table, the streaming attribute of the S table, and the available memory of the coprocessor Size, determines the memory size of the resident area in the coprocessor and the memory size of the flow area.

Optionally, since the R table and the S table are both flow attributes, the control module may not set a resident area in the coprocessor, and all available memory in the coprocessor is used to set a flow area, Improve memory utilization.

S503. The control module performs an operation on the R table to obtain an R table hash table.

S504. The control module divides the R table hash table and the S table into a plurality of sub-R table hash tables and a plurality of sub-S tables, and each of the plurality of sub-R table hash tables has a hash table and a hash table. Each of the plurality of S tables has a one-to-one correspondence.

The plurality of sub-S tables may be, for example, an S1 table or an S2 table in FIG. 8, and the plurality of sub-R table hash tables may be, for example, an R1 table hash table or an R2 table hash table in FIG. 8.

S505. The control module sends the jth sub-R table hash table and the j-th sub-S table in the plurality of sub-R table hash tables to the i-th sub-streaming area of the coprocessor.

Optionally, the main processor may include a first buffer area and a second buffer area, and the control module may use the first buffer area and the second buffer area to list each of the plurality of sub-R tables and the same The child S table corresponding to each child R table is sent to the flow area of the coprocessor.

S506. The coprocessor performs hash connection processing on the jth sub-R table hash table and the j-th sub-S table in the i-th sub-streaming area.

S507. The coprocessor identifies the i-th sub-streaming area as idle.

Optionally, the coprocessor may include a plurality of consecutive sub-flow regions, and the coprocessor may sequentially read the sub-R table in each of the consecutive plurality of sub-flow regions and corresponding to the sub-R table. The child S table, and the child S table and the child R table in each sub-flow region are processed.

S508. The control module sends the j+1th sub-R table hash table and the j+1th sub-S table in the plurality of sub-R table hash tables to the i+1th sub-streaming area of the coprocessor.

S509. The coprocessor performs hash connection processing on the j+1th sub-R table hash table and the j+1th sub-S table in the i+1th sub-flow region.

S510. The coprocessor represents the i+1th sub-streaming area as idle.

S511. The coprocessor according to the plurality of sub-R table hash tables in each of the sub-R table hash tables and the plurality of sub-S tables The hash connection result of each sub-S table determines the hash processing result of the R table and the S table.

Specifically, the coprocessor can obtain the data connection processing result corresponding to each sub S table in the plurality of sub S tables according to the method described in S505-S510, and further obtain the data connection processing of the R table and the S table. As a result, the data connection processing result of the R table and the S table is transmitted to the main processor.

S512. The coprocessor sends the R table and the hash processing result of the S table to the main processor through the control module.

The data processing method of the embodiment of the present application completes data transmission and data processing between the main processor and the coprocessor through a control module between the main processor and the coprocessor, thereby reducing the workload of the main processor and simplifying System code that would integrate data processing, data transfer, and memory management.

FIG. 6 shows a schematic flowchart of still another data processing method 600 of the embodiment of the present application. The method 600 can be applied to the system as shown in FIG. 2, and the method flow of the data processing system in the hash connection processing in the fact table and the plurality of dimension tables of the fact table is described in detail in conjunction with FIG. .

It should be understood that the fact table is used to store at least one fact record, each fact record corresponds to a row in the fact table, and includes a key value column and a metric value column, wherein the value in the key value column corresponds to the dimension of the fact record, for example, Commodity origin, commodity price, merchandise quantity, transaction date, merchandise category, merchandise name, etc.; the value in the metric column corresponds to the subject matter of the fact table, for example, sales or sales volume, and so on.

It should also be understood that the dimension table is used to store the dimensional characteristics of the fact record. The dimension table may include a name column and an attribute column. For example, the commodity origin includes three columns in the dimension table, a place name (locationName), a place number (locationId), and Address, where the origin name is the name column, and the origin number and address are attribute columns. The key-value column in the fact table can correspond to the attribute column in the dimension table, and each key-value column in the fact table can correspond to a dimension table.

The primary key, which uniquely identifies a column in the table, any two rows in a table have different primary key values, and the primary key does not allow null values. In general, the primary key of a table is the first column of the table.

Foreign key, used to join two tables, if the column M in the table A corresponds to the column N in the table B, that is, the column M and the column N correspond to the same attribute, for example, both correspond to the order number, and the column M For the primary key in Table A, column N is the foreign key in Table B. The foreign key of the fact table corresponds to the primary key of the dimension table.

S601. The control module receives a fact table and a plurality of dimension tables sent by the main processor, where the fact table is associated with each dimension table of the plurality of dimension tables, and the data volume of the fact table is greater than the coprocessor. The amount of memory available, the total amount of data for the multiple dimension tables that have been hashed is less than the amount of memory available to the coprocessor.

Optionally, the plurality of dimension tables may be, for example, the first dimension table and the second dimension table in FIG. 9 .

S602. The control module identifies the multiple dimension table as a resident attribute according to the data volume of the multiple dimension tables, the data volume of the fact table, the memory size available to the coprocessor, and the processing context, and the fact The table is identified as a flow attribute.

Specifically, the control module may send the plurality of dimension tables to the resident area of the coprocessor in one time, and send the fact table in batches to the flow area of the coprocessor.

Optionally, before the control module sends the multiple dimension table and the fact table to the coprocessor, the control module may perform, according to the connection filter, each dimension table of the plurality of dimension tables and the fact table. Filtering, filtering out the records that are not related to the current hash connection, and reducing the amount of data of the fact table and the dimension table that need to be transmitted, but the embodiment of the present application does not limit this.

S603. The control module performs hash processing on each dimension table in the plurality of dimension tables to obtain each dimension table hash table in the plurality of dimension tables.

Optionally, the multiple dimension table hash table may be, for example, the first dimension table hash table and the second dimension table hash table in FIG. 9, and the first dimension table is hashed to obtain the first A dimension table hash table, the second dimension table is hashed to obtain the second dimension table hash table.

S604. The control module transmits the plurality of dimension table hash tables to a resident area of the coprocessor.

Optionally, the control module may simultaneously transmit the plurality of dimension table hash tables to the coprocessor, or may sequentially transmit each dimension table hash table in the plurality of dimension table hash tables to the association. The processor is not limited in this embodiment.

Optionally, before the S604, the control module may: according to the data volume of the multiple dimension tables, the data volume of the fact table, the resident attribute of the multiple dimension tables, the flow attribute of the fact table, and the co-processing The amount of memory available to the device determines the memory size of the resident area in the coprocessor and the memory size of the flow area.

Optionally, the coprocessor may store the multiple dimension table hash table in the resident storage area of the coprocessor, or may be based on the memory size of the resident storage area and the multiple dimension table hash table The amount of data is divided into a plurality of sub-resident areas, and the plurality of dimension table hash tables are respectively stored in the plurality of sub-resident areas, which is not limited in this embodiment of the present application. .

S605. The control module performs compression processing on the fact table to obtain a fact compression table.

Alternatively, as shown in FIG. 9, the fact table is subjected to compression processing to obtain a fact compression table.

Optionally, the control module may further perform data compression processing on the fact table filtered by the connection filter, thereby further reducing the data amount of the fact table.

S606. The control module divides the fact compression table into a plurality of sub-facts compression tables.

S607. The control module sends the jth sub-facts compression table in the plurality of fact compression tables to the i-th sub-streaming area of the coprocessor.

Specifically, the main processor includes a first buffer area and a second buffer area, and the control module may send, by using the first buffer area and the second buffer area, each sub-facts compression table in the plurality of sub-facts compression tables to the Coprocessor flow area.

Optionally, the control module may divide the fact compression table into a plurality of sub-facts compression table according to the memory size of the buffer area, which is not limited in this embodiment of the present application.

S608, the coprocessor hash-joins the j-th sub-fact compression table in the i-th sub-flow region and each dimension table hash table in the plurality of dimension table hash tables to obtain the j-th sub-fact The result of the hash join corresponding to the compressed table.

S609. The coprocessor identifies the i-th sub-streaming area as idle.

Optionally, the coprocessor may include a plurality of consecutive sub-flow regions, and the coprocessor may sequentially read the sub-fact compression table in each of the consecutive plurality of sub-flow regions, and compress the sub-facts The table is hashed with each dimension table hash table in multiple dimension table hash tables in the residency region.

S610. The control module sends the j+1th sub-facts compression table in the plurality of fact compression tables to the (i+1)th sub-streaming area of the coprocessor.

S611, the coprocessor hash-joins the j+1th sub-fact compression table in the i+1th sub-flow region and each dimension table hash table in the plurality of dimension table hash tables, to obtain the The hash join result corresponding to the j+1th sub-fact compression table.

S612. The coprocessor identifies the i+1th sub-streaming area as idle.

S613. The coprocessor compresses the hash connection node corresponding to each sub-fact compression table in the plurality of sub-facts compression table. If the result of the connection between the plurality of dimension tables and the fact table is obtained.

Specifically, the coprocessor can obtain the data connection processing result corresponding to each sub-facts compression table in the plurality of sub-facts compression tables according to the method described in S607-S612, and further obtain the fact table and the plurality of dimension tables. The data connection processing result of each dimension table in the medium is transmitted to the main processor by the result of the data connection processing of the fact table and each dimension table in the plurality of dimension tables.

S614. The coprocessor sends the fact table and the hash connection result of each dimension table in the plurality of dimension tables to the main processor through the control module.

In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in a processor or an instruction in a form of software. The steps of the method disclosed in the embodiments of the present application may be directly implemented as a hardware processor, or may be performed by a combination of hardware and software modules in the processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in a memory, and the processor executes instructions in the memory, in combination with hardware to perform the steps of the above method. To avoid repetition, it will not be described in detail here.

Those skilled in the art will appreciate that the various method steps and elements described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, in order to clearly illustrate hardware and software. Interchangeability, the steps and composition of the various embodiments have been generally described in terms of function in the foregoing description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. Different methods may be used to implement the described functionality for each particular application, but such implementation should not be considered to be beyond the scope of the application.

A person skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, or an electrical, mechanical or other form of connection.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the embodiments of the present application.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application is essential or the part contributing to the prior art, or all or part of the technical solution may be in the form of a software product. The computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all of the methods described in the various embodiments of the present application or Part of the steps. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program code. .

The foregoing is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any equivalents can be easily conceived by those skilled in the art within the technical scope disclosed in the present application. Modifications or substitutions are intended to be included within the scope of the present application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

A data processing system, comprising: a main processor, a coprocessor, and a control module;

The main processor is configured to send data to be processed to the control module;

The control module is configured to receive the to-be-processed data sent by the main processor, and send the to-be-processed data to the coprocessor;

The coprocessor is configured to receive the to-be-processed data sent by the control module, process the to-be-processed data, obtain a processing result of the to-be-processed data, and pass the processing result of the to-be-processed data The control module sends to the main processor;

The main processor is further configured to receive a processing result of the to-be-processed data sent by the coprocessor through the control module.
The data processing system according to claim 1, wherein the control module is configured to send the to-be-processed data to the coprocessor at one time, or send the to-be-processed data in batches to the Coprocessor

The control module is further configured to: when the to-be-processed data is sent to the coprocessor, carry a storage attribute of the to-be-processed data, where the storage attribute includes a resident attribute or a flow attribute, where the resident attribute represents The to-be-processed data can be accessed multiple times, and the streaming attribute indicates that the to-be-processed data can only be accessed once.
The data processing system according to claim 2, wherein the control module is specifically configured to:

Determining the to-be-processed according to a processing context of the to-be-processed data, an amount of data of the to-be-processed data, and a memory size available to the coprocessor before transmitting the to-be-processed data to the coprocessor The storage properties of the data.
The data processing system according to claim 2 or 3, wherein the control module is further configured to:

Determining the co-processing according to the data amount of the to-be-processed data, the storage attribute of the to-be-processed data, and the available memory size of the coprocessor before transmitting the to-be-processed data to the coprocessor The memory size of the resident area in the memory of the device and the memory size of the streaming area, wherein the data stored in the resident area can be accessed multiple times, the data in the streaming area can only be accessed once, and Deleted or overwritten after being accessed.
The data processing system according to claim 4, wherein the flow area of the coprocessor comprises a plurality of consecutive sub-flow areas;

The control module is specifically configured to:

And sending the to-be-processed data in batches to each of the consecutive plurality of sub-flow regions;

The coprocessor is specifically used to:

And sequentially reading data in each of the plurality of sub-flow regions, and processing data in each of the sub-flow regions, after processing the data in each of the sub-flow regions, Each of the sub-flow areas is identified as idle;

After the data processing in the last sub-streaming area of the consecutive plurality of sub-flow areas is completed, and the first sub-streaming area of the consecutive plurality of sub-flow areas is identified as idle, continuing from the continuous The first sub-streaming area of the plurality of sub-flow areas begins to read data.
The data processing system according to any one of claims 2 to 5, wherein the control module is specifically configured to:

If the to-be-processed data includes the first data block and the second data block, the data amount of the first data block is smaller than the memory size available to the coprocessor, and the data amount of the second data block is greater than the Coprocessor available Save size,

Determining that a storage attribute of the first data block is a resident attribute, and transmitting the first data block to a resident area of the coprocessor

Determining, that the storage attribute of the second data block is a flow attribute, and dividing the second data block into a plurality of second sub-blocks sent to the flow area of the co-processor in batches.
The data processing system according to claim 6, wherein the coprocessor is specifically configured to:

Processing the first data block in the resident area and each of the plurality of second sub-data blocks in the flow area to obtain the first data block and a processing result of each of the second sub-blocks;

And processing result of the first data block and the second data block according to a processing result of each of the first data block and each of the plurality of second sub data blocks.
The data processing system according to any one of claims 2 to 7, wherein the control module is further configured to:

If the to-be-processed data includes a first data block and a second data block, the data amount of the second data block is greater than a memory size available to the coprocessor, and the hashed first data block The amount of data is greater than the amount of memory available to the coprocessor.

Determining, that the storage attribute of the first data block is a flow attribute, and dividing the first data block into a plurality of first sub-blocks sent to the flow area of the coprocessor in batches,

Determining, that the storage attribute of the second data block is a flow attribute, and dividing the second data block into a plurality of second sub-blocks sent to the flow area of the coprocessor in batches.
The data processing system of claim 8 wherein said coprocessor is further configured to:

Processing each of the first one of the plurality of first sub-blocks and the second of the plurality of second sub-blocks in the flow area to obtain each of the Processing results of the first sub-block and the each of the second sub-blocks;

Obtaining the first data block and the first data block according to a processing result of each of the first one of the plurality of first sub-blocks and each of the plurality of second sub-blocks The processing result of the second data block.
The data processing system according to any one of claims 2 to 9, wherein the main processor comprises a first buffer area and a second buffer area;

The control module is specifically configured to:

And storing the to-be-processed data in the first buffer area, and after the first buffer area is full, sending data in the first buffer area to the coprocessor;

The remaining pending data is further stored to the second buffer area, and after the second buffer area is full, the data in the second cache is sent to the coprocessor.
The data processing system according to claim 10, wherein the control module is further configured to continue to send the data to be processed while transmitting data in the first buffer area to the coprocessor Storing to the second buffer area, and after the second buffer area is full, transmitting data in the second buffer area to the coprocessor.
A data processing method, which is applied to a data processing system, the data processing system includes a main processor, a coprocessor, and a control module, and the data processing method includes:

The main processor sends the to-be-processed data to the control module;

The control module receives the to-be-processed data sent by the main processor, and sends the to-be-processed data to the coprocessor;

The coprocessor receives the to-be-processed data sent by the control module, processes the to-be-processed data, obtains a processing result of the to-be-processed data, and passes the processing result of the to-be-processed data through the Control module is sent to the main processor;

The main processor receives a processing result of the to-be-processed data sent by the coprocessor through the control module.
The data processing method according to claim 12, wherein the control module sends the to-be-processed data to the coprocessor, including:

Sending the to-be-processed data to the coprocessor at one time, or sending the to-be-processed data to the coprocessor in batches, where

And storing, when the data to be processed is sent to the coprocessor, a storage attribute of the to-be-processed data, where the storage attribute includes a resident attribute or a flow attribute, where the resident attribute indicates that the to-be-processed data can be With multiple accesses, the flow attribute indicates that the pending data can only be accessed once.
The data processing method according to claim 13, wherein before the controlling module sends the to-be-processed data to the coprocessor, the method further includes:

The control module determines a storage attribute of the to-be-processed data according to a processing context of the to-be-processed data, an amount of data of the to-be-processed data, and a memory size available to the coprocessor.
The data processing method according to claim 13 or 14, wherein before the sending the to-be-processed data to the coprocessor, the method further comprises:

Determining, by the control module, a memory size of a resident area in a memory of the coprocessor according to an amount of data of the to-be-processed data, a storage attribute of the to-be-processed data, and a memory size available to the coprocessor The memory size of the streaming area, wherein the data stored in the resident area can be accessed multiple times, the data in the streaming area can only be accessed once, and deleted or overwritten after being accessed.
The data processing method according to claim 15, wherein the flow area of the coprocessor comprises a plurality of consecutive sub-flow areas;

The control module sends the to-be-processed data to the coprocessor in batches, including:

The control module sequentially sends the to-be-processed data in batches to each of the consecutive plurality of sub-flow regions;

Processing, by the coprocessor, the data to be processed, and obtaining the processing result of the data to be processed, including:

The coprocessor sequentially reads data in each of the plurality of sub-flow regions, and processes data in each of the sub-stream regions, and data in each of the sub-flow regions After the processing is completed, each of the sub-flow areas is identified as idle;

After the data processing in the last sub-streaming area of the consecutive plurality of sub-flow areas is completed, and the first sub-streaming area of the consecutive plurality of sub-flow areas is identified as idle, the coprocessor continues to The first of the plurality of consecutive sub-flow regions begins to read data.
The data processing method according to any one of claims 13 to 16, wherein the control module sends the to-be-processed data to the coprocessor, including:

If the to-be-processed data includes the first data block and the second data block, the data amount of the first data block is smaller than the memory size available to the coprocessor, and the data amount of the second data block is greater than the Coprocessor available Save size,

The control module determines that a storage attribute of the first data block is a resident attribute, and sends the first data block to a resident area of the coprocessor,

The control module determines that the storage attribute of the second data block is a flow attribute, and divides the second data block into a plurality of second sub-blocks that are sequentially sent to the flow area of the coprocessor in batches.
The data processing method according to claim 17, wherein the coprocessor processes the data to be processed to obtain a processing result of the data to be processed, including:

Processing, by the coprocessor, the first data block in the resident area and each of the plurality of second sub data blocks in the flow area to obtain the a processing result of the first data block and each of the second sub-blocks;

Processing, by the coprocessor, processing of the first data block and the second data block according to processing results of each of the first data block and each of the plurality of second sub data blocks result.
The data processing method according to any one of claims 13 to 18, wherein the control module sends the to-be-processed data to the coprocessor, including:

If the to-be-processed data includes a first data block and a second data block, the data amount of the second data block is greater than a memory size available to the coprocessor, and the hashed first data block The amount of data is greater than the amount of memory available to the coprocessor.

The control module determines that the storage attribute of the first data block is a flow attribute, and divides the first data block into a plurality of first sub-blocks that are sent in batches to a flow area of the coprocessor,

The control module determines that the storage attribute of the second data block is a flow attribute, and divides the second data block into a plurality of second sub-blocks that are sent in batches to a flow area of the coprocessor.
The data processing method according to claim 19, wherein the coprocessor processes the data to be processed to obtain a processing result of the data to be processed, including:

Processing, by the coprocessor, each of the plurality of first subblocks and the second of the plurality of second subblocks in the streaming region Obtaining a processing result of each of the first sub-blocks and each of the second sub-blocks;

The coprocessor obtains the first according to a processing result of each of the first sub-blocks of the plurality of first sub-blocks and each of the second sub-blocks of the plurality of second sub-blocks The processing result of the data block and the second data block.
The data processing method according to any one of claims 13 to 20, wherein the main processor comprises a first buffer area and a second buffer area;

Before the control module sends the to-be-processed data to the coprocessor, the method further includes:

The control module stores the to-be-processed data to the first buffer area until the first buffer area is full;

After the control module saves the first buffer area, storing the remaining data to be processed to the second buffer area until the second buffer area is full;

The control module sends the to-be-processed data to the coprocessor, including:

The control module sends data in the first buffer area to the coprocessor;

The control module sends data in the second buffer to the coprocessor after transmitting data in the first cache to the coprocessor.
The data processing method according to claim 21, wherein said control module treats said to be in place The data is sent to the coprocessor, including:

While transmitting data in the first buffer to the coprocessor, the control module continues to store remaining data to be processed to the second buffer and saves in the second buffer After being full, the data in the second buffer area is sent to the coprocessor.