WO2023125463A1

WO2023125463A1 - Heterogeneous computing framework-based processing method and apparatus, and device and medium

Info

Publication number: WO2023125463A1
Application number: PCT/CN2022/142134
Authority: WO
Inventors: 罗恒锋
Original assignee: 北京字跳网络技术有限公司
Priority date: 2021-12-28
Filing date: 2022-12-27
Publication date: 2023-07-06
Also published as: CN116360971A

Abstract

The embodiments of the present disclosure relate to a heterogeneous computing framework-based processing method and apparatus, and a device and a medium. The method comprises: creating a corresponding application program interface (API) for each algorithm module in at least one heterogeneous processing unit, wherein the heterogeneous processing unit is generated by encapsulating at least one algorithm module; according to a data processing task, calling an interface of the at least one heterogeneous processing unit to create a heterogeneous computing engine framework; and according to the heterogeneous computing engine framework, calling an API of a required algorithm module to execute the data processing task, and then outputting a task processing result to a target device.

Description

Processing method, device, equipment and medium based on heterogeneous computing framework

Cross References to Related Applications

This application is based on the Chinese application with the application number 202111629485.7 and the filing date is December 28, 2021, and claims its priority. The disclosure content of the Chinese application is hereby incorporated into this application as a whole.

technical field

The present disclosure relates to the field of computer technology, and in particular to a processing method, device, device and medium based on a heterogeneous computing framework.

Background technique

With the diversification and complexity of business requirements, the demand for logical computing power of the computing system is getting higher and higher. However, the current computing system is directly implemented on the algorithm module according to the business requirements of the upper layer, such as the CPU algorithm module and the GPU algorithm module. . If the upper-layer business needs to expand or change the algorithm implementation, the implementation between the algorithm module and the upper layer should also be modified synchronously.

Contents of the invention

The disclosure provides a processing method, device, device and medium based on a heterogeneous computing framework.

In a first aspect, an embodiment of the present disclosure provides a processing method based on a heterogeneous computing framework, the method including:

Create a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each said heterogeneous processing unit is generated by encapsulating at least one algorithm module;

Create a heterogeneous computing engine framework by invoking an interface of at least one heterogeneous processing unit according to a data processing task, wherein the heterogeneous computing engine framework includes at least one data input terminal, at least one said heterogeneous processing unit, and at least one data output terminal; and

Call the application program interface API of the required algorithm module according to the heterogeneous computing engine framework to execute the data processing task, and output the task processing result to the target device.

In some embodiments, creating a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit includes:

Create a calling interface and task running interface corresponding to each algorithm module.

In some embodiments, the algorithm module includes: any one or more of a central processing unit CPU, a digital signal processing module DSP, an image processor module GPU, an application specific integrated circuit module ASIC, and a field programmable logic gate array module FPGA a combination.

In some embodiments, the method also includes:

performing segmentation processing on the input data of the heterogeneous processing unit to generate multiple data blocks;

Establishing multiple computing threads corresponding to the multiple data blocks in the heterogeneous processing unit, wherein the multiple computing threads are used to execute the multiple data blocks in parallel to generate corresponding multiple data processing results; as well as

Merge the multiple data processing results to generate output data corresponding to the heterogeneous processing units.

In some embodiments, the method also includes:

determining a plurality of subtasks corresponding to the data processing task, and establishing a corresponding subflow chart for each of the subtasks;

Simultaneously input the data of the at least one data input terminal into multiple sub-flowcharts corresponding to the multiple sub-tasks, wherein the multiple sub-flowcharts are used to perform heterogeneous calculations in parallel to generate multiple sub-task results; and

All the multiple subtask results are provided to the data output terminal, wherein the data output terminal is used to combine the multiple subtask results.

In some embodiments, the method also includes:

A self-defined algorithm node is inserted into the framework of the heterogeneous computing engine, wherein the self-defined algorithm node is connected to the data input end or the output end of the heterogeneous processing unit, and the self-defined algorithm node is used Calculate the data provided by the data input terminal or the output terminal of the heterogeneous processing unit according to a self-defined algorithm, and feed back the calculated result to the data input terminal or the heterogeneous processing unit In the next node connected to the output terminal of .

In some embodiments, the method also includes:

A position termination node is inserted into the framework of the heterogeneous computing engine, wherein the position termination node is connected to the output end of the heterogeneous processing unit, and the position termination node is used to stop the output end of the heterogeneous processing unit Subsequent processing nodes continue to calculate, and output the data provided by the output terminals of the heterogeneous processing units as processing results.

In a second aspect, an embodiment of the present disclosure further provides a processing device based on a heterogeneous computing framework, the device comprising:

A first creating module, configured to create a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each of the heterogeneous processing units is generated by encapsulating at least one algorithm module;

The second creation module is used to create a heterogeneous computing engine framework by invoking the interface of at least one heterogeneous processing unit according to the data processing task, wherein the heterogeneous computing engine framework includes at least one data input terminal, and at least one of the heterogeneous processing units unit, and at least one data output; and

The acquisition module is used to call the application program interface API of the required algorithm module according to the heterogeneous computing engine framework to perform the data processing task, and output the task processing result to the target device.

In a third aspect, the present disclosure provides a computer-readable storage medium, where computer programs/instructions are stored in the computer-readable storage medium, and when the computer programs/instructions are executed by a processor, the processor can realize the above method.

In a fourth aspect, the present disclosure provides an electronic device, including: a processor; and a memory for storing instructions executable by the processor; the processor is used for reading the instructions from the memory, and Execute the above method.

In a fifth aspect, the present disclosure provides a computer program product, where the computer program product includes a computer program/instruction, and when the computer program/instruction is executed by a processor, the above method is implemented.

In a sixth aspect, the present disclosure provides a computer program, where the computer program includes instructions, and when the instructions are executed by a processor, the above method is implemented.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

FIG. 1 is a schematic flowchart of a processing method based on a heterogeneous computing framework provided by an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a heterogeneous processing unit provided by an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a heterogeneous computing engine framework provided by an embodiment of the present disclosure;

FIG. 4 is a schematic flowchart of another processing method based on a heterogeneous computing framework provided by an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of another heterogeneous computing engine framework provided by an embodiment of the present disclosure;

FIG. 6 is a schematic flowchart of another processing method based on a heterogeneous computing framework provided by an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of another heterogeneous computing engine framework provided by an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of inserting a custom algorithm node provided by an embodiment of the present disclosure;

FIG. 9 is a schematic diagram of an insertion position termination node provided by an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of inserting a custom algorithm node and a position termination node provided by an embodiment of the present disclosure;

FIG. 11 is a schematic structural diagram of a processing device based on a heterogeneous computing framework provided by an embodiment of the present disclosure; and

Fig. 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this respect.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more" multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

The current computing system has low computing efficiency, takes a long time to calculate, is not suitable for expansion, is not flexible, and cannot meet the computing power requirements of the current business.

Compared with related technologies, the technical solutions provided by the embodiments of the present disclosure have the following advantages:

To sum up, the processing method based on the heterogeneous computing framework of the embodiment of the present disclosure creates a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each of the heterogeneous processing units is implemented through Generated by encapsulating at least one algorithm module; calling at least one interface of a heterogeneous processing unit according to a data processing task to create a heterogeneous computing engine framework, wherein the heterogeneous computing engine framework includes at least one data input terminal, and at least one of the heterogeneous A processing unit, and at least one data output terminal; call the application program interface API of the required algorithm module according to the heterogeneous computing engine framework to execute the data processing task, and output the task processing result to the target device. In the embodiment of the present disclosure, a heterogeneous computing engine framework is established through a heterogeneous processing unit according to a data processing task, and the data processing task is connected through the heterogeneous computing engine framework. Adjust the heterogeneous computing engine framework to meet the corresponding changes, so it is suitable for expansion and has strong flexibility. It can improve processing efficiency for large-scale algorithm systems, and integrates the characteristics of different algorithm modules in heterogeneous processing units through the heterogeneous computing engine framework. , so as to take advantage of the computing power of heterogeneous processing units, improve computing efficiency, reduce computing time, and improve the performance of heterogeneous processing units, thereby meeting the current business needs for computing power. At the same time, the hierarchical design of the architecture, the modular design, the cascading of algorithms, and the intelligent scheduling of hybrid computing are realized through the heterogeneous computing engine framework, thereby improving the efficiency of the framework design.

In order to solve the above problems, an embodiment of the present disclosure provides a processing method based on a heterogeneous computing framework, which will be introduced in conjunction with specific embodiments below.

FIG. 1 is a schematic flowchart of a processing method based on a heterogeneous computing framework provided by an embodiment of the present disclosure. The method can be executed by a processing device based on a heterogeneous computing framework, where the device can be implemented by software and/or hardware. Generally, Can be integrated in electronic equipment. As shown in Figure 1, the method includes:

Step 101, create a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each heterogeneous processing unit is generated by encapsulating at least one algorithm module.

In the embodiments of the present disclosure, the data processing task is implemented based on heterogeneous processing units, and the number of heterogeneous processing units implementing the data processing task may be one or multiple. The heterogeneous processing unit is an abstract execution layer for data processing tasks. At least one algorithm module is encapsulated in the heterogeneous processing unit. If there are multiple algorithm modules encapsulated in a heterogeneous processing unit, the types of the multiple algorithm modules can be The same can also be different. Wherein, the differences of different types of algorithm modules include but are not limited to: one or more of different process architectures, different instruction sets, and different functions.

For example, the algorithm module includes: a central processing unit (Central Processing Unit, CPU), a digital signal processing module (Digital Signal Processing, DSP), an image processor module (Graphics Processing Unit, GPU), an application-specific integrated circuit module ( Application Specific Integrated Circuit, ASIC), field programmable logic gate array module (Field Programmable Gate Array, FPGA). A heterogeneous processing unit may be packaged with any one of the above-mentioned algorithm modules, or may be packaged with any number of the above-mentioned algorithm modules. For example, a CPU, GPU, and DSP can be packaged in a heterogeneous processing unit.

In the embodiment of the present disclosure, it is necessary to create a corresponding application program interface (Application Programming Interface, API) for each algorithm module in advance, and the application program interface includes: a call interface (backend interface) capable of invoking the corresponding algorithm module and capable of controlling the corresponding The task running interface (runtime interface) that the algorithm module runs. By calling the application programming interface, the algorithm module can be operated normally, so as to realize the data processing task on this basis.

FIG. 2 is a schematic diagram of a heterogeneous processing unit provided by an embodiment of the present disclosure. Referring to FIG. 2 , Backend represents an invocation interface, Runtime represents a task execution interface, and Process represents a heterogeneous processing unit. It should be noted that the subsequent embodiments And in the accompanying drawings, for simplicity of expression, the involved heterogeneous processing units may also be simply described as Process. Assume that the image recognition function is implemented through a Process, which can be packaged with GPU and CPU. Before performing heterogeneous computing, it is necessary to create the GPU Backend interface and GPU Runtime interface corresponding to the GPU, and create the CPU Backend interface and CPU Runtime interface corresponding to the CPU. . Wherein, the GPU Backend interface and the CPU Backend interface are included in the Backend of Figure 2; the GPU Runtime interface and the CPU Runtime interface are included in the Runtime of Figure 2.

Step 102, call at least one heterogeneous processing unit interface to create a heterogeneous computing engine framework according to the data processing task, wherein the heterogeneous computing engine framework includes at least one data input terminal, at least one heterogeneous processing unit, and at least one data output terminal .

It is understandable that different algorithm modules have corresponding task types with high processing efficiency. For example, GPU has high efficiency when processing image type tasks, and DSP has high efficiency when processing digital signal type tasks. Therefore, in order to release the computing power of each algorithm module According to the data processing tasks and the characteristics of each algorithm module, the interface of at least one heterogeneous processing unit can be called to create a heterogeneous computer engine framework.

It should be noted that in the embodiment of the present disclosure, the heterogeneous computing engine framework includes one or more data input ports for input data and one or more data output ports for output data, and at least one heterogeneous Processing unit, the at least one heterogeneous processing unit may be packaged with one or more algorithm modules, for example, a heterogeneous computing engine framework may include a heterogeneous processing unit, and the one heterogeneous processing unit is packaged with two or, the framework of the heterogeneous computing engine may include two heterogeneous processing units, each of which is packaged with an algorithm module, and the types of the two algorithm modules are different.

Among them, the connection relationship between the data input terminal, heterogeneous processing unit and data output terminal in the above-mentioned heterogeneous computing engine framework can be matched and set according to the data processing task, etc., so when the data processing task changes, it is necessary to expand or change the algorithm to achieve In the case of , the data processing task can be realized by adjusting the heterogeneous computing engine framework. For example, the cascade processing of multiple algorithm modules can be realized through the connection relationship in the heterogeneous computing engine framework, and the multiplexing of algorithm modules can also be realized through the connection relationship in the heterogeneous computing engine framework (for example: image pre-processing and image post-processing). Reuse of relevant algorithms in processing), etc. In addition, the number of heterogeneous processing units in the heterogeneous computing engine framework can be determined according to the performance of a single heterogeneous processing unit, compatibility of devices, characteristics of algorithms executed by the heterogeneous processing unit, and the like. Embodiments of the present disclosure do not limit the aforementioned heterogeneous computing engine framework.

FIG. 3 is a schematic diagram of a heterogeneous computing engine framework provided by an embodiment of the present disclosure. For a clearer description, refer to FIG. 3. The heterogeneous computing engine framework includes a data input terminal and a data output terminal, and 7 Process, wherein part of the Process is packaged with a CPU, and another part of the Process is packaged with a GPU. If image processing is performed based on the heterogeneous computing engine framework, due to the image pre-processing part (Process2 and Process3) and image post-processing ( Process5 and Process6) can be reused, so Process2 can be the same as Process5, and Process3 can be the same as Process6. Specifically, the data transmission direction in the framework of the heterogeneous computing engine is from the data input end to Process1, from Process1 to Process2 and Process4, from Process2 to Process3, from Process3 and Process1 to Process4, from Process4 to Process5 and Process7, from Process5 to Process6, from Process6 and Process4 to Process7, from Process7 to data output.

Step 103, call the application program interface API of the required algorithm module according to the framework of the heterogeneous computing engine to execute the data processing task, and output the task processing result to the target device.

After the heterogeneous computing engine framework is determined through the above steps, the business processing results of data processing tasks can be obtained based on the heterogeneous computing engine framework. Specifically, it is necessary to call the application program interface corresponding to the required algorithm module according to the heterogeneous computing engine framework The API performs data processing tasks, thereby realizing the orderly scheduling of the operation of the algorithm modules, and then outputs the task processing results to the target device. Wherein, the target device may be selected according to the application scenario, which is not limited in the embodiment of the present disclosure, for example, a mobile phone, a microcomputer, and the like.

Take the realization of data processing tasks through a Process as an example. The Process is packaged with CPU and GPU. If the CPU needs to be used according to the framework of the heterogeneous computing engine, the CPU Backend interface is called to start the CPU, and the CPU Runtime interface is called to control the operation of the CPU. , similarly, if the GPU needs to be used and run according to the framework of the heterogeneous computing engine, the GPU Backend interface and the GPU Runtime interface should be called. The order of calling is determined and will not be repeated here.

To sum up, the processing method based on the heterogeneous computing framework of the embodiment of the present disclosure creates a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each heterogeneous processing unit is packaged by at least Generated by an algorithm module; call at least one heterogeneous processing unit interface to create a heterogeneous computing engine framework according to the data processing task, wherein the heterogeneous computing engine framework includes at least one data input terminal, at least one heterogeneous processing unit, and at least one Data output terminal: call the application program interface API of the required algorithm module according to the heterogeneous computing engine framework to perform data processing tasks, and output the task processing results to the target device.

In related technologies, according to the data processing tasks, the corresponding algorithms are directly run on the algorithm modules such as CPU and GPU, and the data processing tasks are connected by the specific algorithms. If the algorithm needs to be expanded or changed, the realization of the data processing tasks is also Synchronous modification is required, so it is not suitable for expansion, inflexible, and the processing efficiency for large-scale algorithm systems is very low.

In the embodiment of the present disclosure, a heterogeneous computing engine framework is established through heterogeneous processing units according to data processing tasks, and the data processing tasks are connected through the heterogeneous computing engine framework. The application program interface between the computing engine framework and data processing tasks does not need to be modified synchronously, so it is suitable for expansion and has strong flexibility. It can improve the processing efficiency for large-scale algorithm systems, and integrates heterogeneous processing units through the heterogeneous computing engine framework. The characteristics of different algorithm modules in the system can take advantage of the computing power of heterogeneous processing units, improve computing efficiency, reduce computing time, and improve the performance of heterogeneous processing units, so as to meet the current business needs for computing power.

At the same time, the hierarchical design of the architecture, the modular design, the cascading of algorithms, and the intelligent scheduling of hybrid computing are realized through the heterogeneous computing engine framework, thereby improving the efficiency of the framework design.

Based on the above-mentioned embodiments, in order to further improve the computing capability of the heterogeneous computing framework, the method of block multi-backend parallel execution can be adopted, and the specific description is as follows: FIG. 4 is another processing based on the heterogeneous computing framework provided by the embodiment of the present disclosure The schematic flow chart of the method, as shown in Figure 4, also includes the following steps in the method:

Step 401, performing split processing on the input data of the heterogeneous processing units to generate multiple data blocks.

In the embodiment of the present disclosure, in order to enable the algorithm module to run its suitable business, thereby improving the processing efficiency of the input data by the heterogeneous processing units, the input data is segmented. In some embodiments, the input data may be divided according to the data type of the input data to obtain corresponding data blocks, and the data in the same data block obtained by the division may belong to the same data type.

Step 402, establishing multiple computing threads corresponding to multiple data blocks in the heterogeneous processing unit, wherein the multiple computing threads are used to execute multiple data blocks in parallel to generate multiple corresponding data processing results.

In the embodiment of the present disclosure, multiple calculation threads corresponding to the data block can be established, so as to call the application program interface (for example: Backend interface and Runtime interface) based on the thread, and process the data block based on the thread, for Each data block generates a corresponding data processing result.

Step 403, combining and processing multiple data processing results to generate output data corresponding to heterogeneous processing units.

After the multiple data processing results are obtained, the multiple data processing results may be merged to generate output data corresponding to the heterogeneous processing unit.

For example, as shown in FIG. 5, the input data is divided into two data blocks, the data related to digital signal processing is divided into the third data block, and other types of data are divided into The data is divided into first data blocks. Set up the first calculation thread Thread1 for processing the first data block, call the first calling interface Backend1 and the first task running interface Runtime1 based on Thread1, thereby run the CPU to process the first data block, and generate the first data processing result; establish The second calculation thread Thread2 for processing the second data block calls the second calling interface Backend2 and the second task running interface Runtime2 based on Thread2, thereby running the GPU to process the second data block and generating the second data processing result; The third computing thread Thread3 for processing the third data block calls the third calling interface Backend3 and the third task running interface Runtime3 based on Thread3, thereby running the DSP to process the third data block and generating a third data processing result. It should be noted that the above Thread1, Thread2, and Thread3 may be executed in parallel in the Process, so as to improve the efficiency of input data processing. Further, the first data processing result, the second data processing result and the third data processing result are combined to obtain output data.

To sum up, the processing method based on the heterogeneous computing framework of the embodiment of the present disclosure can divide the input data into blocks and establish computing threads corresponding to the data blocks, thereby further improving the processing efficiency of the hardware and reducing the processing time.

In some embodiments, a processing method with a higher granularity than the previous embodiment may be adopted, and a method of multi-task and multi-backend parallelism may be adopted. The specific description is as follows: FIG. 6 is another framework based on heterogeneous computing provided by an embodiment of the present disclosure A schematic flow chart of the processing method, as shown in Figure 6, also includes the following steps in the method:

Step 601, determining multiple subtasks corresponding to the data processing task, and establishing a corresponding subflowchart for each subtask.

In the embodiments of the present disclosure, in order to improve the processing efficiency of heterogeneous processing units for input data, the data processing tasks are subdivided. In some embodiments, the data processing tasks can be subdivided according to the business type. After processing, multiple subtasks can be obtained, and then a corresponding subflowchart can be established for each subtask. The number of heterogeneous processing units in the subtask flowchart and the number and type of algorithm modules in each heterogeneous processing unit are disclosed in this disclosure. The embodiment is not limited, and those skilled in the art can design the sub-flowchart according to the processing capabilities of the heterogeneous processing units and the complexity of the sub-tasks.

Step 602, simultaneously input data of at least one data input terminal to multiple sub-flowcharts corresponding to multiple sub-tasks, wherein the multiple sub-flowcharts are used to execute heterogeneous calculations in parallel to generate multiple sub-task results.

In an embodiment of the present disclosure, data from at least one data input terminal may be input into the above-mentioned multiple sub-flowcharts at the same time, and then heterogeneous calculations are performed according to each sub-flowchart, thereby generating multiple subtask results.

Step 603, providing multiple subtask results to the data output terminal, wherein the data output terminal is used to combine the multiple subtask results.

In the embodiment of the present disclosure, the results of multiple subtasks are merged through the data output terminal to obtain output data.

For example, see Figure 7. Assuming that the target task is to collect images and perform face recognition, the target task is divided into three subtasks: image noise reduction, image feature extraction, and processing the collected digital signals. Therefore, for image noise reduction Set up the first sub-flowchart, the Process in the first sub-flowchart is encapsulated with CPU; the second sub-flowchart is established for image feature extraction, the Process in the second sub-flowchart is encapsulated with GPU; the second is established for digital signal processing Three sub-flowcharts, the Process in the third sub-flowchart is encapsulated with DSP. Input the collected data into the above three sub-flowcharts, execute heterogeneous calculations in parallel according to the corresponding sub-flowcharts, and then generate the results of the first subtask corresponding to the first subtask and the results of the second subtask corresponding to the second subtask and the result of the third subtask corresponding to the third subtask, and then the data output end combines the results of the above three subtasks to obtain the output data.

To sum up, the processing method based on the heterogeneous computing framework of the embodiment of the present disclosure can divide the target task into multiple subtasks, and establish a corresponding subflowchart for each subtask, thereby further improving the processing efficiency of the hardware and reducing the processing time. duration.

Based on the above embodiments, it is also possible to insert a custom algorithm node in the heterogeneous computing engine framework, specifically including:

Insert a custom algorithm node into the framework of the heterogeneous computing engine, where the custom algorithm node is connected to the data input terminal or the output terminal of the heterogeneous processing unit, and the custom algorithm node is used to process the data input terminal according to the custom algorithm, or the data provided by the output terminal of the heterogeneous processing unit, and feed back the calculated result to the next node connected to the data input terminal or the output terminal of the heterogeneous processing unit. Among them, the custom algorithm includes an algorithm written by the user according to the requirement, an algorithm selected by the user from multiple provided algorithms according to the requirement, and the like.

An example is as follows. FIG. 8 is a schematic diagram of inserting a self-defined algorithm node provided by an embodiment of the present disclosure. As shown in FIG. 8, a first self-defined algorithm node is inserted between the data input terminal and the first heterogeneous processing unit. And a second custom algorithm node is inserted between the first heterogeneous processing unit and the second heterogeneous processing unit. When performing heterogeneous computing, after the data is input from the data input terminal, it is processed by the first custom algorithm node and input to the first heterogeneous processing unit, and then processed by the first heterogeneous processing unit and input to the second custom algorithm node , is processed by the second custom algorithm node and input to the second heterogeneous processing unit, and the second heterogeneous processing unit continues to process.

In some embodiments, a location termination node can also be inserted into the heterogeneous computing engine framework, specifically including:

A position termination node is inserted into the framework of the heterogeneous computing engine, where the position termination node is connected to the output end of the heterogeneous processing unit, and the position termination node is used to stop the processing nodes after the output end of the heterogeneous processing unit to continue computing, and the heterogeneous The data provided by the output terminal of the structural processing unit is output as the processing result.

An example is as follows. FIG. 9 is a schematic diagram of an insertion position termination node provided by an embodiment of the present disclosure. As shown in FIG. 9 , it is necessary to output the processing result of the second heterogeneous processing unit after the second heterogeneous processing unit, then The node may be terminated at the output connection location of the second heterogeneous processing unit. When performing heterogeneous computing, after the data is input from the data input terminal, after being processed by the first heterogeneous processing unit, it is input into the second heterogeneous processing unit, and after being processed by the second heterogeneous processing unit, the second heterogeneous The data output by the processing unit is output as a processing result.

Understandably, it is also possible to simultaneously insert a custom algorithm node and a location termination node in the framework of a heterogeneous computing engine, for example as follows, FIG. 10 is a schematic diagram of inserting a custom algorithm node and a location termination node provided by an embodiment of the present disclosure , as shown in FIG. 10 , insert a custom algorithm node between the data output terminal and the first heterogeneous processing unit, and output the processing result of the first heterogeneous processing unit after the first heterogeneous processing unit. When performing heterogeneous computing, after the data is input from the data input terminal, after being processed by a custom algorithm node, it is input to the first heterogeneous processing unit, and the first heterogeneous processing unit performs processing, and the first heterogeneous processing unit The output data is output as the processing result.

To sum up, the processing method based on the heterogeneous computing framework of the embodiment of the present disclosure can insert a custom algorithm node and/or a location termination node in the heterogeneous computing engine framework, and the user can flexibly insert functions that meet their own or scenario requirements, Moreover, the output nodes of the computing framework are determined flexibly, thereby improving the scalability of the heterogeneous computing framework, and making the applicable scenarios of the heterogeneous computing framework more abundant.

FIG. 11 is a schematic structural diagram of a processing device based on a heterogeneous computing framework provided by an embodiment of the present disclosure. The device can be implemented by software and/or hardware, and can generally be integrated into an electronic device. As shown in Figure 11, the device includes:

The first creation module 1101 is configured to create a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each heterogeneous processing unit is generated by encapsulating at least one algorithm module;

The second creation module 1102 is configured to call at least one interface of a heterogeneous processing unit according to a data processing task to create a heterogeneous computing engine framework, wherein the heterogeneous computing engine framework includes at least one data input terminal, and at least one of the heterogeneous a processing unit, and at least one data output;

The obtaining module 1103 is configured to call the application program interface API of the required algorithm module according to the framework of the heterogeneous computing engine to execute the data processing task, and output the task processing result to the target device.

In some embodiments, the first creation module 1101 is configured to:

Create a call interface and a task running interface corresponding to each algorithm module, wherein the algorithm module includes: a central processing unit CPU, a digital signal processing module DSP, an image processor module GPU, an application-specific integrated circuit module ASIC, field programmable Any one or combination of logic gate array modules FPGA.

In some embodiments, the device also includes:

A segmentation module, configured to perform segmentation processing on the input data of the heterogeneous processing unit to generate multiple data blocks;

The first establishment module is configured to establish a plurality of calculation threads corresponding to the plurality of data blocks in the heterogeneous processing unit, wherein the plurality of calculation threads are used to perform parallel execution on the plurality of data blocks to generate corresponding Multiple data processing results of ;

A generating module, configured to combine and process the multiple data processing results to generate output data corresponding to the heterogeneous processing units.

In some embodiments, the device also includes:

A second building module, configured to determine a plurality of subtasks corresponding to the data processing task, and establish a corresponding subflow chart for each of the subtasks;

The input module is configured to simultaneously input the data of the at least one data input terminal to multiple sub-flowcharts corresponding to the multiple sub-tasks, wherein the multiple sub-flowcharts are used to perform heterogeneous calculations in parallel to generate multiple sub-task results ;

An output module, configured to provide the multiple subtask results to the data output terminal, wherein the data output terminal is used to combine the multiple subtask results.

In some embodiments, the device also includes:

A first insertion module, configured to insert a self-defined algorithm node into the framework of the heterogeneous computing engine, wherein the self-defined algorithm node is connected to the data input end or the output end of the heterogeneous processing unit, The self-defined algorithm node is used to calculate the data provided by the data input terminal or the output terminal of the heterogeneous processing unit according to a self-defined algorithm, and feed back the calculated result to the data input terminal, Or in the next node connected to the output end of the heterogeneous processing unit.

In some embodiments, the device also includes:

The second insertion module is used to insert a position termination node into the framework of the heterogeneous computing engine, wherein the position termination node is connected to the output end of the heterogeneous processing unit, and the position termination node is used to stop the The processing nodes after the output end of the heterogeneous processing unit continue to calculate, and output the data provided by the output end of the heterogeneous processing unit as a processing result.

The processing device based on the heterogeneous computing framework provided by the embodiments of the present disclosure can execute the processing method based on the heterogeneous computing framework provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method.

In order to implement the above embodiments, the present disclosure further proposes a computer program product, including computer programs/instructions, and when the computer program/instructions are executed by a processor, the processing method based on the heterogeneous computing framework in the above embodiments is implemented.

In order to implement the above embodiments, the present disclosure also proposes a computer program, including/instructions, and when the instructions are executed by a processor, the processing method based on the heterogeneous computing framework in the above embodiments is implemented.

Referring to FIG. 12 in detail below, it shows a schematic structural diagram of an electronic device 1200 suitable for implementing an embodiment of the present disclosure. The electronic device 1200 in the embodiment of the present disclosure may include, but not limited to, mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals ( Mobile terminals such as car navigation terminals) and stationary terminals such as digital TVs, desktop computers and the like. The electronic device shown in FIG. 12 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 12, an electronic device 1200 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 1201, which may be randomly accessed according to a program stored in a read-only memory (ROM) 1202 or loaded from a storage device 1208. Various appropriate actions and processes are executed by programs in the memory (RAM) 1203 . In the RAM 1203, various programs and data necessary for the operation of the electronic device 1200 are also stored. The processing device 1201, ROM 1202, and RAM 1203 are connected to each other through a bus 1204. An input/output (I/O) interface 1205 is also connected to the bus 1204 .

Typically, the following devices can be connected to the I/O interface 1205: input devices 1206 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 1207 such as a computer; a storage device 1208 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1209. The communication means 1209 may allow the electronic device 1200 to perform wireless or wired communication with other devices to exchange data. While FIG. 12 shows electronic device 1200 having various means, it is to be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 1209, or from storage means 1208, or from ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the processing method based on the heterogeneous computing framework of the embodiment of the present disclosure are executed.

It should be noted that the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server can communicate using any currently known or future network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital data in any form or medium The communication (eg, communication network) interconnections. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device:

Create a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each heterogeneous processing unit is generated by encapsulating at least one algorithm module; call at least one heterogeneous processing unit according to the data processing task Create a heterogeneous computing engine framework through the interface, where the heterogeneous computing engine framework includes at least one data input terminal, at least one heterogeneous processing unit, and at least one data output terminal; call the application of the required algorithm module according to the heterogeneous computing engine framework The program interface API executes data processing tasks and outputs task processing results to the target device. In the embodiment of the present disclosure, a heterogeneous computing engine framework is established through a heterogeneous processing unit according to a data processing task, and the data processing task is connected through the heterogeneous computing engine framework. Adjust the heterogeneous computing engine framework to meet the corresponding changes, so it is suitable for expansion and has strong flexibility. It can improve processing efficiency for large-scale algorithm systems, and integrates the characteristics of different algorithm modules in heterogeneous processing units through the heterogeneous computing engine framework. , so as to take advantage of the computing power of heterogeneous processing units, improve computing efficiency, reduce computing time, and improve the performance of heterogeneous processing units, thereby meeting the current business needs for computing power. At the same time, the hierarchical design of the architecture, the modular design, the cascading of algorithms, and the intelligent scheduling of hybrid computing are realized through the heterogeneous computing engine framework, thereby improving the efficiency of the framework design.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of a unit does not constitute a limitation of the unit itself under certain circumstances.

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principle. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

A processing method, the processing method is based on a heterogeneous computing framework, comprising:

Create a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each said heterogeneous processing unit is generated by encapsulating at least one algorithm module;

Create a heterogeneous computing engine framework by invoking at least one interface of the heterogeneous processing unit according to the data processing task, wherein the heterogeneous computing engine framework includes at least one data input terminal, at least one heterogeneous processing unit, and at least one data output; and

Call the application program interface API of the required algorithm module according to the heterogeneous computing engine framework to execute the data processing task, and output the task processing result to the target device.
The processing method according to claim 1, wherein said creating a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit comprises:

Create a calling interface and task running interface corresponding to each algorithm module.
The processing method according to claim 1 or 2, wherein the algorithm module comprises: a central processing unit CPU, a digital signal processing module DSP, an image processor module GPU, an application specific integrated circuit module ASIC, a field programmable logic gate array Any one or combination of multiple FPGAs.
The processing method according to any one of claims 1-3, further comprising:

performing segmentation processing on the input data of the heterogeneous processing unit to generate multiple data blocks;

Establishing multiple computing threads corresponding to the multiple data blocks in the heterogeneous processing unit, wherein the multiple computing threads are used to execute the multiple data blocks in parallel to generate corresponding multiple data processing results; as well as

Merge the multiple data processing results to generate output data corresponding to the heterogeneous processing units.
The processing method according to any one of claims 1-3, further comprising:

determining a plurality of subtasks corresponding to the data processing task, and establishing a corresponding subflow chart for each of the subtasks;

Simultaneously input the data of the at least one data input terminal into multiple sub-flowcharts corresponding to the multiple sub-tasks, wherein the multiple sub-flowcharts are used to perform heterogeneous calculations in parallel to generate multiple sub-task results; and

All the multiple subtask results are provided to the data output terminal, wherein the data output terminal is used to combine the multiple subtask results.
The processing method according to any one of claims 1-5, further comprising:

A self-defined algorithm node is inserted into the framework of the heterogeneous computing engine, wherein the self-defined algorithm node is connected to the data input end or the output end of the heterogeneous processing unit, and the self-defined algorithm node is used Calculate the data provided by the data input terminal or the output terminal of the heterogeneous processing unit according to a self-defined algorithm, and feed back the calculated result to the data input terminal or the heterogeneous processing unit In the next node connected to the output terminal of .
The processing method according to any one of claims 1-6, further comprising:

A position termination node is inserted into the framework of the heterogeneous computing engine, wherein the position termination node is connected to the output end of the heterogeneous processing unit, and the position termination node is used to stop the output end of the heterogeneous processing unit Subsequent processing nodes continue to calculate, and output the data provided by the output terminals of the heterogeneous processing units as processing results.
A processing device based on a heterogeneous computing framework, comprising:

A first creating module, configured to create a corresponding application program interface API for each algorithm module in at least one heterogeneous processing unit, wherein each of the heterogeneous processing units is generated by encapsulating at least one algorithm module;

The second creation module is used to create a heterogeneous computing engine framework by invoking the interface of at least one heterogeneous processing unit according to the data processing task, wherein the heterogeneous computing engine framework includes at least one data input terminal, and at least one of the heterogeneous processing units unit, and at least one data output; and

The obtaining module is used to call the application program interface API of the required algorithm module according to the heterogeneous computing engine framework to execute the data processing task, and output the task processing result to the target device.
An electronic device comprising:

processor; and

memory for storing instructions executable by the processor;

The processor is configured to read the instruction from the memory, and execute the instruction to implement the processing method described in any one of claims 1-7.
A computer-readable storage medium, the storage medium stores computer programs/instructions, and when the computer programs/instructions are executed by a processor, the processing method described in any one of claims 1-7 above is executed.
A computer program product, the computer program product comprising a computer program/instruction, when the computer program/instruction is executed by a processor, the processing method according to any one of claims 1-7 is realized.
A computer program, wherein the computer program includes instructions, and when the instructions are executed by a processor, the processing method according to any one of claims 1-7 is implemented.