WO2021088964A1

WO2021088964A1 - Inference system, inference method, electronic device and computer storage medium

Info

Publication number: WO2021088964A1
Application number: PCT/CN2020/127026
Authority: WO
Inventors: 林立翔; 李鹏; 游亮; 龙欣
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2019-11-08
Filing date: 2020-11-06
Publication date: 2021-05-14
Also published as: CN112784989B; TW202119255A; CN112784989A

Abstract

Provided are an inference system, an inference method, an electronic device and a computer storage medium. The inference system comprises a first computing device and a second computing device that are connected to each other, with the first computing device being provided with an inference client, and the second computing device comprising an inference acceleration resource and an inference server, wherein the inference client is used for acquiring model information of a computing model for inference and data to be inferred, and for respectively sending the model information and said data to the inference server in the second computing device; and the inference server is used for loading and calling, by means of the inference acceleration resource, the computing model indicated by the model information, and performing, by means of the computing model, inference processing on said data, and feeding back the result of the inference processing to the inference client.

Description

Reasoning system, reasoning method, electronic equipment and computer storage medium

This application claims the priority of the Chinese patent application filed on November 8, 2019 with the application number 201911089253.X and the title of the invention "inference system, inference method, electronic equipment and computer storage medium", the entire content of which is incorporated herein by reference Applying.

Technical field

The embodiments of the present invention relate to the field of computer technology, in particular to an inference system, an inference method, electronic equipment, and a computer storage medium.

Background technique

Deep learning is generally divided into two parts: training and inference. The training part searches for and solves the optimal parameters of the model, and the inference part allows the trained model to be deployed in an online environment for actual use. . Taking the field of artificial intelligence as an example, after inference is deployed, the input can be converted into specific target output through neural network derivation and calculation. For example, object detection on pictures, classification of text content, etc., are widely used in scenes such as vision, voice, and recommendation.

Currently, most inferences rely on hardware computing resources with inference accelerator cards such as GPU (Graphics Processing Unit). For example, in artificial intelligence reasoning, one way is to connect the GPU to the host computer through a PCIE (Peripheral Component Interconnect Express, high-speed serial computer expansion bus standard) slot. Among them, the pre-processing and other business logic involved in the inference are calculated by the CPU, and the processing of the inference is sent to the GPU through the PCIE slot for calculation, forming a typical heterogeneous computing scenario. For example, in the electronic device 100 shown in FIG. 1, both a CPU 102 and a GPU 104 are provided. The GPU 104 can be set on the electronic device motherboard 108 through the PCIE slot 106, and interact with the CPU 102 through the motherboard circuit on the motherboard 108 . In a reasoning process, the CPU 102 first processes relevant data or information, and then sends the processed data or information to the GPU 104 through the PCIE slot 106. The GPU 104 uses the calculations in the GPU 104 according to the received data or information. The model performs inference processing, and then returns the inference processing result to the CPU 102, and the CPU 102 performs corresponding subsequent processing.

However, the above method has the following problems: a heterogeneous computing machine with the same CPU and GPU is required, and the specifications of the CPU/GPU in the heterogeneous computing machine are fixed. This fixed CPU/GPU performance ratio limits the reasoning involved The deployment of applications makes it impossible to meet the needs of a wide range of reasoning scenarios.

Summary of the invention

In view of this, the embodiment of the present invention provides a reasoning solution to solve some or all of the above-mentioned problems.

According to a first aspect of the embodiments of the present invention, an inference system is provided, including a first computing device and a second computing device connected to each other, wherein the first computing device is provided with an inference client, and the second The computing device is provided with inference acceleration resources and inference server; wherein: the inference client is used to obtain model information and data to be inferred of the calculation model for inference, and respectively send the model information and the data to be inferred To the inference server in the second computing device; the inference server is used to speed up resource loading through inference and call the calculation model indicated by the model information, and perform inference on the data to be inferred through the calculation model Process and feed back the result of the inference processing to the inference client.

According to a second aspect of the embodiments of the present invention, there is provided an inference method, the method comprising: obtaining model information of a calculation model for inference, and sending the model information to a target computing device to instruct the target calculation The device uses the inference acceleration resource set in the target computing device to load the calculation model indicated by the model information; obtains the data to be inferred, and sends the data to be inferred to the target computing device to instruct the target calculation The device uses the inference acceleration resource to call the loaded calculation model, and uses the calculation model to perform inference processing on the data to be inferred; and receives the result of the inference processing fed back by the target computing device.

According to a third aspect of the embodiments of the present invention, there is provided another reasoning method, the method comprising: obtaining model information of a calculation model used for reasoning sent by a source computing device, and loading the model information indication through the reasoning acceleration resource The calculation model; obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model; to the source computing device The result of the inference processing is fed back.

According to a fourth aspect of the embodiments of the present invention, there is provided an electronic device, including: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface complete each other through the communication bus. Inter-communication; the memory is used to store at least one executable instruction, the executable instruction causes the processor to perform the operation corresponding to the inference method described in the second aspect, or the executable instruction causes the The processor executes operations corresponding to the inference method described in the third aspect.

According to a fifth aspect of the embodiments of the present invention, there is provided a computer storage medium having a computer program stored thereon, and when the program is executed by a processor, the reasoning method described in the second aspect is implemented; or, the third aspect is implemented The reasoning method described.

According to the reasoning scheme provided by the embodiment of the present invention, the reasoning process is deployed in different first and second computing devices, where the second computing device is provided with inference acceleration resources, and the main reasoning process can be performed through the calculation model, and The first computing device can then be responsible for data processing before and after the inference processing. In addition, an inference client is deployed in the first computing device, and an inference server is deployed in the second computing device. During inference, the first computing device and the second computing device interact through the inference client and the inference server. The inference client can first send the model information of the calculation model to the inference server, and the inference server uses inference acceleration resources to load the corresponding calculation model; then, the inference client sends the data to be inferred to the inference server, and the inference server receives After the data to be inferred, inference processing can be performed through the loaded calculation model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized by different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem that the fixed specifications of the CPU/GPU of the existing heterogeneous computing machines limit the deployment of inference-related applications, which makes it impossible to meet the needs of a wide range of inference scenarios The problem.

In addition, for users, when using applications involving reasoning, reasoning calculations can be seamlessly transferred to remote devices with reasoning acceleration resources through the reasoning client and the reasoning server, and the reasoning client and the reasoning server can be seamlessly transferred. The interaction between them is unaware to the user. Therefore, it can ensure that the business logic of the application involving reasoning and the user's usage habits for reasoning services remain unchanged, and the reasoning is realized at low cost and the user experience is improved.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some of the embodiments described in the embodiments of the present invention. For those of ordinary skill in the art, other drawings may be obtained based on these drawings.

Fig. 1 is a schematic structural diagram of an electronic device with inference calculation resources in the prior art;

Figure 2a is a structural block diagram of an inference system according to the first embodiment of the present invention;

Figure 2b is a schematic structural diagram of an example of an inference system according to an embodiment of the present invention;

Figure 3a is a structural block diagram of an inference system according to the second embodiment of the present invention;

Figure 3b is a schematic structural diagram of an example of an inference system according to an embodiment of the present invention;

Figure 3c is a schematic diagram of the inference process using the inference system shown in Figure 3b;

Fig. 3d is a schematic diagram of interaction for reasoning using the reasoning system shown in Fig. 3b;

Fig. 4 is a flowchart of an inference method according to the third embodiment of the present invention;

Fig. 5 is a flowchart of an inference method according to the fourth embodiment of the present invention;

Fig. 6 is a flowchart of an inference method according to the fifth embodiment of the present invention;

Fig. 7 is a flowchart of an inference method according to the sixth embodiment of the present invention;

Fig. 8 is a schematic structural diagram of an electronic device according to a seventh embodiment of the present invention;

Fig. 9 is a schematic structural diagram of an electronic device according to the eighth embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the description is The embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art should fall within the protection scope of the embodiments of the present invention.

The specific implementation of the embodiments of the present invention will be further described below in conjunction with the drawings of the embodiments of the present invention.

Example one

Referring to Fig. 2a, a structural block diagram of an inference system according to the first embodiment of the present invention is shown.

The inference system of this embodiment includes a first computing device 202 and a second computing device 204 connected to each other, wherein the first computing device 202 is provided with an inference client 2022, and the second computing device 204 is provided with an inference server 2042 and Inference acceleration resources 2044.

Wherein, the inference client 2022 is used to obtain model information and data to be inferred of the calculation model for inference, and respectively send the model information and the data to be inferred to the inference server 2042 in the second computing device 204 The reasoning server 2042 is used to load and call the calculation model indicated by the model information through the reasoning acceleration resource 2044, perform reasoning processing on the data to be reasoned through the calculation model, and feedback the reasoning processing to the reasoning client 2022 the result of.

In a feasible implementation, the inference client 2022 in the first computing device 202 first obtains model information of the computing model for inference, and sends the model information to the inference server 2042 in the second computing device 204; The inference server 2042 in the second computing device 204 loads the calculation model indicated by the model information through the inference acceleration resource 2044; the inference client 2022 in the first computing device 202 then obtains the data to be inferred, and sends the data to be inferred To the inference server 2042 in the second computing device 204; the inference server 2042 in the second computing device 204 uses the inference acceleration resource 2044 to call the loaded calculation model, and perform inference processing on the inference data through the calculation model and The result of the inference processing is fed back to the inference client 2022.

In the above recommendation system, because the second computing device 204 has the reasoning acceleration resource 2044 used for reasoning, it can effectively load a calculation model for reasoning and perform reasoning calculations with a large amount of data. Also, because the second computing device 204 where the inference acceleration resource 2044 is located and the first computing device 202 are independently set, the inference acceleration resource 2044, such as the GPU, does not need to follow a fixed specification setting with the processor resources in the first computing device 202, such as the CPU, so that The realization of the inference acceleration resource 2044 is more flexible and diverse. Among them, the inference acceleration resource 2044 can be implemented in various forms including but not limited to GPU and NPU. Therefore, it is only necessary to configure a resource such as a CPU for conventional data processing in 202 in the first computing device.

The calculation model for reasoning may be any appropriate calculation model set according to business requirements, which may be applicable to deep learning frameworks (including but not limited to Tensorflow framework, Mxnet framework, and PyTorch framework). In a feasible manner, the second computing device 204 can be pre-configured with a resource pool of the computing model. If the computing model to be used is in the resource pool, it can be directly loaded and used; if it is not in the resource pool, it can be downloaded from Obtained from the first computing device 202. In another feasible manner, the second computing device 204 may not have a pre-set computing model resource pool. When inference is needed, the required computing model is obtained from the first computing device 202 and then stored locally. After multiple inferences, different calculation models can be obtained and finally stored as a resource pool of the calculation model. The obtained different computing models can come from different first computing devices 202, that is, the second computing device 204 can provide inference services for different first computing devices 202, so as to obtain different first computing devices 202. Calculation model.

The model information of the calculation model sent by the inference client 2022 to the inference server 2042 may uniquely identify the calculation model, for example, it may be the identification information of the calculation model, such as an identification ID number. But it is not limited to this. In a feasible way, the model information of the calculation model can also be the verification information of the calculation model, such as MD5 information. The verification information can identify the calculation model on the one hand, and can also perform the calculation model on the other hand. Verification, through one type of information to achieve multiple functions, reducing the cost of information processing. The model information can be obtained when the first computing device 202 loads the model.

Hereinafter, a specific example is used to illustrate the structure of the above reasoning system, as shown in Fig. 2b.

In Figure 2b, the first computing device 202 is implemented as a terminal device, that is, the first terminal device, in which a processor CPU is provided for corresponding service processing, and the second computing device 204 is also implemented as a terminal device, that is, the second terminal. The device, which is provided with the inference acceleration resource GPU. In addition, the first computing device 202 is loaded with a deep learning framework and a reasoning client set in the deep learning framework; the second computing device 204 is correspondingly set with a reasoning server. In this embodiment, it is also set that the second computing device 204 is provided with a resource pool of computing models, in which multiple computing models, such as computing models A, B, C, and D are stored.

It should be understood by those skilled in the art that the above examples are only illustrative. In practical applications, both the first computing device 202 and the second computing device 204 can be implemented as terminal devices, or both can be implemented as servers, or the first computing device 202 and the second computing device 204 can be implemented as a server. One computing device 202 is implemented as a server and the second computing device 204 is implemented as a terminal device or vice versa, which is not limited in the embodiment of the present invention.

Based on the reasoning system in Figure 2b, a process of using the reasoning system for reasoning is as follows.

Taking image recognition as an example, the corresponding model information can be obtained when the deep learning framework is loaded into the model. The inference client first sends the information of the calculation model to the second terminal device, and the second terminal device receives the calculation through the inference server. Model information. Assuming that the information of the calculation model indicates that the calculation model to be used is calculation model A, and the resource pool of the second terminal device stores calculation models A, B, C, and D, then the second terminal device will directly access the resource from the resource through the GPU. Load calculation model A in the pool. Furthermore, the second terminal device obtains the data to be inferred, such as the image to be recognized, from the first terminal device through the inference server and the inference client, and then the second terminal device uses the GPU to call the calculation model A to perform target object recognition on the image, such as Identify whether there are people in the image. After the recognition, the second terminal device will send the recognition result to the inference client of the first terminal device through the inference server, and the inference client will hand it over to the CPU for subsequent processing, such as adding AR special effects.

It should be noted that, in the embodiments of the present invention, unless otherwise specified, quantities such as "multiple" and "multiple" related to "multiple" mean two or more.

According to the inference system provided in this embodiment, the inference processing is deployed in different first and second computing devices. The second computing device is provided with inference acceleration resources, and the main inference processing can be performed through the computing model. A computing device can then be responsible for data processing before and after inference processing. In addition, an inference client is deployed in the first computing device, and an inference server is deployed in the second computing device. During inference, the first computing device and the second computing device interact through the inference client and the inference server. The inference client can first send the model information of the calculation model to the inference server, and the inference server uses inference acceleration resources to load the corresponding calculation model; then, the inference client sends the data to be inferred to the inference server, and the inference server receives After the data to be inferred, inference processing can be performed through the loaded calculation model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized by different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem that the fixed specifications of the CPU/GPU of the existing heterogeneous computing machines limit the deployment of inference-related applications, which makes it impossible to meet the needs of a wide range of inference scenarios The problem.

Example two

This embodiment further optimizes the reasoning system in the first embodiment, as shown in Fig. 3a.

As described in the first embodiment, the inference system of this embodiment includes: a first computing device 202 and a second computing device 204 connected to each other, wherein the first computing device 202 is provided with an inference client 2022, and the second computing device The reasoning server 2042 and the reasoning acceleration resource 2044 are provided in 204.

The inference client 2022 in the first computing device 202 is used to obtain model information of the computing model for inference, and send the model information to the inference server 2042 in the second computing device 204; The reasoning server 2042 is used to load the calculation model indicated by the model information through the reasoning acceleration resource 2044; the reasoning client 2022 in the first computing device 202 is also used to obtain the data to be reasoned, and send the data to be reasoned to the second The inference server 2042 in the computing device 204; the inference server 2042 in the second computing device 204 is also used to use the inference acceleration resource 2044 to call the loaded calculation model, and use the calculation model to perform inference processing on the inference data to be inferred. The result of the inference processing is fed back to the inference client 2022.

In one feasible manner, the first computing device 202 and the second computing device 204 are connected to each other through an elastic network. The elastic network includes, but is not limited to, an ENI (Elastic Network Interface) network. The elastic network has better scalability and flexibility. The first computing device 202 and the second computing device 204 are connected through the elastic network, so that the reasoning system also has better scalability and flexibility. However, it is not limited to this. In actual applications, the first computing device 202 and the second computing device 204 can be connected in any suitable manner or network, so that the data interaction between the two can be smoothly realized.

In this embodiment, optionally, the inference client 2022 can be implemented as a component embedded in the deep learning framework in the first computing device 202, or the inference client can be implemented as a component that can be called by the deep learning framework. The callable file. The deep learning framework provides a platform for deep learning. Based on the deep learning framework, programmers can easily deploy various computing models to achieve different inference functions. Implementing the inference client 2022 in the form of a component or callable file suitable for a deep learning framework on the one hand makes it more compatible and applicable, on the other hand, it also greatly reduces the implementation cost of decoupling inference computing resources. Similarly, the reasoning server 2042 can also be implemented in the form of a component or a callable file.

Based on the above structure, the reasoning system of this embodiment can conveniently exchange corresponding data and information through the reasoning client 2022 and the reasoning server 2042, and realize reasoning processing by remotely calling the reasoning acceleration resource 2044.

In addition, in this embodiment, the inference client 2022 is also configured to send the calculation model to the inference server 2042 when it is determined that the calculation model does not exist in the second computing device 204.

Optionally, the model information of the calculation model is identification information or verification information of the calculation model; the reasoning server 2042 is further configured to determine that the second computing device 204 is in the second computing device 204 through the identification information or the verification information. Whether the calculation model exists, and the determination result is returned to the inference client 2022. However, it is not limited to this, and other methods of determining whether the second computing device 204 has the calculation model are also applicable, for example, the second computing device 204 broadcasts the calculation model it has at regular intervals; or, when needed Or at regular intervals, the first computing device 202 actively sends a message to inquire about the resource status of the computing model in the second computing device 204, and so on.

For example, if the resource pool of the calculation model is not preset in the second computing device 204 or there is no required calculation model in the resource pool, the inference client 2022 will send the calculation model in the first computing device 202 to the second computing device 204, including but not limited to the structure of the calculation model and the data it contains. In the case that the calculation model information sent by the inference client 2022 to the inference server 2042 is identification information or verification information, the inference server 2042 will first determine the second computing device 204 according to the received identification information or verification information. Whether the required calculation model exists in the database, and the determination result is returned to the inference client 2022. If the result of the determination indicates that the required calculation model does not exist in the second computing device 204, the first computing device 202 obtains the calculation model locally and sends it to the second computing device 204, based on the inference of the second computing device 204 Accelerate resources to run the calculation model and perform inference processing.

In this way, it can be effectively ensured that the second computing device 204 with inference acceleration resources can successfully complete inference processing.

In addition, in a feasible manner, the reasoning client 2022 is also used to obtain a reasoning request requesting the calculation model to perform reasoning processing on the data to be reasoned, and perform semantic analysis on the reasoning request, according to the semantics The analysis result determines the processing function in the calculation model to be called, and sends the information of the processing function to the inference server 2042; the inference server 2042 is performing the calculation on the data to be inferred through the calculation model. During inference processing, inference processing is performed on the data to be inferred by calling the processing function indicated by the processing function information in the loaded calculation model.

In some specific business applications of reasoning, the business may not need all the reasoning functions of the calculation model, but only part of the functions. For example, a certain reasoning is used to classify text content, and the current business only needs the calculation function in it to add the corresponding text vector. In this case, when the reasoning client 2022 receives a request for a calculation model to calculate the text After the vector performs the reasoning request of the addition calculation, through semantic analysis of the reasoning request, it is determined that only the COMPUTE() function in the calculation model needs to be called, and then the information of the function can be sent to the reasoning server 2042. After the reasoning server 2042 obtains the information of the function, it can directly call the COMPUTE() function in the calculation model to perform the addition calculation of the text vector.

It can be seen that in this way, the use of the calculation model is more accurate, the reasoning efficiency of the calculation model is greatly improved, and the reasoning burden is reduced.

In a feasible way, the processing function information can be the API interface information of the processing function. Through the API interface information, the processing function in the calculation model to be used can be quickly determined, and the corresponding interface information of the function can also be directly obtained. It can be used directly in the subsequent inference processing.

Optionally, one or more types of inference acceleration resources are provided in the second computing device 204; when the inference acceleration resources include multiple types, different types of inference acceleration resources have different usage priorities; the reasoning server 2042 is based on The preset load balancing rules and the priority of various types of inference acceleration resources, use the inference acceleration resources.

For example, in addition to the GPU provided in the second computing device 204, an NPU or other inference acceleration resources may also be provided. There is a certain priority among multiple inference acceleration resources, and the priority can be set in any appropriate manner, such as setting according to the running speed or manually, etc., which is not limited in the embodiment of the present invention. Further optionally, a CPU may also be provided in the second computing device 204. In this case, you can set the GPU to use the highest priority, the NPU second, and the CPU priority to be the lowest.

In this way, when high-priority inference acceleration resources are heavily loaded, lower-priority inference acceleration resources can be used for inference processing. On the one hand, it ensures that inference processing can be effectively executed, and on the other hand, it can also reduce the cost of inference acceleration resources. It should be noted that the number of a certain type of reasoning acceleration resource may be one or multiple, which is set by a person skilled in the art according to requirements, and the embodiment of the present invention does not limit this. In addition, in addition to the above-mentioned load balancing according to the priority of the preset load balancing rules, those skilled in the art can also set other appropriate load balancing rules according to actual needs, which is not limited in the embodiment of the present invention.

Hereinafter, a specific example is used to describe the inference system in this embodiment.

As shown in Figure 3b, unlike the traditional CPU and inference acceleration resources such as GPU in the same electronic device architecture, in this example, the CPU and inference acceleration resources are decoupled into two parts, namely the CPU client machine in Figure 3b. Client machine) and Server accelerator pools (background reasoning accelerator card machine). Among them, the front-end client machine is a user-operable machine for performing reasoning services, and the back-end reasoning accelerator card machine is used for reasoning calculations, and the communication between the two is realized through ENI. There are multiple inference frameworks in the front-end client machine, as shown in the figure are "Tensorflow inference code", "pyTorch inferce code" and "Mxnet inference code". There are a variety of inference accelerator cards in the background inference accelerator card machine, shown in the figure as "GPU", "Ali-NPU" and "Other Accelerator".

In order to forward the inference business of the front-end client machine to the back-end accelerator card for execution and return the result of the inference to achieve user-side insensible support, this example provides two components that reside in the front-end client machine and the back-end accelerator card machine respectively. It is the EAI client module (that is, the reasoning client) and the Service daemon (that is, the reasoning server).

Among them, the EAI client module is a component in the front-end client machine, and its functions include: a) communicate with the back-end Service daemon through a network connection; b) analyze the semantics of the calculation model and inference requests; c) analyze the semantics of the calculation model and The analysis result of the inference request is sent to the backend Service daemon; d) The inference result sent by the Service daemon is received and returned to the deep learning framework. In one implementation, the EAI client module can be implemented as a plugin module embedded in the functional code of a deep learning framework (such as Tensorflow/pyTorch/Mxnet, etc.). When the inference business loads the calculation model through the deep learning framework, the EAI client module It will intercept the loaded calculation model, analyze the semantics of the calculation model to generate calculation model information, such as verification information (may be MD5 information), transfer calculation model and/or calculation model information, and transfer subsequent operations to the backend The service daemon performs actual inference calculations.

Service daemon is a resident service component of the background reasoning accelerator card machine. Its functions include: a) receiving the calculation model information sent by the EAI client module and the analysis result of the reasoning request; b) according to the calculation model information and the analysis result of the reasoning request , Select the best reasoning accelerator card in the background reasoning accelerator card machine; c) Send the reasoning calculation to the reasoning accelerator card; d) Receive the reasoning result calculated by the reasoning accelerator card and return it to the EAI client module.

Among them, GPU, Ali-NPU and Other Accelerator have a certain priority. For example, from high to low, it is GPU->Ali-NPU->Other Accelerator. In actual use, GPU is used first. If GPU resources are If Ali-NPU is not enough, if Ali-NPU resources are still insufficient, use Other Accelerator.

It can be seen that, unlike the traditional binding of CPU and GPU inference accelerator cards to one machine through the PCIE card slot, the elastic remote inference in this example combines the CPU client machine and the inference accelerator card (server accelerator) through the elastic network card. Pools) decoupling, for users, it is no longer necessary to buy a machine with the same CPU and GPU.

The reasoning process based on the reasoning system shown in Figure 3b is shown in Figure 3c, including: Step ①, EAI client module intercepts and analyzes the semantics of the calculation model when the user starts the reasoning task through the deep learning framework and loads the calculation model to obtain the calculation model Further, obtain the user’s reasoning request, analyze the reasoning request, and obtain the information of the processing function to be used in the calculation model; step ②, the EAI client module is connected to the service daemon through the elastic network, and the information of the calculation model and the processing function are connected The information is forwarded to the back-end Service daemon; step ③, the service daemon selects the optimal inference accelerator card according to the information of the calculation model and the information of the processing function, and loads the calculation model through the inference accelerator card to perform inference calculation; step ④, inference The accelerator card returns the result of the inference calculation to the service daemon; step ⑤, the service daemon forwards the result of the inference calculation to the EAI client daemon through the elastic network; step ⑥, the EAI client daemon returns the result of the inference calculation to the deep learning framework.

As a result, the user performs inference services on the front-end client machine, and the EAI client module and Service daemon automatically forward the user's inference service to the remote inference accelerator card for inference calculation in the background, and return the result of the inference calculation to the front-end client machine for deep learning The framework achieves the user's non-sense and flexible reasoning, and can enjoy the reasoning acceleration service without changing the reasoning code. Moreover, users do not need to purchase a machine with a GPU, and only need to use an ordinary CPU machine to complete the same inference acceleration effect, and there is no need to modify any code logic.

In a specific example where the deep learning framework is the Tensorflow framework, the interaction between the front-end client machine and the back-end inference accelerator card machine is shown in Figure 3d.

The reasoning interaction includes: Step 1, the front-end client machine loads the calculation model through the Tensorflow framework; Step 2, the EAI client module intercepts the calculation model and verifies the model; Step 3, the EAI client module establishes a channel with the Service daemon, and transmits the calculation model; Steps 4. Service daemon analyzes the calculation model, and selects the optimal reasoning accelerator from the accelerator card pool according to the analysis result; step 5, the selected reasoning accelerator is loaded into the calculation model; step 6, the user enters a picture/text and initiates a reasoning request; Step 7, Tensorflow framework obtains user input, EAI client module intercepts the reasoning request, and parses out the information of the processing function to be used and the data to be inferred by the user; Step 8, EAI client module transmits the information of the processing function and the data to be inferred To the Service daemon; step 9, the service daemon forwards the processing function information and the data to be processed to the inference accelerator; step 10, the inference accelerator performs inference calculation through the calculation model and sends the inference result of the inference calculation to the service daemon; step 11, service The daemon transmits the inference results to the EAI client module; step 12, the EAI client module receives the inference results and transfers the inference results to the Tensorflow framework; step 13, the Tensorflow framework displays the inference results to the user.

As a result, the flexible reasoning process under the Tensorflow framework is realized.

Example three

Referring to Fig. 4, a flow chart of an inference method according to the third embodiment of the present invention is shown. The reasoning method of this embodiment describes the reasoning method of the present invention from the perspective of the first computing device.

The reasoning method of this embodiment includes the following steps:

Step S302: Obtain the model information of the calculation model for inference, and send the model information to the target computing device to instruct the target computing device to use the inference acceleration resource set in the target computing device to load the model information The indicated calculation model.

The implementation of the target computing device in this embodiment can refer to the second computing device in the foregoing embodiment.

The execution of this step can refer to the relevant parts of the inference client in the foregoing multiple embodiments. For example, the model information of the calculation model can be obtained when the calculation model is loaded in the deep learning framework, and then sent to the target computing device. After the target computing device receives the model information, it loads the corresponding computing model through corresponding inference acceleration resources such as GPU.

Step S304: Obtain the data to be inferred, and send the data to be inferred to the target computing device to instruct the target computing device to use the inference acceleration resource to call the loaded calculation model, and to use the calculation model to Describe the data to be inferred for inference processing.

Among them, as mentioned above, the data to be inferred is any appropriate data for inference calculation using a calculation model. After the target computing device loads the calculation model, the data to be inferred can be sent to the target computing device. After the target computing device receives the data to be inferred, it uses inference acceleration resources such as the computing model loaded by the GPU to perform inference processing on the inferred data.

Step S306: Receive the result of the inference processing fed back by the target computing device.

After the target computing device uses the GPU loaded computing model to perform inference processing on the inference data to be inferred, the result of the inference processing is obtained and sent to the execution subject of this embodiment such as the inference client, and the inference client receives the inference processing result. result.

In specific implementation, the inference method of this embodiment can be implemented by the inference client of the first computing device in the foregoing embodiment, and the specific implementation of the foregoing process can also refer to the operation of the inference client in the foregoing embodiment, which will not be repeated here.

Through this embodiment, inference processing is deployed in different computing devices, where inference acceleration resources are set in the target computing device, and the main inference processing can be performed through the computing model, while the current computing device that executes the inference method of this embodiment It can be responsible for the data processing before and after the inference processing. During inference, the current computing device can first send the model information of the computing model to the target computing device, and the target computing device uses inference acceleration resources to load the corresponding computing model; then, the current computing device sends the data to be inferred to the target computing device. After the target computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized by different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem that the fixed CPU/GPU specifications of existing heterogeneous computing machines limit the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios The problem.

In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to the remote target computing device with inference acceleration resources, and the interaction between the current computing device and the target computing device is important to the user. It is imperceptible. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits for reasoning business remain unchanged, and the reasoning is realized at low cost and the user experience is improved.

Example four

The reasoning method of this embodiment still describes the reasoning method of the present invention from the perspective of the first computing device. Referring to Fig. 5, a flowchart of an inference method according to Embodiment 4 of the present invention is shown, and the inference method includes:

Step S402: Obtain model information of the calculation model for inference, and send the model information to the target computing device.

In a feasible manner, the model information of the calculation model is identification information or verification information of the calculation model.

In a specific implementation, the first computing device may send the identification information or verification information to the target computing device, such as the second computing device in the foregoing embodiment, the target computing device determines according to the identification information or verification information Whether there is a corresponding calculation model locally, and the judgment result is fed back to the first computing device, so that the first computing device can determine whether there is a calculation model in the target computing device.

Step S404: If it is determined according to the model information that the calculation model does not exist in the target computing device, the calculation model is sent to the target computing device, and the target computing device is instructed to use the target computing device The reasoning acceleration resource set in is loaded into the calculation model.

When the model information of the calculation model is the identification information or verification information of the calculation model, this step may be implemented as: if the identification information or the verification information is used, it is determined that the target computing device does not exist The calculation model sends the calculation model to the target computing device. Including: sending the structure of the calculation model and its data to the target computing device.

If the required calculation model does not exist in the target computing device, the calculation model can be sent to the target computing device. The target computing device obtains and stores the calculation model, and if the calculation model is used again in subsequent use, the target computing device can directly obtain it locally. As a result, it is ensured that the reasoning process can be smoothly realized regardless of whether the required calculation model is available in the target computing device.

Step S406: Obtain the data to be inferred, and send the data to be inferred to the target computing device to instruct the target computing device to use the inference acceleration resource to call the loaded calculation model, and to use the calculation model to Describe the data to be inferred for inference processing.

In a feasible manner, the obtaining the data to be inferred and sending the data to be inferred to the target computing device may include: obtaining a reasoning request requesting the calculation model to perform inference processing on the data to be inferred, And perform semantic analysis on the reasoning request; determine the processing function in the calculation model to be called according to the result of the semantic analysis, and send the processing function information and the data to be reasoned to the target computing device to indicate The target computing device performs inference processing on the data to be inferred by calling the processing function indicated by the information of the processing function in the loaded calculation model.

Wherein, the information of the processing function may optionally be API interface information of the processing function.

Step S408: Receive the result of the inference processing fed back by the target computing device.

Example five

Referring to Fig. 6, there is shown a flow chart of an inference method according to the fifth embodiment of the present invention. The reasoning method of this embodiment describes the reasoning method of the present invention from the perspective of the second computing device.

The reasoning method of this embodiment includes the following steps:

Step S502: Obtain the model information of the calculation model used for inference sent by the source computing device, and load the calculation model indicated by the model information through the inference acceleration resource.

In this embodiment, the source computing device may be the first computing device in the foregoing embodiment, and the model information includes, but is not limited to, identification information and/or verification information.

Step S504: Obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model.

After the inference acceleration resource is loaded into the calculation model, when the data to be inferred sent from the source computing device is received, the calculation model loaded by the inference acceleration resource can be used to perform inference processing on it.

Step S506: Feed back the result of the inference processing to the source computing device.

In specific implementation, the inference method of this embodiment can be implemented by the inference server of the second computing device in the foregoing embodiment, and the specific implementation of the foregoing process can also refer to the operation of the inference server in the foregoing embodiment, which will not be repeated here.

Through this embodiment, inference processing is deployed in different computing devices, where the current computing device that executes the inference method of this embodiment is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model, and the source computing device It can be responsible for the data processing before and after the inference processing. When performing inference, the source computing device can first send the model information of the computing model to the current computing device, and the current computing device uses the inference acceleration resource to load the corresponding computing model; then, the source computing device sends the data to be inferred to the current computing device. After the current computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized by different computing devices, one of which is equipped with an inference acceleration resource such as a GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem that the fixed CPU/GPU specifications of existing heterogeneous computing machines limit the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios The problem.

In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to a remote device with inference acceleration resources, and the interaction between the source computing device and the current computing device is irrelevant to the user. Perceptual, therefore, it can be ensured that the business logic of the application involving reasoning and the user's usage habits for reasoning business remain unchanged, and the reasoning is realized at low cost and the user experience is improved.

Example Six

The reasoning method of this embodiment still describes the reasoning method of the present invention from the perspective of the second computing device. Referring to FIG. 7, there is shown a flowchart of an inference method according to Embodiment 6 of the present invention, and the inference method includes:

Step S602: According to the model information of the calculation model, it is determined that the calculation model does not exist locally, then the source calculation device is requested for the calculation model, and after the calculation model is obtained from the source calculation device, pass the The inference acceleration resource is loaded into the calculation model.

In this embodiment, the model information of the calculation model may be the identification information or verification information of the calculation model; then, this step may be implemented as: according to the identification information or the verification information, it is determined that the local Computing model, request the computing model from the source computing device, and after obtaining the computing model from the source computing device, load the computing model through the inference acceleration resource. The calculation model sent by the source computing device includes, but is not limited to, the structure of the calculation model and its corresponding data.

In addition, in an optional manner, the reasoning acceleration resource includes one or more types; when the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities; then, The calculation model indicated by the inference acceleration resource loading the model information includes: according to a preset load balancing rule and the priority of multiple types of the inference acceleration resource, the inference acceleration resource is used to load the model information indication Calculation model.

Wherein, the load balancing rule and the priority can be appropriately set by those skilled in the art according to actual needs.

Step S604: Obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model.

In a feasible manner, this step can be implemented as: acquiring the data to be inferred sent by the source computing device and the information of the processing function in the computing model to be called, and calling the processing in the loaded computing model The processing function indicated by the function information performs inference processing on the data to be inferred. Wherein, the information of the processing function may be obtained by the source computing device by parsing the reasoning request.

Optionally, the information of the processing function is API interface information of the processing function.

Step S606: Feed back the result of the inference processing to the source computing device.

Through this embodiment, inference processing is deployed in different computing devices. The current computing device that executes the inference method of this embodiment is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model, and the source computing device It can be responsible for the data processing before and after the inference processing. When performing inference, the source computing device can first send the model information of the computing model to the current computing device, and the current computing device uses the inference acceleration resource to load the corresponding computing model; then, the source computing device sends the data to be inferred to the current computing device. After the current computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized by different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem that the fixed CPU/GPU specifications of existing heterogeneous computing machines limit the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios The problem.

Example Seven

Referring to FIG. 8, there is shown a schematic structural diagram of an electronic device according to the seventh embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

As shown in FIG. 8, the electronic device may include a processor (processor) 702, a communication interface (Communications Interface) 704, a memory (memory) 706, and a communication bus 708.

among them:

The processor 702, the communication interface 704, and the memory 706 communicate with each other through the communication bus 708.

The communication interface 704 is used to communicate with other electronic devices or servers.

The processor 702 is configured to execute the program 710, and specifically can execute the relevant steps in the inference method embodiment in the third or fourth embodiment.

Specifically, the program 710 may include program code, and the program code includes computer operation instructions.

The processor 702 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), for example, at least one disk memory.

The program 710 may specifically be used to cause the processor 702 to perform the following operations: obtain model information of a calculation model for inference, and send the model information to a target computing device to instruct the target computing device to use the target computing device The set inference acceleration resources are loaded into the calculation model indicated by the model information; the data to be inferred is obtained, and the data to be inferred is sent to the target computing device to instruct the target computing device to use the inference acceleration resource to call and load The calculation model is used to perform inference processing on the data to be inferred through the calculation model; and the result of the inference processing feedback from the target computing device is received.

In an optional implementation manner, the program 710 is further configured to cause the processor 702 to send the calculation model to the target computing device if it is determined that the calculation model does not exist in the target computing device.

In an optional implementation manner, the model information of the calculation model is identification information or verification information of the calculation model; the program 710 is also used to enable the processor 702 to determine whether the target computing device is in the When the calculation model does not exist, before sending the calculation model to the target computing device, determine whether the calculation model exists in the target computing device through the identification information or the verification information.

In an optional implementation manner, the program 710 is further configured to enable the processor 702 to obtain the data to be inferred and send the data to be inferred to the target computing device: Perform a reasoning request for inference processing on the data to be inferred, and perform semantic analysis on the reasoning request; determine the processing function in the calculation model to be called according to the result of the semantic analysis, and combine the information of the processing function and the data to be inferred Sent to the target computing device to instruct the target computing device to perform inference processing on the data to be inferred by calling the processing function indicated by the processing function information in the loaded calculation model.

In an optional implementation manner, the information of the processing function is API interface information of the processing function.

For the specific implementation of each step in the program 710, reference may be made to the corresponding description in the corresponding steps and units in the above-mentioned corresponding inference method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that, for convenience and concise description, the specific working process of the devices and modules described above can be referred to the corresponding process description in the foregoing method embodiment, which will not be repeated here.

Through the electronic device of this embodiment, the inference processing is deployed in different computing devices, where the target computing device is provided with inference acceleration resources, and the main inference processing can be performed through the computing model, and the inference method of this embodiment is executed. Current electronic equipment can be responsible for data processing before and after inference processing. When performing inference, the current electronic device can first send the model information of the calculation model to the target computing device, and the target computing device uses the inference acceleration resource to load the corresponding calculation model; then, the current electronic device sends the data to be inferred to the target computing device, After the target computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized by different computing devices, one of which is equipped with an inference acceleration resource such as a GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem that the fixed CPU/GPU specifications of existing heterogeneous computing machines limit the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios The problem.

In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to the remote target computing device with inference acceleration resources, and the current interaction between the electronic device and the target computing device is important to the user. It is imperceptible. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits for reasoning business remain unchanged, and the reasoning is realized at low cost and the user experience is improved.

Example eight

9, there is shown a schematic structural diagram of an electronic device according to the eighth embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

As shown in FIG. 9, the electronic device may include: a processor (processor) 802, a communication interface (Communications Interface) 804, a memory (memory) 806, and a communication bus 808.

among them:

The processor 802, the communication interface 804, and the memory 806 communicate with each other through the communication bus 808.

The communication interface 804 is used to communicate with other electronic devices or servers.

The processor 802 is configured to execute the program 810, and specifically can execute the relevant steps in the inference method embodiment in the fifth or sixth embodiment.

Specifically, the program 810 may include program code, and the program code includes computer operation instructions.

The processor 802 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), for example, at least one disk memory.

The program 810 may specifically be used to cause the processor 802 to perform the following operations: obtain model information of a calculation model used for inference sent by a source computing device, load the calculation model indicated by the model information through the inference acceleration resource, and obtain the source calculation For the data to be inferred sent by the device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model; and feed back the result of the inference processing to the source computing device.

In an optional implementation manner, the model information of the calculation model is identification information or verification information of the calculation model; the program 810 is also used to enable the processor 802 to obtain the reasoning information sent by the source computing device. When the model information of the calculation model is loaded into the calculation model indicated by the model information through the inference acceleration resource: according to the identification information or the verification information, it is determined that the calculation model does not exist locally, and then the source calculation device Request the calculation model, and after obtaining the calculation model from the source computing device, load the calculation model through the inference acceleration resource.

In an optional implementation manner, the program 810 is further configured to enable the processor 802 to obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and use the calculation model When performing inference processing on the data to be inferred: obtain the data to be inferred sent by the source computing device and the information of the processing function in the calculation model to be called, and call the processing function in the loaded calculation model. The processing function indicated by the information performs inference processing on the data to be inferred.

In an optional implementation manner, the reasoning acceleration resource includes one or more types; when the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities; program 810 It is also used to enable the processor 802 to load the calculation model indicated by the model information through the inference acceleration resource: according to preset load balancing rules and the priority of multiple types of the inference acceleration resource, use the inference acceleration resource to load Enter the calculation model indicated by the model information.

For the specific implementation of each step in the program 810, reference may be made to the corresponding description in the corresponding steps and units in the above-mentioned corresponding inference method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that, for convenience and concise description, the specific working process of the devices and modules described above can be referred to the corresponding process description in the foregoing method embodiment, which will not be repeated here.

Through the electronic device of this embodiment, inference processing is deployed in different computing devices. Among them, the current electronic device that executes the inference method of this embodiment is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model. The source computing device can then be responsible for the data processing before and after the inference processing. When performing inference, the source computing device can first send the model information of the calculation model to the current electronic device, and the current electronic device uses the inference acceleration resource to load the corresponding calculation model; then, the source computing device sends the data to be inferred to the current electronic device. After the current electronic device receives the data to be inferred, it can perform inference processing through the loaded calculation model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized by different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem that the fixed CPU/GPU specifications of existing heterogeneous computing machines limit the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios The problem.

In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to a remote device with inference acceleration resources, and the interaction between the source computing device and the current electronic device is irrelevant to the user. Perceptual, therefore, it can be ensured that the business logic of the application involving reasoning and the user's usage habits for reasoning business remain unchanged, and the reasoning is realized at low cost and the user experience is improved.

It should be pointed out that according to the needs of implementation, each component/step described in the embodiment of the present invention can be split into more components/steps, or two or more components/steps or partial operations of components/steps can be combined into New components/steps to achieve the purpose of the embodiments of the present invention.

The above method according to the embodiments of the present invention can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or implemented by The computer code downloaded from the network is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium, so that the method described here can be stored in a general-purpose computer, a special-purpose processor, or a programmable Or such software processing on a recording medium of dedicated hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is used by the computer, When the processor or hardware accesses and executes, it implements the inference method described here. In addition, when a general-purpose computer accesses the code for implementing the inference method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the inference method shown here.

A person of ordinary skill in the art may be aware that the units and method steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the embodiments of the present invention.

The above implementations are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those of ordinary skill in the relevant technical field can also make various modifications without departing from the spirit and scope of the embodiments of the present invention. Changes and modifications, therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.

Claims

An inference system, characterized by comprising a first computing device and a second computing device connected to each other, wherein the first computing device is provided with an inference client, and the second computing device is provided with an inference acceleration resource And reasoning server;

among them:

The inference client is used to obtain model information and data to be inferred of a calculation model for inference, and respectively send the model information and the data to be inferred to the inference server in the second computing device;

The reasoning server is used to load and call the calculation model indicated by the model information through the reasoning acceleration resource, perform reasoning processing on the data to be reasoned through the calculation model, and feed back the reasoning processing to the reasoning client the result of.
The reasoning system according to claim 1, wherein the reasoning client is further configured to send the calculation model to the reasoning service when it is determined that the calculation model does not exist in the second computing device end.
The reasoning system according to claim 2, wherein the model information of the calculation model is identification information or verification information of the calculation model;

The reasoning server is further configured to determine whether the calculation model exists in the second computing device through the identification information or the verification information, and return the determination result to the reasoning client.
The reasoning system according to claim 1, wherein:

The reasoning client is also used to obtain a reasoning request requesting the calculation model to perform reasoning processing on the data to be reasoned, perform semantic analysis on the reasoning request, and determine the calculation model to be called according to the semantic analysis result The processing function of, sending the information of the processing function to the inference server;

When the inference server performs inference processing on the data to be inferred through the calculation model, it calls the processing function indicated by the information of the processing function in the loaded calculation model to perform the inference process on the data to be inferred. Data is processed by reasoning.
The reasoning system according to claim 4, wherein the information of the processing function is API interface information of the processing function.
The reasoning system according to claim 1, wherein one or more types of reasoning acceleration resources are provided in the second computing device;

When the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities;

The reasoning server uses the reasoning acceleration resource according to the preset load balancing rules and the priority of the multiple types of the reasoning acceleration resource.
The reasoning system according to any one of claims 1 to 6, wherein the first computing device and the second computing device are connected to each other through an elastic network.
The reasoning system according to any one of claims 1-6, wherein the reasoning client is a component inside a deep learning framework embedded in the first computing device, or the reasoning client is a A callable file called by the deep learning framework.
A reasoning method, characterized in that the method includes:

Acquire model information of a calculation model for inference, and send the model information to a target computing device to instruct the target computing device to use the inference acceleration resource set in the target computing device to load the calculation indicated by the model information model;

Obtain the data to be inferred, and send the data to be inferred to the target computing device to instruct the target computing device to use the inference acceleration resource to call the loaded calculation model, and use the calculation model to analyze the to-be inferred Data is processed by reasoning;

Receiving the result of the inference processing fed back by the target computing device.
The method according to claim 9, wherein the method further comprises:

If it is determined that the calculation model does not exist in the target computing device, the calculation model is sent to the target computing device.
The method according to claim 10, wherein the model information of the calculation model is identification information or verification information of the calculation model;

When it is determined that the calculation model does not exist in the target computing device, before sending the calculation model to the target computing device, the method further includes: passing the identification information or the verification Information to determine whether the calculation model exists in the target computing device.
The method according to claim 9, wherein the acquiring the data to be inferred and sending the data to be inferred to the target computing device comprises:

Acquiring a reasoning request requesting the calculation model to perform reasoning processing on the data to be reasoned, and performing semantic analysis on the reasoning request;

Determine the processing function in the calculation model to be called according to the semantic analysis result, and send the information of the processing function and the data to be inferred to the target computing device to instruct the target computing device to load by calling The processing function indicated by the information of the processing function in the calculation model performs inference processing on the data to be inferred.
The method according to claim 12, wherein the information of the processing function is API interface information of the processing function.
A reasoning method, characterized in that the method includes:

Acquire model information of a calculation model used for inference sent by a source computing device, and load the calculation model indicated by the model information through the inference acceleration resource;

Obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model;

The result of the inference processing is fed back to the source computing device.
The method according to claim 14, wherein the model information of the calculation model is identification information or verification information of the calculation model;

The acquiring the model information of the calculation model used for inference sent by the source computing device, and loading the calculation model indicated by the model information through the inference acceleration resource includes:

According to the identification information or the verification information, it is determined that the calculation model does not exist locally, the calculation model is requested from the source computing device, and after the calculation model is obtained from the source computing device, pass The reasoning acceleration resource is loaded into the calculation model.
The method according to claim 14, wherein the acquiring the data to be inferred sent by the source computing device, using the inference acceleration resource to call the loaded calculation model, and using the calculation model to analyze the data to be inferred Data is processed by reasoning, including:

Obtain the data to be inferred sent by the source computing device and the information of the processing function in the calculation model to be invoked, and by calling the processing function indicated by the information of the processing function in the loaded calculation model, the data to be inferred Data is processed by reasoning.
The method according to claim 16, wherein the information of the processing function is API interface information of the processing function.
The method according to claim 14, wherein the reasoning acceleration resource includes one or more types;

When the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities;

The calculation model indicated by the inference acceleration resource loading the model information includes: according to a preset load balancing rule and the priority of multiple types of the inference acceleration resource, the inference acceleration resource is used to load the model information indication Calculation model.
An electronic device, comprising: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus;

The memory is used to store at least one executable instruction, the executable instruction causes the processor to perform an operation corresponding to the inference method according to any one of claims 9-13, or the executable instruction causes The processor executes operations corresponding to the inference method according to any one of claims 14-18.
A computer storage medium with a computer program stored thereon, and when the program is executed by a processor, the inference method according to any one of claims 9-13 is realized; or, any one of claims 14-18 is realized The reasoning method described.