TW202119255A - Inference system, inference method, electronic device and computer storage medium - Google Patents

Abstract

Provided are an inference system, an inference method, an electronic device and a computer storage medium. The inference system comprises a first computing device and a second computing device that are connected to each other, with the first computing device being provided with an inference client, and the second computing device comprising an inference acceleration resource and an inference server, wherein the inference client is used for acquiring model information of a computing model for inference and data to be inferred, and for respectively sending the model information and said data to the inference server in the second computing device; and the inference server is used for loading and calling, by means of the inference acceleration resource, the computing model indicated by the model information, and performing, by means of the computing model, inference processing on said data, and feeding back the result of the inference processing to the inference client.

Description

Reasoning system, reasoning method, electronic equipment and computer storage medium

The embodiments of the present invention relate to the field of computer technology, and in particular to an inference system, an inference method, an electronic device, and a computer storage medium.

Deep learning is generally divided into two parts: training and inference. The training part is used to search for and solve the optimal parameters of the model, and through the inference part, the trained model can be deployed in the online environment for actual use. . Take the field of artificial intelligence as an example. After inference is deployed, the input can be converted into a specific target output through neural network deduction calculations. For example, object detection on pictures, classification of text content, etc., are widely used in scenes such as vision, voice, and recommendation. At present, most inferences rely on hardware computing resources with inference acceleration cards such as GPU (Graphics Processing Unit, graphics processing unit). For example, in artificial intelligence inference, one way is to connect the GPU to the host computer through a PCIE (Peripheral Component Interconnect Express, high-speed serial computer expansion bus standard) slot. Among them, the pre-processing and other business logic involved in the inference are calculated by the CPU, and the processing of the inference is sent to the GPU for calculation through the PCIE slot, forming a typical heterogeneous computing scenario. For example, in the electronic device 100 shown in FIG. 1, a CPU 102 and a GPU 104 are provided at the same time, and the GPU 104 can be set on the electronic device motherboard 108 through the PCIE slot 106 , and through the motherboard circuit on the motherboard 108 The CPU 102 interacts. In a reasoning process, CPU 102 first of all relevant data or information to be processed, and then the processed data or information sent to the GPU 104 through the PCIE slot 106, GPU 104 according to the received data or information, calculated using the GPU 104 The model performs inference processing, and then returns the inference processing result to the CPU 102 , and the CPU 102 performs corresponding subsequent processing. However, the above method has the following problems: a heterogeneous computing machine with the same CPU and GPU is required, and the specifications of the CPU/GPU in the heterogeneous computing machine are fixed. This fixed CPU/GPU performance ratio limits the reasoning involved The deployment of applications makes it impossible to meet the needs of a wide range of reasoning scenarios.

In view of this, the embodiment of the present invention provides a reasoning solution to solve some or all of the above-mentioned problems. According to a first aspect of the embodiments of the present invention, there is provided an inference system, including a first computing device and a second computing device connected to each other, wherein the first computing device is provided with an inference client, and the second computing device The equipment is equipped with inference acceleration resources and inference servers; wherein: the inference client is used to obtain model information and data to be inferred of the calculation model for inference, and respectively send the model information and the data to be inferred to The inference server in the second computing device; the inference server is used to load and call the calculation model indicated by the model information through the inference acceleration resource, and perform inference processing on the data to be inferred through the calculation model And feedback the result of the reasoning processing to the reasoning client. According to a second aspect of the embodiments of the present invention, there is provided an inference method, the method comprising: obtaining model information of a calculation model for inference, and sending the model information to a target computing device to instruct the target computing device Use the inference acceleration resource set in the target computing device to load the calculation model indicated by the model information; obtain the data to be inferred, and send the data to be inferred to the target computing device to instruct the target computing device Use the calculation model loaded by the inference acceleration resource call to perform inference processing on the data to be inferred through the calculation model; receive the result of the inference processing as feedback from the target computing device. According to a third aspect of the embodiments of the present invention, another reasoning method is provided, the method comprising: acquiring model information of a calculation model used for reasoning sent by a source computing device, and loading the information indicated by the model information through the reasoning acceleration resource Calculation model; obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model; feedback to the source computing device The result of the inference process. According to a fourth aspect of the embodiments of the present invention, there is provided an electronic device including: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface pass through the communication bus. The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operation corresponding to the reasoning method described in the second aspect, or the executable Instruct the processor to perform operations corresponding to the inference method described in the third aspect. According to a fifth aspect of the embodiments of the present invention, there is provided a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the reasoning method as described in the second aspect is realized; or, as described in the third aspect. The reasoning method described. According to the reasoning scheme provided by the embodiment of the present invention, the reasoning process is deployed in different first and second computing devices, wherein the second computing device is provided with inference acceleration resources, and the main reasoning process can be performed through the calculation model, and The first computing device can then be responsible for data processing before and after inference processing. In addition, an inference client is deployed in the first computing device, and an inference server is deployed in the second computing device. During inference, the first computing device and the second computing device interact through the inference client and the inference server. The inference client can first send the model information of the calculation model to the inference server, and the inference server uses the inference acceleration resource to load the corresponding calculation model; then, the inference client sends the data to be inferred to the inference server, and the inference server is receiving After the data to be inferred, inference processing can be performed through the loaded calculation model. In this way, the decoupling of computing resources used for inference is realized. Inference processing through the computing model and data processing other than inference processing can be realized through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of the existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving reasoning, the reasoning calculation can be seamlessly transferred to the remote device with reasoning acceleration resources through the reasoning client and the reasoning server, and the reasoning client and reasoning The interaction between the servers is imperceptible to the user. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits for reasoning business remain unchanged, and the reasoning is realized at low cost and the user experience is improved. .

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the description The embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art should fall within the protection scope of the embodiments of the present invention. The specific implementation of the embodiments of the present invention will be further described below in conjunction with the drawings of the embodiments of the present invention. Embodiment 1 Referring to FIG. 2a, it shows a block diagram of a structure of an inference system according to Embodiment 1 of the present invention. The inference system of this embodiment includes a first computing device 202 and a second computing device 204 connected to each other, wherein the first computing device 202 is provided with an inference client 2022 , and the second computing device 204 is provided with an inference server 2042 and Inference acceleration resources 2044 . Wherein, the inference client 2022 is used to obtain model information and data to be inferred of the calculation model for inference, and respectively send the model information and the data to be inferred to the inference server 2042 in the second computing device 204 The reasoning server 2042 is used to load and call the calculation model indicated by the model information through the reasoning acceleration resource 2044 , perform reasoning processing on the data to be reasoned through the calculation model, and return the reasoning processing to the reasoning client 2022 the result of. In a feasible implementation, the inference client 2022 in the first computing device 202 first obtains model information of the computing model for inference, and sends the model information to the inference server 2042 in the second computing device 204 ; The inference server 2042 in the second computing device 204 loads the computing model indicated by the model information through the inference acceleration resource 2044 ; the inference client 2022 in the first computing device 202 then obtains the data to be inferred, and sends the data to be inferred to second computing device 204 reasoning server end 2042; a second computing device 204 in server end 2042 reasoning using reasoning model acceleration computing resource 2044 calls the loading of data through the treatment reasoning model calculation and inference processing The reasoning client 2022 feeds back the result of the reasoning process. In the above recommendation system, because the second computing device 204 has the reasoning acceleration resource 2044 used for reasoning, it can effectively load a calculation model for reasoning and perform reasoning calculations with a large amount of data. Also, because the second computing device 204 where the inference acceleration resource 2044 is located and the first computing device 202 are independently set, the inference acceleration resource 2044, such as the GPU, does not need to follow a fixed specification setting with the processor resources in the first computing device 202, such as the CPU, so that The realization of the inference acceleration resource 2044 is more flexible and diverse. Among them, the inference acceleration resource 2044 can be implemented in various forms including but not limited to GPU and NPU. Therefore, it is only necessary to configure a resource such as a CPU for routine processing of data in 202 in the first computing device. The calculation model for reasoning may be any appropriate calculation model set according to business requirements, which may be applicable to deep learning frameworks (including but not limited to Tensorflow framework, Mxnet framework, and PyTorch framework). In a feasible manner, the second computing device 204 can be pre-configured with a resource pool of the computing model. If the computing model to be used is in the resource pool, it can be directly loaded and used; if it is not in the resource pool, it can be loaded from the first Obtained from a computing device 202. In another feasible manner, the second computing device 204 may not have a preset resource pool of the computing model. When inference is needed, the required computing model is obtained from the first computing device 202 and then stored locally. After multiple inferences, different calculation models can be obtained and finally stored as a resource pool of the calculation model. The obtained different computing models may come from different first computing devices 202 , that is, the second computing device 204 may provide inference services for different first computing devices 202 , so as to obtain different information from different first computing devices 202. Calculation model. The model information of the calculation model sent by the inference client 2022 to the inference server 2042 can uniquely identify the calculation model, for example, it can be the identification information of the calculation model, such as an identification ID number. But it is not limited to this. In a feasible way, the model information of the calculation model can also be the verification information of the calculation model, such as MD5 information. The verification information can identify the calculation model on the one hand, and can also calibrate the calculation model on the other hand. Experience, realize multiple functions through one kind of information, reduce the cost of information processing. The model information can be obtained when the first computing device 202 loads the model. Hereinafter, a specific example is used to illustrate the structure of the above-mentioned reasoning system, as shown in Fig. 2b. In Figure 2b, the first computing device 202 is implemented as a terminal device, that is, the first terminal device, in which a processor CPU is provided for corresponding service processing, and the second computing device 204 is also implemented as a terminal device, that is, the second terminal. The device, which is provided with the inference acceleration resource GPU. In addition, the first computing device 202 is loaded with a deep learning framework and a reasoning client set in the deep learning framework; the second computing device 204 is correspondingly set with a reasoning server. In this embodiment, it is also set that the second computing device 204 is provided with a resource pool of computing models, in which multiple computing models, such as computing models A, B, C, and D are stored. It should be understood by those skilled in the art that the above examples are only illustrative. In practical applications, the first computing device 202 and the second computing device 204 can both be implemented as terminal devices, or both can be implemented as servers, or, The first computing device 202 is implemented as a server and the second computing device 204 is implemented as a terminal device or vice versa, which is not limited in the embodiment of the present invention. Based on the reasoning system in Figure 2b, a process of using the reasoning system for reasoning is as follows. Take image recognition as an example. When the deep learning framework loads the model, the corresponding model information can be obtained. The inference client sends the calculation model information to the second terminal device, and the second terminal device receives the calculation through the inference server. Model information. Assuming that the information of the calculation model indicates that the calculation model to be used is calculation model A, and the resource pool of the second terminal device stores calculation models A, B, C, and D, then the second terminal device will directly access the resource through the GPU Load calculation model A in the pool. Furthermore, the second terminal device obtains the data to be inferred, such as the image to be recognized, from the first terminal device through the inference server and the inference client, and then the second terminal device uses the GPU to call the computing model A to perform target object recognition on the image, such as Identify whether there are people in the image. After the recognition, the second terminal device will send the recognition result to the inference client of the first terminal device through the inference server, and the inference client will hand it over to the CPU for subsequent processing, such as adding AR special effects. It should be noted that, in the embodiments of the present invention, unless otherwise specified, quantities such as "multiple" and "multiple" related to "multiple" mean two or more. According to the inference system provided in this embodiment, the inference processing is deployed in different first and second computing devices. The second computing device is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model. A computing device can be responsible for data processing before and after inference processing. In addition, an inference client is deployed in the first computing device, and an inference server is deployed in the second computing device. During inference, the first computing device and the second computing device interact through the inference client and the inference server. The inference client can first send the model information of the calculation model to the inference server, and the inference server uses inference acceleration resources to load the corresponding calculation model; then, the inference client sends the data to be inferred to the inference server, and the inference server is receiving After the data to be inferred, inference processing can be performed through the loaded calculation model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving reasoning, the reasoning calculation can be seamlessly transferred to the remote device with reasoning acceleration resources through the reasoning client and the reasoning server, and the reasoning client and reasoning The interaction between the servers is imperceptible to the user. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits for reasoning business remain unchanged, and the reasoning is realized at low cost and the user experience is improved. . Embodiment 2 This embodiment further optimizes the reasoning system in Embodiment 1, as shown in Fig. 3a. As described in the first embodiment, the inference system of this embodiment includes: a first computing device 202 and a second computing device 204 connected to each other, wherein the first computing device 202 is provided with an inference client 2022 , and the second computing device In 204 , an inference server 2042 and an inference acceleration resource 2044 are provided . The first computing device 202 in the UE 2022 reasoning for obtaining information about the model calculation model reasoning, and transmits the information to the second computing device model 204 in reasoning server end 2042; in the second computing device 204 The reasoning server 2042 is used to load the calculation model indicated by the model information through the reasoning acceleration resource 2044 ; the reasoning client 2022 in the first computing device 202 is also used to obtain the data to be reasoned, and send the data to be reasoned to the second the computing device 204 reasoning server end 2042; a second computing device 204 in server end 2042 further reasoning using reasoning model acceleration computing resource 2044 calls the loading of data through the treatment reasoning model calculation and inference processing The result of the inference processing is fed back to the inference client 2022. In one feasible manner, the first computing device 202 and the second computing device 204 are connected to each other through an elastic network. The elastic network includes, but is not limited to, an ENI (Elastic Network Interface) network. The elastic network has better scalability and flexibility. The first computing device 202 and the second computing device 204 are connected through the elastic network, so that the reasoning system also has better scalability and flexibility. However, it is not limited to this. In practical applications, the first computing device 202 and the second computing device 204 can be connected in any suitable manner or network, and the data interaction between the two can be smoothly realized. In this embodiment, optionally, the inference client 2022 can be implemented as a component embedded in the deep learning framework in the first computing device 202 , or the inference client can be implemented as a component that can be called by the deep learning framework. The callable file. The deep learning framework provides a platform for deep learning. Based on the deep learning framework, programmers can easily deploy various computing models to achieve different inference functions. Implementing the inference client 2022 in the form of a component or callable file suitable for a deep learning framework on the one hand makes it more compatible and applicable, on the other hand, it also greatly reduces the implementation cost of decoupling inference computing resources. Similarly, the inference server 2042 can also be implemented in the form of a component or a callable file. Based on the above structure, the reasoning system of this embodiment can conveniently interact with corresponding data and information through the reasoning client 2022 and the reasoning server 2042 , and realize reasoning processing by remotely calling the reasoning acceleration resource 2044. In addition, in this embodiment, the inference client 2022 is also used to send the calculation model to the inference server 2042 when it is determined that the calculation model does not exist in the second computing device 204 . Optionally, the model information of the calculation model is identification information or verification information of the calculation model; the inference server 2042 is also used to determine the identification information or the verification information in the second computing device 204 Whether the calculation model exists, and the determination result is returned to the inference client 2022 . However, it is not limited to this, and other methods of determining whether the second computing device 204 has the calculation model are also applicable, for example, the second computing device 204 broadcasts the calculation model it has at regular intervals; or, when needed Or at regular intervals, the first computing device 202 actively sends a message to inquire about the resource status of the computing model in the second computing device 204, and so on. For example, if the resource pool of the computing model is not preset in the second computing device 204 or there is no required computing model in the resource pool, the inference client 2022 will send the computing model in the first computing device 202 to the second computing device. 204 , including but not limited to the structure of the calculation model and the data it contains. When the information of the calculation model sent by the inference client 2022 to the inference server 2042 is identification information or verification information, the inference server 2042 will first determine the second computing device 204 according to the received identification information or verification information. Whether the required calculation model exists in the database, and the result of the determination is returned to the inference client 2022 . If the result of the determination indicates that the required calculation model does not exist in the second computing device 204 , the first computing device 202 obtains the calculation model locally and sends it to the second computing device 204 , through the inference of the second computing device 204 Accelerate resources to run the calculation model and perform inference processing. In this way, it can be effectively ensured that the second computing device 204 with inference acceleration resources can successfully complete inference processing. In addition, in a feasible manner, the reasoning client 2022 is also used to obtain a reasoning request requesting the calculation model to perform reasoning processing on the data to be reasoned, and perform semantic analysis on the reasoning request, according to the semantic analysis As a result, the processing function in the calculation model to be called is determined, and the information of the processing function is sent to the inference server 2042 ; the inference server 2042 is processing the data to be inferred through the calculation model When inference processing is performed, inference processing is performed on the data to be inferred by calling the processing function indicated by the information of the processing function in the loaded calculation model. In some specific business applications of reasoning, the business may not need all the reasoning functions of the calculation model, but only some of the functions. For example, a certain reasoning is used to classify text content, and the current business only needs the calculation function in it to add the corresponding text vector. In this case, when the reasoning client 2022 receives a request for a calculation model to calculate the text After the vector performs the reasoning request for the addition calculation, through semantic analysis of the reasoning request, it is determined that only the COMPUTE() function in the calculation model needs to be called, and then the information of the function can be sent to the reasoning server 2042 . After the inference server 2042 obtains the information of the function, it can directly call the COMPUTE() function in the calculation model to perform the addition calculation of the text vector. It can be seen that in this way, the use of the calculation model is more accurate, the reasoning efficiency of the calculation model is greatly improved, and the reasoning burden is reduced. In a feasible way, the processing function information can be the API interface information of the processing function. Through the API interface information, the processing function in the calculation model to be used can be quickly determined, and the corresponding function can also be directly obtained Interface information for direct use in subsequent inference processing. Optionally, one or more types of inference acceleration resources are provided in the second computing device 204 ; when the inference acceleration resources include multiple types, the different types of inference acceleration resources have different usage priorities; the inference server 2042 is based on a preset The load balancing rules and the priority of various types of inference acceleration resources, use inference acceleration resources. For example, in addition to the GPU provided in the second computing device 204 , an NPU or other inference acceleration resources may also be provided. There is a certain priority among multiple inference acceleration resources, and the priority can be set in any appropriate manner, such as setting according to operating speed or manually, etc., which is not limited in the embodiment of the present invention. Further optionally, a CPU may also be provided in the second computing device 204. In this case, you can set the GPU to use the highest priority, the NPU second, and the CPU priority to be the lowest. In this way, when high-priority inference acceleration resources are heavily loaded, lower-priority inference acceleration resources can be used for inference processing. On the one hand, it ensures that inference processing can be effectively executed, and on the other hand, it can also reduce the cost of inference acceleration resources. It should be noted that the number of a certain type of reasoning acceleration resource may be one or multiple, which can be set by a person skilled in the art according to requirements, which is not limited in the embodiment of the present invention. In addition, in addition to the above-mentioned load balancing according to the priority of the preset load balancing rules, those skilled in the art can also set other appropriate load balancing rules according to actual needs, which is not limited in the embodiment of the present invention. Hereinafter, a specific example is used to describe the inference system in this embodiment. As shown in Figure 3b, unlike the traditional CPU and inference acceleration resources such as GPU in the same electronic device architecture, in this example, the CPU and inference acceleration resources are decoupled into two parts, namely the CPU client machine in Figure 3b. User machine) and Server accelerator pools (background reasoning accelerator card machine). Among them, the front-end user machine is a machine that can be operated by the user to perform reasoning business, and the back-end reasoning accelerator card machine is used for reasoning calculations. The communication between the two is realized through ENI. There are multiple inference frameworks in the front-end user's machine, as shown in the figure as "Tensorflow inference code", "pyTorch inferce code" and "Mxnet inference code". There are a variety of inference accelerator cards in the background inference accelerator card machine, which are shown as "GPU", "Ali-NPU" and "Other Accelerator" in the figure. In order to forward the reasoning business of the front-end user machine to the back-end accelerator card for execution and return the result of the reasoning to realize the user-side non-sense support, this example provides two components that reside in the front-end user machine and the background accelerator card machine respectively. It is the EAI client module (that is, the inference client) and the service daemon (that is, the inference server). Among them, the EAI client module is a component in the front-end user machine, and its functions include: a) communicate with the back-end service daemon through the network; b) analyze the semantics of the calculation model and inference requests; c) analyze the semantics of the calculation model And the analysis result of the reasoning request is sent to the back-end service daemon; d) The reasoning result sent by the service daemon is received and returned to the deep learning framework. In one implementation, the EAI client module can be implemented as a plugin module and embedded in the function code of a deep learning framework (such as Tensorflow/pyTorch/Mxnet, etc.). When the inference business loads the calculation model through the deep learning framework, the EAI client module It intercepts the loaded calculation model, analyzes the semantics of the calculation model to generate calculation model information, such as verification information (may be MD5 information), transfers calculation model and/or calculation model information, and transfers subsequent operations to the backend The service daemon performs actual inference calculations. Service daemon is the resident service component of the background reasoning accelerator card machine. Its functions include: a) receiving the information of the calculation model sent by the EAI client module and the analysis result of the reasoning request; b) according to the information of the calculation model and the analysis result of the reasoning request, Select the best reasoning accelerator card in the background reasoning accelerator card machine; c) send the reasoning calculation to the reasoning accelerator card; d) receive the reasoning result calculated by the reasoning accelerator card and return it to the EAI client module. Among them, GPU, Ali-NPU and Other Accelerator have a certain priority, for example, GPU->Ali-NPU->Other Accelerator from high to low, then in actual use, the GPU will be used first. If the GPU resource If Ali-NPU is not enough, if Ali-NPU resources are still insufficient, use Other Accelerator. It can be seen that, unlike the traditional binding of CPU and GPU inference accelerator cards to one machine through the PCIE card slot, the flexible remote inference in this example combines the CPU client machine and the inference accelerator card (server) through the flexible network card. accelerator pools) decoupling, for users, it is no longer necessary to buy a machine with the same CPU and GPU. The reasoning process based on the reasoning system shown in Figure 3b is shown in Figure 3c, including: Step ①, when the user starts the reasoning task through the deep learning framework and loads the calculation model, the EAI client module intercepts and analyzes the semantics of the calculation model to obtain the calculation Model information; further, obtain the user’s reasoning request, analyze the reasoning request, and obtain the information of the processing function to be used in the calculation model; step ②, the EAI client module connects with the service daemon through the elastic network to connect the calculation model The information and processing function information are forwarded to the back-end service daemon; step ③, the service daemon selects the optimal inference accelerator card based on the information of the calculation model and processing function information, and loads the calculation model through the inference accelerator card to perform inference Calculation; step ④, the inference accelerator card returns the result of the inference calculation to the service daemon; step ⑤, the service daemon forwards the result of the inference calculation to the EAI client daemon through the elastic network; step o, the EAI client daemon returns the result of the inference calculation Give the deep learning framework. As a result, the user conducts inference business on the front-end user machine, and the EAI client module and service daemon automatically forward the user's inference business to the remote inference accelerator card for inference calculation in the background, and return the result of the inference calculation to the front-end user machine. The deep learning framework enables users to make insensible and flexible reasoning, and they can enjoy reasoning acceleration services without changing the reasoning code. Moreover, users do not need to purchase a machine with a GPU, and only need to use an ordinary CPU machine to achieve the same inference acceleration effect, and there is no need to modify any code logic. In a specific example where the deep learning framework is the Tensorflow framework, the interaction between the front-end user machine and the back-end inference accelerator card machine is shown in Figure 3d. The reasoning interaction includes: step 1, the front-end user machine loads the calculation model through the Tensorflow framework; step 2, the EAI client module intercepts the calculation model and verifies the model; step 3, the EAI client module establishes a channel with the service daemon, and transmits the calculation model; step 4. The service daemon analyzes the calculation model and selects the optimal reasoning accelerator from the accelerator card pool according to the analysis results; step 5, the selected reasoning accelerator is loaded into the calculation model; step 6, the user enters a picture/text and initiates a reasoning request ; Step 7,; Step 8,; Step 9, the service daemon forwards the processing function information and the data to be processed to the inference accelerator; Step 10, the inference accelerator performs inference calculations through the calculation model and sends the inference results of the inference calculations to the service daemon; Step 11, the service daemon transmits the inference results to the EAI client module; Step 12, the EAI client module receives the inference results and transfers the inference results to the Tensorflow framework; Step 13, the Tensorflow framework displays the inference results to the user. As a result, the flexible reasoning process under the Tensorflow framework is realized. According to the inference system provided in this embodiment, the inference processing is deployed in different first and second computing devices. The second computing device is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model. A computing device can be responsible for data processing before and after inference processing. In addition, an inference client is deployed in the first computing device, and an inference server is deployed in the second computing device. During inference, the first computing device and the second computing device interact through the inference client and the inference server. The inference client can first send the model information of the calculation model to the inference server, and the inference server uses inference acceleration resources to load the corresponding calculation model; then, the inference client sends the data to be inferred to the inference server, and the inference server is receiving After the data to be inferred, inference processing can be performed through the loaded calculation model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving reasoning, the reasoning calculation can be seamlessly transferred to the remote device with reasoning acceleration resources through the reasoning client and the reasoning server, and the reasoning client and reasoning The interaction between the servers is imperceptible to the user. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits for reasoning business remain unchanged, and the reasoning is realized at low cost and the user experience is improved. . Embodiment 3 Referring to FIG. 4, it shows a flowchart of an inference method according to Embodiment 3 of the present invention. The reasoning method of this embodiment describes the reasoning method of the present invention from the perspective of the first computing device. The inference method of this embodiment includes the following steps: Step S302: Obtain model information of a calculation model for inference, and send the model information to a target computing device to instruct the target computing device to use the settings in the target computing device The reasoning acceleration resource of is loaded into the calculation model indicated by the model information. The implementation of the target computing device in this embodiment can refer to the second computing device in the foregoing embodiment. The execution of this step can refer to the relevant parts of the inference client in the foregoing multiple embodiments. For example, the model information of the calculation model can be obtained when the calculation model is loaded in the deep learning framework, and then sent to the target computing device. After the target computing device receives the model information, it loads the corresponding computing model through corresponding inference acceleration resources such as GPU. Step S304: Obtain the data to be inferred, and send the data to be inferred to the target computing device to instruct the target computing device to use the inference acceleration resource to call the loaded calculation model, and use the calculation model to compare the data to the target computing device. Describe the data to be inferred for inference processing. Among them, as mentioned above, the data to be inferred is any appropriate data used for inference calculation using a calculation model. After the target computing device loads the calculation model, the data to be inferred can be sent to the target computing device. After the target computing device receives the data to be inferred, it uses inference acceleration resources such as the calculation model loaded by the GPU to perform inference processing on the inferred data. Step S306: Receive the result of the inference processing fed back by the target computing device. After the target computing device uses the calculation model loaded by the GPU to perform inference processing on the inference data to be inferred, the result of the inference processing is obtained and sent to the execution subject of this embodiment such as the inference client, and the inference client receives the inference processing result. result. In specific implementation, the inference method of this embodiment can be implemented by the inference client of the first computing device in the foregoing embodiment, and the specific implementation of the foregoing process can also refer to the operation of the inference client in the foregoing embodiment, which will not be repeated here. Through this embodiment, inference processing is deployed in different computing devices. Among them, the target computing device is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model, and the current computing device that executes the inference method of this embodiment It can be responsible for the data processing before and after the reasoning processing. When performing inference, the current computing device can first send the model information of the computing model to the target computing device, and the target computing device uses inference acceleration resources to load the corresponding computing model; then, the current computing device sends the data to be inferred to the target computing device. After the target computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to the remote target computing device with inference acceleration resources, and the interaction between the current computing device and the target computing device It is imperceptible to users. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits of reasoning services remain unchanged, and the reasoning is realized at low cost and the user experience is improved. Embodiment 4 The reasoning method of this embodiment still describes the reasoning method of the present invention from the perspective of the first computing device. 5, there is shown a flow chart of an inference method according to the fourth embodiment of the present invention. The inference method includes: Step S402: Obtain model information of a calculation model for inference, and send the model information to a target computing device . In a feasible manner, the model information of the calculation model is identification information or verification information of the calculation model. In a specific implementation, the first computing device may send the identification information or verification information to the target computing device, such as the second computing device in the foregoing embodiment, the target computing device determines according to the identification information or verification information Whether there is a corresponding calculation model locally, and the judgment result is fed back to the first computing device, so that the first computing device can determine whether there is a calculation model in the target computing device. Step S404: If it is determined according to the model information that the calculation model does not exist in the target computing device, the calculation model is sent to the target computing device, and the target computing device is instructed to use the target computing device The reasoning acceleration resource set in is loaded into the calculation model. When the model information of the calculation model is the identification information or verification information of the calculation model, this step can be implemented as: if the identification information or the verification information is used, it is determined that the target computing device does not exist The calculation model sends the calculation model to the target calculation device. Including: sending the structure of the calculation model and its data to the target computing device. If the required calculation model does not exist in the target computing device, the calculation model can be sent to the target computing device. The target computing device obtains and stores the calculation model, and if the calculation model is used again in subsequent use, the target computing device can directly obtain it locally. Therefore, it is ensured that the inference processing can be smoothly realized regardless of whether the required calculation model is available in the target computing device. Step S406: Obtain the data to be inferred, and send the data to be inferred to the target computing device to instruct the target computing device to use the inference acceleration resource to call the loaded computing model, and use the computing model to compare the data to the target computing device. Describe the data to be inferred for inference processing. In a feasible manner, the acquiring the data to be inferred and sending the data to be inferred to the target computing device may include: acquiring a reasoning request requesting the calculation model to perform inference processing on the data to be inferred, and Perform semantic analysis on the reasoning request; determine the processing function in the calculation model to be called according to the semantic analysis result, and send the processing function information and the inference data to the target computing device to Instruct the target computing device to perform inference processing on the data to be inferred by calling the processing function indicated by the information of the processing function in the loaded calculation model. Wherein, the information of the processing function can optionally be the API interface information of the processing function. Step S408: Receive the result of the inference processing fed back by the target computing device. In specific implementation, the inference method of this embodiment can be implemented by the inference client of the first computing device in the foregoing embodiment, and the specific implementation of the foregoing process can also refer to the operation of the inference client in the foregoing embodiment, which will not be repeated here. Through this embodiment, inference processing is deployed in different computing devices. Among them, the target computing device is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model, and the current computing device that executes the inference method of this embodiment It can be responsible for the data processing before and after the reasoning processing. When performing inference, the current computing device can first send the model information of the computing model to the target computing device, and the target computing device uses inference acceleration resources to load the corresponding computing model; then, the current computing device sends the data to be inferred to the target computing device. After the target computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to the remote target computing device with inference acceleration resources, and the interaction between the current computing device and the target computing device It is imperceptible to users. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits of reasoning services remain unchanged, and the reasoning is realized at low cost and the user experience is improved. Embodiment 5 Referring to FIG. 6, it shows a flowchart of an inference method according to Embodiment 5 of the present invention. The reasoning method of this embodiment describes the reasoning method of the present invention from the perspective of the second computing device. The reasoning method of this embodiment includes the following steps: Step S502: Obtain the model information of the calculation model used for reasoning sent by the source computing device, and load the calculation model indicated by the model information through the reasoning acceleration resource. In this embodiment, the source computing device may be the first computing device in the foregoing embodiment, and the model information includes, but is not limited to, identification information and/or verification information. Step S504: Obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model. After the inference acceleration resource is loaded into the calculation model, when the data to be inferred sent from the source computing device is received, the calculation model loaded by the inference acceleration resource can be used to perform inference processing on it. Step S506: Feed back the result of the inference processing to the source computing device. In specific implementation, the inference method of this embodiment can be implemented by the inference server of the second computing device in the foregoing embodiment, and the specific implementation of the foregoing process can also refer to the operation of the inference server in the foregoing embodiment, which will not be repeated here. Through this embodiment, inference processing is deployed in different computing devices. Among them, the current computing device that executes the inference method of this embodiment is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model, and the source computing device It can be responsible for data processing before and after the inference processing. During inference, the source computing device can first send the model information of the computing model to the current computing device, and the current computing device uses inference acceleration resources to load the corresponding computing model; then, the source computing device sends the data to be inferred to the current computing device. After the current computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to the remote device with inference acceleration resources, and the interaction between the source computing device and the current computing device is important for the user. The user is imperceptible, therefore, it can ensure that the business logic of the application involving reasoning and the user's usage habits of reasoning business remain unchanged, and the reasoning is realized at a low cost and the user experience is improved. Embodiment 6 The reasoning method of this embodiment is still from the perspective of the second computing device to illustrate the reasoning method of the present invention. Referring to FIG. 7, there is shown a flow chart of an inference method according to the sixth embodiment of the present invention. The inference method includes: Step S602: According to the model information of the calculation model, it is determined that the calculation model does not exist locally, and then the source The computing device requests the computing model, and after obtaining the computing model from the source computing device, loads the computing model through the inference acceleration resource. In this embodiment, the model information of the calculation model can be the identification information or verification information of the calculation model; then, this step can be implemented as follows: according to the identification information or the verification information, it is determined that the local Computing model, request the computing model from the source computing device, and after obtaining the computing model from the source computing device, load the computing model through the inference acceleration resource. The calculation model sent by the source computing device includes, but is not limited to, the structure of the calculation model and its corresponding data. In addition, in an optional manner, the reasoning acceleration resource includes one or more types; when the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities; then, the reasoning The calculation model indicated by the acceleration resource loading the model information includes: according to a preset load balancing rule and the priority of multiple types of the reasoning acceleration resource, using the reasoning acceleration resource to load the calculation model indicated by the model information. Wherein, the load balancing rule and the priority can be appropriately set by those skilled in the art according to actual needs. Step S604: Obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model. In a feasible way, this step can be implemented as: acquiring the data to be inferred sent by the source computing device and the information of the processing function in the computing model to be called, and calling the processing in the computing model loaded The processing function indicated by the information of the function performs inference processing on the data to be inferred. Wherein, the information of the processing function can be obtained by the source computing device by analyzing the reasoning request. Optionally, the information of the processing function is API interface information of the processing function. Step S606: Feed back the result of the inference processing to the source computing device. In specific implementation, the inference method of this embodiment can be implemented by the inference server of the second computing device in the foregoing embodiment, and the specific implementation of the foregoing process can also refer to the operation of the inference server in the foregoing embodiment, which will not be repeated here. Through this embodiment, inference processing is deployed in different computing devices. Among them, the current computing device that executes the inference method of this embodiment is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model, and the source computing device It can be responsible for data processing before and after the inference processing. During inference, the source computing device can first send the model information of the computing model to the current computing device, and the current computing device uses inference acceleration resources to load the corresponding computing model; then, the source computing device sends the data to be inferred to the current computing device. After the current computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to the remote device with inference acceleration resources, and the interaction between the source computing device and the current computing device is important for the user. The user is imperceptible, therefore, it can ensure that the business logic of the application involving reasoning and the user's usage habits of reasoning business remain unchanged, and the reasoning is realized at a low cost and the user experience is improved. Embodiment 7 Referring to FIG. 8, there is shown a schematic structural diagram of an electronic device according to Embodiment 7 of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device. As shown in FIG. 8, the electronic device may include a processor 702, a communications interface 704, a memory 706, and a communication bus 708. Among them: the processor 702, the communication interface 704, and the memory 706 communicate with each other through the communication bus 708. The communication interface 704 is used to communicate with other electronic devices or servers. The processor 702 is configured to execute the program 710, and specifically can execute the relevant steps in the inference method embodiment in the third or fourth embodiment. Specifically, the program 710 may include a program code, and the program code includes a computer operation instruction. The processor 702 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs. The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, or may also include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. The program 710 may specifically be used to cause the processor 702 to perform the following operations: obtain model information of a calculation model for inference, and send the model information to a target computing device to instruct the target computing device to use the target computing device The set inference acceleration resources are loaded into the calculation model indicated by the model information; the data to be inferred is obtained, and the data to be inferred is sent to the target computing device to instruct the target computing device to use the inference acceleration resource to call loading The calculation model is used to perform inference processing on the data to be inferred through the calculation model; and the result of the inference processing returned by the target computing device is received. In an optional implementation manner, the program 710 is further configured to cause the processor 702 to send the calculation model to the target computing device if it is determined that the calculation model does not exist in the target computing device. In an optional implementation manner, the model information of the calculation model is identification information or verification information of the calculation model; the program 710 is also used to enable the processor 702 to determine if the target computing device is not When the calculation model exists, before sending the calculation model to the target computing device, it is determined whether the calculation model exists in the target computing device through the identification information or the verification information. In an optional implementation manner, the program 710 is also used to enable the processor 702 to obtain data to be inferred and send the data to be inferred to the target computing device: obtain a request for the calculation model to respond to the to-be inferred data. The reasoning data is used for the reasoning request for reasoning processing, and the semantic analysis is performed on the reasoning request; the processing function in the calculation model to be called is determined according to the result of the semantic analysis, and the information of the processing function and the to-be reasoning are determined The data is sent to the target computing device to instruct the target computing device to perform inference processing on the data to be inferred by calling the processing function indicated by the information of the processing function in the loaded calculation model. In an optional implementation manner, the information of the processing function is API interface information of the processing function. For the specific implementation of each step in the formula 710, reference may be made to the corresponding description in the corresponding steps and units in the above-mentioned corresponding inference method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working process of the devices and modules described above can be referred to the corresponding process description in the foregoing method embodiment, which will not be repeated here. Through the electronic device of this embodiment, the inference processing is deployed in different computing devices. Among them, the target computing device is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model, and the inference method of this embodiment can be executed. Current electronic equipment can be responsible for data processing before and after inference processing. When performing inference, the current electronic device can first send the model information of the calculation model to the target computing device, and the target computing device uses the inference acceleration resource to load the corresponding calculation model; then, the current electronic device sends the data to be inferred to the target computing device. After the target computing device receives the data to be inferred, it can perform inference processing through the loaded computing model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using applications involving inference, the inference calculation can be seamlessly transferred to the remote target computing device with inference acceleration resources, and the current interaction between the electronic device and the target computing device It is imperceptible to users. Therefore, it can ensure that the business logic of the application involving reasoning and the user's use habits of reasoning services remain unchanged, and the reasoning is realized at low cost and the user experience is improved. Embodiment 8 Referring to FIG. 9, there is shown a schematic structural diagram of an electronic device according to Embodiment 8 of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the electronic device. As shown in FIG. 9, the electronic device may include: a processor 802, a communications interface (Communications Interface) 804, a memory (memory) 806, and a communication bus 808. Among them, the processor 802, the communication interface 804, and the memory 806 communicate with each other through the communication bus 808. The communication interface 804 is used to communicate with other electronic devices or servers. The processor 802 is configured to execute the program 810, and specifically can execute the relevant steps in the inference method embodiment in the fifth or sixth embodiment. Specifically, the program 810 may include a program code, and the program code includes a computer operation instruction. The processor 802 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs. The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. The program 810 can specifically be used to make the processor 802 perform the following operations: obtain model information of a calculation model used for inference sent by the source computing device, load the calculation model indicated by the model information through the inference acceleration resource; obtain the source calculation The data to be inferred sent by the device uses the calculation model loaded by the inference acceleration resource call to perform inference processing on the data to be inferred through the calculation model; and the result of the inference processing is fed back to the source computing device. In an optional implementation, the model information of the calculation model is identification information or verification information of the calculation model; the program 810 is also used to enable the processor 802 to obtain the calculation for reasoning sent by the source computing device. When the model information of the model is loaded into the calculation model indicated by the model information through the inference acceleration resource: According to the identification information or the verification information, it is determined that the calculation model does not exist locally, and then a request is made to the source computing device The calculation model, after obtaining the calculation model from the source calculation device, loads the calculation model through the inference acceleration resource. In an optional implementation manner, the program 810 is also used to enable the processor 802 to obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and use the calculation model to When the data to be inferred performs inference processing: the data to be inferred sent by the source computing device and the information of the processing function in the calculation model to be called are obtained, and the processing function in the calculation model loaded by calling The processing function indicated by the information to perform inference processing on the data to be inferred. In an optional implementation manner, the information of the processing function is API interface information of the processing function. In an optional implementation manner, the reasoning acceleration resource includes one or more types; when the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities; the program 810 is also used to make When the processor 802 loads the calculation model indicated by the model information through the inference acceleration resource: according to a preset load balancing rule and the priority of multiple types of the inference acceleration resource, it uses the inference acceleration resource to load the model information The indicated calculation model. For the specific implementation of each step in the program 810, refer to the corresponding description in the corresponding steps and units in the above-mentioned corresponding inference method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working process of the devices and modules described above can be referred to the corresponding process description in the foregoing method embodiment, which will not be repeated here. Through the electronic device of this embodiment, inference processing is deployed in different computing devices. Among them, the current electronic device that executes the inference method of this embodiment is equipped with inference acceleration resources, and the main inference processing can be performed through the computing model. The source computing device can then be responsible for data processing before and after inference processing. When performing inference, the source computing device can first send the model information of the calculation model to the current electronic device, and the current electronic device uses the inference acceleration resource to load the corresponding calculation model; then, the source computing device sends the data to be inferred to the current electronic device. After the current electronic device receives the data to be inferred, it can perform inference processing through the loaded calculation model. As a result, the decoupling of computing resources used in inference is realized. Inference processing through the computing model and data processing other than inference processing can be achieved through different computing devices, one of which is equipped with inference acceleration resources such as GPU. , There is no need for an electronic device to have both CPU and GPU, which effectively solves the problem of the fixed CPU/GPU specifications of existing heterogeneous computing machines, which restricts the deployment of applications involving inference, which makes it impossible to meet the needs of a wide range of inference scenarios. problem. In addition, for users, when using an application involving inference, the inference calculation can be seamlessly transferred to the remote device with inference acceleration resources, and the interaction between the source computing device and the current electronic device is very important for the user. The user is imperceptible, therefore, it can ensure that the business logic of the application involving reasoning and the user's usage habits of reasoning business remain unchanged, and the reasoning is realized at a low cost and the user experience is improved. It should be pointed out that according to the needs of implementation, each component/step described in the embodiment of the present invention can be split into more components/steps, or two or more components/steps or partial operations of components/steps can be combined into New components/steps to achieve the purpose of the embodiments of the present invention. The above method according to the embodiments of the present invention can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), Or the computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium that is downloaded through the Internet and stored in a local recording medium, so that the method described here can be stored on a general-purpose computer, Such software processing on a dedicated processor or a recording medium of programmable or dedicated hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code (for example, RAM, ROM, flash memory, etc.), when the software or computer code is When accessed and executed by a computer, processor, or hardware, the reasoning method described here is implemented. In addition, when a general-purpose computer accesses the code used to implement the inference method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the inference method shown here. A person of ordinary skill in the art may be aware that the units and method steps of the examples described in the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the embodiments of the present invention. The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those of ordinary skill in the relevant technical field can make various changes without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be limited by the scope of the patent application.

100: electronic equipment 102: CPU 104: GPU 106: PCIE slot 108: Motherboard 202: The first computing device 204: second computing device 2022: Inference Client 2042: Inference server 2044: Reasoning Acceleration Resources S302, S304, S306: steps S402, S404, S406, S408: steps S502, S504, S506: steps S602, S604, S606: steps 702: processor 704: Communication Interface 706: Memory 708: Communication Bus 710: Program 802: processor 804: Communication Interface 806: memory 808: Communication Bus 810: program

In order to more clearly describe the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are merely the present For some of the embodiments recorded in the embodiments of the invention, for those of ordinary skill in the art, other drawings may be obtained based on these drawings. [Figure 1] is a schematic diagram of the structure of an electronic device with inference computing resources in the prior art; [Figure 2a] is a structural block diagram of an inference system according to the first embodiment of the present invention; [Figure 2b] is a schematic structural diagram of an example of an inference system according to an embodiment of the present invention; [Figure 3a] is a block diagram of the structure of an inference system according to the second embodiment of the present invention; [Figure 3b] is a schematic structural diagram of an example of an inference system according to an embodiment of the present invention; [Figure 3c] is a schematic diagram of the inference process using the inference system shown in Figure 3b; [Figure 3d] is an interactive schematic diagram of inference using the inference system shown in Figure 3b; [Figure 4] is a flow chart of an inference method according to the third embodiment of the present invention; [Figure 5] is a flowchart of a reasoning method according to the fourth embodiment of the present invention; [Figure 6] is a flow chart of an inference method according to the fifth embodiment of the present invention; [Fig. 7] is a flowchart of an inference method according to the sixth embodiment of the present invention; [Figure 8] is a schematic structural diagram of an electronic device according to the seventh embodiment of the present invention; [Fig. 9] is a schematic diagram of the structure of an electronic device according to the eighth embodiment of the present invention.

202: The first computing device

204: second computing device

2022: Inference Client

2042: Inference server

2044: Reasoning Acceleration Resources

Claims (20)

An inference system, characterized by comprising a first computing device and a second computing device connected to each other, wherein the first computing device is provided with an inference client, the second computing device is provided with an inference acceleration resource and Inference server among them: The reasoning client is used to obtain model information and data to be inferred of a calculation model for inference, and respectively send the model information and the data to be inferred to the inference server in the second computing device; The reasoning server is used to load and call the calculation model indicated by the model information through the reasoning acceleration resource, perform reasoning processing on the data to be reasoned through the calculation model, and return the reasoning processing to the reasoning client the result of. The inference system according to claim 1, wherein the inference client is further configured to send the calculation model to the inference server when it is determined that the calculation model does not exist in the second computing device. The reasoning system according to claim 2, wherein the model information of the calculation model is identification information or verification information of the calculation model; The inference server is also used to determine whether the calculation model exists in the second computing device through the identification information or the verification information, and return the determination result to the inference client. The reasoning system according to claim 1, wherein: The reasoning client is also used to obtain a reasoning request requesting the calculation model to perform reasoning processing on the data to be reasoned, and perform semantic analysis on the reasoning request, and determine the calculation model to be called according to the semantic analysis result The processing function of, sending the information of the processing function to the inference server; When the inference server performs inference processing on the data to be inferred through the calculation model, by calling the processing function indicated by the information of the processing function in the loaded calculation model, the The data to be inferred is processed inferentially. The reasoning system according to claim 4, wherein the information of the processing function is API interface information of the processing function. The reasoning system according to claim 1, wherein the second computing device is provided with one or more types of reasoning acceleration resources; When the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities; The reasoning server uses the reasoning acceleration resource according to the preset load balancing rules and the priority of the multiple types of the reasoning acceleration resource. The inference system according to any one of claims 1 to 6, wherein the first computing device and the second computing device are connected to each other through an elastic network. The reasoning system according to any one of Claims 1 to 6, wherein the reasoning client is a component inside a deep learning framework embedded in the first computing device, or the reasoning client is a component that can be used by Describes the callable files called by the deep learning framework. A reasoning method, characterized in that the method includes: Obtain model information of a calculation model for inference, and send the model information to a target computing device to instruct the target computing device to use the inference acceleration resource set in the target computing device to load the calculation indicated by the model information model; Obtain the data to be inferred, and send the data to be inferred to the target computing device to instruct the target computing device to use the inference acceleration resource to call the loaded calculation model, and use the calculation model to analyze the to-be inferred Data is processed by reasoning; Receiving the result of the inference processing fed back by the target computing device. The method according to claim 9, the method further comprising: If it is determined that the calculation model does not exist in the target computing device, the calculation model is sent to the target computing device. The method according to claim 10, wherein the model information of the calculation model is identification information or verification information of the calculation model; When it is determined that the calculation model does not exist in the target computing device, before sending the calculation model to the target computing device, the method further includes: using the identification information or the verification Information to determine whether the calculation model exists in the target computing device. The method according to claim 9, wherein the obtaining the data to be inferred and sending the data to be inferred to the target computing device includes: Acquiring a reasoning request requesting the calculation model to perform reasoning processing on the to-be reasoned data, and performing semantic analysis on the reasoning request; The processing function in the computing model to be called is determined according to the semantic analysis result, and the information of the processing function and the data to be inferred are sent to the target computing device to instruct the target computing device to load by calling The processing function indicated by the information of the processing function in the input calculation model performs inference processing on the data to be inferred. The method according to claim 12, wherein the information of the processing function is API interface information of the processing function. A reasoning method, characterized in that the method includes: Obtain the model information of the calculation model used for reasoning sent by the source computing device, and load the calculation model indicated by the model information through the reasoning acceleration resource; Obtain the data to be inferred sent by the source computing device, use the inference acceleration resource to call the loaded calculation model, and perform inference processing on the data to be inferred through the calculation model; The result of the inference processing is fed back to the source computing device. The method according to claim 14, wherein the model information of the calculation model is identification information or verification information of the calculation model; The acquiring the model information of the calculation model used for reasoning sent by the source computing device and loading the calculation model indicated by the model information through the reasoning acceleration resource includes: According to the identification information or the verification information, it is determined that the calculation model does not exist locally, the calculation model is requested from the source computing device, and after the calculation model is obtained from the source computing device, The reasoning acceleration resource is loaded into the calculation model. The method according to claim 14, wherein the acquiring the data to be inferred sent by the source computing device uses the calculation model loaded by the inference acceleration resource call, and performs the calculation on the data to be inferred through the calculation model. Inference processing, including: Obtain the data to be inferred sent by the source computing device and the information of the processing function in the calculation model to be called. By calling the processing function indicated by the information of the processing function in the loaded calculation model, Describe the data to be inferred for inference processing. The method according to claim 16, wherein the information of the processing function is API interface information of the processing function. The method according to claim 14, wherein the reasoning acceleration resource includes one or more types; When the reasoning acceleration resource includes multiple types, different types of reasoning acceleration resources have different usage priorities; The calculation model for loading the model information instruction through the inference acceleration resource includes: according to a preset load balancing rule and the priority of multiple types of the inference acceleration resource, using the inference acceleration resource to load the model information instruction Calculation model. An electronic device, comprising: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, the executable instruction causes the processor to perform an operation corresponding to the reasoning method described in any one of request items 9 to 13, or the executable instruction The processor is caused to perform an operation corresponding to the inference method described in any one of request items 14 to 18. A computer storage medium on which a computer program is stored, and when the program is executed by a processor, the reasoning method as described in any one of claims 9 to 13 is realized; or, as described in any one of claims 14 to 18 The reasoning method described.

Images