WO2020243973A1

WO2020243973A1 - Model-based signal inference method and apparatus

Info

Publication number: WO2020243973A1
Application number: PCT/CN2019/090442
Authority: WO
Inventors: 张珺
Original assignee: 华为技术有限公司
Priority date: 2019-06-06
Filing date: 2019-06-06
Publication date: 2020-12-10
Also published as: CN113966516A

Abstract

Embodiments of the present application provide a model-based signal inference method and apparatus. The model-based signal inference method comprises receiving an input signal, and according to dependency information in a preset model, performing inference of the preset model on the input signal to obtain an output signal, wherein the preset model comprises operators, and each operator comprises an input parameter and an output parameter; the dependency information is used for indicating a dependency relationship between the input parameter or output parameter of at least one of the operators in the inference of a target frame and the input parameter or output parameter of at least one of the operators in the inference of a reference frame when the preset model is used to perform multi-frame inference on the input signal; the inference of the target frame is the inference of at least one frame in the inference of multiple frames; the inference of the reference frame is the inference of at least one frame in the inference of other frames except for the inference of the target frame in the inference of the multiple frames. The inference of the multiple frames having the dependency relationship can be concurrently performed according to the dependency information in the preset model, thereby improving multi-frame inference performance.

Description

Model-based signal reasoning method and device

Technical field

The embodiments of the present application relate to the field of data processing technology, and in particular, to a model-based signal reasoning method and device.

Background technique

With the increasing application of deep learning, various training and reasoning frameworks have emerged. When a model is trained and deployed to a production environment, inference performance is a very important indicator. If the computing power can be fully utilized without changing the hardware configuration, it will greatly enhance the competitiveness of the product.

At present, after the model is trained, it can provide input data and related attribute control for the model through the interface provided by the deep learning framework, perform reasoning, and get the reasoning result. In the inference process, different data can be input for repeated execution. For example, for an image detection model, different frames of the video can be input for continuous inference. If computing resources are abundant, parallel execution can be used to improve execution efficiency.

However, when there is data dependence between inferences, for example, when the input of the n+mth frame inference depends on the output of the nth frame inference, you must wait for the end of the nth frame inference before executing the n+mth frame Reasoning, but cannot improve execution efficiency through parallel execution. In this way, when the computing resources are relatively abundant, the computing power of the computing resources cannot be fully utilized, which reduces the inference performance.

Summary of the invention

The embodiment of the present application provides a model-based signal reasoning method and device, which improves the reasoning performance when there is data dependency between reasoning.

In the first aspect, an embodiment of the present application provides a model-based signal reasoning method, including:

Receive input signals, which are digital signals that can be processed by a computer or processor. According to the dependency information in the preset model, the input signal is inferred from the preset model to obtain the output signal. Wherein, the preset model includes an operator, and the operator includes input parameters and output parameters; the dependency information is used to indicate that when the preset model is used to perform multi-frame inference on the input signal, at least one of the operators in the target frame inference Dependence of the input parameter or output parameter of at least one of the operator in the reference frame reasoning with the input parameter or output parameter of the operator, the target frame reasoning is at least one frame of the multi-frame reasoning, and the reference frame reasoning is the multi-frame At least one frame of reasoning in other frame of reasoning except the target frame of reasoning.

Through the model-based signal reasoning method provided by the first aspect, the dependency information in the preset model can indicate that when the model is used for multi-frame reasoning, the input parameters or output parameters of the operator in the target frame reasoning and the reference frame reasoning The dependency between input parameters or output parameters. According to the dependency information in the preset model, multi-frame reasoning with dependencies can be executed in parallel at the same time, making full use of the computing power of computing resources, and improving the reasoning performance of multi-frame reasoning.

Optionally, in a possible implementation of the first aspect, before performing multi-frame inference of the preset model on the input signal according to the dependency information in the preset model, it may further include: description information of the preset model Dependency information is set in. The description information is used to describe at least one operator included in the preset model and the input parameter and output parameter of each operator in the at least one operator.

Optionally, in a possible implementation of the first aspect, setting the dependency information in the description information of the preset model includes: setting the dependency information in the description information of the preset model by calling the interface of the preset model, The interface of the preset model is used to provide an entrance for information modification of the preset model.

Optionally, in a possible implementation manner of the first aspect, setting dependency information in the description information of the preset model includes: receiving dependency information input by the user, and describing the dependency information in the preset model according to the dependency information input by the user. Set dependent information in the information.

Optionally, in a possible implementation manner of the first aspect, before setting the dependency information in the description information of the preset model, the method further includes: acquiring the description information. Obtain dependency information according to at least one operator included in the description information and the input parameters and output parameters of each operator.

Optionally, in a possible implementation manner of the first aspect, performing the inference of the preset model on the input signal according to the dependency information in the preset model includes: loading the preset model and the dependency information. Schedule each operator of the preset model in the current frame inference according to the dependency information.

Optionally, in a possible implementation of the first aspect, scheduling each operator of the preset model in the current frame inference according to the dependency information includes: determining each operator of the preset model in the current frame inference according to the dependency information The order of execution. Each operator of the preset model in the current frame inference is added to the multiple execution queues according to the execution order; wherein, there is no dependency between the input parameters of the multiple operators that are executed at the same time in the multiple execution queues.

In the second aspect, an embodiment of the present application provides a model-based signal inference device, including a processor and a transmission interface. Among them, the transmission interface is used to receive input signals, and the input signals are digital signals that can be processed by a computer or a processor. The processor is used to perform the inference of the preset model on the input signal according to the dependency information in the preset model to obtain the output signal. Wherein, the preset model includes an operator, and the operator includes input parameters and output parameters; the dependency information is used to indicate that when the preset model is used to perform multi-frame inference on the input signal, at least one of the operators in the target frame inference Dependence of the input parameter or output parameter of at least one of the operator in the reference frame reasoning with the input parameter or output parameter of the operator, the target frame reasoning is at least one frame of the multi-frame reasoning, and the reference frame reasoning is the multi-frame At least one frame of reasoning in other frame of reasoning except the target frame of reasoning.

Optionally, in a possible implementation manner of the second aspect, the processor is further configured to: set dependency information in the description information of the preset model; wherein the description information is used to describe at least one calculation included in the preset model. The input parameters and output parameters of each operator in at least one operator.

Optionally, in a possible implementation of the second aspect, the processor is specifically configured to: set dependency information in the description information of the preset model by calling the interface of the preset model, and the interface of the preset model is used for The preset model provides an entrance for information modification.

Optionally, in a possible implementation manner of the second aspect, the transmission interface is further configured to receive dependency information input by the user, and the processor is specifically configured to set dependency information in the description information of the preset model according to the dependency information input by the user. information.

Optionally, in a possible implementation manner of the second aspect, the processor is further configured to: obtain description information. Obtain dependency information according to at least one operator included in the description information and the input and output parameters of each operator.

Optionally, in a possible implementation of the second aspect, the processor is specifically configured to: load the preset model and dependent information, and schedule each operator of the preset model in the current frame inference according to the dependent information.

Optionally, in a possible implementation manner of the second aspect, the processor is specifically configured to determine the execution sequence of each operator of the preset model in the current frame inference according to the dependency information. Each operator of the preset model in the current frame inference is added to the multiple execution queues according to the execution order; wherein, there is no dependency between the input parameters of the multiple operators that are executed at the same time in the multiple execution queues.

In a third aspect, an embodiment of the present application provides a model-based signal inference device, including: a unit for executing each step of the model-based signal inference method provided in the first aspect.

In a fourth aspect, an embodiment of the present application provides a model-based signal inference device, including a processor, configured to connect to a memory and call a program stored in the memory to execute the model-based signal inference method provided in the first aspect above . The memory can be located inside the device or outside the device. And the processor includes one or more.

In the fifth aspect, embodiments of the present application provide a computer program product containing instructions, which when the instructions run on a computer or processor, cause the computer or processor to execute the model-based signal inference method provided in the first aspect above.

In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium with instructions stored in the computer-readable storage medium. When the instructions run on a computer or a processor, the model-based Signal reasoning method.

In each of the above aspects, optionally, in a possible implementation manner, the dependency information includes the dependency relationship between the input parameter of the operator in the target frame reasoning and the output parameter of the operator in the reference frame reasoning.

In each of the above aspects, optionally, in a possible implementation manner, the dependency information includes a first inter-frame interval, a first operator identifier, and a first parameter identifier, and the first inter-frame interval is used to indicate a connection with the target frame In inference, the first input parameter of the first operator is dependent on the frame interval between the reference frame inference and the target frame inference. The first operator identifier is used to indicate the operator in the reference frame inference that is dependent on the first operator , The first parameter identifier is used to indicate a parameter that is dependent on the first input parameter in the reference frame reasoning.

In the above aspects, optionally, in a possible implementation manner, the dependency information includes the dependency relationship between the output parameter of the operator in the target frame reasoning and the input parameter of the operator in the reference frame reasoning.

In each of the above aspects, optionally, in a possible implementation manner, the dependency information includes a second inter-frame interval, a second operator identifier, and a second parameter identifier, and the second inter-frame interval is used to indicate a connection with the target frame In inference, the interval between reference frame inference and target frame inference for which the first output parameter of the second operator has a dependency relationship, and the second operator identifier is used to indicate the operator in reference frame inference that has a dependency relationship with the second operator , The second parameter identifier is used to indicate a parameter that is dependent on the first output parameter in the reference frame inference.

In each of the above aspects, optionally, in a possible implementation manner, the processable digital signal includes at least one of the following: image data, voice data, and text data.

Description of the drawings

Figure 1 is a schematic diagram of an application scenario to which an embodiment of the application is applicable;

Figure 2 is a schematic structural diagram of a model provided by an embodiment of the application;

FIG. 3 is a flowchart of deploying a deep learning model provided by an embodiment of the application;

FIG. 4 is a diagram of the neural network application architecture provided by an embodiment of the application;

5 is a schematic diagram of the structure of model multi-frame reasoning provided by an embodiment of the application;

FIG. 6 is a schematic diagram of another structure of a model provided by an embodiment of the application;

FIG. 7 is a schematic diagram of description information of a model provided by an embodiment of the application;

FIG. 8 is a flowchart of a model-based signal reasoning method provided by an embodiment of the application;

FIG. 9 is a flowchart of another model-based signal reasoning method provided by an embodiment of the application;

10 is a schematic diagram of an execution queue in multi-frame reasoning provided by an embodiment of the application;

FIG. 11 is a schematic diagram of description information and dependency information of a model provided by an embodiment of the application;

FIG. 12 is another schematic diagram of description information and dependency information of a model provided by an embodiment of this application;

FIG. 13 is a schematic structural diagram of a model-based signal reasoning device provided by an embodiment of the application;

FIG. 14 is a schematic diagram of another structure of a model-based signal reasoning device provided by an embodiment of the application;

FIG. 15 is a schematic diagram of another structure of the model-based signal reasoning device provided by an embodiment of the application.

Detailed ways

The embodiments of the present application are described below in conjunction with the drawings.

The model-based signal reasoning method provided in the embodiment of the present application can be applied to a model-based signal reasoning device. When the model-based signal reasoning device uses the preset model to perform continuous multi-frame reasoning of the preset model, the model-based signal reasoning method provided in the embodiment of the present application can be executed. The embodiment of the present application does not limit the specific implementation of the model-based signal inference device, as long as the device has computing capability and can use the model to perform inference. Exemplarily, FIG. 1 is a schematic diagram of an application scenario to which an embodiment of the application is applicable. As shown in FIG. 1, the model-based signal inference device may include a terminal device 11, a server 12 and a computer 13. Optionally, the terminal device 11, the server 12, and the computer 13 may be connected to the network to obtain data or publish data through the network.

Optionally, the terminal device 11 may also be referred to as user equipment (UE), mobile station (mobile station, MS), or mobile terminal (mobile terminal, MT), etc., which is a way to provide users with voice/data connectivity device of. For example, handheld devices with wireless connectivity, or vehicle-mounted devices. At present, some examples of terminal devices can be: mobile phones (mobile phones), tablet computers, notebook computers, palmtop computers, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, Augmented reality (AR) equipment, wireless terminals in industrial control, wireless terminals in unmanned driving (self-driving), wireless terminals in remote medical surgery, and smart grid (smart grid) ), the wireless terminal in transportation safety, the wireless terminal in the smart city, or the wireless terminal in the smart home.

It should be noted that FIG. 1 is only an example, and does not limit the implementation of the model-based signal inference device.

Below, the concepts involved in the embodiments of the present application will be described.

(1) Model

In the fields of artificial intelligence, machine learning, deep learning, neural networks, etc., models can be used to process data and achieve specific functions. Using the model, the input X can be predicted and analyzed, and the result Y can be output. According to different actual needs, the model can realize different functions, and the specific forms of X and Y can also be different. The model can be reused. Model reuse can be achieved through parameter settings. For example, for different objects in the same scene, model reuse can be achieved by assigning different parameter values.

The model is explained below with examples.

For example, in the field of image processing, an autonomous vehicle can take images of the road ahead of the vehicle. Inputting the image into the preset model can output obstacles in the road to realize obstacle detection. The obstacle can be other vehicles, bicycles, or pedestrians in front of the vehicle.

For another example, in the field of speech processing, the speaker's speech signal will be disturbed by the noise of the surrounding environment. The speaker's speech signal and the signal of the environment noise are mixed into the preset model, the speaker's speech signal can be output, the environment noise can be filtered out, and the signal noise can be reduced.

(2) Operators in the model, their input parameters, and their output parameters

In the model, the realization of the algorithm is not the overall realization. Usually, the algorithm is split, and a model is built according to the execution logic of the algorithm through multiple operators with a single granularity. Among them, each operator has input parameters and output parameters. For an operator, the input parameters of the operator may be the input of the model, or the output parameters of other operators. The output parameter of this operator may be the output of the model or the input parameters of other operators.

It should be noted that the embodiment of the present application does not limit the number of operators included in the model, the number of input parameters and the number of output parameters of each operator.

The following is a description with examples.

Figure 2 is a schematic structural diagram of a model provided by an embodiment of the application. As shown in Figure 2, the model includes 4 operators, namely operators A to D.

Operator A has 1 input parameter x and 1 output parameter a. The input parameter of operator A is also the input x of the model.

Operator B has 1 input parameter a and 1 output parameter b. The input parameter of operator B is the output parameter a of operator A. The output parameter of operator B is the input parameter b of operator D.

Operator C has 1 input parameter a and 1 output parameter c. The input parameter of operator C is the output parameter a of operator A. The output parameter of operator C is the input parameter c of operator D.

Operator D has 2 input parameters b and c, and 1 output parameter y. The input parameter b of operator D is the output parameter b of operator B, and the input parameter c of operator D is the output parameter c of operator C. The output parameter of operator D is the output y of the model.

(3) Model training, inference and deployment

The model needs to be trained before it can be inferred and deployed.

Model training refers to the constant optimization of its own parameters for an initial model, so that the model can achieve corresponding functions. There are many training methods for the model, which are not limited in the embodiment of the present application. For example, the model can be trained through a training data set. Exemplarily, the training data set includes labeled data. For example, when training a model for obstacle detection, the training data set may include multiple images, each of which has been pre-marked with obstacles.

Model reasoning refers to the process of inputting the data to be processed into the trained model to obtain the output result. For example, a model that realizes obstacle detection is also taken as an example. Take an image and input the image into the model to output obstacle information.

Model deployment refers to the process of applying the trained model on the hardware platform to run the model.

The following uses a deep learning model as an example to illustrate the deployment of the model. Fig. 3 is a flowchart of deploying a deep learning model provided by an embodiment of the application. As shown in Figure 3, the deployment model can include:

S301. Load a graph (load graph).

Through the interface provided by the deep learning framework, the preset model can be loaded into the device memory.

S302. Perform model inference (forward).

The interface provided by the deep learning framework provides input data and related attribute control for the loaded preset model, executes the inference (forward), and obtains the inference result. In the inference process of the model, operators are encapsulated in the model, and users generally only care about the input and output of the model, not the input and output of each operator.

Exemplarily, FIG. 4 is a neural network application architecture diagram provided by an embodiment of the application. As shown in Figure 4, the neural network application architecture provided by the embodiment of the present application may include: application program entry 41, model external interface 42, deep learning structure 43, device driver 44, central processing unit 45, graphics processor 46, network The processor 47 and the digital processor 48. Among them, the application program entry 41 is used to select a neural network model. The model external interface 42 is used to call the selected neural network model. The deep learning structure 43 is used to process the input first user image through the neural network model. Exemplarily, the deep learning structure 43 may include an environment manager 431, a model manager 432, a task scheduler 433, a task performer 434, and an event manager 435. Among them, the environment manager 431 is used to control the startup and shutdown of the device-related environment. The model manager 432 is responsible for operations such as loading the neural network model and unloading the neural network model. The task scheduler 433 is used to manage the sequence in which the neural network model is scheduled. The task executor 434 is responsible for executing the task of the neural network model. The event manager 435 is responsible for the notification of various events.

It should be noted that FIG. 4 is only an example, and the embodiment of the present application does not limit the neural network application architecture.

(4) Reasoning per frame, multi-frame reasoning

In the reasoning of the model, different data can be input to repeatedly perform the reasoning. For example, for an image detection model, different frames of the video can be input to continuously perform inference. Among them, for a certain frame in the video, the model performs a frame inference. For multiple frames of the video, the model performs multi-frame inference. In the embodiments of the present application, multi-frame reasoning is also referred to as multiple reasoning, continuous reasoning, multiple frame continuous reasoning, continuous multiple frame reasoning, and continuous multiple reasoning. Every frame of reasoning is also called every frame of reasoning and frame reasoning.

When the model executes multi-frame inference, it can execute inference either serially or in parallel, depending on the type of model interface and the availability of computing resources.

The description will be given below in conjunction with FIG. 5. Fig. 5 is a schematic diagram of the structure of a model multi-frame reasoning provided by an embodiment of the application. It is assumed that the input of each frame of inference of the model does not depend on the output of other frames of inference. As shown in Figure 5, when multi-frame inference is executed serially, the first frame inference is executed at T0, the second frame is executed at T1, and the third frame is executed at T2. When multi-frame inference is executed in parallel, since the input of each frame of inference of the model does not depend on the output of other frame inferences, it is possible to execute 3 frames of inference at the same time at T0. It can be seen that if the computing resources are abundant and the input data of each frame of inference is not dependent, the utilization of resources can be improved and the inference speed can be improved by executing multi-frame inference in parallel.

The following describes the case where the input of each frame of reasoning of the model depends on the output of other frames of reasoning. FIG. 6 is a schematic diagram of another structure of a model provided by an embodiment of the application. As shown in Figure 6, the model includes operators A to D. Among them, the operator A and the operator D are similar to the operator A and the operator D in the model shown in FIG. 2, and will not be repeated here. Operator B has two input parameters. An input parameter of operator B in the second frame of reasoning is the output parameter c of operator C in the first frame of reasoning. Similarly, an input parameter of operator B in the third frame of reasoning is the operator C in the second frame of reasoning. The output parameter c, and so on. The input of each frame of inference depends on the output of the previous frame of inference. At this time, if the parallel execution mode is used for multi-frame inference, the dependency between different frame inferences needs to be considered.

(5) Descriptive information of the model

The description information of the model is used to describe at least one operator included in the model and the input parameters and output parameters of each operator. The embodiment of the present application does not limit the storage method of the description information in the model and the specific content included. For example, the description information of the model can be stored in a file, and the file can be a binary file, a text file, and so on.

The following describes the description information with examples.

FIG. 7 is a schematic diagram of description information of a model provided by an embodiment of the application. The description information shown in FIG. 7 is the description information of the model shown in FIG. 6. As shown in Figure 6 and Figure 7, the model has two input parameters, x and y. According to the description information, the model has 4 operators, which are operators A to D. Among them, the operator A has 1 input parameter x and 1 output parameter a. Operator B has 2 input parameters a and y, and 1 output parameter b. Operator C has 1 input parameter a and 1 output parameter c. Operator D has 2 input parameters b and c, and 1 output parameter d. Among them, the output parameter c of the operator C and the output parameter d of the operator D are marked as true, which is represented as the output of the model. The descriptive information shown in Fig. 7 still cannot reflect the dependence relationship between the multi-frame reasoning of the model.

It should be noted that FIG. 7 is only an example, and does not limit the implementation of the description information.

(6) Model dependency information

The dependency information of the model is used to indicate that the input parameter or output parameter of at least one of the operators in the target frame reasoning and at least one of the operators in the reference frame reasoning is used for multi-frame reasoning using the preset model. The dependency relationship of the input parameters or output parameters of the child, the target frame reasoning is at least one frame reasoning in the multi-frame reasoning, and the reference frame reasoning is at least one frame reasoning in the multi-frame reasoning except for the target frame reasoning. . Detailed descriptions will be given later through various embodiments of the present application.

Currently, multi-frame reasoning is performed in a parallel execution mode, and the constraint condition is that the input of each frame of reasoning does not depend on the output of other frames of reasoning, such as the scene shown in Figure 2 or Figure 5. However, when there is data dependence between multi-frame inferences (such as the scenario shown in Figure 6), the input of the n+mth frame of inference depends on the output of the nth frame of inference, you must wait for the end of the nth frame of inference to execute the first n+m frame reasoning. Even if the input on the n+mth frame inference is only the output of a certain operator in the nth frame of inference, it must wait until all the operators in the nth frame of inference are executed. In this way, when the computing resources are relatively abundant, the computing power of the computing resources is reduced, and the inference performance is reduced.

In view of the foregoing technical problems, the embodiments of the present application provide a model-based signal reasoning method, which can be applied to multi-frame reasoning of the model. For scenarios where the input and output parameters in different frame inferences are dependent, the dependency information of the model can be used to know the dependencies between different operators in different frame inferences, so that parallel execution can be performed in units of operators, making full use of computing resources. Computing power improves inference performance.

The technical solutions of the present application and how the technical solutions of the present application solve the above-mentioned technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

Fig. 8 is a flowchart of a model-based signal reasoning method provided by an embodiment of the application. As shown in Figure 8, the model-based signal reasoning method provided by this embodiment includes:

S801. Receive an input signal, where the input signal is a digital signal that can be processed by a computer or a processor.

Among them, the input signal is a signal to be processed input to a preset model, and is a digital signal that can be processed by a computer or a processor.

It should be noted that the functions implemented by the preset model are different, the application scenarios are different, and the specific implementation forms of the input signal and the processable digital signal and the included content may be different, which is not limited in this embodiment.

Optionally, the input signal may include but is not limited to at least one of the following: a voice signal, a video signal, an image signal, a text signal, a temperature signal, or a pressure signal.

For example, when applied to an image processing scene, an object detection scene, or a face recognition scene, the input signal may be an image signal. The image signal may be a landscape signal taken by a camera, an image signal of a community environment captured by a monitoring device, a facial signal of a face obtained by an access control system, a facial signal of a participant obtained by a conference system, etc.

When used in audio processing scenarios, voice recognition scenarios, and voiceprint recognition scenarios, the input signal may be a voice signal. The voice signal may be a voice signal recorded by a recording device, a voice signal received by a mobile phone or a fixed phone during a call, a voice signal received by a radio from a radio station, or a voice signal of a participant received by the conference system.

When used in language translation and semantic recognition scenarios, the input signal can be a text signal. The text signal may be a TXT text signal, a Word text signal, or a PDF text signal.

Optionally, the processable digital signal may include, but is not limited to, at least one of the following: voice data, image data, video data, text data, temperature data, or pressure data.

S802: Perform the inference of the preset model on the input signal according to the dependency information in the preset model to obtain the output signal. Among them, the preset model includes operators, and the operators include input parameters and output parameters. The dependency information is used to indicate that at least one of the input parameter or output parameter of at least one operator in the target frame reasoning and the operator in the reference frame reasoning when multi-frame reasoning is performed on the input signal using the preset model Dependency of the input parameter or output parameter of the operator, the target frame reasoning is at least one frame of multi-frame reasoning, and the reference frame reasoning is at least one frame of the other frame reasoning except the target frame reasoning. reasoning.

Specifically, when performing continuous multi-frame reasoning of the preset model, different frames have a dependency relationship. The dependency information in the preset model may indicate the dependency relationship. The dependency is specifically the dependency between the input parameter or output parameter of the operator in the target frame inference and the input parameter or output parameter of the operator in the reference frame inference. In this way, based on the dependency information in the preset model, multi-frame reasoning with dependencies can be executed in parallel at the same time, making full use of the computing power of computing resources, and improving the reasoning performance of multi-frame reasoning. Furthermore, in multi-frame reasoning, operators can be used as the execution unit, and operators without dependencies can be executed in parallel at the same time. For operators with dependencies, they can be executed sequentially at different times. Therefore, when performing multi-frame reasoning on a preset model, some operators can be executed in parallel, and some operators can be executed serially, which fully utilizes the computing power of computing resources and improves the inference performance of multi-frame reasoning.

The following uses examples to illustrate the dependency information of the preset model.

In an example, please refer to Figure 6. When the model performs multi-frame reasoning, two consecutive frames of reasoning have a dependency relationship. The dependency information of the preset model may indicate that the input parameter (c) of the operator B in the target frame reasoning has a dependency relationship with the output parameter (c) of the operator C in the reference frame reasoning. Among them, the target frame reasoning can be the next frame reasoning in two consecutive frames of reasoning, and the reference frame reasoning can be the previous frame reasoning of the two consecutive frames of reasoning.

In another example, the dependency information of the preset model may indicate that the input parameters of the operator B in the target frame reasoning have a dependency relationship with the output parameters of the operator C in the reference frame reasoning. Among them, the target frame reasoning can be the Nth frame reasoning and the N+1th frame reasoning, and the reference frame reasoning can be the N-2th frame reasoning. N is an odd number greater than 2. It can be seen that the input parameters of operator B in the third frame of inference and the fourth frame of inference both depend on the output parameters of operator C in the first frame of inference. The input parameters of operator B in the fifth frame of inference and the sixth frame of inference both depend on the output parameters of operator C in the third frame of inference. And so on.

In another example, the dependency information of the preset model may indicate that the input parameters of operator B in the target frame reasoning have a dependency relationship with the output parameters of operator C in the first reference frame reasoning, and the The input parameters are dependent on the output parameters of the operator D in the second reference frame inference. Among them, the target frame reasoning can be the Nth frame reasoning, the first reference frame reasoning can be the N-2th frame reasoning, and the second reference frame reasoning can be the N-1th frame reasoning. N is an integer greater than 2. It can be seen that the input parameters of operator B in the third frame of reasoning depend on the output parameters of operator C in the second frame of reasoning, and the input parameters of operator A in the third frame of reasoning depend on the output of operator D in the first frame of reasoning. parameter. The input parameters of operator B in the fourth frame of reasoning depend on the output parameters of operator C in the third frame of reasoning, and the input parameters of operator A in the fourth frame of reasoning depend on the output parameters of operator D in the second frame of reasoning. And so on.

It should be noted that this embodiment does not limit the storage method of the dependency information in the preset model in the preset model and the specific content included. For example, the dependency information in the preset model can be stored in a file, and the file can be a binary file, a text file, or the like. Optionally, the file and the file corresponding to the description information of the preset model are the same file or different files. For the description information of the preset model, please refer to the description of the aforementioned concept, the principle is similar, and will not be repeated here.

It should be noted that this embodiment does not limit the preset model, and may be any model that performs multi-frame inference.

This embodiment provides a model-based signal reasoning method, including: receiving an input signal, the input signal is a digital signal that can be processed by a computer or a processor, and performing a preset model inference on the input signal according to the dependency information in the preset model, Get the output signal. The model-based signal inference method provided in this embodiment enables multi-frame inference with dependencies to be executed in parallel at the same time based on the dependency information in the preset model, fully utilizes the computing power of computing resources, and improves multi-frame inference Inference performance.

FIG. 9 is a flowchart of another model-based signal reasoning method provided by an embodiment of the application. This embodiment provides an exemplary implementation of S802. As shown in Fig. 9, S802, inferring the input signal with the preset model according to the dependency information in the preset model, may include:

S901. Load the preset model and dependent information.

S902. Schedule each operator of the preset model in the current frame inference according to the dependency information.

Specifically, when performing model inference, load the preset model and dependent information. According to the dependency information, it can be determined whether each operator in the current frame inference has a dependency relationship with the operator in the inference frame before the current frame, so that each operator of the preset model in the current frame inference can be scheduled to execute in parallel. Since the computing power of computing resources can be fully utilized, the reasoning performance of multi-frame reasoning is improved.

Optionally, in S902, scheduling each operator of the preset model in the current frame inference according to the dependency information may include:

Determine the execution order of each operator of the preset model in the current frame inference according to the dependency information.

According to the execution sequence, each operator of the preset model in the current frame inference is added to multiple execution queues. Among them, there is no dependency between the input parameters of the multiple operators executed at the same time in the multiple execution queues.

Specifically, the model-based signal inference device may include multiple processors, and this embodiment does not limit the type and number of processors. For example, the processor may include but is not limited to at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), and a network processing unit (NPU). Each processor can correspond to at least one execution queue. This embodiment does not limit the number of execution queues corresponding to the processor.

In the following, taking the model shown in FIG. 6 as an example, in conjunction with FIG. 10, the multi-frame reasoning of the preset model is exemplified. Assume that the number of execution queues is 4, and the time to execute each operator is the same, which is T. Inference of the model for 3 consecutive frames can create all the operator tasks of the 3 frame inference. Assume that there is no inter-frame dependency in the first frame of inference. Among them, the operators in the first frame of inference are marked as 1A, 1B, 1C, and 1D respectively. The operators in the second frame of inference are marked as 2A, 2B, 2C, and 2D. The operators in the third frame of inference are marked as 3A, 3B, 3C, and 3D. The dependency information in the preset model is used to indicate that the input parameter (c) of the operator B in the target frame reasoning has a dependency relationship with the output parameter (c) of the operator C in the reference frame reasoning. Among them, the target frame reasoning can be the next frame reasoning in two consecutive frames of reasoning, and the reference frame reasoning can be the previous frame reasoning of the two consecutive frames of reasoning.

Specifically, according to the dependency information in the preset model, the execution order of the operators in 3 consecutive frames of inference can be determined. For the second frame of inference, operator 2B depends on the operator 1C in the first frame of inference and the operator 2A in the second frame of inference. Therefore, the execution order of operator 2B needs to be after the first frame operator 1C and the second frame operator 2A. Similarly, for frame 3 inference, operator 3B depends on operator 2C in frame 2 inference and operator 3A in frame 3 inference. The execution order of operator 3B needs to be after the second frame operator 2C and the third frame operator 3A.

It should be noted that for each frame of reasoning, the input parameter or output parameter of at least one operator in each frame of reasoning and the input parameter or input parameter of other operators except the at least one operator can be obtained through the description information in the model. Dependency of output parameters. In the model shown in Figure 6, for the first frame of inference, operator 1B and operator 1C depend on operator 1A, and operator 1D depends on operator 1B and operator 1C. Therefore, the execution order of operators in the first frame of inference is: operator 1A, operator 1B, operator 1C, and operator 1D. Similarly, for the second frame of reasoning, the execution order of the operators is: operator 2A, operator 2B, operator 2C, and operator 2D. For the third frame of inference, the execution order of operators is: operator 3A, operator 3B, operator 3C, and operator 3D.

In summary, according to the execution order, each operator is added to multiple execution queues, as follows:

The first time T (marked as 1T): Operator A can start execution in all three frames of inference. Specifically, operator 1A is in execution queue 1, operator 2A is in execution queue 2, and operator 3A is in execution queue 3.

The second time T (marked as 2T): In the first frame of inference, the operator 1B and the operator 1C are relieved of the dependence on the operator 1A in the first frame of inference, and execution is started. Operator 1B is in execution queue 1, and operator 1C is in execution queue 2. In the second frame of inference, the operator 2C removes the dependence on the operator 2A in the second frame of inference and starts execution. Operator 2C is in execution queue 3. In the third frame of inference, the operator 3C removes the dependence on the operator 3A in the third frame of inference, and starts execution. Operator 3C is in execution queue 4.

The third time T (marked as 3T): In the first frame of inference, the operator 1D removes the dependence on the operator 1B and the operator 1C in the first frame of inference, starts execution, and the operator 1D is in the execution queue 2. In the second frame of inference, the operator 2B relieves the dependence on the operator 1C in the first frame of inference, starts execution, and the operator 2B is in the execution queue 3. In the third frame of inference, the operator 3B relieves the dependence on the operator 2C in the second frame of inference, starts execution, and the operator 3B is in the execution queue 4.

The fourth time T (marked as 4T): In the second frame of inference, the operator 2D relieves the dependence on the operator 2B and the operator 2C in the second frame of inference, and starts execution, and the operator 2D is in the execution queue 3. In the third frame of inference, the operator 3D is relieved of the dependence on the operator 3B and the operator 3C in the third frame of inference, and execution is started, and the operator 3D is in the execution queue 4.

It can be seen that the operators in the 3-frame continuous reasoning can realize parallel scheduling, thereby making full use of the computing power of computing resources and improving the reasoning performance of multi-frame reasoning.

It should be noted that, in the foregoing example, which execution queue the operator is added to is only an example, and this embodiment does not limit this. For example, at the third time T(3T), the operator 1D can also be in the execution queue 1, and at the fourth time T(4T), the operator 2D can also be in the execution queue 1, and the operator 3D can also be Execute in execution queue 2.

In the model-based signal inference method provided in this embodiment, by loading the preset model and dependent information, each operator of the preset model in the current frame inference can be scheduled according to the dependent information, so that the operators can be executed in parallel, making full use of computing resources Its computing power improves the reasoning performance of multi-frame reasoning.

On the basis of any of the foregoing embodiments, optionally, before performing multi-frame inference (S802) of the preset model on the input signal according to the dependency information in the preset model, the model-based signal inference method provided in this embodiment , Can also include:

Set the dependency information in the description information of the preset model. The description information is used to describe at least one operator included in the preset model and the input parameter and output parameter of each operator in the at least one operator.

Specifically, the description information of the preset model is used to describe the basic information of the operators in the preset model, including the number, names, input parameters, and output parameters of the operators. However, the description information cannot reflect the dependency between different frames when the preset model is used for multi-frame reasoning. For example, refer to Figures 6 and 7. In the description information shown in Fig. 7, operator B has two input parameters a and y. However, the dependency relationship between the input parameter y of operator B and operator C cannot be obtained from the description information. As shown in Figure 6, for two adjacent frames of inference, the input parameter y of operator B in the next frame of inference is the output parameter c of operator C in the previous frame of inference. By setting the dependency information in the description information of the preset model, the dependency information can be added to the preset model, so that the multi-frame reasoning with dependencies can be executed in parallel according to the dependency information in the multi-frame reasoning, and the multi-frame reasoning can be improved. Inference performance.

Optionally, in an implementation manner, setting dependency information in the description information of the preset model may include:

Dependent information is set in the description information of the preset model by calling the interface of the preset model, and the interface of the preset model is used to provide an entrance for information modification of the preset model.

Optionally, the interface of the preset model may include at least one of the following: a function interface, an application programming interface (application programming interface, API) interface, and an executable command.

The following is an example to illustrate.

In the first example, an executable command can be called, and the dependency information can be set by adding a new independent parameter (file or character string) in the description information of the preset model. For example, the execution command is:

Modeltrans orgModel:xxx desModel:yyy frameDependDecs:"{OP(B)input(c)}depend{frame(-1)OP(C)output(c)}"

In the second example, for different open source frameworks, such as tensorflow, you can call the function interface and set the dependency information in the description information of the preset model. For example, the python interface is provided as follows:

import xxx

xxx.add_frame_depend(orgGraph,frameDependInfo,destGraph)

In the third example, the description information of the preset model can be a binary file or a TXT file. Refer to Figure 11. FIG. 11 is a schematic diagram of description information and dependency information of a model provided by an embodiment of the application. Among them, the left side in FIG. 11 is the original description information of the preset model, which can be referred to the related description in FIG. 7, which is not repeated here. You can call executable commands to set dependent information in the original description information of the preset model, as shown on the right in Figure 11. Wherein, the dependency information may include the interframe interval -1, which is used to indicate that the input parameter (c) of the operator B in the target frame reasoning has a dependency relationship with the output parameter (c) of the operator C in the reference frame reasoning. Among them, the target frame reasoning can be the next frame reasoning in two consecutive frames of reasoning, and the reference frame reasoning can be the previous frame reasoning of the two consecutive frames of reasoning. For example, the execution command is:

Modeltrans orgModel:{xxx,graphDescFile}desModel:yyy

Optionally, before setting the dependency information in the description information of the preset model, it may also include:

Get the description information.

Obtain dependency information according to at least one operator included in the description information and the input parameters and output parameters of each operator.

In this implementation manner, by acquiring the description information of the preset model, the dependency information can be acquired through the basic information of the operator included in the description information, and the implementation manner is simple.

Optionally, in another implementation manner, setting dependency information in the description information of the preset model may include:

Receive dependency information entered by the user.

Set the dependency information in the description information of the preset model according to the dependency information input by the user.

In this implementation manner, the dependence information can be directly set in the description information of the preset model according to the dependence information input by the user, and the implementation manner is simple.

Below, on the basis of any of the foregoing embodiments, the dependency information in the preset model will be described in detail.

Optionally, in an implementation manner, the dependency information may include the dependency relationship between the input parameter of the operator in the target frame inference and the output parameter of the operator in the reference frame inference.

In this implementation, in terms of time, the target frame reasoning is behind the reference frame reasoning. The input parameters of the operator in the target frame inference depend on the output parameters of the operator in the reference frame inference. For example, as shown in Figure 6, the target frame reasoning can be the second frame reasoning, and the reference frame reasoning can be the first frame reasoning. The input parameters of operator B in the second frame of inference are dependent on the output parameters of operator C in the first frame of inference.

Optionally, the dependency information may include a first inter-frame interval, a first operator identifier, and a first parameter identifier, and the first inter-frame interval is used to indicate a dependency relationship with the first input parameter of the first operator in the target frame inference The interval between the reference frame reasoning and the target frame reasoning, the first operator identifier is used to indicate the operator that has a dependency relationship with the first operator in the reference frame reasoning, and the first parameter identifier is used to indicate the reference frame reasoning and the first operator. An input parameter has a dependency relationship.

The dependency information will be described below in conjunction with Figure 6 and Figure 11.

As shown in Figure 6 and Figure 11, the target frame reasoning can be the second frame reasoning, and the reference frame reasoning can be the first frame reasoning. The first operator in target frame reasoning is operator B. The first input parameter of the first operator in the target frame inference is the parameter c. The first inter-frame interval is the inter-frame interval between the second frame of inference and the first frame of inference, specifically -1. The first operator is identified as the operator C in the first frame of inference. The first parameter is identified as the output parameter c of the operator C in the first frame of inference.

It should be noted that in this example, the first interframe interval, the first operator identifier, and the first parameter identifier are just examples. When the preset models are different, the specific parameters of the parameters are different. The specific fetching is not limited.

Optionally, in another implementation manner, the dependency information includes the dependency relationship between the output parameter of the operator in the target frame reasoning and the input parameter of the operator in the reference frame reasoning.

In this implementation manner, in terms of time, the target frame reasoning is in front of the reference frame reasoning. The input parameters of the operator in the reference frame reasoning depend on the output parameters of the operator in the target frame reasoning. For example, as shown in Figure 6, the target frame reasoning can be the first frame reasoning, and the reference frame reasoning can be the second frame reasoning. The input parameters of operator B in the second frame of inference are dependent on the output parameters of operator C in the first frame of inference.

Optionally, the dependency information may include a second inter-frame interval, a second operator identifier, and a second parameter identifier, and the second inter-frame interval is used to indicate a dependency relationship with the first output parameter of the second operator in the target frame inference The interval between the reference frame reasoning and the target frame reasoning, the second operator identifier is used to indicate the operator that has a dependency relationship with the second operator in the reference frame reasoning, and the second parameter identifier is used to indicate the reference frame reasoning and the first A parameter whose output parameter has a dependency relationship.

The dependency information will be described below in conjunction with Figure 6 and Figure 12.

As shown in Figure 6 and Figure 12, the target frame reasoning can be the first frame reasoning, and the reference frame reasoning can be the second frame reasoning. The second operator in target frame reasoning is operator C. The first output parameter of the second operator in the target frame inference is the parameter c. The second inter-frame interval is the interval between the second frame of inference and the first frame of inference, and is specifically 1. The second operator is identified as operator B in frame 2 inference. The second parameter is identified as the input parameter c of operator B in the second frame of inference.

It should be noted that, in this example, the indications of the second inter-frame interval, the second operator identifier, and the second parameter identifier are just examples. When the preset models are different, the specific indications of the parameters are different. The specific fetching is not limited.

FIG. 13 is a schematic structural diagram of a model-based signal reasoning device provided by an embodiment of the application. The model-based signal reasoning device provided in this embodiment can execute the model-based signal reasoning method provided in the embodiment of this application. As shown in FIG. 13, the model-based signal reasoning device provided in this embodiment may include:

The receiving module 21 is configured to receive an input signal, the input signal being a digital signal that can be processed by a computer or a processor;

The processing module 22 is configured to perform inference of the preset model on the input signal according to the dependency information in the preset model to obtain an output signal;

Wherein, the preset model includes an operator, the operator includes an input parameter and an output parameter; the dependency information is used to indicate that when the preset model is used to perform multi-frame inference on the input signal, the target frame is inferred The dependency relationship between the input parameter or output parameter of at least one of the operators in the operator and the input parameter or output parameter of the at least one operator in the reference frame reasoning, and the target frame reasoning is For at least one frame of reasoning in the multi-frame reasoning, the reference frame reasoning is at least one frame of reasoning in the multi-frame reasoning other than the target frame reasoning.

Optionally, the processing module 22 is further configured to:

The dependency information is set in the description information of the preset model; wherein the description information is used to describe at least one operator included in the preset model and the input of each operator in the at least one operator Parameters and output parameters.

Optionally, the processing module 22 is specifically configured to:

The dependency information is set in the description information of the preset model by calling the interface of the preset model, and the interface of the preset model is used to provide an entrance for information modification for the preset model.

Optionally, the receiving module 21 is further configured to receive the dependency information input by the user;

The processing module 22 is specifically configured to set the dependency information in the description information of the preset model according to the dependency information input by the user.

Optionally, the processing module 22 is further configured to:

Obtain the description information;

The dependency information is acquired according to at least one operator included in the description information and the input parameter and output parameter of each operator.

Optionally, the dependency information includes a dependency relationship between an input parameter of an operator in the target frame inference and an output parameter of the operator in the reference frame inference.

Optionally, the dependency information includes a first inter-frame interval, a first operator identifier, and a first parameter identifier, and the first inter-frame interval is used to indicate the first interval of the first operator in the inference with the target frame. The interval between the reference frame reasoning whose input parameters have a dependency relationship and the target frame reasoning, and the first operator identifier is used to indicate an operator that has a dependency relationship with the first operator in the reference frame reasoning, The first parameter identifier is used to indicate a parameter that has a dependency relationship with the first input parameter in the reference frame inference.

Optionally, the dependency information includes the dependency relationship between the output parameter of the operator in the target frame inference and the input parameter of the operator in the reference frame inference.

Optionally, the dependency information includes a second inter-frame interval, a second operator identifier, and a second parameter identifier, and the second inter-frame interval is used to indicate the first interval of the second operator in inference with the target frame. The interval between the reference frame reasoning whose output parameter has a dependency relationship and the target frame reasoning, and the second operator identifier is used to indicate an operator that has a dependency relationship with the second operator in the reference frame reasoning, The second parameter identifier is used to indicate a parameter that is dependent on the first output parameter in the reference frame inference.

Optionally, the processing module 22 is specifically configured to:

Loading the preset model and the dependency information;

Scheduling each operator of the preset model in the current frame inference according to the dependency information.

Optionally, the processing module 22 is specifically configured to:

Determine the execution order of the operators of the preset model in the current frame reasoning according to the dependency information;

Each operator of the preset model in the current frame reasoning is added to multiple execution queues according to the execution order; wherein, the input parameters of the multiple operators that are executed at the same time in the multiple execution queues There is no dependency between.

Optionally, the processable digital signal includes at least one of the following: image data, voice data, and text data.

The model-based signal reasoning device provided in this embodiment can execute the model-based signal reasoning method provided in the embodiment of this application. The technical principles and technical effects are similar, and will not be repeated here.

It should be understood that the division of modules in the above device is only a division of logical functions, and may be fully or partially integrated into one physical entity in actual implementation, or may be physically separated. In addition, the modules in the device may be all implemented in the form of software called through processing elements, or all may be implemented in the form of hardware, some modules may also be implemented in the form of software called through the processing elements, and some modules are implemented in the form of hardware. For example, each module can be a separately set up processing element, or it can be integrated in a certain chip of the device for implementation. In addition, it can also be stored in the memory in the form of a program, which is called and executed by a certain processing element of the device. Features. In addition, all or part of these modules can be integrated together or implemented independently. The processing element described here can also become a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules may be implemented by an integrated logic circuit of hardware in a processor element or implemented in a form of being called by software through a processing element.

In an example, the modules in the above device may be one or more integrated circuits configured to implement the above method, for example: one or more application specific integrated circuits (ASIC), or one or more Data signal processor (digital signal processor, DSP), or, one or more field programmable gate arrays (FPGA), or a combination of at least two of these integrated circuits. For another example, when the modules in the above device can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a CPU or other processors that can call programs. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

In an example, the above receiving module may be the interface circuit of the above device, which is used to transmit signals with other devices.

FIG. 14 is a schematic diagram of another structure of a model-based signal inference device provided by an embodiment of the application. The model-based signal reasoning device provided in this embodiment can execute the model-based signal reasoning method provided in the embodiment of this application. As shown in FIG. 14, the model-based signal inference device provided in this embodiment may include a processor 31 and a transmission interface 32. Wherein, the transmission interface 32 can communicate with other devices, and the processor 31 can execute the model-based signal reasoning method provided in the embodiment of the present application. Optionally, the model-based signal inference device may further include a memory.

It should be noted that this embodiment does not limit the type of model-based signal inference device. For example, the model-based signal inference device may be a terminal device, a server, or a computer as shown in FIG. 1.

It should be understood that the processor may be a general-purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the implementation of this application. The methods, steps and logic block diagrams disclosed in the examples. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor.

In the embodiment of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), for example Random access memory (random access memory, RAM). The memory is any medium that can be used to carry or store desired program codes in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The memory in the embodiments of the present application may also be a circuit or any other device capable of realizing a storage function, for storing program instructions and/or data.

FIG. 15 is a schematic diagram of another structure of the model-based signal reasoning device provided by an embodiment of the application. As shown in FIG. 15, the hardware architecture of the model-based signal inference device can be applied to SOC and application processor (application processor, AP), and can execute the model-based signal inference method provided in the embodiments of the present application.

Exemplarily, the model-based signal inference device may include at least one CPU, at least one memory, GPU, decoder, dedicated video/graphics processor, receiving interface, transmitting interface, etc. Optionally, the model-based signal inference device may also include a microprocessor and a microcontroller (microcontroller unit, MCU), etc. In an optional situation, the above-mentioned parts of the model-based signal inference device are coupled through connectors. It should be understood that, in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or Indirect connection through other devices, such as various interfaces, transmission lines or buses, etc. These interfaces are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces, which are not limited in this embodiment . In an optional case, the above-mentioned parts are integrated on the same chip. In another optional situation, the CPU, GPU, decoder, receiving interface, and sending interface can be integrated on a chip, and the internal parts of the chip access external memory through a bus. The dedicated video/graphics processor can be integrated with the CPU on the same chip, or it can exist as a separate processor chip, for example, the dedicated video/graphics processor can be a dedicated image signal processor (ISP) . The chip involved in the embodiments of this application is a system manufactured on the same semiconductor substrate by an integrated circuit process, also called a semiconductor chip, which can be manufactured on a substrate using an integrated circuit process (usually a semiconductor such as silicon) The outer layer of the integrated circuit formed on the material) is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various types of functional devices, and each type of functional device includes transistors such as logic gate circuits, metal-oxide-semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include capacitors and resistors. Or inductance and other components. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

Optionally, the CPU may be a single-CPU (single-CPU) processor or a multi-core (multi-CPU) processor. Optionally, the CPU may be a processor group composed of multiple processors, and the multiple processors are coupled to each other through one or more buses. In an optional situation, part of the processing of the image signal or video signal is completed by the GPU, part is completed by a dedicated video/graphics processor, and may also be completed by software code running on a general-purpose CPU or GPU.

The device may also include a memory, which may be used to store computer program instructions, including various computer program codes including an operating system (OS), various user application programs, and program codes for executing the solutions of the present application. The memory can also be used to store video data, image data, and so on. The CPU may be used to execute the computer program code stored in the memory to implement the method in the embodiment of the present application. Optionally, the memory may be a non-power-down volatile memory, such as an embedded multimedia card (EMMC), universal flash storage (UFS) or read-only memory (ROM). ), or other types of static storage devices that can store static information and instructions, or volatile memory, such as RAM or other types of dynamic storage devices that can store information and instructions, or It is electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs) , Optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other computer-readable storage that can be used to carry or store program code in the form of instructions or data structures and that can be accessed by a computer Medium, but not limited to this.

The receiving interface may be a data input interface of the processor chip. In an optional case, the receiving interface may be a mobile industry processor interface (MIPI) or a high definition multimedia interface (MIPI). multimedia interface, HDMI) or Display Port (DP), etc.

Claims

A model-based signal reasoning method, which is characterized in that it includes:

Receiving an input signal, the input signal being a digital signal that can be processed by a computer or a processor;

Perform the inference of the preset model on the input signal according to the dependency information in the preset model to obtain the output signal;

Wherein, the preset model includes an operator, the operator includes an input parameter and an output parameter; the dependency information is used to indicate that when the preset model is used to perform multi-frame inference on the input signal, the target frame is inferred The dependency relationship between the input parameter or output parameter of at least one of the operators in the operator and the input parameter or output parameter of the at least one operator in the reference frame reasoning, and the target frame reasoning is For at least one frame of reasoning in the multi-frame reasoning, the reference frame reasoning is at least one frame of reasoning in the multi-frame reasoning other than the target frame reasoning.
The method according to claim 1, wherein before the multi-frame reasoning of the preset model is performed on the input signal according to the dependency information in the preset model, the method further comprises:

The dependency information is set in the description information of the preset model; wherein the description information is used to describe at least one operator included in the preset model and the input of each operator in the at least one operator Parameters and output parameters.
The method according to claim 2, wherein the setting the dependency information in the description information of the preset model comprises:

The dependency information is set in the description information of the preset model by calling the interface of the preset model, and the interface of the preset model is used to provide an entrance for information modification for the preset model.
The method according to claim 2, wherein the setting the dependency information in the description information of the preset model comprises:

Receiving the dependency information input by the user;

The dependency information is set in the description information of the preset model according to the dependency information input by the user.
The method according to claim 2 or 3, wherein before setting the dependency information in the description information of the preset model, the method further comprises:

Obtain the description information;

The dependency information is acquired according to at least one operator included in the description information and the input parameter and output parameter of each operator.
The method according to any one of claims 1 to 5, wherein the dependency information includes the dependency between the input parameters of the operator in the target frame reasoning and the output parameters of the operator in the reference frame reasoning relationship.
The method according to claim 6, wherein the dependency information includes a first inter-frame interval, a first operator identifier, and a first parameter identifier, and the first inter-frame interval is used to indicate a connection with the target frame In the inference, the first input parameter of the first operator has a dependency relationship between the reference frame inference and the target frame inference, and the first operator identifier is used to indicate the reference frame inference and the first The operator has a dependency relationship, and the first parameter identifier is used to indicate a parameter that has a dependency relationship with the first input parameter in the reference frame reasoning.
The method according to any one of claims 1 to 5, wherein the dependency information includes the dependency between the output parameter of the operator in the target frame reasoning and the input parameter of the operator in the reference frame reasoning relationship.
The method according to claim 8, wherein the dependency information includes a second inter-frame interval, a second operator identifier, and a second parameter identifier, and the second inter-frame interval is used to indicate a relationship with the target frame In inference, the interval between the reference frame inference and the target frame inference on which the first output parameter of the second operator has a dependency relationship, and the second operator identifier is used to indicate that the reference frame inference is related to the second The operator has a dependency relationship, and the second parameter identifier is used to indicate a parameter that has a dependency relationship with the first output parameter in the reference frame inference.
The method according to any one of claims 1 to 5, wherein the inference of the preset model on the input signal according to dependency information in the preset model comprises:

Loading the preset model and the dependency information;

Scheduling each operator of the preset model in the current frame inference according to the dependency information.
The method according to claim 10, wherein the scheduling each operator of the preset model in the current frame inference according to the dependency information comprises:

Determine the execution order of the operators of the preset model in the current frame reasoning according to the dependency information;

Each operator of the preset model in the current frame reasoning is added to multiple execution queues according to the execution order; wherein, the input parameters of the multiple operators that are executed at the same time in the multiple execution queues There is no dependency between.
The method according to any one of claims 1 to 5, wherein the processable digital signal includes at least one of the following: image data, voice data, and text data.
A model-based signal reasoning device, which is characterized by comprising: a processor and a transmission interface;

The transmission interface is used to receive an input signal, and the input signal is a digital signal that can be processed by a computer or a processor;

The processor is configured to perform inference of the preset model on the input signal according to the dependency information in the preset model to obtain an output signal;

Wherein, the preset model includes an operator, the operator includes an input parameter and an output parameter; the dependency information is used to indicate that when the preset model is used to perform multi-frame inference on the input signal, the target frame is inferred The dependency relationship between the input parameter or output parameter of at least one of the operators in the operator and the input parameter or output parameter of the at least one operator in the reference frame reasoning, and the target frame reasoning is For at least one frame of reasoning in the multi-frame reasoning, the reference frame reasoning is at least one frame of the other frame reasoning in the multi-frame reasoning except the target frame reasoning.
The device according to claim 13, wherein the processor is further configured to:

The dependency information is set in the description information of the preset model; wherein the description information is used to describe at least one operator included in the preset model and the input of each operator in the at least one operator Parameters and output parameters.
The device according to claim 14, wherein the processor is specifically configured to:

The dependency information is set in the description information of the preset model by calling the interface of the preset model, and the interface of the preset model is used to provide an entrance for information modification for the preset model.
The device according to claim 14, wherein:

The transmission interface is also used to receive the dependency information input by the user;

The processor is specifically configured to set the dependency information in the description information of the preset model according to the dependency information input by the user.
The device according to claim 14 or 15, wherein the processor is further configured to:

Obtain the description information;

The dependency information is acquired according to at least one operator included in the description information and the input parameter and output parameter of each operator.
The device according to any one of claims 13 to 17, wherein the dependency information includes the dependency between the input parameter of the operator in the target frame inference and the output parameter of the operator in the reference frame inference relationship.
The apparatus according to claim 18, wherein the dependency information includes a first inter-frame interval, a first operator identifier, and a first parameter identifier, and the first inter-frame interval is used to indicate a connection with the target frame In the inference, the first input parameter of the first operator has a dependency relationship between the reference frame inference and the target frame inference, and the first operator identifier is used to indicate the reference frame inference and the first The operator has a dependency relationship, and the first parameter identifier is used to indicate a parameter that has a dependency relationship with the first input parameter in the reference frame reasoning.
The apparatus according to any one of claims 13 to 19, wherein the dependency information includes the dependency between the output parameter of the operator in the target frame reasoning and the input parameter of the operator in the reference frame reasoning relationship.
The apparatus according to claim 20, wherein the dependency information includes a second inter-frame interval, a second operator identifier, and a second parameter identifier, and the second inter-frame interval is used to indicate a connection with the target frame In inference, the interval between the reference frame inference and the target frame inference on which the first output parameter of the second operator has a dependency relationship, and the second operator identifier is used to indicate that the reference frame inference is related to the second The operator has a dependency relationship, and the second parameter identifier is used to indicate a parameter that has a dependency relationship with the first output parameter in the reference frame inference.
The device according to any one of claims 13 to 21, wherein the processor is specifically configured to:

Loading the preset model and the dependency information;

Scheduling each operator of the preset model in the current frame inference according to the dependency information.
The device according to claim 22, wherein the processor is specifically configured to:

Determine the execution order of the operators of the preset model in the current frame reasoning according to the dependency information;

Each operator of the preset model in the current frame reasoning is added to multiple execution queues according to the execution order; wherein, the input parameters of the multiple operators that are executed at the same time in the multiple execution queues There is no dependency between.
The device according to any one of claims 13 to 23, wherein the processable digital signal comprises at least one of the following: image data, voice data, and text data.
A computer-readable storage medium, characterized in that instructions are stored in the computer-readable storage medium, and when the instructions are run on a computer or a processor, the instructions described in any one of claims 1 to 12 are implemented. Model-based signal reasoning method.