CN114840719A

CN114840719A - Detection model post-processing method, medium, electronic device, and program product

Info

Publication number: CN114840719A
Application number: CN202210521601.1A
Authority: CN
Inventors: 章小龙; 许礼武; 黄敦博; 陈柏韬
Original assignee: ARM Technology China Co Ltd
Current assignee: ARM Technology China Co Ltd
Priority date: 2022-05-13
Filing date: 2022-05-13
Publication date: 2022-08-02

Abstract

The application relates to the field of artificial intelligence and discloses a detection model post-processing method, medium, electronic equipment and program product. The method comprises the following steps: determining a candidate bounding box on a processing object, and carrying out adjustment operation on the candidate bounding box; acquiring bounding box data and area adjusting data corresponding to the adjusting operation, wherein the area adjusting data comprises first adjusting data and second adjusting data; determining a table lookup parameter associated with the second adjustment data, and acquiring a table lookup result corresponding to the table lookup parameter from a lookup table; inputting the bounding box data and the first adjustment data into the first operation part to obtain a first operation result; determining a second operation result of the second operation part based on a linear operation of the bounding box data and the table look-up result; determining adjusted bounding box data based on a linear operation of the first operation result and the second operation result. Thus, the running speed of the electronic equipment can be improved.

Description

Detection model post-processing method, medium, electronic device, and program product

Technical Field

The present application relates to the field of artificial intelligence, and in particular, to a detection model post-processing method, medium, electronic device, and program product.

Background

With the rapid development of Artificial Intelligence (AI), neural network models are increasingly widely applied in the field of artificial intelligence. Deep learning takes Deep Neural Networks (DNNs) as a model, and achieves very remarkable results in key fields of many human intelligence, such as image recognition, target detection, reinforcement learning, semantic analysis and the like.

Because the Neural Network is a resource-intensive algorithm, it not only needs a large amount of computation cost, but also consumes a large amount of memory, and an arithmetic Unit, such as a Neural-Network Processing Unit (NPU), for operating the Neural Network model is usually a fixed-point arithmetic Unit, and it does not have a circuit capable of directly implementing nonlinear operations such as exponential function, division, etc. in each operator of the Neural Network model, but implements the operations by iterative solution, which consumes a large amount of time. In order to increase the operation speed of the neural network model, the neural network model is usually quantized to obtain a fixed-point operation neural network model, and then the neural network model is operated by an electronic device.

The mainstream model quantization scheme at present is to use fixed point quantization for a backbone network (backbone) model, and floating point calculation is adopted in a model post-processing part to ensure the inference precision of a neural network model. However, this method still consumes a lot of computation cost, occupies a lot of hardware resources of the NPU, and affects the speed of the electronic device running the neural network model.

Disclosure of Invention

In view of this, embodiments of the present application provide a detection model post-processing method, medium, electronic device, and program product.

In a first aspect, an embodiment of the present application provides a detection model post-processing method, which is applied to an electronic device, and includes:

determining a candidate bounding box on a processing object, and performing an adjustment operation on the candidate bounding box, wherein the adjustment operation comprises a first operation part and a second operation part, the first operation part comprises a first constant division operation factor, and the second operation part comprises a second constant division operation factor and at least one nonlinear operation factor;

acquiring bounding box data and area adjusting data corresponding to the adjusting operation, wherein the area adjusting data comprises first adjusting data and second adjusting data;

determining a table lookup parameter associated with the second adjustment data, and acquiring a table lookup result corresponding to the table lookup parameter from a lookup table;

inputting the bounding box data and the first adjustment data into the first operation part to obtain a first operation result;

determining a second operation result of the second operation part based on a linear operation of the bounding box data and the table look-up result;

determining adjusted bounding box data based on a linear operation of the first operation result and the second operation result.

It is understood that, during the operation of the detection model, if the adjustment operation for the candidate bounding box is detected, such as an operation of adjusting the anchor frame generated by the model by a proposal operator, the adjustment data may be obtained by determining a table look-up parameter associated with the second adjustment data, selecting from a predetermined table look-up based on the table look-up parameter, obtaining fixed point number table look-up result of the part including the second constant division operation factor and the non-linear operation factor in the adjustment operation, determining the fixed point number operation result of the second operation part according to the table look-up result, further determining the operation result of the whole adjustment operation without constant division operation and complex nonlinear operation, the operation precision is ensured, the operation amount is reduced, hardware resources occupied by the electronic equipment when the detection model is operated are reduced, and the operation speed of the electronic equipment including the adjustment operation of the constant division operation factor and the nonlinear operation factor is improved.

In one possible implementation of the first aspect, the inputting the bounding box data and the first adjustment data into the first operation part to obtain a first operation result includes:

determining an input quantization parameter for the region adjustment data;

constant division operation is carried out on the input quantization parameter based on the first constant division operation factor to obtain a first quantization parameter;

performing inverse quantization on the first adjustment data based on the first quantization parameter to obtain first inverse quantization result data of a first sub-operation including the first constant division operation factor;

determining a first operation result of the first operation portion based on a linear operation of the bounding box data and the first inverse quantization result data.

It can be understood that, when the operation result of the first operation part is calculated, the first adjustment data is dequantized based on the first quantization parameter, and then is subjected to linear operation with the bounding box data, so that the quantization parameter of the bounding box data can be reserved for transmission, and the normal operation of the operation process is ensured.

In one possible implementation of the first aspect, the inverse quantizing the first adjustment data based on the first quantization parameter to obtain first inverse quantization result data of a first sub-operation including the first constant division operation factor includes:

determining a first scaling coefficient and a first shift number corresponding to the first quantization parameter, wherein the first scaling coefficient and the first shift number are fixed-point numbers of a first target quantization bit number, and the first target quantization bit number is determined by the first quantization bit number of the bounding box data and a second quantization bit number of the area adjustment data;

and multiplying the first adjustment data by the first scaling coefficient, and shifting the obtained product to the right by the first shift digit to obtain the first inverse quantization result data.

In one possible implementation of the first aspect, the bounding box data comprises first bounding box data and second bounding box data;

the determining a first operation result of the first operation portion based on a linear operation of the bounding box data and the first inverse quantization result data includes:

and multiplying the first inverse quantization result data by the first bounding box data, and then adding the first inverse quantization result data and the second bounding box data to obtain a first operation result of the first operation part.

It can be understood that, when the electronic device performs the adjustment operation on the candidate bounding box, the first constant division operation factor can be directly fused into the quantization parameter of the first adjustment data, so that the constant division operation on the first adjustment data is removed, the precision loss in the operation process is reduced, the speed of the electronic device for operating the adjustment operation including the constant division operation factor and the nonlinear operation factor is further increased, and the performance of the electronic device is improved.

In a possible implementation of the first aspect, the obtaining a table lookup result corresponding to the table lookup parameter from the table lookup includes:

determining an index value corresponding to the table look-up parameter in a look-up table; the table lookup parameter and the index value are fixed-point numbers, and the floating-point number corresponding to the table lookup parameter is the same as the floating-point number corresponding to the index value in numerical value;

obtaining a table look-up result corresponding to the index value in the look-up table; the index value of the lookup table and the lookup result are both values of a second target quantization bit number, and the second target quantization bit number is determined by the first quantization bit number of the bounding box data.

In one possible implementation of the first aspect, the determining a second operation result of the second operation part based on a linear operation of the bounding box data and the table lookup result includes:

determining a second quantization parameter of the table look-up result;

performing inverse quantization on the table look-up result based on the second quantization parameter to obtain second inverse quantization result data;

and multiplying the second inverse quantization result data by the first bounding box data to obtain a second operation result of the second operation part.

It can be understood that when the operation result of the second operation part is calculated, the table look-up result is inversely quantized based on the second quantization parameter, and then the table look-up result and the bounding box data are linearly operated, so that the quantization parameter of the bounding box data can be reserved for transmission, and the normal operation of the operation process is ensured.

In a possible implementation of the first aspect, the inverse quantizing the table lookup result based on the second quantization parameter to obtain second inverse quantization result data includes:

determining a second scaling coefficient and a second shift number corresponding to the second quantization parameter, wherein the second scaling coefficient and the second shift number are both integers;

and multiplying the table look-up result by the second scaling coefficient, and shifting the obtained product to the right by the second shift digit to obtain second inverse quantization result data.

It can be understood that, by representing the first quantization parameter and the second quantization parameter in the form of fixed-point multiplication and shift, the operation amount of the electronic device can be further reduced, so as to improve the operation speed of the electronic device.

In a second aspect, the present application provides a readable medium, where the readable medium contains instructions, and when the instructions are executed by a processor of an electronic device, the instructions cause the electronic device to implement the first aspect and any one of the detection model post-processing methods provided in the various possible implementations of the first aspect.

In a third aspect, the present application provides an electronic device, including: a memory to store instructions for execution by one or more processors of an electronic device; and a processor, which is one of the processors of the electronic device, and is configured to execute instructions to enable the electronic device to implement the detection model post-processing method according to the first aspect and any one of the various possible implementations of the first aspect.

In a fourth aspect, the present application provides a computer program product, where the computer program product includes instructions for implementing the detection model post-processing method according to the first aspect described above and any one of the various possible implementations of the first aspect described above.

Drawings

FIG. 1 illustrates a schematic structural diagram of a detection model, according to some embodiments of the present application;

FIG. 2 illustrates a schematic diagram of a procedure for a proposal operator, according to some embodiments of the present application;

FIG. 3 illustrates a schematic diagram of another exemplary operation of a proposal operator, according to some embodiments of the present disclosure;

FIG. 4 illustrates a flow diagram of a method of inspection model post-processing, according to some embodiments of the present application;

FIG. 5 illustrates a flow chart for calculating a result of a first operation, according to some embodiments of the present application;

FIG. 6 illustrates a schematic diagram of a detection model post-processing apparatus, according to some embodiments of the present application;

FIG. 7 illustrates a schematic structural diagram of an electronic device, according to some embodiments of the present application.

Detailed Description

Illustrative embodiments of the present application include, but are not limited to, detection model post-processing methods, media, and electronic devices. Embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

For a clearer understanding of the present application, the structure of the neural network model will now be described.

FIG. 1 illustrates a schematic structural diagram of a detection model, according to some embodiments of the present application. As shown in fig. 1, the detection model 10 includes a Backbone Network (Backbone)11, a Region suggestion Network (RPN) 12, and a Region-of-interest header Network (RoiHead) 13. The main network 11 performs feature extraction on an input image through a plurality of convolution kernels to obtain a plurality of feature maps; the area suggestion network 12 is configured to generate a candidate bounding box corresponding to the output feature map of the backbone network 11, and perform adjustment operation on the candidate bounding box (anchor box) based on the output feature map through a proposal operator to obtain an adjusted bounding box (i.e., an area of interest); the head network 13 of the region of interest completes region feature pooling and classification regression to obtain a final prediction result and calculate loss.

It is understood that the structure of the detection model 10 is only an example, and in other embodiments, the detection model may be any neural network model including a proposal operator, and is not limited herein.

It will be appreciated that the propofol operator can be defined by the following formula, with reference to figure 2 of the drawings.

Where, (tx, ty, th, tw) is region adjustment data (encoding frame data) indicating the offset of the candidate bounding box (anchor box); (xcenter _ a, ycenter _ a, ha, wa) as bounding box data (anchor box data) indicating the center position and size of the candidate bounding box (anchor box); (xmin, ymin, xmax, ymax) is adjusted bounding box data indicating the position of the adjusted bounding box. m and n are hyper-parameters, which are determined by the detection model before being deployed to the electronic device, that is, for a detection model to be operated, the hyper-parameters m and n are known quantities.

Substituting the calculation formula of xcenter, ycenter, h, w into the calculation formula of xmin, ymin, xmax, ymax can obtain:

the following describes the quantization process of the propofol operator.

For convenience of description, in the embodiments of the present application, x _ f represents a floating point number of a variable x, x _ q represents a fixed point number of the variable x, and scale _ x represents a quantization parameter quantized into x _ q by x _ f. Assuming that x _ q is the number of symmetrically quantized n-bit fixed points, the relationship of x _ f, scale _ x, and x _ q can be determined by the following equations (5) and (6).

Wherein max (x) represents the maximum value of x, min (x) represents the minimum value of x, abs (x) represents the absolute value of x, round (x) represents rounding x, and n is the number of quantization bits.

It is understood that round (x) may also be a function of other functions, for example, a floor (x) function, a fix (x) function, and the like, which may discard a fractional part for retaining an integer part, and the embodiment of the present application does not limit this.

It can be understood that, since the NPU can only perform fixed-point operation, when calculating the adjusted bounding box data, all the operation factors in the calculation formula need to be quantized into fixed-point numbers. Since the quantization processes of equations (1) to (4) are similar, equation (1) will be described as an example.

It can be understood that the exponential operation is a nonlinear part in the whole proposal operator, and can be simply realized in a table look-up manner, but the characteristic of the analysis algorithm can be known, as shown in fig. 3, the constant division before the exponential operation and the averaging operation after the exponential operation are fused into a look-up table (LUT table), so that the quantization precision can be improved, and the performance can be greatly improved.

Thus, the portion that will include constant division and exponential operators

As a whole, it can be obtained from equation (1) and equation (6):

wherein (x) _ q represents the fixed point number corresponding to x, sacle _1 represents the quantization parameter of the region adjustment data tx, sacle _2 represents the quantization parameter of the candidate bounding box data wa, xcenter _ a, and sacle _ tw represents the quantization parameter of the candidate bounding box data wa, xcenter _ a

The quantization parameter of the whole.

It is understood that if the quantization parameter that retains the bounding box data is selected for transfer, i.e. such that scale _3 ═ cycle _2 in the above equation, then it can be found that:

as can be seen from equation (7), the fixed point number can be directly used for calculation in addition to the constant division operation factor and the exponential operation factor. Therefore, the constant division operation factor and the exponential operation factor are quantized as the key of the operator quantization, and whether the quantization is accurate or not directly determines the quantization precision of the whole operator.

For easy understanding, the part of equation (7) for adjusting the center coordinates of the candidate bounding box may be used

As a first operation part, a part of the equation (7) for adjusting and equally dividing the width of the candidate bounding box

After the second operation part determines the operation result of the first operation part and the second operation result of the second operation part, the first operation result and the second operation result may be subtracted from each other to obtain the fixed point number result of xmin.

In particular, for a first sub-operation comprising a first constant division operation factor in the first operation portion

Instead of performing a constant division quantization operation, a constant division factor may be fused into the quantization parameter, sacle _1, of tx. Namely, when the operation result of the first operation part is calculated in the on-line operation process, the constant division operation is carried out on the quantization parameter, namely, the fixed number is not operated on tx _ q, and the first quantization parameter is obtained

And then calculating through tx _ q _ sacle _ tx _ wa _ q + xcenter _ a _ q to obtain the operation result of the first operation part.

Wherein tx _ q scale _ tx is equivalent to a first sub-operation comprising the first constant division factorCalculating machine

In order to solve the problem, a bit expansion mode can be adopted in the inverse quantization process to ensure the quantization precision.

In some embodiments, the first quantization parameter scale _ tx may be first subjected to a fi operation, and the first quantization parameter scale _ tx may be converted into a form of multiplication and SHIFT, i.e., fi (scale _ tx) is calculated to obtain the first scaling factor and the first SHIFT number (Q1, SHIFT 1). Wherein Q1 and SHIFT1 are fixed-point numbers of a first target quantization bit number, which can be determined by a first quantization bit number of the bounding box data and a second quantization bit number of the region adjustment data. That is, the operation result of the first operation portion may be calculated by tx _ Q1 > > SHIFT1 wa _ Q + xcenter _ a _ Q, where ">" represents a right SHIFT operation.

In some embodiments, the first target number of quantization bits may be a difference value between a first number of quantization bits of the bounding box data and a second number of quantization bits of the region adjustment data. For example, as shown in fig. 3, assuming that the first quantization bit number of the bounding box data is 16 bits and the second quantization bit number of the region adjustment data is 8 bits, the first target quantization bit number of the first scaling factor Q1 and the first SHIFT number SHIFT1 may be 8 bits.

Specifically, for the second operation portion including constant division operation factor and exponential operation factor

The part can be used as a whole, and when the operation result of the second operation part is calculated in the on-line operation process, the operation result can be obtained in a table look-up mode

Looking up the table result by partial fixed point number and then passing through

And calculating to obtain an operation result of the second operation part.

The table lookup result is multiplied by the sacle _ tw to form an inverse quantization process, and the quantization parameters of the bounding box data can be reserved in the process of multiplying the bounding box data through inverse quantization operation, so that the subsequent quantization requirements are met. Because the inverse quantization can bring about floor errors, the inverse quantization operation can not be directly fused into a lookup table offline from the viewpoint of quantization precision so as to reduce the floor errors.

In some embodiments, scale _ tw may be converted into a form of multiplication and shift, that is, a second scaling coefficient and a second shift number (q2, shift2) corresponding to scale _ tw are determined, where q2 and shift2 are both integers. That is, can pass through

Calculating the operation result of the second operation part, wherein ">>"indicates a right shift operation.

The process of building the look-up table is described below.

(1) The index value in the lookup table and the quantization parameter of the lookup table result can be calculated according to the quantization parameter, cycle _1, of the region adjustment data tw as follows:

where n0 is the quantization bit number of the region adjustment data tw, n1 is a second target quantization bit number, which represents the index value in the lookup table and the quantization bit number of the lookup table result, and the second target quantization bit number may be determined by the first quantization bit number of the bounding box data.

In some embodiments, the second target number of quantization bits may be equal to the first number of quantization bits of the bounding box data. For example, as shown in fig. 3, assuming that the first quantization bit number of the bounding box data is 16 bits, the second target quantization bit number may also be 16 bits.

(2) Can be made based on the quantization parameter scale _ twThe table look-up function is

The look-up table of (2). The index values and the table look-up results in the look-up table are fixed point numbers of the second target quantization bit number, and the table look-up result corresponding to each index value is a fixed point number result obtained by calculating the index value as input data of a table look-up function in advance.

It can be understood that, in the on-line calculation stage, the fixed point number of the area adjustment data tw may be directly used as a table lookup parameter, and an index value in the table lookup equal to the table lookup parameter may be used as an index value corresponding to the table lookup parameter.

In some embodiments, on the premise of ensuring that the floating point number corresponding to the index value is the same as the floating point number corresponding to the table lookup parameter, the method for determining the index value may also be adjusted according to an actual application scenario, which is not limited in the embodiments of the present application.

The following describes a detection model post-processing method provided by the embodiment of the present application.

FIG. 4 illustrates a flow diagram of a detection model post-processing method, according to some embodiments of the present application. The main execution body of the process is an electronic device, as shown in fig. 4, the process includes the following steps:

s401: determining a candidate bounding box on a processing object, and performing an adjustment operation on the candidate bounding box, wherein the adjustment operation comprises a first operation part and a second operation part, the first operation part comprises a first constant division operation factor, and the second operation part comprises a second constant division operation factor and at least one nonlinear operation factor.

In this embodiment of the application, the processing object may be a processing object of the detection model, such as an image; the candidate bounding box (anchor box) may be generated by other operators in the detection model, such as the anchor generators operator.

In the process of operating the detection model, if the operation of the propsal operator is detected, the method provided by the embodiment of the application implements adjustment operation on the candidate bounding box through the propsal operator to adjust the position of the boundary, so as to obtain adjusted data of the bounding box.

For example, in the process of operating the detection model 10, the electronic device may detect a proposal operator when the area suggestion network 12 is operated, and trigger the detection model post-processing method provided in the embodiment of the present application.

S402: and acquiring the bounding box data and the area adjusting data corresponding to the adjusting operation, wherein the area adjusting data comprises first adjusting data and second adjusting data.

In this embodiment of the present application, the bounding box data may be generated by an operator in the detection model, and the region adjustment data may be obtained by decoding a feature map generated in the detection model. Specifically, the bounding box data includes first bounding box data (wa, ha) indicating the width and height of the candidate bounding box and second bounding box data (xcenter _ a, ycenter _ a) indicating the center coordinates of the candidate bounding box, the first adjustment data includes adjustment data (tx, ty) of the center coordinates of the candidate bounding box, and the second adjustment data includes adjustment data (tw, th) of the width and height of the candidate bounding box.

S403: and determining the table look-up parameters associated with the second adjustment data, and acquiring the table look-up result corresponding to the table look-up parameters from the table look-up.

In the embodiment of the present application, the fixed point number of the second adjustment data may be used as a table lookup parameter, and a corresponding table lookup result may be obtained from a pre-constructed lookup table.

Specifically, the obtaining the table lookup result corresponding to the table lookup parameter from the lookup table may include:

Specifically, a part of the second operation part of the proposal operator, which includes constant division operation and nonlinear operation, may be set as a lookup table item to equate the second operation part to a product of the lookup table item and the first bounding box data, and a lookup table for storing a correspondence between an index value determined based on the second adjustment data and a corresponding lookup table item lookup result is pre-constructed, where the lookup table result corresponding to each index value is a fixed-point number result calculated by using the index value as input data in advance. In the process of adjusting the candidate bounding box through the proposal operator, the NPU may determine an index value of the table lookup based on the second adjustment data, and query a table lookup result of a table lookup item corresponding to the index value from the table lookup according to the index value. Therefore, the NPU does not need to perform constant division operation and complex nonlinear operation, the speed of the NPU for adjusting the candidate bounding box is increased, and the running speed of the detection model is increased.

In some embodiments, when constructing the lookup table, the lookup table may be constructed as a lookup table with a second target quantization bit number, that is, each index value and the corresponding lookup table result in the lookup table are fixed-point numbers of the second target quantization bit number, and the second target quantization bit number is the same as the first quantization bit number of the bounding box data, so that the quantization bit number of the lookup table result obtained from the lookup table is the same as the quantization bit number of the bounding box data.

S404: and inputting the bounding box data and the first adjusting data into the first operation part to obtain a first operation result.

In an embodiment of the present application, the adjusting the first adjustment data is adjusting center coordinates of a candidate bounding box, and as shown in fig. 5, the inputting the bounding box data and the first adjustment data into the first operation part to obtain a first operation result may include:

s501: an input quantization parameter for the region adjustment data is determined.

In this embodiment, the quantization parameters of the first region adjustment data and the second region adjustment data are both the input quantization parameters.

S502: and performing constant division operation on the input quantization parameter based on the first constant division operation factor to obtain a first quantization parameter.

For example, when calculating xmin in the adjusted bounding box data, constant division may be performed on the quantization parameter sacle _1 of the first region adjustment data (i.e., the input quantization parameter) to obtain a first quantization parameter

S503: and performing inverse quantization on the first adjustment data based on the first quantization parameter to obtain first inverse quantization result data of a first sub-operation including the first constant division operation factor.

In this embodiment of the present application, the fixed-point number of the first adjustment data is multiplied by the first quantization parameter, so as to obtain a first dequantization result of the first sub-operation including the first constant division operation factor. It can be understood that, when the operation result of the first operation part is calculated, the first adjustment data is dequantized based on the first quantization parameter, and then is subjected to linear operation with the bounding box data, so that the quantization parameter of the bounding box data can be reserved for transmission, and the normal operation of the operation process is ensured.

In some embodiments, the inverse quantizing the first adjustment data based on the first quantization parameter to obtain first inverse quantization result data of a first sub-operation including the first constant division operation factor may include:

Specifically, fi operation may be performed on the first quantization parameter, the first quantization parameter is converted into a form of multiplication and SHIFT, a first scaling coefficient and a first SHIFT number (Q1, SHIFT1) corresponding to the first quantization parameter are obtained, and first inverse quantization result data calculated by tx _ Q × Q1 > > SHIFT1 is obtained, where ">" represents a right SHIFT operation.

It can be understood that, by representing the first quantization parameter in the form of fixed-point number multiplication and shift, the operation amount of the electronic device can be further reduced, so as to improve the operation speed of the electronic device.

S504: determining a first operation result of the first operation portion based on a linear operation of the bounding box data and the first inverse quantization result data.

In an embodiment of the present application, the determining a first operation result of the first operation part based on a linear operation of the bounding box data and the first dequantization result data may include:

For example, when calculating xmin in the adjusted bounding box data, the calculation result of the first arithmetic part may be obtained by multiplying the calculated first inverse quantization result data tx _ Q1 > > SHIFT1 by the fixed point number wa _ Q of the first bounding box data, and adding the resultant product to the fixed point number xcenter _ a _ Q of the second bounding box data, where ">" represents a right SHIFT operation.

In some embodiments, in calculating the operation result of the first operation part, the fixed-point number of the region adjustment data may be multiplied by the first scaling coefficient, the resultant product may be multiplied by the fixed-point number of the first bounding box data, and the resultant product may be shifted to the right by the first shift bit and added to the fixed-point number of the second bounding box data.

For example, when calculating xmin in the adjusted bounding box data, the operation result of the first operation portion may be calculated by tx _ Q1 wa _ Q > > SHIFT1+ xcenter _ a _ Q, where ">" represents a right SHIFT operation. It can be understood that the operation speed of the NPU when adjusting the candidate bounding box can be further increased by adjusting the operation sequence in the calculation process.

It can be understood that when the electronic device performs the adjustment operation on the candidate bounding box, the first constant division operation factor can be directly fused into the quantization parameter of the first adjustment data, so that the constant division operation on the first adjustment data is removed, the precision loss in the operation process is reduced, the speed of the electronic device for operating the adjustment operation including the constant division operation factor and the nonlinear operation factor is further increased, and the performance of the electronic device is improved.

S405: and determining a second operation result of the second operation part based on the linear operation of the bounding box data and the table look-up result.

In this embodiment of the application, the determining the second operation result of the second operation part based on the linear operation of the bounding box data and the table lookup result may include:

determining a second quantization parameter of the table look-up result;

Specifically, the calculating of the second quantization parameter of the table lookup result according to (8), and the performing inverse quantization on the table lookup result based on the second quantization parameter to obtain second inverse quantization result data may include:

In some embodiments, the second quantization parameter may be converted into a form of multiplication and shift, and a second scaling coefficient and a second shift number corresponding to the second quantization parameter are obtained (q2, shift 2).

In some embodiments, when constructing the lookup table, the second quantization parameter may be converted into a form of multiplication and shift to obtain a second scaling coefficient and a second shift number (q2, shift2) corresponding to the second quantization parameter, and the second scaling coefficient and the second shift number (q2, shift2) are stored in the lookup table in advance. A second scaling coefficient and a second shift number (q2, shift2) corresponding to the second quantization parameter pre-stored in the lookup table may be obtained in an online calculation process.

For example, when calculating xmin in the adjusted bounding box data, it is possible to pass

Calculating to obtain second inverse quantization result data, and multiplying the second inverse quantization result data by the fixed point number wa _ q of the first boundary frame data to obtain the operation result of the second operation part, wherein ">>"indicates a right shift operation.

It can be understood that, by representing the second quantization parameter in the form of fixed-point number multiplication and shift, the operation amount of the electronic device can be further reduced, so as to improve the operation speed of the electronic device.

In some embodiments, in calculating the operation result of the second operation part, the table lookup result may be multiplied by the second scaling factor, the obtained product may be multiplied by the fixed-point number of the first bounding box data, and the obtained product may be shifted to the right by the second shift digit.

The operation result of the second operation section is calculated. It can be understood that the operation speed of the NPU when adjusting the candidate bounding box can be further increased by adjusting the operation sequence in the calculation process.

S406: determining adjusted bounding box data based on a linear operation of the first operation result and the second operation result.

In an embodiment of the present application, the adjusted data of the bounding box includes minimum abscissa data, maximum abscissa data, minimum ordinate data, and maximum ordinate data (xmin, ymin, xmax, ymax) of the bounding box. For the minimum abscissa data and the minimum ordinate data, the corresponding first operation result and second operation result may be subtracted, and for the maximum abscissa data and the maximum ordinate data, the corresponding first operation result and second operation result may be added.

It is understood that, in the process of performing a linear operation on the first operation result and the second operation result, the quantization parameter output of the bounding box data may be selected, that is, the quantization parameter finally output by the proposal operator is the quantization parameter scale2 of the bounding box data.

In summary, in the process of running the detection model, if the adjustment operation for the candidate bounding box is detected, such as an operation of adjusting the anchor frame generated by the model by a proposal operator, the adjustment data may be obtained by determining a table look-up parameter associated with the second adjustment data, selecting from a predetermined table look-up based on the table look-up parameter, obtaining fixed point number table look-up result of the part including the second constant division operation factor and the non-linear operation factor in the adjustment operation, determining the fixed point number operation result of the second operation part according to the table look-up result, further determining the operation result of the whole adjustment operation without constant division operation and complex nonlinear operation, the operation precision is ensured, the operation amount is reduced, hardware resources occupied by the electronic equipment when the detection model is operated are reduced, and the operation speed of the electronic equipment including the adjustment operation of the constant division operation factor and the nonlinear operation factor is improved.

Fig. 6 shows a schematic structural diagram of an apparatus 600 for post-processing an inspection model according to some embodiments of the present application, where the apparatus 600 is disposed in an electronic device, and as shown in fig. 6, the apparatus 600 may include:

a determining module 610, configured to determine a candidate bounding box on a processing object, and perform an adjustment operation on the candidate bounding box, where the adjustment operation includes a first operation portion and a second operation portion, the first operation portion includes a first constant division operation factor, and the second operation portion includes a second constant division operation factor and at least one nonlinear operation factor;

an obtaining module 620, configured to obtain bounding box data and area adjustment data corresponding to the adjustment operation, where the area adjustment data includes first adjustment data and second adjustment data;

a table lookup module 630, configured to determine a table lookup parameter associated with the second adjustment data, and obtain a table lookup result corresponding to the table lookup parameter from a lookup table;

a first operation module 640, configured to input the bounding box data and the first adjustment data into the first operation portion to obtain a first operation result;

a second operation module 650, configured to determine a second operation result of the second operation part based on a linear operation of the bounding box data and the table lookup result;

a third operation module 660, configured to determine adjusted bounding box data based on linear operation of the first operation result and the second operation result.

It should be noted that, when the apparatus provided in the foregoing embodiment implements the functions thereof, only the division of the functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the apparatus may be divided into different functional modules to implement all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and specific implementation processes thereof are detailed in the corresponding method embodiments and are not described herein again.

It can be understood that the detection model post-processing method provided in the embodiment of the present application may be applied to any electronic device capable of operating a neural network model, including but not limited to a mobile phone, a wearable device (such as a smart watch), a tablet computer, a desktop, a laptop, a handheld computer, a notebook, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a Personal Digital Assistant (PDA), an Augmented Reality (AR)/Virtual Reality (VR) device, and the like, and the embodiment of the present application is not limited. In order to facilitate understanding of the technical solution of the embodiment of the present application, an electronic device 100 is taken as an example to describe a structure of an electronic device to which the detection model post-processing method provided in the embodiment of the present application is applied.

Further, fig. 7 illustrates a schematic structural diagram of an electronic device 100, according to some embodiments of the present application. As shown in fig. 7, the electronic device 100 includes one or more processors 101, a system Memory 102, a Non-Volatile Memory (NVM) 103, a communication interface 104, an input/output (I/O) device 105, and system control logic 106 for coupling the processors 101, the system Memory 102, the NVM 103, the communication interface 104, and the input/output (I/O) device 105. Wherein:

the processor 101 may include one or more Processing units, for example, Processing modules or Processing circuits that may include a central Processing Unit (cpu), (central Processing Unit), an image processor (gpu), (graphics Processing Unit), a digital Signal processor (dsp), (digital Signal processor), a microprocessor MCU (Micro-programmed Control Unit), an AI (Artificial Intelligence) processor, or a Programmable logic device fpga (field Programmable Gate array), a Neural Network Processor (NPU), and the like, may include one or more single-core or multi-core processors. In some embodiments, the NPU may be configured to run an instruction corresponding to the detection model post-processing method provided in the embodiments of the present application.

The system Memory 102 is a volatile Memory, such as a Random-Access Memory (RAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), and the like. The system memory is used to temporarily store data and/or instructions, for example, in some embodiments, the system memory 102 may be used to store the lookup table 30 described above.

Non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and/or any suitable non-volatile storage device, such as a Hard Disk Drive (HDD), Compact Disc (CD), Digital Versatile Disc (DVD), Solid-State Drive (SSD), and the like. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a Secure Digital (SD) memory card or the like. In other embodiments, non-volatile memory 103 may be used to store the look-up table 30 as described above.

In particular, system memory 102 and non-volatile storage 103 may each include: a temporary copy and a permanent copy of instruction 107. The instructions 107 may include: when executed by at least one of the processors 101, cause the electronic device 100 to implement the detection model post-processing methods provided by the embodiments of the present application.

The communication interface 104 may include a transceiver to provide a wired or wireless communication interface for the electronic device 100 to communicate with any other suitable device over one or more networks. In some embodiments, the communication interface 104 may be integrated with other components of the electronic device 100, for example the communication interface 104 may be integrated in the processor 101. In some embodiments, the electronic device 100 may communicate with other devices through the communication interface 104, for example, the electronic device 100 may obtain the neural network model and the look-up table 30 corresponding to the neural network model from other electronic devices through the communication interface 104.

Input/output (I/O) devices 105 may include input devices such as a keyboard, mouse, etc., output devices such as a display, etc., and a user may interact with electronic device 100 through input/output (I/O) devices 105.

System control logic 106 may include any suitable interface controllers to provide any suitable interfaces for the other modules of electronic device 100. For example, in some embodiments, system control logic 106 may include one or more memory controllers to provide an interface to system memory 102 and non-volatile memory 103.

In some embodiments, at least one of the processors 101 may be packaged together with logic for one or more controllers of the System control logic 106 to form a System In Package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same Chip with logic for one or more controllers of the System control logic 106 to form a System-on-Chip (SoC).

It is understood that the configuration of electronic device 100 shown in fig. 7 is merely an example, and in other embodiments, electronic device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of these implementations. Embodiments of the application may be implemented as computer programs or program code executing on programmable systems comprising at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.

Program code may be applied to input instructions to perform the functions described herein and generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this Application, a processing system includes any system having a Processor such as, for example, a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor.

The program code may be implemented in a high level procedural or object oriented programming language to communicate with a processing system. The program code can also be implemented in assembly or machine language, if desired. Indeed, the mechanisms described in this application are not limited in scope to any particular programming language. In any case, the language may be a compiled or interpreted language.

In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or via other computer readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including, but not limited to, floppy diskettes, optical disks, Read-Only memories (CD-ROMs), magneto-optical disks, Read-Only memories (ROMs), Random Access Memories (RAMs), Erasable Programmable Read-Only memories (EPROMs), Electrically Erasable Programmable Read-Only memories (EEPROMs), magnetic or optical cards, flash Memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared digital signals, etc.) using the Internet to transmit information in an electrical, optical, acoustical or other form of propagated signals. Thus, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

In the drawings, some features of the structures or methods may be shown in a particular arrangement and/or order. However, it is to be understood that such specific arrangement and/or ordering may not be required. Rather, in some embodiments, the features may be arranged in a manner and/or order different from that shown in the illustrative figures. In addition, the inclusion of a structural or methodical feature in a particular figure is not meant to imply that such feature is required in all embodiments, and in some embodiments, may not be included or may be combined with other features.

It should be noted that, in the embodiments of the apparatuses in the present application, each unit/module is a logical unit/module, and physically, one logical unit/module may be one physical unit/module, or may be a part of one physical unit/module, and may also be implemented by a combination of multiple physical units/modules, where the physical implementation manner of the logical unit/module itself is not the most important, and the combination of the functions implemented by the logical unit/module is the key to solve the technical problem provided by the present application. Furthermore, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units/modules which are not so closely related to solve the technical problems presented in the present application, which does not indicate that no other units/modules exist in the above-mentioned device embodiments.

It is noted that in the examples and specification of this patent, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the use of the verb "comprise a" to define an element does not exclude the presence of another, same element in a process, method, article, or apparatus that comprises the element.

While the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A detection model post-processing method is applied to electronic equipment and is characterized by comprising the following steps:

2. The method of claim 1, wherein inputting the bounding box data and the first adjustment data into the first operation portion results in a first operation result, comprising:

determining an input quantization parameter for the region adjustment data;

3. The method of claim 2, wherein the inverse quantizing the first adjustment data based on the first quantization parameter to obtain first inverse quantization result data of a first sub-operation including the first constant division operation factor, comprises:

4. The method of claim 2, wherein the bounding box data comprises first bounding box data and second bounding box data;

5. The method of claim 1, wherein the obtaining the table lookup result corresponding to the table lookup parameter from the table lookup comprises:

6. The method of claim 4, wherein determining the second operation result of the second operation portion based on the linear operation of the bounding box data and the table lookup result comprises:

determining a second quantization parameter of the table look-up result;

7. The method of claim 6, wherein the inverse quantizing the table lookup result based on the second quantization parameter to obtain second inverse quantization result data comprises:

8. A readable medium containing instructions that, when executed by a processor of an electronic device, cause the electronic device to implement the detection model post-processing method of any one of claims 1 to 7.

9. An electronic device, comprising:

a memory to store instructions for execution by one or more processors of an electronic device;

and a processor, which is one of the processors of the electronic device, for executing the instructions to cause the electronic device to implement the detection model post-processing method of any one of claims 1 to 7.

10. A computer program product, characterized in that it comprises instructions for implementing a detection model post-processing method according to any one of claims 1 to 7.