WO2022160229A1

WO2022160229A1 - Apparatus and method for processing candidate boxes by using plurality of cores

Info

Publication number: WO2022160229A1
Application number: PCT/CN2021/074313
Authority: WO
Inventors: 林鑫; 汪昊; 刘虎
Original assignee: 华为技术有限公司
Priority date: 2021-01-29
Filing date: 2021-01-29
Publication date: 2022-08-04
Also published as: CN116762092A

Abstract

Disclosed is an apparatus for processing candidate boxes by using a plurality of cores. The apparatus comprises a plurality of single-core processors. A plurality of first single-core processors from among the plurality of single-core processors are used for executing the following processes in parallel: acquiring a portion of candidate boxes from all candidate boxes of an image to be detected; and acquiring a suppression relationship between each candidate box in the portion of candidate boxes and each candidate box in all the candidate boxes. A second single-core processor from among the plurality of single-core processors is used for acquiring a target candidate box according to the suppression relationship acquired by each first single-core processor and a confidence corresponding to each candidate box, wherein the second single-core processor is any one of the plurality of first single-core processors, or the second single-core processor is different from any one of the plurality of first single-core processors. By means of the solution provided in the present application, the calculation speed of an NMS algorithm can be improved.

Description

Apparatus and method for processing candidate frames using multiple cores

technical field

The present application relates to the technical field of data processing, and in particular, to an apparatus and method for processing candidate frames using multiple cores.

Background technique

Computer vision is an integral part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. The knowledge of the data and information that people need to be photographed. To put it figuratively, it is to install eyes (cameras/camcorders) and brains (algorithms) on the computer to identify, track and measure the target instead of the human eye, so that the computer can perceive the environment. Because perception can be viewed as extracting information from sensory signals, computer vision can also be viewed as the science of how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision is the use of various imaging systems to replace the visual organ to obtain input information, and then the computer replaces the brain to process and interpret the input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously. Object detection is a technique commonly used in the field of computer vision.

Nonmaximum suppression (NMS) is widely used in computer vision algorithms, such as image recognition, edge detection or object detection. The classifier is an important part of the NMS algorithm, which can be used to detect objects in the image, such as whether it is a face, etc. For an object in the image, the classifier will generate multiple candidate frames. The classifier computes a confidence level, also known as a score, for each candidate box. In order to accurately detect or recognize objects, only one optimal candidate frame is reserved for an object, and the content in the optimal candidate frame is used as the recognized or detected object.

Specifically, the candidate frame with the highest classifier score in all candidate frames in the image is selected, the overlapping area between other candidate frames and the candidate frame with the highest classifier score is traversed and calculated, and according to the relationship between the overlapping area and the preset threshold, the frames in the other candidate frames are deleted. Some candidate boxes. The candidate frame with the highest classifier score is selected again from the other candidate frames that have not been deleted, and the traversal calculation of the overlapping area is performed again until all other candidate frames except the optimal candidate frame are deleted. In a more complex image, the number of candidate boxes is very large, which leads to an increase in the traversal calculation amount, which increases the calculation amount of the NMS algorithm, resulting in the problem of long calculation time of the NMS algorithm.

SUMMARY OF THE INVENTION

The embodiments of the present application provide an apparatus and method for processing candidate frames by using multiple cores, which improves the calculation speed of the NMS algorithm and improves the efficiency of obtaining target candidate frames.

To achieve the above purpose, the embodiments of the present application provide the following technical solutions:

A first aspect of the present application provides an apparatus for processing candidate frames using multiple cores, and the apparatus for processing candidate frames using multiple cores may include multiple single-core processors. All candidate frames of the image to be detected can be processed by the device, wherein each candidate frame in all the candidate frames of the image to be detected corresponds to a sequence number, and the sequence numbers of any two candidate frames are different. In a possible implementation manner, the sequence numbers corresponding to each candidate frame are consecutive. In a possible implementation, different sequence numbers may be randomly assigned to each candidate frame. In a possible implementation manner, sequence numbers may be assigned to all candidate frames according to the order of the confidence levels of all candidate frames from high to low. In a possible implementation manner, sequence numbers may be assigned to all candidate frames according to the order of confidence of all candidate frames from low to high. The multiple first single-core processors among the multiple single-core processors are configured to execute the following process in parallel: acquiring some candidate frames from all the candidate frames. In a possible implementation, some candidate frames may be acquired from all candidate spaces according to their respective identification information. The identification information is unique, the identification information of any two first single-core processors is different, and the partial candidate frames obtained by any two first single-core processors are different; each candidate frame in the obtained partial candidate frame is the same as Suppression relationship between each candidate box in all candidate boxes. In a possible implementation, the suppression relationship between each candidate frame and itself may not be obtained. The second single-core processor among the plurality of single-core processors is configured to obtain the final candidate frame according to the suppression relationship obtained by each first single-core processor and the corresponding confidence level of each candidate frame. The second single-core processor is any one of the multiple first single-core processors, or the second single-core processor is different from any one of the multiple first single-core processors.

It can be known from the solution provided in the first aspect that the solution provided by the present application processes the computation without dependencies in parallel by a multi-core processor, so as to speed up the processing progress. Among them, computations without dependencies can be understood as computations independent of the ordering of confidence. In the solution provided in this application, the step of acquiring the inhibition relationship may be performed by a multi-core processing model. Each first single-core processor obtains a part of candidate frames respectively, and calculates the suppression relationship between each candidate frame in the obtained candidate frame and each candidate frame in all candidate frames. Through the multi-core processor, the inhibition relationship between any two candidate frames in all candidate frames can be obtained at one time, and then the calculation related to the confidence level is calculated through the second single-core processing. In the solution provided in this application, according to The obtained inhibition relationship and the corresponding confidence of each candidate frame are used to obtain the final candidate frame. In this application, the final candidate frame is also sometimes referred to as the target candidate frame, both of which represent the remaining candidate frames after all candidate frames are processed by the NMS algorithm.

In a possible implementation manner, each first single-core processor is specifically configured to: acquire the area of each candidate frame in all candidate frames. Obtain the overlap area between each candidate frame in some candidate frames and each candidate frame in all candidate frames. Obtain the overlapping area of each candidate frame in some candidate frames and each candidate frame in all candidate frames. According to the relationship between the overlapping area and the ratio of the overlapping area and the preset threshold, the suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames is obtained. In this embodiment, a process of obtaining the suppression relationship between any two candidate frames is given. In the solution provided in this application, the process of obtaining the suppression relationship between any two candidate frames is processed by multi-core processing. The parallel execution of the processor increases the speed of the calculation. In addition, in some possible implementations, at least one of the above steps is performed in parallel by a multi-core processor, and the remaining steps may be performed by a single-core processor, such as obtaining the area of each candidate frame in all candidate frames, obtaining some candidate frames The overlapping area between each candidate frame in the frame and each candidate frame in all candidate frames, and the overlapping area between each candidate frame in some candidate frames and each candidate frame in all candidate frames, these three The step is obtained by a multi-core processor, and according to the relationship between the overlap area and the ratio of the overlap area and the preset threshold, the suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames can be obtained in the first step. Two single-core processors execute.

In a possible implementation manner, each first single-core processor is specifically configured to acquire, from the sequence of candidate frames, some adjacent candidate frames in order according to their respective identification information. The sequence of candidate frames is obtained by sorting all the candidate frames according to the order of the confidence levels from high to low. In this embodiment, the candidate frames obtained by the multi-core processor are the candidate frames sorted according to the confidence, which is convenient for subsequent calculation.

In a possible implementation manner, each first single-core processor is specifically configured to: obtain from the candidate frame sequence the same number of adjacent partial candidate frames in order according to the respective identification information. In this implementation manner, the number of partial candidate frames obtained by each first single-core processor is the same, for example, the first first single-core processor obtains candidate frames with serial numbers 1-20, and the second The core processor obtains candidate boxes with sequence numbers 21-40, and so on. Through such a design, subsequent calculations are facilitated, and the performance of each first single-core processing can be the same.

In a possible implementation manner, the second single-core processor is further configured to: obtain N bit sequences from each first single-core processor, where N is the number of partial candidate frames, and the N bit sequences use In order to represent the suppression relationship between each candidate frame in the partial candidate frame and each candidate frame in the all candidate frames, each bit sequence in the N bit sequences is used to represent the first candidate frame and the suppression relationship between each candidate frame in all the candidate frames, the first candidate frame is one candidate frame in the partial candidate frame. When the second single-core processor is not identical to any one of the plurality of first single-core processors, each first single-core processor sends N bit sequences to the second single-core processor. In other words, when the second single-core processor is not identical to any one of the plurality of first single-core processors, the second single-core processor receives the sequence of N bits sent by each of the first single-core processors . When the second single-core processor is any one of the plurality of first single-core processors, the second single-core processor obtains N bit sequences from itself, and from each of the other first single-core processors A single-core processor each acquires N bit sequences. In other words, when the second single-core processor is one of the plurality of first single-core processors, the second single-core processor can obtain the N bit sequences generated by itself, and receive every other first single-core processor. A sequence of N bits sent by the core processor. In this embodiment, the suppression relationship between one candidate frame and each candidate frame in all the candidate frames can be represented by a bit sequence, which is beneficial to the subsequent calculation process and also increases the diversity of the scheme.

In a possible implementation, each bit sequence may include M bits, each of the M bits is used to indicate that the first candidate frame is suppressed by the second candidate frame, or to indicate that the first candidate frame is not Suppressed by a second candidate frame, which is one candidate frame among all the candidate frames. M is the number of all candidate frames. In this possible implementation, in order to reduce the amount of calculation, reduce the amount of data that needs to be transmitted, and improve the calculation speed of the algorithm, each of the M positions can use 1 bit to indicate the inhibition relationship, for example, when 1 bit is 0, it means The second candidate frame is suppressed by the first candidate frame, and when 1 bit is 1, it indicates that the second candidate frame is not suppressed by the first candidate frame.

In a possible implementation manner, each first single-core processor is further configured to: perform an initialization operation on each bit sequence according to the obtained sorting of some candidate frames, so that each bit sequence after initialization is The first P bits are used to indicate that the first candidate frame is suppressed by the second candidate frame, so that the last M-P bits of each of the bit sequences after initialization are used to indicate that the first candidate frame is not suppressed by the second candidate frame. , P is the sequence number of the first candidate frame in the sorting of some candidate frames. When obtaining the suppression relationship, each candidate frame does not need to obtain the suppression relationship with itself, and does not need to consider the suppression relationship between candidate frames whose confidence levels are ranked before itself, so when initializing the bit sequence, M bits The first P bits are used to indicate that the first candidate frame is suppressed by the second candidate frame, so that the M-P bits in the M bits are used to indicate that the first candidate frame is not suppressed by the second candidate frame, and P is the first candidate frame. The sequence number of the candidate frame in the sorting of some candidate frames.

In a possible implementation manner, the second single-core processor is specifically configured to: perform multiple processing on the acquired bit sequence to obtain the target candidate frame, wherein any one processing in the multiple processing includes: according to the sequence number The sequence obtains the bit sequence to be processed from the acquired bit sequence, and the sequence number sequence is used to indicate the sequence number of each bit sequence, wherein the sequence number of the bit sequence to be processed is determined according to the sequence number of the second candidate frame. The sequence is used to represent the suppression relationship between the second candidate frame and each candidate frame in all candidate frames, the second candidate frame is a candidate frame in all candidate frames, and the sequence number of each candidate frame in all candidate frames is based on The candidate box sequence is obtained. Obtain an updated global sequence according to the bit sequence to be processed and the global sequence obtained by the previous processing, and the updated global sequence indicates candidate frames that are not suppressed by the second candidate frame and other candidate frames ranked before the second candidate frame , the updated global sequence is used to obtain the target candidate frame. In this embodiment, a specific solution for obtaining the final candidate frame by the second single-core processor is given.

In a possible implementation manner, the second single-core processor is specifically configured to: generate a sequence number sequence according to the sequence of candidate frames, where the sequence of candidate frames is used to indicate the confidence level of the candidate frame. The sequence number sequence is screened multiple times to obtain multiple sequence numbers, and the multiple sequence numbers are used to obtain the final candidate frame, wherein any one of the multiple screenings may include: obtaining the bit sequence of a specific candidate frame according to the sequence number obtained by the previous screening , the bit sequence of the specific candidate frame is used to represent the suppression relationship between the specific candidate frame and the second candidate frame. According to the inhibition relationship between the bit sequence of the specific candidate frame and the global sequence obtained by the last screening, the updated global sequence is obtained. The updated global sequence indicates at least one sequence number, and the candidate frame corresponding to at least one sequence number is not sorted by the specific candidate frame and the sequence number. Other candidate boxes before a specific candidate box are suppressed. In this embodiment, a specific solution for obtaining the final candidate frame by the second single-core processor is given.

In a possible implementation manner, the second single-core processor is specifically configured to: acquire the bit sequence to be processed from the acquired bit sequence according to the sequence number of the acquired bit sequence, and the sequence number of the bit sequence to be processed is based on the sequence number of the bit sequence to be processed. The sequence number of the second candidate frame is determined, and the bit sequence to be processed is used to represent the suppression relationship between the second candidate frame and each candidate frame in all candidate frames, and the second candidate frame is a candidate frame in all candidate frames. , the serial number of each candidate frame in all candidate frames is obtained according to the sequence of candidate frames. According to the bit sequence to be processed and the processed bit sequence, the target candidate frame is obtained. The target candidate frame is a candidate frame that is not suppressed by the second candidate frame and other candidate frames sorted before the second candidate frame. Each processed candidate frame The bit sequence is used to represent the suppression relationship between each other candidate frame ranked before the second candidate frame and each candidate frame in all candidate frames.

In a possible implementation manner, some candidate frames obtained by each first single-core processor constitute all candidate frames.

A second aspect of the present application provides a method for processing a candidate frame using multiple cores, which may include: acquiring a candidate frame of an image to be detected. Part of the candidate frames are respectively obtained from all the candidate frames by using the respective identification information of the multiple first single-core processors. The suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames is acquired by each first single-core processor. The second single-core processor obtains the final candidate frame according to the inhibition relationship obtained by each first single-core processor and the corresponding confidence level of each candidate frame.

In a possible implementation manner, acquiring, by each first single-core processor, the suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames may include: by each candidate frame The first single-core processor obtains the area of each candidate box in all candidate boxes. The overlapping area between each candidate frame in some candidate frames and each candidate frame in all candidate frames is obtained by each first single-core processor. The overlapping area of each candidate frame in some candidate frames and each candidate frame in all candidate frames is obtained by each first single-core processor. The suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames is obtained by each first single-core processor according to the relationship between the overlapping area and the ratio of the overlapping area and the preset threshold.

In a possible implementation manner, the method may further include: sorting all the candidate frames according to the order of the confidence levels corresponding to each candidate frame from high to low, so as to obtain a sequence of candidate frames. Obtaining part of the candidate frames from all the candidate frames by using the respective identification information of the multiple first single-core processors may include: obtaining, from the sequence of candidate frames, the adjacent candidate frames by using the respective identification information of the multiple first single-core processors. Some candidate boxes.

In a possible implementation manner, obtaining, from the candidate frame sequence, some adjacent candidate frames in order by using the respective identification information of the multiple first single-core processors may include: using the respective identification information of the multiple first single-core processors The identification information obtains the same number of adjacent partial candidate frames from the sequence of candidate frames.

In a possible implementation manner, the method may further include: obtaining, through the second single-core processor, N bit sequences from each of the first single-core processors, where N is the number of partial candidate frames, and among the N bit sequences Each bit sequence of is used to represent the suppression relationship between the first candidate frame and the second candidate frame, the first candidate frame is a candidate frame in some candidate frames, and the second candidate frame is each candidate frame in all candidate frames frame.

In a possible implementation, each bit sequence may include M bits, each of the M bits is used to indicate that the first candidate frame is suppressed by the second candidate frame, or to indicate that the first candidate frame is not Suppressed by the second candidate frame, M is the number of all candidate frames.

In a possible implementation manner, the method may further include: performing an initialization operation on each bit sequence by each first single-core processor according to the order of the obtained partial candidate frames, so that the first P in the M bits are initialized. bits are used to indicate that the first candidate frame is suppressed by the second candidate frame, so that the last M-P bits of the M bits are used to indicate that the first candidate frame is not suppressed by the second candidate frame, and P is the first candidate frame in the partial The sequence number in the ordering of candidate boxes.

In a possible implementation manner, obtaining the final candidate frame by the second single-core processor according to the suppression relationship obtained by each first single-core processor and the corresponding confidence level of each candidate frame may include: according to the sequence of candidate frames , to generate a sequence of serial numbers. The sequence number sequence is screened multiple times to obtain multiple sequence numbers, and the multiple sequence numbers are used to obtain the final candidate frame, wherein any one of the multiple screenings may include: obtaining the bit sequence of a specific candidate frame according to the sequence number obtained by the previous screening , the bit sequence of the specific candidate frame is used to represent the suppression relationship between the specific candidate frame and the second candidate frame. According to the inhibition relationship between the bit sequence of the specific candidate frame and the global sequence obtained by the last screening, the updated global sequence is obtained. The updated global sequence indicates at least one sequence number, and the candidate frame corresponding to at least one sequence number is not sorted by the specific candidate frame and the sequence number. Other candidate boxes before a specific candidate box are suppressed.

In a possible implementation manner, the target candidate frame is obtained by the second single-core processor according to the inhibition relationship obtained by each first single-core processor and the confidence level corresponding to each candidate frame in all the candidate frames, The method includes: performing multiple processing on the acquired bit sequence by the second single-core processor to obtain the target candidate frame, wherein any one processing in the multiple processing includes: acquiring the to-be-processed bit sequence from the acquired bit sequence according to the sequence number sequence Bit sequence, the sequence number sequence is used to indicate the sequence number of each bit sequence, where the sequence number of the bit sequence to be processed is determined according to the sequence number of the second candidate frame, and the bit sequence to be processed is used to indicate the second candidate frame and all candidate frames. The suppression relationship between each candidate frame in the frame, the second candidate frame is a candidate frame in all candidate frames, and the sequence number of each candidate frame in all candidate frames is obtained according to the sequence of candidate frames. Obtain an updated global sequence according to the bit sequence to be processed and the global sequence obtained by the previous processing, and the updated global sequence indicates candidate frames that are not suppressed by the second candidate frame and other candidate frames ranked before the second candidate frame , the updated global sequence is used to obtain the target candidate frame.

A third aspect of the present application provides a neural network device. The neural network device may include a device for processing candidate frames using multiple cores. The device for processing candidate frames using multiple cores is the device for processing candidate frames using multiple cores described in the first aspect.

A fourth aspect of the present application provides an apparatus for processing candidate boxes using multiple cores, which may include: a memory for storing computer-readable instructions. It may also include a processor coupled to the memory for executing computer readable instructions in the memory to perform the method as described in the second aspect.

A fifth aspect of the present application provides a chip system, the chip system may include a processor and a communication interface, the processor obtains program instructions through the communication interface, and the method described in the second aspect is implemented when the program instructions are executed by the processor.

A sixth aspect of the present application provides a computer-readable storage medium, which may include a program that, when executed by a processing unit, executes the method described in the second aspect.

A seventh aspect of the present application provides a computer program product, which, when the computer program product is run on a computer, causes the computer to perform the method of the second aspect.

The beneficial effects of the solutions described in the second to seventh aspects can be understood with reference to the beneficial effects of the solutions described in the first aspect, and details are not repeated here.

The solution provided in this application separates the parallel computing part and the data-dependent part of the calculation in the NMS algorithm. The inhibition relationship between candidate boxes is obtained through a multi-core processor, and the parallel execution capability of the multi-core processor is used to increase the flexibility of calculation and improve the calculation speed of the NMS algorithm. On the premise of no need to add special processing instructions, the suppression relationship between any candidate frame in all candidate frames and each candidate frame in all candidate frames can be obtained, and then the bit sequence matrix can be obtained. Among them, each bit sequence in the bit sequence matrix can indicate the suppression relationship between the two candidate boxes through 1 bit, and the amount of data is small, which effectively reduces the amount of data transmitted by the multi-core processor to the second single-core processing, and shortens the calculation time. The second single-core processor may extract the bit sequence corresponding to the unsuppressed candidate frame from the bit sequence matrix, and delete the suppressed bit sequence. For example, the vreduce instruction may be used to implement this process. The solution provided in this application can speed up the calculation process of the NMS algorithm and quickly obtain the final candidate frame.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic diagram of the architecture of an RPN;

FIG. 2a is a schematic structural diagram of an apparatus for processing candidate frames using multiple cores provided by an embodiment of the present application;

FIG. 2b is a schematic structural diagram of an apparatus for processing candidate frames using multiple cores provided by an embodiment of the present application;

3 is a schematic diagram of each single-core processor in a multi-core processor in an embodiment of the present application acquiring some candidate frames;

Fig. 4 is the schematic flow chart of simultaneously obtaining the area of a plurality of candidate frames by SIMD instruction;

5 is a schematic flowchart of simultaneously obtaining the overlap area between each candidate frame in the first candidate frame and all candidate frames through SIMD instructions;

FIG. 6 is a schematic diagram of the suppression relationship between the first candidate frame and the second candidate frame represented by a bit sequence in an embodiment of the present application;

7a is a schematic diagram of the second single-core processor screening serial numbers in an embodiment of the present application;

7b is a schematic diagram of the second single-core processor screening serial numbers in an embodiment of the present application;

7c is a schematic diagram of the second single-core processor screening serial numbers in an embodiment of the present application;

8 is a schematic diagram of the second single-core processor screening serial numbers in an embodiment of the present application;

9 is a schematic flowchart of a method for processing candidate frames using multiple cores provided by an embodiment of the present application;

10 is a schematic flowchart of a method for processing candidate frames using multiple cores provided by an embodiment of the present application;

FIG. 11 is a schematic flowchart of a method for processing candidate frames using multiple cores provided by an embodiment of the present application;

12 is a schematic flowchart of the first single-core processor acquiring the suppression relationship between candidate frames in an embodiment of the present application;

FIG. 13 is a schematic flowchart of obtaining a final candidate frame by a second single-core processor in an embodiment of the present application;

FIG. 14 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Those of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The present application provides an apparatus and method for processing candidate frames using multiple cores. The computing speed of the NMS algorithm can be improved by the device for processing candidate frames using multiple cores provided in this application.

Since this application involves a large amount of NMS-related knowledge, in order to better understand the solution provided by this application, NMS and related knowledge are first introduced below.

The basic idea of the NMS algorithm is to search for local maxima and suppress elements that are not maxima. NMS is widely used in video tracking (viedo tracking) and object recognition (object recognition) and other fields. For example, it is widely used in edge detection, face detection, target detection and other scenarios. Taking the task of target detection as an example to illustrate, the key to target detection is to accurately locate the target of interest from the scene and correctly determine the type of the target. Object detection systems usually employ two stages to locate and identify objects of interest, namely the candidate region stage and the region detection stage. The candidate region stage aims to find hundreds or thousands of candidate boxes from the possible positions and scales of the target, so that the target is all contained in these candidate boxes. The region detection stage further identifies and locates the potential targets in these candidate frames, so as to accurately determine the category of the target. With the advent of deep learning, most object detection systems are based on deep neural networks. At present, the common model in the candidate region stage is the region proposal network (RPN). As shown in FIG. 1 , it is a schematic diagram of the architecture of an RPN. The input of RPN200 is a feature map (future map). The feature map can be obtained by extracting the features of the image to be processed through a convolutional neural network (CNN). A convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional/pooling layer 120 . The convolutional/pooling layer 120 may include layers as examples 121-126, in one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, and layer 124 is a pooling layer , 125 is a convolution layer, 126 is a pooling layer; in another implementation manner, 121 and 122 are a convolutional layer, 123 is a pooling layer, 124 and 125 are a convolutional layer, and 126 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or it can be used as the input of another convolutional layer to continue the convolution operation. Taking the convolution layer 121 as an example, the convolution layer 121 may include many convolution operators, which are also called kernels, and their role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator can be essentially a weight matrix. This weight matrix is usually pre-defined. In the process of convolving an image, the weight matrix is usually pixel by pixel along the horizontal direction on the input image ( Or two pixels after two pixels...depending on the value of stride), which completes the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will be extended to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension of the convolutional output, but in most cases a single weight matrix is not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to extract unwanted noise in the image. Perform fuzzification... The dimensions of the multiple weight matrices are the same, and the dimension of the feature maps extracted from the weight matrices with the same dimensions are also the same, and then the multiple extracted feature maps with the same dimensions are combined to form the output of the convolution operation . The weight values in these weight matrices need to be obtained through a lot of training in practical applications, and each weight matrix formed by the weight values obtained by training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions. When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features; with the convolutional neural network As the depth of the network 100 deepens, the features extracted by the later convolutional layers (eg 126) become more and more complex, such as features such as high-level semantics. Features with higher semantics are more suitable for the problem to be solved. Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer, that is, each layer 121-126 as shown in 120 in Figure 1, which can be a convolutional layer followed by a layer The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. Taking image data as an example, in the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a max pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the average value of the pixel values in the image within a certain range. The max pooling operator can take the pixel with the largest value within a specific range as the result of max pooling. Also, just as the size of the weight matrix used in the convolutional layer should be related to the size of the image, the operators in the pooling layer should also be related to the size of the image. The size of the output image after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer. In addition to the convolutional layer/pooling layer 120, the convolutional neural network 100 may also include other structures, such as a hidden layer, an output layer, etc., because it has nothing to do with this solution, and will not be introduced. After the convolutional layer/pooling layer 120 obtains the feature map, the feature map is input into the RPN200. The RPN200 network uses a sliding window to capture the features of each position of the image on the feature map, and maps the features of each position to k anchor windows of different scales and aspect ratios, where k is generally a positive integer greater than 1, such as k is 3. When generating a candidate frame, the features of each position are input into an intermediate layer (eg, regression network, fully connected network, etc.), and each anchor window is determined to include or exclude the object to be detected. This paper also refers to the middle layer as a classifier. An anchor window including the object to be detected can be considered as a candidate frame, and the candidate frame is also scored, and a candidate frame with a higher score indicates a higher probability of including the object to be detected. In a possible implementation manner, the RPN network may further include other processing steps, such as performing compression (reshape) processing on the acquired feature map. This embodiment of the present application does not limit the RPN network may include more processing steps.

For the same object to be detected, a large number of candidate frames may be generated, and these candidate frames may overlap with each other. At this time, it is necessary to use NMS to find the best candidate frame and eliminate redundant candidate frames. In the implementation process of traditional NMS, a large number of obtained candidate frames, also known as bounding boxes, are serially processed, and the candidate frame with the highest confidence is selected in each round, and then all remaining candidate frames are concerned with the selected candidate frame. The candidate boxes for the overlap ratio (the overlap ratio of the area), which will be suppressed in this round. The candidate frame selected in this round will be deleted from the candidate frame list and will not appear in the next round. Then start the next round, repeat the above process, select the candidate frame with the highest confidence, and suppress the candidate frame with high overlap rate. For example, it is assumed that multiple candidate frames are sorted according to the degree of confidence from high to low, and candidate frame 1, candidate frame 2, candidate frame 3, candidate frame 4, and candidate frame 5 are obtained. Among them, candidate frame 1 has the highest confidence, and focus on the remaining candidate frame 2, candidate frame 3, candidate frame 4, candidate frame 5 and the overlap rate of candidate frame 1. It is assumed that candidate frame 4 and candidate frame 1 have a high overlap rate ( The high overlap rate can be understood as the overlap rate exceeding the preset threshold), then the candidate frame 4 is suppressed by the candidate frame 1 . In this round, candidate frame 1 is selected as the first final candidate frame, and candidate frame 4 is determined to be suppressed. In the next round, select the candidate frame with the highest confidence from the candidate frame 2, the candidate frame 3 and the candidate frame 5, that is, the candidate frame 2, pay attention to the overlap rate of the remaining candidate frame 3, candidate frame 5 and candidate frame 2, Assuming that candidate frame 3 and candidate frame 2 have a high overlap rate, candidate frame 3 is suppressed by candidate frame 2. In this round, candidate frame 2 is selected as the second final candidate frame, and at the same time, it is determined that candidate frame 4 is suppressed. In the next round, only candidate frame 5 is left, then candidate frame 5 is not suppressed by any candidate frame, and candidate frame 5 is the third final candidate frame. It can be seen from the above process that the candidate frame selected in each round depends on the selection result of the previous round. For example, when selecting the second final candidate frame, it depends on the result of the first round selection, and when selecting the second final candidate frame, it depends on Based on the results of the first round of selection and the second round of selection results, this brings great difficulty to the parallel optimization of the NMS algorithm.

In order to solve the above technical problems, the embodiment of the present application provides an apparatus for processing candidate frames using multiple cores, which splits the calculation process of the NMS algorithm, and executes the calculations without dependencies in parallel, so as to improve the calculation speed of the NMS algorithm. This will be described in detail below.

An apparatus for processing candidate frames by using multiple cores provided in this embodiment of the present application may include multiple first single-core processors 2021 and second single-core processors 203 . The second single-core processor 203 is any one of the multiple first single-core processors 2021 , or the second single-core processor 203 is different from any one of the multiple first single-core processors 2021 . Referring to FIG. 2a , it is a schematic structural diagram of an apparatus for processing candidate frames using multiple cores provided by an embodiment of the present application. As shown in FIG. 2a, when the second single-core processor 203 is different from any one of the multiple first single-core processors 2021, the multiple first single-core processors 2021 can be regarded as a multi-core processor as a whole 202. Referring to FIG. 2b, it is a schematic structural diagram of another apparatus for processing candidate frames by using multiple cores provided by an embodiment of the present application. As shown in FIG. 2b, the apparatus includes a plurality of first single-core processors. It can be considered that the structure shown in FIG. 2b is based on FIG. 2a, and the second single-core processor 203 is regarded as a plurality of first single-core processors. Any of the core processors 2021.

Each first single-core processor 2021 is configured to obtain partial candidate frames from all candidate frames of the image to be detected, and obtain the difference between each candidate frame in the partial candidate frame and each candidate frame in all candidate frames inhibition relationship. How to obtain the candidate frame of the image to be detected can be understood with reference to the process of how to obtain the candidate frame in FIG. 1 , and details are not repeated here. In a possible implementation manner, each first single-core processor 2021 may acquire some candidate frames from all candidate frames of the image to be detected according to the respective identification information. In a possible implementation manner, some candidate frames obtained by each first single-core processor 2021 constitute all candidate frames. It should be noted that, in this application, an apparatus for processing candidate frames by using multiple cores may also be referred to as a data processing apparatus, and the two have the same meaning.

Wherein, each candidate frame in all the candidate frames corresponds to a sequence number, and the sequence numbers of any two candidate frames are different. In a possible implementation manner, the sequence numbers corresponding to each candidate frame are consecutive. In a possible implementation, different sequence numbers may be randomly assigned to each candidate frame. In a possible implementation manner, sequence numbers may be assigned to all candidate frames according to the order of the confidence levels of all candidate frames from high to low. In a possible implementation manner, sequence numbers may be assigned to all candidate frames according to the order of confidence of all candidate frames from low to high. For example, all candidate frames include candidate frame A, candidate frame B, candidate frame C, candidate frame D and candidate frame E, sort the candidate frames according to the confidence of each candidate frame from high to low, and obtain the confidence of candidate frame E The confidence degree of candidate frame C is the second, the confidence degree of candidate frame A is the next, the confidence degree of candidate frame D is lower than that of candidate frame A, and the confidence degree of candidate frame B is the lowest. Then according to the order of confidence of all candidate frames from high to low, the candidate frame E is assigned the sequence number 1, the candidate frame C is assigned the sequence number 2, the candidate frame A is assigned the sequence number 3, the candidate frame D is assigned the sequence number 4, and the candidate frame B is assigned Serial number 5.

After the apparatus for processing candidate frames by using multiple cores obtains all the candidate frames, all the candidate frames may be sent to each of the first single-core processors 2021 in the multi-core processor 202 . In a possible implementation manner, all candidate frames sorted according to the confidence may be sent to each first single-core processor 2021 in the multi-core processor 202 . In a possible implementation manner, all candidate boxes that are not sorted according to the confidence may also be sent to each first single-core processor 2021 in the multi-core processor 202 .

After each first single-core processor 2021 obtains all the candidate frames, it selects some candidate frames from all the candidate frames as candidate frames to be processed according to the respective identification information, and some candidate frames obtained by any two first single-core processors 2021 The boxes are not the same. The identification information is unique, the identification information of any two first single-core processors 2021 is different, and the partial candidate frames obtained by any two first single-core processors 2021 are different.

In a possible implementation manner, the number of partial candidate frames acquired by each first single-core processor 2021 may be the same. For example, assuming that there are N candidate frames in total and there are 5 first single-core processors 2021 in total, the number of partial candidate frames obtained by each single-core processor is N/5. For example, assuming that N is the first single-core processor 2021, the first first single-core processor 2021 can obtain candidate frames corresponding to serial numbers 1-20 from all candidate frames according to the preset identification information, and the second The first single-core processor 2021 obtains candidate frames corresponding to serial numbers 21-40 from all candidate frames according to the preset identification information, and the third first single-core processor 2021 obtains from all candidate frames according to the preset identification information The candidate frame corresponding to the serial number 41-60, the fourth first single-core processor 2021 obtains the candidate frame corresponding to the serial number 61-80 from all the candidate frames according to the preset identification information, and the fifth first single-core processor 2021 The candidate frames corresponding to the serial numbers 81-100 are obtained from all the candidate frames according to the preset identification information. In this example, each of the first single-core processors 2021 obtains some candidate frames from all the candidate frames according to the respective identification information, and the number of the obtained partial candidate frames is the same. When each first single-core processor 2021 obtains all candidate frames sorted by confidence, in the above example, the first single-core processor 2021 obtains the top 20 candidate frames in the confidence ranking, and the second The single-core processor 203 obtains the candidate boxes from the 21st to the 40th in the confidence order, the third single-core processor obtains the candidate boxes from the 41st to the 60th in the confidence order, and the fourth single-core processor obtains the The candidate boxes from the 61st to the 80th in the confidence order are obtained by the fifth single-core processor, and the candidate boxes from the 81st to the first single-core processor 2021 in the confidence order are obtained.

As shown in FIG. 3 , in the embodiment of the present application, each single-core processor in the multi-core processor 202 obtains a schematic diagram of some candidate frames. In such an embodiment, the sorted candidate boxes are used as input to the apparatus for processing candidate boxes using multiple cores. It is assumed that there are 2560 sequence numbers in total, and each sequence number corresponds to a candidate frame. The smaller the value of the sequence number, the higher the confidence of the candidate frame. The sorted candidate frames are copied to the multi-core processor 202 , and specifically to each of the first single-core processors 2021 in the multi-core processor 202 . As shown in FIG. 3 , each first single-core processor 2021 selects its own partial candidate frame from the 2560 candidate frames according to its own identification information. In the example shown in Figure 3, the input sorted candidate frames are evenly divided into R regional tiles (tiling), and each regional tile includes the same number of candidate boxes, as indicated by the black boxes in Figure 3 The region obtained by each first single-core processor 2021 is divided into blocks, or understood as a partial candidate frame obtained by each first single-core processor 2021 .

In a possible implementation manner, the number of partial candidate frames acquired by each first single-core processor 2021 may be different. For example, there are a total of N candidate boxes, N is the first single-core processor 2021, there are a total of five first single-core processors 2021, and the first first single-core processor 2021 can be selected from all the first single-core processors 2021 according to the preset identification information The candidate frame obtains the candidate frame corresponding to the serial number 1-20, the second first single-core processor 2021 obtains the candidate frame corresponding to the serial number 21-60 from all the candidate frames according to the preset identification information, and the third first single-core processor 2021 obtains the candidate frame corresponding to the serial number 21-60. The core processor 2021 obtains the candidate frames corresponding to the sequence numbers 61-70 from all the candidate frames according to the preset identification information, and the fourth first single-core processor 2021 obtains the sequence numbers 71-70 from all the candidate frames according to the preset identification information. The candidate frame corresponding to 85, the fifth first single-core processor 2021 obtains the sequence numbers 86-100 from all the candidate frames according to the preset identification information.

After each first single-core processor 2021 acquires some candidate frames from all candidate frames according to the respective identification information, it acquires the suppression relationship between each candidate frame in the partial candidate frame and each candidate frame in all candidate frames . This is a process of parallel processing. After each first single-core processor 2021 obtains the suppression relationship between each candidate frame in its own partial candidate frame and each candidate frame in all candidate frames, After the results output by a single-core processor 2021 are summed up, the suppression relationship between each candidate frame in all candidate frames and all other candidate frames can be obtained. The number of partial candidate frames obtained by each of the above-mentioned first single-core processors 2021 may be the same. The description continues. The first single-core processor obtains a candidate frame with a sequence number of 1 and each candidate frame in all N candidate frames. At the same time as the suppression relationship between the boxes, the second single-core processor obtains the suppression relationship between the candidate box with serial number 21 and each candidate box in all N candidate boxes, and the third single-core processor obtains The suppression relationship between the candidate frame with serial number 41 and each candidate frame in all N candidate frames, while the fourth single-core processor obtains the candidate frame with serial number 61 and each candidate frame in all N candidate frames At the same time, the fifth single-core processor obtains the suppression relationship between the candidate frame with serial number 81 and each candidate frame in all N candidate frames. When the first single-core processor obtains the suppression relationship between each candidate frame in the candidate frames corresponding to the sequence numbers 1 to 20 and each candidate frame in all the candidate frames, the second single-core processor obtains the sequence number The suppression relationship between each candidate frame in the candidate frames corresponding to 21 to 40 and each candidate frame in all candidate frames, while the third single-core processor obtains each candidate frame in the candidate frames corresponding to serial numbers 41 to 60. The suppression relationship between the frame and each candidate frame in all candidate frames, and the fourth single-core processor obtains the information of each candidate frame in the candidate frames corresponding to serial numbers 61 to 80 and each candidate frame in all candidate frames. At the same time, the fifth single-core processor obtains the suppression relationship between each candidate frame in the candidate frame corresponding to the serial number 81 to the first single-core processor 2021 and each candidate frame in all the candidate frames.

Exemplarily, a method for calculating the suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames is given below.

Get the area of each candidate box in all candidate boxes. In a possible implementation manner, each first single-core processor 2021 acquires the area of each candidate frame in all candidate frames. Since the calculation process of the area of each candidate frame is independent, a single instruction stream with multiple data streams (single instruction multiple data) can be used, so that each first single-core processor 2021 in the multi-core processor 202 can simultaneously calculate multiple candidates area of the box. In another possible implementation, each first single-core processor 2021 calculates the area of each candidate frame in the partially obtained candidate frame, and according to the calculation result of each first single-core processor 2021, then The area of each candidate frame in all candidate frames can be obtained. Each first single-core processor 2021 may acquire the area of the candidate box calculated by the other first single-core processors 2021 . For example, in the above example, the first single-core processor 2021 may obtain the area of each candidate box in the candidate boxes whose serial numbers are from 21 to the first single-core processor 2021 . In a possible implementation, a single-core processor may acquire the area of each candidate frame in all candidate frames, and send the acquired area of each candidate frame in all candidate frames to another first single-core processor for processing device 2021.

Each candidate frame corresponds to a coordinate, which is used to represent the coordinate information of the candidate frame on the feature map, and the area of the candidate frame can be obtained according to the coordinates of a candidate frame. For example, a candidate frame can usually be represented by the coordinates of the upper left corner and the lower right corner of the candidate frame. Assuming that the coordinates of the upper left corner are (x1, y1) and the coordinates of the lower right corner are (x2, y2), the area of the candidate frame can be Expressed as (x2-x1+1)*(y2-y1+1). As shown in FIG. 4 , it is a schematic flowchart of simultaneously obtaining the areas of multiple candidate boxes through SIMD instructions. In the example of Figure 4, there are 16 candidate frames in total, the abscissa of the upper left corner of each candidate frame is x1, the ordinate of the upper left corner of each candidate frame is y1, and the abscissa of the lower right corner of each candidate frame is x2 , and the ordinate of the lower right corner of each candidate box is y2. According to the formula (x2-x1+1)*(y2-y1+1), the area of each candidate frame in the 16 candidate frames can be obtained simultaneously.

After obtaining the area of each candidate frame in all candidate frames, in order to calculate the suppression relationship between any two candidate frames, it is also necessary to calculate the overlap ratio between the two candidate frames. Specifically, the overlapping area between each candidate frame in some candidate frames and each candidate frame in all candidate frames can be obtained. Obtain the overlapping area of each candidate frame in some candidate frames and each candidate frame in all candidate frames. According to the relationship between the overlapping area and the ratio of the overlapping area and the preset threshold, the suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames is obtained. A detailed description will be given below.

For each first single-core processor 2021 , each first single-core processor 2021 calculates the overlapping area between each obtained candidate frame and all the candidate frames for each of the obtained candidate frames. Assume that the overlapping area between the first candidate frame and each candidate frame in all other candidate frames is currently calculated. Assuming that the coordinates of the upper left corner of the first candidate frame are (sel-x1, sel-y1), the coordinates of the lower right corner are (sel-x2, sel-y2), and the coordinates of the upper left corner of a candidate frame in all candidate frames are (x1, y1), the coordinates of the lower right corner are (x2, y2). Then the overlapping area between the two candidate boxes can be expressed as [min(sel-x2, x2)-max(sel-x1, x1)+1]*[min(sel-y2, y2)-max(sel- y1, y1)+1]. As shown in FIG. 5 , it is a schematic flowchart of simultaneously obtaining the overlapping area between the first candidate frame and each candidate frame in all candidate frames through SIMD instructions. In the example of Fig. 5, there are 16 candidate frames in total. The coordinates of the upper left corner of the currently selected candidate frame (the first candidate frame) are (sel-x1, sel-y1), and the coordinates of the lower right corner are (sel-x1, sel-y1). x2, sel-y2), the coordinates of the upper left corner of each candidate frame in all candidate frames are (x1, y1), and the coordinates of the lower right corner are (x2, y2). According to the formula [min(sel-x2, x2)-max(sel-x1, x1)+1]*[min(sel-y2, y2)-max(sel-y1, y1)+1], it can be obtained at the same time The overlap area between the selected candidate box and each candidate box in all candidate boxes.

For each first single-core processor 2021 , each first single-core processor 2021 calculates the overlapping area between each obtained candidate frame and all the candidate frames for each obtained candidate frame. It has been described above that the area of each candidate frame is obtained according to the coordinates of the candidate frame, and the overlapping area between the two candidate frames can be obtained by superimposing the areas of any two candidate frames. Specifically, for example, it is assumed that the coordinates of the upper left corner of the first candidate frame are (sel-x1, sel-y1), the coordinates of the lower right corner are (sel-x2, sel-y2), and the upper left corner of a candidate frame in all candidate frames is The coordinates of the corner are (x1, y1), and the coordinates of the lower right corner are (x2, y2). Then the overlapping area between the two candidate boxes can be expressed as [(x2-x1+1)*(y2-y1+1)]+[(sel-x2-sel-x1+1)*(sel-y2- sel-y1+1)]. In a possible implementation manner, the overlapping area between the first candidate frame and each candidate frame in all candidate frames can be obtained simultaneously through SIMD instructions, for example, according to the formula [(x2-x1+1)*(y2-y1 +1)]+[(sel-x2-sel-x1+1)*(sel-y2-sel-y1+1)] simultaneously obtains the difference between the first candidate frame and each candidate frame in all candidate frames Overlay area.

According to the relationship between the ratio of the overlapping area between each candidate frame and other candidate frames to the overlapping area between the candidate frame and other candidate frames and the preset threshold, obtain the relationship between each candidate frame and other candidate frames. inhibit relationship. If the ratio is less than the threshold, it means that the overlap ratio of the two candidate frames is small, which means that the inhibition relationship between the two candidate frames is not suppressed. If the ratio is greater than the threshold, it means that the overlap rate of the two candidate frames is large, which means that the inhibition relationship between the two candidate frames is suppressed. The candidate box is suppressed.

In a possible implementation, according to the sequence number of each candidate frame in some candidate frames, each candidate frame can be regarded as the selected candidate frame in turn, and the selected candidate frame and each of the all candidate frames can be obtained. Suppression relationship between candidate boxes. In a possible implementation, multiple candidate frames can also be selected as the selected candidate frames at the same time, and the difference between each candidate frame in the multiple selected candidate frames and each candidate frame in all the candidate frames is obtained at one time. inhibitory relationship between.

In a possible implementation, the suppression relationship between the selected candidate frame and each candidate frame in all candidate frames can be represented by a bit sequence. For example, each bit sequence is used to represent the suppression relationship between the first candidate frame and the second candidate frame. The first candidate frame is a candidate frame in some candidate frames, and the second candidate frame is each candidate frame in all candidate frames. frame. Each candidate frame corresponds to a bit sequence, and each bit sequence includes M positions, where M is the number of all candidate frames. Assuming that the selected candidate frame is the first candidate frame, the bit sequence corresponding to the first candidate frame is the first bit sequence, the bit sequence includes M positions, and each position in the M positions is used to represent the global candidate frame The suppression relationship between one candidate box and the first candidate box. In a possible implementation manner, the suppression relationship between the first candidate frame and each candidate frame may be determined according to the sequence numbers of the candidate frames and sequentially according to the sequence numbers. In one possible implementation, each of the M positions may indicate the suppression relationship through a plurality of bits. In a possible implementation, in order to reduce the amount of calculation, reduce the amount of data to be transmitted, and improve the calculation speed of the algorithm, each position in the M positions can indicate the suppression relationship by 1 bit, for example, when 1 bit is 0, it indicates the first The second candidate frame is suppressed by the first candidate frame, and when 1 bit is 1, it indicates that the second candidate frame is not suppressed by the first candidate frame. In order to more intuitively show the suppression relationship between the first candidate frame and the second candidate frame represented by the bit sequence, please refer to FIG. 6 , which is the first candidate frame and the second candidate frame represented by the bit sequence in this embodiment of the application. Schematic diagram of the inhibition relationship between. In this embodiment, it is assumed that for each candidate frame, the suppression relationship between the first candidate frame and each candidate frame is determined according to the sequence number of the candidate frame from small to large. As shown in Figure 6, assuming that there are 16 candidate frames in total, the first bit sequence corresponding to the candidate frame with sequence number 1, in the first bit sequence shown in Figure 6, the first position, the third position, the first bit sequence If the 6th position, the 9th position, the 12th position, the 13th position and the 15th position are 1, and the rest of the positions are 0, it can be considered that the candidate frame with sequence number 3 is not suppressed by the candidate frame with sequence number 1, The candidate frame with sequence number 6 is not suppressed by the candidate frame with sequence number 1, the candidate frame with sequence number 9 is not suppressed by the candidate frame with sequence number 1, the candidate frame with sequence number 12 is not suppressed by the candidate frame with sequence number 1, and the sequence number is The candidate frame of 13 is not suppressed by the candidate frame with the sequence number 1, the candidate frame with the sequence number 15 is not suppressed by the candidate frame with the sequence number 1, and the candidate frames with the sequence number are all suppressed by the candidate frame with the sequence number 1, that is, the sequence number is 2. , 4, 5, 7, 8, 10, 11, and 16 candidates are suppressed by the 1 candidate. For another example, the second bit sequence corresponding to the candidate frame with sequence number 2, in the second bit sequence shown in FIG. 6, the first position, the second position, the fourth position to the seventh position, the ninth position The first position and the 14th position are 1, and the remaining positions are 0. It can be considered that the candidate frame with the sequence number 1, the sequence number 4 to 7, the sequence number 9 and the sequence number 14 is not suppressed by the candidate frame with the sequence number 2, and the remaining sequence numbers are not suppressed by the candidate frame with the sequence number 2. The candidate frames of , are suppressed by the candidate frame with sequence number 2, that is, the candidate frame with sequence number 3, sequence number 8, sequence numbers 10 to 13, sequence number 15, and sequence number 16 are suppressed by the candidate frame with sequence number 2.

In a possible implementation manner, M bit sequences may be initially set, and a first number of initial bit sequences may be configured for each first single-core processor 2021 , and the first number is obtained by the first single-core processor 2021 The number of partial candidate boxes. When obtaining the suppression relationship, each candidate frame does not need to obtain the suppression relationship with itself, and does not need to consider the suppression relationship between candidate frames whose confidence levels are ranked before itself, so when initializing the bit sequence, M bits The first P bits are used to indicate that the first candidate frame is suppressed by the second candidate frame, so that the M-P bits in the M bits are used to indicate that the first candidate frame is not suppressed by the second candidate frame, and P is the first candidate frame. The sequence number of the candidate frame in the sorting of some candidate frames. For example, referring to Table 1, it is assumed that there are 15 candidate frames in total, each candidate frame corresponds to a sequence number, the 15 sequence numbers are consecutive, and the smaller the sequence number, the higher the confidence. Then, when initializing the bit sequence with the serial number of 7, for example, when indicated by 1 bit, the first 7 bits are 1, and the last 8 bits are 0.

Table 1:

1

0

In a possible implementation, since each candidate frame does not need to obtain the inhibition relationship with itself when obtaining the suppression relationship, and does not need to consider the suppression relationship between candidate frames whose confidence levels are ranked before itself, it is also possible to After the bit sequence of each candidate frame is acquired, each acquired bit sequence is processed by checking the bit sequence. The check bit makes the first P bits in the M bits all used to indicate that the first candidate frame is suppressed by the second candidate frame, and the last M-P bits in the M bits are used to indicate that the first candidate frame is not suppressed by the second candidate frame. The candidate frame is suppressed, and P is the sequence number of the first candidate frame in the sorting of some candidate frames. For example, referring to Table 1, Table 1 can also be understood as the parity bit sequence of the bit sequence numbered 7, and the parity bit sequence of the bit sequence numbered 7 and the bit sequence numbered 7 are ORed bitwise Calculation is performed to obtain a bit sequence with a sequence number of 7 after verification.

Through the above steps, for each first single-core processor 2021, N bit sequences can be obtained, where N is the number of partial candidate frames obtained by the first single-core processor 2021. For all the first single-core processors 2021, a total of M bit sequences can be obtained.

Each first single-core processor 2021 sends the acquired suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames to the second single-core processor 203 . In a possible implementation manner, each first single-core processor 2021 sends the acquired N bit sequences to the second single-core processor 203 . In a possible implementation manner, each first single-core processor 2021 sends the obtained N bit sequences after checking to the second single-core processor 203 . In a possible implementation manner, each first single-core processor 2021 may send N bit sequences to the storage module, and the storage module sorts the N bit sequences according to the sequence numbers of the candidate boxes, and stores the sorted N bit sequences. A sequence of bits is used as input to the second single-core processor 203 . When the second single-core processor is one of the plurality of first single-core processors, the second single-core processor may acquire the N bit sequences generated by itself, and receive each other The N bit sequences sent by a single-core processor will not be repeated below.

The second single-core processor 203 is configured to obtain the final candidate frame according to the suppression relationship obtained by each first single-core processor 2021 and the corresponding confidence level of each candidate frame.

In a possible implementation, the second single-core processor 203 may first obtain the bit sequence corresponding to the candidate frame with the highest confidence, and may obtain each bit sequence corresponding to the candidate frame with the highest confidence according to the bit sequence corresponding to the candidate frame with the highest confidence. The inhibition relationship between each candidate frame and the candidate frame with the highest confidence is obtained, and the candidate frame that is not inhibited by the candidate frame with the highest confidence is obtained, and the next round of screening process is entered. In the next round of screening process, select the bit sequence corresponding to the candidate frame with the highest confidence, and obtain the candidate frame that is not suppressed by the candidate frame with the highest confidence according to the bit sequence corresponding to the candidate frame with the highest confidence in this round, and enter In the next round of screening process, the above process is repeated until the preset stop condition is met. The stopping condition can be understood as outputting a preset number of final candidate frames, or traversing all candidate frames, or other set stopping conditions. In order to better understand this process, here is an example to illustrate. Continue to refer to FIG. 6 , the candidate frame with sequence number 1 is the candidate frame with the highest confidence in all candidate frames. According to the first bit sequence corresponding to sequence number 1, sequence number 3, sequence number 6, sequence number 9, sequence number 12, sequence number 13 and The candidate frame corresponding to the sequence number 15 is not suppressed by the candidate frame whose sequence number is 1. Then the candidate boxes corresponding to the sequence number 3, the sequence number 6, the sequence number 9, the sequence number 12, the sequence number 13, and the sequence number 15 are subjected to the next round of screening. Among them, the confidence of the candidate frame of sequence number 3 is the highest confidence of the candidate frame of sequence number 3, sequence number 6, sequence number 9, sequence number 12, sequence number 13 and sequence number 15, then according to the bit sequence corresponding to sequence number 3 (for example, the third bit sequence) to obtain the inhibition relationship between the candidate frames of sequence number 3 and sequence number 6, sequence number 9, sequence number 12, sequence number 13, and sequence number 15, and after these two rounds of screening, two final candidate frames are obtained, which are the ones of sequence number 1. The candidate frame and the candidate frame of sequence number 3, and so on, repeat the above-mentioned screening process until the preset stop condition is reached.

In a possible implementation, a sequence number sequence is generated according to the sequence of candidate boxes.

The sequence number sequence is screened multiple times to obtain multiple sequence numbers, and the multiple sequence numbers are used to obtain the final candidate frame, wherein any one screening in the multiple screening includes: obtaining the bit sequence of a specific candidate frame according to the sequence number obtained by the previous screening, The bit sequence of the specific candidate frame is used to represent the inhibition relationship between the specific candidate frame and the second candidate frame; according to the inhibition relationship between the bit sequence of the specific candidate frame and the global sequence obtained from the last screening, the updated global sequence is obtained, and the updated global sequence is updated. The latter global sequence indicates at least one sequence number, and the candidate frame corresponding to at least one sequence number is not suppressed by the specific candidate frame and other candidate frames ranked before the specific candidate frame. In the following, an example is given by taking 1 bit to represent the suppression relationship as an example.

For example, all candidate frames are sorted in the order of confidence from high to low, so that each candidate frame corresponds to a sequence number, then each sequence number in the sequence number sequence is consistent with the sequence number corresponding to each candidate frame. For example, assuming that there are 15 candidate frames in total, after sorting all candidate frames in the order of confidence from high to low, the sequence number of the first candidate frame is 1, then the sequence number 1 in the sequence number sequence is used to obtain the first candidate frame. For a bit sequence of a candidate frame, for example, the sequence number of the seventh candidate frame is 7, the sequence number 7 in the sequence number sequence is used to obtain the bit sequence of the seventh candidate frame.

The second single-core processor 203 initializes the global sequence, the initialized global sequence includes M bits, and each bit is set to 1, which is used to indicate that in the initial state, each candidate frame in all candidate frames is not suppressed , that is, each candidate frame in all candidate frames is reserved. Among the M bits, the first bit to the Mth bit are respectively used to indicate the suppression relationship between the currently selected candidate frame and each candidate frame corresponding to the confidence level from high to low.

Judging from the bit sequence of the candidate frame with sequence number 1, and performing the AND operation on the bit sequence of the candidate frame with sequence number 1 and the global sequence, and using the global sequence after the AND operation as the updated global sequence. 7a, in the example shown in FIG. 7a, a total of 16 candidate frames are included, and the updated global sequence is obtained after the bit sequence of the candidate frame numbered 1 and the global sequence are ANDed. The position of 0 in the updated global sequence indicates that the candidate frame at this position is suppressed by the candidate frame with sequence number 1, and the position of 1 in the updated global sequence indicates that the candidate frame at this position is not suppressed by the candidate frame with sequence number 1 . In the example shown in Figure 7a, the candidate frames corresponding to sequence numbers 2 to 6, sequence number 10, sequence number 12, and sequence number 16 are not suppressed by the candidate frame corresponding to sequence number 1, and candidate frames corresponding to other sequence numbers are suppressed by sequence number 1. Through the global sequence obtained after the first update, the serial numbers 2 to 6, the serial number 10, the serial number 12 and the serial number 16 are obtained. Since the serial numbers are assigned to each candidate frame in the order of confidence from high to low, the higher the serial number, the higher the serial number. small, the higher the confidence. Next, start the judgment from the bit sequence of the candidate frame with sequence number 2, perform AND operation on the bit sequence of the candidate frame with sequence number 2 and the global sequence obtained after the first update, and use the global sequence after the AND operation as the updated global sequence. The global sequence of , specifically the global sequence after the second update. Refer to Figure 7b for understanding. Figure 7b continues to illustrate on the basis of Figure 7a. In the example shown in Figure 7b, that is, the candidate frames corresponding to sequence numbers 4-6 are not suppressed by the candidate frame corresponding to sequence number 2, and other sequence numbers are not suppressed by the candidate frame corresponding to sequence number 2. The corresponding candidate frame is suppressed by the candidate frame corresponding to sequence number 2. Since the suppression relationship between the candidate frame corresponding to sequence number 1 and the candidate frame corresponding to sequence number 2 has been judged, the judgment will not be repeated here. The candidate frame corresponding to sequence number 1 has already been determined. Selected as the first final candidate box. Through the global sequence after the second update, the sequence numbers 4 to 6 are obtained. Then, starting from the bit sequence of the candidate frame of sequence number 4, perform AND operation on the bit sequence corresponding to the candidate frame of sequence number 4 and the global sequence obtained after the second update, and use the global sequence after the AND operation as the updated global sequence. The global sequence, specifically as the global sequence after the third update. According to the global sequence after the third update, the sequence number for the next screening can be obtained, and so on. The process of each cycle will select the sequence number that is reserved after this AND operation and the selected sequence number for determining the final The sequence number of the candidate frame, and then the sequence number to be reserved next time is screened according to the reserved sequence number, and the above-mentioned cyclic screening process is repeated until the stop condition is satisfied. When the stopping condition is satisfied, the final candidate frame is obtained according to the sequence number corresponding to the position of 1 in the global sequence obtained by the final update. 7c for understanding, assuming that the final obtained global sequence is shown in FIG. 7c, then it is determined that the sequence numbers corresponding to the position of 1 in the global sequence are sequence number 1 to sequence number 6, sequence number 10, sequence number 12 and sequence number 15, then determine the sequence number The candidate frames corresponding to 1 to 6, the candidate frame corresponding to 10, the candidate frame corresponding to 12, and the candidate frame corresponding to 15 are the final candidate frames. If the sequence number assigned to each candidate box is in the order of confidence from high to low, the confidence level ranks the 1st to 6th candidate boxes, the 10th candidate box, the 12th and the 15th candidate box. The candidate frame is the final candidate frame.

In a possible implementation manner, the output of the second single-core processor 203 may be the coordinates of the final candidate frame, such as outputting the coordinates of the upper left corner and the lower right corner of each final candidate frame.

In the process that the second single-core processor 203 performs multiple screenings on the sequence number sequence, actually each screening only needs to acquire the first few sequence numbers, or the first one sequence number. For example, in the above example, the global sequence obtained after the first update actually only needs to obtain the sequence number 2, and then the next screening can be performed without the need to obtain the sequence numbers 3-6, sequence numbers 10, sequence numbers 12 and sequence numbers 16. ; Through the updated global sequence after the second update, in fact, only the sequence number 4 needs to be obtained, and the next screening can be performed without the need to obtain the sequence number 5 and the sequence number 6. Therefore, in a possible implementation, an early-stop indicator can be designed in an apparatus utilizing multiple cores to process candidate boxes. For example, the indicator is used to indicate that only a preset number of sequence numbers are acquired each time, such as acquiring one or three, so as to reduce redundant computation. Referring to FIG. 8 , when the indicator instructs to acquire three sequence numbers, the device using the multi-core processing candidate frame acquires the sequence numbers of the first three positions of 1 for the updated global sequence, and then stops acquiring the sequence numbers of other positions of 1.

In some possible scenarios, the amount of data input to the candidate frame of the device using multi-core processing candidate frames is too large, or the number of the first single-core processors 2021 included in the multi-core processor 202 is too small, etc., which may cause the first The amount of data that needs to be processed by a single-core processor 2021 , or the amount of data acquired by the first single-core processor 2021 exceeds the maximum storage space of the first single-core processor 2021 . For these scenarios, in a possible implementation, all candidate frames may be input into the apparatus for processing candidate frames by using multiple cores. For example, all candidate frames are divided into multiple groups of candidate frames, such as a first group of candidate frames and a second group of candidate frames, and each first single-core processor 2021 obtains from the first group of candidate frames according to their respective identification information For some candidate frames, each first single-core processor 2021 obtains the suppression relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames, and the second single-core processor 203 obtains the suppression relationship between each candidate frame in the partial candidate frames and each candidate frame in all candidate frames. The suppression relationship obtained by the core processor 2021 and the confidence level corresponding to each candidate frame in the first group of candidate frames are used to obtain the final candidate frame in the first group of candidate frames. Each first single-core processor 2021 then obtains some candidate frames from the second group of candidate frames according to their respective identification information, and each first single-core processor 2021 obtains each candidate frame in the partial candidate frame and all candidate frames. For the suppression relationship between each candidate frame, the second single-core processor 203 obtains the second group of candidates according to the suppression relationship obtained by each first single-core processor 2021 and the corresponding confidence level of each candidate frame in the second group of candidate frames The final candidate box in the box. Referring to FIG. 9 for illustration, the multi-core processor 202 processes the first group of candidate frames, and sends the processed data to the second single-core processor 203. For this part of the process, refer to the solutions described in FIGS. 1 to 8 . For understanding, it will not be repeated here. The multi-core processor 202 processes the second group of candidate frames, and sends the processed data to the second single-core processor 203. So far, the second single-core processor 203 obtains the last update obtained for the first group of candidate frames. The latter global sequence (hereinafter referred to as the first set of global sequences). The first set of global sequences and the bit sequences corresponding to the candidate frames with the highest confidence in the second set of candidate frames are ANDed to obtain the first updated global sequence for the second set of candidate frames, and the subsequent cyclic calculation process For understanding, refer to the solutions described in FIG. 1 to FIG. 8 , and details are not repeated here. That is, the global sequence in the initialization state of the second group of candidate boxes is the first group of global sequences. All final candidate boxes can be obtained according to the last updated global sequence of the second group of candidate boxes.

In a possible implementation manner, if the number of final candidate frames obtained according to the first group of candidate frames already meets the preset requirement, the final candidate frame may not be obtained according to the second group of candidate frames. In addition, it should be noted that the above-mentioned two groups of candidate frames are only exemplary, and in fact, all candidate frames may be divided into more groups of candidate frames.

In a possible implementation, all candidate frames may also be input into the device using multi-core processing candidate frames at one time, and the total number of partial candidate frames obtained by each first single-core processor 2021 is less than the total number of all candidate frames. , for example, the total number of partial candidate frames acquired by each first single-core processor 2021 is half of the total number of all candidate frames. How each first single-core processor 2021 processes some of the obtained candidate frames, and how the second single-core processor 203 outputs the final candidate frame according to the data sent by each first single-core processor 2021 The specific introduction has been made above, and will not be repeated here. In a possible implementation, when the total number of partial candidate frames acquired by each first single-core processor 2021 is less than the total number of all candidate frames, if the second single-core processor 203 has output the final candidate of the preset data frame, the remaining candidate frames can no longer be processed. For example, when the total number of partial candidate frames acquired by each first single-core processor 2021 is half of the total number of candidate frames, if the second single-core processor 203 has output a preset number of candidate frames for this half number of candidate frames For the final candidate frame, the apparatus for processing candidate frames using multi-core does not process the remaining half of the candidate frames.

Referring to FIG. 10 , it is a schematic flowchart of a method for processing candidate frames using multiple cores provided by an embodiment of the present application. Specifically, the method includes the following steps:

1001. Obtain a candidate frame of an image to be detected.

How to obtain the candidate frame of the image to be detected can be understood with reference to the process of how to obtain the candidate frame in FIG. 1 , and details are not repeated here.

In a preferred embodiment, the obtained candidate frames of the image to be detected are candidate frames that have been sorted in descending order of confidence, and in the order of confidence from high to low, each candidate frame corresponds to A sequence number, the smaller the sequence number, the higher the confidence of the candidate frame. Referring to FIG. 11 , a total of 2560 candidate frames are obtained, each candidate frame corresponds to a sequence number, and a total of 2560 sequence numbers are included.

1002. Obtain some candidate frames from all candidate frames by using the respective identification information of the multiple first single-core processors.

Because the identification information of the first single-core processors is different, the partial candidate frames obtained by the first single-core processors are different. It should be noted that each single-core processor of the plurality of first single-core processors can obtain all candidate frames, but only some of the candidate frames obtained by each of them are used as candidate frames to be processed, and the candidate frames to be processed follow step 1003 to be processed. Continuing to refer to FIG. 11 , for example, it is assumed that X first single-core processors are included in total, and each single-core processor obtains 16 candidate frames from all candidate frames as candidate frames to be processed. In addition, the value of each bit in the bit sequence is only for illustration, and does not represent the real suppression relationship.

1003. Acquire, through each first single-core processor, an inhibition relationship between each candidate frame in some candidate frames and each candidate frame in all candidate frames.

1004. Obtain, by the second single-core processor, a final candidate frame according to the inhibition relationship obtained by each first single-core processor and the confidence level corresponding to each candidate frame.

In a preferred embodiment, the processes executed by each first single-core processor are completely consistent. Referring to FIG. 12 , it is a schematic flowchart of obtaining the suppression relationship between candidate frames by the first single-core processor in an embodiment of the present application. Taking one of the single-core processors as an example, the first single-core processing model can perform the following steps:

1201. Obtain the area of each candidate frame.

1202. Initialize the bit sequence according to the sequence numbers of some candidate frames.

1203. Calculate the overlapping area of each candidate frame in the obtained partial candidate frames and each candidate frame in all the candidate frames.

1204. Calculate the overlapping area of each candidate frame in some of the obtained candidate frames and each candidate frame in all the candidate frames.

1205. Calculate the ratio of the overlapping area and the overlapping area to obtain a bit sequence of each candidate frame.

Wherein, steps 1201 to 1205 can be understood with reference to the relevant steps performed by the first single-core processor 2021 in the embodiments corresponding to FIG. 2a and FIG. 2b, and details are not repeated here. 11, when obtaining the suppression relationship, each candidate frame does not need to obtain the suppression relationship with itself, and does not need to consider the suppression relationship between candidate frames whose confidence is ranked before itself, Therefore, the bit sequence of each candidate frame output in step 1205, the bit sequence corresponding to the candidate frame of serial number P, and the first M-P bits do not need to be considered in the calculation, for example, all the first M-P bits can be set to 1. In addition, referring to FIG. 11 , each bit sequence output by each first single-core processor may be formed into a bit sequence matrix according to the serial number. This matrix of bit sequences can be stored in a memory unit as an input to the second single core processor. In addition, it should be noted that the values in the bit sequence matrix are schematic illustrations and do not represent the real suppression relationship.

Referring to FIG. 13 , it is a schematic flowchart of obtaining a final candidate frame by the second single-core processor in an embodiment of the present application.

As shown in Figure 13, the second single-core processor may perform the following steps:

1301. Generate a sequence number sequence according to the sequence number of the candidate frame sequence and initialize the global sequence.

1302. Obtain a candidate frame according to the sequence number obtained from the previous screening, perform an AND operation according to the bit sequence of the obtained candidate frame and the last updated global sequence, and update the global sequence again.

1303. Obtain the sequence number of the next screening according to the updated global sequence.

1304. Repeat step 1302 and step 1303 until the stop condition is satisfied.

1305. Obtain a final candidate frame according to the final updated global sequence.

Steps 1301 to 1305 can be understood with reference to the relevant steps performed by the second single-core processor 203 in the embodiment corresponding to FIG. 2a , and details are not repeated here. When the second single-core processor 203 is any one of the plurality of first single-core processors 2021, steps 1301 to 1305 may refer to the related processes performed by the first single-core processor 2021 in the embodiment corresponding to FIG. 2b. The steps are understood, and details are not repeated here. Continuing to refer to FIG. 11 , the second single-core processor performs multiple screening on the sequence number sequence to obtain the final candidate frame.

An apparatus for processing candidate frames using multiple cores and a method for processing candidate frames using multiple cores provided by the present application have been introduced above, and the solutions provided by the embodiments of the present application are adopted. It can be understood that, in order to realize the above-mentioned functions, the above-mentioned apparatus for utilizing multi-core processing candidate frames includes corresponding hardware structures and/or software modules for executing each function. Those skilled in the art should easily realize that the present application can be implemented in hardware or in the form of a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Describing from the hardware structure, the apparatus for utilizing the multi-core processing candidate frame in FIG. 2a to FIG. 13 can be implemented by one entity device, or can be implemented jointly by multiple entity devices, and can also be a logical function module in one entity device, This embodiment of the present application does not specifically limit this.

In addition to the structure of the apparatus for processing candidate frames using multiple cores described above, the apparatus for processing candidate frames using multiple cores can also be implemented by the computer device shown in FIG. 14 . Schematic diagram of the hardware structure. It includes: a communication interface 1401 and a processor 1402, and may also include a memory 1403.

The communication interface 1401 can use any device such as a transceiver for communicating with other devices or communication networks. In this solution, the end-side device can use the communication interface 1401 to communicate with the server, such as uploading a model or downloading a model. In a possible implementation manner, the communication interface 1401 may use technologies such as Ethernet, radio access network (RAN), and wireless local area networks (WLAN) to communicate with the server.

The processor 1402 includes but is not limited to a central processing unit (CPU), a network processor (NP), an application-specific integrated circuit (ASIC) or a programmable logic device (programmable logic device, PLD) one or more. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a general-purpose array logic (generic array logic, GAL) or any combination thereof. Processor 1402 is responsible for communication lines 1404 and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions.

Memory 1403 may be read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types of storage devices that can store information and instructions It can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, CD-ROM storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or capable of carrying or storing desired program code in the form of instructions or data structures and capable of being executed by a computer Access any other medium without limitation. The memory may exist independently and be connected to the processor 1402 through the communication line 1404 . The memory 1403 may also be integrated with the processor 1402. If the memory 1403 and the processor 1402 are separate devices, the memory 1403 and the processor 1402 are connected, for example, the memory 1403 and the processor 1402 can communicate through a communication line. The communication interface 1401 and the processor 1402 can communicate through a communication line, and the communication interface 1401 can also be directly connected to the processor 1402 .

Communication lines 1404 , which may include any number of interconnected buses and bridges, link together various circuits including one or more processors 1402 , represented by processor 1402 , and memory, represented by memory 1403 . Communication lines 1404 may also link together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and, therefore, will not be described further herein. When the second single-core processor is any one of the multiple first single-core processors, it can be considered that the processor 1402 in this application includes the multi-core processor 202 and the second single-core processor in the embodiment corresponding to FIG. 2a 203; when the second single-core processor is different from any one of the multiple first single-core processors, it may be considered that the processor 1402 in this application includes the first single-core processor in the embodiment corresponding to FIG. 2b 2021. It should be understood that the above is only an example provided by the embodiments of the present application, and an apparatus utilizing a multi-core processing candidate block may have more or less components than those shown, two or more components may be combined, or Different configurations of components are possible.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product.

The device for utilizing a multi-core processing candidate frame provided by the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or circuit etc. When the training device is a chip, the processing unit can execute the computer execution instructions stored in the storage unit, so that the chip executes the steps performed by the device using the multi-core processing candidate frame described in the embodiments shown in FIG. 2a to FIG. 13 . Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.

Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processor (graphics processing unit, GPU), a digital signal processor (digital signal processor, DSP), application specific integrated circuit (ASIC) or field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, Discrete hardware components, etc. A general purpose processor may be a microprocessor or it may be any conventional processor or the like.

From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware. Special components, etc. to achieve. Under normal circumstances, all functions completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuit, etc. However, a software program implementation is a better implementation in many cases for this application. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that make contributions to the prior art. The computer software products are stored in a readable storage medium, such as a floppy disk of a computer. , U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or CD, etc., including several instructions to make a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present application.

Embodiments of the present application further provide a computer-readable storage medium, where a program for training a model is stored in the computer-readable storage medium, and when it runs on a computer, the computer executes the program shown in FIG. 9 or FIG. 10 above. The examples describe steps in the method.

Embodiments of the present application also provide a computer-readable storage medium, where a program for data processing is stored in the computer-readable storage medium, and when the computer-readable storage medium runs on a computer, the computer executes the program shown in FIG. 6 , FIG. 8 , and FIG. 9. Steps in the method described in the embodiment shown in FIG. 11 . Or cause the computer to execute the steps in the method described in the embodiment shown in FIG. 12 above.

The embodiments of the present application also provide a digital processing chip. The digital processing chip integrates circuits and one or more interfaces for realizing the above-mentioned processor or the functions of the processor. When a memory is integrated in the digital processing chip, the digital processing chip can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate the memory, it can be connected with the external memory through the communication interface. The digital processing chip implements the actions performed by the training device/translation device in the above embodiment according to the program codes stored in the external memory.

Embodiments of the present application also provide a computer program product, where the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a server, data center, etc., which includes one or more available media integrated. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: ROM, RAM, magnetic disk or optical disk, etc.

The device and method for utilizing multi-core processing candidate frames provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described with specific examples in this article. The descriptions of the above embodiments are only used to help understanding The method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be It is construed as a limitation of this application.

The terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. The term "and/or" in this application is only an association relationship to describe associated objects, which means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, independently There are three cases of B. In addition, the character "/" in this article generally indicates that the related objects before and after are an "or" relationship. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover non-exclusive inclusion, eg, a process, method, system, product or device comprising a series of steps or modules not necessarily limited to those expressly listed Rather, those steps or modules may include other steps or modules not expressly listed or inherent to these processes, methods, products or apparatus. The naming or numbering of the steps in this application does not mean that the steps in the method flow must be executed in the time/logical sequence indicated by the naming or numbering, and the named or numbered process steps can be implemented according to the The technical purpose is to change the execution order, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. In practical applications, there may be other divisions. For example, multiple modules may be combined or integrated into another system, or some features may be ignored. , or not implemented, in addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms. There are no restrictions in the application. In addition, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed into multiple circuit modules, and some or all of them may be selected according to actual needs. module to achieve the purpose of the solution of this application.

Claims

An apparatus for processing candidate frames using multiple cores, wherein the apparatus comprises multiple single-core processors,

The multiple first single-core processors in the multiple single-core processors are configured to execute the following processes in parallel:

Obtain some candidate frames from all candidate frames of the image to be detected;

obtaining the suppression relationship between each candidate frame in the partial candidate frame and each candidate frame in the all candidate frames;

The second single-core processor in the plurality of single-core processors is configured to obtain the suppression relationship according to each of the first single-core processors and the corresponding relationship between each candidate frame in the all candidate frames. Confidence, obtain a target candidate frame, where the second single-core processor is any one of the multiple first single-core processors, or the second single-core processor is associated with the multiple first single-core processors None of the single-core processors are the same.
The apparatus according to claim 1, wherein each of the first single-core processors is specifically used for:

The partial candidate frames are acquired from all the candidate frames according to the respective identification information.
The apparatus according to claim 2, wherein each of the first single-core processors is specifically used for:

According to the respective identification information, the partial candidate frames with the same number and adjacent in order are obtained from the candidate frame sequence, wherein the candidate frame sequence is used to indicate the confidence level of all the candidate frames.
The apparatus according to any one of claims 1 to 3, wherein the second single-core processor is further configured to:

Obtain N bit sequences from each of the first single-core processors, where N is the number of the partial candidate frames, and the N bit sequences are used to indicate that each candidate frame in the partial candidate frames is the same as the Suppression relationship between each candidate box in the all candidate boxes.
The apparatus according to claim 4, wherein each of the bit sequences includes M bits, where M is the number of all the candidate frames, and the M bits are used to represent the first candidate frame and all the candidate frames. Suppression relationship between each candidate frame in the candidate frame, the first candidate frame is one candidate frame in the partial candidate frame.
The apparatus according to claim 4 or 5, wherein the second single-core processor is specifically used for:

Obtain the bit sequence to be processed from the acquired bit sequence according to the acquired sequence number of the bit sequence, the sequence number of the to-be-processed bit sequence is determined according to the sequence number of the second candidate frame, the to-be-processed bit sequence is determined The bit sequence is used to represent the suppression relationship between the second candidate frame and each candidate frame in all the candidate frames, the second candidate frame is one candidate frame in the all candidate frames, and the all candidate frames The serial number of each candidate frame in the candidate frame is obtained according to the candidate frame sequence;

Obtain the target candidate frame according to the to-be-processed bit sequence and the processed bit sequence, where the target candidate frame is not selected by the second candidate frame and other candidate frames sorted before the second candidate frame Suppressed candidate frames, each of the processed bit sequences is used to represent a suppression relationship between each other candidate frame ranked before the second candidate frame and each candidate frame in all the candidate frames.
The apparatus according to any one of claims 1 to 6, wherein a part of the candidate frames acquired by each first single-core processor constitutes all the candidate frames.
The apparatus according to any one of claims 1 to 7, wherein each of the first single-core processors is specifically configured to:

Obtain the area of each candidate frame in all the candidate frames;

obtaining the overlapping area between each candidate frame in the partial candidate frame and each candidate frame in the all candidate frames;

obtaining the overlapping area between each candidate frame in the partial candidate frame and each candidate frame in the entire candidate frame;

According to the relationship between the overlapping area and the ratio of the overlapping area and a preset threshold, the suppression relationship between each candidate frame in the partial candidate frame and each candidate frame in the all candidate frames is acquired.
A method for processing candidate frames using multiple cores, comprising:

Through the multiple first single-core processors, the following processes are executed in parallel:

Obtain partial candidate frames from all candidate frames of the image to be detected by each of the first single-core processors;

Obtain, by each of the first single-core processors, the suppression relationship between each candidate frame in the partial candidate frame and each candidate frame in the entire candidate frame;

The target candidate frame is obtained by the second single-core processor according to the inhibition relationship obtained by each of the first single-core processor and the confidence level corresponding to each candidate frame in the all candidate frames, wherein the the second single-core processor is any one of the multiple first single-core processors, or the second single-core processor is different from any one of the multiple first single-core processors .
The method according to claim 9, characterized in that the obtaining, by each of the first single-core processors, some candidate frames from all candidate frames of the image to be detected comprises:

The partial candidate frame is acquired from the sequence of candidate frames through the respective identification information of each of the first single-core processors.
The method according to claim 10, wherein the obtaining the partial candidate frame from the candidate frame sequence through the respective identification information of each of the first single-core processors comprises:

The partial candidate frames with the same number and adjacent in order are obtained from the candidate frame sequence through the respective identification information of each of the first single-core processors, wherein the candidate frame sequence is used to indicate the The confidence level of all the candidate boxes described above.
The method according to any one of claims 9 to 11, wherein the method further comprises:

N bit sequences are obtained from each of the first single-core processors by the second single-core processor, and the N bit sequences are used to represent each candidate frame in the partial candidate frame and the entire candidate frame. Suppression relationship between each candidate box in the candidate box.
The method according to claim 12, wherein each of the bit sequences includes M bits, where M is the number of all the candidate frames, and the M bits are used to represent the first candidate frame and all the candidate frames. Suppression relationship between each candidate frame in the candidate frame, the first candidate frame is one candidate frame in the partial candidate frame.
The method according to claim 12 or 13, wherein the suppression relationship obtained by the second single-core processor according to each of the first single-core processors, and the The confidence corresponding to each candidate frame is obtained, and the target candidate frame is obtained, including:

Obtain the bit sequence to be processed from the acquired bit sequence according to the acquired sequence number of the bit sequence, the sequence number of the to-be-processed bit sequence is determined according to the sequence number of the second candidate frame, the to-be-processed bit sequence is determined The bit sequence is used to represent the suppression relationship between the second candidate frame and each candidate frame in all the candidate frames, the second candidate frame is one candidate frame in the all candidate frames, and the all candidate frames The serial number of each candidate frame in the candidate frame is obtained according to the candidate frame sequence;

Obtain the target candidate frame according to the to-be-processed bit sequence and the processed bit sequence, where the target candidate frame is not selected by the second candidate frame and other candidate frames sorted before the second candidate frame Suppressed candidate frames, each of the processed bit sequences is used to represent a suppression relationship between each other candidate frame ranked before the second candidate frame and each candidate frame in all the candidate frames.
The method according to any one of claims 9 to 14, wherein part of the candidate frames obtained by each first single-core processor constitutes all the candidate frames.
The method according to any one of claims 9 to 15, wherein the obtaining each candidate frame and all the candidate frames in the partial candidate frames through each of the first single-core processors The suppression relationship between each candidate box of , including:

Obtain the area of each candidate frame in all candidate frames through each of the first single-core processors;

Obtain, by each of the first single-core processors, the overlapping area between each candidate frame in the partial candidate frame and each candidate frame in the entire candidate frame;

Obtain, by each of the first single-core processors, the overlapping area between each candidate frame in the partial candidate frame and each candidate frame in the entire candidate frame;

Obtain each candidate frame in the partial candidate frame and the whole candidate frame by each of the first single-core processors according to the relationship between the overlapping area and the ratio of the overlapping area and a preset threshold. Suppression relationship between each candidate box.
A device for processing candidate frames using multiple cores, comprising:

memory for storing computer-readable instructions;

A processor coupled to the memory for executing computer readable instructions in the memory to perform a method as described in any one of claims 9 to 16.
A chip system, characterized in that the chip system includes a processor and a communication interface, the processor obtains program instructions through the communication interface, and when the program instructions are executed by the processor, claims 9 to 10 are implemented. The method of any of 16.
A computer-readable storage medium, characterized by comprising a program that, when executed by a processing unit, executes the method according to any one of claims 9 to 16.
A computer program product, characterized in that, when the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 9 to 16.