JP2022532322A - Image processing methods and devices, electronic devices and storage media - Google Patents

Abstract

Embodiments of the present disclosure provide an image processing method, an apparatus, an electronic device, a storage medium, and a program product, wherein the image processing method performs M-level feature extraction on an image to be processed, and obtaining an M-level first feature map of an image, wherein the scale of the first feature map for each level of the M-level first feature map is different, M being an integer greater than 1; performing scaling and fusion on feature map sets corresponding to the first feature maps of levels, respectively, to obtain second feature maps of M levels, each of said feature map sets corresponding to said first 1 feature map and a first feature map adjacent to said first feature map; and performing target detection on said M-level second feature map to obtain a target detection result for said image to be processed. and including. Embodiments of the present disclosure can improve target detection effectiveness. [Selection drawing] Fig. 1a

Description

[Cross-reference to related applications]
This application is submitted based on a Chinese patent application with an application number of 20100306929.2 filed with the Chinese Patent Office on April 17, 2020, claiming the priority of the Chinese patent application. The entire contents of the Chinese patent application are incorporated herein by reference.
[Technical field]
The present disclosure relates to the field of computer technology, and in particular to image processing methods and devices, electronic devices and storage media.

In the process of processing an image by deep learning, it is usually necessary to detect a target in the image (for example, an object, an animal, a pedestrian) and determine information such as the position or category of the target in the image. However, the scale of the target in the image can be quite different, such as short-distance and long-distance sheep in the image. In the related technique, the detection effect of the target with a large difference in the scale of the image is not good.

The present disclosure proposes a technical plan for image processing.

According to one aspect of the present disclosure, an image processing method is provided, in which the image processing method performs M (M is an integer greater than 1) level feature extraction on the image to be processed. To obtain the M-level first feature map of the image to be processed, the scale of the first feature map of each level of the M-level first feature map is different, and it corresponds to the first feature map of each level. Scale adjustment and fusion are performed for each feature map set to obtain an M-level second feature map, and each feature map set is adjacent to the first feature map and the first feature map. Includes a first feature map to be processed, and performing target detection on the M-level second feature map to obtain a target detection result of the image to be processed.

In some embodiments of the present disclosure, the feature map set corresponding to the first feature map at the i-th (i is an integer and 1 <i <M) level is the first at the i-1 level. Scale adjustment and fusion are performed for each of the feature map sets corresponding to the feature map, the first feature map of the i-level, and the first feature map of the i + 1 level. To obtain the second feature map of the M level, the scale of the first feature map of the i-1 level is reduced to obtain the third feature map of the first i level, and to obtain the third feature map of the first i level. Performing a transformation so that the scale is not changed for the first feature map of the level to obtain the third feature map of the second i-level and expanding the scale of the first feature map of the i + 1 level. Then, the third feature map of the third i-level is obtained, the third feature map of the first i-level, the third feature map of the second i-level, and the third feature map. Including fusing with the third feature map of the i-level to obtain the second feature map of the i-level, wherein the first i-level third feature map and the second feature map are described. The scale of the third feature map of the third i-level and the third feature map of the third i-level is the same.

In this way, for the feature map set corresponding to the first feature map of the i-level, the first feature map of the i-1 level having a large scale is set to the same scale as the first feature map of the i-level. By reducing the scale and expanding the smaller scale i + 1 level first feature map to the same scale as the i level first feature map, the scale of each feature map of the feature map set can be unified.

In some embodiments of the present disclosure, the feature map set corresponding to the first level first feature map includes the first level first feature map and the second level first feature map, and each of the above levels. Performing scale adjustment and fusion for each feature map set corresponding to the first feature map of the above to obtain an M-level second feature map is a scale for the first level first feature map. Perform the transformation so that is not changed to obtain the first level third feature map and scale the second level first feature map to the second first level. Obtaining the third feature map and fusing the first level third feature map and the second first level third feature map to obtain the first level second feature map. Here, the scales of the first level third feature map and the second first level third feature map are the same.

In this way, in the case of the first level first feature map, there is no previous level feature map, and only the first level first feature map itself and the adjacent second level first feature map are processed. The scale of the obtained first level third feature map and the second first level third feature map can be the same. The first level third feature map and the second first level third feature map are added to obtain the first level second feature map. As a result, it is possible to realize the fusion of the adjacent feature maps of the first level.

In some embodiments of the present disclosure, the feature map set corresponding to the first feature map of the M level includes the first feature map of the M-1 level and the first feature map of the M level. Performing scale adjustment and fusion for each feature map set corresponding to the first feature map of each level to obtain the second feature map of the M level is the first feature map of the M-1 level. To obtain the first M-level third feature map by reducing the scale of, and to perform conversion so that the scale does not change with respect to the first M-level first feature map, the second Obtaining the third feature map of the third M level and fusing the third feature map of the first M level and the second feature map of the second M level to obtain the second feature map of the second M level. Including obtaining a feature map, where the scales of the first M-level third feature map and the second M-level third feature map are the same.

In this way, in the case of the first feature map of the M level, there is no feature map of the next level, and the first feature map of the M level itself and the first feature map of the adjacent M-1 level Only can be processed, and the scales of the obtained first M-level third feature map and the second M-level third feature map are the same. The third feature map of the first M level and the third feature map of the second M level can be added to obtain the second feature map of the M level. As a result, it is possible to realize the fusion of the adjacent feature maps of the first level.

In some embodiments of the present disclosure, reducing the scale of the i-1 level first feature map to obtain the first i level third feature map is by means of the first convolution layer. The size of the convolution kernel of the first convolution layer is N × N, including convolving the first feature map of the i-1 level to obtain the third feature map of the first i level. , The step size is n, N, n is an integer greater than 1, and the scale of the first feature map of the i-1 level is n times the scale of the first feature map of the i level. It is possible to obtain the second i-level third feature map by performing the transformation so that the scale is not changed with respect to the i-level first feature map by the second convolution layer. Containing the first feature map of the level to obtain the third feature map of the second i-level, the size of the convolution kernel of the second convolution layer is N × N, and the step size is 1. Therefore, to expand the scale of the first feature map of the i + 1 level to obtain the third feature map of the third i level is to obtain the third feature map of the third i level by the third convolution layer and the upsampling layer. The size of the convolution kernel of the third convolution layer is N × N, including convolving and upsampling the first feature map to obtain the third i-level third feature map. The size is 1.

By setting different convolution layers in this way, it is possible to realize processing for each feature map of the feature map set corresponding to the first level first feature map, and for the subsequent fusion processing, The scale of each feature map in the feature map set can be unified.

In some embodiments of the present disclosure, performing a transformation so that the scale is not changed with respect to the first level first feature map to obtain the first first level third feature map. The size of the convolution kernel of the second convolution layer is N ×, including convolving the first level first feature map with the second convolution layer to obtain the first first level third feature map. N, the step size is 1, N is an integer greater than 1, and the scale of the first level first feature map is expanded to obtain the second first level third feature map. That includes convolving and upsampling the second level first feature map with the third convolution layer and upsampling layer to obtain the second first level third feature map. The size of the convolution kernel of the third convolution layer is N × N, and the step size is 1.

By setting different convolution layers in this way, it is possible to realize processing for each feature map of the feature map set corresponding to the first level first feature map.

In some embodiments of the present disclosure, reducing the scale of the M-1 level first feature map to obtain the first M level third feature map is by means of the first convolutional layer. The size of the convolution kernel of the first convolution layer is N × N, including convolving the first feature map of the first M-1 level to obtain the third feature map of the first M level. , The step size is n, N, n is an integer greater than 1, and the scale of the first feature map of the i-1 level is n times the scale of the first feature map of the i level. It is possible to obtain the third feature map of the second M level by performing the transformation so that the scale is not changed with respect to the first feature map of the M level by the second convolution layer. Containing the first feature map of the level to obtain the third feature map of the second M level, the size of the convolution kernel of the second convolution layer is N × N and the step size is 1. Is.

By setting different convolution layers in this way, it is possible to realize processing for each feature map of the feature map set corresponding to the first feature map of the M level.

In some embodiments of the present disclosure, the second convolution layer and the third convolution layer include a deformable convolution layer or an expansion convolution layer.

In this way, if the second and third convolution layers are deformable convolution layers, the offset is learned by setting a separate convolution layer, and then both the input feature map and the offset. Either can be used as an input for a deformable convolution layer, offset sampling points, and then convolved. When the 2nd and 3rd convolutional layers are expansion convolutional layers, presetting the expansion rate of the expansion convolution helps to properly adjust the receptive fields of the convolution, further enhancing the effect of feature map fusion. Can be improved.

In some embodiments of the present disclosure, the image processing method is implemented by an image processing network, which performs scale adjustment and fusion P times with respect to the M-level first feature map. Containing series-connected P (P is a positive integer) level fusion network block for each level of fusion network block, a plurality of first convolution layers, a plurality of second convolution layers and a plurality of first convolution layers. Obtaining an M-level second feature map by performing scale adjustment and fusion on each feature map set corresponding to the first feature map of each level, including the three convolutional layers, is the M-level. The first feature map is input to the first level fusion network block and the first fused M level fourth feature map is output, and j-1 (j is an integer and 1 <j. The 4th feature map of the M level fused at the <P) th time is input to the fusion network block of the jth level, and the 4th feature map of the M level fused at the jth time is output, and P. -Including inputting the M-level fourth feature map fused for the first time into the P-level fusion network block and outputting the M-level second feature map.

In this way, the fusion effect can be further improved by processing the image with the directly connected P-level fusion network block.

In some embodiments of the present disclosure, each level of fusion network block further comprises a normalization layer, and the j-1st fused M-level fourth feature map is input to the j-level fusion network block. Then, the fourth feature map of the M level fused at the jth time is output by the first convolution layer, the second convolution layer and the third convolution layer of the j-level fusion network block. Scale adjustment and fusion are performed on the feature map sets corresponding to the first fused M-level fourth feature map, respectively, to obtain the j-th fused M-level intermediate feature map. The combined batch normalization process is executed on the M-level intermediate feature map fused by the normalization layer at the jth time to obtain the M-level fourth feature map fused at the jth time. ,including.

In this way, by executing the combined batch normalization process on the M-level intermediate feature map fused by the normalization layer for the jth time, the training process is effectively stabilized and the performance is further improved. Combined batch normalization can have a significant effect, especially if the batch of detection tasks is relatively small.

In some embodiments of the present disclosure, the image processing method is implemented by an image processing network, which further comprises a regression network and a classification network, which is a target for the M-level second feature map. To execute the detection and obtain the target detection result of the processed image, the M-level second feature map is input to the regression network to determine the image frame corresponding to the target in the processed image. In addition, the M-level second feature map is input to the classification network to determine the category of the target in the processed image, and the target detection result is the image corresponding to the target. Includes the frame and the category of the target.

In this way, the regression network and the classification network are used to realize the regression task and the classification task of target detection, respectively.

According to one aspect of the present disclosure, an image processing apparatus is provided, in which the image processing apparatus performs M (M is an integer larger than 1) level feature extraction on the image to be processed, and the above-mentioned image processing apparatus is performed. It is a feature extraction module configured to obtain an M-level first feature map of the image to be processed, and the scale of the first feature map at each level of the M-level first feature map is different from that of the feature extraction module. It is a scale adjustment and fusion module configured to perform scale adjustment and fusion for each feature map set corresponding to the first feature map of each level to obtain an M level second feature map. Each of the feature map sets performs target detection on the scale adjustment and fusion module including the first feature map and the first feature map adjacent to the first feature map, and the M level second feature map. A target detection module configured to obtain a target detection result of the image to be processed is provided.

According to one aspect of the present disclosure, an electronic device is provided, wherein the electronic device comprises a processor and a memory in which a processor-executable instruction is stored, wherein the processor is stored in the memory. It is configured to call the command given and execute the above image processing method.

According to one aspect of the present disclosure, a computer-readable storage medium in which a computer program instruction is stored is provided, and the above-mentioned image processing method is realized when the computer program instruction is executed by a processor.

According to one aspect of the present disclosure, a computer program product is provided, the computer program product comprising one or more instructions, the one or more instructions causing the processor to perform the image processing method described above. Let me.

In the embodiment of the present disclosure, M-level feature extraction is performed on the image to be processed to obtain an M-level first feature map, and each first feature map and an adjacent feature map are fused to M. The second feature map of the level can be obtained and the target detection can be performed on the second feature map of the M level to obtain the target detection result, whereby the features between the adjacent layers of the first feature map of the M level can be obtained. By fusing related information, the effect of target detection can be effectively improved.

It should be understood that the general description described above and the detailed description described below are merely examples and explanations and do not limit the present disclosure. Hereinafter, detailed description of the exemplary embodiments with reference to the drawings will reveal other features and embodiments of the present disclosure.

The flowchart of the image processing method which concerns on embodiment of this disclosure is shown. Schematic representations of four different methods for generating combinations of multidimensional features are shown. The schematic diagram of the operation principle of the deformable convolution layer is shown. A schematic diagram of batch normalization related to related techniques is shown. A schematic diagram of batch normalization related to related techniques is shown. The schematic diagram of the union batch normalization which concerns on the Example of this disclosure is shown. The schematic diagram of the detector which concerns on a related technique is shown. The schematic diagram of the image processing network which concerns on embodiment of this disclosure is shown. The block diagram of the image processing apparatus which concerns on embodiment of this disclosure is shown. The block diagram of the electronic device which concerns on embodiment of this disclosure is shown. The block diagram of the electronic device which concerns on embodiment of this disclosure is shown.

The above drawings are incorporated herein to constitute a portion thereof, and these drawings are used to illustrate examples consistent with the present disclosure and to illustrate the technical solutions of the present disclosure together with the specification. Will be done.

Hereinafter, various exemplary examples, features and embodiments of the present disclosure will be described in detail with reference to the drawings. The same reference number in the drawing indicates an element of the same or similar function. Various embodiments of the examples are shown in the drawings, but it is not necessary to draw the drawings to scale unless otherwise specified.

The term "exemplary" in the specification means "used as an example, example or description". Here, any example described as "exemplary" need not be construed as more appropriate or superior to the other examples.

As used herein, the term "and / or" simply refers to the relationships of related objects, indicating that three types of relationships can exist, for example, A and / or B, where A exists independently. , A and B exist at the same time, and represent three cases such as the case where B exists independently. Further, the term "at least one" as used herein refers to any combination of one or more of a plurality, and includes, for example, at least one of A, B, C. This indicates that it contains any one or more elements selected from the set composed of A, B and C.

Further, in order to better illustrate the present disclosure, a number of details are provided in the following embodiments. It will be appreciated by those skilled in the art that this disclosure can be carried out without some details being provided. In some embodiments, the methods, means, elements and circuits well known to those of skill in the art will not be described in detail in order to emphasize the gist of the present disclosure.

FIG. 1a shows a flowchart of an image processing method according to an embodiment of the present disclosure, and as shown in FIG. 1a, the image processing method includes the following steps.

In step S11, M (M is an integer larger than 1) level feature extraction is performed on the processed image to obtain the M level first feature map of the processed image, and the M level is obtained. The scale of the first feature map at each level of the first feature map is different.

In step S12, scale adjustment and fusion are performed on the feature map sets corresponding to the first feature maps of each level to obtain the second feature map of the M level, and here, each of the feature map sets. Includes the first feature map and a first feature map adjacent to the first feature map.

In step S13, the target detection is executed for the second feature map of the M level, and the target detection result of the processing target image is obtained.

In some embodiments of the present disclosure, the image processing method is performed by an electronic device such as a terminal device or a server, where the terminal device is a user device (UE: User Computing), mobile device, user terminal, terminal, cellular. It may be a telephone, a cordless telephone, a mobile information terminal (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like, and the method is computer-readable in which a processor is stored in a memory. It can be realized by a method of calling an instruction, or the method can be executed by a server.

For example, the image to be processed may be an image including a target (for example, an object, an animal, a pedestrian, etc.), and the image to be processed may be an image collected by an image collecting device (for example, a camera). It may be obtained by another method, and the present disclosure is not particularly limited thereto.

In some embodiments of the present disclosure, in step S11, feature maps are extracted from different layers of the network and processed by, for example, performing feature extraction at multiple levels on the image to be processed by the feature pyramid network. It is possible to obtain a first feature map (also called a feature pyramid) at the M (M is an integer greater than 1) level of the image. Here, the scale of the first feature map of each level of the first feature map of M level is different. The characteristic pyramid network may include at least an M-layer convolution layer, a pooling layer, and the like, and the present disclosure is not particularly limited to the network structure of the characteristic pyramid network. Detection using single-scale images can reduce storage and computational costs.

FIG. 1b is a schematic representation of four different methods for generating a combination of multidimensional features, wherein FIG. 1b shows a diagram (a) showing a characterized image pyramid and features of a single scale. FIG. 1b includes a diagram (b) showing, a diagram (c) showing a pyramid feature hierarchical structure, and a diagram (d) showing a feature pyramid network, as shown in FIG. 1b. The resulting image pyramid builds a feature pyramid using the image pyramid. Features are calculated independently for images of each scale, and the speed of predictive output is slow. FIG. 1b (b) shows the case of a single scale feature, where the detection system speeds up the detection and outputs a prediction by using only the single scale feature. FIG. 1b (c) shows the case of the pyramid feature hierarchical structure, and the prediction is output by using the pyramid feature hierarchical structure a plurality of times. The feature pyramid network shown in FIG. 1b (d) has a high speed of outputting predictions as in FIGS. (B) and (c), and is more accurate than them. In this way, the top-down process of the feature pyramid network expands the top-level small feature map to the same size as the adjacent feature map by upsampling. The advantage of this is that it not only uses the strong semantic features of the top layer, but also the high resolution information of the bottom layer.

In the subsequent processing, when the M-level first feature map is directly fused, the semantic information between different layers can be fused, but the correlation of the features between adjacent layers cannot be expressed. In this case, step S12 can realize the fusion of the first feature map of each level and the first feature map adjacent thereto.

In some embodiments of the present disclosure, in step S12, scale adjustment and fusion are performed on the feature map sets corresponding to the first feature maps of each level to obtain the second feature map of M level. Each feature map set may include the first feature map and a first feature map adjacent to the first feature map. For example, for any first feature map, the scale of 2q adjacent feature maps (that is, acquiring q (q ≧ 1) feature maps before and after each) is the same as that of the first feature map. Adjusted to a scale, the adjusted 2q feature maps and the first feature map are added to obtain a second feature map corresponding to the first feature map, the present disclosure of which is particularly relative to the value of q. Not limited.

In some embodiments of the present disclosure, the scale of the feature map set of the first feature map (including the first feature map and adjacent 2q feature maps) may be unified to a specific scale, for example, features. Enlarge all feature maps in the map set by many times the scale of the first feature map, or reduce them to a fraction of the scale of the first feature map. Then, each adjusted feature map is added to obtain a second feature map corresponding to the first feature map. The present disclosure is not particularly limited to the scale range and method of scale adjustment to the feature map set.

In this way, the correlation of the feature map dimension and the correlation of the spatial dimension can be obtained, and the accuracy of the fused feature map can be improved.

In some embodiments of the present disclosure, in step S13, target detection can be performed on the M-level second feature map to obtain the target detection result of the image to be processed. For example, regression processing and classification processing are executed for the second feature map of M level, respectively. After the regression processing, the location image area (that is, the detection frame) of the target in the processing target image can be determined, and after the classification processing, the target category in the processing target image can be determined. The target detection result of the processing target image may include a target location image area (that is, a detection frame) in the processing target image, a target category, and the like.

According to the embodiment of the present disclosure, M-level feature extraction is performed on the image to be processed to obtain an M-level first feature map, and each first feature map and an adjacent feature map are fused. The second feature map of the M level can be obtained, and the target detection result can be obtained by performing the target detection on the second feature map of the M level, whereby the adjacent layers of the first feature map of the M level can be obtained. It is possible to effectively improve the effect of target detection by fusing the related information of the features.

In some embodiments of the present disclosure, the scale of the first feature map at each level of the M level first feature map acquired in step S11 may be gradual, eg, the first level. The scale of the first feature map is 512 × 512, the scale of the first feature map of the second level is 256 × 256, and the scale of the first feature map of the third level is 128 × 128. The present disclosure is not particularly limited to the value of the scale of the first feature map of M level.

In some embodiments of the present disclosure, for the first feature map of the i (i is an integer and 1 <i <M) level of the first feature map of the M level, the first of the i level. The feature map set corresponding to the feature map includes the first feature map of the i-1 level, the first feature map of the i level, and the first feature map of the i + 1 level. Here, step S12 is
Reducing the scale of the first feature map of the first i-1 level to obtain the third feature map of the first i level,
Performing a transformation on the i-level first feature map so that the scale is not changed to obtain a second i-level third feature map.
By expanding the scale of the first feature map of the first i + 1 level to obtain the third feature map of the third i level,
The third feature map of the first i-level, the third feature map of the second i-level, and the third feature map of the third i-level are fused to form the third feature map of the i-level. 2 Including getting a feature map
Here, the scales of the first i-level third feature map, the second i-level third feature map, and the third i-level third feature map are the same.

For example, for the feature map set corresponding to the first feature map of the i-level, the first feature map of the i-1 level having a large scale is reduced to the same scale as the first feature map of the i-level. By enlarging the first feature map of the i + 1 level, which has a smaller scale, to the same scale as the first feature map of the i level, the scale of each feature map of the feature map set can be unified.

In some embodiments of the present disclosure, the scale of the first level first feature map can be scaled down to obtain the first level i third feature map, the first level i first. A transformation can be performed on the feature map so that the scale does not change to obtain the second i-level third feature map, and the scale of the first i + 1 level first feature map is scaled up to 3 A third feature map of the third i-level can be obtained. Here, the scales of the first i-level third feature map, the second i-level third feature map, and the third i-level third feature map are the same.

In some embodiments of the present disclosure, methods such as convolution and downsampling achieve scale reduction, and deconvolution, upsampling, and convolution with a step size smaller than 1 achieve scale expansion. However, the convolution or other processing method having a step size of 1 can realize a conversion in which the scale is not changed, and the present disclosure is not particularly limited thereto.

In some embodiments of the present disclosure, the first i-level third feature map, the second i-level third feature map, and the third i-level third feature map are directly added together. Alternatively, by adding and fusing according to preset weights, the i-level second feature map can be obtained, and the i-level second feature map scale and the i-level first feature map scale can be obtained. Is the same. As a result, it is possible to realize the fusion of adjacent feature maps and improve the feature extraction effect.

In some embodiments of the present disclosure, reducing the scale of the i-1 level first feature map to obtain the first i level third feature map is by means of the first convolution layer. The size of the convolution kernel of the first convolution layer is N × N (N), including convolving the first feature map of the i-1 level to obtain the third feature map of the first i level. Is an integer greater than 1), the step size is n (n is an integer greater than 1), and the scale of the i-1 level first feature map is the i-level th. It is n times the scale of one feature map,
Performing a transformation on the i-level first feature map so that the scale is not changed to obtain a second i-level third feature map is performed by the second convolution layer of the i-level. The size of the convolution kernel of the second convolution layer is N × N and the step size is 1, including convolving the first feature map to obtain the second i-level third feature map. ,
Enlarging the scale of the first feature map of the i + 1 level to obtain the third feature map of the third i level is to obtain the first feature of the i + 1 level by the third convolution layer and the upsampling layer. The size of the convolution kernel of the third convolution layer is N × N and the step size is 1 including convolving and upsampling the map to obtain the third feature map of the third i-level. Is.

For example, by setting different convolution layers, it is possible to realize processing for each feature map of the feature map set corresponding to the first level first feature map.

In some embodiments of the present disclosure, the first convolutional layer convolves the first level i-1 feature map to obtain the first level i third feature map. The size of the convolution kernel of the first convolution layer is N × N (N is an integer greater than 1), the step size is n (n is an integer greater than 1), and the i-. The scale of the 1st level 1st feature map is n times the scale of the 1st level 1st feature map, and the scale can be reduced by convolution. For example, if the scale of the first feature map of the i-1 level is 256 × 256 and the scale of the first feature map of the i level is 128 × 128, then n = 2, that is, the i-th. Both the length and width of the 1st level 1st feature map are twice the length and width of the ist level 1st feature map. After convolution, the scale of the first i-level third feature map obtained is 128 × 128. Here, N is, for example, 3, and the present disclosure is not particularly limited to the values of N and n.

In some embodiments of the present disclosure, a second convolutional layer can be used to convolve a first level i-level feature map to obtain a second i-level third feature map, the second convolutional layer. The size of the convolution kernel of is N × N, and the step size is 1, that is, it is possible to realize a conversion in which the scale is not changed by convolution. For example, the scale of the first feature map of the i-level is 128 × 128, and the scale of the third feature map of the second i-level acquired after convolution is 128 × 128. As will be appreciated, those skilled in the art will be able to implement transformations that do not change the scale using other methods, and the present disclosure is not particularly limited thereto.

In some embodiments of the present disclosure, the third convolution layer and the upsampling layer convolve the first feature map of the i + 1 level to perform n times upsampling, and the third feature of the third i level. A map can be obtained, the size of the convolution kernel of the third convolution layer is N × N, the step size is 1, that is, expansion of the scale can be realized by convolution and upsampling. For example, if the scale of the first feature map of the i + 1 level is 64 × 64 and the scale of the first feature map of the i level is 128 × 128, n = 2. After convolution and double upsampling, the scale of the third feature map of the third i-level obtained is 128 × 128. As will be appreciated, those skilled in the art can achieve scale expansion using other methods such as deconvolution and convolution with a step size of 1 / n, and the present disclosure is not particularly limited thereto.

As a result, by unifying the scale of each feature map of the feature map set, it can be useful for the subsequent fusion process.

In some embodiments of the present disclosure, the first i-level third feature map, the second i-level third feature map, and the third i-level third feature map are directly populated. Add up to get the second feature map of the i-level. The overall processing process is as follows.

Y ^ i = Upsample (w ^ 1 * x ^ (i + 1)) + w ^ 0 * x ^ i + w ^ (-1) * _ (s = 2) x ^ (i-1) Equation (1)
The corresponding code is:

In equation (1), Y ^ i represents the second feature map of the i-th level, and x ^ (i + 1), x ^ i and x ^ (i-1) are the second features of the i + 1 level, respectively. Represents a 1-feature map, an i-level 1st feature map, and a 1st-level first feature map, where w ^ 1, w ^ 0, and w ^ (-1) are the third convolution layer and the third, respectively. The weights of the two convolution layers and the first convolution layer are represented, * represents a convolution operation, s represents a step size, and Upsample represents an upsampling operation.

The processing process of equation (1) is also referred to as pyramid convolution or scale space convolution. The pyramid convolution process can obtain a second feature map in which adjacent layer information is fused, and can effectively improve the subsequent target detection effect.

In some embodiments of the present disclosure, with respect to the first level first feature map of the M level first feature map, the feature map set corresponding to the first level first feature map is the first level first feature map. Includes 1 feature map and 2nd level 1st feature map. Here, step S12 is
To obtain the first level third feature map by performing a transformation so that the scale is not changed with respect to the first level first feature map.
Enlarging the scale of the first level first feature map to obtain the second first level third feature map,
Including fusing the first level third feature map and the second first level third feature map to obtain a first level second feature map.
The scales of the first level third feature map and the second first level third feature map are the same.

For example, in the case of the first level first feature map, there is no previous level feature map, and only the first level first feature map itself and the adjacent second level first feature map can be processed. ..

In some embodiments of the present disclosure, transformations are performed so that the scale is not changed for the first level first feature map to obtain the first first level third feature map and the second level. The scale of the first feature map of can be expanded to obtain the third feature map of the second i-level. Here, the scales of the first level third feature map and the second first level third feature map are the same.

In some embodiments of the present disclosure, the first level third feature map and the second first level third feature map are added to obtain a first level second feature map. Can be done. As a result, it is possible to realize the fusion of the adjacent feature maps of the first level.

In some embodiments of the present disclosure, performing a transformation so that the scale is not changed with respect to the first level first feature map to obtain the first first level third feature map. The size of the convolution kernel of the second convolution layer is N ×, including convolving the first level first feature map with the second convolution layer to obtain the first first level third feature map. N (N is an integer greater than 1), the step size is 1, and
Enlarging the scale of the first level first feature map to obtain the second first level third feature map is to obtain the second level first feature by means of the third convolution layer and the upsampling layer. The size of the convolution kernel of the third convolution layer is N × N, and the step size is 1, including convolving and upsampling the map to obtain the third feature map of the second first level. be.

That is, by setting different convolution layers, it is possible to realize processing for each feature map of the feature map set corresponding to the first level first feature map. The second convolution layer can convolve the first level first feature map to obtain the first first level third feature map, i.e., a transformation can be achieved in which the convolution does not change the scale. 3 The convolution layer and the upsampling layer convolve the first feature map of the second level and perform n times upsampling to obtain the third feature map of the second first level, that is, convolution and upsampling. It is possible to realize the expansion of the scale. The processing method is the same as the above description, and is not described repeatedly here.
By this method, it is possible to unify the scale of each feature map of the feature map set and bring convenience to the subsequent fusion.

In some embodiments of the present disclosure, for the M-level first feature map of the M-level first feature map, the feature map set corresponding to the M-level first feature map is of the M-1 level. Includes a first feature map and the M-level first feature map. Here, step S12 is
By reducing the scale of the first feature map of the first M-1 level to obtain the third feature map of the first M level,
To obtain the second M-level third feature map by performing a transformation so that the scale is not changed for the first M-level feature map.
Including fusing the first M-level third feature map and the second M-level second feature map to obtain a second M-level feature map.
Here, the scales of the first M-level third feature map and the second M-level third feature map are the same.

For example, in the case of the first feature map of the first M level, there is no feature map of the next level, and only the first feature map of the M level and the adjacent first feature map of the M-1 level are processed. can do.

In some embodiments of the present disclosure, the scale of the first feature map of the M-1 level can be scaled down to obtain the third feature map of the first M level, the first of the M level. A transformation can be performed on the feature map so that the scale is not changed to obtain a second M-level third feature map. Here, the scales of the first M-level third feature map and the second M-level third feature map are the same.

In some embodiments of the present disclosure, the first M-level third feature map and the second M-level third feature map are added to obtain a second M-level feature map. Can be done. As a result, it is possible to realize the fusion of the adjacent feature maps of the first level.

In some embodiments of the present disclosure, reducing the scale of the M-1 level first feature map to obtain the first M level third feature map is by means of the first convolution layer. Convolving the first feature map of the first M-1 level to obtain the third feature map of the first M level, the size of the convolution kernel of the first convolution layer is N × N (N). Is an integer greater than 1), the step size is n (n is an integer greater than 1), and the scale of the first feature map of the i-1 level is the i-level. It is n times the scale of the first feature map,
Performing a transformation on the M-level first feature map so that the scale is not changed to obtain a second M-level third feature map is performed by the second convolution layer of the M-level. The size of the convolution kernel of the second convolution layer is N × N and the step size is 1, including convolving the first feature map to obtain the second M-level third feature map. ..

That is, by setting different convolution layers, it is possible to realize processing for each feature map of the feature map set corresponding to the first feature map of the M level. The first convolution layer convolves the first feature map of the M-1 level to obtain the third feature map of the first M level, that is, the convolution realizes the reduction of the scale, and the second convolution layer. The first M-level feature map can be convoluted to obtain a second M-level third feature map, i.e., the conversion can be performed so that the scale is not changed by the convolution. The processing method is the same as the above description, and is not described repeatedly here. By this method, it is possible to unify the scale of each feature map of the feature map set and bring convenience to the subsequent fusion.

In some embodiments of the present disclosure, the second convolution layer and the third convolution layer include a deformable convolution layer or an expansion convolution layer.

FIG. 1c is a schematic diagram of the operating principle of a deformable convolutional layer, which includes an input feature map 11, a deformable convolutional layer 12, a convolution 13, an offset 14, and an output feature map 15. As shown in FIG. 1c, first, the offset 14 is learned using a separate convolution 13 and the input feature map 11 is shared. After that, both the input feature map 11 and the offset 14 are used as inputs of the deformable convolution layer 12, the sampling points are offset, and then the convolution is performed to acquire the output feature map 15.

After the pyramid convolution has moved to the bottom layer, the normal convolution of the pyramid convolution can be replaced with a deformable or inflatable convolution, but shares weight with the bottom layer convolution. This allows the receptive fields to be dynamically adjusted at different positions in the feature map to achieve alignment with the normal convolution of the underlying feature map. In this case, the adjusted pyramid convolution is also referred to as a pyramid convolution with a uniform scale.

That is, in the case of the feature map set corresponding to the first feature map of the i-level, the first convolution layer corresponding to the first feature map of the i-1 level is a normal convolution, and the first feature of the i-level. The second convolution layer corresponding to the map and the third convolution layer corresponding to the first feature map of the i + 1 level are deformable convolutions or expansion convolutions.

In some embodiments of the present disclosure, when the second convolution layer and the third convolution layer are deformable convolution layers, the feature map input after learning the offset by setting a separate convolution layer. And offsets can both be used as inputs to the deformable convolution layer to offset the sampling points before convolution.

In some embodiments of the present disclosure, when the second and third convolutional layers are inflatable convolutional layers, the receptive fields of the convolution are appropriately adjusted by presetting the expansion rate of the inflatable convolution. Can be useful for. The present disclosure is not particularly limited to the setting of the expansion rate.

This makes it possible to appropriately adjust the receptive fields of the convolution and further improve the fusion effect of the feature map.

In some embodiments of the present disclosure, the image processing method according to the embodiments of the present disclosure can be realized by an image processing network, and the image processing network can extract features at a plurality of levels with respect to the image to be processed. May include a feature pyramid network for performing.

In some embodiments of the present disclosure, the image processing network is serially connected P (where P is positive) for performing scaling and fusion P times with respect to the M-level first feature map. It may include level fusion network blocks (which are integers), and each level fusion network block includes a plurality of first convolution layers, a plurality of second convolution layers, and a plurality of third convolution layers.

In some embodiments of the present disclosure, the scaling and fusion processes can be performed multiple times, which can be implemented by P-level fusion network blocks, each level of fusion network block ( PConv) includes a plurality of first convolution layers, a plurality of second convolution layers, and a plurality of third convolution layers for processing each feature map set composed of adjacent feature maps. P is, for example, 4, and the present disclosure is not particularly limited to the value of P.

In some embodiments of the present disclosure, each level of fusion network block can process multiple feature map sets, where each feature map set is one convolution for convolving each feature map of the feature map set. Corresponds to the stratum. For example, for a feature map set including a first feature map at the i-1 level, a first feature map at the i-level, and a first feature map at the i + 1 level, the convolutional layer set corresponding to the feature map set is , 1st convolution layer, 2nd convolution layer, 3rd convolution layer and upsampling layer, 1st feature map of i-1 level, 1st feature map of i level and 1st feature map of i + 1 level. Is for folding each.

In some embodiments of the present disclosure, step S12 is
The first M-level feature map is input to the first-level fusion network block, and the first fused M-level fourth feature map is output.
The M-level fourth feature map fused at the j-1 (j is an integer and 1 <j <P) th time is input to the j-level fusion network block, and the fusion is performed at the jth time. Outputting the 4th feature map of M level and
P-The first fusion of the M-level fourth feature map is input to the P-level fusion network block, and the M-level second feature map is output.

For example, the M-level first feature map is input to the first-level fusion network block, the first scale adjustment and fusion are performed, and the first fused M-level fourth feature map is output. The first fused M-level fourth feature map can be input to the next level fused network block. The j-1 (j is an integer and 1 <j <P) th-fused M-level fourth feature map is input to the j-level fusion network block, and the j-th scale adjustment and The fusion can be executed and the M-level fourth feature map fused can be output. P-1 The 4th M-level feature map fused is input to the P-level fusion network block, the P-th scale adjustment and fusion are performed, and the M-level 2nd feature map is output. be able to.

Thereby, the fusion effect can be further improved.

In some embodiments of the present disclosure, each level of fusion network block further comprises a normalization layer for normalizing the post-fusion feature map. Here, inputting the j-1st fused M-level fourth feature map into the j-level fusion network block and outputting the j-1st fused M-level fourth feature map is possible.
For the feature map set corresponding to the M-level fourth feature map fused at the j-1st time by the first convolution layer, the second convolution layer, and the third convolution layer of the j-level fusion network block. Performing scale adjustment and fusion, respectively, to obtain the M-level intermediate feature map fused at the jth time,
The combined batch normalization process is executed on the M-level intermediate feature map fused by the normalization layer at the jth time to obtain the M-level fourth feature map fused at the jth time. including.

For example, the j-th scale adjustment and fusion is the fourth feature of the M-level fused by the first convolution layer, the second convolution layer, and the third convolution layer of the j-level fusion network block. Scale adjustment and fusion can be performed on the feature map sets corresponding to the maps, respectively, to obtain the M-level intermediate feature map fused at the jth time.

For example, the input parameters for batch normalization are B = {x _{1 ... m} }, magnification multiple γ, and offset coefficient β (parameters to be learned).

The output of batch normalization is as follows.

Returns the learned expansion multiple γ and offset coefficient β.

Here, the formula (2) is a formula of the response of the network after standardization, the formula (3) is a formula for calculating the average value of the batch processing data, and the formula (4) is the batch processing data. Equation (5) is an equation for normalization, and equation (6) is an equation for scale conversion and offset.

In some embodiments of the present disclosure, the j-level fusion network block can process a plurality of feature map sets corresponding to the j-1th fused M-level fourth feature map, each. The feature map set corresponds to one convolutional layer set for folding each feature map of the feature map set. For example, for a feature map set including a first feature map at the i-1 level, a first feature map at the i-level, and a first feature map at the i + 1 level, the convolutional layer set corresponding to the feature map set is , 1st convolution layer, 2nd convolution layer, 3rd convolution layer and upsampling layer, 1st feature map of i-1 level, 1st feature map of i level and 1st feature map of i + 1 level. Is for folding each.

In some embodiments of the present disclosure, the normalization layer statistics the statistical values (eg, mean and variance) of the M-level intermediate feature map fused in the j-th time, and the M-level fused in the j-th time. The union batch normalization process is executed for the intermediate feature map of the above, and the normalized result is determined as the fourth feature map of the M level fused in the jth time.

2a and 2b show a schematic diagram of batch normalization according to the related technique, and FIG. 2c shows a schematic diagram of the combined batch normalization according to the embodiment of the present disclosure. Here, after the folding layer 21 processing, a plurality of feature maps (in FIG. 2a, FIG. 2b and FIG. 2c, two feature maps will be described as an example) are output, and the batch normalization layer (abbreviated as BN) 22 is used. Batch normalization is executed for each of the plurality of feature maps, and after batch normalization, activation is performed by the activation layer (for example, ReLU layer) 23. Here, γ and β represent the expansion multiple and the offset coefficient, respectively, which can be obtained by learning, and μ and σ represent the mean value and the standard deviation, respectively, which can be obtained statistically.

In a related technique, as shown in FIG. 2a, the two batch normalization layers 22 are made to share the magnification multiple γ and the offset coefficient β so that the mean value μ and standard deviation σ of each feature map are statistic, respectively. As shown in FIG. 2b, the two batch normalization layers 22 are trained with the magnification multiple γ and the offset coefficient β, respectively, and the mean value μ and standard deviation σ of each feature map are statistic. be able to.

In the combined batch normalization process according to the embodiment of the present disclosure, as shown in FIG. 2c, the two batch normalization layers 22 share the expansion multiple γ and the offset coefficient β, and the average value of all the feature maps. It is possible to jointly statistic μ and standard deviation σ.

Joint statistics of feature map statistics for all scales can effectively stabilize the training process and further improve performance, especially if the batch of detection tasks is relatively small. It can bring about an excellent effect by the conversion.

In some embodiments of the present disclosure, the image processing network may further include a regression network and a classification network for achieving the regression and classification tasks of target detection. Here, the recurrent network and the classification network may include a convolution layer, an activation layer, a fully connected layer, and the like, and the present disclosure is not particularly limited to the network structure of the recurrent network and the classification network.

In the embodiments of the present disclosure, step S13 is
The second feature map of the M level is input to the regression network to determine the image frame corresponding to the target in the image to be processed.
The second feature map of the M level is input to the classification network to determine the category of the target in the image to be processed, and the target detection result is the image frame corresponding to the target and the target. Includes, and may include.

For example, the regression task and the classification task of target detection can be realized according to the second feature map of M level. By inputting the M-level second feature map into the regression network and performing regression processing, it is possible to acquire an image frame corresponding to the target in the image to be processed, and input the M-level second feature map into the classification network. By processing the image, the category of the target in the image to be processed can be determined. Here, the processing target image target detection result may include an image frame corresponding to the target and a category of the target.

Detectors in related techniques typically design regression headers and classification headers for regression and classification tasks, respectively. The image processing network according to the embodiment of the present disclosure uses a P-level fusion network block (using a pyramid convolution) as a combined header for a regression task and a classification task, with a slight difference between the two tasks to the receptor field. Only according to, non-shared convolutions are added to the regression and classification networks, which significantly reduces the amount of computation and does not compromise performance.

FIG. 3a shows a schematic diagram of a detector according to a related technique, and FIG. 3b shows a schematic diagram of an image processing network according to an embodiment of the present disclosure.

As shown in FIG. 3a, detectors in related arts design regression headers 31 and classification headers 32, respectively, for regression and classification tasks, each of which is a multi-level network block (eg, eg). The feature map is processed by the convolution block), the regression task and the classification task are realized by the network block at the last level, and the regression task acquires the coordinates of the four vertices of the detection frame of K targets in the image and classifies them. By the task, K target categories in the image (set to have a total of C categories) are acquired. Here, the network block at each level may include a convolution layer, an activation layer, a fully connected layer, and the like, and the present disclosure is not particularly limited thereto.

As shown in FIG. 3b, the image processing network according to the embodiment of the present disclosure uses a P-level fusion network block (also referred to as a P convolution block) as a combination header 33 for a regression task and a classification task, M. After processing the first feature map of the level by the combination header 33, the second feature map of the M level is obtained. The M-level second feature map is processed by inputting it into the network block of each additional header 34 of the regression network and the classification network, and the last level network block (including the convolution layer, the activation layer, the fully connected layer, etc.). Realize the regression task and the classification task with. The additional header 34 of the regression network and the classification network may include at least one convolution layer. Different convolutional parameters can be set for the convolutional layers of the two additional headers 34 according to the subtle differences between the receptive field return task and the classification task, and the present disclosure is not particularly limited thereto.

As shown in FIG. 3b, the regression task acquires the coordinates of the four vertices of the detection frame of the K targets in the image, and the classification task acquires the categories of the K targets in the image (total of C). Set if there is a category). The present disclosure is not particularly limited to the network structure of the additional header 34 network block and the last level network block.

As a result, the image processing network according to the embodiment of the present disclosure can significantly reduce the amount of calculation and can not impair the performance.

In some embodiments of the present disclosure, the image processing network can be trained prior to applying the image processing network according to the embodiments of the present disclosure. That is, by inputting the sample images in the training set into the image processing network and processing them by the feature pyramid network, the P-level fusion network block, the regression network, and the classification network, the sample target detection results of the sample images are acquired, and a plurality of them are obtained. After training, if the network loss is determined according to the difference between the sample target detection result and the labeling result of the sample image of the sample image, the parameters of the image processing network are adjusted according to the network loss, and the training conditions (for example, network convergence) are satisfied. Get the image processing network of. The present disclosure is not particularly limited to the training process.

In some embodiments of the present disclosure, in order to further utilize the correlation of features between adjacent layers of a feature pyramid, a pyramid convolution is proposed as a three-dimensional convolution form, i.e., the dimensional and spatial dimensions of the feature map. At the same time, pay attention to the correlation of. The image processing method according to the embodiment of the present disclosure fuses the relevant information of the features between the adjacent layers of the feature pyramid by spatially large-scale pyramid convolution, and better obtains the correlation between the feature map dimension and the spatial dimension. can do. This solves the problem in the field of object detection that when a feature pyramid extracts features of different scales, it overlooks the correlation of features between adjacent layers and is only interested in semantic information between different layers.

In some embodiments of the present disclosure, combined batch normalization and scale spatial convolution are naturally combined to provide global statistics for all scale feature maps and effectively stabilize the training process. The performance is further improved, which allows batch normalization to be applied even when the batch is small. This solves the problem that batch normalization is not always fully applied in the field of object detection because it is a practical application and accurate statistics cannot be obtained when the data batch is small.

In some embodiments of the present disclosure, in order to reduce the difference between the conventional feature pyramid and the Gaussian pyramid, the image processing method according to the embodiments of the present disclosure may use a deformable convolution instead of the conventional convolution. The network handles the extraction of different scales more reasonably and efficiently by improving the pyramid convolution to a scale-uniform convolution, thereby reducing the difference between the normal feature pyramid and the Gaussian pyramid. To do so. The one-stage detector uses a shared header module to perform further feature extraction, which can significantly reduce the amount of computation, not impair performance, and increase the inference speed. This solves the problem of irrational design of the parameters of the current feature pyramid and shared header modules.

In some embodiments of the present disclosure, the image processing method according to the embodiments of the present disclosure can significantly improve the performance of a one-stage detector with very small speed loss in a data set with large scale changes. It can also detect that a two-stage detector is also effective.

The image processing method according to the embodiment of the present disclosure can be applied to scenes such as object detection and pedestrian detection, and the scene where the scale change of the object is large (for example, the object is located at a short distance and a long distance from the camera). The detection task can be realized, and the detection performance and the detection speed can be improved at the same time.

The embodiments of the above-mentioned methods described in the present disclosure can be combined with each other to generate a combined embodiment without violating the principle and logic, and the number of papers is limited. Please understand that I will not explain it repeatedly.

One of ordinary skill in the art can understand that in the above method of the embodiment, the execution order of each step is determined by their function and possible internal logic.

It should be noted that the present disclosure further provides image processing devices, electronic devices, computer-readable storage media, and programs that can be used to realize any of the image processing methods provided in the present disclosure, and the corresponding technical techniques. The solution and description may refer to the corresponding description of the method portion and will not be repeated herein.

FIG. 4 shows a block diagram of the image processing apparatus according to the embodiment of the present disclosure, and as shown in FIG. 4, the image processing apparatus is
A feature extraction module 41 configured to perform M (M is an integer greater than 1) level feature extraction on the processed image to obtain an M level first feature map of the processed image. The feature extraction module 41 and the feature extraction module 41, which have different scales of the first feature map of each level of the first feature map of the M level,
A scale adjustment and fusion module 42 configured to perform scale adjustment and fusion for each feature map set corresponding to the first feature map of each level to obtain an M level second feature map. , Each of the feature map sets includes a scale adjustment and fusion module 42, including the first feature map and a first feature map adjacent to the first feature map.
A target detection module 43 configured to execute target detection on the M-level second feature map and obtain a target detection result of the processed image is provided.

In some embodiments of the present disclosure, the feature map set corresponding to the first feature map at the i-th (i is an integer and 1 <i <M) level is the first at the i-1 level. The scale adjustment and fusion module includes a feature map, an i-level first feature map and a first i + 1 level first feature map, and the scale adjustment and fusion module scales down the i-1 level first feature map to 1 A first scale reduction submodule configured to obtain a third i-level feature map and a conversion performed so that the scale is not changed for the i-level first feature map, 2 The first conversion submodule configured to obtain the third feature map of the third i-level and the third feature of the third i-level by expanding the scale of the first feature map of the i + 1 level. A first scale magnifying submodule configured to obtain a map, the first i-level third feature map, the second i-level third feature map, and a third i-level. The first fusion submodule configured to fuse with the third feature map of the first i-level to obtain the second feature map of the i-level, and the third feature map of the first i-level. The scales of the second i-level third feature map and the third i-level third feature map are the same. In some embodiments of the present disclosure, the feature map set corresponding to the first level first feature map includes said first level first feature map and second level first feature map, said scale adjustment. And the fusion module is configured to perform a transformation on the first level first feature map so that the scale is not changed to obtain a first level third feature map. The submodule, the second scale expansion submodule configured to expand the scale of the second level first feature map to obtain the second first level third feature map, and the first. A second fusion submodule configured to fuse the first level third feature map and the second first level third feature map to obtain a first level second feature map. In addition, the scales of the first level third feature map and the second first level third feature map are the same. The feature map set corresponding to the first feature map of the M level includes the first feature map of the M-1 level and the first feature map of the M level, and the scale adjustment and fusion module is the M. A second scale reduction submodule configured to scale down a -1 level first feature map to obtain a first M level third feature map, and the M level first feature map. A third conversion submodule configured to perform a conversion so that the scale is not changed for the second M level to obtain a third feature map of the second M level, and a third of the first M level. The first feature map comprises a third fusion submodule configured to fuse the feature map with the second M level second feature map to obtain a second M level feature map. The scales of the third feature map of the second M level and the third feature map of the second M level are the same. In some embodiments of the present disclosure, the first scale reduction submodule arrangement convolves the first feature map of the i-1 level with the first convolution layer and is of the first i level. It is configured to obtain a third feature map, the size of the convolution kernel of the first convolution layer is N × N (N is an integer greater than 1), and the step size is n (n is greater than 1). The scale of the first feature map of the i-1 level is n times the scale of the first feature map of the i-level, and the first conversion submodule is the second convolution. The layer is configured to convolve the i-level first feature map to obtain the second i-level third feature map, and the size of the convolution kernel of the second convolution layer is N × N. Yes, the step size is 1, and the 1st scale expansion submodule convolves and upsamples the 1st feature map of the i + 1 level by the 3rd convolution layer and the upsampling layer, and the 3rd. The size of the convolution kernel of the third convolution layer is N × N, and the step size is 1. In some embodiments of the present disclosure, the second conversion submodule convolves the first level first feature map with the second convolution layer to obtain the first level third feature map. Configured to obtain, the size of the convolution kernel of the second convolution layer is N × N (N is an integer greater than 1), the step size is 1, and the second scale expansion submodule is The third convolution layer and the upsampling layer are configured to convolve and upsample the second level first feature map to obtain the second first level third feature map. The size of the convolution kernel of the convolution layer is N × N, and the step size is 1. In some embodiments of the present disclosure, the second scale reduction submodule convolves the first feature map of the first M-1 level with the first convolution layer and the first M level. It is configured to obtain a three-feature map, the size of the convolution kernel of the first convolution layer is N × N (N is an integer greater than 1), and the step size is n (n is an integer greater than 1). The scale of the first feature map of the i-1 level is n times the scale of the first feature map of the i-level, and the third conversion submodule is the second convolution layer. Convolves the first feature map of the M level to obtain the third feature map of the second M level, and the size of the convolution kernel of the second convolution layer is N × N. , The step size is 1. In some embodiments of the present disclosure, the second convolution layer and the third convolution layer include a deformable convolution layer or an expansion convolution layer. In some embodiments of the present disclosure, the image processing apparatus is implemented by an image processing network, which performs scale adjustment and fusion P times with respect to the M-level first feature map. Containing series-connected P (P is a positive integer) level fusion network block for each level of fusion network block, a plurality of first convolution layers, a plurality of second convolution layers and a plurality of first convolution layers. The scale adjustment and fusion module, which includes three convolutional layers, inputs the M-level first feature map into the first-level fusion network block and outputs the first-fused M-level fourth feature map. The j-level fusion of the first fusion submodule configured to do so and the j-1 (j is an integer and 1 <j <P) th-time fused M-level fourth feature map. A second fusion submodule configured to input to the network block and output the jth fused M-level fourth feature map, and a P-1st fused M-level fourth feature map. Is provided in a third fusion submodule configured to input the P-level fusion network block and output the M-level second feature map. In some embodiments of the present disclosure, the fusion network block at each level further comprises a normalization layer, wherein the second fusion submodule is a first convolution layer, a second convolution layer of the j-level fusion network block. And, by the third convolution layer, scale adjustment and fusion were performed for the feature map set corresponding to the M-level fourth feature map fused in the j-1th time, respectively, and the fusion was performed in the jth time. An M-level intermediate feature map is obtained, and a union batch normalization process is executed on the M-level intermediate feature map fused by the normalization layer at the jth time, and the M-level intermediate feature map fused at the jth time is executed. It is configured to obtain a fourth feature map. In some embodiments of the present disclosure, the image processing apparatus is realized by an image processing network, the image processing network further includes a regression network and a classification network, and the target detection module is the M-level second. The feature map is input to the regression network, and the regression submodule configured to determine the image frame corresponding to the target in the processed image and the M-level second feature map are input to the classification network. The classification submodule is configured to determine the category of the target in the image to be processed, and the target detection result includes the image frame corresponding to the target and the category of the target. Equipped with a module.

In some embodiments, the functions or modules of the apparatus provided in the embodiments of the present disclosure can be configured to perform the methods described in the embodiments of the image processing methods described above, and the realization thereof. Can refer to the description of the embodiment of the image processing method described above, and is not repeated here for the sake of brevity.

The embodiments of the present disclosure further provide a computer-readable storage medium in which computer program instructions are stored, and realize the above-mentioned image processing method when the computer program instructions are executed by a processor. The computer-readable storage medium may be a volatile computer-readable storage medium or a non-volatile computer-readable storage medium. The embodiments of the present disclosure further propose electronic devices, wherein the electronic device comprises a processor and a memory configured to store processor executable instructions, wherein the processor is stored by the memory. It is configured to call the issued instruction to execute the above image processing method. The embodiments of the present disclosure further provide a computer program product containing computer readable code, and when the computer readable code is executed in the device, the processor in the device is according to any one of the above embodiments. Execute a command to realize the provided image processing method. The embodiments of the present disclosure further provide another computer program product, wherein the computer program product is configured to store a computer-readable instruction, and when the instruction is executed, the computer implements any one of the above. Make sure to perform the operation of the image processing method provided in the example. The electronic device can be provided as a terminal, a server or other form of device.

FIG. 5 shows a block diagram of the electronic device 800 according to the embodiment of the present disclosure. For example, the electronic device 800 may be a terminal such as a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, or a mobile information terminal.

Referring to FIG. 5, the electronic device 800 includes processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816. It can be equipped with one or more of the components.

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations relating to displays, telephone calls, data communications, camera operations and recording operations. The processing component 802 may include one or more processors 820 for executing instructions to perform all or part of the steps of the image processing method described above. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and the other components. For example, the processing component 802 may include a multimedia module for facilitating the interaction between the multimedia component 808 and the processing component 802. The memory 804 is configured to store various types of data to support operations in the electronic device 800. Examples of these data include instructions, contact data, phonebook data, messages, photographs, videos, etc. of any application or method running on the electronic device 800. Memory 804 includes static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), and programmable read-only memory (PROM). It can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The power component 806 provides power to various components of the electronic device 800. The power component 806 can include a power management system, one or more power sources, and other components related to the generation, management, and distribution of power for the electronic device 800. The multimedia component 808 includes a screen provided as an output interface between the electronic device 800 and the user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen comprises a touch panel, the screen can be implemented as a touch screen for receiving input signals from the user. The touch panel comprises one or more touch sensors for detecting touches, swipes and gestures on the touch panel. The touch sensor can not only detect the boundaries of the touch or swipe operation, but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 comprises one front camera and / or rear camera. When the electronic device 800 is in an operating mode such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or may have a focal length and an optical zoom function. The audio component 810 is configured to output and / or input an audio signal. For example, the audio component 810 comprises one microphone (MIC), and the microphone is configured to receive an external audio signal when the electronic device 800 is in an operating mode such as a call mode, a recording mode and a voice recognition mode. To. The received audio signal may be further stored in memory 804 or may be transmitted via the communication component 816. In some embodiments, the audio component 810 further comprises a speaker for outputting an audio signal. The I / O interface 812 provides an interface between the processing component 802 and the peripheral interface module, which peripheral interface module may be a keyboard, click wheel, buttons, or the like. These buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button. The sensor component 814 comprises one or more sensors for providing the electronic device 800 with a state assessment of each aspect. For example, the sensor component 814 can detect the on / off state of the electronic device 800 and the relative position of the component such as the display or keypad of the electronic device 800, and the sensor component 814 can also detect the electronic device 800 or the electronic device 800 or. It is also possible to detect a change in the position of a component of the electronic device 800, the presence or absence of contact of the electronic device 800 with the user, the orientation or acceleration / deceleration of the electronic device 800, and the change in the temperature of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without physical contact. The sensor component 814 can also further include an optical sensor such as a CMOS or CCD image sensor for use in imaging applications. In some embodiments, the sensor component 814 can further include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor. The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further comprises a Near Field Communication (NFC) module to facilitate short range communication. For example, NFC modules can be represented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies. In an exemplary embodiment, the electronic device 800 is configured to perform the image processing method described above, one or more integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing. It can be embodied by an integrated circuit (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic element. In an exemplary embodiment, a computer-readable storage medium, such as a memory 804 containing computer program instructions, is further provided, wherein the computer program instructions are executed by the processor 820 of the electronic device 800, whereby the image processing method described above. Can be carried out.

FIG. 6 shows a block diagram of the electronic device 1900 according to the embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server. Referring to FIG. 6, the electronic device 1900 is represented as a processing component 1922 including one or more processors and a memory resource configured to store instructions (such as an application) that can be executed by the processing component 1922. It includes a memory 1932. An application stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Further, the processing component 1922 is configured to execute the above image processing method by executing an instruction.

The electronic device 1900 also has an input / output with a power component 1926 configured to perform power management of the electronic device 1900 and a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network. (I / O) interface 1958 and can be provided. The electronic device 1900 can be operated on the basis of an operating system stored in memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like. In an exemplary embodiment, a non-volatile computer readable storage medium, such as a memory 1932 containing computer program instructions, is further provided and the above computer program instructions are executed by the processing component 1922 of the electronic device 1900 to perform the above image. The processing method can be carried out.

The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer readable storage medium, which includes computer readable program instructions for the processor to realize various aspects of the embodiments of the present disclosure.

The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by the instruction executing device. The computer-readable storage medium can be, for example, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination described above, but is not limited thereto. Examples of computer-readable storage media (non-exhaustive list) are portable computer disksets, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access. Memory (SRAM), Portable Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD), Memory Stick, Floppy Disk, Punch Card or Groove Convex Structure with Instructions Stored, and any of the above-mentioned suitable. Includes mechanical coding equipment such as combinations. The computer-readable storage medium used here is radio waves, other electromagnetic waves propagating freely, electromagnetic waves propagating through waveguides or other propagating media (such as optical pulses through fiber optic cables), or wires. It should not be interpreted as a temporary signal, such as an electronic signal transmitted via.

The computer-readable program instructions described herein are downloaded from a computer-readable storage medium to each computing / processing device or via a network such as the Internet, local area networks, wide area networks and / or wireless networks to external computers. Alternatively, it can be downloaded to an external storage device. The network may include copper wire transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and / or edge servers and the like. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and issues the computer-readable program instructions for storage in the computer-readable storage medium of other computing / processing equipment. Forward.

The computer programming instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcodes, firmware instructions, state setting data, or one or more programmings. The programming language may be source code or target code programmed in any combination of languages, such as object-oriented programming languages such as Smalltalk, C ++, and common programming languages such as "C" language or similar programming languages. Includes procedural programming languages. Computer-readable program instructions may be executed entirely on the user's computer, some may be executed on the user's computer, or some may be executed as a single independent software package, some of which may be executed by the user. It may be run on one computer and other parts may be run on a remote computer, or it may be run entirely on a remote computer or server. In the case of a remote computer, the remote computer may connect to the user's computer or connect to an external computer via any type of network, including a local area network (LAN) or wide area network (WAN). Yes (for example, you can use an internet service provider to access an external computer over the internet). In some embodiments, computer-readable instruction state information is used to customize an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA) or programmable logic array (PLA). , Computer-readable program instructions can be executed, thereby realizing each aspect of the present disclosure.

Here, each aspect of the present disclosure has been described with reference to the flowcharts and / or blocks of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure, but each block of the flowchart and / or block diagram has been described. It should be understood that each block combination of the flowchart and / or the block diagram can be realized by computer-readable program instructions.

These computer-readable program instructions can be provided to the processor of a general purpose computer, dedicated computer or other programmable data processing device, whereby these instructions can be provided to the processor of the computer or other programmable data processing device. When executed by, it creates a means to realize the function / operation specified by one or more blocks in the flowchart and / or the block diagram. These computer-readable program instructions may be stored in a computer-readable storage medium, and the computer, programmable data processing device and / or other device may operate in a particular manner in response to these instructions. Thus, the computer-readable medium in which the instructions are stored can include a product that includes instructions for each aspect of the function / operation specified in one or more blocks in the flowchart and / or block diagram.

Also, by loading computer-readable program instructions into a computer, other programmable data processing device, or other device, the computer, programmable number of processing devices, or other device can perform a series of operational steps. Thereby generating a process realized by a computer, thereby one or more in a flowchart and / or a block diagram by instructions executed by a computer, other programmable number processing device, or other equipment. The function / operation specified by the block can be realized.

The flowcharts and block diagrams in the accompanying drawings show the feasible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram can represent a part of a module, program segment or instruction, the module, program segment or part of the instruction to realize a specified logical function. Includes one or more executable instructions of. In some alternative implementations, the functions displayed in blocks can also be performed in a different order than they are displayed in the drawing. For example, two consecutive blocks can actually be executed in parallel and, depending on the related functions, in reverse order. Each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated system based on the hardware performing the specified function or operation, or with dedicated hardware. Note that this can be achieved by a combination of computer instructions.

The computer program product can be realized by hardware, software or a combination thereof. In an exemplary embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a software product, such as a software development kit (SDK). ..

Although each embodiment of the present disclosure has been described above, the above description is exemplary, not exhaustive, and is not limited to each disclosed embodiment. Many modifications and changes will be apparent to those of skill in the art without departing from the scope and spirit of each of the embodiments described. The choice of terminology used herein best describes the principles of each embodiment, actual application or technical improvement in the market, or one of ordinary skill in the art will understand each embodiment disclosed herein. Intended to be able to.

In the embodiment of the present disclosure, M-level feature extraction is performed on the image to be processed to obtain an M-level first feature map, and each first feature map and an adjacent feature map are fused to M. The second feature map of the level can be obtained, the target detection can be performed on the second feature map of the M level to obtain the target detection result, whereby the features between the adjacent layers of the first feature map of the M level can be obtained. By fusing related information, the effect of target detection can be effectively improved.

Claims (25)

It ’s an image processing method.
Performing M (M is an integer larger than 1) level feature extraction on the processed image to obtain the M level first feature map of the processed image, that is, the M level. The scale of the first feature map at each level of the first feature map is different,
Scale adjustment and fusion are performed on the feature map sets corresponding to the first feature maps of each level to obtain the second feature map of the M level, and each of the feature map sets is described above. It includes a first feature map and a first feature map adjacent to the first feature map.
Including performing target detection on the M-level second feature map to obtain a target detection result of the processed image.
Image processing method. The feature map set corresponding to the first feature map of the i-th (i is an integer and 1 <i <M) level is the first feature map of the i-1 level and the first feature of the i-level. Includes map and 1st feature map for i + 1 level
Performing scale adjustment and fusion on the feature map set corresponding to the first feature map of each level to obtain the second feature map of M level can be obtained.
Reducing the scale of the first feature map of the first i-1 level to obtain the third feature map of the first i level,
Performing a transformation on the i-level first feature map so that the scale is not changed to obtain a second i-level third feature map.
By expanding the scale of the first feature map of the first i + 1 level to obtain the third feature map of the third i level,
The first i-level third feature map, the second i-level third feature map, and the third i-level third feature map are fused to form the i-level. Including getting a second feature map
The scales of the first i-level third feature map, the second i-level third feature map, and the third i-level third feature map are the same.
The image processing method according to claim 1. The feature map set corresponding to the first level first feature map includes the first level first feature map and the second level first feature map, and the feature map corresponding to the first level first feature map. Performing scale adjustment and fusion for each pair to obtain an M-level second feature map is
To obtain the first level third feature map by performing a transformation so that the scale is not changed with respect to the first level first feature map.
Enlarging the scale of the first level first feature map to obtain the second first level third feature map,
Including fusing the first level third feature map and the second first level third feature map to obtain a first level second feature map.
The scales of the first level third feature map and the second first level third feature map are the same.
The image processing method according to claim 1 or 2. The feature map set corresponding to the first feature map of the M level includes the first feature map of the M-1 level and the first feature map of the M level.
Performing scale adjustment and fusion on the feature map set corresponding to the first feature map of each level to obtain the second feature map of M level can be obtained.
By reducing the scale of the first feature map of the first M-1 level to obtain the third feature map of the first M level,
To obtain the second M-level third feature map by performing a transformation so that the scale is not changed for the first M-level feature map.
Including fusing the first M-level third feature map and the second M-level second feature map to obtain a second M-level feature map.
The scales of the first M-level third feature map and the second M-level third feature map are the same.
The image processing method according to any one of claims 1 to 3. Reducing the scale of the first feature map of the first i-1 level to obtain the third feature map of the first i level
The size of the convolution kernel of the first convolution layer, including convolving the first feature map of the i-1 level with the first convolution layer to obtain the third feature map of the first i level. Is N × N (N is an integer greater than 1), the step size is n (n is an integer greater than 1), and the scale of the first feature map of the i-1 level is It is n times the scale of the first feature map of the i-th level.
Performing a transformation on the i-level first feature map so that the scale is not changed to obtain a second i-level third feature map can be achieved.
The size of the convolution kernel of the second convolution layer is N ×, including convolving the first feature map of the i-level with the second convolution layer to obtain the third feature map of the second i-level. N, the step size is 1,
Enlarging the scale of the first feature map of the first i + 1 level to obtain the third feature map of the third i level
The third convolution layer and the upsampling layer include convolving and upsampling the first feature map of the i + 1 level to obtain the third feature map of the third i level. The size of the convolution kernel of the convolution layer is N × N, and the step size is 1.
The image processing method according to any one of claims 2 to 4. Performing a transformation on the first level first feature map so that the scale is not changed to obtain the first level third feature map is
The size of the convolution kernel of the second convolution layer is N ×, including convolving the first level first feature map with the second convolution layer to obtain the first first level third feature map. N (N is an integer greater than 1), the step size is 1, and
Enlarging the scale of the first level first feature map to obtain the second first level third feature map
The third convolution layer and the upsampling layer include convolving and upsampling the second level first feature map to obtain a second first level third feature map, the third convolution. The size of the layer convolution kernel is N × N and the step size is 1.
The image processing method according to claim 3. Reducing the scale of the first feature map of the first M-1 level to obtain the third feature map of the first M level is possible.
The size of the convolution kernel of the first convolution layer, including convolving the first feature map of the M-1 level with the first convolution layer to obtain the third feature map of the first M level. Is N × N (N is an integer greater than 1), the step size is n (n is an integer greater than 1), and the scale of the first feature map of the i-1 level is , N times the scale of the first feature map of the i-th level.
Performing a transformation on the M-level first feature map so that the scale is not changed to obtain a second M-level third feature map is
The size of the convolution kernel of the second convolution layer is N ×, including convolving the first feature map of the M level with the second convolution layer to obtain the third feature map of the second M level. N, step size is 1,
The image processing method according to claim 4. The second convolution layer and the third convolution layer include a deformable convolution layer or an expansion convolution layer.
The image processing method according to any one of claims 5 to 7. The image processing method is realized by an image processing network, and the image processing network is connected in series to P (P) for performing scale adjustment and fusion P times with respect to the M-level first feature map. Contains level fusion network blocks (where is a positive integer), and each level fusion network block comprises a plurality of first convolution layers, a plurality of second convolution layers, and a plurality of third convolution layers.
Performing scale adjustment and fusion on the feature map set corresponding to the first feature map of each level to obtain the second feature map of M level can be obtained.
The first M-level feature map is input to the first-level fusion network block, and the first fused M-level fourth feature map is output.
The M-level fourth feature map fused at the j-1 (j is an integer and 1 <j <P) th time is input to the j-level fusion network block, and the fusion is performed at the jth time. Outputting the 4th feature map of M level and
P-1 The M-level fourth feature map fused for the first time is input to the P-level fusion network block, and the M-level second feature map is output.
The image processing method according to any one of claims 5 to 8. Each level of fusion network block further contains a normalization layer,
Inputting the j-1st fused M-level fourth feature map into the j-level fusion network block and outputting the j-1st fused M-level fourth feature map is possible.
For the feature map set corresponding to the M-level fourth feature map fused at the j-1st time by the first convolution layer, the second convolution layer, and the third convolution layer of the j-level fusion network block. Performing scale adjustment and fusion, respectively, to obtain the M-level intermediate feature map fused at the jth time,
The combined batch normalization process is executed on the M-level intermediate feature map fused by the normalization layer at the jth time to obtain the M-level fourth feature map fused at the jth time. including,
The image processing method according to claim 9. The image processing method is realized by an image processing network, which further includes a regression network and a classification network, performs target detection on the M-level second feature map, and performs the processing target image. To obtain the target detection result of
The second feature map of the M level is input to the regression network to determine the image frame corresponding to the target in the image to be processed.
The second feature map of the M level is input to the classification network to determine the category of the target in the image to be processed, and the target detection result is the image frame corresponding to the target and the target. Including, including, including,
The image processing method according to any one of claims 1 to 10. It is an image processing device
A feature extraction module configured to perform M (M is an integer greater than 1) level feature extraction on the processed image to obtain the M level first feature map of the processed image. Therefore, the scale of the first feature map of each level of the first feature map of the M level is different from that of the feature extraction module.
It is a scale adjustment and fusion module configured to perform scale adjustment and fusion for each feature map set corresponding to the first feature map of each level to obtain an M level second feature map. Each of the feature map sets includes a scale adjustment and fusion module comprising the first feature map and a first feature map adjacent to the first feature map.
A target detection module configured to execute target detection on the M-level second feature map and obtain a target detection result of the processed image is provided.
Image processing device. The feature map set corresponding to the first feature map of the i-th (i is an integer and 1 <i <M) level is the first feature map of the i-1 level and the first feature of the i-level. Includes map and 1st feature map for i + 1 level
The scale adjustment and fusion module is a first scale reduction submodule configured to reduce the scale of the i-1 level first feature map to obtain a first i level third feature map. And a first conversion submodule configured to perform a transformation on the i-level first feature map so that the scale is not changed to obtain a second i-level third feature map. The first scale expansion submodule configured to expand the scale of the first feature map of the i + 1 level to obtain the third feature map of the third i level, and the first i level. The third feature map of the third level, the third feature map of the second i-level, and the third feature map of the third i-level are fused to obtain the second feature map of the i-level. The first fusion submodule configured in the above, the first i-level third feature map, the second i-level third feature map, and the third i-level third feature map. The scale is the same as the 3 feature maps,
The image processing apparatus according to claim 12. The feature map set corresponding to the first level first feature map includes the first level first feature map and the second level first feature map, and the scale adjustment and fusion module is the first level. A second conversion submodule configured to perform a transformation so that the scale is not changed for the first feature map to obtain a first level third feature map, and the second level second. The second scale expansion submodule configured to magnify the scale of one feature map to obtain the second first level third feature map, and the first level third feature map and said. It comprises a second fusion submodule configured to fuse with a second first level third feature map to obtain a first level second feature map, the first level said. The scales of the third feature map and the third feature map of the second first level are the same.
The image processing apparatus according to claim 12 or 13. The feature map set corresponding to the first feature map of the M level includes the first feature map of the M-1 level and the first feature map of the M level, and the scale adjustment and fusion module is the M. A second scale reduction submodule configured to scale down a -1 level first feature map to obtain a first M level third feature map, and the M level first feature map. A third conversion submodule configured to perform a conversion so that the scale is not changed for the second M level to obtain a third feature map of the second M level, and a third of the first M level. The first feature map comprises a third fusion submodule configured to fuse the feature map with the second M level second feature map to obtain a second M level feature map. The scales of the third feature map of the second M level and the third feature map of the second M level are the same.
The image processing apparatus according to any one of claims 12 to 14. The first scale reduction submodule is configured to convolve the first level i-1 feature map with the first convolution layer to obtain the first level third feature map. The size of the convolution kernel of the first convolution layer is N × N (N is an integer greater than 1), the step size is n (n is an integer greater than 1), and the i-. The scale of the 1st level 1st feature map is n times the scale of the 1st level 1 feature map, and the 1st transformation submodule is the 1st feature map of the ist level by the 2nd convolution layer. Is configured to convolve to obtain the third feature map of the second i-level, the size of the convolution kernel of the second convolution layer is N × N, the step size is 1, and the first. The one-scale expansion submodule convolves and upsamples the first feature map of the i + 1 level by the third convolution layer and the upsampling layer to obtain the third feature map of the third i level. The size of the convolution kernel of the third convolution layer is N × N, and the step size is 1.
The image processing apparatus according to any one of claims 13 to 15. The second conversion submodule is configured to fold the first level first feature map with the second convolution layer to obtain the first level third feature map, the second convolution. The size of the layer convolution kernel is N × N (N is an integer greater than 1), the step size is 1, and the second scale expansion submodule is provided by the third convolution layer and the upsampling layer. It is configured to convolve and upsample the first level first feature map to obtain the second first level third feature map, and the size of the convolution kernel of the third convolution layer is N ×. N, and the step size is 1.
The image processing apparatus according to claim 15. The second scale reduction submodule is configured to convolve the first M-1 level feature map with the first convolution layer to obtain the first M level third feature map. The size of the convolution kernel of the first convolution layer is N × N (N is an integer greater than 1), the step size is n (n is an integer greater than 1), and the i-. The scale of the 1st level 1st feature map is n times the scale of the ist level 1st feature map, and the 3rd transformation submodule is the 1st feature map of the M level by the 2nd convolution layer. Is configured to convolve to obtain the second M-level third feature map, the size of the convolution kernel of the second convolution layer is N × N, and the step size is 1.
The image processing apparatus according to claim 16. The second convolution layer and the third convolution layer include a deformable convolution layer or an expansion convolution layer.
The image processing apparatus according to any one of claims 16 to 18. The image processing apparatus is realized by an image processing network, and the image processing network is connected in series to P (P) for performing scale adjustment and fusion P times with respect to the M-level first feature map. Contains level fusion network blocks (where is a positive integer), each level of fusion network block comprising a plurality of first convolutional layers, a plurality of second convolutional layers and a plurality of third convolutional layers, said scale adjustment and. The fusion module is configured to input the M-level first feature map into the first-level fusion network block and output the first-fused M-level fourth feature map. Input the module and the M-level fourth feature map fused to the j-1 (j is an integer and 1 <j <P) th time into the j-level fusion network block, and the jth time. Input the 2nd fusion submodule configured to output the fused M-level 4th feature map and the P-1st fused M-level 4th feature map into the P-level fusion network block. A third fusion submodule configured to output the M-level second feature map.
The image processing apparatus according to any one of claims 16 to 19. The fusion network block at each level further comprises a normalization layer, and the second fusion submodule is the j- by the first convolution layer, the second convolution layer and the third convolution layer of the j-level fusion network block. Scale adjustment and fusion are performed on the feature map sets corresponding to the first fused M-level fourth feature map, respectively, to obtain the j-th fused M-level intermediate feature map. It is configured to execute the union batch normalization process on the M-level intermediate feature map fused at the jth time by the normalization layer to obtain the M-level fourth feature map fused at the jth time. ,
The image processing apparatus according to claim 20. The image processing apparatus is realized by an image processing network, the image processing network further includes a regression network and a classification network, and the target detection module inputs the M-level second feature map into the regression network. , The regression submodule configured to determine the image frame corresponding to the target in the processed image and the M-level second feature map are input to the classification network to enter the target in the processed image. A classification submodule configured to determine a category of, wherein the target detection result comprises a classification submodule including an image frame corresponding to the target and the target category.
The image processing apparatus according to any one of claims 13 to 21. It ’s an electronic device,
With the processor
With memory configured to store processor executable instructions,
The processor is configured to call an instruction stored in the memory to execute the image processing method according to any one of claims 1 to 11.
Electronics. A computer-readable storage medium that stores computer program instructions.
The image processing method according to any one of claims 1 to 11 is realized when the computer program instruction is executed by the processor.
Computer-readable storage medium. A computer program product that contains one or more instructions.
The one or more instructions causes the processor to execute the image processing method according to any one of claims 1 to 11.
Computer program product.

Images