CN115100417A - Image processing method, storage medium, and electronic device - Google Patents

Abstract

The invention discloses an image processing method, a storage medium and an electronic device. The method includes: acquiring a target image, wherein the target image includes a target object; performing multi-scale feature extraction on the target image to obtain a plurality of first feature maps; and using a multi-branch network structure to perform feature fusion on the plurality of first feature maps , obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; at least one second feature map is detected to obtain the detection result of the target object. The present invention solves the technical problem of low image detection accuracy in the related art.

Description

Image processing method, storage medium, and electronic device

technical field

The present invention relates to the field of image processing, in particular, to an image processing method, a storage medium and an electronic device.

Background technique

At present, in computer vision tasks, deep learning technology is generally used to detect objects in images. The high complexity of deep learning technology results in low detection efficiency. For low-complexity detection methods, the detection accuracy is high. Poor degree.

For the above problems, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image processing method, a storage medium, and an electronic device, so as to at least solve the technical problem of low image detection accuracy in the related art.

According to an aspect of the embodiments of the present application, an image processing method is provided, including: acquiring a target image, wherein the target image includes a target object; performing multi-scale feature extraction on the target image to obtain a plurality of first feature maps; The branch network structure performs feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; The feature map is detected, and the detection result of the target object is obtained.

According to an aspect of the embodiments of the present application, an image processing method is provided, including: acquiring a target remote sensing image, wherein the target remote sensing image includes a target object; and performing multi-scale feature extraction on the target remote sensing image to obtain a plurality of first feature maps ; use a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on a plurality of first feature maps through a plurality of branches; A second feature map is used for detection to obtain the detection result of the target object.

According to an aspect of the embodiments of the present application, an image processing method is provided, including: acquiring an agricultural remote sensing image, wherein the agricultural remote sensing image includes crops; performing multi-scale feature extraction on the agricultural remote sensing image to obtain a plurality of first feature maps; A multi-branch network structure is used to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; The second feature map is detected to obtain the detection result of the crop.

According to an aspect of the embodiments of the present application, an image processing method is provided, comprising: acquiring a remote sensing image of a building, wherein the remote sensing image of the building includes a target building; and performing multi-scale feature extraction on the remote sensing image of the building to obtain multiple The first feature map; using a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to feature the plurality of first feature maps through multiple branches. Fusion; detecting at least one second feature map to obtain a detection result of the target building.

According to an aspect of the embodiments of the present application, an image processing method is provided, comprising: acquiring a target image by a cloud server, wherein the target image contains a target object; and the cloud server performs multi-scale feature extraction on the target image to obtain a plurality of first features Figure; the cloud server utilizes a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches ; the cloud server detects at least one second feature map to obtain a detection result of the target object.

In the embodiment of the present application, first acquire a target image, wherein the target image includes a target object; perform multi-scale feature extraction on the target image to obtain multiple first feature maps; use a multi-branch network structure to perform multiple first feature maps Feature fusion to obtain at least one second feature map, wherein the multi-branch network structure is used to indicate that feature fusion is performed on multiple first feature maps through multiple branches to obtain at least one second feature map, wherein the multi-branch network structure is used. By performing feature fusion on multiple first feature maps through multiple branches, and detecting at least one second feature map, a detection result of the target object is obtained, thereby improving the accuracy of the detection result of the target object. It is easy to notice that feature fusion can be performed on the first feature map through multiple branches, thereby improving the accuracy of the obtained at least one second feature map, and multiple branches can also reduce the amount of parameters in the fusion process, so that the The efficiency of fusion is improved, thereby solving the technical problem of low image detection accuracy in the related art.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1 is a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing an image processing method according to an embodiment of the present application;

2 is a flowchart of an image processing method according to Embodiment 1 of the present application;

3 is a schematic diagram of a giraffe target detector according to an embodiment of the present application;

4 is a schematic diagram of a detector according to an embodiment of the present application;

5 is a schematic diagram of a multi-branch network structure according to an embodiment of the present application;

6 is a schematic diagram of detection by a target detection model according to an embodiment of the present application;

7 is a schematic diagram of a target detection model training according to an embodiment of the present application;

8 is a flowchart of an image processing method according to Embodiment 2 of the present application;

9 is a flowchart of an image processing method according to Embodiment 3 of the present application;

10 is a flowchart of an image processing method according to Embodiment 4 of the present application;

11 is a flowchart of an image processing method according to Embodiment 5 of the present application;

12 is a schematic diagram of an image processing apparatus according to Embodiment 6 of the present application;

13 is a schematic diagram of an image processing apparatus according to Embodiment 7 of the present application;

14 is a schematic diagram of an image processing apparatus according to Embodiment 8 of the present application;

15 is a schematic diagram of an image processing apparatus according to Embodiment 9 of the present application;

FIG. 16 is a structural block diagram of a computer terminal according to an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

First of all, some nouns or terms that appear in the process of describing the embodiments of the present application are suitable for the following explanations:

GFPN: Generalized-Feature Pyramid Networks, the generalized feature pyramid network structure, acts on the detection feature fusion part, and is used for feature fusion at different scales.

GFocalV2: Generalized Focal Loss V2, a special network structure for matching the target and the predicted value in target detection for feature detection.

YOLOv1 YOLOv2 YOLOv3 YOLOv4 YOLOv5 YOLOX: A series of special image-based detection methods.

At present, image-based object detection is a basic technology in machine vision tasks, and is widely used in remote sensing, security, homeland, water conservancy, retail and other industries. Object detection technology based on deep learning is the current mainstream method, but the computational complexity of this scheme is often very high, and it is difficult to meet practical use.

The present application provides an image processing method, which can improve the accuracy of the detection result while improving the image detection efficiency.

Example 1

According to an embodiment of the present application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although A logical order is shown in the flowcharts, but in some cases steps shown or described may be performed in an order different from that herein.

The method embodiment provided in Embodiment 1 of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. FIG. 1 is a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing an image processing method according to an embodiment of the present application. As shown in FIG. 1 , the computer terminal 10 (or the mobile device 10 ) may include one or more processors (102a, 102b, . A processing device such as an MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, may also include: display, input/output interface (I/O interface), universal serial bus (USB) port (may be included as one of the ports of the BUS bus), network interface, power supply and/or or camera. Those of ordinary skill in the art can understand that the structure shown in FIG. 1 is only a schematic diagram, which does not limit the structure of the above electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .

It should be noted that the one or more processors and/or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuit may be embodied in whole or in part as software, hardware, firmware or any other combination. Furthermore, the data processing circuitry may be a single stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computer terminal 10 (or mobile device). As referred to in the embodiments of the present application, the data processing circuit acts as a kind of processor control (eg, selection of a variable resistance termination path connected to an interface).

The memory 104 can be used to store software programs and modules of the application software, such as a program instruction/data storage device corresponding to the image processing method in the embodiment of the present application. This kind of function application and data processing is to realize the above-mentioned image processing method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, memory 104 may further include memory located remotely from the processor, and these remote memories may be connected to computer terminal 10 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Transmission means 106 are used to receive or transmit data via a network. A specific example of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10 . In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF) module, which is used for wirelessly communicating with the Internet.

The display may be, for example, a touch screen type liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

It should be noted here that, in some optional embodiments, the computer device (or mobile device) shown in FIG. 1 may include hardware elements (including circuits), software elements (including a computer stored on a computer-readable medium) code), or a combination of both hardware and software elements. It should be noted that FIG. 1 is only one example of a specific embodiment, and is intended to illustrate the types of components that may be present in a computer device (or mobile device) as described above.

Under the above operating environment, the present application provides an image processing method as shown in FIG. 2 . FIG. 2 is a flowchart of an image processing method according to Embodiment 1 of the present application. As shown in Figure 2, the method includes:

Step S202, acquiring a target image.

Among them, the target image contains the target object.

The above-mentioned target image may be an image containing the target object to be detected, wherein the target image may be a remote sensing image obtained by a drone and/or a satellite, and the target image may also be an image captured by a photographing device.

The above-mentioned target object may be a specific object to be detected in the target image.

In an agricultural scene, the target image may be an agricultural remote sensing image, and the target object may be a crop to be detected.

In a building scene, the target image may be a remote sensing image of a building, and the target object may be a building to be detected.

In an optional embodiment, in order to better process the target image, the acquired target image may be transmitted to a corresponding processing device for processing, for example, directly transmitted to the user's computer terminal (for example, a laptop computer) , personal computer, etc.) for processing, or transmitted to the cloud server through the user's computer terminal for processing. It should be noted that, since a large amount of computing resources are required to process the target image, in the embodiments of the present application, the processing device is a cloud server as an example for description.

For example, in order to facilitate the user to upload the target image, an interactive interface can be provided to the user, wherein the interactive interface includes controls such as "Select Image", "Upload", "Image Display", etc. The user can click the "Select Image" button to determine the need to upload and upload the target image to the cloud server for processing by clicking the "Upload" button. In addition, in order to facilitate the user to confirm whether the selected target image is the target image that needs to be processed, the selected target image can be displayed in the "Image Display" area. After the user confirms that it is correct, the data can be uploaded by clicking the "Upload" button.

It should be noted that the client and the cloud server can exchange data through a specific interface. The client can pass the description page of the target object selected by the user into the interface function, and use it as a parameter of the interface to realize the description of the target object. The purpose of uploading the page to the cloud server.

Step S204, performing multi-scale feature extraction on the target image to obtain a plurality of first feature maps.

In an optional embodiment, multi-scale feature extraction may be performed on the target image, so as to obtain multiple first feature maps of different scales, and the multiple first feature maps may be fused through a preset feature fusion strategy , to improve the detection accuracy of the feature map.

In another optional embodiment, the feature extraction layer in the giraffe target detector (GiraffeDet) may be used to perform multi-scale feature extraction on the target image to obtain multiple first feature maps, wherein the multiple first feature maps different scales. Optionally, the feature extraction layer includes multiple scaling layers (scales), and each scaling layer corresponds to a different size, and multi-scale feature extraction can be performed on the target image according to the multiple scaling layers to obtain multiple first feature maps.

Step S206, using a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map.

The multi-branch network structure is used to perform feature fusion on a plurality of first feature maps through a plurality of branches.

The above-mentioned multi-branch network structure may include two or more branches, and the specific number of branches may be set by the user, and the number of branches may also be flexibly adjusted according to requirements.

In an optional embodiment, a multi-branch network may be used to repeatedly perform feature fusion on multiple first feature maps according to a preset feature fusion strategy to obtain multiple second feature maps, wherein the feature fusion strategy Feature fusion strategy that can be used by GiraffeDet.

FIG. 3 is a schematic diagram of a giraffe target detector according to an embodiment of the present application. As shown in FIG. 3 , S1-S5 are multiple first feature maps of different scales extracted by the feature extraction layer. The feature maps are fused repeatedly, and the feature maps obtained from the fusion are connected in the form of ^log2N to obtain the feature maps that need to be detected in the end. In the fusion part, the method shown in Fig. 3 can be used to up and down sample feature maps of different sizes, wherein the upward arrow represents the down-sampling process, and the upward arrow represents the up-sampling process. After the first feature map is spliced, it is input into the corresponding multi-branch network structure, and the first feature maps of different sizes are fused through the multi-branch network structure to obtain the fused features, such as S5_0. A feature map and the fused feature continue to be fused, and the final example is S5_N, where N represents the number of stacking, S5_0 in Figure 3 is obtained by the fusion of S4 and S5, and S4_0 is obtained by the fusion of S3, S4 and S5 It is obtained that S5_1 is obtained by the fusion of S5, S5_0 and S4_0, wherein, there is a cross-connection before S5_1 and S5, and the cross-connection is connected in the form of log2 ^N.

In an optional embodiment, the framework for detecting the target object in the target image may be backbone (feature extraction network)-neck (shallow high-resolution detail feature map and low-resolution semantic feature map are fused) -head (detector), where, at present, the calculation ratio of backbone:neck:head is generally 2-4:1:2. Among them, the calculation ratio of the fusion part is small, resulting in low subsequent calculation accuracy. In order to solve this problem Problem, this application adjusts the calculation ratio to 1:5:1, and increases the calculation ratio of the fusion part, which can further improve the detection accuracy.

It should be noted that the above calculation ratio refers to the allocation of calculation volume. Assuming that a total of 100GFLOPs of calculation volume is required, then the calculation of the backbone part will probably account for 2/5*100=40GFLOPS; the specific calculation is the corresponding convolution in the neural network. such a standard operation.

Step S208: Detect at least one second feature map to obtain a detection result of the target object.

The above detection result can be a target detection frame, wherein the target detection frame is used to mark the target object in the target image; the above detection result can also include the category of the target object, wherein the category of the target object can be marked in the target detection. next to the box or elsewhere as specified.

In an optional embodiment, at least one second feature map can be obtained by fusion according to multiple first feature maps, and different detectors can be used to detect the second feature maps of different scales. The detector structure is the same, that is, the length, width and height corresponding to different detectors are the same, but the parameters in the detectors are different. Because the accuracy of the second feature map of different scales is different, the Detecting the second feature map can further improve the detection effect of the feature map of the scale, thereby improving the detection accuracy of the feature map of the scale. The parameters of the detector can be adjusted to make the detection accuracy of the detector higher, so as to detect the second feature map with higher accuracy, thereby improving the accuracy of detection, and the parameters of the detector can be adjusted to make the detection accuracy of the detector lower , so as to detect the second feature map with lower accuracy, thereby improving the detection accuracy. Through this setting, the detection accuracy can be higher under the same calculation amount.

In another optional embodiment, GFOcalV2 Head (detection algorithm) is used as the basic detector structure.

FIG. 4 is a schematic diagram of a detector according to an embodiment of the present application. The detector shown on the left is the detector used by the target, and the same detector is used to detect the feature maps of different scales; the detector shown on the right The detector is a method of using a detector in the present application, and for feature maps of different scales, detectors with different parameters but the same structure are used for detection, thereby improving the detection accuracy.

In another optional embodiment, a target image may be acquired, wherein the target image includes a target object; a feature extraction layer in the target detection model may be used to perform multi-scale feature extraction on the target image to obtain multiple first feature maps ; The multi-branch network structure in the target detection model can be used to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used for multiple first feature maps through multiple branches. Perform feature fusion; the detector in the target detection model can be used to detect at least one second feature map to obtain a detection result of the target object.

Through the above steps, first obtain a target image, wherein the target image contains the target object; perform multi-scale feature extraction on the target image to obtain a plurality of first feature maps; use a multi-branch network structure to perform feature fusion on the plurality of first feature maps, Obtain at least one second feature map, wherein the multi-branch network structure is used to represent feature fusion of multiple first feature maps through multiple branches to obtain at least one second feature map, wherein the multi-branch network structure is used to obtain at least one second feature map through multiple branches. Each branch performs feature fusion on a plurality of first feature maps, and detects at least one second feature map to obtain a detection result of the target object, thereby improving the accuracy of the detection result of the target object. It is easy to notice that feature fusion can be performed on the first feature map through multiple branches, thereby improving the accuracy of the obtained at least one second feature map, and multiple branches can also reduce the amount of parameters in the fusion process, so that the The efficiency of fusion is improved, thereby solving the technical problem of low image detection accuracy in the related art.

In the above embodiments of the present application, the multi-branch network structure includes: a first branch and a second branch, wherein the output of the first branch is connected to the output of the second branch.

The above-mentioned first branch may include a 1×1 convolutional layer, and the above-mentioned second branch may include N 1×1 and 3×3 convolutional blocks, wherein the N convolutional blocks can be connected in sequence. . It should be noted that 3×3 in the second branch can be replaced with other convolutional layers, for example, 3×3 can be replaced with 5×5, but not limited to this.

In an optional embodiment, the multi-branch network structure may include: a first convolution layer, a first branch, a second branch, and the output of the first convolution layer is connected to the input of the first branch and the second branch, The output of the first branch is connected to the output of the second branch.

The above-mentioned first convolutional layer may be 1×1.

FIG. 5 is a schematic diagram of a multi-branch network structure according to an embodiment of the present application. As shown in FIG. 5 , a plurality of first feature maps to be merged can be processed and then merged to obtain a merged feature map. The merged feature maps can be respectively The input is processed into the first branch and the second branch to obtain two output feature maps output by the two branches, and the two output feature maps can be spliced to obtain the second feature map.

In another optional embodiment, the multi-branch network structure may further include multiple branches, which is not limited here, wherein the first branch may include multiple sub-branches, and the second branch may also include multiple sub-branches .

In the above embodiments of the present application, using a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map includes: performing channel merging on multiple first feature maps to obtain a combined feature map; using The first branch performs convolution processing on the merged feature map to obtain the first output feature; the second branch performs convolution processing on the merged feature map to obtain the second output feature; the first output feature and the second output feature are channel merged , obtain at least one second feature map.

The above-mentioned first branch may include convolution layers with different convolution kernels, and the combined feature maps are processed by convolution layers corresponding to convolution kernels of different sizes, which can reduce the calculation amount of the convolution operation, thereby improving the fusion efficiency. efficiency.

In an optional embodiment, the first convolutional layer may be used to directly combine the channels of the plurality of first feature maps to obtain the above-mentioned combined feature maps. It is also possible to use the first convolution layer to perform convolution operations on multiple first feature maps to obtain multiple third feature maps, so that the sizes of multiple first feature maps can be unified, which is convenient for the subsequent merging process. The channels of multiple third feature maps are merged to achieve the effect of splicing multiple third feature maps to obtain a merged feature map, and the first branch can be used to perform convolution processing on the merged feature map to obtain the first output feature, The second branch may be used to perform convolution processing on the merged feature to obtain a second output feature, and the channels of the first output feature and the second output feature may be merged to obtain a second feature map.

Further, the second feature map can be used as the first feature map, and a multi-branch network can be used to continue to process multiple first feature maps to obtain a new second feature map. A plurality of second feature maps are obtained.

In the above embodiments of the present application, the first branch includes: at least one convolution block, each convolution block in the multiple convolution blocks includes multiple sub-convolutional layers, and the convolution kernels of the multiple sub-convolutional layers are different.

In an optional embodiment, the number of convolution blocks may be determined according to the number of input first feature maps. If the number of input first feature maps is N, the number of convolution blocks is N.

The sizes of the convolution kernels of the above-mentioned multiple sub-convolutional layers are different, wherein the convolution kernels corresponding to the convolutional layers located in front of the convolutional blocks in the multiple sub-convolutional layers may be smaller than those corresponding to the convolutional layers located behind the convolutional blocks. convolution kernel.

The above-mentioned multiple sub-convolutional layers can be 1×1 and 3×3, respectively.

The multiple sub-convolutional layers included in the aforementioned multiple convolutional blocks may be the same. The multiple sub-convolutional layers included in the aforementioned multiple convolutional blocks may be different.

In the above embodiments of the present application, using the first branch to perform convolution processing on the merged feature map to obtain the first output feature includes: using at least one convolution block to perform convolution processing on the merged feature map to obtain the first output feature.

The above-mentioned convolution block includes a first sub-convolution layer and a second sub-convolution layer, wherein the first sub-convolution layer can be a convolution layer with a smaller convolution kernel, and the second sub-convolution layer can be Convolutional layers with larger convolution kernels. The first subconvolutional layer can be 1×1, and the second subconvolutional layer can be 3×3.

In an optional embodiment, when a convolution block is included, the first sub-convolution layer in the convolution block can be used to perform a convolution operation on the merged feature to obtain a processing result. Input into the second sub-convolutional layer, and use the second sub-convolutional layer to perform convolution operation on the processing result to obtain the first output feature.

In the above embodiments of the present application, using the second branch to perform convolution processing on the combined feature map to obtain the second output feature includes: using three convolution layers to perform convolution processing on the combined feature map to obtain the second output feature.

The above-mentioned third convolutional layer may be 1×1.

In the above-mentioned embodiment of the present application, the method further includes: using a target detection model to detect the target image to obtain a detection result of the target object, wherein the target detection model is obtained by training based on the target sample image, and the target sample image is obtained through the sample detection frame. A sample image is obtained by data augmentation.

In an optional embodiment, the target image can be input into the target detection model, and the target image can be detected by using the target detection model to obtain the detection result of the target object. Among them, the main framework of the target detection model can be backbone-neck-head, and the backbone can be used to extract multi-scale features of the target image to obtain multiple first feature maps, and then use the neck to use the multi-branch network structure to perform multiple first feature maps. Feature fusion is performed on the feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; finally, the head is used to detect at least one second feature map, Get the detection result of the target object.

In an optional embodiment, the target detection model may be a commonly used detection model, and the sample images sampled for training are different from the conventionally used sample images, mainly by performing data enhancement on multiple sample images through the sample detection frame , get the target sample image.

In another optional embodiment, the target detection model may include a feature extraction layer, a multi-branch network structure, and a detection layer, wherein the feature extraction layer is used to perform multi-scale feature extraction on the target image to obtain multiple The first feature map, the multi-branch network structure is used to fuse multiple first feature maps to obtain at least one second feature map, and the detection layer is used to detect the at least one second feature map to obtain the detection result of the target object. The multi-branch network structure may include a first branch and a second branch, wherein the output of the first branch is connected to the output of the second branch.

FIG. 6 is a schematic diagram of detection by a target detection model according to an embodiment of the present application. The target image to be detected can be input into the target detection model, and the target detection model can output the detection result of the target object in the target image. The result is obtained by annotating the target object through the target detection frame.

In the above-mentioned embodiments of the present application, the method further includes: acquiring a plurality of sample images and sample detection frames corresponding to the plurality of sample images, wherein the sample detection frames are used to mark target objects in the sample images; Set the number of sample detection frames as target detection frames; perform data enhancement on the target object corresponding to the target detection frame to obtain target sample images; use the target sample images to train the initial detection model to obtain the target detection model.

In an optional embodiment, multiple sample images may be mixed first to obtain a mixed image, and then a preset number of sample detection frames included in the multiple sample detection frames included in the mixed image are determined as target detection frames.

In another optional embodiment, multiple sample images and sample detection frames corresponding to the multiple sample images may be randomly selected in the training data set, and the multiple sample images may be detected according to the box level (region level) of the multiple sample images. For image data enhancement, you can first perform global data enhancement on multiple sample images to obtain multiple enhanced sample images, and then randomly select a preset number of sample detection frames from the enhanced multiple sample images as target detection frames By performing local data enhancement, data enhancement can be performed on the target object corresponding to the target detection frame to obtain the target sample image, and finally the initial detection model is trained by using the target sample image to obtain the target model.

In another optional embodiment, a preset number of sample detection frames may be randomly selected from a plurality of sample images as target detection frames to perform local data enhancement, and data enhancement may be performed on the target object corresponding to the target detection frame to obtain Enhanced multiple sample images, and then perform global data enhancement on the enhanced multiple sample images to obtain target sample images, and finally use the target sample images to train the initial detection model to obtain the target model.

The above-mentioned global data enhancement may be performing color change, rotation, contrast enhancement, random erasure, scaling, cropping, etc. on multiple sample images. The above-mentioned local data enhancement may be performing color change, rotation, contrast enhancement, random erasing, scaling, etc. on the target object in the target detection frame.

In the above embodiments of the present application, determining a preset number of sample detection frames in multiple sample images as target detection frames includes: splicing multiple sample images to obtain an initial sample image; Mixing to obtain a mixed image; determining a preset number of sample detection frames in the mixed image as target detection frames.

In an optional embodiment, a plurality of sample images can be stitched in sequence to obtain an initial sample image; optionally, a plurality of sample images can be stitched onto a picture with a white background, and the picture with a white background can be filled , to obtain the initial sample image. If multiple sample images do not fill the white background image, you can continue to randomly select a part of the sample image from the training data to fill the white background image until it is filled. Optionally, random splicing can also be performed on multiple sample images. The multiple sample images may also be stitched in other ways, which are not limited here.

The aforementioned preset number may be a preset number, and the aforementioned preset number may also be determined according to the number of sample detection frames, for example, the preset number may be 30% of the sample detection frames.

In another optional embodiment, a mosaic module (mosaic) can be used to sequentially stitch multiple sample images to obtain an initial sample image, wherein the mosaic is used to enrich the detection background and detection objects in the target image, so as to enrich the detection background and detection objects in the target image. data set. After the initial sample image is obtained, the initial sample image can be scaled and cropped to obtain the processed initial sample image, and the processed initial sample image and the preset sample image can be mixed by using a mixing module (mixup) to obtain a mixed image , in order to further enhance the data to enrich the features in the sample image. A preset number of sample detection frames included in the multiple sample detection frames included in the mixed image may be determined as target detection frames.

FIG. 7 is a schematic diagram of training a target detection model according to an embodiment of the present application. First, multiple sample images including sample detection frames are input into the mosaic module, and the mosaic module is used to stitch the multiple sample images to obtain an initial sample. image, and then use the mixing module to mix the initial sample image and the preset sample image to obtain a mixed image, and finally use the region level module to determine the preset number of sample detection frames in the mixed image as the target detection frame, and the corresponding target detection frame. The target object is enhanced with data to obtain the target sample image; the initial detection model is trained by using the target sample image to obtain the target detection model.

In the above-mentioned embodiment of the present application, the method further includes: outputting a detection result; receiving a first feedback result, wherein the first feedback result is obtained by modifying the channel in the merged feature map according to the detection result; based on the first feedback result Update the merged feature map.

In an optional embodiment, the detection result can be output and displayed to the user's client terminal, and the user can modify the channel in the merged feature map according to the detection result to obtain the first feedback result, so as to be compatible with the first feedback result according to the first feedback result. The merged feature map is updated with a feedback result, and detection can be performed according to the updated merged feature map to obtain a detection result with higher accuracy.

The neural network structure proposed in this application can use a lightweight network structure (csp_darknet) as the backbone, a large proportion of csp_GFPN as the neck, and a scale-decouple (different scale) GFocalv2 as the head. At the same time, combined with the training method of data augmentation based on learning, the target object in the target image can be detected quickly and accurately. Significantly reduces resource usage when deploying neural network structures.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence. As in accordance with the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

From the description of the above embodiments, those skilled in the art can clearly understand that the image processing method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases The former is a better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present invention.

Example 2

According to an embodiment of the present application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although A logical order is shown in the flowcharts, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 8 is a flowchart of an image processing method according to Embodiment 2 of the present application. As shown in FIG. 8 , the method may include the following steps:

Step S802, acquiring a remote sensing image of the target.

Among them, the target remote sensing image contains the target object.

Step S804, performing multi-scale feature extraction on the target remote sensing image to obtain a plurality of first feature maps.

Step S806, using a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map.

The multi-branch network structure is used to perform feature fusion on a plurality of first feature maps through a plurality of branches.

Step S808: Detect at least one second feature map to obtain a detection result of the target object.

In the above embodiments of the present application, the multi-branch network structure includes: a first branch and a second branch, wherein the output of the first branch is connected to the output of the second branch.

In the above embodiments of the present application, using a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map includes: performing channel merging on multiple first feature maps to obtain a combined feature map; using The first branch performs convolution processing on the merged feature map to obtain the first output feature; the second branch performs convolution processing on the merged feature map to obtain the second output feature; the first output feature and the second output feature are channel merged , obtain at least one second feature map.

In the above embodiments of the present application, the first branch includes: at least one convolution block, each convolution block in the multiple convolution blocks includes multiple sub-convolutional layers, and the convolution kernels of the multiple sub-convolutional layers are different.

In the above-mentioned embodiment of the present application, the method further includes: using a target detection model to detect the target image to obtain a detection result of the target object, wherein the target detection model is obtained by training based on the target sample image, and the target sample image is obtained through the sample detection frame. A sample image is obtained by data augmentation.

In the above-mentioned embodiments of the present application, the method further includes: acquiring a plurality of sample images and sample detection frames corresponding to the plurality of sample images, wherein the sample detection frames are used to mark target objects in the sample images; Set the number of sample detection frames as target detection frames; perform data enhancement on the target object corresponding to the target detection frame to obtain target sample images; use the target sample images to train the initial detection model to obtain the target detection model.

In the above embodiments of the present application, determining a preset number of sample detection frames in multiple sample images as target detection frames includes: splicing multiple sample images to obtain an initial sample image; Mixing to obtain a mixed image; determining a preset number of sample detection frames in the mixed image as target detection frames.

In the above-mentioned embodiment of the present application, the method further includes: outputting a detection result; receiving a first feedback result, wherein the first feedback result is obtained by modifying the channel in the merged feature map according to the detection result; based on the first feedback result Update the merged feature map.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 3

According to an embodiment of the present application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although A logical order is shown in the flowcharts, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 9 is a flowchart of an image processing method according to Embodiment 3 of the present application. As shown in FIG. 9 , the method may include the following steps:

Step S902, obtaining agricultural remote sensing images.

Among them, agricultural remote sensing images contain crops.

Step S904, performing multi-scale feature extraction on the agricultural remote sensing image to obtain a plurality of first feature maps.

Step S906, using a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map.

The multi-branch network structure is used to perform feature fusion on a plurality of first feature maps through a plurality of branches.

Step S908, detecting at least one second feature map to obtain a detection result of the crop.

In the above embodiments of the present application, the multi-branch network structure includes: a first branch and a second branch, wherein the output of the first branch is connected to the output of the second branch.

In the above embodiments of the present application, using a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map includes: performing channel merging on multiple first feature maps to obtain a combined feature map; using The first branch performs convolution processing on the merged feature map to obtain the first output feature; the second branch performs convolution processing on the merged feature map to obtain the second output feature; the first output feature and the second output feature are channel merged , obtain at least one second feature map.

In the above embodiments of the present application, the first branch includes: at least one convolution block, each convolution block in the multiple convolution blocks includes multiple sub-convolutional layers, and the convolution kernels of the multiple sub-convolutional layers are different.

In the above-mentioned embodiment of the present application, the method further includes: using a target detection model to detect agricultural remote sensing images to obtain detection results of crops, wherein the target detection model is obtained by training based on target sample images, and the target sample images are paired with multiple sample detection frames. A sample image is obtained by data augmentation.

In the above-mentioned embodiments of the present application, the method further includes: acquiring a plurality of sample images and sample detection frames corresponding to the plurality of sample images, wherein the sample detection frames are used to label crops in the sample images; The number of sample detection frames is the target detection frame; the crops corresponding to the target detection frame are enhanced to obtain the target sample image; the target sample image is used to train the initial detection model to obtain the target detection model.

In the above embodiments of the present application, determining a preset number of sample detection frames in multiple sample images as target detection frames includes: splicing multiple sample images to obtain an initial sample image; Mixing to obtain a mixed image; determining a preset number of sample detection frames in the mixed image as target detection frames.

In the above-mentioned embodiment of the present application, the method further includes: outputting a detection result; receiving a first feedback result, wherein the first feedback result is obtained by modifying the channel in the merged feature map according to the detection result; based on the first feedback result Update the merged feature map.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 4

According to an embodiment of the present application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although A logical order is shown in the flowcharts, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 10 is a flowchart of an image processing method according to Embodiment 4 of the present application. As shown in FIG. 10 , the method may include the following steps:

Step S1002, acquiring remote sensing images of buildings.

Among them, the building remote sensing image includes the target building.

In step S1004, multi-scale feature extraction is performed on the remote sensing image of the building to obtain a plurality of first feature maps.

Step S1006, using a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map.

The multi-branch network structure is used to perform feature fusion on a plurality of first feature maps through a plurality of branches.

Step S1008: Detect at least one second feature map to obtain a detection result of the target building.

In the above embodiments of the present application, the multi-branch network structure includes: a first branch and a second branch, wherein the output of the first branch is connected to the output of the second branch.

In the above embodiments of the present application, using a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map includes: performing channel merging on multiple first feature maps to obtain a combined feature map; using The first branch performs convolution processing on the merged feature map to obtain the first output feature; the second branch performs convolution processing on the merged feature map to obtain the second output feature; the first output feature and the second output feature are channel merged , obtain at least one second feature map.

In the above embodiments of the present application, the first branch includes: at least one convolution block, each convolution block in the multiple convolution blocks includes multiple sub-convolutional layers, and the convolution kernels of the multiple sub-convolutional layers are different.

In the above-mentioned embodiment of the present application, the method further includes: using a target detection model to detect the remote sensing image of the building to obtain the detection result of the target building, wherein the target detection model is obtained by training based on the target sample image, and the target sample image is detected by the sample The box is obtained by data augmentation of multiple sample images.

In the above-mentioned embodiments of the present application, the method further includes: acquiring a plurality of sample images and sample detection frames corresponding to the plurality of sample images, wherein the sample detection frames are used to mark target buildings in the sample images; A preset number of sample detection frames are target detection frames; data enhancement is performed on the target buildings corresponding to the target detection frames to obtain target sample images; the initial detection model is trained by using the target sample images to obtain a target detection model.

In the above embodiments of the present application, determining a preset number of sample detection frames in multiple sample images as target detection frames includes: splicing multiple sample images to obtain an initial sample image; Mixing to obtain a mixed image; determining a preset number of sample detection frames in the mixed image as target detection frames.

In the above-mentioned embodiment of the present application, the method further includes: outputting a detection result; receiving a first feedback result, wherein the first feedback result is obtained by modifying the channel in the merged feature map according to the detection result; based on the first feedback result Update the merged feature map.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 5

According to an embodiment of the present application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although A logical order is shown in the flowcharts, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 11 is a flowchart of an image processing method according to Embodiment 5 of the present application. As shown in FIG. 11 , the method may include the following steps:

Step S1102, the cloud server acquires the target image.

Among them, the target image contains the target object.

Step S1104, the cloud server performs multi-scale feature extraction on the target image to obtain a plurality of first feature maps.

Step S1106, the cloud server uses a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map.

The multi-branch network structure is used to perform feature fusion on a plurality of first feature maps through a plurality of branches.

Step S1108, the cloud server detects at least one second feature map to obtain a detection result of the target object.

In the above embodiments of the present application, the multi-branch network structure includes: a first branch and a second branch, wherein the output of the first branch is connected to the output of the second branch.

In the above-mentioned embodiment of the present application, the cloud server uses a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map, including: the cloud server performs channel merging on multiple first feature maps to obtain the merged feature map; the cloud server uses the first branch to perform convolution processing on the combined feature map to obtain a first output feature; the cloud server uses the second branch to perform convolution processing on the combined feature map to obtain a second output feature; The output feature and the second output feature are channel combined to obtain at least one second feature map.

In the above embodiments of the present application, the first branch includes: at least one convolution block, each convolution block in the multiple convolution blocks includes multiple sub-convolutional layers, and the convolution kernels of the multiple sub-convolutional layers are different.

In the above-mentioned embodiment of the present application, the method further includes: the cloud server detects the target image by using the target detection model to obtain the detection result of the target object, wherein the target detection model is obtained by training based on the target sample image, and the target sample image passes through the sample detection frame. Data augmentation is performed on multiple sample images.

In the above-mentioned embodiment of the present application, the method further includes: the cloud server obtains a plurality of sample images and sample detection frames corresponding to the plurality of sample images, wherein the sample detection frames are used to mark target objects in the sample images; the cloud server determines a plurality of sample detection frames. The preset number of sample detection frames in the sample image is the target detection frame; the cloud server performs data enhancement on the target object corresponding to the target detection frame to obtain the target sample image; the cloud server uses the target sample image to train the initial detection model to obtain the target detection model Model.

In the above embodiments of the present application, the cloud server determines a preset number of sample detection frames in the plurality of sample images as target detection frames, including: the cloud server stitches the plurality of sample images to obtain an initial sample image; the cloud server stitches the initial sample images Mixing with a preset sample image to obtain a mixed image; the cloud server determines a preset number of sample detection frames in the mixed image as target detection frames.

In the above-mentioned embodiment of the present application, the method further includes: the cloud server outputs a detection result; the cloud server receives a first feedback result, wherein the first feedback result is obtained by modifying the channel in the merged feature map according to the detection result; The server updates the merged feature map based on the first feedback result.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 6

According to an embodiment of the present application, an image processing apparatus for implementing the above image processing method is also provided. FIG. 12 is a schematic diagram of an image processing apparatus according to Embodiment 6 of the present application. As shown in FIG. 12 , the apparatus 1200 It includes: an acquisition module 1202 , an extraction module 1204 , a fusion module 1206 , and a detection module 1208 .

The acquisition module is used to acquire target remote sensing images, wherein the agricultural remote sensing images contain target objects; the extraction module is used to perform multi-scale feature extraction on the target remote sensing images to obtain a plurality of first feature maps; the fusion module is used to utilize multi-branch network The structure performs feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; the detection module is used for at least one feature fusion. The second feature map is detected to obtain the detection result of the target object.

It should be noted here that the above-mentioned acquisition module 1202, extraction module 1204, fusion module 1206, and detection module 1208 correspond to steps S202 to S208 in Embodiment 1, and examples and applications implemented by the four modules and corresponding steps The scenario is the same, but is not limited to the content disclosed in the above Embodiment 1. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

In the above embodiments of the present application, the multi-branch network structure includes: a first branch and a second branch, wherein the output of the first branch is connected to the output of the second branch.

In the above embodiments of the present application, the fusion module includes: a first processing unit and a merging unit.

The merging unit is used for channel merging multiple first feature maps to obtain a merged feature map; the first processing unit is used to perform convolution processing on the merged feature map using the first branch to obtain a first output feature; the first processing The unit is further configured to perform convolution processing on the merged feature map using the second branch to obtain a second output feature; the merging unit is further configured to perform channel merging on the first output feature and the second output feature to obtain at least one second feature map.

In the above embodiments of the present application, the first branch includes: at least one convolution block, each convolution block in the multiple convolution blocks includes multiple sub-convolutional layers, and the convolution kernels of the multiple sub-convolutional layers are different.

In the above-mentioned embodiment of the present application, the detection module is further configured to detect the target image by using the target detection model to obtain the detection result of the target object, wherein the target detection model is obtained by training based on the target sample image, and the target sample image is obtained through the sample detection frame to many A sample image is obtained by data augmentation.

In the above embodiments of the present application, the apparatus further includes: an acquisition module, a determination module, an enhancement module, and a training module.

Wherein, the acquisition module is used to acquire multiple sample images and sample detection frames corresponding to the multiple sample images, wherein the sample detection frame is used to mark the target objects in the sample images; the determination module is used to determine the preset quantity in the multiple sample images The sample detection frame is the target detection frame; the enhancement module is used to enhance the target object corresponding to the target detection frame to obtain the target sample image; the training module is used to train the initial detection model by using the target sample image to obtain the target detection model.

In the above embodiments of the present application, the determination module includes: a splicing unit, a mixing unit, and a determination unit.

Wherein, the splicing unit is used for splicing multiple sample images to obtain an initial sample image; the mixing unit is used for mixing the initial sample image and the preset sample image to obtain a mixed image; the determining unit is used for determining the preset quantity in the mixed image The sample detection frame of is the target detection frame.

In the above embodiments of the present application, the device further includes: an output module, a receiving module, and an updating module.

The output module is used to output the detection result; the receiving module is used to receive the first feedback result, wherein the first feedback result is used to modify the channel in the merged feature map according to the detection result; the update module is used to obtain based on the first feedback Feedback results update the merged feature map.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 7

According to an embodiment of the present application, an image processing apparatus for implementing the above image processing method is also provided. FIG. 13 is a schematic diagram of an image processing apparatus according to Embodiment 7 of the present application. As shown in FIG. 13 , the apparatus 1300 It includes: an acquisition module 1302 , an acquisition module 1302 , an acquisition module 1302 , and an acquisition module 1302 .

The acquisition module is used to acquire agricultural remote sensing images, wherein the agricultural remote sensing images include crops; the extraction module is used to perform multi-scale feature extraction on the agricultural remote sensing images to obtain a plurality of first feature maps; the fusion module is used to utilize a multi-branch network structure Perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; the detection module is used for at least one first feature map. Two feature maps are detected to obtain the detection results of crops.

It should be noted here that the above-mentioned acquisition module 1302, extraction module 1304, fusion module 1306, and detection module 1308 correspond to steps S802 to S808 in Embodiment 2, and examples and applications implemented by the four modules and corresponding steps The scenario is the same, but is not limited to the content disclosed in the above Embodiment 1. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 8

According to an embodiment of the present application, an image processing apparatus for implementing the above image processing method is also provided. FIG. 14 is a schematic diagram of an image processing apparatus according to Embodiment 8 of the present application. As shown in FIG. 14 , the apparatus 1400 It includes: an acquisition module 1402 , an extraction module 1404 , a fusion module 1406 , and a detection module 1408 .

The acquisition module is used to acquire remote sensing images of buildings, wherein the remote sensing images of buildings include target buildings; the extraction module is used to perform multi-scale feature extraction on the remote sensing images of buildings to obtain multiple first feature maps; the fusion module is used for A multi-branch network structure is used to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; the detection module uses For detecting at least one second feature map, a detection result of the target building is obtained.

It should be noted here that the above-mentioned acquisition module 1402, extraction module 1404, fusion module 1406, and detection module 1408 correspond to steps S902 to S908 in Embodiment 3, and examples and applications implemented by the four modules and corresponding steps The scenario is the same, but is not limited to the content disclosed in the above Embodiment 1. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

In the above-mentioned embodiment of the present application, the method further includes: using a target detection model to detect the remote sensing image of the building to obtain the detection result of the target building, wherein the target detection model is obtained by training based on the target sample image, and the target sample image is detected by the sample The box is obtained by data augmentation of multiple sample images.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 9

According to an embodiment of the present application, an image processing apparatus for implementing the above image processing method is also provided. FIG. 15 is a schematic diagram of an image processing apparatus according to Embodiment 9 of the present application. As shown in FIG. 15 , the apparatus 1500 It includes: an acquisition module 1502 , an extraction module 1504 , a fusion module 1506 , and a detection module 1508 .

The acquisition module is used to obtain the target remote sensing image through the cloud server, wherein the target remote sensing image contains the target object; the extraction module is used to perform multi-scale feature extraction on the target remote sensing image through the cloud server to obtain a plurality of first feature maps; the fusion module It is used to perform feature fusion on multiple first feature maps by using a multi-branch network structure through a cloud server to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches. Fusion; the detection module is used to detect at least one second feature map through the cloud server to obtain the detection result of the target object.

It should be noted here that the above-mentioned acquisition module 1502, extraction module 1504, fusion module 1506, and detection module 1508 correspond to steps S1002 to S1008 in Embodiment 4, and examples and applications implemented by the four modules and corresponding steps The scenario is the same, but is not limited to the content disclosed in the above Embodiment 1. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 10

An embodiment of the present invention may provide an electronic device, the electronic device includes: a memory and a processor, where the processor is configured to run a program stored in the memory, the electronic device may be a computer terminal, and the computer terminal may be a computer terminal in a group of computer terminals. Any computer terminal device. Optionally, in this embodiment, the above-mentioned computer terminal may also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the above-mentioned computer terminal may be located in at least one network device among multiple network devices of a computer network.

In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image processing method: acquiring a target image, wherein the target image includes a target object; performing multi-scale feature extraction on the target image to obtain a plurality of first feature maps; A multi-branch network structure is used to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; The second feature map is detected to obtain the detection result of the target object.

Optionally, FIG. 16 is a structural block diagram of a computer terminal according to an embodiment of the present application. As shown in FIG. 16 , the computer terminal A may include: one or more (only one is shown in the figure) processors and memories.

The memory can be used to store software programs and modules, such as program instructions/modules corresponding to the image processing method and apparatus in the embodiments of the present application, and the processor executes various functional applications by running the software programs and modules stored in the memory. And data processing, that is, to realize the above-mentioned image processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory located remotely from the processor, and these remote memories may be connected to Terminal A through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The processor can call the information and the application program stored in the memory through the transmission device to perform the following steps: acquire a target image, wherein the target image includes the target object; perform multi-scale feature extraction on the target image to obtain a plurality of first feature maps; A multi-branch network structure is used to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; The second feature map is detected to obtain the detection result of the target object.

Optionally, the above-mentioned processor may also execute the program code of the following steps: performing channel merging on a plurality of first feature maps to obtain a combined feature map; using the first branch to perform convolution processing on the combined feature map to obtain a first output feature ; use the second branch to perform convolution processing on the merged feature map to obtain a second output feature; perform channel merging on the first output feature and the second output feature to obtain at least one second feature map.

Optionally, the above-mentioned processor may also execute the program code of the following steps: using the target detection model to detect the target image to obtain the detection result of the target object, wherein the target detection model is obtained by training based on the target sample image, and the target sample image is obtained through the sample image. The detection frame is obtained by data augmentation of multiple sample images.

Optionally, the above-mentioned processor may also execute the program code of the following steps: acquiring multiple sample images and sample detection frames corresponding to the multiple sample images, wherein the sample detection frames are used to mark the target objects in the sample images; A preset number of sample detection frames in the sample image are target detection frames; data enhancement is performed on the target object corresponding to the target detection frame to obtain target sample images; the initial detection model is trained by using the target sample images to obtain target detection models.

Optionally, the above-mentioned processor may also execute the program code of the following steps: splicing a plurality of sample images to obtain an initial sample image; mixing the initial sample image and the preset sample image to obtain a mixed image; Let the number of sample detection boxes be the target detection boxes.

Optionally, the above-mentioned processor may also execute the program code of the following steps: outputting a detection result; receiving a first feedback result, wherein the first feedback result is used to modify the channel in the merged feature map according to the detection result; The first feedback result updates the merged feature map.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain a target remote sensing image, wherein the target remote sensing image includes a target object; perform multi-scale feature extraction on the target remote sensing image to obtain a plurality of first remote sensing images. feature map; using a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; The at least one second feature map is detected to obtain a detection result of the target object.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain an agricultural remote sensing image, wherein the agricultural remote sensing image includes crops; perform multi-scale feature extraction on the agricultural remote sensing image to obtain a plurality of first features Figure; perform feature fusion on multiple first feature maps by using a multi-branch network structure to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches; At least one second feature map is detected to obtain a detection result of the crop.

The processor can call the information and the application program stored in the memory through the transmission device to perform the following steps: obtain a remote sensing image of a building, wherein the remote sensing image of the building includes the target building; perform multi-scale feature extraction on the remote sensing image of the building to obtain multiple first feature maps; using a multi-branch network structure to perform feature fusion on the multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used for multiple first feature maps through multiple branches Perform feature fusion; perform detection on at least one second feature map to obtain a detection result of the target building.

The processor can call the information and the application program stored in the memory through the transmission device to perform the following steps: the cloud server obtains the target image, wherein the target image includes the target object; the cloud server performs multi-scale feature extraction on the target image to obtain a plurality of first images. a feature map; the cloud server uses a multi-branch network structure to perform feature fusion on a plurality of first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform a multi-branch network structure on the plurality of first feature maps through a plurality of branches. Feature fusion; the cloud server detects at least one second feature map to obtain a detection result of the target object.

Using the embodiment of the present application, first acquire a target image, where the target image contains a target object; perform multi-scale feature extraction on the target image to obtain a plurality of first feature maps; use a multi-branch network structure to characterize the plurality of first feature maps Fusion to obtain at least one second feature map, wherein the multi-branch network structure is used to represent that feature fusion is performed on multiple first feature maps through multiple branches to obtain at least one second feature map, wherein the multi-branch network structure is used for Feature fusion is performed on multiple first feature maps through multiple branches, and at least one second feature map is detected to obtain a detection result of the target object, thereby improving the accuracy of the detection result of the target object. It is easy to notice that feature fusion can be performed on the first feature map through multiple branches, thereby improving the accuracy of the obtained at least one second feature map, and multiple branches can also reduce the amount of parameters in the fusion process, so that the The efficiency of fusion is improved, thereby solving the technical problem of low image detection accuracy in the related art.

Those of ordinary skill in the art can understand that the structure shown in FIG. 15 is only a schematic diagram, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, an applause computer, and a mobile internet device (Mobile Internet Devices, MID) , PAD and other terminal equipment. FIG. 15 does not limit the structure of the above electronic device. For example, the computer terminal 10 may also include more or less components than those shown in FIG. 15 (eg, network interface, display device, etc.), or have a different configuration than that shown in FIG. 15 .

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can Including: flash disk, read-only memory (Read-Only Memory, ROM), random access device (RandomAccess Memory, RAM), magnetic disk or optical disk, etc.

Example 11

Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the above-mentioned storage medium may be used to store the program code executed by the image processing method provided in the above-mentioned first embodiment.

Optionally, in this embodiment, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

Optionally, in this embodiment, the storage medium is configured to store program codes for performing the following steps: acquiring a target image, where the target image includes a target object; performing multi-scale feature extraction on the target image to obtain a plurality of first a feature map; using a multi-branch network structure to perform feature fusion on multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform feature fusion on multiple first feature maps through multiple branches ; Detect at least one second feature map to obtain a detection result of the target object.

Optionally, the above-mentioned storage medium is further configured to store program codes for performing the following steps: performing channel merging on a plurality of first feature maps to obtain a combined feature map; using the first branch to perform convolution processing on the combined feature map, obtaining a first output feature; using the second branch to perform convolution processing on the combined feature map to obtain a second output feature; performing channel merging on the first output feature and the second output feature to obtain at least one second feature map.

Optionally, the above-mentioned storage medium is also set to store program codes for performing the following steps: using a target detection model to detect the target image to obtain a detection result of the target object, wherein the target detection model is obtained by training based on the target sample image, The target sample image is obtained by performing data enhancement on multiple sample images through the sample detection frame.

Optionally, the above-mentioned storage medium is also configured to store program codes for performing the following steps: acquiring a plurality of sample images and sample detection frames corresponding to the plurality of sample images, wherein the sample detection frames are used to mark objects in the sample images. object; determine a preset number of sample detection frames in multiple sample images as target detection frames; perform data enhancement on the target object corresponding to the target detection frame to obtain target sample images; use the target sample images to train the initial detection model to obtain the target Check the model.

Optionally, the above-mentioned storage medium is further configured to store program codes for performing the following steps: splicing a plurality of sample images to obtain an initial sample image; mixing the initial sample image and the preset sample image to obtain a mixed image; Determine a preset number of sample detection frames in the mixed image as target detection frames.

Optionally, the above-mentioned storage medium is further configured to store program codes for performing the following steps: outputting a detection result; receiving a first feedback result, wherein the first feedback result is used to pair the channels in the merged feature map according to the detection result. Modification is performed to obtain; the merged feature map is updated based on the first feedback result.

Optionally, in this embodiment, the storage medium is set to store program codes for performing the following steps: acquiring a target remote sensing image, wherein the target remote sensing image includes a target object; performing multi-scale feature extraction on the target remote sensing image to obtain multiple first feature maps; using a multi-branch network structure to perform feature fusion on the multiple first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used for multiple first feature maps through multiple branches Perform feature fusion; perform detection on at least one second feature map to obtain a detection result of the target object.

Optionally, in this embodiment, the storage medium is configured to store program codes for performing the following steps: acquiring an agricultural remote sensing image, wherein the agricultural remote sensing image includes crops; performing multi-scale feature extraction on the agricultural remote sensing image to obtain multiple A multi-branch network structure is used to perform feature fusion on the plurality of first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used to perform a multi-branch network structure on the plurality of first feature maps through multiple branches. Feature fusion; detecting at least one second feature map to obtain a detection result of crops.

Optionally, in this embodiment, the storage medium is configured to store program codes for performing the following steps: acquiring a remote sensing image of a building, wherein the remote sensing image of the building includes a target building; Feature extraction, to obtain a plurality of first feature maps; using a multi-branch network structure to perform feature fusion on the plurality of first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used for multiple The first feature map performs feature fusion; at least one second feature map is detected to obtain a detection result of the target building.

Optionally, in this embodiment, the storage medium is configured to store program codes for performing the following steps: the cloud server acquires a target image, wherein the target image contains the target object; the cloud server performs multi-scale feature extraction on the target image, Obtain a plurality of first feature maps; the cloud server uses a multi-branch network structure to perform feature fusion on the plurality of first feature maps to obtain at least one second feature map, wherein the multi-branch network structure is used for multiple A feature map performs feature fusion; the cloud server detects at least one second feature map to obtain a detection result of the target object.

Using the embodiment of the present application, first acquire a target image, where the target image contains a target object; perform multi-scale feature extraction on the target image to obtain a plurality of first feature maps; use a multi-branch network structure to characterize the plurality of first feature maps Fusion to obtain at least one second feature map, wherein the multi-branch network structure is used to represent that feature fusion is performed on multiple first feature maps through multiple branches to obtain at least one second feature map, wherein the multi-branch network structure is used for Feature fusion is performed on multiple first feature maps through multiple branches, and at least one second feature map is detected to obtain a detection result of the target object, thereby improving the accuracy of the detection result of the target object. It is easy to notice that feature fusion can be performed on the first feature map through multiple branches, thereby improving the accuracy of the obtained at least one second feature map, and multiple branches can also reduce the amount of parameters in the fusion process, so that the The efficiency of fusion is improved, thereby solving the technical problem of low image detection accuracy in the related art.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The apparatus embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (14)

1. An image processing method, comprising:

acquiring a target image, wherein the target image comprises a target object;

performing multi-scale feature extraction on the target image to obtain a plurality of first feature maps;

performing feature fusion on the plurality of first feature maps by using a multi-branch network structure to obtain at least one second feature map, wherein the multi-branch network structure is used for performing feature fusion on the plurality of first feature maps through a plurality of branches;

and detecting the at least one second characteristic diagram to obtain a detection result of the target object.

2. The method of claim 1, wherein the multi-drop network architecture comprises: a first branch, a second branch, wherein an output of the first branch is connected with an output of the second branch.

3. The method of claim 2, wherein performing feature fusion on the plurality of first feature maps using a multi-drop network architecture to obtain at least one second feature map comprises:

channel merging is carried out on the first feature maps to obtain a merged feature map;

performing convolution processing on the merged feature map by using the first branch to obtain a first output feature;

performing convolution processing on the merged feature map by using the second branch to obtain a second output feature;

and carrying out channel combination on the first output characteristic and the second output characteristic to obtain the at least one second characteristic diagram.

4. The method of claim 3, wherein the first branch comprises: at least one convolution block, each of the plurality of convolution blocks comprising a plurality of sub-convolution layers, the plurality of sub-convolution layers having different convolution kernels.

5. The method of claim 1, further comprising:

and detecting the target image by using a target detection model to obtain the detection result of the target object, wherein the target detection model is obtained by training based on a target sample image, and the target sample image is obtained by performing data enhancement on a plurality of sample images through a sample detection frame.

6. The method of claim 5, further comprising:

obtaining the plurality of sample images and the sample detection frames corresponding to the plurality of sample images, wherein the sample detection frames are used for marking target objects in the sample images;

determining a preset number of sample detection frames in the plurality of sample images as target detection frames;

performing data enhancement on the target object corresponding to the target detection frame to obtain a target sample image;

and training an initial detection model by using the target sample image to obtain the target detection model.

7. The method of claim 6, wherein determining a preset number of sample detection frames in the plurality of sample images as target detection frames comprises:

splicing the plurality of sample images to obtain an initial sample image;

mixing the initial sample image and a preset sample image to obtain a mixed image;

determining the preset number of the sample detection frames in the mixed image as the target detection frame.

8. The method of claim 3, further comprising:

outputting the detection result;

receiving a first feedback result, wherein the first feedback result is obtained by modifying a channel in the combined feature map according to the detection result;

updating the merged feature map based on the first feedback result.

9. An image processing method, comprising:

acquiring a target remote sensing image, wherein the target remote sensing image comprises a target object;

carrying out multi-scale feature extraction on the target remote sensing image to obtain a plurality of first feature maps;

performing feature fusion on the plurality of first feature maps by using a multi-branch network structure to obtain at least one second feature map, wherein the multi-branch network structure is used for performing feature fusion on the plurality of first feature maps through a plurality of branches;

and detecting the at least one second characteristic diagram to obtain a detection result of the target object.

10. An image processing method, comprising:

obtaining a building remote sensing image, wherein the building remote sensing image comprises a target building;

carrying out multi-scale feature extraction on the building remote sensing image to obtain a plurality of first feature maps;

performing feature fusion on the plurality of first feature maps by using a multi-branch network structure to obtain at least one second feature map, wherein the multi-branch network structure is used for performing feature fusion on the plurality of first feature maps through a plurality of branches;

and detecting the at least one second characteristic diagram to obtain a detection result of the target building.

11. The method of claim 10, further comprising:

and detecting the building remote sensing image by using a target detection model to obtain the detection result of the target building, wherein the target detection model is obtained by training based on a target sample image, and the target sample image is obtained by performing data enhancement on a plurality of sample images through a sample detection frame.

12. An image processing method, comprising:

the method comprises the steps that a cloud server obtains a target image, wherein the target image comprises a target object;

the cloud server performs multi-scale feature extraction on the target image to obtain a plurality of first feature maps;

the cloud server performs feature fusion on the plurality of first feature maps by using a multi-branch network structure to obtain at least one second feature map, wherein the multi-branch network structure is used for performing feature fusion on the plurality of first feature maps through a plurality of branches;

and the cloud server detects the at least one second characteristic diagram to obtain a detection result of the target object.

13. A computer-readable storage medium, comprising a stored program, wherein the program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method of any one of claims 1 to 12.

14. An electronic device, comprising: a memory and a processor for executing a program stored in the memory, wherein the program when executed performs the method of any one of claims 1 to 12.

Images