CN117372307A - Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method - Google Patents

Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method Download PDF

Info

Publication number
CN117372307A
CN117372307A CN202311627840.6A CN202311627840A CN117372307A CN 117372307 A CN117372307 A CN 117372307A CN 202311627840 A CN202311627840 A CN 202311627840A CN 117372307 A CN117372307 A CN 117372307A
Authority
CN
China
Prior art keywords
image
illumination
module
enhancement
unmanned aerial
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311627840.6A
Other languages
Chinese (zh)
Other versions
CN117372307B (en
Inventor
马晨瑛
徐诚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Aeronautics and Astronautics
Original Assignee
Nanjing University of Aeronautics and Astronautics
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Aeronautics and Astronautics filed Critical Nanjing University of Aeronautics and Astronautics
Priority to CN202311627840.6A priority Critical patent/CN117372307B/en
Publication of CN117372307A publication Critical patent/CN117372307A/en
Application granted granted Critical
Publication of CN117372307B publication Critical patent/CN117372307B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention belongs to the technical field of collaborative image enhancement, and discloses a multi-unmanned aerial vehicle collaborative detection distributed image enhancement method, which comprises the following steps: respectively inputting a plurality of initial images into an image enhancement model of a corresponding unmanned aerial vehicle to carry out image enhancement to obtain a plurality of enhanced images, and inputting the plurality of enhanced images into a panoramic stitching module to carry out panoramic stitching to obtain a panoramic stitching image; the image enhancement model comprises an illumination estimation module, an illumination enhancement module and a reflection component enhancement module; the illumination estimation module is formed by connecting a plurality of convolutional neural networks in series; the reflection component enhancement module consists of a plurality of filtering submodules which are connected in sequence; the illumination enhancement module is an encoder-decoder network structure which introduces an attention mechanism; the panorama stitching module is composed of a decomposition network, an image stitching module and a weight fusion module which are sequentially connected. The technical scheme of the invention can enhance the acquired images in a distributed and coordinated way, and improves the efficiency and effect of panoramic stitching processing of the ground control center.

Description

Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method
Technical Field
The invention belongs to the technical field of collaborative image enhancement, and particularly relates to a multi-unmanned aerial vehicle collaborative detection distributed image enhancement method.
Background
With the progress of unmanned aerial vehicle technology and the decrease of cost, unmanned aerial vehicle's application in various fields gradually increases. Unmanned aerial vehicle has flexibility, high mobility and lower risk characteristics, makes it become the ideal platform of carrying out various detection tasks. Some detection tasks require monitoring of a wide area or acquisition of high resolution image data, such as land mapping, urban planning, resource exploration, etc. The range and sensor coverage of a single drone is limited, while the coordinated operation of multiple drones can expand the detection range and provide more detailed data. Because unmanned aerial vehicle carries out data acquisition under different time, different positions and different weather conditions, the illumination degree in the detection area can be different. This may result in images acquired by the drone having different brightness, contrast and color profiles. The problem of uneven illumination may exist in panoramic stitching the images directly, so that the stitched images show obvious brightness variation and discontinuity. This may cause problems for subsequent image analysis and target detection, degrading the performance of the overall detection system.
To solve the problem of uneven illumination, researchers have proposed methods and techniques: (1) illumination correction: the acquired image is subjected to illumination correction, so that the image has consistency in brightness, contrast and color. The illumination correction algorithm can adjust the brightness value of each pixel according to the brightness distribution of the image, so that illumination equalization of the image is realized; (2) high dynamic range image stitching: for scenes with large illumination differences, high dynamic range (High Dynamic Range, HDR) image stitching techniques may be employed. According to the technology, a plurality of images with different exposure degrees are fused, so that a wider brightness range is realized in a panoramic image, and illumination non-uniformity is reduced; (3) image fusion and fade transitions: when image stitching is performed, the transition of the area of uneven illumination can be smoothed by using the methods of image fusion and gradual transition. By gradually changing the edges of the image, the transition between different illumination areas is more natural, and obvious illumination non-uniformity is reduced; (4) target detection and compensation: in unmanned aerial vehicle collaborative detection, target areas of interest can be identified using a target detection algorithm, and then illumination compensation is performed on these areas. By processing the target area in a targeted manner, the influence of uneven illumination on target detection can be reduced, and the visibility and accuracy of the target are improved.
Although the above method has certain advantages in solving the problem of uneven illumination, there are some potential disadvantages: (1) The illumination correction method may cause excessive processing or distortion. In global illumination correction, details of certain local areas may be lost or become unrealistic. Overcorrection can lead to overcorrection of brightness and contrast of the image, possibly affecting the visibility and detail of the image; (2) The HDR image is better in effect of processing scenes with larger illumination difference, but for scenes with discontinuous illumination change or complex transition, the illumination non-uniformity can not be completely eliminated, and the problems of unnatural transition or inconsistent brightness of partial areas of the spliced image can be caused; (3) Image fusion and gradual transition methods require more complex algorithms and processing steps, which may increase computational and processing complexity, which may result in increased processing time or require higher computational resources, limiting the ability to process in real-time or quickly; (4) The target detection and compensation method needs to accurately detect and position the target and perform illumination compensation treatment, which may be limited by the performance of the target detection algorithm and may cause the problem of missed detection or false detection; furthermore, compensating the target may introduce additional errors or distortions.
Disclosure of Invention
The invention aims to provide a multi-unmanned aerial vehicle collaborative detection distributed image enhancement method so as to solve the problems in the prior art.
In order to achieve the above object, the present invention provides a multi-unmanned aerial vehicle collaborative detection distributed image enhancement method, including:
acquiring a plurality of initial images acquired by a plurality of unmanned aerial vehicles;
respectively inputting a plurality of initial images into an image enhancement model of a corresponding unmanned aerial vehicle to carry out image enhancement to obtain a plurality of enhanced images, and inputting the plurality of enhanced images into a panoramic stitching module of a ground control center to carry out panoramic stitching to obtain a panoramic stitching image;
the image enhancement model comprises an illumination estimation module, an illumination enhancement module and a reflection component enhancement module which are sequentially connected; the illumination estimation module is formed by connecting a plurality of convolutional neural networks in series; the reflection component enhancement module consists of a plurality of filtering submodules which are connected in sequence; the illumination enhancement module is an encoder-decoder network structure which introduces an attention mechanism;
the panorama stitching module is composed of a decomposition network, an image stitching module and a weight fusion module which are sequentially connected.
Optionally, the training process of the image enhancement model specifically includes:
acquiring training data; the training data comprises an initial training image and a corresponding enhanced image;
and inputting the training data into an illumination estimation module, an illumination enhancement module and a reflection component enhancement module in sequence, and training by taking the minimum loss between the output result of the reflection component enhancement module and the enhancement image corresponding to the initial training image as a target to obtain the image enhancement model.
Optionally, the processing procedure of the image enhancement model includes:
inputting the initial image into the illumination estimation module for illumination estimation to obtain illumination estimation component data;
taking the illumination estimation component data as the input of the illumination enhancement module, and performing illumination component enhancement processing to obtain an illumination component enhanced image;
dividing the initial image and the illumination estimation component data, taking the operation result as the input of the reflection component enhancement module, and enhancing the reflection component to obtain a reflection component enhanced image;
and fusing the illumination component enhanced image and the reflection component enhanced image to obtain the enhanced image.
Optionally, inputting the initial image into the illumination estimation module for illumination estimation to obtain illumination estimation component data, which specifically includes:
estimating the brightness of the initial image stage by stage through a plurality of convolutional neural networks in the illumination estimation module to obtain illumination estimation component data;
the calculation formula for acquiring the illumination estimation component data is as follows:
in the method, in the process of the invention,is provided with training parameters->For learning illumination; />Residual term representing t-phase,/>Representing t->The illumination component of the phase is used to determine,Tis the total number of stages; y represents an original image; />A mapping function representing the entire illumination estimation phase; the same +.>Architecture.
Optionally, the illumination estimation component data is used as input of the illumination enhancement module to perform illumination component enhancement processing to obtain an illumination component enhanced image, which specifically includes:
performing illumination component enhancement on the illumination estimation component data through an encoder-decoder network in the illumination enhancement module, inputting the illumination estimation component data into an encoder to perform global feature capture and local feature capture, and further performing detail recovery and structure recovery on an output result of the encoder through a decoder;
adding an attention module in the encoder-decoder network, applying an attention mechanism to the output result of the encoder through the attention module to obtain an attention map, and fusing the attention map with the output result of the decoder to obtain an illumination component enhanced image;
the calculation formula for obtaining the attention map is as follows:
in the method, in the process of the invention,Ain order to take care of the force of the drawing,indicating return to maximum in three color channels, < >>For the original reference illumination component image, x represents the input illumination component image.
Optionally, division operation is performed on the initial image and illumination estimation component data, and the operation result is used as input of the reflection component enhancement module to perform reflection component enhancement, so as to obtain a reflection component enhanced image, which specifically includes:
and dividing the initial image and the illumination estimation component data to obtain reflection component data, inputting the reflection component data into the reflection component enhancement module, and sequentially carrying out denoising, contrast and white balance enhancement processing on the reflection component data through a plurality of filtering submodules in the reflection component enhancement module to obtain the reflection component enhanced image.
Optionally, the filtering submodules are composed of a median filtering submodule, a histogram equalization filtering submodule and a white balance filtering submodule which are sequentially connected.
Optionally, the processing procedure of the panorama stitching module includes:
inputting the enhanced image into a decomposition network for decomposition to obtain a high-frequency image and a low-frequency image:
wherein L and G respectively represent Laplacian and Gaussian pyramid, j represents the number of layers, j=0 is the original image, and expansion represents expansion operation;
taking the high-frequency image and the low-frequency image as the input of the image stitching module, acquiring feature matching points between the high-frequency image and the low-frequency image based on a feature point matching method, calculating a homography matrix based on the feature matching points and the data, further carrying out affine transformation on the high-frequency image and the low-frequency image by utilizing the homography matrix to obtain a high-frequency stitched image and a low-frequency stitched image,
inputting the high-frequency spliced image and the low-frequency spliced image into the weight fusion module for fusion,
and carrying out weight fusion and crack removal processing on each spliced image according to illumination estimation component data of the image enhancement model corresponding to each spliced image to obtain the panoramic spliced image.
Optionally, after obtaining the panoramic stitched image, the method further includes: reversely optimizing illumination estimation component data of the image enhancement model of each unmanned aerial vehicle by utilizing an information conservation principle;
the reverse optimization of the illumination estimation component data of the image enhancement model of each unmanned aerial vehicle by utilizing the information conservation principle comprises the following specific processes:
determining weight values occupied by each unmanned aerial vehicle according to the illumination consistency loss values, carrying out normalization processing on the weight values occupied by each unmanned aerial vehicle, and fusing the illumination estimation component data by using a weighted average method after the normalization processing is finished to obtain a global illumination fusion component：
In the method, in the process of the invention,for the weighting factor>Estimating a component, namely a local parameter, for illumination of the unmanned aerial vehicle;
based on the principle of conservation of information, fusing the global illumination componentsAnd reversely optimizing the illumination estimation component data in the image enhancement model distributed to each unmanned aerial vehicle according to the weight.
The invention has the technical effects that:
the invention provides a multi-unmanned aerial vehicle collaborative detection distributed image enhancement method, which comprises the following steps: acquiring a plurality of initial images acquired by a plurality of unmanned aerial vehicles; respectively inputting a plurality of initial images into an image enhancement model of a corresponding unmanned aerial vehicle to carry out image enhancement to obtain a plurality of enhanced images, and inputting the plurality of enhanced images into a panoramic stitching module of a ground control center to carry out panoramic stitching to obtain a panoramic stitching image; the image enhancement model comprises an illumination estimation module, an illumination enhancement module and a reflection component enhancement module which are sequentially connected; the illumination estimation module is formed by connecting a plurality of convolutional neural networks in series; the reflection component enhancement module consists of a plurality of filtering submodules which are connected in sequence; the illumination enhancement module is an encoder-decoder network structure which introduces an attention mechanism; the panorama stitching module is composed of a decomposition network, an image stitching module and a weight fusion module which are sequentially connected.
Aiming at the problem that the illumination of spliced images after the collaborative detection of multiple unmanned planes is uneven, the invention provides a novel image enhancement technology, and introduces the enhancement effect of restricting the image enhancement network by illumination consistency loss;
according to the invention, through an offline centralized training image enhancement network and integrating a trained model into the unmanned aerial vehicle, when the unmanned aerial vehicle cluster executes a collaborative detection task, a single unmanned aerial vehicle can enhance the acquired image in a distributed collaborative manner, and the enhanced image and local parameters are transmitted to a ground control center, so that the efficiency and effect of panoramic stitching processing by the ground control center can be improved;
the invention adopts an attention mechanism in the image enhancement technology: the attention mechanism module adopts U-Net network learning attention force diagram, and guides the network self-adaptive guiding enhancement network to correctly enhance the low-illumination area through the network, thereby avoiding excessively enhancing the normal illumination area and improving the detail restoration and illumination balance of the image.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
The accompanying drawings, which are included to provide a further understanding of the application, illustrate and explain the application and are not to be construed as limiting the application. In the drawings:
FIG. 1 is a flow chart of an embodiment of the present invention;
FIG. 2 is a schematic diagram of an image enhancement model according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an illumination estimation module according to an embodiment of the present invention;
FIG. 4 is a schematic view of a light enhancement module according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a reflection component enhancement module according to an embodiment of the present invention;
fig. 6 is a schematic view of a panorama stitching module structure according to an embodiment of the present invention;
fig. 7 is a global parameter feedback flow chart in an embodiment of the invention.
Detailed Description
Various exemplary embodiments of the invention will now be described in detail, which should not be considered as limiting the invention, but rather as more detailed descriptions of certain aspects, features and embodiments of the invention.
Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although the invention has been described with reference to a preferred method, any method similar or equivalent to those described herein can be used in the practice or testing of the present invention. All documents mentioned in this specification are incorporated by reference for the purpose of disclosing and describing the methodologies associated with the documents. In case of conflict with any incorporated document, the present specification will control.
It will be apparent to those skilled in the art that various modifications and variations can be made in the specific embodiments of the invention described herein without departing from the scope or spirit of the invention. Other embodiments will be apparent to those skilled in the art from consideration of the specification of the present invention. The specification and examples are exemplary only.
As used herein, the terms "comprising," "including," "having," "containing," and the like are intended to be inclusive and mean an inclusion, but not limited to.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Examples
As shown in fig. 1 to 7, in this embodiment, a method for collaborative detection of distributed image enhancement by multiple unmanned aerial vehicles is provided, including:
acquiring a plurality of initial images acquired by a plurality of unmanned aerial vehicles;
respectively inputting a plurality of initial images into an image enhancement model of a corresponding unmanned aerial vehicle to carry out image enhancement to obtain a plurality of enhanced images, and inputting the plurality of enhanced images into a panoramic stitching module of a ground control center to carry out panoramic stitching to obtain a panoramic stitching image;
the image enhancement model comprises an illumination estimation module, an illumination enhancement module and a reflection component enhancement module which are sequentially connected; the illumination estimation module is formed by connecting a plurality of convolutional neural networks in series; the reflection component enhancement module consists of a plurality of filtering submodules which are connected in sequence; the illumination enhancement module is an encoder-decoder network structure which introduces an attention mechanism;
the panorama stitching module is composed of a decomposition network, an image stitching module and a weight fusion module which are sequentially connected.
The embodiment provides a multi-unmanned aerial vehicle collaborative detection distributed image enhancement method, which can enhance illumination consistency images aiming at the problem of uneven illumination after panoramic stitching of images acquired by unmanned aerial vehicles in multiple areas, estimates illumination components of the images through a lightweight convolution network through an offline centralized training image enhancement network, extracts image reflection components according to retinal cortex theory (Retinal Cortex Theory, retinex), adopts a network structure of an encoder-decoder for illumination components of the images, and introduces an attention module of a U-Net network; the noise in the image is removed by adopting a filter aiming at the reflection component, and the brightness, the contrast and the color distribution of the image collected by the unmanned aerial vehicle can be identical by respectively enhancing the illumination component and the reflection component and then fusing. Embedding the trained image enhancement model into unmanned aerial vehicles, carrying out distributed collaborative enhancement on acquired images by each unmanned aerial vehicle, and then transmitting the enhanced images and local parameters to a ground control center for panoramic stitching and global information processing.
The image enhancement technology is integrated in the unmanned aerial vehicle as an image preprocessing step, the unmanned aerial vehicle clusters are distributed to cooperatively enhance the acquired images, the enhanced images and local parameters are transmitted to a ground control center to be subjected to global information processing and are transmitted to a panoramic stitching module to be subjected to panoramic stitching, the global information processing is used for jointly processing information of each unmanned aerial vehicle in the unmanned aerial vehicle clusters, and the global parameters are fed back to each unmanned aerial vehicle, so that the cooperation among each unmanned aerial vehicle is improved.
The image enhancement technology comprises an illumination estimation module, an illumination enhancement module and a reflection component enhancement module: the illumination estimation module consists of a plurality of small-sized light-weight convolutional neural networks connected in series, wherein the network comprises three different convolutional blocks, and the first convolutional block comprises a 3 multiplied by 3 convolutional kernel and a modified linear unit leak ReLU layer; the second convolution block comprises a 3 x 3 convolution kernel, a bulk normalization layer (Batch Normalization, BN) and a modified linear unit leak ReLU layer; the third convolution block comprises a 3 x 3 convolution kernel and an activation function Sigmoid layer, which estimates the illumination of the input image stage by stage through the convolution network, wherein the basic stage illumination estimation formula is:
in the method, in the process of the invention,is provided with training parameters->For learning illumination; />Residual term representing t-phase,/>Representing t->The illumination component of the phase is used to determine,Tis the total number of stages; y represents an original image; />A mapping function representing the entire illumination estimation phase; each phase uses a phaseSame->Architecture.
The illumination estimation module uses a smooth canonical loss function and a data fidelity loss function to ensure that the estimated illumination is smooth and has pixel consistency with each stage input, the loss function of the module is:
wherein,representing the weight coefficient, ++>Representing data fidelity loss function,/->Representing a smooth canonical loss function; />In the expression, T is the total number of stages, < >>And->Respectively representing illumination components of the t stage and the t-1 stage; />In the expression, N is the total number of pixels, i denotes the ith pixel, +.>Representing i adjacent pixels in its 5 x 5 window,/i>Representing weights, where c represents the image channel in YUV color space, +.>The standard deviation of the gaussian kernel is shown.
Residual connection:
the key feature of the illumination estimation module is that an enhanced image is generated through residual connection, and in the forward propagation process, the output of each convolution block is added to the previous feature map, so that the gradient disappearance problem in the training process is relieved, and the network is easier to learn residual information. The final output feature map is added to the input image to generate an enhanced image.
Output limit:
the pixel value of the output image (ill u) is limited to between 0.0001 and 1 to ensure that the generated image is not too bright or too dark.
The illumination estimation module realizes the illumination estimation function of the image through the connection of the depth convolution neural network and the residual error, and the proper activation function and the standardization layer.
The reflection component enhancement module consists of three filters, including a median filter, a contrast filter and a white balance filter. The median filter can remove salt and pepper noise and impulse noise in the reflection component, and can keep the edge and detail of the image, the basic idea of the median filter is to sequence pixels in a neighborhood window with a fixed size, take the pixel value at the middle position as a new value of the pixel, and the contrast filter adopts histogram equalization, so that the contrast is increased by widening gray values with a large number of pixels in the enhanced image (namely, gray values with a main effect on a picture), and merging gray values with a small number of pixels (namely, gray values with no main effect on the picture), so that the contrast is increased and the image is clear; the white balance filter finds a near white region containing a reference white point by analyzing the YCrCb (Y: luminance, chromaticity: cr and Cb, cr components representing red color difference, reflecting the difference between red and luminance; cb components representing blue color difference, reflecting the difference between blue and luminance) coordinate space of a picture using a dynamic threshold algorithm, defining certain points as reference white points by setting a threshold, and calculating the gain from the reference pointsParameters, adjusting white balance of the original image, correcting color offset and unbalance in the image, enabling the color of the image to be more accurate and natural, and enhancing the reflection component image r through the three filters to obtain an enhanced reflection component。
The illumination component enhancement module adopts a network structure of an encoder-decoder and introduces an attention module, the encoder is composed of three convolution layers and a pooling layer, the size and the channel number of a feature map are gradually reduced through convolution operation to capture global and local features in an image, the decoder is composed of three deconvolution layers and an up-sampling layer, and the detail and the structure of the image are restored through gradually increasing the size and the channel number of the feature map; the attention module adopts a U-Net network, the network is guided to adaptively guide the enhancement network to correctly enhance a low-illumination area through the network, the excessive enhancement of a normal illumination area is avoided, in addition, the illumination consistency loss is adopted to calculate the mean square error between an expected value of an illumination component and the enhanced illumination component to restrict the learning of the illumination enhancement network, so that the enhanced images acquired by a plurality of unmanned aerial vehicles have illumination consistency, the spliced images are even in illumination, and the illumination consistency loss function expression is as follows:
where N represents the number of images involved in the training,representing the i-th enhanced illumination component image,/->Representing a preset illumination component image.
The unmanned aerial vehicle cluster distributed collaborative enhancement is characterized in that the unmanned aerial vehicle cluster distributed collaborative enhancement is integrated into each unmanned aerial vehicle through the image enhancement network training model, when the unmanned aerial vehicle cluster executes regional detection tasks, each unmanned aerial vehicle applies the integrated image enhancement model to carry out parallel collaborative enhancement on the collected images of the region, then each unmanned aerial vehicle transmits the enhanced images and local parameters to a ground control center, the ground control center carries out global processing on the information of each unmanned aerial vehicle, the processed global information is returned to the unmanned aerial vehicle, the synergy among the unmanned aerial vehicles is increased, and the enhanced images of each unmanned aerial vehicle are transmitted into a panoramic stitching module to carry out panoramic stitching.
The panoramic stitching module is arranged in the ground control center and comprises a decomposition network, an image stitching module and a weight fusion module, wherein the image input decomposition network is decomposed into images with different high and low frequencies, the high-frequency images and the low-frequency images are respectively input into the image stitching module to generate a high-frequency stitching image and a low-frequency stitching image, and then the stitched high-frequency images and the stitched low-frequency images are input into the weight fusion module for fusion. The decomposition network carries out multi-layer decomposition on the original image through the Gaussian pyramid, then carries out up-sampling on each layer of the Gaussian pyramid, and makes difference between the previous layer Gaussian pyramid image and the expansion image of the current layer image to obtain a Laplacian pyramid, and the image is decomposed into high-frequency and low-frequency images with different frequency bands:
wherein L and G represent laplacian and gaussian pyramids, respectively, wherein top-level images of the two pyramids are identical, j represents the number of layers, j=0 at the lowest layer is the original image, and expand represents expansion operation. The image stitching module obtains feature matching point pairs through the existing method based on feature point matching, calculates a homography matrix by using the matched feature point pairs, carries out affine transformation on images by using the homography matrix, carries out weight fusion and crack removal processing on the images according to local parameters transmitted by each unmanned aerial vehicle, and repeats the processes until all image processing is completed. The weight fusion module performs linear fusion on the high-frequency and low-frequency spliced images by using different weights, and adopts a linearly decreasing weight coefficient sequence, so that the weight of a low-frequency part is higher and the weight of a high-frequency part is lower on a lower level of the pyramid.
According to the multi-unmanned aerial vehicle collaborative detection distributed image enhancement technology, as shown in fig. 1, offline centralized training is performed on an image enhancement network, then a trained model is integrated in unmanned aerial vehicles, when an unmanned aerial vehicle cluster performs collaborative detection tasks, the distributed collaborative enhancement is performed on collected images, the enhanced images and local parameters are transmitted to a ground control center for global information processing and panoramic stitching, the ground control center links unmanned aerial vehicles through global information processing, and the global parameters are fed back to each unmanned aerial vehicle, so that the collaborative performance among each unmanned aerial vehicle is improved.
The image enhancement network is composed of an illumination estimation module, an illumination component enhancement module and a reflection component enhancement module. The steps of the offline centralized training image enhancement module are described below:
step 1, selecting images with different illumination conditions in a low illumination enhancement data set LOL as a training set, and training an image enhancement network;
step 2, carrying out illumination estimation on the image through an illumination estimation module 1, wherein the illumination estimation module estimates the brightness of the input image step by step through a plurality of light convolution networks, and the weights of all the stages are shared and the network structures are the same;
step 3, the estimated illumination component imageThe light component enhancement module 2 performs adaptive enhancement processing by the light consistency loss function +.>Constraining the enhanced multiple images to have illumination consistency, inputting the estimated illumination component images into an encoder-decoder network by an illumination component enhancement module for illumination component enhancement, generating an attention map by an attention mechanism by a feature map in the encoder, fusing the attention map with the feature map of the decoder, adding information in the feature map, guiding a network adaptive guiding enhancement network to correctly enhance low illumination areas, avoidingThe area without excessively enhancing normal illumination realizes illumination component enhancement, the attention mechanism adopts U-Net network, and the value range of attention diagram is [0,1 ]]The calculation formula is as follows:
wherein,indicating return to maximum in three color channels, < >>For the original reference illumination component image, x represents the input illumination component image.
Step 4, removing the illumination component estimated in the step 2 in the input image to obtain a reflection component according to the Retinex theory, inputting the reflection component into a reflection component enhancement module 3 for denoising, contrast ratio and white balance enhancement to solve the problems of excessive noise and the like of the reflection component caused by low illumination or overexposure, denoising the input reflection component through a median filter, and then increasing the contrast ratio and adjusting the white balance through histogram equalization filtering and a dynamic threshold algorithm to make the reflection image clearer and more natural;
step 5, according to Retinex theory, the enhanced illumination component image is obtainedAnd reflection component image +.>Fusion is carried out to obtain enhanced image +.>The formula of the fused image is: />. To ensure fused image +.>Keeping consistent with the original image, adopting a reconstruction loss function:
wherein the method comprises the steps ofRepresenting an enhanced illumination component image, +.>Representing a reflected component image, +.>Representing the corresponding artwork.
The total loss of image enhancement network training can be expressed as:
wherein,representing the weight coefficient, ++>Representing data fidelity loss function,/->Representing a smooth canonical loss function, +.>Representing the illumination uniformity loss function,/->Representing the reconstruction loss function.
The unmanned aerial vehicle collaborative distributed enhancement steps are as follows:
step 1, integrating an offline centralized training image enhancement model into an unmanned aerial vehicle;
step 2, each unmanned aerial vehicle in the unmanned aerial vehicle cluster utilizes the distributed collaborative enhancement of the image enhancement model to acquire an image;
and 3, transmitting the enhanced image and the local parameters to a ground control center by each unmanned aerial vehicle for global information processing, inputting the enhanced image into a panorama splicing module for panorama splicing, and enabling the ground control center to link the unmanned aerial vehicles through global information processing and feed the global parameters back to each unmanned aerial vehicle by the ground control center, so that the synergy between each unmanned aerial vehicle is improved.
The global information processing flow adopts distributed federal processing to perform global information processing, local parameters are transmitted to a ground control center to perform global information processing, the global information processing is utilized to jointly process information of each unmanned aerial vehicle in the unmanned aerial vehicle cluster, and the global parameters are fed back to each unmanned aerial vehicle.
The local parameters transmitted to the ground control center by the unmanned aerial vehicle are illumination estimation componentsIn global information processing, determining weight values occupied by each unmanned aerial vehicle according to illumination consistency loss values, normalizing, and fusing illumination by using a weighted average method to obtain an optimal fusion result +.>：
Wherein the method comprises the steps ofRepresenting the weighting coefficients>Representing the estimated illumination component of the acquired image by the unmanned aerial vehicle.
The multi-machine collaborative distributed federal image enhancement adopts the principle of information conservation, distributes the fusion result of global information processing into each unmanned aerial vehicle according to weights, carries out reverse optimization on illumination estimation component data in each image enhancement model, and provides global illumination information for the next image enhancement of the unmanned aerial vehicle. Setting information sharing factors according to the information conservation theorem to meet the following conditions:
representing the information factor assigned by the drone.
The illumination component of the next image enhancement of the unmanned aerial vehicle is as follows:
wherein,representing the estimated illumination component of the unmanned aerial vehicle on the acquired image with image enhancement, < >>Information factor representing unmanned aerial vehicle allocation, +.>Representing the global illumination fusion component.
The panoramic stitching module is arranged at the ground control center, and the stitching steps are as follows:
the method comprises the steps that images acquired by a plurality of reinforced unmanned aerial vehicles are input into a panoramic stitching module for reinforcement, the panoramic stitching module comprises a decomposition network, an image stitching module and a weight fusion module, the image input decomposition network is decomposed into images with different high and low frequencies, the high-frequency images and the low-frequency images are input into the image stitching module to generate high and low-frequency stitched images, and then the stitched high and low-frequency images are input into the weight fusion module for fusion.
The foregoing is merely a preferred embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (9)

1. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method is characterized by comprising the following steps of:
acquiring a plurality of initial images acquired by a plurality of unmanned aerial vehicles;
respectively inputting a plurality of initial images into an image enhancement model of a corresponding unmanned aerial vehicle to carry out image enhancement to obtain a plurality of enhanced images, and inputting the plurality of enhanced images into a panoramic stitching module of a ground control center to carry out panoramic stitching to obtain a panoramic stitching image;
the image enhancement model comprises an illumination estimation module, an illumination enhancement module and a reflection component enhancement module which are sequentially connected; the illumination estimation module is formed by connecting a plurality of convolutional neural networks in series; the reflection component enhancement module consists of a plurality of filtering submodules which are connected in sequence; the illumination enhancement module is an encoder-decoder network structure which introduces an attention mechanism;
the panorama stitching module is composed of a decomposition network, an image stitching module and a weight fusion module which are sequentially connected.
2. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method according to claim 1, wherein the training process of the image enhancement model specifically comprises:
acquiring training data; the training data comprises an initial training image and a corresponding enhanced image;
and inputting the training data into an illumination estimation module, an illumination enhancement module and a reflection component enhancement module in sequence, and training by taking the minimum loss between the output result of the reflection component enhancement module and the enhancement image corresponding to the initial training image as a target to obtain the image enhancement model.
3. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method according to claim 1, wherein the processing of the image enhancement model comprises:
inputting the initial image into the illumination estimation module for illumination estimation to obtain illumination estimation component data;
taking the illumination estimation component data as the input of the illumination enhancement module, and performing illumination component enhancement processing to obtain an illumination component enhanced image;
dividing the initial image and the illumination estimation component data, taking the operation result as the input of the reflection component enhancement module, and enhancing the reflection component to obtain a reflection component enhanced image;
and fusing the illumination component enhanced image and the reflection component enhanced image to obtain the enhanced image.
4. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method according to claim 3, wherein the initial image is input into the illumination estimation module for illumination estimation to obtain illumination estimation component data, and the method specifically comprises the following steps:
estimating the brightness of the initial image stage by stage through a plurality of convolutional neural networks in the illumination estimation module to obtain illumination estimation component data;
the calculation formula for acquiring the illumination estimation component data is as follows:
，
in the method, in the process of the invention,is provided with training parameters->For learning illumination; />Residual term representing t-phase,/>Representing tThe illumination component of the phase is used to determine,Tis the total number of stages; y represents an original image; />A mapping function representing the entire illumination estimation phase; the same +.>Architecture.
5. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method according to claim 3, wherein the illumination estimation component data is used as the input of the illumination enhancement module to perform illumination component enhancement processing to obtain an illumination component enhanced image, and the method specifically comprises the following steps:
performing illumination component enhancement on the illumination estimation component data through an encoder-decoder network in the illumination enhancement module, inputting the illumination estimation component data into an encoder to perform global feature capture and local feature capture, and further performing detail recovery and structure recovery on an output result of the encoder through a decoder;
adding an attention module in the encoder-decoder network, applying an attention mechanism to the output result of the encoder through the attention module to obtain an attention map, and fusing the attention map with the output result of the decoder to obtain an illumination component enhanced image;
the calculation formula for obtaining the attention map is as follows:
，
in the method, in the process of the invention,Ain order to take care of the force of the drawing,indicating return to maximum in three color channels, < >>For the original reference illumination component image, x represents the input illumination component image.
6. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method according to claim 3, wherein the method is characterized in that the initial image and illumination estimation component data are subjected to division operation, the operation result is used as the input of the reflection component enhancement module, reflection component enhancement is performed, and a reflection component enhanced image is obtained, and specifically comprises the following steps:
and dividing the initial image and the illumination estimation component data to obtain reflection component data, inputting the reflection component data into the reflection component enhancement module, and sequentially carrying out denoising, contrast and white balance enhancement processing on the reflection component data through a plurality of filtering submodules in the reflection component enhancement module to obtain the reflection component enhanced image.
7. The method of claim 6, wherein,
the filtering submodules are composed of a median filtering submodule, a histogram equalization filtering submodule and a white balance filtering submodule which are connected in sequence.
8. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method according to claim 1, wherein the processing procedure of the panorama stitching module comprises:
inputting the enhanced image into a decomposition network for decomposition to obtain a high-frequency image and a low-frequency image:
，
wherein L and G respectively represent Laplacian and Gaussian pyramid, j represents the number of layers, j=0 is the original image, and expansion represents expansion operation;
taking the high-frequency image and the low-frequency image as the input of the image stitching module, acquiring feature matching points between the high-frequency image and the low-frequency image based on a feature point matching method, calculating a homography matrix based on the feature matching points and the data, further carrying out affine transformation on the high-frequency image and the low-frequency image by utilizing the homography matrix to obtain a high-frequency stitched image and a low-frequency stitched image,
inputting the high-frequency spliced image and the low-frequency spliced image into the weight fusion module for fusion,
and carrying out weight fusion and crack removal processing on each spliced image according to illumination estimation component data of the image enhancement model corresponding to each spliced image to obtain the panoramic spliced image.
9. The multi-unmanned aerial vehicle collaborative detection distributed image enhancement method of claim 8, further comprising, after obtaining the panoramic stitched image: reversely optimizing illumination estimation component data of the image enhancement model of each unmanned aerial vehicle by utilizing an information conservation principle;
the reverse optimization of the illumination estimation component data of the image enhancement model of each unmanned aerial vehicle by utilizing the information conservation principle comprises the following specific processes:
determining weight values occupied by each unmanned aerial vehicle according to the illumination consistency loss values, carrying out normalization processing on the weight values occupied by each unmanned aerial vehicle, and fusing the illumination estimation component data by using a weighted average method after the normalization processing is finished to obtain a global illumination fusion component：
，
In the method, in the process of the invention,for the weighting factor>Estimating a component, namely a local parameter, for illumination of the unmanned aerial vehicle;
based on the principle of conservation of information, fusing the global illumination componentsAnd reversely optimizing the illumination estimation component data in the image enhancement model distributed to each unmanned aerial vehicle according to the weight.
CN202311627840.6A 2023-12-01 2023-12-01 Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method Active CN117372307B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311627840.6A CN117372307B (en) 2023-12-01 2023-12-01 Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311627840.6A CN117372307B (en) 2023-12-01 2023-12-01 Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method

Publications (2)

Publication Number Publication Date
CN117372307A true CN117372307A (en) 2024-01-09
CN117372307B CN117372307B (en) 2024-02-23

Family

ID=89398663

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311627840.6A Active CN117372307B (en) 2023-12-01 2023-12-01 Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method

Country Status (1)

Country Link
CN (1) CN117372307B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210272236A1 (en) * 2019-02-28 2021-09-02 Tencent Technology (Shenzhen) Company Limited Image enhancement method and apparatus, and storage medium
CN116152120A (en) * 2023-04-20 2023-05-23 南京大学 Low-light image enhancement method and device integrating high-low frequency characteristic information
CN116596799A (en) * 2023-06-02 2023-08-15 南京邮电大学 Low-illumination image enhancement method based on channel space compound attention
CN116645305A (en) * 2023-05-19 2023-08-25 西安电子科技大学 Low-light image enhancement method based on multi-attention mechanism and Retinex

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210272236A1 (en) * 2019-02-28 2021-09-02 Tencent Technology (Shenzhen) Company Limited Image enhancement method and apparatus, and storage medium
CN116152120A (en) * 2023-04-20 2023-05-23 南京大学 Low-light image enhancement method and device integrating high-low frequency characteristic information
CN116645305A (en) * 2023-05-19 2023-08-25 西安电子科技大学 Low-light image enhancement method based on multi-attention mechanism and Retinex
CN116596799A (en) * 2023-06-02 2023-08-15 南京邮电大学 Low-illumination image enhancement method based on channel space compound attention

Also Published As

Publication number Publication date
CN117372307B (en) 2024-02-23

Similar Documents

Publication Publication Date Title
CN112734650B (en) Virtual multi-exposure fusion based uneven illumination image enhancement method
CN111986120A (en) Low-illumination image enhancement optimization method based on frame accumulation and multi-scale Retinex
CN112419181B (en) Method for enhancing detail of wide dynamic infrared image
CN107292830B (en) Low-illumination image enhancement and evaluation method
Liu et al. Survey of natural image enhancement techniques: Classification, evaluation, challenges, and perspectives
CN110163807B (en) Low-illumination image enhancement method based on expected bright channel
CN110428389B (en) Low-light-level image enhancement method based on MSR theory and exposure fusion
CN114998141B (en) Space environment high dynamic range imaging method based on multi-branch network
CN110009574B (en) Method for reversely generating high dynamic range image from low dynamic range image
CN102646267B (en) Degraded image restoration method and system
CN112801925A (en) Underwater image enhancement method for maximally eliminating water ripple influence based on information entropy
Wang et al. Low-light image enhancement based on virtual exposure
CN115578297A (en) Generalized attenuation image enhancement method for self-adaptive color compensation and detail optimization
CN116681606A (en) Underwater uneven illumination image enhancement method, system, equipment and medium
CN112927162A (en) Low-illumination image oriented enhancement method and system
CN110675332A (en) Method for enhancing quality of metal corrosion image
Parihar et al. A comprehensive analysis of fusion-based image enhancement techniques
CN115731146A (en) Multi-exposure image fusion method based on color gradient histogram feature light stream estimation
Wen et al. Autonomous robot navigation using Retinex algorithm for multiscale image adaptability in low-light environment
Huang et al. Underwater image enhancement based on color restoration and dual image wavelet fusion
CN117611467A (en) Low-light image enhancement method capable of balancing details and brightness of different areas simultaneously
CN117422653A (en) Low-light image enhancement method based on weight sharing and iterative data optimization
CN117372307B (en) Multi-unmanned aerial vehicle collaborative detection distributed image enhancement method
Huang et al. An end-to-end dehazing network with transitional convolution layer
CN114998173A (en) High dynamic range imaging method for space environment based on local area brightness adjustment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant