CN115239603A - Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network - Google Patents

Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network Download PDF

Info

Publication number
CN115239603A
CN115239603A CN202211166059.9A CN202211166059A CN115239603A CN 115239603 A CN115239603 A CN 115239603A CN 202211166059 A CN202211166059 A CN 202211166059A CN 115239603 A CN115239603 A CN 115239603A
Authority
CN
China
Prior art keywords
image
neural network
enhanced
dim light
branch
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211166059.9A
Other languages
Chinese (zh)
Inventor
许达文
张祥建
张旻晋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengdu Shihaixintu Microelectronics Co ltd
Original Assignee
Chengdu Shihaixintu Microelectronics Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengdu Shihaixintu Microelectronics Co ltd filed Critical Chengdu Shihaixintu Microelectronics Co ltd
Priority to CN202211166059.9A priority Critical patent/CN115239603A/en
Publication of CN115239603A publication Critical patent/CN115239603A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/17—Terrestrial scenes taken from planes or by drones
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Remote Sensing (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention provides an unmanned aerial vehicle aerial image dim light enhancing method based on a multi-branch neural network, which comprises the steps of obtaining unmanned aerial vehicle aerial dim light images under different scenes and constructing a dim light image data set; selecting image data in a dim light image data set, inputting a pre-established multi-branch dim light enhancement neural network, and outputting a color image with the same size as an input image by performing feature extraction and feature enhancement on the input image data at different levels; fusing the color image characteristics of different branches to obtain a final enhanced image; constructing model loss of the multi-branch dim light enhanced neural network, taking the enhanced image as a training sample, and training the multi-branch dim light enhanced neural network until the model loss is converged to obtain a trained network model; and (4) enhancing the aerial dark light image of the unmanned aerial vehicle to be tested by using the trained model. The proposal effectively solves the problem of poor image quality of aerial images under the dark light condition.

Description

Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network
Technical Field
The invention relates to the technical field of computer vision image processing, in particular to an unmanned aerial vehicle aerial image dim light enhancing method based on a multi-branch neural network.
Background
Under the dim light environment, because the restriction of illumination condition, the obvious defects such as the dark local feature of the color image colour that obtains is not outstanding can appear information loss and unexpected noise in the dark space and lead to unmanned aerial vehicle to take photo by plane the rate of accuracy of discernment and the impression directly perceived of image when can't receive sufficient light at the shooting in-process, all can produce huge influence to the effect of taking photo by plane.
Disclosure of Invention
In order to make up for the defects in the prior art, the invention provides the unmanned aerial vehicle aerial image dim light enhancing method based on the multi-branch neural network, which solves the problem of poor image quality of an aerial image under the dim light condition, improves the accuracy of aerial identification of the unmanned aerial vehicle, and enhances the shooting effect of the unmanned aerial vehicle in the dim light environment.
The purpose of the invention is realized by adopting the following technical scheme:
an unmanned aerial vehicle aerial image dim light enhancing method based on a multi-branch neural network comprises the following steps:
acquiring unmanned aerial vehicle aerial dark light images in different scenes, and constructing a dark light image data set;
selecting image data in a dim light image data set, inputting a pre-established multi-branch dim light enhancement neural network, and outputting a color image with the same size as an input image by performing feature extraction and feature enhancement on the input image data at different levels;
fusing the color image characteristics of different branches to obtain a final enhanced image;
constructing model loss of a multi-branch dim light enhanced neural network, and training the multi-branch dim light enhanced neural network by taking the enhanced image as a training sample until the model loss is converged to obtain a trained network model;
and enhancing the aerial dark light image of the unmanned aerial vehicle to be tested by using the trained model.
Preferably, the constructing the dim-light image dataset comprises: acquiring photos shot under normal light and dark light states in different scenes through preset angles and positions of the unmanned aerial vehicle, and performing image cutting and zooming on the photos to a 255X 255-sized three-channel color image;
collecting three-channel color image data, and dividing the collected data in proportion; of these, 80% are used as training set and 20% are used as test set.
Preferably, the pre-establishing of the multi-branch dim light enhancement neural network comprises:
acquiring image data in a training set as an input image, performing feature extraction of different levels on the input image through a feature extraction module, and performing feature enhancement on the extracted image features through an enhancement module to generate a color image with the same size as the input image; the feature extraction module is a 10-layer convolutional neural network, and each layer uses 32 3X3 convolutional kernels for feature extraction; after each convolution the ReLu activation function is set for non-linear mapping.
Preferably, the enhancement module comprises a 10-layer convolutional neural network, and each layer of network is formed by connecting 8 3X3 convolutional kernels, 16 5X5 deconvolution kernels, 8 5X5 deconvolution kernels and 3 5X5 deconvolution kernels in series; after each convolution a ReLu activation function is set for non-linear mapping.
Preferably, the enhanced image is an image feature enhanced by fusion module using 3-channel 1X1 convolution kernel fusion.
Preferably, the model loss of the multi-branch dim light enhancement neural network comprises structure loss and context loss;
wherein, the structure loss is used for measuring the difference between the enhanced image and the corresponding normal light image and guiding the learning process;
the context loss is used for measuring the difference between the enhanced image and the normal light image corresponding to the enhanced image.
Further, the structural loss is determined by:
Figure 605343DEST_PATH_IMAGE001
wherein,
Figure 725746DEST_PATH_IMAGE002
in order to be a loss of the structure,
Figure 974325DEST_PATH_IMAGE003
the average values of the pixel values of the enhanced image and the normal light image are respectively;
Figure 572796DEST_PATH_IMAGE004
is the variance of the two pixel values;
Figure 856010DEST_PATH_IMAGE005
is a hyper-parameter.
Further, the context loss is determined by:
Figure 147314DEST_PATH_IMAGE006
wherein E and G are respectively an enhanced image and a real image,
Figure 148768DEST_PATH_IMAGE007
the sizes of the characteristic graphs in the pre-trained VGG-19 network are respectively described;
Figure 315045DEST_PATH_IMAGE008
and representing the feature mapping obtained by the jth convolutional layer of the ith feature map in the pre-trained VGG-19 network.
Preferably, the constructing a model loss of the multi-branch dim light enhanced neural network, taking the enhanced image as a training sample, and training the multi-branch dim light enhanced neural network until the model loss converges, so as to obtain a trained network model includes: inputting corresponding training samples according to the sample input quantity of the predefined model training, and performing iterative training on the model; for the characteristic color extracted from the VGG-19 network of the normal light image in the context loss, the output of the fourth convolution layer of the third block of the VGG-19 network is used as a context loss extraction layer in the training process;
and updating network parameters through an Adam optimization algorithm, and saving the model with the highest test result as a final model after iteration is finished.
Preferably, the enhancing the unmanned aerial vehicle aerial dark light image to be tested by using the trained model comprises:
and acquiring a dark light image shot by the unmanned aerial vehicle in a dark light environment, zooming the shot dark light image to 255X255, inputting the zoomed dark light image into a trained model, and acquiring an enhanced picture.
Compared with the prior art, the invention has the following beneficial effects:
the invention provides a dark light enhancement method for aerial images of unmanned aerial vehicles based on a multi-branch neural network, which comprises the steps of firstly, obtaining aerial dark light images of unmanned aerial vehicles under different scenes to construct a dark light image data set; selecting image data in a dim light image data set, inputting a pre-established multi-branch dim light enhancement neural network, and outputting a color image with the same size as an input image by performing feature extraction and feature enhancement on the input image data at different levels; secondly, fusing the color image characteristics of different branches to obtain a final enhanced image; finally, constructing model loss of the multi-branch dim light enhanced neural network, taking the enhanced image as a training sample, and training the multi-branch dim light enhanced neural network until the model loss is converged to obtain a trained network model; and enhancing the aerial dark light image of the unmanned aerial vehicle to be tested by using the trained model. Therefore, the problem of poor image quality of aerial images under the dark light condition is effectively solved.
The invention provides innovation from the method, does not rely on the inherent mode of the traditional unmanned aerial vehicle shooting, and makes up the shooting defects of dark color image, unobtrusive local characteristics and the like caused by the limitation of illumination conditions. The accuracy of unmanned aerial vehicle aerial photography discernment is improved, shooting effect under the reinforcing unmanned aerial vehicle dim light environment.
Drawings
In order to more clearly illustrate the detailed description of the invention or the technical solutions in the prior art, the drawings used in the detailed description or the prior art description will be briefly described below. Throughout the drawings, like elements or portions are generally identified by like reference numerals. In the drawings, elements or portions are not necessarily drawn to scale.
FIG. 1 shows a flow chart of a method for enhancing dim light of an aerial image of an unmanned aerial vehicle based on a multi-branch neural network, provided by the invention;
FIG. 2 shows a schematic structural diagram of a system for enhancing the dim light of an aerial image of an unmanned aerial vehicle based on a multi-branch neural network, provided by the invention;
fig. 3 shows a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and therefore are only examples, and the protection scope of the present invention is not limited thereby.
It is to be noted that, unless otherwise specified, technical or scientific terms used herein shall have the ordinary meaning as understood by those skilled in the art to which the invention pertains.
As shown in fig. 1, the embodiment of the present invention provides a method for enhancing dim light of an aerial image of an unmanned aerial vehicle based on a multi-branch neural network, the method comprising the following steps:
s1, acquiring unmanned aerial vehicle aerial photography dark light images in different scenes, and constructing a dark light image data set;
s2, selecting image data in the dim light image data set, inputting a pre-established multi-branch dim light enhancement neural network, and outputting a color image with the same size as an input image by performing different levels of feature extraction and feature enhancement on the input image data; fusing the color image characteristics of different branches to obtain a final enhanced image;
s3, constructing model loss of the multi-branch dim light enhanced neural network, and training the multi-branch dim light enhanced neural network by taking the enhanced image as a training sample until the model loss is converged to obtain a trained network model;
and S4, enhancing the aerial photography dim light image of the unmanned aerial vehicle to be tested by using the trained model.
In step S1, the process of constructing the dim-light image data set includes: acquiring photos shot under normal light and dark light states in different scenes through preset angles and positions of the unmanned aerial vehicle, and cutting and zooming the photos into three-channel color images with the size of 255X 255;
collecting three-channel color image data, and dividing the collected data in proportion; of these, 80% are training sets and 20% are test sets. A training set for storing image data for model training; and the test set is used for storing the image data to be tested.
In step S2, establishing a multi-branch dim light enhancing neural network includes:
acquiring image data in a training set as an input image, performing feature extraction of different levels on the input image through a feature extraction module, and performing feature enhancement on the extracted image features through an enhancement module to generate a color image with the same size as the input image; the feature extraction module is a 10-layer convolutional neural network, and each layer uses 32 3X3 convolutional kernels for feature extraction; after each convolution a ReLu activation function is set for non-linear mapping.
The enhancement module comprises 10 layers of convolutional neural networks, wherein each layer of convolutional neural network is formed by connecting 8 3X3 convolutional kernels, 16 5X5 deconvolution kernels, 8 5X5 deconvolution kernels and 3X 5 deconvolution kernels in series; after each convolution the ReLu activation function is set for non-linear mapping.
The enhanced image in step S2 is obtained by fusing the enhanced image features by a fusion module using a 3-channel 1X1 convolution kernel.
In step S3, model loss of the multi-branch dim light enhanced neural network comprises structure loss and context loss; wherein, the structure loss is used for measuring the difference between the enhanced image and the corresponding normal light image and guiding the learning process;
and context loss for measuring the difference between the enhanced image and the normal light image corresponding to the enhanced image.
The structural loss is determined by the following formula:
Figure 718344DEST_PATH_IMAGE001
wherein,
Figure 180550DEST_PATH_IMAGE002
in order to be a loss of the structure,
Figure 669300DEST_PATH_IMAGE003
the average values of the pixel values of the enhanced image and the normal light image are respectively;
Figure 875153DEST_PATH_IMAGE004
is the variance of the two pixel values;
Figure 132959DEST_PATH_IMAGE009
is a hyper-parameter.
The context loss is determined by:
Figure 31645DEST_PATH_IMAGE006
wherein E and G are respectively an enhanced image and a real image,
Figure 7691DEST_PATH_IMAGE007
the sizes of all characteristic graphs in the pre-trained VGG-19 network are respectively described;
Figure 751657DEST_PATH_IMAGE008
representing the feature map obtained by the jth convolutional layer of the ith feature map in the pre-trained VGG-19 network. The VGG-19 network is a classical convolutional neural network structure, which comprises 5 blocks, each block comprises 2,4 convolutional layers respectively, and different convolutional layers in each block comprise the same convolutional kernel size, 64,128,256,512 respectively. The output sizes of the convolution layers in the blocks are the same, and are respectively as follows: 112X 112, 56X 56, 28X 28, 14X 14 and 7X 7. That is, the value of i in the above formula is 1-5, and the value of j corresponds to the number of convolution layers in each block, which is 1-2,1-4, 1-4, and 1-4, respectively.W i,j AndH i,j is the same in each block, i.e. in five blocksW i,j AndH i,j the values of (b) are the length and width of the output feature map, respectively (112 ), (56, 56), (28, 28), (14, 14), (7, 7).C i,j The number of convolution kernels in each block, which is the same for each block, is 64,128,256,512, respectively. In step S3, it is mentioned later that "for the characteristic color extracted in the context loss normal light image VGG, the output of the fourth convolution layer of the third block of the VGG-19 network is used as the context loss extraction layer in the training process", and therefore, when the calculation formula is specifically executed, i =3, j =4,W i,j =28 andH i,j =28，C i,j =256”。
in step S3, training the multi-branch dim light enhanced neural network with the enhanced image as a training sample includes: inputting corresponding training samples according to the sample input quantity of the predefined model training, and performing iterative training on the model; for the characteristic color extracted from the VGG-19 network model of the normal light image in the context loss, the output of a fourth convolution layer of a third block of the VGG-19 network is used as a context loss extraction layer in the training process;
and updating network parameters through an Adam optimization algorithm, and saving the model with the highest test result as a final model after iteration is finished.
In the step S4, the step of enhancing the aerial photography dim light image of the unmanned aerial vehicle to be tested by using the trained model comprises the following steps:
and acquiring a dark light image shot by the unmanned aerial vehicle in a dark light environment, zooming the shot dark light image to 255X255, inputting the zoomed dark light image into a trained model, and acquiring an enhanced picture.
Example 1: the embodiment 1 of the invention provides an unmanned aerial vehicle aerial image dim light enhancement system based on a multi-branch neural network by combining the inventive principles of the above contents and further based on the same technical concept of the unmanned aerial vehicle aerial image dim light enhancement method based on the multi-branch neural network.
As shown in fig. 2, in this embodiment, the unmanned aerial vehicle aerial image dim light enhancement system 100 based on the multi-branch neural network includes:
the building module 110 is configured to obtain the dark light images of the unmanned aerial vehicle aerial photography in different scenes, and build a dark light image dataset;
the image acquisition module 120 is configured to select image data from the dim-light image data set, input a pre-established multi-branch dim-light enhancement neural network, perform feature extraction and feature enhancement on the input image data at different levels, and output a color image having the same size as the input image; fusing the color image characteristics of different branches to obtain a final enhanced image;
the model training module 130 is configured to construct a model loss of the multi-branch dim light enhanced neural network, train the multi-branch dim light enhanced neural network with the enhanced image as a training sample until the model loss converges, and obtain a trained network model;
and the image enhancement module 140 is used for enhancing the aerial dark light image of the unmanned aerial vehicle to be tested by using the trained model.
Example 2: referring to fig. 3, embodiment 2 provides an electronic device 200 according to an embodiment of the present application. The electronic device 200 includes: a transceiver 210, a memory 220, a communication bus 230, and a processor 240.
The elements of the transceiver 210, the memory 220, and the processor 240 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, the components may be electrically coupled to each other via one or more communication buses 230 or signal lines. The transceiver 210 is used for transceiving data. The memory 220 is used for storing a computer program, such as a software functional module shown in fig. 2, that is, the unmanned aerial vehicle aerial image dim light enhancement system 100 based on the multi-branch neural network. The unmanned aerial vehicle aerial image dim light enhancement system 100 based on the multi-branch neural network includes at least one software functional module stored in the memory 220 in the form of software or firmware (firmware) or solidified in an Operating System (OS) of the electronic device 200. The processor 240 is configured to execute executable modules stored in the memory 220, for example, software functional modules or computer programs included in the unmanned aerial vehicle image dim light enhancement system 100 based on the multi-branch neural network.
At this time, the processor 240 is configured to obtain the dark light images of the unmanned aerial vehicle aerial photography in different scenes, and construct a dark light image dataset;
selecting image data in a dim light image data set, inputting a pre-established multi-branch dim light enhancement neural network, and outputting a color image with the same size as an input image by performing feature extraction and feature enhancement on the input image data at different levels; fusing the color image characteristics of different branches to obtain a final enhanced image;
constructing model loss of a multi-branch dim light enhanced neural network, and training the multi-branch dim light enhanced neural network by taking the enhanced image as a training sample until the model loss is converged to obtain a trained network model;
and (4) enhancing the aerial dark light image of the unmanned aerial vehicle to be tested by using the trained model.
The Memory 220 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
Processor 240 may be an integrated circuit chip having signal processing capabilities, such as an IMX6 chip running a Vxworks (embedded real-time operating system). The Processor may also be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor 240 may be any conventional processor or the like.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present invention, and they should be construed as being included in the following claims and description.

Claims (10)

1. An unmanned aerial vehicle aerial image dim light enhancing method based on a multi-branch neural network is characterized by comprising the following steps:
acquiring unmanned aerial vehicle aerial photography dim light images in different scenes, and constructing a dim light image data set;
selecting image data in a dim light image data set, inputting a pre-established multi-branch dim light enhancement neural network, and outputting a color image with the same size as an input image by performing feature extraction and feature enhancement on the input image data at different levels; fusing the color image characteristics of different branches to obtain a final enhanced image;
constructing model loss of the multi-branch dim light enhanced neural network, and training the multi-branch dim light enhanced neural network by taking the enhanced image as a training sample until the model loss is converged to obtain a trained network model;
and (4) enhancing the aerial dark light image of the unmanned aerial vehicle to be tested by using the trained model.
2. The method of claim 1, wherein constructing a dim-light image dataset comprises: acquiring photos shot under normal light and dark light states in different scenes through preset angles and positions of the unmanned aerial vehicle, and performing image cutting and zooming on the photos to a 255X 255-sized three-channel color image;
collecting three-channel color image data, and dividing the collected data in proportion; of these, 80% are used as training set and 20% are used as test set.
3. The method of claim 1, wherein the pre-establishing of the multi-branch dim light enhancing neural network comprises:
acquiring image data in a training set as an input image, performing feature extraction of different levels on the input image through a feature extraction module, and performing feature enhancement on the extracted image features through an enhancement module to generate a color image with the same size as the input image; the feature extraction module is a 10-layer convolutional neural network, and each layer uses 32 convolution kernels of 3X3 to extract features; after each convolution a ReLu activation function is set for non-linear mapping.
4. The method of claim 1, wherein the enhancement module comprises 10 layers of convolutional neural network, each layer of network consisting of 8 3X3 convolutional kernels, 16 5X5 deconvolution kernels, 8 5X5 deconvolution kernels, and 3 5X5 deconvolution kernels in series; after each convolution a ReLu activation function is set for non-linear mapping.
5. The method of claim 1, the enhanced image being an image feature enhanced by a fusion module using a 3-channel 1X1 convolution kernel fusion.
6. The method of claim 1, wherein the model loss of the multi-branch dim light-enhancing neural network comprises structural loss, contextual loss;
wherein, the structure loss is used for measuring the difference between the enhanced image and the corresponding normal light image and guiding the learning process;
the context loss is used for measuring the difference between the enhanced image and the normal light image corresponding to the enhanced image.
7. The method of claim 6, wherein the structural loss is determined by:
Figure 966431DEST_PATH_IMAGE001
wherein,
Figure 86834DEST_PATH_IMAGE002
in order to be a loss of the structure,
Figure 725625DEST_PATH_IMAGE003
the average values of the pixel values of the enhanced image and the normal light image are respectively;
Figure 589676DEST_PATH_IMAGE004
the variance of the two pixel values;
Figure 482677DEST_PATH_IMAGE005
is a hyper-parameter.
8. The method of claim 7, wherein the context loss is determined by:
Figure 836298DEST_PATH_IMAGE006
wherein E and G are respectively an enhanced image and a real image,
Figure 572173DEST_PATH_IMAGE007
the sizes of all characteristic graphs in the pre-trained VGG-19 network are respectively described;
Figure 364548DEST_PATH_IMAGE008
representing the feature map obtained by the jth convolutional layer of the ith feature map in the pre-trained VGG-19 network.
9. The method of claim 1, wherein constructing a model loss of the multi-branch dim-light-enhanced neural network, and training the multi-branch dim-light-enhanced neural network with the enhanced image as a training sample until the model loss converges to obtain a trained network model comprises: inputting corresponding training samples according to the sample input quantity of the predefined model training, and performing iterative training on the model; for the characteristic color extracted from the VGG-19 network of the normal light image in the context loss, the output of the fourth convolution layer of the third block of the VGG-19 network is used as a context loss extraction layer in the training process;
and updating network parameters through an Adam optimization algorithm, and saving the model with the highest test result as a final model after iteration is finished.
10. The method of claim 1, wherein the enhancing the drone to be tested aerial scout image using the trained model comprises:
and acquiring a dim light image shot by the unmanned aerial vehicle in a dim light environment, zooming the shot dim light image to 255X255 size, inputting the zoomed dim light image into a trained model, and acquiring an enhanced picture.
CN202211166059.9A 2022-09-23 2022-09-23 Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network Pending CN115239603A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211166059.9A CN115239603A (en) 2022-09-23 2022-09-23 Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211166059.9A CN115239603A (en) 2022-09-23 2022-09-23 Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network

Publications (1)

Publication Number Publication Date
CN115239603A true CN115239603A (en) 2022-10-25

Family

ID=83667043

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211166059.9A Pending CN115239603A (en) 2022-09-23 2022-09-23 Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network

Country Status (1)

Country Link
CN (1) CN115239603A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108986050A (en) * 2018-07-20 2018-12-11 北京航空航天大学 A kind of image and video enhancement method based on multiple-limb convolutional neural networks
CN112183236A (en) * 2020-09-10 2021-01-05 佛山聚卓科技有限公司 Unmanned aerial vehicle aerial photography video content identification method, device and system
CN112819707A (en) * 2021-01-15 2021-05-18 电子科技大学 End-to-end anti-blocking effect low-illumination image enhancement method
CN114998145A (en) * 2022-06-07 2022-09-02 湖南大学 Low-illumination image enhancement method based on multi-scale and context learning network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108986050A (en) * 2018-07-20 2018-12-11 北京航空航天大学 A kind of image and video enhancement method based on multiple-limb convolutional neural networks
CN112183236A (en) * 2020-09-10 2021-01-05 佛山聚卓科技有限公司 Unmanned aerial vehicle aerial photography video content identification method, device and system
CN112819707A (en) * 2021-01-15 2021-05-18 电子科技大学 End-to-end anti-blocking effect low-illumination image enhancement method
CN114998145A (en) * 2022-06-07 2022-09-02 湖南大学 Low-illumination image enhancement method based on multi-scale and context learning network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
FEIFAN LV等: "MBLLEN: Low-light Image/Video Enhancement Using CNNs", 《CONFERENCE: BMVC》 *

Similar Documents

Publication Publication Date Title
US11521011B2 (en) Method and apparatus for training neural network model for enhancing image detail
US20220188999A1 (en) Image enhancement method and apparatus
CN108921161B (en) Model training method and device, electronic equipment and computer readable storage medium
CN108304821B (en) Image recognition method and device, image acquisition method and device, computer device and non-volatile computer-readable storage medium
CN111598799A (en) Image toning enhancement method and image toning enhancement neural network training method
CN108764372A (en) Construction method and device, mobile terminal, the readable storage medium storing program for executing of data set
CN112446380A (en) Image processing method and device
CN109102885B (en) Automatic cataract grading method based on combination of convolutional neural network and random forest
WO2019223513A1 (en) Image recognition method, electronic device and storage medium
CN110276831B (en) Method and device for constructing three-dimensional model, equipment and computer-readable storage medium
CN112465840A (en) Semantic segmentation model training method, semantic segmentation method and related device
US11138693B2 (en) Attention-driven image manipulation
CN110991412A (en) Face recognition method and device, storage medium and electronic equipment
CN114565539B (en) Image defogging method based on online knowledge distillation
WO2024153156A1 (en) Image processing method and apparatus, and device and medium
CN111862040A (en) Portrait picture quality evaluation method, device, equipment and storage medium
CN110717864A (en) Image enhancement method and device, terminal equipment and computer readable medium
CN111354058B (en) Image coloring method and device, image acquisition equipment and readable storage medium
CN115239603A (en) Unmanned aerial vehicle aerial image dim light enhancing method based on multi-branch neural network
CN114175093A (en) Detection device and detection method for display panel, electronic device and readable medium
CN111583166A (en) Image fusion network model construction and training method and device
CN116137023A (en) Low-illumination image enhancement method based on background modeling and detail enhancement
Yuan et al. A plug-and-play image enhancement model for end-to-end object detection in low-light condition
US11606544B2 (en) Neural network based auto-white-balancing
CN111612690B (en) Image splicing method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20221025

RJ01 Rejection of invention patent application after publication