CN113723231A - Low-illumination semantic segmentation model training method, semantic segmentation method and semantic segmentation device - Google Patents
Low-illumination semantic segmentation model training method, semantic segmentation method and semantic segmentation device Download PDFInfo
- Publication number
- CN113723231A CN113723231A CN202110940177.XA CN202110940177A CN113723231A CN 113723231 A CN113723231 A CN 113723231A CN 202110940177 A CN202110940177 A CN 202110940177A CN 113723231 A CN113723231 A CN 113723231A
- Authority
- CN
- China
- Prior art keywords
- semantic segmentation
- low
- image
- loss function
- illumination
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 107
- 238000005286 illumination Methods 0.000 title claims abstract description 41
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000012549 training Methods 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 5
- 238000003062 neural network model Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 239000004576 sand Substances 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 2
- 230000009466 transformation Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000011541 reaction mixture Substances 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a training method of a low-illumination semantic segmentation model, a semantic segmentation method and a device, wherein the training method of the low-illumination semantic segmentation model comprises the following steps: acquiring an image data set under low illumination; respectively carrying out enhancement processing on the acquired image data through an image enhancement model to obtain an enhanced image data set and an enhanced image loss function; respectively performing semantic segmentation on the acquired enhanced image data through a semantic segmentation model to acquire a semantic segmentation image data set and a semantic segmentation loss function; obtaining a total loss function according to the enhanced image loss function and the semantic segmentation loss function; and adjusting the semantic segmentation model according to a total loss function to obtain a low-illumination semantic segmentation model. The method is used for obtaining accurate segmentation precision in a low-illumination scene.
Description
Technical Field
The invention relates to image processing and semantic segmentation technologies, in particular to a low-illumination semantic segmentation model training method, a semantic segmentation method and a semantic segmentation device.
Background
The semantic segmentation provides semantic information of a driving environment for a control system of the automobile, helps a driving system to judge driving, and is important for applications such as automatic driving and geological survey. At present, the semantic algorithm for the city street view of automatic driving only acts on an ideal scene with sufficient illumination, the robustness of other scenes is not high, and particularly, when low-illumination scenes such as night and the like are processed, an accurate semantic segmentation result is difficult to obtain.
Disclosure of Invention
The purpose of the invention is as follows: in order to overcome the defects of the prior art, the invention provides a low-illumination semantic segmentation model training method, a semantic segmentation method and a semantic segmentation device, so as to improve the semantic segmentation accuracy under a low-illumination scene.
The technical scheme is as follows: in a first aspect, the present invention provides a training method for a low-light semantic segmentation model, where the method includes:
acquiring an image data set under low illumination; respectively carrying out enhancement processing on the acquired image data through an image enhancement model to obtain an enhanced image data set and an enhanced image loss function; respectively performing semantic segmentation on the acquired enhanced image data through a semantic segmentation model to acquire a semantic segmentation image data set and a semantic segmentation loss function; obtaining a total loss function according to the enhanced image loss function and the semantic segmentation loss function; and adjusting the semantic segmentation model according to a total loss function to obtain a low-illumination semantic segmentation model.
Optionally, the acquisition of the low-light image dataset is based on a cityscaps dataset.
Optionally, the function of the image enhancement model is represented as:
wherein the content of the first and second substances,
representing normal images output after image enhancement, ILAnd (4) representing a mapping function of the low-illumination image f (x) representing the brightness adjustment of the image, wherein a Sigmod activation function is adopted, and theta is a network parameter of the enhancement network.
Optionally, the enhanced image loss function is expressed as:
where N represents the total number of pixels in the input image,
and
and the output pixel result of the network and the label pixel at the corresponding position are respectively represented, theta is a network parameter of the enhancement network, and beta is set to be 0.1 in the training process.
Optionally, the semantic segmentation model is obtained based on a ghostNet neural network model training.
Optionally, the semantic segmentation loss function is expressed as:
wherein, yiRepresenting the probability of presoftmax belonging to class i, e being a constant.
Optionally, the total loss function is obtained by performing cascade training on an enhanced image loss function and a semantic segmentation loss function.
Optionally, the total loss function is represented as:
L＝βDLD+βSLS (4)
where L represents the total loss function of the entire network, LDAnd LSRepresenting the loss function, beta, of the enhancement network and the segmentation networkSAnd betaDTo representThe weight of the function is lost.
In a second aspect, a method for low-illumination semantic segmentation, the method comprising:
acquiring a source image, and carrying out image enhancement on the source image through an image enhancement model to obtain an enhanced source image; and performing semantic segmentation on the enhanced source image through a low-illumination semantic segmentation model to obtain segmentation precision, wherein the low-illumination semantic segmentation model is obtained through training by using the method of any one of the first aspect.
In a third aspect, a low-illumination semantic segmentation apparatus, the apparatus comprising:
the enhancement unit is used for carrying out image enhancement on the source image through the image enhancement model to obtain an enhanced source image; and the segmentation unit is used for performing semantic segmentation on the enhanced source image through a low-illumination semantic segmentation model to obtain segmentation precision, wherein the low-illumination semantic segmentation model is obtained through training by using the method of any one of the third aspect.
Has the advantages that: according to the method, the semantic segmentation model is adjusted according to the total loss function, the image loss is enhanced, and the segmentation precision is improved; a semantic segmentation model is established based on a ghostNet neural network model, the total sum of parameters required in a Ghost module and the calculation complexity are reduced, and the segmentation efficiency is improved; and the image enhancement model adopts a self-coding structure with jump links, so that the image brightness is improved, and more image details can be acquired.
Drawings
Fig. 1 is a schematic flow chart of a training method of a low-illumination semantic segmentation model according to an embodiment of the present invention.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present invention is not limited thereby.
Example 1:
as shown in FIG. 1, the invention relates to a low-illumination semantic segmentation model training method, which comprises the following steps:
the Cityscapes data set is selected as a main data set of an experiment, and the Gamma value of the image is adjusted firstly, namely the image is subjected to inverse Gamma transformation. Gamma transformation is an image operation for performing nonlinear adjustment on an image input gray value, and can make the gray value of an image before transformation and the gray value of the image after transformation have an exponential nonlinear relationship, as shown in formula (1):
wherein IinIoutEach represents an input/output image, β represents a Gamma value, and α represents a transform coefficient. After Gamma conversion, in order to obtain a better low-illumination effect of the image, a PS tool is also used for adjusting the brightness and the contrast of the city maps street view image after Gamma conversion, so that the finally manually adjusted low-illumination street view data is similar to the actual low-illumination dark scene as much as possible in vision to obtain the low-illumination street view image with good visual effect, thus on the basis of the original data set containing the street view image of normal weather and the corresponding semantic segmentation label, a new corresponding low-illumination image is generated, each group of data contains three city street view images with low illumination respectively, and the city street view image and the semantic segmentation label in normal day. The artwork and segmentation labels for these datasets are from the cityscaps dataset.
The obtained low-illumination image is subjected to image enhancement processing by using a self-coding structure with jump connection, the number of parameters can be effectively reduced through dimension reduction of a coder, representative characteristics of the image are obtained, and a decoder achieves the denoising effect, as shown in formula (2):
in the formula (2), the reaction mixture is,
representing normal images output after image enhancement, ILIndicating low lightThe image f (x) represents a mapping function of image brightness adjustment, wherein a Sigmod activation function is adopted, and theta is a network parameter of the enhancement network.
When a normal image is obtained, the loss of an image enhancement network structure can be generated, and Mean Absolute Error (MAE) and a quadratic norm (L) are adopted in the training of the enhancement network structure2Error) to calculate the total error of the enhancement network, as shown in equation (3):
n represents the total number of pixels in the input image,
and
label pixels respectively representing the output pixel result and the corresponding position of the network; l is2The regularization term can prevent the network from over-fitting, improve the convergence rate of the network, and beta is set to 0.1 during the training process.
The method is characterized in that the method selects the ghostNet as a semantic segmentation network, and the ghostNet is used as a lightweight network which is applied to generating more characteristic maps by a series of simple linear operations used by the convolutional layer. Compared with the common convolutional neural network, the total number of parameters and the calculation complexity required in the Ghost module are reduced under the condition of not changing the size of the output feature map, and all convolutions in the module are replaced by pointwise convolutions in order to improve the efficiency.
The loss function of semantic segmentation, adopting the cross entropy loss function of softmax, comprises two steps for each pixel in the image, and the mathematical expression of the two steps is shown in the formula (4) (5):
formula (4), yiRepresenting the probability of presoftmax belonging to class i, e being a constant. Firstly, softmax is calculated for the last layer output of the cascade segmentation network to obtain the probability that each pixel belongs to a certain class, then cross entropy loss L is calculated for the predicted probability vector, each pixel is taken as a sample to calculate the cross entropy loss, and then the average value is taken as the final segmentation network loss Ls。
The mathematical expression of the final total loss function of two networks which jointly train their losses through the cascade network is shown as the formula (6)
L＝βDLD+βSLS (6)
Where L represents the total loss function of the entire network, LDAnd LSRepresenting the loss function, beta, of the enhancement network and the segmentation networkSAnd betaDRepresenting the weight of the loss function.
And adjusting the semantic segmentation model established based on the ghostNet neural network model according to the total loss function to obtain the low-illumination semantic segmentation model.
Example 2:
a method of low-light semantic segmentation, the method comprising:
acquiring a source image, and carrying out image enhancement on the source image through an image enhancement model to obtain an enhanced source image;
performing semantic segmentation on the enhanced source image through a low-illumination semantic segmentation model to obtain segmentation precision,
wherein the low-light semantic segmentation model is obtained by training according to the method of any one of embodiment 1.
Example 3:
a low-light semantic segmentation apparatus, the apparatus comprising:
the enhancement unit is used for carrying out image enhancement on the source image through the image enhancement model to obtain an enhanced source image;
a segmentation unit for performing semantic segmentation on the enhanced source image through a low-illumination semantic segmentation model to obtain segmentation accuracy,
wherein the low-light semantic segmentation model is obtained by training according to the method of any one of embodiment 1.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A training method of a low-illumination semantic segmentation model is characterized by comprising the following steps:
acquiring an image data set under low illumination;
respectively carrying out enhancement processing on the acquired image data through an image enhancement model to obtain an enhanced image data set and an enhanced image loss function;
respectively performing semantic segmentation on the acquired enhanced image data through a semantic segmentation model to acquire a semantic segmentation image data set and a semantic segmentation loss function;
obtaining a total loss function according to the enhanced image loss function and the semantic segmentation loss function;
and adjusting the semantic segmentation model according to a total loss function to obtain a low-illumination semantic segmentation model.
2. The method for training the low-light semantic segmentation model according to claim 1, wherein the low-light image dataset is obtained based on a cityscaps dataset.
3. The method for training the low-light semantic segmentation model according to claim 1, wherein the function of the image enhancement model is expressed as:
wherein the content of the first and second substances,
representing normal images output after image enhancement, ILAnd (4) representing a mapping function of the low-illumination image f (x) representing the brightness adjustment of the image, wherein a Sigmod activation function is adopted, and theta is a network parameter of the enhancement network.
4. The method for training the low-light semantic segmentation model according to claim 3, wherein the enhanced image loss function is expressed as:
where N represents the total number of pixels in the input image,
and
and the output pixel result of the network and the label pixel at the corresponding position are respectively represented, theta is a network parameter of the enhancement network, and beta is set to be 0.1 in the training process.
5. The method for training the low-illumination semantic segmentation model according to claim 1, wherein the semantic segmentation model is obtained based on a ghostNet neural network model training.
6. The method for training the low-light semantic segmentation model according to claim 5, wherein the semantic segmentation loss function is expressed as:
wherein, yiRepresenting the probability of presoftmax belonging to class i, e being a constant.
7. The method for training the low-illumination semantic segmentation model according to claim 1, wherein the total loss function is obtained by training an enhanced image loss function and a semantic segmentation loss function in a cascade manner.
8. The method for training the low-light semantic segmentation model according to claim 7, wherein the total loss function is expressed as:
L＝βDLD+βSLS (4)
where L represents the total loss function of the entire network, LDAnd LSRepresenting the loss function, beta, of the enhancement network and the segmentation networkSAnd betaDRepresenting the weight of the loss function.
9. A low-light semantic segmentation method, comprising:
acquiring a source image, and carrying out image enhancement on the source image through an image enhancement model to obtain an enhanced source image;
performing semantic segmentation on the enhanced source image through a low-illumination semantic segmentation model to obtain segmentation precision,
wherein the low-light semantic segmentation model is trained by the method of any one of claims 1-8.
10. An apparatus for low-light semantic segmentation, the apparatus comprising:
the enhancement unit is used for carrying out image enhancement on the source image through the image enhancement model to obtain an enhanced source image;
a segmentation unit for performing semantic segmentation on the enhanced source image through a low-illumination semantic segmentation model to obtain segmentation accuracy,
wherein the low-light semantic segmentation model is trained by the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110940177.XA CN113723231A (en) | 2021-08-17 | 2021-08-17 | Low-illumination semantic segmentation model training method, semantic segmentation method and semantic segmentation device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110940177.XA CN113723231A (en) | 2021-08-17 | 2021-08-17 | Low-illumination semantic segmentation model training method, semantic segmentation method and semantic segmentation device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113723231A true CN113723231A (en) | 2021-11-30 |
Family
ID=78676694
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110940177.XA Pending CN113723231A (en) | 2021-08-17 | 2021-08-17 | Low-illumination semantic segmentation model training method, semantic segmentation method and semantic segmentation device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113723231A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109410129A (en) * | 2018-09-28 | 2019-03-01 | 大连理工大学 | A kind of method of low light image scene understanding |
| US20200218961A1 (en) * | 2017-09-27 | 2020-07-09 | Google Llc | End to End Network Model for High Resolution Image Segmentation |
| CN111507343A (en) * | 2019-01-30 | 2020-08-07 | 广州市百果园信息技术有限公司 | Training of semantic segmentation network and image processing method and device thereof |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
| US20210241107A1 (en) * | 2019-03-26 | 2021-08-05 | Tencent Technology (Shenzhen) Company Limited | Method, apparatus, device, and storage medium for training image semantic segmentation network |
-
2021
- 2021-08-17 CN CN202110940177.XA patent/CN113723231A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200218961A1 (en) * | 2017-09-27 | 2020-07-09 | Google Llc | End to End Network Model for High Resolution Image Segmentation |
| CN109410129A (en) * | 2018-09-28 | 2019-03-01 | 大连理工大学 | A kind of method of low light image scene understanding |
| CN111507343A (en) * | 2019-01-30 | 2020-08-07 | 广州市百果园信息技术有限公司 | Training of semantic segmentation network and image processing method and device thereof |
| US20210241107A1 (en) * | 2019-03-26 | 2021-08-05 | Tencent Technology (Shenzhen) Company Limited | Method, apparatus, device, and storage medium for training image semantic segmentation network |
| AU2020103901A4 (en) * | 2020-12-04 | 2021-02-11 | Chongqing Normal University | Image Semantic Segmentation Method Based on Deep Full Convolutional Network and Conditional Random Field |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111882002B (en) | MSF-AM-based low-illumination target detection method | |
| CN113052210B (en) | Rapid low-light target detection method based on convolutional neural network | |
| CN112184577B (en) | Single image defogging method based on multiscale self-attention generation countermeasure network | |
| CN110400275B (en) | Color correction method based on full convolution neural network and characteristic pyramid | |
| CN110363770B (en) | Training method and device for edge-guided infrared semantic segmentation model | |
| CN104217404A (en) | Video image sharpness processing method in fog and haze day and device thereof | |
| CN109903250B (en) | Underwater image sharpening processing method based on multi-scale gradient domain contrast stretching | |
| CN113066025B (en) | Image defogging method based on incremental learning and feature and attention transfer | |
| CN111709888B (en) | Aerial image defogging method based on improved generation countermeasure network | |
| CN107292830A (en) | Low-light (level) image enhaucament and evaluation method | |
| CN110807744B (en) | Image defogging method based on convolutional neural network | |
| CN113420794B (en) | Binaryzation Faster R-CNN citrus disease and pest identification method based on deep learning | |
| CN116402679A (en) | Lightweight infrared super-resolution self-adaptive reconstruction method | |
| CN115035171A (en) | Self-supervision monocular depth estimation method based on self-attention-guidance feature fusion | |
| CN114677560A (en) | Deep learning algorithm-based lane line detection method and computer system | |
| CN112686804A (en) | Image super-resolution reconstruction method and device for mine low-illumination environment | |
| CN113723231A (en) | Low-illumination semantic segmentation model training method, semantic segmentation method and semantic segmentation device | |
| CN116452450A (en) | Polarized image defogging method based on 3D convolution | |
| CN116542865A (en) | Multi-scale real-time defogging method and device based on structural re-parameterization | |
| CN115496764A (en) | Dense feature fusion-based foggy image semantic segmentation method | |
| CN115631108A (en) | RGBD-based image defogging method and related equipment | |
| CN114862724A (en) | Contrast type image defogging method based on exponential moving average knowledge distillation | |
| Wang et al. | Low-light traffic objects detection for automated vehicles | |
| CN113763261A (en) | Real-time detection method for far and small targets under sea fog meteorological condition | |
| CN111986109A (en) | Remote sensing image defogging method based on full convolution network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |