CN116402724A - RYB format RAW image color restoration method - Google Patents
RYB format RAW image color restoration method Download PDFInfo
- Publication number
- CN116402724A CN116402724A CN202310671470.XA CN202310671470A CN116402724A CN 116402724 A CN116402724 A CN 116402724A CN 202310671470 A CN202310671470 A CN 202310671470A CN 116402724 A CN116402724 A CN 116402724A
- Authority
- CN
- China
- Prior art keywords
- layer
- stage structure
- image
- images
- layers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000013528 artificial neural network Methods 0.000 claims abstract description 26
- 230000008569 process Effects 0.000 claims abstract description 7
- 238000011084 recovery Methods 0.000 claims abstract description 7
- 101100248200 Arabidopsis thaliana RGGB gene Proteins 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 210000000988 bone and bone Anatomy 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 4
- 239000003086 colorant Substances 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a RAW image color restoration method of RYB format, which comprises a neural network with a two-stage structure, wherein the first-stage structure comprises ten layers of structures which are sequentially connected, and the second-stage structure is a recovery layer for restoring the output of the first-stage structure. According to the invention, the RAW graph in RYB format is processed through the neural network, the neural network is a pixel-level processed neural network, and each pixel value of the image can be modified and transformed in the network application process; by the network training method provided by the invention, the trained neural network can successfully recover the RYB format RAW image into a full-color RGB image.
Description
Technical Field
The invention relates to an image processing method, in particular to an image color restoration method.
Background
Currently, most of the mainstream CMOS image sensors output RAW data in Bayer format. The implementation principle is that a layer of optical filter is covered on the surface of the image sensor, so that a certain pixel point only receives light of one color, and the pixel value obtained by the image sensor is the pixel value of the single color. The bayer filter array is arranged using RGB three colors, typically in a set of 2x2 matrices, the number of colors being R: g: b=1: 2:1, each element represents a pixel, so that each pixel point has only one color, and common arrangement forms are RGGB, BGGR, GRBG and the like. If the green filter in the 2x2 matrix is replaced by a yellow filter and is in RYYB arrangement, the spectrum response of Y (yellow) in the sensor is wider, the spectrum coverage is wider, more photons can be sensed, the total light inlet quantity can be improved by about 30-40%, the signal-to-noise ratio of Luminance is obviously improved in a dark light scene, and the night shooting effect is better.
However, the conventional RGB three-color-based algorithm such as the Demosaic algorithm, the AWB algorithm, etc. cannot accurately restore the image colors to the RAW image arranged in RYYB.
Disclosure of Invention
The invention aims to: in view of the above prior art, a RYB format RAW image color reproduction method is proposed to perform color reproduction on a RYB format RAW image captured by an image sensor, and output an RGB format image.
The technical scheme is as follows: a RAW image color reproduction method of RYB format, comprising: inputting RAW images arranged in RYYB into a neural network for color reduction processing, and outputting RGB images;
the neural network comprises a two-stage structure, and the first-stage structure comprises a ten-layer structure which is sequentially connected: the first layer is a preprocessing layer, R, Y, Y, B pixels in the image are respectively extracted into independent channels through convolution, the input size of the first layer is 448 x 3, and the output size of the first layer is 448 x 4; the second to fifth layers are all convolution layers, each layer is formed by linking Conv layers, BN layers and ReLU layers, and the output sizes of the second to fifth layers are respectively as follows: 112 x 24, 56 x 96, 28 x 192, 14 x 384; the sixth to tenth layers are deconvolution layers, each layer is formed by linking an Upsample layer, a BN layer and a ReLU layer, and the output sizes of the sixth to tenth layers are respectively as follows: 14 x 384, 28 x 192, 56 x 96, 112 x 24, 448 x 4; wherein, the output of the seventh layer, the eighth layer and the ninth layer respectively correspond to the output of the fourth layer, the third layer and the second layer for residual operation;
the second stage structure of the neural network is a recovery layer for recovering the output of the first stage structure, the recovery layer is composed of an Upsample layer and 3 Conv layer links with the size of 1*1, and the final output size is 448×448×3.
Further, when training the neural network, marking the first-stage structure as a backbond, and constructing a structure Tophalf parallel to the first-stage structure, wherein the Tophalf is formed by the same structure as the first to fifth layers in the first-stage structure;
training the first stage structure first includes:
step 1: the method comprises the steps that an image sensor which outputs RAW images arranged in RGGB and RAW images arranged in RYYB are adopted to respectively shoot a plurality of scenes in the same mode, and the obtained images are respectively recorded into an image set P1 and an image set P2, so that a training set is formed;
step 2: inputting the images in the training set into the first-stage structure for training;
in the training process, inputting the images in the image set P1 into the Tophalf, inputting the corresponding images in the image set P2 into the backbox, and then respectively calculating the difference Dis of the output images of the sixth layer, the seventh layer, the eighth layer and the ninth layer in the backbox relative to the second layer, the third layer, the fourth layer and the fifth layer in the Tophalf n ：
In the Backbone i,j Outputting pixel values at the ith row and the jth column of the image for the corresponding layer in the backlight, tophalf i,j Outputting pixel values at the ith row and the jth column of the image for the corresponding layer in the Tophalf; wherein n is 6,7,8 and 9, which respectively correspond to a sixth layer, a seventh layer, an eighth layer and a ninth layer in the back bone;
will make each degree of difference Dis n The values are added according to the weights to obtain a Loss value Loss Dis ：
Loss Dis =0.4Dis 6 +0.6Dis 7 +0.8Dis 8 +1.0Dis 9
Performing back propagation according to the loss value to optimize network parameters of the first-stage structure;
step 3: training the second stage structure after completing the training of the first stage structure, comprising: inputting the images in the image set P2 into the trained first stage structure, inputting the images with the final output size of 448 x 4 of the first stage structure into the second stage structure, and then obtaining the loss value LossR between the images with the final output size of 448 x 3 of the second stage structure and the corresponding images in the image set P1:
wherein P1 i,j For the pixel value at the ith row and j column of the corresponding image in the image set P1, a discover i,j Final outputting pixel values at row i and column j of the image for the second stage structure;
and carrying out back propagation according to the loss value LossR to optimize network parameters of the second-stage structure.
The beneficial effects are that: the existing Demosaic algorithm is only applicable to RGB format images, and no RYB format RAW image processing algorithm capable of effectively improving the picture effect exists. According to the invention, the RAW graph in RYB format is processed through the neural network, the neural network is a pixel-level processed neural network, and each pixel value of the image can be modified and transformed in the network application process; by the network training method provided by the invention, the trained neural network can successfully recover the RYB format RAW image into a full-color RGB image.
Drawings
FIG. 1 is a schematic diagram of a first stage structure of a neural network according to the present invention;
FIG. 2 is a schematic diagram of the Tophalf network constructed during the neural network training process of the present invention;
FIG. 3 is a schematic diagram showing the difference values obtained during the training of the first stage structure of the neural network according to the present invention;
fig. 4 is a schematic diagram of a network structure of a second stage structure of the neural network according to the present invention.
Detailed Description
The invention is further explained below with reference to the drawings.
A RAW image color reproduction method of RYB format, comprising: and inputting the RAW images arranged in RYYB into a neural network for color reduction processing, and outputting RGB images.
The neural network includes a two-stage structure. As shown in fig. 1, the first-stage structure includes ten layers of structures connected in sequence: the first layer is a preprocessing layer, R, Y, Y, B pixels in the image are respectively extracted into independent channels through convolution, the input size of the first layer is 448 x 3, and the output size of the first layer is 448 x 4; the second to fifth layers are all convolution layers, each layer is composed of a convolution (Conv) layer, a BN layer and an activation function (ReLU) layer in a linked mode, and the output sizes of the second to fifth layers are respectively as follows: 112 x 24, 56 x 96, 28 x 192, 14 x 384; the sixth to tenth layers are deconvolution layers, each layer is composed of an up sampling (Upsample) layer, a BN layer and a ReLU layer which are linked, and the output sizes of the sixth to tenth layers are respectively as follows: 14 x 384, 28 x 192, 56 x 96, 112 x 24, 448 x 4. Wherein, the outputs of the seventh, eighth and ninth layers respectively correspond to the outputs of the fourth, third and second layers, and residual operations (Res) are performed.
As shown in fig. 4, the second stage structure of the neural network is a recovery layer for recovering the output of the first stage structure, and the recovery layer is composed of an Upsample layer and 3 Conv layer links with a size of 1*1, and the final output size is 448×448×3.
When training the above neural network, the first-stage structure is denoted as a backbond, and a structure Tophalf parallel to the first-stage structure is constructed, which is composed of the same structures as the first to fifth layers in the first-stage structure, as shown in fig. 2.
First training a first stage structure, comprising:
step 1: the image sensors which output RAW images arranged in RGGB and RAW images arranged in RYYYB are adopted to respectively shoot a plurality of scenes in the same way, namely, different image sensors are adopted to respectively shoot the same scene, other shooting parameters and conditions are the same, and the obtained images are respectively recorded into an image set P1 and an image set P2, so that a training set is formed. In the training process of this embodiment, the training set images are not less than 3000.
Step 2: the images in the training set are input into the first stage structure for training.
In the training process, the images in the image set P1 are input into Tophalf, the corresponding images in the image set P2 are input into the backbox, and then, as shown in FIG. 3, the difference degree Dis of the output images of the sixth layer, the seventh layer, the eighth layer and the ninth layer in the backbox relative to the second layer, the third layer, the fourth layer and the fifth layer in Tophalf is calculated respectively n ：
In the Backbone i,j Output the pixel value at the ith row and j column of the image for the corresponding layer in the backlight, tophalf i,j Outputting pixel values at the ith row and the jth column of the image for the corresponding layer in Tophalf; and n is 6,7,8 and 9, and corresponds to a sixth layer, a seventh layer, an eighth layer and a ninth layer in the back bond respectively.
Will make each degree of difference Dis n The values are added according to the weights to obtain a Loss value Loss Dis ：
Loss Dis =0.4Dis 6 +0.6Dis 7 +0.8Dis 8 +1.0Dis 9
The network parameters of the first phase structure are optimized by back propagation based on the loss values.
Step 3: training the second stage structure after training the first stage structure is completed.
Inputting the images in the image set P2 into a trained first stage structure, inputting the images with the final output size of 448 x 4 of the first stage structure into a second stage structure, and then obtaining loss values LossR between the images with the final output size of 448 x 3 of the second stage structure and the corresponding images in the image set P1:
wherein P1 i,j For pixel values at the ith row, column, j, of the corresponding image in image set P1, a discover i,j The pixel values at row j column of the i-th row of the image are finally output for the second stage structure.
The network parameters of the second phase structure are optimized by back propagation according to the loss value LossR.
Inputting the RAW graph of RYYYB into a trained neural network, and when the network is normally used, the Tophalf structure is not started, and the network finally outputs the RGB image with normal color.
The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (2)
1. A RAW image color reproduction method of RYB format, comprising: inputting RAW images arranged in RYYB into a neural network for color reduction processing, and outputting RGB images;
the neural network comprises a two-stage structure, and the first-stage structure comprises a ten-layer structure which is sequentially connected: the first layer is a preprocessing layer, R, Y, Y, B pixels in the image are respectively extracted into independent channels through convolution, the input size of the first layer is 448 x 3, and the output size of the first layer is 448 x 4; the second to fifth layers are all convolution layers, each layer is formed by linking Conv layers, BN layers and ReLU layers, and the output sizes of the second to fifth layers are respectively as follows: 112 x 24, 56 x 96, 28 x 192, 14 x 384; the sixth to tenth layers are deconvolution layers, each layer is formed by linking an Upsample layer, a BN layer and a ReLU layer, and the output sizes of the sixth to tenth layers are respectively as follows: 14 x 384, 28 x 192, 56 x 96, 112 x 24, 448 x 4; wherein, the output of the seventh layer, the eighth layer and the ninth layer respectively correspond to the output of the fourth layer, the third layer and the second layer for residual operation;
the second stage structure of the neural network is a recovery layer for recovering the output of the first stage structure, the recovery layer is composed of an Upsample layer and 3 Conv layer links with the size of 1*1, and the final output size is 448×448×3.
2. The method for color reproduction of a RAW image in RYB format according to claim 1, wherein the neural network is trained by marking the first-stage structure as a Backbone and constructing a structure Tophalf parallel to the first-stage structure, the Tophalf being composed of the same structure as the first to fifth layers in the first-stage structure;
training the first stage structure first includes:
step 1: the method comprises the steps that an image sensor which outputs RAW images arranged in RGGB and RAW images arranged in RYYB are adopted to respectively shoot a plurality of scenes in the same mode, and the obtained images are respectively recorded into an image set P1 and an image set P2, so that a training set is formed;
step 2: inputting the images in the training set into the first-stage structure for training;
in the training process, inputting the images in the image set P1 into the Tophalf, inputting the corresponding images in the image set P2 into the backbox, and then respectively calculating the sixth layer, the seventh layer, the eighth layer and the ninth layer in the backbox relative to the second layer in the TophalfThe difference degree Dis of the output images of the third layer, the fourth layer and the fifth layer n ：
In the Backbone i,j Outputting pixel values at the ith row and the jth column of the image for the corresponding layer in the backlight, tophalf i,j Outputting pixel values at the ith row and the jth column of the image for the corresponding layer in the Tophalf; wherein n is 6,7,8 and 9, which respectively correspond to a sixth layer, a seventh layer, an eighth layer and a ninth layer in the back bone;
will make each degree of difference Dis n The values are added according to the weights to obtain a Loss value Loss Dis ：
Loss Dis =0.4Dis 6 +0.6Dis 7 +0.8Dis 8 +1.0Dis 9
Performing back propagation according to the loss value to optimize network parameters of the first-stage structure;
step 3: training the second stage structure after completing the training of the first stage structure, comprising: inputting the images in the image set P2 into the trained first stage structure, inputting the images with the final output size of 448 x 4 of the first stage structure into the second stage structure, and then obtaining the loss value LossR between the images with the final output size of 448 x 3 of the second stage structure and the corresponding images in the image set P1:
wherein P1 i,j For the pixel value at the ith row and j column of the corresponding image in the image set P1, a discover i,j Final outputting pixel values at row i and column j of the image for the second stage structure;
and carrying out back propagation according to the loss value LossR to optimize network parameters of the second-stage structure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310671470.XA CN116402724B (en) | 2023-06-08 | 2023-06-08 | RYB format RAW image color restoration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310671470.XA CN116402724B (en) | 2023-06-08 | 2023-06-08 | RYB format RAW image color restoration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116402724A true CN116402724A (en) | 2023-07-07 |
| CN116402724B CN116402724B (en) | 2023-08-11 |
Family
ID=87010896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310671470.XA Active CN116402724B (en) | 2023-06-08 | 2023-06-08 | RYB format RAW image color restoration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116402724B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111127336A (en) * | 2019-11-18 | 2020-05-08 | 复旦大学 | Image signal processing method based on self-adaptive selection module |
| WO2020114087A1 (en) * | 2018-12-04 | 2020-06-11 | 北京达佳互联信息技术有限公司 | Method and device for image conversion, electronic equipment, and storage medium |
| CN112529775A (en) * | 2019-09-18 | 2021-03-19 | 华为技术有限公司 | Image processing method and device |
| CN114022732A (en) * | 2021-11-03 | 2022-02-08 | 北京理工大学 | Extremely dark light object detection method based on RAW image |
| CN115797228A (en) * | 2023-01-30 | 2023-03-14 | 深圳市九天睿芯科技有限公司 | Image processing device, method, chip, electronic equipment and storage medium |
-
2023
- 2023-06-08 CN CN202310671470.XA patent/CN116402724B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020114087A1 (en) * | 2018-12-04 | 2020-06-11 | 北京达佳互联信息技术有限公司 | Method and device for image conversion, electronic equipment, and storage medium |
| CN112529775A (en) * | 2019-09-18 | 2021-03-19 | 华为技术有限公司 | Image processing method and device |
| CN111127336A (en) * | 2019-11-18 | 2020-05-08 | 复旦大学 | Image signal processing method based on self-adaptive selection module |
| CN114022732A (en) * | 2021-11-03 | 2022-02-08 | 北京理工大学 | Extremely dark light object detection method based on RAW image |
| CN115797228A (en) * | 2023-01-30 | 2023-03-14 | 深圳市九天睿芯科技有限公司 | Image processing device, method, chip, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116402724B (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111127336B (en) | Image signal processing method based on self-adaptive selection module | |
| US7907791B2 (en) | Processing of mosaic images | |
| CN102122388B (en) | For obtaining equipment and the method for high dynamic range images | |
| US6781626B1 (en) | System and method of color interpolation | |
| US7283663B2 (en) | Interpolation of edge portions of a digital image | |
| US6757012B1 (en) | Color selection for sparse color image reconstruction | |
| US20070159542A1 (en) | Color filter array with neutral elements and color image formation | |
| CN101272503B (en) | Gridding noise elimination method and device for remaining image definition | |
| US8248496B2 (en) | Image processing apparatus, image processing method, and image sensor | |
| CN105847772A (en) | Imaging system with clear filter pixels | |
| CN111402145B (en) | Self-supervision low-illumination image enhancement method based on deep learning | |
| CN111598789B (en) | Sparse color sensor image reconstruction method based on deep learning | |
| CN109785252A (en) | Based on multiple dimensioned residual error dense network nighttime image enhancing method | |
| CN116128735B (en) | Multispectral image demosaicing structure and method based on densely connected residual error network | |
| US20130293750A1 (en) | Image sensing apparatus, method of controlling operation of same and image sensing system | |
| CN100369459C (en) | Image signal processing apparatus | |
| WO2011076974A1 (en) | Pixel information reproduction using neural networks | |
| CN111932459A (en) | Video image processing method and device, electronic equipment and storage medium | |
| JP4190886B2 (en) | Prevention of green non-uniformity in image sensors | |
| CN116402724B (en) | RYB format RAW image color restoration method | |
| CN117274060B (en) | Unsupervised end-to-end demosaicing method and system | |
| US8692910B2 (en) | Image processing device, image signal correction method, correction matrix calculation method, and imaging device | |
| US20240029460A1 (en) | Apparatus and method for performing image authentication | |
| CN103621070A (en) | Imaging device and imaging program | |
| CN110544210B (en) | Bayer CFA image recovery method based on fuzzy boundary interpolation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |