CN113808043A - Camera imaging method, device, medium and equipment - Google Patents
Camera imaging method, device, medium and equipment Download PDFInfo
- Publication number
- CN113808043A CN113808043A CN202111085826.9A CN202111085826A CN113808043A CN 113808043 A CN113808043 A CN 113808043A CN 202111085826 A CN202111085826 A CN 202111085826A CN 113808043 A CN113808043 A CN 113808043A
- Authority
- CN
- China
- Prior art keywords
- image data
- raw
- neural network
- domain
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 29
- 238000013528 artificial neural network Methods 0.000 claims abstract description 97
- 238000012549 training Methods 0.000 claims abstract description 51
- 238000000034 method Methods 0.000 claims abstract description 16
- 238000012545 processing Methods 0.000 claims description 31
- 238000010606 normalization Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 14
- 238000012937 correction Methods 0.000 claims description 14
- 108091006146 Channels Proteins 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 12
- 230000015654 memory Effects 0.000 claims description 10
- 238000005070 sampling Methods 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 230000000694 effects Effects 0.000 abstract description 13
- 239000010410 layer Substances 0.000 description 89
- 238000010586 diagram Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 12
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000011176 pooling Methods 0.000 description 4
- 230000011514 reflex Effects 0.000 description 4
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000003705 background correction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 206010034960 Photophobia Diseases 0.000 description 1
- 238000012356 Product development Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 208000013469 light sensitivity Diseases 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
Provided herein are a camera imaging method, apparatus, medium, and device, the method including: acquiring image data of a plurality of Raw domains using a target device, and acquiring images of a plurality of sRGB domains corresponding to the image data of the plurality of Raw domains using a reference device; normalizing the image data pairs of the plurality of Raw fields; training a neural network by using the normalized image data of the plurality of Raw domains and the corresponding images of the plurality of sRGB domains; and applying the trained neural network to the target device. When the target equipment shoots, the neural network can generate an image of an sRGB domain according to image data of an Raw domain, the image effect of the sRGB domain is close to that of professional shooting equipment, and the shooting quality is improved.
Description
Technical Field
This document relates to the field of image restoration, and more particularly, to a camera imaging method, apparatus, medium, and device.
Background
With the popularization of smart phones, more and more people begin to use mobile phones to create photographic works. In the related art, the sensor of the mobile phone acquires data by using a bayer array, and each pixel point of the image sensor only obtains a pixel value of a certain channel in R, G, B channels based on the bayer array, so as to obtain RAW domain data. To output a complete sRGB domain image, three-channel pixel values of each pixel point need to be complemented, and a series of processing is performed on RAW domain data through an ISP flow (including sub-modules such as black level correction, lens shading correction, dead pixel correction, interpolation, white balance, color correction, gamma correction, tone mapping, and the like, and in addition, modules such as automatic exposure, noise reduction, and sharpening) built in a mobile phone. The quality of images finally obtained by the smart phone is far from the quality of professional camera equipment, and the quality of mobile phone photography needs to be improved. Moreover, it is a tedious task to adjust a plurality of modules independently, which consumes a lot of manpower and time, and causes problems of information loss accumulation, detail loss, high noise, and the like.
Disclosure of Invention
To overcome the problems in the related art, provided herein are a camera imaging method, apparatus, medium, and device.
According to a first aspect herein, there is provided a camera imaging method comprising:
acquiring image data of a plurality of Raw domains using a target device, and acquiring images of a plurality of sRGB domains corresponding to the image data of the plurality of Raw domains using a reference device;
normalizing the image data pairs of the plurality of Raw fields;
training a neural network by using the normalized image data of the plurality of Raw domains and the corresponding images of the plurality of sRGB domains;
and applying the trained neural network to the target equipment so that the target equipment generates an image of an sRGB domain according to the shot image data of the Raw domain.
Based on the scheme, the normalization processing comprises the following steps:
and carrying out black level correction processing on the image data of the Raw domain, or dividing the value of each pixel point in the image data of the Raw domain by a preset value.
Based on the foregoing scheme, training a neural network using preprocessed image data of multiple Raw domains and images of multiple sRGB domains includes:
and taking the preprocessed image data of the Raw domain as input data of a neural network, taking an image of an sRGB domain corresponding to the image data of the Raw domain as an output image of the neural network, and training the neural network to obtain a mapping relation between the image data of the Raw domain and the image of the sRGB domain.
Based on the foregoing solution, the neural network is an N-layer structure, each of the second layer to the nth layer includes a plurality of convolution stacking modules, and taking the image data of the preprocessed Raw domain as the input data of the neural network includes:
converting the image data of the Raw domain into 4-channel data according to the channel sequence of B, Gb, R and Gr, and inputting the 4-channel data into a second layer of the neural network; taking the data processed and pooled by a convolution stacking module in the second layer as the input data of the third layer, and so on until reaching the Nth layer;
the method for using the image of the sRGB domain corresponding to the image data of the Raw domain as the output image of the neural network comprises the following steps:
and taking the image in the sRGB domain as the output of the first layer of the neural network, and taking the image obtained by carrying out N-1 times of downsampling on the image in the sRGB domain as the output image of the Nth layer, wherein N is an integer larger than or equal to 3.
Based on the scheme, the second layer to the (N-1) th layer of the neural network further comprises a merging module, and the merging module is used for merging the input data of each layer with the up-sampling result of the output data of the next layer.
Based on the scheme, the training neural network comprises the following steps:
and training the neural network layer by layer according to the sequence from the Nth layer to the first layer.
According to another aspect herein, there is provided a camera imaging apparatus comprising:
an image acquisition module configured to acquire image data of a plurality of Raw domains using a target device, and acquire images of a plurality of sRGB domains corresponding to the image data of the plurality of Raw domains using a reference device;
the normalization processing module is used for performing normalization processing on the image data pairs of the plurality of Raw domains;
the training module is used for training a neural network by using the preprocessed image data of a plurality of Raw domains and the images of a plurality of sRGB domains;
and the application module is used for applying the trained neural network to the target equipment so that the target equipment generates an image of an sRGB domain according to the shot image data of the Raw domain.
Based on the foregoing scheme, the training module training the neural network using the preprocessed image data of the plurality of Raw domains and the images of the plurality of sRGB domains includes:
and taking the preprocessed image data of the Raw domain as input data of a neural network, taking an image of an sRGB domain corresponding to the image data of the Raw domain as an output image of the neural network, and training the neural network to obtain a mapping relation between the image data of the Raw domain and the image of the sRGB domain.
According to another aspect herein, there is provided a computer readable storage medium having stored thereon a computer program which, when executed, implements the steps of a camera imaging method.
According to another aspect herein, there is provided a computer device comprising a processor, a memory and a computer program stored on the memory, the processor implementing the steps of the camera imaging method when executing the computer program.
The target equipment is used for acquiring image data of a plurality of Raw domains, the reference equipment is used for acquiring images of an sRGB domain corresponding to the image data of the Raw domains, the image data of the Raw domains and the images of the sRGB domain which are in one-to-one correspondence are used for training the neural network, and the trained neural network is applied to the target equipment. And an ISP flow is not required to be built in the target equipment, so that independent adjustment of each module in the ISP flow is avoided, time is saved, and the product development progress is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification. In the drawings:
fig. 1 is a flowchart illustrating a camera imaging method according to an exemplary embodiment.
FIG. 2 is a schematic diagram of a neural network architecture, shown in accordance with an exemplary embodiment.
Fig. 3 is a block diagram of a camera imaging device shown in accordance with an example embodiment.
Fig. 4 is a block diagram illustrating a camera imaging device according to an exemplary embodiment.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention, and it is obvious that the described embodiments are some but not all of the embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments herein without making any creative effort, shall fall within the scope of protection. It should be noted that the embodiments and features of the embodiments may be arbitrarily combined with each other without conflict.
With the popularization of smart phones, more and more people begin to use mobile phones to create photographic works. In the related art, the sensor of the mobile phone acquires data by using a bayer array, and each pixel point of the image sensor only obtains a pixel value of a certain channel in R, G, B channels based on the bayer array, so as to obtain RAW domain data. To output a complete sRGB domain image, three-channel pixel values of each pixel point need to be complemented, and a series of processing is performed on RAW domain data through an ISP flow (including sub-modules such as black level correction, lens shading correction, dead pixel correction, interpolation, white balance, color correction, gamma correction, tone mapping, and the like, and in addition, modules such as automatic exposure, noise reduction, and sharpening) built in a mobile phone. The quality of images finally obtained by the smart phone is far from the quality of professional camera equipment, and the quality of mobile phone photography needs to be improved. Moreover, it is a tedious task to adjust a plurality of modules independently, which consumes a lot of manpower and time, and causes problems of information loss accumulation, detail loss, high noise, and the like.
The camera imaging method provided by the invention can be applied to electronic equipment such as a smart phone and a tablet personal computer, and can also be applied to other electronic equipment with a camera function, such as a camera and smart home equipment.
Fig. 1 is a flowchart illustrating a camera imaging method according to an exemplary embodiment. Referring to fig. 1, the camera imaging method includes at least steps S11 to S14, described in detail as follows:
step S11 is to acquire image data of a plurality of Raw domains using the target device, and acquire images of a plurality of sRGB domains corresponding to the image data of the plurality of Raw domains using the reference device.
In the present application, the target device means a device on which a camera to be adjusted is set, and the reference device means a camera device having a desired photographing effect. For example, the target device may be a smart phone in development, and various debugging needs to be performed on the camera function in the smart phone, so that the debugged target device can be normally marketed for sale; the reference device may be a professional camera device, such as a single lens reflex camera. Of course, the target device may be another electronic device having a camera function, and the reference device may be another camera device having an ideal photographing effect.
And shooting image data of a plurality of Raw fields by using the target equipment, wherein the Raw image data is Raw data of a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) image sensor for converting captured light source signals into digital signals. Images of a plurality of sRGB domains corresponding to image data of a plurality of Raw domains are acquired using a reference device, and the sRGB (standard Red Green blue) general color standard is a color standard commonly used in digital cameras, digital video cameras, scanners, displays, and the like.
The acquired image data of the plurality of Raw domains and the images of the plurality of sRGB domains correspond one to one, so that the neural network can be trained conveniently. It should be understood by those skilled in the art that the pixel sizes of the acquired image data of the Raw domain and the image of the sRGB domain in one-to-one correspondence are the same, and the image contents are the same. For example, a smartphone is used as a target device, a professional single-lens reflex camera is used as a reference device, an image pair of the same scene is shot simultaneously, image data of an Raw domain and an image of an sRGB domain, which correspond to each other one by one and have the same pixel size and the same image content, are cut out from the obtained image pair, and by shooting different scenes for multiple times and performing cutting processing, image data of multiple Raw domains and images of multiple sRGB domains can be obtained as a training set for training a neural network. A plurality of image data of the Raw domain different from the training set and a plurality of images of the sRGB domain can be obtained as the test set.
In step S12, normalization processing is performed on the pairs of image data in the plurality of Raw domains. Because each pixel point in the image sensor can only sense one color, the pixel points are converted into different numbers according to different light sensitivity intensities, namely pixel values in image data of a Raw domain. The number is different in different image sensors, for example, a pixel point in an 8-bit image sensor is divided into 256 levels from darkest to brightest, and the range of the pixel value in the corresponding image data is 0-255; a pixel point in the 10-bit image sensor is divided into 1024 levels from the darkest to the brightest, and the range of the pixel value in the corresponding image data is 0-1023. Therefore, it is necessary to perform normalization processing on the image data of the Raw domain, and convert the range of pixel values of the processed image data of the Raw domain into 0-1, so as to facilitate feature extraction by the neural network.
In an exemplary embodiment, the normalization process includes:
and carrying out black level correction processing on the image data of the Raw domain, or dividing the value of each pixel point in the image data of the Raw domain by a preset value.
For example, if the image data of the Raw domain acquired by the target device is Raw image data, the normalization process may be implemented by black level correction, which may be expressed as:
y＝(x-bl)/(wl-bl)
where x and y represent data before and after processing, respectively, bl represents black level, and wl represents white level.
If the image data of the Raw domain acquired by the target device has been subjected to preliminary processing, only the pixel value of each pixel is retained, where black level data and white level data are missing, and normalization processing cannot be realized through the above black level correction, at this time, normalization processing may be realized by dividing the value of each pixel in the image data of the Raw domain by a preset value, for example, for an 8-bit image sensor, the preset value may be 255, and for a 10-bit image sensor, the preset value may be 4 × 255. Through normalization processing, the pixel value of each pixel point in the image data of the Raw domain is changed to be between 0 and 1, so that the input of the neural network is facilitated, and the neural network is trained.
And step S13, training a neural network by using the normalized image data of the plurality of Raw domains and the corresponding images of the plurality of sRGB domains.
After a sufficient amount of image data of the Raw domain and images of the corresponding sRGB domain are obtained, normalization processing is completed on the image data of the Raw domain, and then the neural network can be trained by using the image data of the Raw domain and the images of the corresponding sRGB domain.
In an exemplary embodiment, training a neural network using preprocessed image data of a plurality of Raw domains and images of a plurality of sRGB domains comprises:
and taking the preprocessed image data of the Raw domain as input data of a neural network, taking an image of an sRGB domain corresponding to the image data of the Raw domain as an output image of the neural network, and training the neural network to obtain a mapping relation between the image data of the Raw domain and the image of the sRGB domain.
And respectively taking the image data of the Raw domain and the image of the sRGB domain which are in one-to-one correspondence as the input and the output of the neural network, and training the neural network, so that the neural network can learn the mapping relation between the image data of the Raw domain and the image of the sRGB domain.
And step S14, applying the trained neural network to the target device so that the target device generates an image of the sRGB domain according to the captured image data of the Raw domain.
After the trained neural network is applied to the target equipment, the neural network can obtain an sRGB domain image according to the image data and the mapping relation of the Raw domain shot by the target equipment, so that developers are not required to establish ISP flows, and a large amount of labor and time consumed by independently adjusting modules in a plurality of ISP flows are avoided. In addition, the training neural network uses the sRGB domain image of the reference device as an output, so that after the image data of the Raw domain captured by the target device is processed by the neural network, the finally obtained image effect approaches the effect of the sRGB domain image of the reference device. For example, after the trained neural network is applied to the smart phone by using the smart phone as the target device and the professional single lens reflex as the reference device, the image shot by the smart phone has the image effect shot by the professional single lens reflex, so that the imaging quality of the target device is improved.
In an exemplary embodiment, the neural network is an N-layer structure, each layer includes a plurality of convolution stacking modules (ConvMultiBlock), and taking the preprocessed Raw domain image data as the input data of the neural network includes:
converting image data of the Raw domain into 4-channel data according to the channel sequence of B, Gb, R and Gr, and inputting the 4-channel data into a second layer of the neural network; and taking the data processed and pooled (Maxpooling) in the second layer as input data of a third layer, and so on until the Nth layer, wherein N is an integer greater than or equal to 3.
The step of using the image of the sRGB domain corresponding to the image data of the Raw domain as the output image of the neural network includes:
and taking the image of the sRGB domain as the output of the first layer of the neural network, and taking the image of the sRGB domain after N-1 times of downsampling as the output image of the Nth layer.
In this embodiment, the neural network has a multilayer structure, where the number of layers is N, and N is an integer greater than or equal to 3.
FIG. 2 is a schematic diagram of a neural network architecture, shown in accordance with an exemplary embodiment. Referring to fig. 2, in a neural network, each layer includes a plurality of convolutional stacked modules (Conv multiblocks), each of which includes 2 convolutional layers (Conv), each of which is followed by an example normalization layer in (instance normalization) and a nonlinear activation layer lreul (leakey relu).
Taking the training data as the image data with the length and the width of 448 pixels as an example, before the data after normalization processing is input into the neural network, because the image data of the Raw domain is stored in the form of a Bayer array, the pixel values of the four channels of R, Gr, B and Gb are arranged into a single-layer vector according to a certain rule. Before entering the convolutional neural network, the data also needs to be divided into pixel values representing four channels separately and arranged in the channel order of B, Gb, R, and Gr. The Raw domain data is an array of 1 × 448 × 448 size, and should be converted into an array of 4 × 224 × 224 size to be input to the second layer of the neural network, and an image of the sRGB domain of 1 × 448 × 448 size is output as the first layer of the neural network. And meanwhile, taking the data processed and pooled by the convolution stacking module in the second layer as the input data of the third layer, and so on until the Nth layer. And taking the image obtained by carrying out N-1 times of downsampling on the image in the sRGB domain as an output image of the Nth layer.
By adopting the neural network with a multilayer structure, the learning result of the neural network can be supervised layer by layer, and the training effect is improved.
In an exemplary embodiment, the second to N-1 layers of the neural network further include a merging layer (Concat) for merging the input data of each layer with the up-sampled result of the output data of the next layer.
As shown in fig. 2, there are 5 layers, wherein the second to 4 th layers further include a merging module (Concat) for merging the input data of each layer with the up-sampling result of the output data of the next layer.
When the neural network with the multilayer structure is trained, the connection between upper and lower layer data is kept, and the training effect is improved.
In an exemplary embodiment, training the neural network comprises:
and training the neural network layer by layer according to the sequence from the Nth layer to the first layer.
The training of the neural network is started from the lowest layer, after the training of the next layer is finished, the training of the previous layer is added, the training is finished by accumulating layer by layer, and the training of the whole neural network is finished, so that the training effect is improved.
The neural network herein will be described in detail with reference to fig. 2 as an example. As shown in fig. 2, a 5-layer neural network is built, image data of a Raw domain of 448 pixels in length and width is converted into 4-channel data, the data size of each channel is 224 × 224, and the data of 4 channels is Input to the layer 2 (level1) of the neural network as Input data (Input). The Input data are transmitted into the second layer after being processed by the first convolution stacking module and pooling of the second layer (level1), and by analogy, the Input data (Input) enter the lowest layer (level4), namely the 5 th layer, after passing through 3 convolution stacking modules (ConvMultiBlock) and 3 times pooling (MaxPooling). In the 5 th layer, a combination module (Conv + signal) including 4 convolution stack modules (ConvMultiBlock), 3 residual modules (sensor Summation), 1 × 1 convolution layers and S-type growth curve (Sigmoid) functions, wherein the output of the third residual block is up-sampled (Upsampling) and then transferred to the 4 th layer (level3), and meanwhile, the output of the third residual block enters the combination module of the 1 × 1 convolution layers and the S-type growth curve (Sigmoid) functions, the output is the output of the 5 th layer (level4), and the output of the 5 th layer is an image obtained by down-sampling the image of sRGB domain whose output data is 448 × 448 size 4 times, and the size is 28 × 28. The method comprises the steps of sequentially including a convolution stacking module, a merging module, a convolution stacking module, a 1 × 1 convolution layer and a combination module of an S-type growth curve (Sigmoid) function in a 4 th layer, wherein Input data (Input) enters the 4 th layer (level3) after passing through 2 convolution stacking modules (ConvMultiBlock) and 2 times of pooling (Maxpooling), the data processed by the first convolution stacking module and the data transmitted upwards after up-sampling (Upsampling) of the output of a third residual block in the 5 th layer simultaneously enter the merging module, the merging module (Concat) combines the output of the first convolution stacking module (ConvMultiBlock) and a characteristic diagram from the level4 together, the data processed by the second convolution stacking module (ConvBlockBlock) enters the third layer after up-sampling on the one hand, the data processed by the 1 × 1 convolution layer and a combination module of the S-type growth curve (Sigmoid) function on the other hand serve as combined output data of the 4 × 4 th layer, and the data of the image data of the 4 × 448 is down-sampled and output as GB image data of the size of the 4 th layer after down-sampling, the size is 56 × 56. The structures of the 3 rd layer and the 2 nd layer are the same as the structure of the 4 th layer, and the data processing process is also the same, so the description is not repeated. The layer 1 combination module, which includes only 1 × 1 convolutional layers and S-type growth curve (Sigmoid) functions, inputs the output upsampled data of the second convolutional stacking module in the second layer, and outputs an image of the sRGB domain of size 448 × 448.
In the training process, the 5 th layer (level4) is trained, and the sRGB domain data obtained by down-sampling the sRGB domain image of 448 × 448 size by four times with reference to the standard (Ground Truth), i.e., the output of the 5 th layer, has a size of 28 × 28. The input data of layer 5 is data after 3 times pooling of arrays of size 4 × 224 × 224, with size 28 × 28. After the 5 th layer is trained, the mapping relation between the image data of the Raw domain with the size of 28 × 28 and the sRGB domain data with the same size is obtained, the training result is upwards transferred to the 4 th layer, and then the 4 th layer is trained until reaching the first layer. And finally obtaining the mapping relation between the Raw domain image data and the sRGB domain image.
And testing the neural network by using the image in the test set, evaluating the training effect, ending the training if the training effect is in accordance with the expectation, applying the trained neural network to the target equipment, finishing the processing of the image data of the Raw domain by the neural network after the target equipment acquires the image data of the Raw domain, enabling the final imaging effect of the target equipment to be close to the imaging effect of the reference equipment, simplifying the image processing process and reducing the labor input and time cost for debugging the camera.
Fig. 3 is a block diagram of a camera imaging device shown in accordance with an exemplary embodiment. Referring to fig. 3, the image forming apparatus includes: an image acquisition module 301, a normalization processing module 302, a training module 303, and an application module 304.
The image acquisition module 301 is configured to acquire image data of a plurality of Raw domains using a target device, and acquire images of a plurality of sRGB domains corresponding to the image data of the plurality of Raw domains using a reference device.
The normalization processing module 302 is configured to normalize pairs of image data for a plurality of Raw domains.
The training module 303 is configured to train a neural network using the preprocessed image data of the plurality of Raw domains and the images of the plurality of sRGB domains.
The application module 304 is configured to apply the trained neural network to the target device, so that the target device generates an image in the sRGB domain from the captured image data in the Raw domain.
In an exemplary embodiment, the normalization processing module 302 normalizes pairs of image data of a plurality of Raw fields including:
and carrying out black level correction processing on the image data of the Raw domain, or dividing the value of each pixel point in the image data of the Raw domain by a preset value.
In an exemplary embodiment, the training module training the neural network using the preprocessed image data of the plurality of Raw domains and the images of the plurality of sRGB domains includes:
and taking the preprocessed image data of the Raw domain as input data of the neural network, taking an image of the sRGB domain corresponding to the image data of the Raw domain as an output image of the neural network, and training the neural network to obtain a mapping relation between the image data of the Raw domain and the image of the sRGB domain.
Fig. 4 is a block diagram illustrating a method for a camera imaging device 400 according to an exemplary embodiment. Referring to fig. 4, the apparatus 400 includes a processor 401, and the number of processors may be set to one or more as necessary. The device 400 also includes a memory 402 for storing instructions, such as an application program, that are executable by the processor 401. The number of the memories can be set to one or more according to needs. Which may store one or more application programs. The processor 401 is configured to execute instructions to perform the camera imaging method described above.
As will be appreciated by one skilled in the art, the embodiments herein may be provided as a method, apparatus (device), or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied in the medium. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer, and the like. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices) and computer program products according to embodiments herein. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such article or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of additional like elements in the article or device comprising the element.
While the preferred embodiments herein have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following appended claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of this disclosure.
It will be apparent to those skilled in the art that various changes and modifications may be made herein without departing from the spirit and scope thereof. Thus, it is intended that such changes and modifications be included herein, provided they come within the scope of the appended claims and their equivalents.
Claims (10)
1. A camera imaging method, comprising:
acquiring image data of a plurality of Raw domains by using a target device, and acquiring images of a plurality of sRGB domains corresponding to the image data of the plurality of Raw domains by using a reference device;
normalizing the image data pairs of the plurality of Raw fields;
training a neural network by using the normalized image data of the plurality of Raw domains and the corresponding images of the plurality of sRGB domains;
and applying the trained neural network to the target equipment so that the target equipment generates an image of an sRGB domain according to the shot image data of the Raw domain.
2. The camera imaging method of claim 1, wherein the normalization process comprises:
and performing black level correction processing on the image data of the Raw domain, or dividing the value of each pixel point in the image data of the Raw domain by a preset value.
3. The camera imaging method of claim 1,
the training neural network using the preprocessed image data of the plurality of Raw domains and the images of the plurality of sRGB domains comprises:
and taking the preprocessed image data of the Raw domain as input data of the neural network, taking an image of an sRGB domain corresponding to the image data of the Raw domain as an output image of the neural network, and training the neural network to obtain a mapping relation between the image data of the Raw domain and the image of the sRGB domain.
4. The camera imaging method according to claim 3, wherein the neural network is an N-layer structure, each of the second to N-th layers includes a plurality of convolution stacking modules, and the using the image data of the preprocessed Raw domain as the input data of the neural network includes:
converting the image data of the Raw domain into 4-channel data according to the channel sequence of B, Gb, R and Gr, and inputting the 4-channel data into a second layer of the neural network; taking the data processed and pooled by the convolution stacking module in the second layer as the input data of the third layer, and so on until reaching the Nth layer;
the taking the image of the sRGB domain corresponding to the image data of the Raw domain as the output image of the neural network includes:
and taking the image of the sRGB domain as the output of the first layer of the neural network, and taking the image of the sRGB domain after N-1 times of downsampling as the output image of the Nth layer, wherein N is an integer more than or equal to 3.
5. The camera imaging method as claimed in claim 4, wherein the second to N-1 layers of the neural network further comprise a combining module for combining the input data of each layer with the up-sampling result of the output data of the next layer.
6. The camera imaging method as claimed in any one of claims 4 to 5, wherein said training neural network comprises:
and training the neural network layer by layer according to the sequence from the Nth layer to the first layer.
7. A camera imaging apparatus, comprising:
an image acquisition module configured to acquire image data of a plurality of Raw domains using a target device, and acquire images of a plurality of sRGB domains corresponding to the image data of the plurality of Raw domains using a reference device;
the normalization processing module is used for performing normalization processing on the image data pairs of the plurality of Raw domains;
the training module is used for training a neural network by using the preprocessed image data of a plurality of Raw domains and the images of a plurality of sRGB domains;
and the application module is used for applying the trained neural network to the target equipment so that the target equipment generates an image of an sRGB domain according to the shot image data of the Raw domain.
8. The camera imaging apparatus of claim 7, wherein the training module training the neural network using the preprocessed image data of the plurality of Raw domains and the images of the plurality of sRGB domains comprises:
and taking the preprocessed image data of the Raw domain as input data of the neural network, taking an image of an sRGB domain corresponding to the image data of the Raw domain as an output image of the neural network, and training the neural network to obtain a mapping relation between the image data of the Raw domain and the image of the sRGB domain.
9. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed, implements the steps of the method according to any one of claims 1-6.
10. A computer arrangement comprising a processor, a memory and a computer program stored on the memory, characterized in that the steps of the method according to any of claims 1-6 are implemented when the computer program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111085826.9A CN113808043A (en) | 2021-09-16 | 2021-09-16 | Camera imaging method, device, medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111085826.9A CN113808043A (en) | 2021-09-16 | 2021-09-16 | Camera imaging method, device, medium and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113808043A true CN113808043A (en) | 2021-12-17 |
Family
ID=78941236
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111085826.9A Pending CN113808043A (en) | 2021-09-16 | 2021-09-16 | Camera imaging method, device, medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113808043A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111383171A (en) * | 2018-12-27 | 2020-07-07 | Tcl集团股份有限公司 | Picture processing method, system and terminal equipment |
| CN111402146A (en) * | 2020-02-21 | 2020-07-10 | 华为技术有限公司 | Image processing method and image processing apparatus |
| CN112166455A (en) * | 2019-09-26 | 2021-01-01 | 深圳市大疆创新科技有限公司 | Image processing method, image processing device, movable platform and machine-readable storage medium |
| CN113112424A (en) * | 2021-04-08 | 2021-07-13 | 深圳思谋信息科技有限公司 | Image processing method, image processing device, computer equipment and storage medium |
-
2021
- 2021-09-16 CN CN202111085826.9A patent/CN113808043A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111383171A (en) * | 2018-12-27 | 2020-07-07 | Tcl集团股份有限公司 | Picture processing method, system and terminal equipment |
| CN112166455A (en) * | 2019-09-26 | 2021-01-01 | 深圳市大疆创新科技有限公司 | Image processing method, image processing device, movable platform and machine-readable storage medium |
| CN111402146A (en) * | 2020-02-21 | 2020-07-10 | 华为技术有限公司 | Image processing method and image processing apparatus |
| CN113112424A (en) * | 2021-04-08 | 2021-07-13 | 深圳思谋信息科技有限公司 | Image processing method, image processing device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| CHEN CHEN 等: "Learning to See in the Dark", 《ARXIV》, pages 1 - 10 * |
| OLAF RONNEBERGER 等: "U-Net: Convolutional Networks for Biomedical Image Segmentation", 《ARXIV》, pages 1 - 8 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3350767B1 (en) | Exposure-related intensity transformation | |
| CN109636754B (en) | Extremely-low-illumination image enhancement method based on generation countermeasure network | |
| WO2019233394A1 (en) | Image processing method and apparatus, storage medium and electronic device | |
| US20200084363A1 (en) | Temporally aligned exposure bracketing for high dynamic range imaging | |
| CN115442515B (en) | Image processing method and apparatus | |
| RU2397542C2 (en) | Method and device for creating images with high dynamic range from multiple exposures | |
| US11625815B2 (en) | Image processor and method | |
| AU2018346909A1 (en) | Image signal processor for processing images | |
| CN110428366A (en) | Image processing method and device, electronic equipment, computer readable storage medium | |
| US10425599B2 (en) | Exposure selector for high-dynamic range imaging and associated method | |
| CN106797437A (en) | High dynamic range images using multiple images synthesize | |
| JP2023525702A (en) | Image adjustment based on machine learning | |
| CN111598799A (en) | Image toning enhancement method and image toning enhancement neural network training method | |
| CN113079325B (en) | Method, apparatus, medium, and device for imaging billions of pixels under dim light conditions | |
| WO2020187424A1 (en) | Image processor | |
| WO2021063118A1 (en) | Method and apparatus for image processing | |
| CN110930341A (en) | Low-illumination image enhancement method based on image fusion | |
| US11977319B2 (en) | Saliency based capture or image processing | |
| WO2016202926A1 (en) | Method and device for producing a digital image | |
| Ueda et al. | Underwater image synthesis from RGB-D images and its application to deep underwater image restoration | |
| CN104010134B (en) | For forming the system and method with wide dynamic range | |
| CN113808043A (en) | Camera imaging method, device, medium and equipment | |
| CN111383188B (en) | Image processing method, system and terminal equipment | |
| US11889175B2 (en) | Neural network supported camera image or video processing pipelines | |
| KR20190100833A (en) | Apparatus for generating high dynamic range image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |