CN115457157A - Image simulation method, image simulation device and electronic equipment - Google Patents
Image simulation method, image simulation device and electronic equipment Download PDFInfo
- Publication number
- CN115457157A CN115457157A CN202211114952.7A CN202211114952A CN115457157A CN 115457157 A CN115457157 A CN 115457157A CN 202211114952 A CN202211114952 A CN 202211114952A CN 115457157 A CN115457157 A CN 115457157A
- Authority
- CN
- China
- Prior art keywords
- image
- simulation
- optical imaging
- radiation intensity
- image sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/20—Drawing from basic elements, e.g. lines or circles
Abstract
The embodiment of the invention provides an image simulation method, an image simulation device and electronic equipment. The image simulation method comprises the following steps: determining the radiation intensity distribution of the simulated field light based on the spectral power distribution of the simulated field light and the input image information; based on the radiation intensity distribution, determining optical imaging data of the simulated field of view rays after passing through the optical imaging system by using imaging parameters of the optical imaging system; determining an image conversion voltage generated by the optical imaging data at the image sensor using a first photoelectric conversion parameter of the image sensor; sampling the image conversion voltage based on a color filter array of the image sensor to obtain an original simulation image; and performing noise adding simulation on the original simulation image based on noise statistical distribution to obtain a simulated noisy image. The scheme of the embodiment of the invention realizes the forward reliable simulation of the original image and reduces the cost of the image simulation.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to an image simulation method, an image simulation device and electronic equipment.
Background
With the rapid development of the fields of monitoring cameras, smart phones, automatic driving and the like, people increasingly have outstanding requirements on high-quality images and videos, and the requirements on the performance and the imaging quality of an image sensor are higher and higher.
Raw image data directly output by an image sensor such as the Bayer-RGGB format may also be referred to as image Raw data, and the Raw image data is subjected to a series of processing such as RGB color interpolation to obtain standard RGB (standard RGB, sRGB) image data, i.e., a visualized image.
The processing of Raw image data is an important part of the image imaging process of an image sensor and plays a key role in the imaging quality, and in recent years, a deep learning method is applied to the processing of the Raw image data to realize good performance.
In contrast, it is easy to acquire standard RGB image data, and Raw image data is not visual data that is finally output, and it is difficult to acquire Raw image data, so a reliable and low-cost image simulation method is required.
Disclosure of Invention
Embodiments of the present invention provide an image simulation method, an image simulation apparatus, and an electronic device to at least partially solve the above problems.
According to a first aspect of the embodiments of the present invention, there is provided an image simulation method, including: determining the radiation intensity distribution of the simulated view field light rays based on the spectral power distribution of the simulated view field light rays and the input image information; based on the radiation intensity distribution, determining optical imaging data of the simulated field of view rays after passing through the optical imaging system by using imaging parameters of the optical imaging system; determining an image conversion voltage generated by the optical imaging data at an image sensor using a first photoelectric conversion parameter of the image sensor; sampling the image conversion voltage based on a color filter array of the image sensor to obtain an original simulation image; and performing noise adding simulation on the original simulation image based on noise statistical distribution to obtain a simulated noisy image.
In another implementation of the invention, the input image information is color channel values of the visualized image. The determining the radiation intensity distribution of the simulated field of view light based on the spectral power distribution of the simulated field of view light and the input image information comprises: multiplying the relative spectral power distribution of each color channel by the spectral power distribution of the light source to obtain the spectral power distribution of each color channel, wherein the spectral power distribution indicates the mapping relation between the wavelength of the light source and the total radiation intensity, and the relative spectral power distribution indicates the mapping relation between the radiation intensity of the wavelength of each color channel and the total radiation intensity; and multiplying the spectral power distribution of each color channel with the color channel value of the visual image to obtain the radiation intensity distribution of the simulated field light.
In another implementation of the invention, the method further comprises: and determining a linear RGB image obtained by carrying out inverse gamma conversion on the standard RGB image as the visual image.
In another implementation of the present invention, the first photoelectric conversion parameter of the image sensor includes a photoelectric conversion efficiency of m color channels of the image sensor, m being an integer greater than 2; the determining an image conversion voltage generated by the optical imaging data at the image sensor using a first photoelectric conversion parameter of the image sensor comprises: and at least multiplying the radiation intensity distribution indicated by the optical imaging data and the photoelectric conversion efficiency of the c color channel of the image sensor to obtain an image conversion voltage generated by irradiating the pixel corresponding to the c color channel with light, wherein c is an integer and is more than or equal to 1 and less than or equal to m.
In another implementation of the invention, the input image information comprises n image frames, n being an integer greater than 2. The multiplying at least the radiation intensity distribution indicated by the optical imaging data and the photoelectric conversion efficiency of the c-th color channel of the image sensor to obtain the image conversion voltage generated by the light irradiating on the pixel corresponding to the c-th color channel comprises: determining the light of the ith image frame and the ith image conversion current generated on the pixel of the c-th color channel based on the product of the radiation intensity distribution of the light of the ith image frame and the photoelectric conversion efficiency of the c-th color channel of the image sensor, wherein i is an integer, and is more than or equal to 1 and less than or equal to n; determining an ith image conversion voltage based on a product between the ith image conversion current and an exposure time of the ith image frame; and summing the n image conversion voltages to obtain an image conversion voltage corresponding to the c color channel.
In another implementation manner of the present invention, sampling the image conversion voltage based on a color filter array of the image sensor to obtain an original simulation image includes: constructing an image sampling function based on a color filter array of the image sensor; and inputting the image conversion voltage into the image sampling function to obtain an original simulation image.
In another implementation manner of the present invention, based on noise statistical distribution, performing noise-adding simulation on the original simulation image to obtain a simulated noisy image, including: carrying out normalization processing on the original simulation image to obtain a normalized image of the original simulation image; determining a noise statistical distribution of the normalized image based on a second photoelectric conversion parameter of the image sensor; and based on the noise statistical distribution, carrying out noise adding processing on the normalized image to obtain a simulated noisy image.
In another implementation of the present invention, the determining a noise statistical distribution of the normalized image based on the second photoelectric conversion parameter of the image sensor includes: determining a noise Gaussian distribution of the normalized image based on the following formula: sigma 2 ＝α*I norm + beta; wherein α = CG AG I max /((I max -I black )*V max ) (ii) a And β = σ 2 R *AG 2 *I 2 max /((I max -I black ) 2 *V 2 max )。σ 2 Variance of the noise Gaussian distribution; i is norm Representing the normalized image; CG and AG are the conversion gain and the analog gain of the image sensor, respectively; v max And σ R The maximum voltage swing and the noise standard deviation of the image sensor are respectively; i is max And I black Respectively the maximum value and the minimum value after analog-digital conversion and quantization of the image sensor.
In another implementation manner of the present invention, determining optical imaging data of the simulated field of view ray after passing through the optical imaging system by using an imaging parameter of the optical imaging system based on the radiation intensity distribution includes: constructing an imaging function of an optical imaging system based on imaging parameters of the optical imaging system, wherein the imaging function represents the change of the light ray radiation intensity distribution before and after imaging of the optical imaging system; and inputting the radiation intensity distribution of the simulated field of view light into the imaging function to obtain the optical imaging data of the simulated field of view light.
According to a second aspect of the embodiments of the present invention, there is provided an image simulation apparatus including: the first simulation module is used for determining the radiation intensity distribution of the simulated field light based on the spectral power distribution of the simulated field light and input image information; the second simulation module is used for determining optical imaging data of the simulated field of view rays after passing through the optical imaging system by utilizing the imaging parameters of the optical imaging system based on the radiation intensity distribution; a third simulation module for determining an image conversion voltage generated by the optical imaging data at the image sensor by using a first photoelectric conversion parameter of the image sensor; the fourth simulation module is used for sampling the image conversion voltage based on the color filter array of the image sensor to obtain an original simulation image; and the fifth simulation module is used for performing noise adding simulation on the original simulation image based on the noise statistical distribution to obtain a simulation noisy image.
According to a third aspect of embodiments of the present invention, there is provided an electronic apparatus, including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus; the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the corresponding operation of the method according to the first aspect.
In the scheme of the embodiment of the invention, the simulation of the radiation intensity distribution of the simulated field light is realized by simulating the spectral power distribution of the field light and inputting image information, the simulation of the optical imaging process is realized by the imaging parameters of the optical imaging system, the simulation of the photoelectric conversion process is realized by the photoelectric conversion parameters based on the image sensor, the simulation of the image sampling process is realized based on the color filter array of the image sensor, and the simulation of the noise adding process is realized based on the noise statistical distribution. Because the generation of the input image information sequentially requires the radiation intensity distribution of field-of-view rays, an optical imaging process, a photoelectric conversion process, an image sampling process and a noise adding process, compared with the traditional reverse simulation process, the scheme of the embodiment of the invention realizes the forward reliable simulation of the original image with noise by inputting the image information. In addition, the spectral power distribution of the simulated field light, the imaging parameters of the optical imaging system, the photoelectric conversion parameters of the image sensor and the color filter array of the image sensor can be simulated into a data processing process, so that the cost of image simulation is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present invention, and it is also possible for a person skilled in the art to obtain other drawings based on the drawings.
FIG. 1 is a flow diagram of steps in an image processing flow according to one example.
FIG. 2 is a flow chart of steps of an image simulation method according to one embodiment of the present invention.
Fig. 3 shows a schematic diagram of the image sampling process of the embodiment of fig. 2.
FIG. 4 is a schematic diagram of a sequence of image frames for a dynamic exposure process simulation of FIG. 2.
Fig. 5 is a block diagram of an image simulation apparatus according to another embodiment of the present invention.
Fig. 6 is a schematic structural diagram of an electronic device according to another embodiment of the invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described in detail below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments of the present invention shall fall within the scope of the protection of the embodiments of the present invention.
The following further describes specific implementation of the embodiments of the present invention with reference to the drawings.
FIG. 1 shows an exemplary image processing flow. Generally, the image processing flow may process an image acquired by an image processor in RGB format, for example, an output image of the image processor may be referred to as Raw image data, and a pixel value of each pixel is a channel value of any one of an R channel, a G channel, or a B channel.
The image processing flow of the present example involves an image acquisition module 110, a color interpolation module 120, a color correction module 130, and a gamma correction module 140. The image acquisition module 110 may be implemented by an optical imaging system and a pixel sensor, and acquires Raw image data.
After Raw image data of the image sensor is acquired, each pixel value in the input image is corrected in the color interpolation module 120, and corrected image data is obtained. It should be understood that the color interpolation module 120 may be a part of Image Signal Processing (ISP), and the input Image may be obtained after performing Black Level Correction (BLC), lens Shading Correction (LSC), dead Pixel Correction (DPC), and other processes.
The linear RGB image is output through the processing of the color interpolation module 120. Further, in the color correction module 130, the respective channel values of the R channel, the G channel, and the B channel are corrected. Then, in the gamma correction module 140, correction between the pixel values of the linear RGB image and the display luminance is performed, resulting in a standard RGB image, i.e., a visualized image. The Raw image data is not a visual image which is finally output, and the difficulty in acquiring the Raw image data is high.
Therefore, a series of image simulation schemes are improved, the reliability of image simulation can be improved, the cost of image simulation is lowered, and the problem caused by difficulty in acquisition of Raw image data is solved.
Fig. 2 illustrates an image simulation method according to an embodiment of the present invention, which may be performed using any computing device having data processing capabilities, such as a terminal device, a desktop computer, or a server for image simulation processing. In addition, the image acquisition system simulated by the image simulation method can be an electronic device with image shooting capability, such as a digital camera, a mobile phone and the like. The image acquisition system comprises an optical imaging system and a pixel sensor and can be used for subsequent image signal processing.
The image simulation method of the embodiment includes:
s210: and determining the radiation intensity distribution of the simulated field light based on the spectral power distribution of the simulated field light and the input image information.
It should be understood that the visualized image may be a standard RGB image, or may be a linear RGB image obtained by performing inverse gamma conversion on the standard RGB image.
It should also be understood that the Spectral Power Distribution (SPD) of the simulated field of view light is indicative of the radiant Power value of the light waves at each wavelength in the simulated field of view light.
S220: and determining optical imaging data of the simulated field of view light after passing through the optical imaging system by using the imaging parameters of the optical imaging system based on the radiation intensity distribution.
It should be understood that the optical imaging system is a lens group, and the imaging parameters include the focal length, focal ratio, magnification, etc. of the lens group; or the focal length, focal ratio, magnification, etc. of each lens in the lens group. The simulated field light is used as the incident light of the optical imaging system, and the emergent light is obtained after passing through the optical imaging system. The optical imaging data of the simulated field of view rays may be optical imaging data of the emergent rays, for example, a radiation intensity distribution of the emergent rays.
The relationship between the radiation intensity distribution of the incident light and the radiation intensity distribution of the emergent light is correlated with an imaging function, which can be determined based on imaging parameters of the optical imaging system.
S230: an image conversion voltage generated by the optical imaging data at the image sensor is determined using the first photoelectric conversion parameter of the image sensor.
It will be appreciated that when performing photoelectric conversion, the photodetectors of the pixel sensor are capable of detecting photons of the outgoing light, and the photodetectors of each pixel form a current and a voltage by detecting the corresponding conversion of the photons into electrons. The image conversion voltage may be characterized as a voltage map, and when the pixel sensor performs photoelectric conversion based on different color channels, a voltage map corresponding to each color channel may be generated. For example, for a black and white image, the pixel sensor generates a voltage map for the grayscale channel, and for the RGB channel, the pixel sensor generates a voltage map for each of the three channels.
S240: and sampling the image conversion voltage based on a color filter array of the image sensor to obtain an original simulation image.
It should be understood that the image sensor may be implemented as a pixel sensor, i.e., an image sensor that implements photoelectric conversion based on an array of photosensitive pixels. The Color Filter array herein may be a CFA (Color Filter Arrays), image sampling using the CFA is a means for improving the optical imaging quality, and the Color Filter array according to the embodiment of the present invention may be any, for example, bayer Filter, RCCC, RCCB, etc. As a specific example, the color filter array may be a Bayer-RGGB array. The voltage map can be sampled based on a color filter array to obtain an original simulation image. It should also be understood that the original simulated image is referred to herein as simulated raw data (original image data)
S250: and performing noise adding simulation on the original simulation image based on the noise statistical distribution to obtain a simulated noisy image.
It should be understood that the noise statistical distribution may be a statistical distribution such as a gaussian distribution, a normal distribution, or the like that reflects the regularity of the noise in the image. As one example, the noise statistical distribution may be a statistical distribution preset based on experience. As another example, a photoelectric conversion parameter related to image noise generation in the pixel sensor may be determined, and the noise statistical distribution may be a statistical distribution determined from such photoelectric conversion parameter.
In the scheme of the embodiment of the invention, the simulation of the radiation intensity distribution of the simulated field light is realized by simulating the spectral power distribution of the field light and inputting image information, the simulation of the optical imaging process is realized by the imaging parameters of an optical imaging system, the simulation of the photoelectric conversion process is realized by the photoelectric conversion parameters based on an image sensor, the simulation of the image sampling process is realized based on the color filter array of the image sensor, and the simulation of the noise adding process is realized based on the noise statistical distribution. Because the generation of the input image information sequentially requires the radiation intensity distribution of field-of-view rays, an optical imaging process, a photoelectric conversion process, an image sampling process and a noise adding process, compared with the traditional reverse simulation process, the scheme of the embodiment of the invention realizes the forward reliable simulation of the original image with noise by inputting the image information. In addition, the spectral power distribution of the simulated field light, the imaging parameters of the optical imaging system, the photoelectric conversion parameters of the image sensor and the color filter array of the image sensor can be simulated into a data processing process, so that the cost of image simulation is reduced.
In other examples, the input image information is a color channel value of the visualization image. Determining the radiation intensity distribution of the simulated field of view light based on the spectral power distribution of the simulated field of view light and the input image information, comprising: multiplying the relative spectral power distribution of each color channel by the spectral power distribution of the light source to obtain the spectral power distribution of each color channel, wherein the spectral power distribution indicates the mapping relation between the wavelength of the light source and the total radiation intensity, and the relative spectral power distribution indicates the mapping relation between the radiation intensity of the wavelength of each color channel and the total radiation intensity; and multiplying the spectral power distribution of each color channel with the color channel value of the visual image to obtain the radiation intensity distribution of the simulated field light. The field of view light is generated by the light source, and the spectral power distribution of the field of view light generating the visual image is reliably simulated by adopting the relative spectral power distribution of each color channel and the spectral power distribution of the light source.
It should be understood that the visualized image may be a standard RGB image, or may be a linear RGB image obtained by performing inverse gamma conversion on the standard RGB image. Specifically, the radiation intensity distribution of the simulated field ray can be calculated by the following formula 1:
P scene ＝(lRGB*SPD RGB )*SPD ill (formula 1)
Wherein, P scene Representing the radiation intensity distribution of the simulated field of view rays; lRGB represents a channel value of an R channel, a G channel, or a B channel of the linear RGB image; SPD RGB Representing the relative spectral power distribution of the R, G or B channel of a linear RGB image, i.e. the mapping between the radiation intensity of the wavelength of the R, G or B channel and the total radiation intensity; SPD ill Represents the spectral power distribution of the light source, i.e. the mapping between the wavelength of the light source and the total radiation intensity.
For example, the spectral power distribution of the light source may be a spectral power distribution vector, with each element in the vector indicating a radiation power value of the light wave at each discretized wavelength, which may have the same or different wavelength intervals, e.g., between the wavelength range of 400nm-700nm, with sampling wavelength intervals of 10nm, as evidenced by 31 discretized wavelengths.
As another example, the color channel values of the linear RGB image may be expressed as a matrix of pixel values H x W x 3,H representing the number of pixels of the linear RGB image in the height dimension, W representing the number of pixels of the linear RGB image in the width dimension, and 3 representing the number of channels, i.e., R channel, G channel, and B channel.
For another example, the relative spectral power distribution of each color channel may be a relative spectral power distribution matrix, and each element in the relative spectral power distribution matrix represents a relative radiation power value of each discretized wavelength field light wave corresponding to the R channel, the G channel, and the B channel.
For another example, the radiation intensity distribution of the simulated field of view light may be a radiation intensity distribution matrix, each element in the radiation intensity distribution matrix indicating a radiation power value of the field of view light wave of each discretized wavelength corresponding to the R, G, and B channels.
In other examples, determining optical imaging data of the simulated field of view rays after passing through the optical imaging system based on the radiation intensity distribution by using imaging parameters of the optical imaging system includes: and then, inputting the radiation intensity distribution of the simulated field-of-view light into the imaging function to obtain the radiation intensity distribution of the emergent light. Imaging parameters based on the optical imaging system can reliably represent the imaging characteristics of a lens, so that an imaging function can be reliably constructed through the imaging parameters, and the simulation reliability of the optical imaging process is ensured.
Specifically, the imaging function can be constructed by equation 2:
P optics ＝F(P scene (ii) a FN, M) (equation 2);
wherein F represents an imaging function; p is optics Representing the radiation intensity distribution of the emergent ray; FN (F-Number) denotes the focal ratio of the optical imaging system; m denotes an optical magnification of the optical imaging system.
It should be understood that the imaging function may also include the focal length FL of the camera lens as a variable. The optical imaging system may be a lens group, for example, a camera lens group.
Without considering factors such as optical diffraction, MTF (Modulation Transfer Function) characteristics of an optical imaging system, and off-optical axis luminance nonuniformity, as a specific example of an imaging Function, a mathematical expression thereof is: p optics ＝P scene *π/(1+4*FN 2 *(1+abs(M)) 2 ) Where abs (·) denotes the absolute value. The imaging function of the present example improves the simulation efficiency of the optical imaging process.
In another example, the first photoelectric conversion parameter of the image sensor includes a photoelectric conversion efficiency of m color channels of the image sensor, m being an integer greater than 2. Accordingly, determining an image conversion voltage generated by the optical imaging data at the image sensor using the first photoelectric conversion parameter of the image sensor comprises: and at least multiplying the radiation intensity distribution of the emergent ray by the photoelectric conversion efficiency of the c color channel of the image sensor to obtain the image conversion voltage generated by the emergent ray irradiating on the pixel corresponding to the c color channel, wherein c is an integer and is more than or equal to 1 and less than or equal to m. Since the photoelectric conversion efficiency of each color channel is determined, the photoelectric conversion characteristics of each color channel are simulated, thereby reliably simulating the photoelectric conversion process.
In other examples, sampling the image conversion voltage based on a color filter array of the image sensor to obtain an original simulation image includes: based on a color filter array of the image sensor, an image sampling function is constructed, and then, an image conversion voltage is input to the image sampling function to obtain an original simulation image. The image sampling process is reliably simulated by constructing the image sampling function through the color filter array.
For example, the Color Filter array herein may be a Color Filter Array (CFA), image sampling by using the CFA is a means for improving optical imaging quality, and the Color Filter array according to the embodiment of the present invention may be any Color Filter array, such as Bayer Filter, RCCC, RCCB, and the like. As a specific example, the color filter array may be a Bayer-RGGB array.
The image sampling function can be constructed using the following equation 3:
I R ＝∑F(V c (ii) a CFA) (formula 3)
Wherein F represents the image sampling function, I R Representing Raw image data, V c Representing the image conversion voltage generated by the pixel of the c-th color channel of the m color channels.
Fig. 3 illustrates a CFA-based image sampling process, for example, where the image conversion voltages generated by the pixels of the respective color channels may be characterized as voltage maps, such as the R-channel voltage map 31, the G-channel voltage map 32, and the B-channel voltage map 33 shown in fig. 3.
Each of the R channel voltage map 31, the G channel voltage map 32, and the B channel voltage map 33 may be sampled by using the color filter array 30, that is, the color filter array 30 is used as a unit, and the channel voltage maps are shifted in each channel voltage map, so that the same channel pixels of each channel voltage map and the color filter array 30 are retained (i.e., sampled), and the R channel voltage map 310, the G channel voltage map 320, and the B channel voltage map 330 after sampling are obtained respectively. And then, superposing the sampled voltage maps of the channels to obtain an original simulation image.
Further, in an example photoelectric conversion process, the visualization image includes n image frames captured in the input video, n being an integer greater than 2, and the static exposure and dynamic exposure processes may be simulated based on the n image frames.
Specifically, at least multiplying the radiation intensity distribution indicated by the optical imaging data by the photoelectric conversion efficiency of the c-th color channel of the image sensor to obtain an image conversion voltage generated by irradiating the pixel corresponding to the c-th color channel with the emergent light, includes: the method comprises the steps of determining the product between the radiation intensity distribution of emergent rays of an ith image frame and the photoelectric conversion efficiency of a c color channel of an image sensor, determining the emergent rays of the ith image frame and the ith image conversion current generated on pixels of the c color channel, then determining the ith image conversion voltage based on the product between the ith image conversion current and the exposure time of the ith image frame, and then summing n image conversion voltages to obtain the image conversion voltage corresponding to the c color channel. The differences of the respective color channel values in different image frames can record dynamically changing field light by calculating the image conversion current of each image frame, and then the dynamic exposure is reliably simulated by accumulating the discretized image conversion voltage, that is, the simulated blurring effect is increased.
In one example, for a static exposure process, the image conversion current can be calculated by equation 4, and the image conversion voltage can be further calculated by equation 5:
C c ＝P optics *QE c *Δ*q*(S a *F f ) (equation 4);
V c ＝C c * IT CG/q (equation 5);
wherein, C c Representing the image conversion current corresponding to the c-th color channel; p optics Representing the radiation intensity distribution of the emergent ray; v c Indicating the image conversion voltage corresponding to the c-th color channel.
Furthermore, QE c Represents the photoelectric conversion efficiency of the c-th color channel, and Δ represents the wavelength sampling interval; q represents a charge unit constant; s a Represents the physical area of the pixel; f f A fill factor representing a pixel; IT denotes an exposure time, CG is a conversion gain parameter of the image sensor, and q denotes a charge unit constant.
That is, in the case where the visualized image is an image frame, the image conversion voltage corresponding to the c-th color channel of the image frame may be determined based on the product relationship between the image conversion current corresponding to the c-th color channel of the image frame and the exposure time IT of the image frame.
In addition, when the static exposure process simulation is executed, the Image simulation method of the embodiment of the invention can be realized by adopting software to cooperate with an optical imaging system and an Image processing system, for example, the ISET (Image Systems Evaluation Toolkit) is a set of integrated software engineering and can be used for capturing and processing simulated field light. By simulating the optical imaging system and the image processing system, ISET can achieve a conversion of the visualized image to Raw image data under specific camera parameters.
In another example, for dynamic exposure process simulation, the visualized image is n image frames, and the accumulated image conversion voltage corresponding to the c-th color channel can be calculated based on equation 6 and equation 7:
Vc＝∑(C c,i *IT i ) CG/q, (equation 6)
IT＝∑IT i (formula 7)
Where i denotes the ith image frame among the n image frames. That is, the product relationship between the image conversion current corresponding to the c-th color channel of each image frame and the exposure time of the image frame itself is calculated to obtain the image conversion voltage of the image frame, and then the image conversion voltages of the n image frames are accumulated to simulate the accumulated image conversion voltage within the total exposure time IT.
In a more specific example, the input video frame rate is 500fps (i.e., the frame rate is 2 ms), and n image frames of the visualization image are obtained from the input video, for example, the n image frames may be consecutive image frames.
The image frame sequence construction process is described below with reference to fig. 4 to obtain different dynamic exposure effects with different exposure times. As shown in fig. 4, a long exposure time and a short exposure time are set, and a long-and-short exposure frame sequence is constructed. The long exposure time may be N times the short exposure time, e.g., the long exposure time is 2 times the short exposure time, the exposure time of the short exposure frame is 2.5ms, and the exposure time of the long exposure frame is 5ms.
The image frame sequence of fig. 4 is formed by a sequence of consecutive long and short exposure frames, e.g. the first long and short exposure frame sequence comprises a long exposure frame N1 Long And short exposure frame N1 Short The second long-short exposure frame sequence comprises a long exposure frame N2 Long And short exposure frame N2 Short (not shown).
In the simulation long exposure frame N1 Long Taking the input long exposure frame [1,2,3]Respectively corresponding to the exposure time [2,2,1 ]]ms, the image conversion voltage of the long exposure frame is calculated using equation 6.
In the simulation short exposure frame N1 Short Then, an input short exposure frame [3,4 ] can be taken]Respectively corresponding to the exposure time [1,1.5]ms, the image conversion voltage of the short exposure frame is calculated using equation 6.
Continuous simulation long exposure frame N2 Lon Taking the input long exposure frame [4,5,6,7]Respectively corresponding to the exposure time [0.5,2,2,0.5]ms, the image conversion voltage of the long exposure frame is calculated using equation 6.
In other examples, in the image simulation method, as an example of performing noise-adding simulation on the original simulation image, normalization processing may be performed on the original simulation image to obtain a normalized image of the original simulation image, then, noise statistical distribution of the normalized image is determined based on the second photoelectric conversion parameter of the image sensor, and then, noise-adding processing may be performed on the normalized image based on the noise statistical distribution to obtain a simulated noisy image. It should be understood that the noise of the normalized image includes, but is not limited to, photon noise, read noise, and the like. The noise statistical distribution may be a gaussian distribution or a normal distribution. The noise statistical distribution of the normalized image reliably reflects the mechanism of generating noise in the pixel sensor, so that the noisy original image corresponding to the normalized image is reliably simulated.
Specifically, a noise model may be constructed based on the statistical distribution of noise, the noise model performing normalization processing and noise addition processing. And inputting the original simulation image into a noise model to obtain a simulation noisy image.
More specifically, determining a noise statistical distribution of the normalized image based on the second photoelectric conversion parameter of the image sensor includes: determining a noise gaussian distribution of the normalized image based on equations 8, 9, and 10 as follows:
σ 2 ＝α*I norm + beta (equation 8)
α＝CG*AG*I max /((I max -I black )*V max ) (formula 9)
And β = σ 2 R *AG 2 *I 2 max /((I max -I black ) 2 *V 2 max ) (equation 10)
Wherein σ 2 is the variance of the noise gaussian distribution; i is norm Representing a normalized image; CG and AG are Conversion Gain (Conversion Gain) and Analog Gain (Analog Gain) of the image sensor, respectively; v max And σ R The maximum voltage swing and the noise standard deviation of the image sensor are respectively; I.C. A max And I black Respectively, the quantized maximum and minimum values of the analog-to-digital conversion of the image sensor. Noise Gaussian distribution is constructed through the photoelectric conversion parameters of the image sensor, a mechanism for generating noise in the pixel sensor is reliably simulated, and accordingly a noisy original image is reliably simulated. Furthermore, by modifying the AG of the image sensor, different image shots can be simulated quicklyNoise of Raw image data under the standard is taken.
When the original simulation image is normalized, formula 11 can be adopted to normalize the original simulation image I R Performing maximum voltage swing based clamping:
I R (I R >V max )＝V max (equation 11);
then, the original simulation image after the clamping processing is normalized by adopting a formula 12 to obtain a normalized image I norm ：
I norm ＝I R /max(I R ) (in accordance with the equation 12),
where max (·) denotes a max operation;
then, the normalized image may be subjected to denoising processing by using formula 13 to obtain a noisy original image I noise ：
I noise ＝I norm +f(0,σ 2 ) (equation 13) of the above-mentioned formula,
wherein f (·) represents a gaussian distribution.
The clamping processing of the maximum voltage swing reflects the reliable parameter range of the pixel processor, and the reliability of the original simulation image subjected to the clamping processing is ensured.
In addition, an image data pair can be constructed based on the normalized image and the noisy original image, and when the deep learning method is used for image noise analysis, the image data pair can be used as training data for the image noise analysis, so that the reliability of the training data and the number of samples are improved.
Fig. 5 is a block diagram of an image simulation apparatus according to another embodiment of the present invention, the image simulation apparatus corresponding to the image simulation method of fig. 2, including:
the first simulation module 510 determines a radiation intensity distribution of the simulated field of view light based on the spectral power distribution of the simulated field of view light and the input image information.
The second simulation module 520 determines optical imaging data of the simulated field of view light after passing through the optical imaging system based on the radiation intensity distribution and by using the imaging parameters of the optical imaging system.
The third simulation module 530 determines an image conversion voltage generated by the optical imaging data at the image sensor using the first photoelectric conversion parameter of the image sensor.
The fourth simulation module 540 samples the image conversion voltage based on the color filter array of the image sensor to obtain an original simulation image.
The fifth simulation module 550 performs noise-adding simulation on the original simulation image based on the noise statistical distribution to obtain a simulated noisy image.
In the scheme of the embodiment of the invention, the simulation of the radiation intensity distribution of the simulated field light is realized by simulating the spectral power distribution of the field light and inputting image information, the simulation of the optical imaging process is realized by the imaging parameters of the optical imaging system, the simulation of the photoelectric conversion process is realized by the photoelectric conversion parameters based on the image sensor, the simulation of the image sampling process is realized based on the color filter array of the image sensor, and the simulation of the noise adding process is realized based on the noise statistical distribution. Because the generation of the input image information sequentially requires the radiation intensity distribution of field-of-view rays, an optical imaging process, a photoelectric conversion process, an image sampling process and a noise adding process, compared with the traditional reverse simulation process, the scheme of the embodiment of the invention realizes the forward reliable simulation of the original image with noise by inputting the image information. In addition, the spectral power distribution of the simulated field light, the imaging parameters of the optical imaging system, the photoelectric conversion parameters of the image sensor and the color filter array of the image sensor can be simulated into a data processing process, so that the cost of image simulation is reduced.
In other examples, the input image information is a color channel value of the visualization image. The first simulation module is specifically configured to: multiplying the relative spectral power distribution of each color channel by the spectral power distribution of the light source to obtain the spectral power distribution of each color channel, wherein the spectral power distribution indicates the mapping relation between the wavelength of the light source and the total radiation intensity, and the relative spectral power distribution indicates the mapping relation between the radiation intensity of the wavelength of each color channel and the total radiation intensity; and multiplying the spectral power distribution of each color channel with the color channel value of the visual image to obtain the radiation intensity distribution of the simulated field light.
In other examples, the first simulation module is further to: and determining a linear RGB image obtained by performing inverse gamma conversion on the standard RGB image as the visual image.
In other examples, the first photoelectric conversion parameter of the image sensor includes a photoelectric conversion efficiency of m color channels of the image sensor, m being an integer greater than 2. The third simulation module is specifically configured to: and at least multiplying the radiation intensity distribution indicated by the optical imaging data by the photoelectric conversion efficiency of the c color channel of the image sensor to obtain an image conversion voltage generated by irradiating the pixel corresponding to the c color channel with the emergent light, wherein c is an integer and is more than or equal to 1 and less than or equal to m.
In other examples, the input image information includes n image frames, n being an integer greater than 2. The third simulation module is specifically configured to: determining the product of the radiation intensity distribution of the emergent ray of the ith image frame and the photoelectric conversion efficiency of the c color channel of the image sensor, wherein i is an integer, and is more than or equal to 1 and less than or equal to n; determining an ith image conversion voltage based on a product between an ith image conversion current and an exposure time of an ith image frame; and summing the n image conversion voltages to obtain an image conversion voltage corresponding to the c color channel.
In other examples, the fourth simulation module is specifically configured to: and constructing an image sampling function based on the color filter array of the image sensor, and inputting the image conversion voltage into the image sampling function to obtain an original simulation image.
In other examples, the fifth simulation module is to: carrying out normalization processing on the original simulation image to obtain a normalized image of the original simulation image; determining a noise statistical distribution of the normalized image based on a second photoelectric conversion parameter of the image sensor; and based on the noise statistical distribution, carrying out noise adding processing on the normalized image to obtain a simulated noisy image.
In some other examples, the fifth simulation module is specifically configured to: determining a noise Gaussian distribution of the normalized image based on the following formula: sigma 2 ＝α*I norm + beta; wherein α = CG AG I max /((I max -I black )*V max ) (ii) a And β = σ 2 R *AG 2 *I 2 max /((I max -I black ) 2 *V 2 max ) (ii) a Wherein σ 2 Variance of the noise Gaussian distribution; i is norm Representing the normalized image; CG and AG are the conversion gain and the analog gain of the image sensor, respectively; v max And σ R The maximum voltage swing and the noise standard deviation of the image sensor are respectively; i is max And I black Respectively the maximum value and the minimum value after analog-digital conversion and quantization of the image sensor.
In other examples, the second simulation module is specifically configured to: constructing an imaging function of an optical imaging system based on imaging parameters of the optical imaging system, wherein the imaging function represents the change of the light radiation intensity distribution before and after imaging of the optical imaging system; and inputting the radiation intensity distribution of the simulated field of view rays into the imaging function to obtain the radiation intensity distribution of the emergent rays.
The apparatus of this embodiment is used to implement the corresponding method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again. In addition, the functional implementation of each module in the apparatus of this embodiment can refer to the description of the corresponding part in the foregoing method embodiments, and is not repeated herein.
Referring to fig. 6, a schematic structural diagram of an electronic device according to another embodiment of the present invention is shown, and the specific embodiment of the present invention does not limit the specific implementation of the electronic device.
As shown in fig. 6, the electronic device may include: a processor (processor) 602, a communication Interface 604, a memory 606 in which a program 610 is stored, and a communication bus 608.
The processor, the communication interface, and the memory communicate with each other via a communication bus.
And the communication interface is used for communicating with other electronic equipment or servers.
And the processor is used for executing the program, and particularly can execute the relevant steps in the method embodiment.
In particular, the program may include program code comprising at least one executable instruction.
The processor may be a processor CPU, or an Application Specific Integrated Circuit ASIC (Application Specific Integrated Circuit), or one or more Integrated circuits configured to implement embodiments of the present invention. The intelligent device comprises one or more processors which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And the memory is used for storing programs. The memory may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program may specifically be adapted to cause a processor to execute the image simulation method of fig. 2.
In addition, for specific implementation of each step in the program, reference may be made to corresponding steps and corresponding descriptions in units in the foregoing method embodiments, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.
It should be noted that, according to implementation requirements, each component/step described in the embodiment of the present invention may be divided into more components/steps, and two or more components/steps or partial operations of the components/steps may also be combined into a new component/step to achieve the purpose of the embodiment of the present invention.
The above-described method according to an embodiment of the present invention may be implemented in hardware, firmware, or as software or computer code storable in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk, or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware such as an ASIC or FPGA. It will be appreciated that a computer, processor, microprocessor controller, or programmable hardware includes memory components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by a computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code for implementing the methods illustrated herein, execution of the code transforms the general-purpose computer into a special-purpose computer for performing the methods illustrated herein.
Those of ordinary skill in the art will appreciate that the various illustrative elements and method steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.
The above embodiments are only for illustrating the embodiments of the present invention and not for limiting the embodiments of the present invention, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention, so that all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.
Claims (11)
1. An image simulation method, comprising:
determining the radiation intensity distribution of the simulated field light based on the spectral power distribution of the simulated field light and the input image information;
based on the radiation intensity distribution, determining optical imaging data of the simulated field of view rays after passing through the optical imaging system by using imaging parameters of the optical imaging system;
determining an image conversion voltage generated by the optical imaging data at the image sensor using a first photoelectric conversion parameter of the image sensor;
sampling the image conversion voltage based on a color filter array of the image sensor to obtain an original simulation image;
and performing noise adding simulation on the original simulation image based on noise statistical distribution to obtain a simulated noisy image.
2. The method of claim 1, wherein the input image information is color channel values of a visual image;
the determining the radiation intensity distribution of the simulated field of view light based on the spectral power distribution of the simulated field of view light and the input image information comprises:
multiplying the relative spectral power distribution of each color channel by the spectral power distribution of the light source to obtain the spectral power distribution of each color channel, wherein the spectral power distribution indicates the mapping relation between the wavelength of the light source and the total radiation intensity, and the relative spectral power distribution indicates the mapping relation between the radiation intensity of the wavelength of each color channel and the total radiation intensity;
and multiplying the spectral power distribution of each color channel with the color channel value of the visual image to obtain the radiation intensity distribution of the simulated field light.
3. The method of claim 2, further comprising:
and determining a linear RGB image obtained by carrying out inverse gamma conversion on the standard RGB image as the visual image.
4. The method of claim 1, wherein the first photoelectric conversion parameter of the image sensor comprises a photoelectric conversion efficiency of m color channels of the image sensor, m being an integer greater than 2;
the determining an image conversion voltage generated by the optical imaging data at the image sensor using a first photoelectric conversion parameter of the image sensor comprises:
and at least multiplying the radiation intensity distribution indicated by the optical imaging data and the photoelectric conversion efficiency of the c color channel of the image sensor to obtain an image conversion voltage generated by irradiating the pixel corresponding to the c color channel with light, wherein c is an integer and is more than or equal to 1 and less than or equal to m.
5. The method of claim 4, wherein the input image information comprises n image frames, n being an integer greater than 2;
the multiplying at least the radiation intensity distribution indicated by the optical imaging data and the photoelectric conversion efficiency of the c-th color channel of the image sensor to obtain the image conversion voltage generated by the light irradiating on the pixel corresponding to the c-th color channel comprises:
determining the light of the ith image frame and the ith image conversion current generated on the pixel of the c-th color channel based on the product of the radiation intensity distribution of the light of the ith image frame and the photoelectric conversion efficiency of the c-th color channel of the image sensor, wherein i is an integer and is more than or equal to 1 and less than or equal to n;
determining an ith image conversion voltage based on a product between the ith image conversion current and an exposure time of the ith image frame;
and summing the n image conversion voltages to obtain an image conversion voltage corresponding to the c color channel.
6. The method of claim 1, wherein sampling the image conversion voltage based on a color filter array of the image sensor to obtain an original simulation image comprises:
constructing an image sampling function based on a color filter array of the image sensor;
and inputting the image conversion voltage into the image sampling function to obtain an original simulation image.
7. The method of claim 1, wherein the performing noise-adding simulation on the original simulation image based on the noise statistical distribution to obtain a simulated noisy image comprises:
carrying out normalization processing on the original simulation image to obtain a normalized image of the original simulation image;
determining a noise statistical distribution of the normalized image based on a second photoelectric conversion parameter of the image sensor;
and based on the noise statistical distribution, carrying out noise adding processing on the normalized image to obtain a simulated noisy image.
8. The method of claim 7, wherein determining the noise statistical distribution of the normalized image based on the second photoelectric conversion parameter of the image sensor comprises:
determining a noise Gaussian distribution of the normalized image based on the following formula: sigma 2 ＝α*I norm +β；
Wherein α = CG AG I max /((I max -I black )*V max )；
And β = σ 2 R *AG 2 *I 2 max /((I max -I black ) 2 *V 2 max )；
Wherein σ 2 Variance of the noise Gaussian distribution; i is norm Representing the normalized image; CG and AG are the conversion gain and the analog gain of the image sensor, respectively; v max And σ R Maximum voltage swing and noise standard deviation of the image sensor respectively；I max And I black Respectively the maximum value and the minimum value after analog-digital conversion and quantization of the image sensor.
9. The method of claim 1, wherein determining optical imaging data of the simulated field of view rays after passing through an optical imaging system using imaging parameters of the optical imaging system based on the radiation intensity distribution comprises:
constructing an imaging function of an optical imaging system based on imaging parameters of the optical imaging system, wherein the imaging function represents the change of the light radiation intensity distribution before and after imaging of the optical imaging system;
and inputting the radiation intensity distribution of the simulated view field light rays into the imaging function to obtain optical imaging data of the simulated view field light rays.
10. An image simulation apparatus, comprising:
the first simulation module is used for determining the radiation intensity distribution of the simulated field light based on the spectral power distribution of the simulated field light and input image information;
the second simulation module is used for determining optical imaging data of the simulated field of view rays after passing through the optical imaging system by utilizing the imaging parameters of the optical imaging system based on the radiation intensity distribution;
a third simulation module for determining an image conversion voltage generated by the optical imaging data at the image sensor by using a first photoelectric conversion parameter of the image sensor;
the fourth simulation module is used for sampling the image conversion voltage based on the color filter array of the image sensor to obtain an original simulation image;
and the fifth simulation module is used for performing noise adding simulation on the original simulation image based on the noise statistical distribution to obtain a simulation noisy image.
11. An electronic device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus; the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the corresponding operation of the method according to any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211114952.7A CN115457157A (en) | 2022-09-14 | 2022-09-14 | Image simulation method, image simulation device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211114952.7A CN115457157A (en) | 2022-09-14 | 2022-09-14 | Image simulation method, image simulation device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115457157A true CN115457157A (en) | 2022-12-09 |
Family
ID=84303299
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211114952.7A Pending CN115457157A (en) | 2022-09-14 | 2022-09-14 | Image simulation method, image simulation device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115457157A (en) |
-
2022
- 2022-09-14 CN CN202211114952.7A patent/CN115457157A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220044363A1 (en) | Techniques for Controlled Generation of Training Data for Machine Learning Enabled Image Enhancement | |
| Wei et al. | A physics-based noise formation model for extreme low-light raw denoising | |
| CN109636754B (en) | Extremely-low-illumination image enhancement method based on generation countermeasure network | |
| US9635332B2 (en) | Saturated pixel recovery in light-field images | |
| US7038185B1 (en) | Camera for directly generating a gradient image | |
| US20170256036A1 (en) | Automatic microlens array artifact correction for light-field images | |
| US6646246B1 (en) | Method and system of noise removal for a sparsely sampled extended dynamic range image sensing device | |
| CN107071234B (en) | Lens shadow correction method and device | |
| EP2618560B1 (en) | Dark current correction for shutterless cameras | |
| US9858644B2 (en) | Bayer color filter array based high dynamic range video recording method and device | |
| KR101225056B1 (en) | Apparatus and method for reducing noise from image sensor | |
| CN112118388A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| WO2023086194A1 (en) | High dynamic range view synthesis from noisy raw images | |
| KR20190113889A (en) | Imaging Noise Reduction Systems and Methods | |
| CN108600631B (en) | Image processing method, image processing device, computer-readable storage medium and electronic equipment | |
| WO2022006556A1 (en) | Systems and methods of nonlinear image intensity transformation for denoising and low-precision image processing | |
| US10867392B2 (en) | Spatially multiplexed exposure | |
| CN115457157A (en) | Image simulation method, image simulation device and electronic equipment | |
| US11861814B2 (en) | Apparatus and method for sensing image based on event | |
| JP4110044B2 (en) | Imaging method | |
| CN114331893A (en) | Method, medium and electronic device for acquiring image noise | |
| CN114979500B (en) | Image processing method, image processing apparatus, electronic device, and readable storage medium | |
| CN114494080A (en) | Image generation method and device, electronic equipment and storage medium | |
| CN115205731A (en) | Video noise reduction device and method and terminal | |
| EP3913572A1 (en) | Loss function for image reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |