JP7497216B2 - Image processing device and method, imaging device, program, and storage medium - Google Patents

Description

The present invention relates to a technology for encoding and recording images acquired by an image sensor in which the exposure time can be controlled for each pixel.

In conventional imaging devices, raw image information (RAW data) captured by an imaging sensor is de-Bayered (demosaiced) and converted into signals consisting of luminance and color difference, and each signal is then subjected to so-called development processing, such as noise removal, optical distortion correction, and image optimization. The developed luminance and color difference signals are then typically compressed and encoded, and recorded on a recording medium.

On the other hand, there are also imaging devices that store unprocessed imaging data (RAW data) obtained by the imaging sensor immediately after development on a recording medium. When RAW data is recorded, it can be saved in a state that maintains a rich color gradation without compromising the color information from the imaging sensor, allowing for a high degree of editing freedom. However, the problem with RAW data is that the amount of recorded data is enormous, requiring a lot of free space on the recording media. For this reason, it is desirable to compress and encode the RAW data so that the amount of data recorded can be reduced.

Incidentally, as disclosed in Patent Literature 1, an imaging device that can obtain an image with a wide dynamic range in a single shot by arranging pixels with different exposure times on the same plane is known as a device for obtaining a high dynamic range image. Patent Literature 1 also discloses a synthesis method for generating a high dynamic range image in development when using this imaging device.

JP 2013-21660 A

However, the conventional technology disclosed in the above-mentioned Patent Document 1 does not disclose a method for encoding the RAW data before synthesis.

In addition, when using an imaging device such as that described in Patent Document 1, when encoding the raw data before synthesis, the level difference between pixels with different exposure times arranged on the same plane is large, resulting in the generation of many high-frequency components and reduced encoding efficiency. This poses the problem of a large amount of data when recording the raw data.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to make it possible to reduce the amount of data when encoding and recording RAW data that contains a mixture of pixel signals with different exposure times.

The image processing device of the present invention comprises a generating means for generating a plurality of RAW data from RAW data corresponding to a first exposure time and RAW data corresponding to a second exposure time longer than the first exposure time, both obtained from an image sensor capable of capturing images with different exposure times for each pixel, and an encoding means for encoding the plurality of RAW data generated by the generating means, wherein the generating means generates differential RAW data from the difference between RAW data obtained by applying a gain to RAW data corresponding to one of the first exposure time and the second exposure time and RAW data corresponding to the other of the first exposure time and the second exposure time, and the encoding means encodes the differential RAW data and the RAW data corresponding to the other of the first exposure time and the second exposure time .

According to the present invention, it is possible to reduce the amount of data when encoding and recording RAW data that contains a mixture of pixel signals with different exposure times.

1 is a block diagram showing the functional configuration of a digital camera that is a first embodiment of an image processing apparatus of the present invention. FIG. 2 is a diagram showing a pixel array of an imaging unit. FIG. 4 is a diagram showing the pixel arrangement of the imaging unit and settings of exposure time. 5A to 5C are views showing a method of separating RAW data according to the first embodiment. 5A to 5C are views showing a method of separating RAW data according to the first embodiment. 5A to 5C are views showing a method of separating RAW data according to the first embodiment. 5A to 5C are views showing a method of separating RAW data according to the first embodiment. FIG. 13 is a diagram showing RAW data output when pixels have the same exposure time. FIG. 4 is a block diagram showing the configuration of a RAW encoding unit. FIG. 13 is a diagram showing an example of frequency transformation (subband division). FIG. 13 is a diagram showing an example of a unit for generating a quantization parameter. FIG. 13 is a diagram showing an example of generation of a quantization parameter. FIG. 13 is a diagram showing an example of generation of a quantization parameter. FIG. 13 is a diagram showing an example of generation of a quantization parameter. 10A to 10C are diagrams showing a method of separating RAW data according to the second embodiment. 13A to 13C are diagrams showing a method of separating RAW data according to the third embodiment. 13A and 13B are diagrams showing pixel arrangements and exposure time settings according to the fourth embodiment. 13A and 13B are diagrams showing replacement of pixel arrays in the fourth embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
1 is a block diagram showing the functional configuration of a digital camera 100 which is a first embodiment of an image processing device of the present invention. The digital camera 100 is configured to include an image capturing unit 101, a separation unit 102, a RAW encoding unit 103, a recording processing unit 104, a recording medium 105, a memory I/F (memory interface) 106, and a memory 107.

The imaging unit 101 includes a lens optical system capable of optical zooming, including an optical lens, an aperture, a focus control unit, and a lens driving unit, and an image sensor (imaging element) in which multiple pixels having photoelectric conversion elements are arranged two-dimensionally.

The image sensor photoelectrically converts the subject image formed by the lens optical system at each pixel, then performs analog-to-digital conversion using an A/D conversion circuit, outputting a digital signal (pixel data, RAW data) for each pixel. Image sensors that can be used include CCD image sensors and CMOS image sensors.

In this embodiment, each pixel of the image sensor is provided with a color filter of R (red), G1/G2 (green), or B (blue) as shown in FIG. 2. The raw data output from the imaging unit 101 is stored in the memory 107 via the memory I/F 106.

The separation unit 102 is a circuit or module that separates the RAW data acquired by the imaging unit 101 into RAW data for each exposure time. The RAW data stored in the memory 107 is read via the memory I/F 106, separated into RAW data for each exposure time, and output to the RAW encoding unit 103.

The RAW encoding unit 103 is a circuit or module that performs calculations on the RAW data, and encodes the RAW data input from the separation unit 102. The RAW encoding unit 103 stores the encoded data generated by encoding in the memory 107 via the memory I/F 106.

The recording processing unit 104 reads various data, such as encoded data stored in the memory 107, via the memory I/F 106 and records it on the recording medium 105. The recording medium 105 is a large-capacity randomly accessible recording medium, such as a non-volatile memory.

The memory I/F 106 arbitrates memory access requests from each processing unit and controls reading and writing to the memory 107. The memory 107 is a volatile memory such as an SDRAM, and operates as a storage means. The memory 107 provides a storage area for storing various data such as the image data and audio data described above, or various data output from each processing unit constituting the digital camera 100.

Next, the pixel arrangement structure of the imaging unit 101 will be described with reference to FIG. 2. As shown in FIG. 2, the imaging unit 101 has an R pixel, a G1 pixel, a G2 pixel, and a B pixel arranged in 2×2 pixel units, and is characterized in that the same color is arranged in 2×2 pixel units. The imaging unit 101 has a structure in which a total of 4×4 pixels is the minimum unit, and this minimum unit is repeatedly arranged.

The setting of the exposure time in an image sensor having the pixel array structure shown in FIG. 2 and capable of controlling the exposure time for each pixel (capturing images with different exposure times for each pixel) will be described with reference to FIG. 3. As shown in FIG. 3, the horizontal direction is x and the vertical direction is y, with the column number represented by the x coordinate and the row number represented by the y coordinate. Numbers in parentheses indicate the coordinates indicating the position of each pixel on the image sensor. White pixels represent short-time exposure pixels and gray pixels represent long-time exposure pixels. In this embodiment, short-time exposure pixels that perform short-time exposure and long-time exposure pixels that perform long-time exposure are set in a zigzag pattern in the column direction as shown in FIG. 3.

For example, the exposure times of the four R pixels at the top left of FIG. 3 are set as follows: R(1,1) is a short-exposure pixel, R(2,1) is a long-exposure pixel, R(1,2) is a long-exposure pixel, and R(2,2) is a short-exposure pixel. In this way, short-exposure pixels and long-exposure pixels are set alternately in each column, and short-exposure pixels and long-exposure pixels are set alternately in each row. If we trace only the short-exposure pixels in the y direction in the first and second columns, from the top, the first column is a short-exposure pixel in the first row, the second column is a short-exposure pixel in the second row, the first column is a short-exposure pixel in the third row, and the second column is a short-exposure pixel in the fourth row. Similarly, if we trace only the long-exposure pixels in the y direction in the first and second columns, from the top, the second column is a long-exposure pixel in the first row, the first column is a long-exposure pixel in the second row, the second column is a long-exposure pixel in the third row, and the first column is a long-exposure pixel in the fourth row.

As described above, the pixel array structure and its exposure time settings are such that pixels of the same color are set in 2 x 2 pixel units, and among these four pixels, two short-exposure pixels (one of the two types of exposure times) and two long-exposure pixels (the other of the two types of exposure times) are arranged.

If the RAW data is to be encoded while it is acquired by the imaging unit 101, that is, while pixels with different exposure times are mixed, the level difference between the pixels with different exposure times is large, resulting in a large amount of high-frequency components and a large amount of recorded RAW data. Therefore, in this embodiment, the separation unit 102 separates the RAW data for each exposure time, matches the levels between the pixels, and suppresses the high-frequency components, thereby reducing the amount of recorded RAW data.

Next, the separation method will be described with reference to Figs. 4A to 4D. As shown in Figs. 4A to 4D, the separation unit 102 separates the RAW data input from the imaging unit 101 into RAW data in a Bayer array structure composed only of short-time exposure pixels, and RAW data in a Bayer array structure composed only of long-time exposure pixels, and outputs the data to the RAW encoding unit 103.

Specifically, RAW data consisting only of short-exposure pixels is separated into two planes of short-exposure RAW data, shown as RAW data 401a in FIG. 4A and RAW data 401b in FIG. 4B. RAW data 401a is short-exposure RAW data consisting of short-exposure pixels surrounded by diamond shapes in odd rows and odd columns, as shown in FIG. 4A. RAW data 401b is short-exposure RAW data consisting of short-exposure pixels surrounded by diamond shapes in even rows and even columns, as shown in FIG. 4B.

In the same manner, the RAW data consisting only of long exposure pixels is separated into two planes of long exposure RAW data shown as RAW data 401c in FIG. 4C and RAW data 401d in FIG. 4D. RAW data 401c is long exposure RAW data consisting of long exposure pixels surrounded by a diamond shape in odd rows and even columns as shown in FIG. 4C. RAW data 401d is long exposure RAW data consisting of long exposure pixels surrounded by a diamond shape in even rows and odd columns as shown in FIG. 4D. The RAW encoding unit 103 encodes the RAW data 401a, 401b, 401c, and 401d input from the separation unit 102 in a Bayer array.

Up to this point, we have explained the separation method of the separation unit 102 when the exposure times of pixels arranged on the same plane are different using the pixel array in Figure 2. Next, we will explain the processing of the separation unit 102 when the exposure times of all pixels are the same, with reference to Figure 5.

In this case, the separation unit 102 calculates the pixel average value for every four pixels of the same color component surrounded by a gray diamond as shown in FIG. 5 for the RAW data acquired by the imaging unit 101, constructs RAW data 501, and outputs it to the RAW encoding unit 103. Specifically, separation is performed by calculating the arithmetic average for each same color component as shown in the following Equations 1 to 4.

Next, the detailed configuration and processing flow of the RAW encoding unit 103 that processes the short-exposure RAW data 401a and 401b and the long-exposure RAW data 401c and 401d will be described with reference to the block diagram shown in FIG. 6.

The RAW encoding unit 103 is configured with a channel conversion unit 601, a frequency conversion unit 602, a quantization parameter generation unit 603, a quantization unit 604, an encoding unit 605, and a quantization parameter encoding unit 606.

The channel conversion unit 601 converts the Bayer array RAW data input from the separation unit 102 into multiple channels. For example, it converts each of the R, G1, G2, and B in the Bayer array into four channels. Alternatively, it further converts R, G1, G2, and B into four channels using the following conversion formulas 5 to 8.

Y = (R + G1 + G2 + B) / 4 Equation 5
C0 = R - B Equation 6
C1 = (G0 + G1) / 2 - (R + B) / 2 Equation 7
C2 = G0 - G1 Equation 8
Although a configuration example in which conversion to four channels is shown here, the number of channels and the conversion method may be other than those described above.

The frequency transform unit 602 performs frequency transform processing using a discrete wavelet transform at a predetermined decomposition level (hereafter referred to as lev) on a channel-by-channel basis, and outputs the generated subband data (transform coefficients) to the quantization parameter generation unit 603 and the quantization unit 604.

Figure 7(a) shows the configuration of a filter bank for realizing the discrete wavelet transform associated with the subband decomposition process with lev=1. When the discrete wavelet transform process is performed in the horizontal and vertical directions, the signal is decomposed into one low-frequency subband (LL) and three high-frequency subbands (HL, LH, HH) as shown in Figure 7(b).

The transfer functions of the low-pass filter (hereafter referred to as lpf) and high-pass filter (hereafter referred to as hpf) shown in FIG. 7(a) are shown in Equation 9 and Equation 10, respectively.

lpf(z)=(-z ^-2 +2z ^- 1+6+ ^2z1 - ^z2 )/8 Equation 9
hpf(z)=(-z ^-1 +2- ^z1 )/2 Equation 10
When lev is greater than 1, hierarchical subband decomposition is performed on the low frequency subband (LL). Note that, although the discrete wavelet transform is configured with a 5-tap lpf and a 3-tap hpf as shown in Equations 9 and 10, a filter configuration with a different number of taps and different coefficients may be used.

The quantization parameter generation unit 603 generates a quantization parameter for performing a quantization process for each predetermined subband data unit on the subband data (transformation coefficients) generated by the frequency transformation unit 602. The generated quantization parameter is input to the quantization parameter coding unit 606 and is also supplied to the quantization unit 604.

The quantization unit 604 performs quantization processing on the subband data (transformation coefficients) output from the frequency conversion unit 602 based on the quantization parameters supplied from the quantization parameter generation unit 603, and outputs the quantized subband data (transformation coefficients) to the encoding unit 605.

The encoding unit 605 performs predictive differential entropy encoding for each subband in raster scan order on the quantized subband data (transform coefficients) output from the quantization unit 604, and stores the generated encoded RAW data in the memory 107. Note that the prediction method and entropy encoding method may be other methods.

The quantization parameter encoding unit 606 is a processing unit that encodes the quantization parameters input from the quantization parameter generation unit 603. The quantization parameter is encoded using a common encoding method with the encoding unit 605, and the generated encoded quantization parameters are stored in the memory 107.

Next, the relationship between the subband data, channel data, and RAW data when generating quantization parameters with the above-mentioned specified subband unit being 4x4 will be explained with reference to Figure 8.

As shown in FIG. 8, the 4x4 subbands correspond to 8x8 pixels in each channel, and also correspond to blocks of 16x16 pixels in each RAW data. Therefore, in this case, it is necessary to store the quantization parameters in memory 107 for each RAW data block of 16x16 pixels in the short-exposure RAW data 401a, 401b and the long-exposure RAW data 401c, 401d.

In order to reduce the amount of data for the quantization parameters, it is effective to use a common quantization parameter for the short-exposure RAW data 401a, 401b and the long-exposure RAW data 401c, 401d. In this case, the amount of data can be reduced to 1/2. In this embodiment, in order to further reduce the amount of data, the quantization parameter generated with the exposure time closer to the correct exposure is used as a reference to calculate the other quantization parameter. This makes it possible to reduce the amount of data for the quantization parameters to 1/4. The reason why the exposure closer to the correct exposure is used as a reference here is that in overexposed and underexposed images in which highlight blowout and shadow crush occur, a quantization parameter that accurately matches the characteristics of the subject cannot be generated.

As a specific example, if the short exposure is considered to be closer to the correct exposure, the formula for calculating the quantization parameter of the long exposure RAW data is shown in Equation 11, based on the quantization parameter generated from the short exposure RAW data.

L_Qp = α × S_Qp + β Equation 11
here,
L_Qp: quantization parameter of long exposure RAW data, S_Qp: quantization parameter of short exposure RAW data, α: slope, and β: intercept.

In this embodiment, the quantization parameter for the long exposure RAW data is calculated based on the quantization parameter generated from the short exposure RAW data. However, the quantization parameter for the short exposure RAW data may be generated based on the quantization parameter generated from the long exposure RAW data. Also, the quantization parameter may be calculated by setting α and β for each of the short exposure and long exposure without using either the short exposure or long exposure as the reference.

Next, a method for determining α and β shown in Equation 11 will be described. Although α and β may be any value, in this embodiment, a detailed method for determining the parameters will be described. As in the above example, if the short exposure is closer to the correct exposure, the long exposure will be overexposed because the exposure time is longer than that of the short exposure. Therefore, in the area where the brightness is medium to bright in the short exposure, the pixel value reaches a saturation level in the long exposure, and it is highly likely that the pixel value output according to the brightness of the subject cannot be performed. On the other hand, it is possible to obtain more detailed information than in the short exposure in the dark area. Therefore, the quantization parameter of the long exposure RAW data is made larger in the area where the brightness is determined to be medium to bright in the short exposure RAW data compared to the short exposure. In addition, by making the quantization parameter the same for the area determined to be dark, it is possible to reduce the data amount of the quantization parameter while maintaining image quality.

Specifically, the following will be described with reference to Figures 9A to 9C. Figure 9A shows an example of setting quantization parameters in short-exposure RAW data according to the brightness of the short-exposure RAW data. Also, Figure 9B shows an example of setting quantization parameters in long-exposure RAW data according to the brightness of the short-exposure RAW data. Note that the brightness index may be evaluated using the 1LL subband, which corresponds to the quantization parameter generation unit described above. The magnitude relationship of each quantization parameter is shown in Equations 12 to 14.

Q0<Q1<Q2 Equation 12
Q1 < Q3 Equation 13
Q2<Q4 Equation 14
First, the quantization parameter in the short-exposure RAW data is set so that the darker the area, the smaller the quantization parameter (Q0<Q1<Q2) in consideration of visual characteristics. On the other hand, in the long-exposure RAW data, the quantization parameter is set so that the area corresponding to the dark area of the short-exposure RAW data has the same Q0 as the short-exposure RAW data, and the area corresponding to the medium to bright area has a larger quantization parameter than the short-exposure RAW data (Q1<Q3, Q2<Q4).

Figure 9C shows a graph for calculating the quantization parameter of long exposure RAW data based on the quantization parameter generated from the short exposure RAW data. The horizontal axis represents the quantization parameter of the short exposure RAW data (S_Qp), and the vertical axis represents the quantization parameter of the long exposure RAW data (L_Qp). α and β shown in Equation 11 can be set to satisfy the relationship shown in Equations 12 to 14.

Note that α and β are stored in memory 107, just like the encoded data, and are recorded together with the encoded data on recording medium 105 via memory I/F 106. Also, a flag indicating whether the reference quantization parameter is for short exposure or long exposure is stored in memory 107, and is recorded together with the encoded data on recording medium 105 via memory I/F 106. However, if neither short exposure nor long exposure is used as the reference, and α and β are provided for each exposure time, then there is no need to have a flag.

Also, to handle both of the above cases, first have a flag indicating whether or not there is a reference exposure time, and then, if there is a reference exposure time, have a flag indicating whether the reference is a short exposure or a long exposure. In this case, too, each flag information is stored in memory 107 and recorded on recording medium 105 together with the encoded data via memory I/F 106.

As described above, in this embodiment, the separation unit 102 separates the RAW data for each exposure time, eliminating the level difference between the pixels to be encoded, and suppressing high frequency components, making it possible to reduce the amount of recorded RAW data. In addition, by using the quantization parameter calculated for one piece of RAW data as a reference to determine the quantization parameter for the other piece of RAW data with a different exposure time, it is possible to reduce the amount of recorded RAW data.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the second embodiment, the method of separating RAW data in the separation unit 102 is different from that in the first embodiment. Note that the configuration of the digital camera in the second embodiment is the same as that in the first embodiment, so a duplicated description will be omitted and only the differences will be described.

In the first embodiment, the separation unit 102 outputs the raw data separated into pixels with the same exposure time, specifically, two planes of raw data consisting of only short-exposure pixels and two planes of raw data consisting of only long-exposure pixels, to the raw encoding unit 103.

In contrast, in the second embodiment, in order to further reduce the amount of data, the separation unit 102 adds pixel values of nearby pixels with the same exposure time and color component, calculates the average pixel value, and outputs it to the RAW encoding unit 103.

The processing of the separation unit 102 in this embodiment will be described with reference to FIG. 10. For RAW data input from the imaging unit 101 containing a mixture of pixels with different exposure times, the separation unit 102 calculates the arithmetic average of the pixel values of the pixels enclosed in the rectangle shown in FIG. 10(a), i.e., short-exposure pixels and pixel values of the same color component, and separates the data into short-exposure RAW data 1001a. Specifically, the separation is performed by calculating the arithmetic average for each same color component, as shown in the following Equations 15 to 18.

Similarly, pixels surrounded by a rectangle in FIG. 10(b) that are long-exposure pixels and have the same color component are averaged and separated into long-exposure RAW data 1001b. Specifically, the average of each same color component is calculated and separated as shown in the following Equations 19 to 22.

As described above, in the second embodiment, the RAW data acquired by the imaging unit 101 is averaged and separated by the separation unit 102, making it possible to reduce the amount of data output to the RAW encoding unit 103 to half that of the first embodiment.

Third Embodiment
Next, a third embodiment of the present invention will be described. In the third embodiment, the method of separating RAW data in the separation unit 102 is different from that of the first and second embodiments. Note that the configuration of the digital camera of this embodiment is the same as that of the first and second embodiments, so a duplicated description will be omitted and only the differences will be described.

In the second embodiment, the separation unit 102 calculates the average of adjacent pixels with the same exposure time and the same color component, and outputs it to the RAW encoding unit 103. In the third embodiment, in order to further reduce the amount of data compared to the second embodiment, a gain is multiplied to the RAW data of one exposure time to match the other exposure time, and the difference value is output to the RAW encoding unit 103. In other words, the RAW encoding unit 103 encodes the RAW data of the average for one exposure time, and encodes the RAW data of the difference value (difference RAW data) for the other exposure time.

The processing in the separation unit 102 in this embodiment will be described with reference to FIG. 11. First, as in the second embodiment, the separation unit 102 adds nearby pixels with the same exposure time and the same color component, calculates the average, and obtains RAW data 1001a and 1001b, as shown in FIG. 10(a) and FIG. 10(b). Next, the first row of the long-exposure RAW data 1001b and the first row of the short-exposure RAW data 1001a are multiplied by a gain γ corresponding to the long-exposure RAW data, and an offset ε is added to obtain a difference value. At this time, the gain γ and the offset ε may be determined in advance by calculating backward from the exposure time, or may be determined using a histogram of the acquired short-exposure pixels and long-exposure pixels.

Specifically, the difference in the first row is calculated as shown in Equations 23 to 26 below.

This is performed not only on the first row, but also on the second, third, and fourth rows, and the calculated difference value is output to the RAW encoding unit 103. Note that in this embodiment, correction is performed on the short-exposure RAW data, but correction may also be performed on the long-exposure RAW data. However, from the perspective of rounding processing, the accuracy of the difference value is improved by multiplying the short-exposure RAW data by gain γ to calculate the difference.

As described above, in the third embodiment, the RAW data is not output directly to the RAW encoding unit 103, but is instead output as a difference value, making it possible to further reduce the amount of recorded RAW data compared to the second embodiment.

Fourth Embodiment
Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, a pixel arrangement different from those in the first to third embodiments, specifically, a pixel arrangement shown in FIG.

In the first to third embodiments described above, the imaging unit 101 has a structure in which the minimum unit is a 4×4 array of 16 pixels, each of which is made up of four different pixels R, G1, G2, and B as shown in FIG. 2, and this minimum unit is repeatedly arranged.

In contrast, FIG. 12 shows the pixel array and exposure time settings of the imaging unit 101 in the fourth embodiment. With the horizontal direction as x and the vertical direction as y, the column number is represented by the x coordinate and the row number is represented by the y coordinate. Numbers in parentheses indicate the coordinates of the position of each pixel on the image sensor. White pixels represent short-exposure pixels, and gray pixels represent long-exposure pixels. In this way, in FIG. 12, short-exposure pixels and long-exposure pixels are set alternately in two-column units in a pixel array of a Bayer array consisting of an R, G1, G2, and B array.

Even with the pixel array and exposure time settings of FIG. 12, the processing described in the first to third embodiments can be performed by replacing the pixel array structure shown in FIG. 2 as shown in FIG. 13.

As described above, in the fourth embodiment, even if the pixel arrangement is changed, it is possible to perform the same processing as described in the first to third embodiments.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

101: imaging unit, 102: separation unit, 103: RAW encoding unit, 104: recording processing unit, 105: recording medium, 106: memory interface, 107: memory

Claims (16)

a generating means for generating a plurality of RAW data from RAW data corresponding to a first exposure time and RAW data corresponding to a second exposure time longer than the first exposure time, the RAW data being obtained from an image sensor capable of capturing images with different exposure times for each pixel;
An encoding means for encoding the plurality of RAW data generated by the generating means;
Equipped with
the generating means generates differential RAW data from a difference between RAW data obtained by applying a gain to RAW data corresponding to one of the first exposure time and the second exposure time and RAW data corresponding to the other of the first exposure time and the second exposure time;
The image processing apparatus according to claim 1, wherein the encoding means encodes the difference RAW data and RAW data corresponding to the other of the first exposure time and the second exposure time . The image processing device according to claim 1, characterized in that the generating means generates RAW data in a Bayer array structure for each exposure time. the generating means generates differential RAW data from a difference between RAW data obtained by applying a gain to the RAW data corresponding to the first exposure time and the RAW data corresponding to the second exposure time;
3. The image processing apparatus according to claim 1, wherein the encoding means encodes the differential raw data and the raw data corresponding to the second exposure time. 2. The image processing device according to claim 1, wherein the generating means generates one piece of RAW data corresponding to the first exposure time by adding together multiple pixel data of the same color component for the first exposure time, generates one piece of RAW data corresponding to the second exposure time by adding together multiple pixel data of the same color component for the second exposure time, and generates differential RAW data from a difference between RAW data obtained by applying a gain to the generated piece of RAW data corresponding to one of the first exposure time and the second exposure time and the generated piece of RAW data corresponding to the other of the first exposure time and the second exposure time . 5. The image processing device according to claim 4, wherein the generating means generates one piece of RAW data corresponding to the first exposure time and one piece of RAW data corresponding to the second exposure time by calculating an averaging of a plurality of pixel data. 2. The image processing apparatus according to claim 1, wherein the gain is determined in accordance with an exposure time. 2 . The image processing apparatus according to claim 1 , wherein the gain is determined in accordance with a histogram of pixels having the first exposure time and a histogram of pixels having the second exposure time. 2. The image processing device according to claim 1, wherein the generating means generates differential RAW data from a difference between RAW data obtained by multiplying RAW data corresponding to one of the first exposure time and the second exposure time by a gain and adding a predetermined offset, and RAW data corresponding to the other of the first exposure time and the second exposure time. a control means for controlling an exposure time for each pixel of the image sensor;
9. The image processing device according to claim 1, wherein the generating means generates the plurality of RAW data when the exposure times for the pixels are different, and generates a single RAW data when the exposure times for the pixels are the same. 10. The image processing apparatus according to claim 9 , wherein the generating means generates one piece of RAW data by calculating an average value of image data of the same color in the vicinity when the exposure time for each pixel is the same. 2. The image processing device according to claim 1, wherein the encoding means determines a quantization parameter of the RAW data corresponding to one of the first exposure time and the second exposure time based on a quantization parameter of the RAW data corresponding to the other of the first exposure time and the second exposure time. 12. The image processing apparatus according to claim 11, wherein the encoding means determines a quantization parameter for the RAW data corresponding to the second exposure time based on a quantization parameter for the RAW data corresponding to the first exposure time. An image sensor capable of controlling the exposure time for each pixel;
An image processing device according to any one of claims 1 to 12 ,
An imaging device comprising: a generating step of generating a plurality of pieces of RAW data from RAW data corresponding to a first exposure time and RAW data corresponding to a second exposure time longer than the first exposure time, the RAW data being obtained from an image sensor capable of capturing images with different exposure times for each pixel;
an encoding step of encoding the plurality of RAW data generated in the generating step;
having
In the generating step, differential RAW data is generated from a difference between RAW data obtained by applying a gain to RAW data corresponding to one of the first exposure time and the second exposure time, and RAW data corresponding to the other of the first exposure time and the second exposure time;
An image processing method , comprising: encoding the differential RAW data and RAW data corresponding to the other of the first exposure time and the second exposure time, in the encoding step . A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 12 . 13. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

Images