JP2013162347A - Image processor, image processing method, program, and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a wide dynamic range.SOLUTION: A saturated pixel detector detects a saturated pixel from an image obtained via an imaging element in which plural elements different in exposure time are arrayed regularly. A surrounding pixel identifier identifies surrounding pixels positioned near the saturated pixels and including pixels of elements different from the elements which the saturated pixel belongs to. A prediction arithmetic unit performs prediction arithmetic processing of the image on the basis of the identified surrounding pixels. This technique can be applied to an image processor, for example.

Description

  The present technology relates to an image processing apparatus, an image processing method, a program, and an apparatus, and more particularly, to an image processing apparatus, an image processing method, a program, and an apparatus that can realize a wide dynamic range.

  In general image sensor shooting, all pixels are shot with the same exposure time. However, in such a photographing method, it is difficult to photograph in such a manner that a dark subject and a bright subject coexist as an imaging range such as indoors and outside a window.

  For example, consider a case where a subject including both a room and the outside of a window as shown in FIG. 1B is photographed by a Bayer array image sensor shown in FIG. 1A. The Bayer arrangement is an arrangement method in which a total of four pixels of 2 × 2 (vertical direction × horizontal direction) composed of two R pixels and G pixels and B pixels are repeatedly arranged in the vertical direction and the horizontal direction. In the figure, the R pixel indicated by “R” is a pixel that outputs a red pixel signal, the G pixel indicated by “G” is a pixel that outputs a green pixel signal, and is indicated by “B”. The B pixel is a pixel that outputs a blue pixel signal.

  In order to shoot in accordance with a dark subject in the room, it is necessary to set a long exposure time for each pixel as shown in FIG. 2A. In FIG. 2A, “LR” indicates an R pixel with a long exposure time set, “LG” indicates a G pixel with a long exposure time set, and “LB” indicates a B pixel with a long exposure time set. . When photographing is performed under such conditions, the photographed image is in a so-called “overexposed” state, as shown in FIG.

  On the other hand, in order to shoot in accordance with a bright subject outside the window, it is necessary to set the exposure time of each pixel short as shown in FIG. 3A. In FIG. 3A, “SR” indicates an R pixel for which a short exposure time is set, “SG” indicates a G pixel for which a short exposure time is set, and “SB” indicates a B pixel for which a short exposure time is set. . When shooting is performed under such conditions, the shot image does not represent a dark subject portion and is in a so-called “blackout” state, as shown in FIG. 3B.

  Therefore, it is not possible to obtain an image that expresses both a dark subject portion and a bright subject portion as shown in FIG.

  Therefore, there are a method for preparing a plurality of images taken with different exposure times, a method for combining the images, and a method for realizing a wide dynamic range by interpolating saturated pixels using unsaturated pixels. It has been proposed (see, for example, Patent Documents 1 and 2).

JP 2009-201094 A JP 2006-333313 A

  However, there is no technique for interpolating saturated pixels, and research for realizing a wide dynamic range is underway.

  The present technology has been made in view of such a situation, and makes it possible to realize a wide dynamic range.

  An image processing apparatus according to an aspect of the present technology includes a saturation pixel detection unit that detects a saturation pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged, and the saturation A surrounding pixel specifying unit for specifying a surrounding pixel including a pixel of an element group different from the element group to which the saturated pixel belongs, and a prediction calculation for performing an image prediction calculation process based on the surrounding pixel A part.

  According to an image processing method of one aspect of the present technology, an image processing apparatus detects a saturated pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged, and the saturation A step of specifying a surrounding pixel including a pixel of an element group different from the element group to which the saturated pixel belongs, and performing an image prediction calculation process based on the surrounding pixel.

  A program according to one aspect of the present technology includes a saturation pixel detection unit that detects a saturation pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged; A surrounding pixel specifying unit for specifying a surrounding pixel including a pixel of an element group different from the element group to which the saturated pixel belongs, and a prediction that performs an image prediction calculation process based on the surrounding pixel It is for functioning as an arithmetic unit.

  An apparatus according to one aspect of the present technology detects a saturated pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged, is located in the vicinity of the saturated pixel, and A processor capable of specifying a surrounding pixel including a pixel of an element group different from the element group to which the saturated pixel belongs, and executing an instruction to perform an image prediction calculation process based on the surrounding pixel, and the instruction executable by a computer And a memory for recording.

  In one aspect of the present technology, a saturated pixel is detected from an image acquired via an image sensor in which a plurality of element groups having different exposure times are regularly arranged, and the saturation pixel is located near the saturated pixel. A surrounding pixel including a pixel of an element group different from the element group to which the pixel belongs is specified, and an image prediction calculation process is performed based on the surrounding pixel.

  The program can be provided by being transmitted through a transmission medium or by being recorded on a recording medium.

  The image processing apparatus may be an independent apparatus, or may be an internal block constituting one apparatus.

  According to one aspect of the present technology, a wide dynamic range can be realized.

It is a figure explaining the conventional imaging | photography. It is a figure explaining the conventional imaging | photography which lengthened exposure time. It is a figure explaining the conventional imaging | photography which shortened exposure time. It is a figure explaining the example of an ideal image to obtain | require by imaging | photography. It is a block diagram of a 1st embodiment of an imaging device to which this art is applied. It is a figure which shows the example of the image sensor of FIG. It is a figure which shows the relationship between exposure time and a signal level. It is a figure explaining the process of a gain adjustment circuit. It is a block diagram which shows the 1st structure of a demosaic processing circuit. It is a figure which shows the example of a class tap and a prediction tap. It is a figure which shows the example of a saturation pixel. It is a figure which shows the example of a class tap in case a saturated pixel exists, and a prediction tap. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a figure explaining the process of a demosaic processing circuit. It is a flowchart explaining the demosaic image generation process by an imaging device. It is a block diagram which shows the structural example of a learning apparatus. It is a block diagram which shows the structural example of the image signal generation part for learning. It is a flowchart explaining a learning process. It is a block diagram which shows the 2nd structure of a demosaic processing circuit. It is a figure which shows the example of the class tap in the 2nd structure of a demosaic processing circuit, and a prediction tap. It is a flowchart explaining the demosaic image generation process in 2nd Embodiment. And FIG. 18 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. 1st Embodiment (configuration example which excludes long livestock pixels)
2. Second embodiment (configuration example in which only saturated pixels are excluded)

<1. First Embodiment>
[Configuration example of imaging device]
FIG. 5 is a block diagram illustrating a configuration example of the first embodiment of the imaging apparatus to which the present technology is applied.

  5 includes a lens 11, an image sensor 12, a gain adjustment circuit 13, a demosaic processing circuit 14, a compression processing circuit 15, a recording circuit 16, and a display device 17. The imaging device 1 receives light from a subject (subject light), records an image signal corresponding to the subject on a recording medium, and displays an image obtained from the image signal. The imaging apparatus 1 can be applied to, for example, a video camera, a digital still camera, a mobile phone, and the like.

  The lens 11 condenses subject light on the image sensor 12. The image sensor 12 is composed of an image sensor such as a CCD or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor 12 receives subject light and converts an analog electrical signal corresponding to the amount of received light into A / D (Analog to Digital). Then, it is supplied to the gain adjustment circuit 13 as a digital image signal.

  The color array of the image sensor 12 is the Bayer array shown in FIG. 1, but the exposure time of each pixel is not the same, and the pixel that receives light with a long exposure time (hereinafter referred to as a long pixel) is shorter. It is composed of a combination of pixels that receive light during the exposure time (hereinafter referred to as short-lived pixels). Here, it is assumed that the ratio of the exposure time of the long pixel and the short pixel of the image sensor 12 is N: 1 (N is an integer greater than 1).

  FIG. 6 shows a part of the pixels of the image sensor 12.

  In FIG. 6, the difference in the exposure time of each pixel of the image sensor 12 is expressed in the same manner as in FIGS. That is, in the imaging device 1 image sensor 12, the R pixel and the B pixel are set as short pixels. The G pixel is set to either a short pixel or a long pixel depending on its position. In other words, among the 4 pixels of 2 × 2 regularly arranged, the R pixel and the B pixel are assigned to the short pixel (SR pixel, SB pixel), and the two G pixels are long One pixel (LG pixel) and one short pixel (SG pixel) are allocated.

  FIG. 7 shows the relationship between the pixel exposure time and the signal level (pixel value).

  The signal level and the exposure time are in a proportional relationship. For example, if the exposure time is doubled, the signal level is also doubled. However, if the pixel (element) of the image sensor 12 has an 8-bit output, the signal level (pixel value) is saturated at an 8-bit maximum value, that is, 255.

  Here, for example, the ratio of the exposure time of the long livestock pixel to the short livestock pixel is 10: 1, and the received light quantity is such that the signal level of the subject light incident on the image sensor 12 is 1000 in 10-bit count. Suppose there was. In this case, the pixel value of the long lived pixel is saturated at 255, but the pixel value of the short lived pixel becomes 100 because the exposure time is short, and falls within the 8-bit range. Then, the true pixel value (= 1000) of the subject can be obtained by multiplying 100, which is the pixel value of the short pixel, by 10 according to the exposure ratio. Therefore, a wide dynamic range can be realized by providing short pixels in the image sensor 12.

  However, regarding noise, because of the random nature of noise, there is no proportional relationship between the signal level and the exposure time, and the longer the exposure time, the more advantageous. In other words, with respect to noise, short livestock pixels are disadvantageous compared to long livestock pixels. Generally, when the exposure time is doubled, the noise amount becomes 1 / √2. Therefore, all of the image sensors 12 cannot be short pixels.

  For the above reasons, the image sensor 12 of the imaging apparatus 1 employs a configuration in which a plurality of element groups (short pixel groups and long pixel groups) having different exposure times are regularly arranged.

  In the present embodiment, a part of the R pixel, B pixel, and G pixel is a short pixel, and the remaining part of the G pixel is a long pixel. Some of the pixels may be long livestock pixels, and the remaining part of the G pixels may be short livestock pixels. That is, it is possible to appropriately determine which pixels are set as short livestock pixels and long livestock pixels. Also, the color arrangement is not limited to the Bayer arrangement, and other RGB arrangements, an arrangement in which white pixels for improving sensitivity are added to R, G, and B pixels may be used. That is, the selection and combination of colors and the arrangement method thereof are arbitrary.

  Returning to FIG. 5, the gain adjustment circuit 13 multiplies the pixel value (pixel signal) of the short-lived pixel by a gain N times that is the ratio of the exposure time to match the signal levels of the short-lived pixel and the long-lived pixel. .

  FIG. 8 is a diagram conceptually showing gain processing by the gain adjustment circuit 13. The thick SR, SG, and SB pixels shown on the left side of FIG. 8 indicate a state before gain processing by the gain adjustment circuit 13, and the SR, SG, And SB pixel has shown the state after the gain process adjusted to the signal level of the long livestock pixel. Similarly in FIG. 9 and subsequent figures, the SR, SG, and SB pixels in the narrow frame indicate pixel values after gain processing.

  Returning to FIG. 5, the demosaic processing circuit 14 uses all the R, G, and B colors for each pixel based on the pixel signal of any one of the R, G, and B color components supplied from the gain adjustment circuit 13. A demosaic process for complementing the color component is executed so as to have the pixel signal of the component.

  Here, the demosaic processing circuit 14 saturates a pixel whose signal level is saturated (hereinafter also referred to as a saturated pixel) among the pixels constituting the image supplied from the gain adjustment circuit 13. The demosaic process is executed without using pixels. In other words, the demosaic processing circuit 14 is a pixel that is not saturated (hereinafter also referred to as a non-saturated pixel) when a saturated pixel exists in the pixels that constitute the image supplied from the gain adjustment circuit 13. A pixel value of a saturated pixel is predicted from the pixel value and output.

  In this embodiment, it is assumed that the saturation of the pixel value occurs only in the long livestock pixel and does not saturate in the short livestock pixel. In other words, in the image sensor 12, the exposure ratio N between the long livestock pixel and the short livestock pixel is determined so that the short livestock pixel is not saturated.

  The demosaic processing circuit 14 supplies the image signal after the demosaic processing to the compression processing circuit 15 and the display device 17.

  The compression processing circuit 15 compresses and encodes an image signal at a specified compression ratio in accordance with a predetermined format such as JPEG (Joint Photographic Experts Group), and supplies the image signal to the recording circuit 16. The compression processing circuit 15 may also perform resolution conversion processing of an image signal that reduces or enlarges the size of the captured image.

  The recording circuit 16 records (stores) the compression-encoded image signal in a semiconductor memory as a recording medium. The recording medium for recording the image signal may be a semiconductor memory, an optical disc such as a DVD (Digital Versatile Disc), a hard disk, or the like.

  The display device 17 is configured by an LCD (Liquid Crystal display) or the like, and displays an image corresponding to the image signal supplied from the demosaic processing circuit 14.

  Therefore, according to the imaging device 1 of FIG. 5, the image sensor 12 receives the subject light and outputs an image signal, and the demosaic processing circuit 14 performs the demosaic processing on the image signal as a result of the processing. The image signal is output to the subsequent compression processing circuit 15.

[Detailed configuration example of demosaic processing circuit]
FIG. 9 is a block diagram showing a detailed configuration of the demosaic processing circuit 14 of FIG.

  The demosaic processing circuit 14 includes a saturated pixel presence determination unit 21, a prediction tap extraction unit 22, a class classification unit 23, a coefficient generation unit 24, and a prediction calculation unit 25. The class classification unit 23 includes a class tap extraction unit 31, The class code generating unit 32 is configured.

  The image signal output from the gain adjustment circuit 13 (FIG. 5) and adjusted in gain for the short pixels is sent to the saturated pixel presence determination unit 21, the prediction tap extraction unit 22, and the class tap extraction unit 31. Supplied.

  In the following description, the image signal output from the gain adjustment circuit 13 after the gain adjustment is performed on the short pixel is referred to as a mosaic image, and the image after demosaic processing output from the demosaic processing circuit 14 is referred to as a demosaic image. Say.

  The demosaic processing circuit 14 obtains and outputs a demosaic image from the mosaic image by a prediction calculation using the class classification adaptation process. In the class classification adaptive processing, each pixel of the demosaic image obtained by the prediction calculation is sequentially set as the target pixel, and the processing for obtaining the image signal of the target pixel is executed for all the pixels.

  Specifically, the demosaic processing circuit 14 extracts a plurality of pixels as a prediction tap with respect to the target pixel, and learns using the pixel value of the prediction tap, a teacher image, and a student image (described later in FIG. 26 and the like). A demosaic image is generated by a predetermined prediction calculation using a prediction coefficient obtained in advance.

  The saturated pixel presence determination unit 21 detects a saturated pixel and outputs a detection result. Specifically, the saturated pixel presence determination unit 21 determines whether or not a saturated pixel exists as a plurality of pixels extracted as prediction taps by the prediction tap extraction unit 22 and a class tap by the class tap extraction unit 31. Each of the plurality of extracted pixels is determined. Then, when it is determined that a saturated pixel exists in any of the plurality of pixels extracted as the prediction tap, the saturated pixel presence determination unit 21 supplies a detection signal indicating the determination result to the prediction tap extraction unit 22. . When it is determined that a saturated pixel exists in any of the plurality of pixels extracted as the class tap, the saturated pixel presence determination unit 21 supplies a detection signal indicating the determination result to the class tap extraction unit 31. .

  The prediction tap extraction unit 22 extracts, as a prediction tap, a pixel (a pixel value) constituting the mosaic image used for predicting the pixel value of the target pixel of the demosaic image.

  Specifically, the prediction tap extraction unit 22 is spatially close to the pixel of the mosaic image corresponding to the target pixel (for example, the pixel of the mosaic image that is spatially closest to the target pixel). A plurality of pixels of the mosaic image at the position (near) are extracted as prediction taps.

  Moreover, when the determination result that the saturated pixel exists is supplied from the saturated pixel presence determination unit 21, the prediction tap extraction unit 22 selects the long livestock pixel from among the plurality of pixels of the mosaic image to be extracted as the prediction tap. Only the removed short pixels are extracted as prediction taps.

  The class classification unit 23 performs class classification for classifying the target pixel into any one of several classes.

  The class tap extraction unit 31 extracts a plurality of pixels of the mosaic image used for classifying the target pixel as class taps. Specifically, the class tap extraction unit 31 extracts, as class taps, a plurality of pixels of the mosaic image that are spatially close (near) to the pixels of the mosaic image corresponding to the target pixel.

  In addition, when the determination result that the saturated pixel exists is supplied from the saturated pixel presence determination unit 21, the class tap extraction unit 31 selects the long livestock pixel from among the plurality of pixels of the mosaic image to be extracted as the class tap. Only the removed short pixels are extracted as class taps.

  FIG. 10 illustrates an example of pixels extracted as a class tap and a prediction tap with respect to the target pixel.

  FIG. 10A shows an example of a class tap, and FIG. 10B shows an example of a prediction tap. 10A and 10B, the cross (x) mark shown in the center pixel of 25 pixels of 5 × 5 indicates that it is a pixel of interest, and the circle (◯) indicates a class tap or a prediction tap. Yes.

  In the example of FIG. 10A, the class tap is composed of a total of 13 pixels, that is, 8 pixels adjacent to the target pixel, and 2 pixels away from the target pixel in four horizontal and vertical directions. ing.

  In the example of FIG. 10B, the prediction tap is composed of a total of 25 pixels of 5 × 5 centered on itself with respect to the target pixel.

  As described above, in the prediction tap extraction unit 22 and the class tap extraction unit 31, it is determined in advance which pixel to extract as a class tap or a prediction tap with respect to the target pixel.

  Note that FIG. 10 is an example of a tap structure of prediction taps and class taps, and it is of course possible to employ other tap structures. Further, the prediction tap and the class tap may have the same tap structure or different tap structures.

  Here, for example, among the 25 pixels of 5 × 5 centered on the target pixel, as shown in FIG. 11, the leftmost pixel in the top row and the pixel moved by two pixels rightward from that pixel Assume that two are saturated pixels, and detection signals to that effect are supplied to the prediction tap extraction unit 22 and the class tap extraction unit 31.

  FIG. 12 shows an example of class taps and prediction taps when saturated pixels are detected as shown in FIG.

  FIG. 12A shows an example of a class tap, and FIG. 12B shows an example of a prediction tap.

  When the saturated pixel exists, the long live pixel is excluded from the class tap. Therefore, as shown in FIG. 12A, the class tap is composed of only 8 pixels adjacent to the target pixel.

  Moreover, as shown in FIG. 12B, the prediction tap is composed of 16 pixels excluding long livestock pixels among 5 × 5 25 pixels centered on itself.

  The prediction tap obtained by the prediction tap extraction unit 22 is supplied to the prediction calculation unit 25 of FIG. 9, and the class tap obtained by the class tap extraction unit 31 is supplied to the class code generation unit 32 of FIG.

  The class code generation unit 32 performs class classification based on the pixels constituting the class tap supplied from the class tap extraction unit 31, generates a class code corresponding to the class obtained as a result, and outputs the class code to the coefficient generation unit 24. Supply.

  As a method of classifying, for example, ADRC (Adaptive Dynamic Range Coding) or the like can be adopted.

  In the method using ADRC, the pixel values of the pixels constituting the class tap are subjected to ADRC processing, and the class of the target pixel is determined according to the ADRC code obtained as a result.

In the K-bit ADRC, for example, the maximum value MAX and the minimum value MIN of the pixels constituting the class tap are detected, and DR = MAX−MIN is set as the local dynamic range of the set, and this dynamic range Based on DR, the pixel values constituting the class tap are requantized to K bits. That is, the pixel value of each pixel forming the class taps, the minimum value MIN is subtracted, and the subtracted value is divided (quantized) by DR / 2 ^K. A bit string obtained by arranging the pixel values of the K-bit pixels constituting the class tap in a predetermined order is output as an ADRC code. Therefore, for example, when the class tap is subjected to 1-bit ADRC processing, the pixel value of each pixel constituting the class tap is divided by the average value of the maximum value MAX and the minimum value MIN (rounded down). Thereby, the pixel value of each pixel is set to 1 bit (binarized). Then, a bit string in which the 1-bit pixel values are arranged in a predetermined order is output as an ADRC code. For example, the class code generation unit 32 generates (outputs) an ADRC code obtained by performing ADRC processing on a class tap as a class code.

  Note that the class code generation unit 32 is a data classification method such as DCT (Discrete Cosine Transform), VQ (Vector Quantization), DPCM (Differential Pulse Code Modulation), etc. , And the data amount obtained as a result of data compression on the pixels constituting the class tap may be used as a class classification result.

  The coefficient generation unit 24 stores a prediction coefficient for each class that is obtained in advance by learning described later with reference to FIGS. 26 to 28. The coefficient generation unit 24 then stores the prediction coefficient stored in the address corresponding to the class code supplied from the class code generation unit 32 among the stored prediction coefficients (supplied from the class code generation unit 32). The prediction coefficient of the class represented by the class code) is supplied to the prediction calculation unit 25.

  The prediction calculation unit 25 acquires a prediction tap output from the prediction tap extraction unit 22 and a prediction coefficient output from the coefficient generation unit 24, and uses the prediction tap and the prediction coefficient to generate a pixel of the demosaic image of the target pixel. A predetermined prediction calculation for obtaining a value is performed. The pixel value of the demosaic image obtained here represents pixel signals of all the color components of R, G, and B.

[Processing of demosaic processing circuit]
Next, how to determine the pixel value of the demosaic image by the demosaic processing circuit 14 will be described with reference to FIGS.

  Conventionally, in the case where a saturated pixel exists in a captured image and no countermeasure (processing) for the saturated pixel is included, the pixel value of the saturated pixel is used as it is.

  For example, consider the case of shooting a subject as shown in FIG.

  The figure on the left side of FIG. 13 is an image for explaining the subject. The subject has a brighter brightness on the upper side in the vertical direction and a darker brightness on the lower side.

  The diagram on the right side of FIG. 13 shows the true luminance value of the subject at each position (vertical direction position) on the line a of the left subject. The vertical axis in the diagram on the right side of FIG. 13 indicates the vertical position on the line a, and the horizontal axis indicates the true luminance value (brightness) of the subject.

  The true luminance value of the subject is lower than the saturation level SV of the long pixel of the image sensor 12 below the vertical position Vb on the line a, but at or above the vertical position Vb on the line a, The luminance value is higher than the saturation level SV.

  The subject line a in FIG. 13 is captured by five pixels of the line a ′ in which the long pixels LG pixels and the short pixels SB pixels are alternately arranged in the image sensor 12 shown on the left side of FIG. To do.

  In this case, as shown on the right side of FIG. 14, the pixel values of the two SB pixels are multiplied by a gain corresponding to the exposure ratio N with respect to the pixel value of the saturation level SV or lower, so that the true luminance of the subject The value is equal to the value. On the other hand, the pixel values of the three LG pixels are values of the saturation level SV.

  Therefore, in an imaging device that does not take measures against saturated pixels, the pixel value of the saturation level SV is output for the LG pixel, so that the pixel value deviated from the true luminance value of the subject as shown by the solid line in FIG. 15A. Is obtained.

  On the other hand, the demosaic processing circuit 14 obtains a demosaic image using only the pixel value after the gain processing of the SB pixel, which is a short pixel, without using the pixel value of the LG pixel that has reached the saturation level SV. As shown by the solid line in FIG. 15B, a correct pixel value that matches the true luminance value of the subject can be obtained. That is, according to the imaging device 1, an accurate image corresponding to the subject can be output.

  Conventionally, as a countermeasure against saturated pixels, there is a method of obtaining a pixel value of a saturated pixel by interpolating with pixel values of surrounding pixels.

For example, as shown in FIG. 16, consider a case where a subject in which a bright part and a dark part are sharply switched at a certain boundary (edge) is photographed. The diagram on the right side of FIG. 16 shows the true luminance value of the subject at each position (vertical direction position) on the line b of the subject on the left side of FIG. 16. The true luminance value of the dark portion is P ₁ , and the true portion of the bright portion is true. luminance value is assumed to be P _2.

  The line b of the subject in FIG. 16 is photographed by five pixels of the line b ′ in which the LG pixels of the long pixels and the SB pixels of the short pixels are alternately arranged in the image sensor 12, as shown on the left side of FIG. Suppose that

  In this case, as shown on the right side of FIG. 17, the pixel values of the two SB pixels are pixel values obtained by multiplying the pixel value below the saturation level SV by a gain according to the exposure ratio N. The value is equal to the true luminance value of the subject. On the other hand, among the three LG pixel values, the pixel value of the bottom LG pixel in the figure is a value equal to or lower than the saturation level SV, and therefore matches the true luminance value of the subject. The pixel values of the upper two LG pixels are clipped at the saturation level SV because the true luminance value is larger than the saturation level SV.

  For the pixel value of 5 pixels in the line b ′ of the image sensor 12 obtained in this way, the pixel values of two LG pixels clipped at the saturation level SV are conventionally the pixel values of the surrounding pixels. Obtained by interpolation.

  FIG. 18 shows a state after interpolation of two LG pixels clipped at the saturation level SV.

As shown in FIG. 18, the top LG pixel is corrected to the bright pixel value P ₂ , but the second LG pixel from the top is the bright pixel value P ₂ and the dark pixel. The value is corrected to an intermediate value ((P ₁ + P ₂ ) / 2) of the value P ₁ . As a result, as shown in FIG. 19, the edge portion becomes an intermediate color between the bright portion and the dark portion, resulting in an image with blurred edges.

Here, also in the past, considering that the second LG pixel from the top is the pixel on the edge, an intermediate value ((P ₁ + P ₂ ) / 2) of the pixel values of the bright part and the dark part is set to part of the pixel values P _2, or, it is conceivable to perform processing to isolate either of the dark part of the pixel value P ₁ additionally.

However, there are variations in the pixel values P ₂ and dark pixel value P ₁ itself noises light portion. Then, the pixel value of the second LG pixel obtained by interpolation becomes a pixel value P ₂ on the bright side as shown in FIG. 20 due to variations in them, or on the dark side as shown in FIG. it may become the pixel value P _1. As a result, as shown in FIG. 22, the image that is finally output may be an image in which the pixel value of the edge portion fluctuates for each pixel.

On the other hand, the demosaic processing circuit 14 does not use the pixel value of the LG pixel that has reached the saturation level SV, but uses only the pixel value after gain processing of the SB pixel, which is a short-lived pixel, for each pixel. Ask for. Therefore, as shown in FIG. 23, the edge does not shake for each pixel. In the example of FIG. 23, the pixel value of the edge portion, is output as the pixel value P ₂ of the bright portion. Therefore, the demosaic processing circuit 14 can obtain a correct pixel value that matches the true luminance value of the subject, as shown in FIG. That is, according to the imaging device 1, an accurate image corresponding to the subject can be output.

[Prediction calculation and prediction coefficient learning]
Next, prediction calculation in the prediction calculation unit 25 of the demosaic processing circuit 14 and learning of a prediction coefficient used for the prediction calculation will be described.

  Now, an image signal in which saturated pixels are interpolated is used as a high-quality image signal, and an image signal including the saturated pixels as it is is used as a low-quality image signal, and a prediction tap is extracted from the low-quality image signal. It is assumed that the pixel value of a pixel of a high-quality image signal (hereinafter, appropriately referred to as a high-quality pixel) is obtained (predicted) by a predetermined prediction calculation using the prediction coefficient.

  Assuming that, for example, linear primary prediction calculation is adopted as the predetermined prediction calculation, the pixel value y of the high-quality pixel is obtained by the following linear linear expression.

In Equation (1), x _n represents the pixel value of the pixel of the n-th low-quality image signal (hereinafter referred to as “low-quality pixel” as appropriate) that constitutes the prediction tap for the high-quality pixel y. w _n represents the n th prediction coefficient to be multiplied by the n th low image quality pixel (its pixel value). In equation (1), the prediction tap is assumed to be composed of N low image quality pixels x ₁ , x ₂ ,..., X _N.

  The pixel value y of the high-quality pixel can be obtained not by the linear primary expression shown in Expression (1) but by a higher-order expression of the second or higher order.

Now, when the true value of the pixel value of the high-quality pixel of the k-th sample is expressed as y _k and the predicted value of the true value y _k obtained by the equation (1) is expressed as y _k ′, the prediction error e _k Is expressed by the following equation.

Now, since the predicted value y _k ′ of Equation (2) is obtained according to Equation (1), the following equation is obtained by replacing y _k ′ of Equation (2) according to Equation (1).

In Equation (3), x _{n, k} represents the nth low-quality pixel that constitutes the prediction tap for the high-quality pixel of the k-th sample.

Prediction coefficient w _n of the prediction error e _k and 0 of the formula (3) (or Equation (2)) is, is the optimal for predicting the high-quality pixel, for all the high-quality pixel, such In general, it is difficult to obtain a correct prediction coefficient w _n .

Therefore, as the standard for representing that the prediction coefficient w _n is optimal, for example, when adopting the method of least squares, optimal prediction coefficient w _n is the sum E of square errors expressed by the following formula It can be obtained by minimizing.

However, in Equation (4), K is the high image quality pixel y _k and the low image quality pixels x _{1, k} , x _{2, k} ,..., X _N constituting the prediction tap for the high image quality pixel y _{k. , k} represents the number of samples (the number of learning samples).

The minimum value of the sum E of square errors of Equation (4) (minimum value), as shown in Equation (5), given that the sum E partially differentiated by the prediction coefficient w _n by w _n to 0.

Therefore, when partial differentiation of above equation (3) by the prediction coefficient w _n, the following equation is obtained.

  From the equations (5) and (6), the following equation (7) is obtained.

By substituting equation (3) into e _k in equation (7), equation (7) can be expressed by the normal equation shown in equation (8).

Normal equation of Equation (8), for example, by using a like sweeping-out method (Gauss-Jordan elimination method) can be solved for the prediction coefficient w _n.

By solving each normal equation class of formula (8), the optimum prediction coefficient (here, a prediction coefficient that minimizes the sum E of square errors) to w _n, can be found for each class.

[Flowchart of imaging processing]
With reference to the flowchart of FIG. 25, the demosaic image generation process by the imaging device 1 will be described.

  First, in step S1, the image sensor 12 images a subject. That is, the image sensor 12 receives subject light, converts an analog electrical signal corresponding to the received light amount into a digital electrical signal, and supplies the digital electrical signal to the gain adjustment circuit 13 as an image signal. In the image sensor 12, as shown in FIG. 6, whether the pixel is a long pixel or a short pixel is determined in advance depending on the pixel position, and each pixel is determined in advance depending on whether the pixel is a long pixel or a short pixel. Light is received with the specified exposure time.

  In step S <b> 2, the gain adjustment circuit 13 multiplies the pixel value of the short livestock pixel by N times as an exposure ratio to match the signal levels of the short livestock pixel and the long livestock pixel. The image signal after gain adjustment is output to the demosaic processing circuit 14.

  In step S3, the saturated pixel presence determination unit 21 of the demosaic processing circuit 14 sets a target pixel. That is, the saturated pixel presence determination unit 21 sets a predetermined pixel constituting a demosaic image obtained by prediction calculation as a target pixel to be noted as a prediction calculation target pixel.

  In step S4, the saturated pixel presence determination unit 21 determines whether there is a saturated pixel among the plurality of pixels that are the prediction tap and the class tap. When a saturated pixel is detected among a plurality of pixels serving as a prediction tap, the saturation pixel presence determination unit 21 supplies a detection signal indicating that a saturation pixel exists to the prediction tap extraction unit 22. When a saturated pixel is detected among a plurality of pixels serving as class taps, the saturated pixel presence determination unit 21 supplies a detection signal indicating that a saturated pixel exists to the class tap extraction unit 31.

  In step S5, the class tap extraction unit 31 supplies from the saturated pixel presence determination unit 21 a detection signal indicating that a saturated pixel exists in the class tap set for the target pixel, that is, a saturated pixel exists. Determine whether it was done.

  If it is determined in step S5 that no saturated pixel exists, the process proceeds to step S6, and the class tap extraction unit 31 extracts a plurality of pixels at predetermined positions near the target pixel as class taps, and class codes are obtained. Supply to the generator 32. Here, the plurality of pixels extracted as class taps include both long and short livestock pixels.

  On the other hand, if it is determined in step S5 that a saturated pixel exists, the process proceeds to step S7, and the class tap extraction unit 31 selects a class tap as a class tap from a plurality of pixels in a predetermined position near the target pixel of setting. Only the short livestock pixels excluding the long livestock pixels are extracted as class taps and supplied to the class code generating unit 32.

  In step S <b> 8, the class code generation unit 32 performs a 1-bit ADRC process on the class tap supplied from the class tap extraction unit 31 to obtain the class code of the pixel of interest and supplies it to the coefficient generation unit 24.

  In step S <b> 9, the coefficient generation unit 24 supplies the prediction calculation unit 25 with the prediction coefficient of the class represented by the class code of the target pixel supplied from the class code generation unit 32.

  In step S <b> 10, the prediction tap extraction unit 22 supplies, from the saturation pixel presence determination unit 21, a detection signal indicating that a saturation pixel exists in the prediction tap set for the target pixel, that is, a saturation pixel exists. Determine whether it was done.

  If it is determined in step S10 that no saturated pixel exists, the process proceeds to step S11, and the prediction tap extraction unit 22 extracts a plurality of pixels at a predetermined position near the target pixel as prediction taps, and performs prediction calculation. To the unit 25. The plurality of pixels extracted as prediction taps here include both long and short livestock pixels.

  On the other hand, when it is determined in step S10 that no saturated pixel exists, the process proceeds to step S12, and the prediction tap extraction unit 22 sets a plurality of pixels at a predetermined position in the vicinity of the target pixel set in advance as the prediction tap. Among them, only the short livestock pixels excluding the long livestock pixels are extracted as prediction taps and supplied to the prediction calculation unit 25.

  In step S <b> 13, the prediction calculation unit 25 performs the calculation (prediction calculation) of Expression (1) using the prediction tap supplied from the prediction tap extraction unit 22 and the prediction coefficient supplied from the coefficient generation unit 24. Thus, the pixel value of the target pixel is obtained.

  In step S <b> 14, the prediction calculation unit 25 determines whether all the pixels of the demosaic image as the generated image have been obtained as the target pixel.

  If it is determined in step S14 that all the pixels have not yet been set as the target pixel, the process returns to step S3, and a pixel that has not yet been set as the target pixel is set as the next target pixel to perform the above-described steps S3 to S14. The process is repeated.

  On the other hand, if it is determined in step S14 that all the pixels are the target pixel, the process ends.

  The above processing is repeatedly executed whenever a subject is imaged by the image sensor 12 and an image signal of a frame image (field image) is output from the image sensor 12.

[Configuration example of learning device]
Figure 26 shows an example of the configuration of a learning apparatus 51 performs learning for obtaining prediction coefficients w _n for each class by solving the normal equations in equation (8).

Learning image signal generating unit 61 of the learning apparatus 51 generates tutor data and student data serving as learning image signal used for learning of prediction coefficients w _n.

  The generation of teacher data and student data as learning image signals will be described with reference to FIG. FIG. 27 shows a detailed configuration example of the learning image signal generation unit 61. In the following description of the generation of the learning image signal, it is assumed that the exposure ratio N is 10 (exposure time for long lived pixels: exposure time for short lived pixels = 10: 1).

  The learning image signal generation unit 61 includes at least an integration unit 71, a synthesis unit 72, a gain adjustment circuit 73, and a clip unit 74.

  First, one image obtained by photographing a predetermined subject with an image sensor having the same exposure time for all pixels is input to the integrating unit 71. The image input to the accumulating unit 71 is an image that is the basis of a teacher image and a student image to be generated from now on, and is referred to as a basic image.

  The integrating unit 71 generates an image obtained by integrating the input basic image according to the exposure ratio N, and outputs it as a teacher image. That is, the integrating unit 71 performs a process of multiplying the pixel value of each pixel of the basic image by 10 and outputs the resulting image as a teacher image.

  The composition unit 72 is supplied with one input basic image and the teacher image generated by the integration unit 71. The combining unit 72 selects the pixel value of the basic image or the pixel value of the teacher image so as to correspond to the positions of the long pixel and the short pixel of the image sensor 12 of the imaging device 1 that performs the prediction calculation, A synthesized composite image is generated. That is, the combining unit 72 selects the pixel value at the position corresponding to the teacher image as the pixel value at the position of the long pixel of the image sensor 12, and the pixel value at the position of the short pixel of the image sensor 12 as An image in which pixel values at corresponding positions in the basic image are selected is generated as a composite image. The generated composite image is supplied to the gain adjustment circuit 73.

  The gain adjustment circuit 73 is similar to the case where the gain is applied to the pixel value of the short livestock pixel in order to match the signal level of the short livestock pixel and the long livestock pixel in the imaging apparatus 1 of FIG. Then, a process of multiplying the exposure ratio N gain is executed.

  The clip unit 74 causes the pixel value of one or more predetermined long pixels of the composite image after gain processing supplied from the gain adjustment circuit 73 to be clipped at a preset saturation level. The long-lived pixels clipped at the saturation level correspond to saturated pixels when the image sensor 12 of the imaging device 1 captures an image. Therefore, hereinafter, a pixel clipped at the saturation level of the composite image is also referred to as a saturation pixel. The clip unit 74 outputs the clip-processed image as a student image.

  An additional processing unit 75 that performs processing for adding random noise to the student image generated by the clip unit 74, processing for adding image blur and distortion peculiar to the imaging device 1, and the like is provided at the subsequent stage of the clip unit 74. Can do.

  The learning image signal generation unit 61 generates a large number of sets of teacher images and student images by changing a subject or changing a pixel position to be clipped. A large number of generated teacher images are stored as teacher data in the teacher data storage unit 62, and a large number of generated student images are stored as student data in the student data storage unit 63.

  The teacher data storage unit 62 in FIG. 26 stores teacher data. The pixels constituting the teacher image stored in the teacher data storage unit 62 are sequentially set as the target teacher pixel. The student data storage unit 63 stores student data.

  The saturated pixel presence determination unit 64 performs the presence determination of the saturated pixel similar to the saturated pixel presence determination unit 21 in FIG. That is, the saturated pixel presence determination unit 64 determines whether or not there is a saturated pixel in the prediction tap and class tap pixels set for the teacher pixel of interest, and the determination result is compared with the prediction tap extraction unit 65. The data is output to the class tap extraction unit 66.

  The prediction tap extraction unit 65 extracts a plurality of pixels in the vicinity of the pixel of the student image corresponding to the teacher pixel of interest supplied from the student data storage unit 63 as a prediction tap and supplies the extracted pixel to the addition unit 68. Here, the position of the prediction tap extracted by the prediction tap extraction unit 65 is the same position as the position of the prediction tap extracted by the prediction tap extraction unit 22 of FIG. Further, when a detection signal indicating that a saturated pixel is present is supplied from the saturated pixel presence determination unit 64, the prediction tap extraction unit 65 applies a short pixel to the short-lived pixel as in the prediction tap extraction unit 22 of FIG. Only a plurality of pixels at corresponding positions are extracted.

  The class tap extraction unit 66 extracts a plurality of pixels in the vicinity of the pixel of the student image corresponding to the teacher image of interest supplied from the student data storage unit 63 as class taps and supplies the class tap to the class code generation unit 67. Here, the position of the class tap extracted by the class tap extraction unit 66 is the same position as the position of the class tap extracted by the class tap extraction unit 31 of FIG. Further, when a detection signal indicating that a saturated pixel exists is supplied from the saturated pixel presence determination unit 64, the class tap extraction unit 66 converts the short tap pixel into a short live pixel as in the class tap extraction unit 31 of FIG. Only a plurality of pixels at corresponding positions are extracted.

  The class code generation unit 67 performs the same class classification as the class code generation unit 32 of FIG. 9 based on the class tap output from the class tap extraction unit 66, and adds the class code corresponding to the resulting class. To the unit 68.

  The adding unit 68 reads the target teacher pixel from the teacher data storage unit 62, the target teacher pixel, and the student data constituting the prediction tap configured for the target teacher pixel supplied from the prediction tap extraction unit 65 Is added for each class code supplied from the class code generation unit 67.

That is, the adder 68 is supplied with the teacher data y _k stored in the teacher data storage 62, the prediction tap x _{n, k} output from the prediction tap extractor 65, and the class code output from the class code generator 67. Is done.

Then, the adding unit 68 uses the prediction tap (student data) x _{n, k} for each class corresponding to the class code supplied from the class code generating unit 67 _, and uses the student data in the matrix on the left side of equation (8). An operation corresponding to multiplication (x _{n, k} x _{n ′, k} ) and summation (Σ) is performed.

Further, the adding unit 68 also uses the prediction tap (student data) x _{n, k} and the teacher data y _k for each class corresponding to the class code supplied from the class code generating unit 67, and the equation (8). Is multiplied by the student data x _{n, k} and the teacher data y _k (x _{n, k} y _k ) in the vector on the right side of _, and an operation corresponding to summation (Σ) is performed.

That is, the adding unit 68 performs the left-side matrix component (Σx _{n, k} x _{n ′, k} ) and the right-side vector component in Equation (8) previously obtained for the teacher data set as the target teacher pixel. (Σx _{n, k} y _k ) is stored in its built-in memory (not shown), and the matrix component (Σx _{n, k} x _{n ′, k} ) or vector component (Σx _{n, k} y) _k ), the corresponding component x _{n, k + 1} calculated using the teacher data y _{k + 1} and the student data x _{n, k + 1} for the teacher data newly selected as the teacher pixel of interest. x _{n ′, k + 1} or x _{n, k + 1} y _{k + 1} is added (addition represented by the summation of Expression (8) is performed).

  Then, the addition unit 68 performs the above-described addition using all the teacher data stored in the teacher data storage unit 62 as the target teacher pixel, thereby obtaining the normal equation shown in Expression (8) for each class. Then, the normal equation is supplied to the prediction coefficient calculation unit 69.

Prediction coefficient calculation unit 69 solves the normal equations for each class supplied from the adder 68, for each class, and outputs the determined optimal prediction coefficients w _n.

The coefficient generation unit 24 of the demosaic processing circuit 14 in FIG. 9 stores the prediction coefficient w _m for each class obtained as described above.

  In the above case, the teacher image is an image in which no saturated pixel exists, and the student image is an image in which the pixel value of a predetermined pixel corresponding to the long-lived pixel is clipped at a predetermined saturation level. Thereby, as a prediction coefficient produced | generated by the learning apparatus 51, even if it is a case where the long livestock pixel of the image sensor 12 was saturated in imaging | photography by the imaging device 1, it interpolates appropriately using only the short livestock pixel To generate a pixel value obtained.

  Further, in the case of the learning image signal generation unit 61 in FIG. 27, when an addition processing unit 75 for adding random noise is added, saturated pixels are appropriately interpolated and random noise is removed. A prediction coefficient for generating a (reduced) pixel value can be generated.

  In addition, in the learning image signal generation unit 61 in FIG. 27, when an additional processing unit 75 that performs processing such as adding image blur and distortion peculiar to the imaging device 1 is added, Can be appropriately interpolated, and a prediction coefficient for generating a pixel value from which image blur and distortion are removed (reduced) can be generated.

[Flowchart of learning process]
Next, the learning process of the learning device 51 will be described with reference to the flowchart of FIG.

  First, in step S41, the learning image signal generation unit 61 generates teacher data and student data as learning image signals, the generated teacher data is stored in the teacher data storage unit 62, and the generated student data is stored as student data. It memorize | stores in the memory | storage part 63, respectively.

  In step S <b> 42, the prediction tap extraction unit 65 sets the teacher data stored in the teacher data storage unit 62 as notable teacher pixels that have not yet been noted teacher pixels.

  In step S43, the saturated pixel presence determination unit 64 detects whether there is a saturated pixel among the student data pixels that are the prediction tap and the class tap corresponding to the teacher pixel of interest. The saturated pixel presence determination unit 64 supplies a detection signal to the prediction tap extraction unit 65 when a saturated pixel exists in a pixel that becomes a prediction tap, and when a saturated pixel exists in a pixel that becomes a class tap, The detection signal is supplied to the class tap extraction unit 66.

  In step S <b> 44, the prediction tap extraction unit 65 extracts a prediction tap from the student data stored in the student data storage unit 63 for the teacher pixel of interest and supplies the prediction tap to the adding unit 68. Here, when the detection signal indicating that a saturated pixel exists is supplied from the saturated pixel presence determination unit 64, the prediction tap extraction unit 65 performs prediction using only the short livestock pixels excluding the long livestock pixels. Configure taps.

  In step S <b> 45, the class tap extraction unit 66 extracts the class tap from the student data stored in the student data storage unit 63 for the teacher pixel of interest and supplies the class tap to the class code generation unit 67. Here, when a detection signal indicating that a saturated pixel is present is supplied from the saturated pixel presence determining unit 64, the class tap extracting unit 66 uses only the short lived pixels excluding the long lived pixels. Configure taps.

  In step S46, the class code generator 67 classifies the focused teacher pixel based on the class tap for the focused teacher pixel, and outputs the class code corresponding to the resulting class to the adding unit 68. To do.

  In step S <b> 47, the adding unit 68 reads the target teacher pixel from the teacher data storage unit 62, and configures the prediction tap configured for the target teacher pixel and the target teacher pixel supplied from the prediction tap extraction unit 65. The addition of the equation (8) for the student data is performed for each class code (class) supplied from the class code generation unit 67.

  In step S <b> 48, the prediction tap extraction unit 65 determines whether teacher data that has not yet been focused on as a teacher pixel is stored in the teacher data storage unit 62. If it is determined in step S48 that the teacher data that is not the attention teacher pixel is still stored in the teacher data storage unit 62, the process returns to step S42, and the prediction tap extraction unit 65 has not yet selected the teacher data. Hereinafter, the same processing is repeated with the teacher data as a new focused teacher pixel.

  On the other hand, if it is determined in step S48 that the teacher data that is not the attention teacher pixel is not stored in the teacher data storage unit 62, the process proceeds to step S49. In step S49, the adding unit 68 supplies the matrix on the left side and the vector on the right side in Equation (8) for each class obtained by the processing so far to the prediction coefficient calculating unit 69.

In step S49, the prediction coefficient calculation unit 69 solves the normal equation for each class constituted by the matrix on the left side and the vector on the right side in the equation (8) for each class supplied from the addition unit 68, seeking a prediction coefficient w _n for each class, the process is terminated.

  When the learning process described above is executed by the learning device 51, the prediction coefficient for each class stored in the coefficient generation unit 24 of the demosaic processing circuit 14 in FIG. 9 is obtained.

  It should be noted that due to the number of learning image signals being insufficient, etc., there may occur a class in which the number of normal equations necessary for obtaining a prediction coefficient cannot be obtained. For example, the prediction coefficient calculation unit 69 outputs a default prediction coefficient.

<2. Second Embodiment>
Next, a second embodiment of an imaging apparatus to which the present technology is applied will be described.

  The second embodiment of the imaging apparatus 1 differs from the imaging apparatus 1 of FIG. 5 only in the configuration of the demosaic processing circuit 14. Therefore, the configuration of the demosaic processing circuit 14 of the second embodiment will be described. Hereinafter, the demosaic processing circuit 14 of the first embodiment is referred to as a first configuration of the demosaic processing circuit 14, and the demosaic processing circuit 14 of the second embodiment is referred to as a second configuration of the demosaic processing circuit 14. Called.

[Configuration example of demosaic processing circuit]
FIG. 29 is a block diagram showing a second configuration of the demosaic processing circuit 14.

  The demosaic processing circuit 14 in FIG. 29 differs from the above-described first configuration of the demosaic processing circuit 14 in a saturated pixel presence determination unit 81, a prediction tap extraction unit 82, and a class tap extraction unit 83. In the demosaic processing circuit 14 of FIG. 29, the description of the same part as the first configuration will be omitted as appropriate.

  The saturated pixel presence determination unit 21 having the first configuration described above determines whether or not a saturated pixel exists in a pixel serving as a prediction tap and a class tap with respect to the target pixel, and outputs a determination result.

  In the second configuration of FIG. 29, the saturated pixel presence determination unit 81 determines whether a saturated pixel exists in a pixel that is a prediction tap and a class tap with respect to the target pixel, and determines that a saturated pixel exists. If so, saturated pixel position information indicating the position of the saturated pixel is output.

  That is, the saturated pixel presence determination unit 81 determines whether or not there is a saturated pixel in a plurality of pixels extracted as prediction taps by the prediction tap extraction unit 82. Saturated pixel position information indicating the pixel position is output to the prediction tap extraction unit 82. Further, the saturated pixel presence determination unit 81 determines whether or not a saturated pixel exists in any of the plurality of pixels extracted as the class tap, and when it is determined that the saturated pixel exists, the position of the saturated pixel Is output to the class tap extraction unit 83.

  The saturated pixel position information includes, for example, a plurality of (scheduled) pixels to be predicted taps or class taps in a predetermined order (for example, raster scanning order), saturated pixels “1”, and unsaturated pixels “ Information represented as “0” can be used. In addition, if a data compression technique such as DPCM (predictive coding) is used for saturated pixel position information in which “1” or “0” continues for the number of pixels, the amount of data can be further reduced and prediction taps can be made. The data can be provided to the extraction unit 82 and the class tap extraction unit 83.

  Similar to the prediction tap extraction unit 22 having the first configuration, the prediction tap extraction unit 82 determines pixels (pixel values) constituting the mosaic image used to predict the pixel value of the target pixel of the demosaic image to be predicted. , Extracted as a prediction tap, and supplied to the prediction calculation unit 25.

  Specifically, when the saturated pixel position information indicating that no saturated pixel exists is supplied from the saturated pixel presence determination unit 81, the prediction tap extraction unit 82 predicts a plurality of pixels near the target pixel of the mosaic image. This is extracted as a tap and supplied to the prediction calculation unit 25. In this case, the plurality of pixels extracted as prediction taps are the same as the prediction tap extraction unit 22 of the first configuration.

  On the other hand, when saturated pixel position information indicating that a saturated pixel exists is supplied from the saturated pixel presence determination unit 81, the prediction tap extraction unit 82 selects a saturated pixel from a plurality of pixels near the target pixel of the mosaic image. Pixels excluding the pixels indicated as saturated pixels in the pixel position information are extracted as prediction taps and supplied to the prediction calculation unit 25.

  Similar to the class tap extraction unit 31 having the first configuration, the class tap extraction unit 83 extracts a plurality of pixels of the mosaic image used for classifying the target pixel as class taps. When saturated pixel position information indicating that no saturated pixel exists is supplied from the saturated pixel presence determination unit 81, the class tap extraction unit 83 is a spatially close position to the pixel of the mosaic image corresponding to the target pixel. A plurality of pixels of the mosaic image in the vicinity are extracted as class taps. In this case, the plurality of pixels extracted as class taps are the same as the class tap extraction unit 31 of the first configuration.

  On the other hand, when the saturated pixel position information indicating that a saturated pixel exists is supplied from the saturated pixel presence determination unit 81, the class tap extraction unit 83, among a plurality of pixels of the mosaic image to be extracted as a class tap, Pixels excluding the pixels indicated as saturated pixels in the saturated pixel position information are extracted as class taps.

  The operations of the prediction tap extraction unit 82 and the class tap extraction unit 83 will be described using the example of the class tap and the prediction tap described with reference to FIG.

  The pixel shown in FIG. 10A is determined as the class tap extraction target pixel, and the pixel shown in FIG. 10B is determined as the prediction tap extraction target pixel. Of these, two pixels, the leftmost pixel in the top row and the pixel moved two pixels to the right from the uppermost row shown in FIG. 11, are detected as saturated pixels, and the saturated pixel position information is the prediction tap extraction unit. 82 and the class tap extraction unit 83.

  In this case, the class tap and the prediction tap are as shown in FIG. FIG. 30A shows an example of class taps extracted by the class tap extraction unit 83, and FIG. 30B shows an example of prediction taps extracted by the prediction tap extraction unit 82.

  As apparent from the comparison with FIG. 12A, the class tap extraction unit 83 also extracts LG pixels that are not saturated pixels as class taps. Similarly, in the prediction tap extraction unit 82, LG pixels that are not saturated pixels are also extracted as prediction taps, as is clear as compared with FIG. 12B.

  Similar to the first embodiment, the coefficient generation unit 24 stores the prediction coefficient for each class obtained by the learning process of the learning device 51. However, in the learning process for the second embodiment, saturated pixel detection, class tap extraction, and prediction tap extraction are performed by a saturated pixel presence determination unit 81, a prediction tap extraction unit 82, and a class tap extraction unit. This is different from the learning process for the first embodiment in that it is performed in the same manner as 83.

  In the present embodiment, saturation pixel position information is supplied to the prediction tap extraction unit 82, and the prediction tap extraction unit 82 does not output the pixel value of the saturated pixel to the prediction calculation unit 25. , The saturated pixel portion of the prediction calculation of equation (1) is omitted. However, the prediction calculation unit 25 is omitted by replacing the prediction coefficient of the saturated pixel portion of the prediction calculation of Expression (1) with “0” so that the saturated pixel position information is supplied to the prediction calculation unit 25. You may do it. In this case, the prediction tap extraction unit 82 always outputs the pixel values of a plurality of pixels at a predetermined position with respect to the target pixel to the prediction calculation unit 25 regardless of whether there is a saturated pixel.

[Flowchart of imaging processing]
A demosaic image generation process according to the second embodiment will be described with reference to the flowchart of FIG.

  Steps S61 to S63 are the same as steps S1 to S3 in FIG.

  In step S64, the saturated pixel presence determination unit 81 determines whether there is a saturated pixel among the plurality of pixels that are the prediction tap and the class tap. When a saturated pixel is detected among a plurality of pixels serving as a prediction tap, the saturated pixel presence determination unit 81 supplies saturated pixel position information indicating the position of the saturation pixel to the prediction tap extraction unit 82. When a saturated pixel is detected among a plurality of pixels serving as class taps, the saturated pixel presence determining unit 81 supplies saturated pixel position information indicating the position of the saturated pixel to the class tap extracting unit 83.

  In step S65, the class tap extraction unit 83 determines whether there is a saturated pixel in the class tap set for the target pixel, that is, saturated pixel position information indicating the presence of the saturated pixel is received from the saturated pixel presence determination unit 81. Determine if it was supplied.

  If it is determined in step S65 that no saturated pixel exists, the process proceeds to step S66, and the class tap extraction unit 83 extracts a plurality of pixels at predetermined positions near the target pixel as class taps, and class codes are obtained. Supply to the generator 32.

  On the other hand, if it is determined in step S65 that a saturated pixel exists, the process proceeds to step S67. In step S67, the class tap extraction unit 83 extracts, from the plurality of pixels at predetermined positions near the target pixel to be set as the class tap, the pixel excluding the saturated pixel as the class tap, and the class code generation unit 32. To supply.

  In the next steps S68 and S69, as in steps S8 and S9 of FIG. 25, the class code generation unit 32 obtains the class code of the pixel of interest, and the coefficient generation unit 24 calculates the prediction coefficient corresponding to the class code. 25.

  In step S <b> 70, the prediction tap extraction unit 82 determines whether saturated pixel exists in the prediction tap set for the target pixel, that is, saturated pixel position information indicating the presence of the saturated pixel is received from the saturated pixel presence determination unit 81. Determine if it was supplied.

  If it is determined in step S70 that no saturated pixel exists, the process proceeds to step S71, and the prediction tap extraction unit 82 extracts a plurality of pixels at a predetermined position near the target pixel as prediction taps, and performs prediction calculation. To the unit 25.

  On the other hand, if it is determined in step S70 that no saturated pixel exists, the process proceeds to step S72. In step S <b> 72, the prediction tap extraction unit 22 extracts a pixel excluding a saturated pixel from a plurality of pixels at a predetermined position near the target pixel to be set as a prediction tap as a prediction tap, and outputs the prediction tap to the prediction calculation unit 25. Supply.

  The processing in steps S73 and S74 is the same as the processing in steps S13 and S14 in FIG.

  The above processing is repeatedly executed whenever a subject is imaged by the image sensor 12 and an image signal of a frame image (field image) is output from the image sensor 12.

  As described above, in the first embodiment, the demosaic processing circuit 14 excludes all pixel values of the long pixel from the target pixels of the class tap or the prediction tap when the saturated pixel is detected. Pixels were class taps or prediction taps.

  On the other hand, in the second embodiment, the demosaic processing circuit 14 uses the class tap or the prediction tap as a pixel excluding only the saturated pixel among the target pixels of the class tap or the prediction tap when the saturation pixel is detected. It was.

  In either case of adopting the first embodiment or the second embodiment, the class code is determined using a plurality of pixels in the vicinity of the target pixel not including the saturated pixel. Pixels can be prevented (reduced) from being classified into the wrong class. In addition, since the prediction calculation unit 25 can perform prediction calculation without using the pixel value of the saturated pixel, the imaging device 1 can generate a demosaic image in which saturated pixels are appropriately interpolated. That is, the imaging device 1 can generate a demosaic image that realizes a wide dynamic range.

  Note that the imaging apparatus 1 can selectively execute the first and second embodiments of the demosaic processing circuit 14 described above.

  Conventionally, when the pixel value of a saturated pixel is obtained by interpolation from a non-saturated pixel, the accuracy of the interpolation is in a trade-off relationship with the circuit scale. growing. The imaging apparatus 1 can execute a demosaic process in which saturated pixels are correctly interpolated without increasing the circuit scale.

  In the above-described embodiment, the image sensor 12 is a combination of pixels having two types of exposure time, a pixel that receives light with a long exposure time and a pixel that receives light with a shorter exposure time.

  However, the present technology is not limited to a combination of two types of pixels (pixel groups) having different exposure times, and may be a combination of pixels (pixel groups) having three or more types of exposure times. For example, the image sensor 12 is a pixel (pixel group) having three types of exposure times from the longest exposure time, the long livestock pixel, the middle livestock pixel, and the short livestock pixel (long livestock pixel> medium livestock pixel> short livestock pixel). As a combination of the above, it is possible to detect the saturated pixels of the long-lived pixels and execute the above-described demosaic image generation process. In this case, when the saturated pixel of the long livestock pixel is detected, only the livestock pixel and the short livestock pixel may be used as the prediction tap and the class tap. Livestock pixels may be used. Further, it is possible to detect saturated pixels of long livestock pixels and medium livestock pixels and use only short livestock pixels as prediction taps and class taps, or short livestock pixels and unsaturated long livestock pixels and livestock pixels. May be used.

  In the example described above, the demosaic processing circuit 14 has been described as a configuration that is incorporated as a part of the imaging apparatus 1, but an image signal output from an imaging element in which a plurality of pixel groups having different exposure times are regularly arranged. Can be configured independently as an image processing apparatus that performs demosaic processing as described above. In this case, the demosaic processing circuit 14 can be configured as a single chip (SOC: System On Chip) by itself, or by adding a gain adjustment circuit 13 and a compression processing circuit 15 as necessary. .

  The above-described example is an example in which a saturated pixel is detected from the image signal output from the image sensor 12, and the class classification adaptive process using the class tap and the prediction tap excluding the detected saturated pixel is applied to the demosaic process. is there. However, the class classification adaptive process described above may be applied to processes other than the demosaic process. For example, a rede-mosaic process that converts an R, G, B pixel array into an R, G, B + W (white) pixel array, a scale conversion process that converts the image scale, and a high resolution that converts the image resolution to a high resolution You may apply to a process (super-resolution process) etc. Further, two or more of the demosaic process, the redemosaic process, the scale conversion process, and the high resolution process to which the above-described class classification adaptive process is applied may be executed in series or in parallel.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

  FIG. 32 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are connected to each other via a bus 104.

  An input / output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

  The input unit 106 includes a keyboard, a mouse, a microphone, and the like. The output unit 107 includes a display, a speaker, and the like. The storage unit 108 is composed of a hard disk, a nonvolatile semiconductor memory, or the like. The communication unit 109 includes a network interface or the like. The drive 110 drives a removable recording medium 111 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, the CPU 101 as a processing processor loads, for example, a program stored in the storage unit 108 to the RAM 103 via the input / output interface 105 and the bus 104 and executes the program. The series of processes described above is performed.

  In the computer, the program can be installed in the storage unit 108 via the input / output interface 105 by attaching the removable recording medium 111 to the drive 110. Further, the program can be received by the communication unit 109 and installed in the storage unit 108 via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In addition, the program can be installed in the ROM 102 or the storage unit 108 in advance.

  In the present specification, the steps described in the flowcharts are performed in parallel or in a call even if they are not necessarily processed in time series, as well as in time series in the order described. It may be executed at a necessary timing such as when.

  Embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

In addition, this technique can also take the following structures.
(1)
A saturated pixel detection unit for detecting a saturated pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged;
A surrounding pixel specifying unit that specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
An image processing apparatus comprising: a prediction calculation unit that performs an image prediction calculation process based on the surrounding pixels.
(2)
The image processing apparatus according to (1), wherein the surrounding pixel specifying unit specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes only pixels of an element group different from the element group to which the saturated pixel belongs.
(3)
The plurality of element groups having different exposure times include a first element group having a first exposure time and a second element group having a second exposure time shorter than the first exposure time,
The surrounding pixel specifying unit specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes pixels of the second element group different from the first element group to which the saturated pixel belongs. The image processing apparatus according to any one of (2).
(4)
The surrounding pixel specifying unit is located in the vicinity of the saturated pixel, and is a surrounding pixel composed of a pixel other than the element group to which the saturated pixel belongs and a pixel other than the saturated pixel of the element group to which the saturated pixel belongs. The image processing apparatus according to any one of (1) to (3).
(5)
The plurality of element groups having different exposure times include a first element group having a first exposure time and a second element group having a second exposure time shorter than the first exposure time,
The surrounding pixel specifying unit is located in the vicinity of the saturated pixel, the pixel of the second element group different from the first element group to which the saturated pixel belongs, and the first element other than the saturated pixel The image processing apparatus according to any one of (1) to (4), wherein surrounding pixels including a group of pixels are specified.
(6)
The plurality of element groups having different exposure times are regularly arranged according to color components, and the second element group is one color component of the plurality of color components. The image processing apparatus according to any one of (5).
(7)
The image processing apparatus according to any one of (1) to (6), wherein the arrangement of the color components is a Bayer arrangement, and the color component of the second element group is green.
(8)
The image processing apparatus according to any one of (1) to (7), further including a level adjustment unit that adjusts signal levels of a plurality of element groups having different exposure times.
(9)
The image acquired through the image sensor is a mosaic image,
The image processing apparatus according to any one of (1) to (7), wherein the prediction calculation unit performs prediction calculation processing of an image after demosaicing.
(10)
The prediction calculation unit generates and outputs a higher-resolution image from a low-resolution image acquired through the imaging device by the prediction calculation process. Any one of (1) to (9) An image processing apparatus according to 1.
(11)
The prediction calculation unit generates and outputs an image of a second color arrangement different from the image of the first color arrangement acquired via the image sensor by the prediction calculation processing. The image processing apparatus according to any one of (10).
(12)
A class classification unit that classifies pixels of an image to be predicted into a predetermined class based on surrounding pixels that are located in the vicinity of the saturated pixel and that include pixels that include at least an element group different from the element group to which the saturated pixel belongs. In addition,
The prediction calculation unit uses the prediction coefficient corresponding to the predetermined class classified by the class classification unit and the pixel value of the surrounding pixel specified by the surrounding pixel specifying unit, to predict the image The image processing device according to any one of (1) to (11), wherein processing is performed.
(13)
The image processing device
A saturated pixel is detected from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged,
A peripheral pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
An image processing method including a step of performing an image prediction calculation process based on the surrounding pixels.
(14)
Computer
A saturated pixel detection unit for detecting a saturated pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged;
A surrounding pixel specifying unit that specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
The program for functioning as a prediction calculation part which performs the prediction calculation process of an image based on the said surrounding pixel.
(15)
A saturated pixel is detected from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged,
A peripheral pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
A processor capable of executing an instruction to perform prediction calculation processing of an image based on the surrounding pixels;
And a memory for recording the instructions executable by a computer.

  DESCRIPTION OF SYMBOLS 1 Imaging device, 12 Image sensor, 13 Gain adjustment circuit, 14 Demosaic processing circuit, 21 Saturation pixel presence determination part, 22 Prediction tap extraction part, 23 Class classification part, 24 Coefficient generation part, 25 Prediction calculation part, 31 Class tap extraction Section, 32 class code generation section, 81 saturated pixel presence determination section, 82 prediction tap extraction section, 83 class tap extraction section

Claims (20)

A saturated pixel detection unit for detecting a saturated pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged;
A surrounding pixel specifying unit that specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
An image processing apparatus comprising: a prediction calculation unit that performs an image prediction calculation process based on the surrounding pixels. The image processing apparatus according to claim 1, wherein the surrounding pixel specifying unit specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes only pixels of an element group different from the element group to which the saturated pixel belongs. The plurality of element groups having different exposure times include a first element group having a first exposure time and a second element group having a second exposure time shorter than the first exposure time,
The surrounding pixel specifying unit specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes pixels of the second element group different from the first element group to which the saturated pixel belongs. Image processing apparatus. The plurality of element groups having different exposure times are regularly arranged according to color components, and the second element group is one color component of the plurality of color components. Image processing apparatus. The image processing apparatus according to claim 4, wherein the arrangement of the color components is a Bayer arrangement, and the color component of the second element group is green. The image processing apparatus according to claim 1, further comprising a level adjustment unit that adjusts signal levels of the plurality of element groups having different exposure times. The image acquired through the image sensor is a mosaic image,
The image processing apparatus according to claim 1, wherein the prediction calculation unit performs a prediction calculation process on the demosaiced image. The image processing apparatus according to claim 1, wherein the prediction calculation unit generates and outputs a higher-resolution image from a low-resolution image acquired through the imaging element by the prediction calculation process. The prediction calculation unit generates and outputs an image of a second color arrangement different from the image of the first color arrangement acquired through the imaging device by the prediction calculation processing. Image processing apparatus. The image processing apparatus further includes a class classification unit that classifies pixels of an image to be predicted and calculated into a predetermined class based on surrounding pixels that are located in the vicinity of the saturated pixel and include pixels of a device group different from the device group to which the saturated pixel belongs ,
The prediction calculation unit uses the prediction coefficient corresponding to the predetermined class classified by the class classification unit and the pixel value of the surrounding pixel specified by the surrounding pixel specifying unit, to predict the image The image processing apparatus according to claim 1, wherein processing is performed. The surrounding pixel specifying unit is located in the vicinity of the saturated pixel, and is a surrounding pixel composed of a pixel other than the element group to which the saturated pixel belongs and a pixel other than the saturated pixel of the element group to which the saturated pixel belongs. The image processing apparatus according to claim 1. The plurality of element groups having different exposure times include a first element group having a first exposure time and a second element group having a second exposure time shorter than the first exposure time,
The surrounding pixel specifying unit is located in the vicinity of the saturated pixel, the pixel of the second element group different from the first element group to which the saturated pixel belongs, and the first element other than the saturated pixel The image processing apparatus according to claim 11, wherein a surrounding pixel composed of a group of pixels is specified. The plurality of element groups having different exposure times are regularly arranged according to color components, and the second element group is one color component of the plurality of color components. Image processing apparatus. The image processing apparatus according to claim 13, wherein the arrangement of the color components is a Bayer arrangement, and the color component of the second element group is green. The image acquired through the image sensor is a mosaic image,
The image processing apparatus according to claim 1, wherein the prediction calculation unit performs a prediction calculation process on the demosaiced image. The image processing apparatus according to claim 1, wherein the prediction calculation unit generates and outputs a higher-resolution image from a low-resolution image acquired through the imaging element by the prediction calculation process. The prediction calculation unit generates and outputs an image of a second color arrangement different from the image of the first color arrangement acquired through the imaging device by the prediction calculation processing. Image processing apparatus. The image processing device
A saturated pixel is detected from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged,
A peripheral pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
An image processing method including a step of performing an image prediction calculation process based on the surrounding pixels. Computer
A saturated pixel detection unit for detecting a saturated pixel from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged;
A surrounding pixel specifying unit that specifies a surrounding pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
The program for functioning as a prediction calculation part which performs the prediction calculation process of an image based on the said surrounding pixel. A saturated pixel is detected from an image acquired through an image sensor in which a plurality of element groups having different exposure times are regularly arranged,
A peripheral pixel that is located in the vicinity of the saturated pixel and includes a pixel of an element group different from the element group to which the saturated pixel belongs;
A processor capable of executing an instruction to perform prediction calculation processing of an image based on the surrounding pixels;
And a memory for recording the instructions executable by a computer.

Images