JP2013223209A - Image pickup processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image which has had blurring of colors reduced while also cutting back a calculation amount and a circuit scale, as well as securing the quantity of light and improving an S/N ratio.SOLUTION: Odd-numbered lines of a single-plate color image sensor each have first and second color pixels alternately arranged, and even-numbered lines each have first and third color pixels alternately arranged. Pixel values are read out from a plurality of first color pixels disposed in one and the same line alternately in a first exposure time and a second exposure time which is longer than the first exposure time according to the order in which they are arranged. Pixel values are read out from a plurality of second color pixels disposed in one and the same line alternately in the first and the second exposure time according to the order in which they are arranged. Pixel values are read out from a plurality of third color pixels disposed in one and the same line alternately in the first and the second exposure time according to the order in which they are arranged. An image processing unit generates a new moving image with a designated frame rate on the basis of the pixel values read out in the first and the second exposure time.

Description

  The present application relates to an imaging processing apparatus and an imaging processing method for generating a moving image.

  In the conventional imaging processing apparatus, the number of pixels of the imaging element tends to increase for the purpose of achieving high resolution. Since there is a limit to increasing the size of the entire image sensor, each pixel must be reduced in size. On the other hand, as the pixel size is reduced, the amount of light incident on one pixel of the image sensor decreases. As a result, the signal-to-noise ratio (S / N) of each pixel is reduced, and it is difficult to maintain the image quality.

  Patent Document 1 uses three image sensors that detect red, green, and blue light, and processes signals obtained by controlling the exposure time, thereby achieving high resolution, high frame rate, and high sensitivity. It is disclosed that image restoration is realized. In this technology, two types of resolution image sensors are used. One pixel signal is read out by long-time exposure from one image sensor that generates a high-resolution image, and the pixel signal is read out by short-time exposure from the other image sensor that generates a low-resolution image. Thereby, the light quantity was ensured.

  Furthermore, Patent Document 2 extends the method of Patent Document 1 to a single-panel camera, and even with a single-panel camera, it maintains image quality while suppressing a decrease in the signal-to-noise ratio (S / N) of each pixel. Is disclosed.

JP 2009-105992 A Japanese Patent No. 4598162

  The techniques of Patent Documents 1 and 2 have a relatively large amount of computation and circuit scale. For this reason, there has been a demand for development of a device with a smaller amount of calculation and a smaller circuit scale.

  One non-limiting exemplary embodiment of the present application is an imaging process capable of reducing the amount of color blur while reducing the amount of calculation and the circuit scale while ensuring the light amount and improving the S / N. Provide technology.

  An imaging processing apparatus according to an embodiment of the present invention includes a single-plate color imaging device and an image processing unit. In the odd-numbered line of the single-plate color image pickup device, the first color pixel and the second color pixel are alternately arranged, and in the even line, the first color pixel and the third color pixel are alternately arranged and provided on the same line. From a plurality of the first color pixels, it is possible to read pixel values alternately in a first exposure time and a second exposure time longer than the first exposure time according to the arrangement order, and in the same line It is possible to read pixel values from the plurality of second color pixels provided alternately in the first exposure time and the second exposure time according to the arrangement order, and a plurality of pixels provided on the same line. From the third color pixel, pixel values can be alternately read in the first exposure time and the second exposure time according to the arrangement order, and the image processing unit can read the first exposure time. And pixel values read, based on the pixel value read by said second exposure time to generate a new video image of the predetermined frame rate.

  The image processing unit is a moving image of the first color component of the subject obtained based on the pixel value read during the first exposure time and the pixel value read during the second exposure time. , Receiving the moving image of the second color component and the moving image of the third color component, and (a) reading the first color component, the second color component, and the third color read at the first exposure time. A first difference between each moving image of the color component and a moving image obtained by sampling the new moving image at the first exposure time; and (b) the read out at the second exposure time. Including a second difference between the moving images of the first color component, the second color component, and the third color component, and a moving image obtained by sampling the new moving image at the second exposure time. An evaluation formula is set, and a moving image that satisfies the evaluation formula under a predetermined condition is added to the new moving image. It may be obtained as an image.

  The image processing unit may obtain a moving image that minimizes the evaluation formula as the new moving image.

  The image processing unit detects the movement of the subject using the moving images of the first color component, the second color component, and the third color component read out during the first exposure time. A motion detection unit, a moving image that sets a constraint condition term relating to the motion distribution of the new moving image using the motion detection result, and further minimizes the evaluation formula further including the constraint condition term related to the motion distribution An image generation unit that obtains an image as the new moving image may be provided.

  The image processing unit sets a constraint condition term so that a change in pixel value distribution of the new moving image is reduced or a change in pixel value of the new moving image is constant, and the constraint A moving image that minimizes the evaluation expression further including a conditional term may be obtained as the new moving image.

  The image processing unit may set a constraint condition related to the distribution of the pixel values in one direction of the image.

  The image processing unit may acquire the pixel value for each line from the single-plate color imaging device, and set a constraint condition regarding the distribution of the pixel value for each acquired line.

  The imaging processing apparatus applies the moving image of the first color component, the moving image of the second color component, and the moving image of the third color component obtained based on the pixel signal read out during the first exposure time. A display control unit that performs predetermined image processing for display, and a display unit that displays the first moving image, the second moving image, and the third moving image subjected to image processing by the display control unit; May be further provided.

  According to an exemplary embodiment of the present invention, an imaging processing device generates an image with reduced color blurring while ensuring a light amount and improving S / N while reducing a calculation amount and a circuit scale. can do.

1 is a block diagram illustrating a configuration of an imaging processing apparatus 100 according to an exemplary embodiment of the present invention. 2 is a diagram illustrating an example of a single-plate color imaging element 102. FIG. It is a figure which shows the example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. It is a bird's-eye view which shows the exposure time (reading timing) of each pixel which comprises the frame image extended in a xy direction about several frame images. It is a figure which shows another example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. It is a figure which shows another example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. It is a figure which shows another example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. It is a figure which shows another example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. It is a figure which shows another example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. It is a figure which shows another example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. It is a figure which shows another example of arrangement | positioning of a single time exposure pixel and a long time exposure pixel. 2 is a diagram illustrating an example of a detailed configuration of an image processing unit 105. FIG. 4 is a flowchart illustrating a processing procedure of the imaging processing apparatus 100. (A) And (b) is a figure which shows the base frame and reference frame when performing motion detection by block matching. It is a figure which shows the process sequence of a conjugate gradient method. It is a figure which shows the structural example of the image generation part 202 for performing the process shown in FIG. It is a figure which shows the process sequence of a quasi-Newton method. It is a schematic diagram for demonstrating the vertical vector which makes each pixel value of a moving image an element. 2 is a block diagram illustrating a configuration of an imaging processing apparatus 200 having an image processing unit 105 that does not include a motion detection unit 201. FIG. It is a block diagram which shows the structure of the imaging processing apparatus 400 by other exemplary embodiment of this invention. FIG. 10 is a conceptual diagram of processing for performing interpolation processing on each color image of R, G, and B constituting a short-time exposure image and then generating each color image for reducing and displaying each color image on the display unit 802. FIG. 10 is a conceptual diagram of processing for generating each color image for display on a display unit 802 by performing interpolation processing and aspect ratio changing processing on each color image of R, G, and B constituting a short-time exposure image.

  First, regarding the above-described Patent Documents 1 and 2, problems of the prior art found by the inventors will be described.

  In the above-mentioned Patent Documents 1 and 2, the constraint conditions in the two directions of the x direction and the y direction are used as the smoothness constraint. This requires a two-dimensional memory access for the image. As a result, in Patent Documents 1 and 2, the amount of calculation and the circuit scale have increased.

  Hereinafter, an imaging processing apparatus and the like according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram illustrating a configuration of an imaging processing apparatus 100 according to the present embodiment. The imaging processing apparatus 100 includes an optical system 101, a single plate color imaging element 102, a readout control unit 103, a control unit 104, and an image processing unit 105. Hereinafter, each component of the imaging processing apparatus 100 will be described in detail.

  The optical system 101 is, for example, a camera lens, and forms an image of a subject on the image plane of the image sensor.

  The single-plate color image sensor 102 is an image sensor in which a color filter array of red (R), green (G), and blue (B) is attached to each pixel. FIG. 2 shows an example of the single-plate color image sensor 102. Corresponding to each pixel (photodiode) of the single-plate image pickup device, an array of color filters called a Bayer array is mounted. In the figure, “R”, “G”, and “B” indicate “red”, “green”, and “blue” filters, respectively.

  Each pixel of the single-plate color image sensor 102 receives the light (optical image) connected by the optical system 101 through a red, green or blue filter provided corresponding to each. Each pixel performs photoelectric conversion on a pixel basis, and outputs a pixel value (pixel signal) corresponding to the amount of light incident on each pixel. An image for each color component is obtained from pixel signals of the same color component that are captured at the same frame time. A color image is obtained from images of all the color components.

  Hereinafter, in the present specification and drawings, pixels that detect red, green, and blue light are represented by “R”, “G”, and “B”, respectively. A pixel signal output from each pixel of the image sensor has a pixel value related to any of R, G, and B colors.

  As is apparent from this description, each pixel of the image sensor is a unit of the image sensor that receives light transmitted through the color filters of each color and outputs a signal corresponding to the intensity of the received light.

  The readout control unit 103 reads out an image signal for each predetermined exposure time from the single-plate color imaging device 102 having the arrangement shown in FIG. 2 and acquires it as a moving image. The reading method is as described in detail below. The read operation of the read control unit 103 is executed based on control by the control unit 104.

  FIG. 3 shows an example of the arrangement of the short exposure image and the long exposure image. The pixel signal is read out in units of one line. In FIG. 3, the subscript S for R, G, and B indicates short-time exposure, and the subscript L indicates long-time exposure. In the present specification and drawings, the subscript “S” may represent short exposure, and the subscript “L” may represent long exposure.

  In the specification of the present application, “short-time exposure” means exposure for a time of one frame during general moving image shooting. For example, “short exposure” corresponds to exposure with an exposure time of 1/30 second when the frame rate is 30 frames / second (30 fps). When the frame rate is 60 frames / second (60 fps), it is 1/60 seconds. On the other hand, the long exposure means that the exposure is performed for a time longer than the time required for one frame exposure, for example, about 2 to 10 frames.

  According to the configuration shown in FIG. 3, the following (a) and (b) are understood. That is, (a) G and R are alternately arranged when paying attention to odd lines, and G and B are arranged alternately when paying attention to even lines. Then, (b) paying attention to the same color pixel (R pixel, G pixel or B pixel) on one line parallel to the x axis, the exposure time (or readout time) is “short” → “long” → “short” "→" Long "... alternately.

  FIG. 4 schematically shows the exposure time of each pixel in FIG. The horizontal direction of the drawing is the time axis t. The long exposure image is half of each pixel of R / G / B. In addition, the G pixel and R pixel lines and the B pixel and G pixel lines are repeated every two lines.

  The readout control unit 103 switches the exposure time for each pixel by controlling the readout pulse signal and the reset signal for each pixel of the single-plate color imaging element 102 based on the control signal of the control unit 104. At the time of short exposure, the readout control unit 103 reads out a pixel value every frame time and reads out a pixel signal of one frame image. Further, at the time of long exposure, the readout control unit 103 exposes the single-plate color image sensor 102 over a plurality of frame times, and reads out the pixel signal after the exposure period ends.

  Note that the readout control unit 103 may add pixel signals after reading them out from the element. Specifically, the readout control unit 103 reads out pixel signals from all pixel lines every frame period. And about a certain pixel, after the pixel signal for a continuous several frame is obtained, they are added. At this time, pixel signals having the same pixel coordinate value in a plurality of frames are added. As a result, for the pixel to which the pixel signal is added, the readout control unit 103 can output a pixel signal that is substantially the same as the pixel signal of the pixel that has been exposed over a plurality of frame periods.

  Note that the frame period, which is the exposure time of the long exposure pixels, is determined in advance.

  Further, the pixel arrangement in the imaging processing apparatus of the present application is not limited to FIG. For example, the pixel arrangement and exposure time (or readout time) as shown in FIGS. 5A to 5G may be employed. This pixel arrangement will be described later.

  Refer to FIG. 1 again.

  The image processing unit 105 receives moving image data captured at different exposure times for each pixel, and performs image processing on these to estimate pixel values (for example, R, G, and B pixel values) at each pixel. Then, a high-resolution color moving image is generated.

  FIG. 6 shows an example of a detailed configuration of the image processing unit 105. The image processing unit 105 includes a motion detection unit 201 and an image generation unit 202.

  FIG. 7 is a flowchart illustrating a processing procedure of the imaging processing apparatus 100. First, an outline will be described.

  In step S <b> 401, the image processing unit 105 acquires moving images having different frame rates for each pixel from the read control unit 103.

  In step S402, the motion detection unit 201 of the image processing unit 105 detects the motion for each pixel of the generated image using the short-time exposure image of the acquired images.

  In steps S403 and S404, the image generation unit 202 sets motion constraint conditions and sets smoothness constraints.

  In step S405, the image generation unit 202 generates a high frame rate color image.

  In step S406, the image generation unit 202 generates a color image by performing a known demosaicing process using the generated RGB image.

  In step S407, the image generation unit 202 outputs the generated moving image.

  Details of each step described above will be described below.

  The motion detection unit 201 receives an image (short-time exposure image) formed by pixel signals obtained by exposure for one frame period. The motion detection unit 201 uses a short-time exposure image to detect a motion (optical flow) from a pixel value captured by short-time exposure by an existing known technique such as block matching, a gradient method, or a phase correlation method. Examples of known techniques include P.I. Anandan. “Computational framework and an algorithm for the measurement of visual motion”, International Journal of Computer Vision, Vol. 2, pp. 283-310, 1989 is known.

  FIGS. 8A and 8B show a base frame and a reference frame when motion detection is performed by block matching. The motion detection unit 201 sets a window area A shown in FIG. 8A in a reference frame (an image at time t when attention is to be obtained for motion), and is similar to a pattern in the window area. In the reference frame. As the reference frame, for example, a frame next to the frame of interest is often used.

As shown in FIG. 8B, the search range is normally set in advance with a certain range C based on the position B where the movement amount is zero. In addition, the similarity (degree) of the pattern is determined by the residual sum of squares (SSD: Sum of Square Differences) shown in (Equation 1) or the absolute sum of residuals (SAD: Sum of Absorbed Differences) shown in (Equation 2). ) Is calculated as an evaluation value.

In (Equation 1) and (Equation 2), I (x, y, t) is a spatio-temporal distribution of an image, that is, a pixel value, and x, yεW is a pixel included in the window region of the reference frame. The coordinate value of In the present embodiment, using the pixel value of the image captured by short-time exposure in FIG. 3, the image is interpolated and enlarged for each color of RGB, and then the luminance component is obtained by the following Equation 3 to perform motion detection.

  I (x, y, t) is not limited to (Equation 3), but may be a simple addition of RGB values or the like.

  The motion detection unit 201 searches for a set of (u, v) that minimizes the evaluation value by changing (u, v) within the search range, and sets this as a motion vector between frames. Specifically, by sequentially shifting the set position of the window area, a motion is obtained for each pixel or each block (for example, 8 pixels × 8 pixels), and a motion vector is generated.

  For the distribution of the value of (u, v) in the vicinity of (u, v) that minimizes (Equation 1) or (Equation 2) obtained as described above, a linear or quadratic function is obtained. By applying (a known technique known as an equiangular fitting method or a parabolic fitting method), motion detection with sub-pixel accuracy is performed.

<Generation processing of R, G, B pixel values in each pixel>
The image generation unit 202 minimizes an evaluation function J (evaluation expression) represented by the following expression, and calculates R, G, and B pixel values for each pixel.

Here, H ₁ is a sampling process for long exposure, H ₂ is a sampling process for short exposure, and f is an RGB image having a high spatial resolution and a high time resolution to be newly generated. Λ _L and λ _S are weights. Of the RGB images captured by the single-plate color image sensor 102, g _{L is} an image captured by long exposure and g _S is an image captured by short exposure. M is a power index and Q is a condition to be satisfied by the image f to be generated, that is, a constraint condition.

  The value of the exponent M in (Expression 4) is not particularly limited, but 1 or 2 is preferable from the viewpoint of the amount of calculation.

The first term of (Expression 4) is an image obtained by sampling the RGB image f having a high spatial resolution and a high temporal resolution to be generated by the long exposure sampling process H ₁ , and the actual long exposure image. Is obtained by calculating the difference from g _L. If a sampling process H ₁ for long exposure is determined in advance and f for minimizing this difference is obtained, it can be said that f is the best match with g _L obtained by long exposure. Similarly, for the second term, f which minimizes the difference can be said to be the best match with g _S obtained by short-time exposure. The third term is a constraint condition term for uniquely determining the solution f.

And it can be said that the RGB image f that minimizes the evaluation function J of (Equation 4) satisfies both g _L and g _S obtained by the long exposure and the short exposure comprehensively. The image generation unit 202 generates an RGB image pixel value with improved S / N by calculating an RGB image f that minimizes (Equation 4).

The minimization of J in (Equation 4) is performed by obtaining an RGB image f where ∂J / ∂f = 0. In this calculation, when M = 2 in (Equation 4) and Q is a quadratic form of f, the partial differential ∂J / ∂f by f of J is a linear expression of f. It becomes a linear optimization problem, resulting in the calculation of simultaneous equations Af = b. That is, when (Equation 4) is partially differentiated by f, it is as follows.

Q ₁ is a constant matrix (matrix not including f) when Q is a quadratic form of f.

The RGB image f is calculated by transforming this equation and solving the following simultaneous equations.

  On the other hand, when M = 1, when M = an integer of 3 or more, and when M is not an integer, a nonlinear optimization problem occurs.

  When the calculation becomes a linear optimization problem, the RGB image f that minimizes the evaluation function J can be calculated by the procedure of the conjugate gradient method shown in FIG. FIG. 10 shows a configuration example of the image generation unit 202 for performing the processing shown in FIG. The image generation unit 202 includes a coefficient calculation unit 501, a vector calculation unit 502, and a calculation unit 503.

The coefficient calculation unit 501 calculates a coefficient matrix A. The vector calculation unit 502 calculates a constant vector b. The calculation unit 503 uses the matrix A and the vector b obtained by the calculation to solve simultaneous equations of Af = b. The process shown in FIG. 9 is mainly the process of the calculation unit 503. When the matrix A and the vector b are obtained and an initial value and an end condition are given from the control unit 104, the process of FIG. 9 is started. The end condition in FIG. 9 is when the residual r _{k + 1} at the (k + 1) th step shown in FIG. 9 is sufficiently small.

  Thereby, the G image f that minimizes the evaluation function J can be obtained. Since the algorithm shown in FIG. 9 is known, detailed description is omitted.

On the other hand, when the calculation becomes a nonlinear optimization problem, f that minimizes J can be calculated by the procedure of the quasi-Newton method shown in FIG. The minimization and minimization of J by this method starts with an initial solution of f, and updates the solution f by iterative calculation so as to make J smaller by using the gradient of J with respect to f. In FIG. 11, Kk is set as s _k ≡f _{k + 1} −f _k and y _k = ∇J (f _k ), and is calculated as follows in the DFP method in the quasi-Newton method.

In the BFGS method in the quasi-Newton method, calculation is performed as follows.

  Hereinafter, (Equation 4) will be described in more detail.

The images f, g _L and g _S are vertical vectors whose elements are the pixel values of the moving image. In the following, the vector notation for an image means a vertical vector in which pixel values are arranged line by line and RGB for each RGB as shown in FIG. 12, and the function notation means a spatio-temporal distribution of pixel values. As a pixel value, in the case of a luminance value, one value may be considered per pixel. The number of elements of f is, for example, 2000 × 1000 × 30 = 60000000 if the moving image to be generated is 2000 pixels wide, 1000 pixels long, and 30 frames.

  The number of pixels in the vertical and horizontal directions of the image f and the number of frames used for signal processing are set by the image processing unit 105.

The long exposure sampling process H ₁ is a matrix in which the number of rows is equal to the number of elements of g _{L and} the number of columns is equal to the number of elements of f. The short exposure sampling process H ₂ is a matrix whose number of rows is equal to the number of elements of g _{S and} whose number of columns is equal to the number of elements of f.

  Since the amount of information relating to the number of pixels of a moving image (for example, 2000 pixels wide × 1000 pixels in height) and the number of frames (for example, 30 frames) is too large in a computer that is currently in widespread use, f (4) is minimized. Cannot be obtained in a single process. In the imaging processing apparatus of the present application, as will be described later, it is possible to process each RGB color only for each line (for example, a width of 1000 pixels × a height of 1 pixel), so that memory access can be greatly reduced. it can.

  Thus, the moving image f to be generated can be calculated by repeating the process of obtaining a part of f for the temporal and spatial partial regions.

Next, formulation of the long exposure sampling process H ₁ will be described using a simple example. In the following, a single-plate color imaging device having a Bayer arrangement for an image having a width of 4 pixels (x = 1, 2, 3, 4), a height of 2 pixels (y = 1, 2), and 2 frames (t = 1, 2) Consider an imaging process of g _L when imaging is performed at 102 and time is added for two frames in a line of y = 1. Here, for example, corresponding to the upper left four pixels × 2 pixels in FIG. 3, two pixels (x, y) = (2, 2) where y = 2 and _GL in (4, 2) Assume t = 1,2.

Elements of the obtained G image f are expressed as in the following (Equation 9) including t = 1 and 2.

In (Equation 9), G _{111 to} G ₃₁₂ , R _{211 to} R ₄₂₂ , and B _{121 to} B ₃₂₂ indicate the R, G, and B pixel values in each pixel, and the three subscripts are x, y, and t in order. Indicates the value.

Of the above (Equation 9), in order to obtain a long exposure, R ₂₁₁ and R ₂₁₂ are required for the position (2, 1), G ₃₁₁ and G ₃₁₂ are specified for the position (3, 1), and the position (1, for 2) B ₁₂₁ and B _122, a G ₂₂₁ and G ₂₂₂ for position (2,2). Therefore, a matrix for extracting only those pixel values can be defined as follows.

The matrix H ₁ is set in consideration of the sampling process of long exposure. That is, in order to obtain the long-time exposure pixels in the lines of y = 1 and 2, each pixel value of t = 1, 2 at the positions (x, y) = (2, 1) and (3, 1) And the pixel values of t = 1 and 2 at the positions (x, y) = (1, 2) and (2, 2) need to be added. According to these, the imaging process of the long exposure image is formulated as follows.

Next, formulation of the sampling process H ₂ for short time exposure will be described using a simple example. As in the previous example, an image with a width of 4 pixels (x = 1, 2, 3, 4), a height of 2 pixels (y = 1, 2), and 2 frames (t = 1, 2) is captured in a Bayer array. Consider the case of imaging with an element.

Similar to the example of the previous sampling process H ₁ , this example also assumes the upper left 4 pixels × 2 pixels of FIG. However, since it is a short time exposure, the pixel values of the short time exposure pixels at y = 1, 2 in FIG. 3 are required for t = 1 and 2. That is, the pixel value of Gs in the pixel (x, y) = (1, 1), the pixel value of Rs in (4, 1), the pixel value of Bs in (3, 2), and (4 , 2) the pixel value of Gs is required for t = 1 and 2. The elements of the obtained RGB image f are as shown in (Equation 9).

Of the above (Equation 9), in order to obtain short-time exposure, G ₁₁₁ and G ₁₁₂ for the position (1,1), R ₄₁₁ and R ₄₁₂ for the position (4,1), and the position (3,3) for 2) B ₃₂₁ and B _322, a G ₄₂₁ and G ₄₂₂ for position (4,2). Therefore, a matrix for extracting only those pixel values can be defined as follows.

The first to fourth lines of (Equation 12) relate to time t = 1, and the fifth to eighth lines relate to time t = 2. According to these, the imaging process of the short time exposure line is formulated as follows.

(11) and (Equation 13), an RGB image f captured by changing the exposure time by a single-plate image pickup device of the Bayer array, showing a process of obtaining g _L, g _S. Conversely, the problem of generating f from g _L and g _S is generally called an inverse problem. In the absence of the constraint condition Q, there are an infinite number of fs that minimize (Equation 14) below.

  This can be easily explained because (Equation 14) holds even if an arbitrary value is entered as a pixel value that is not sampled. Therefore, the RGB image f cannot be uniquely solved by minimizing (Equation 14).

  Therefore, in order to obtain a unique solution for the RGB image f, a constraint condition Q is introduced. As Q, a constraint on smoothness regarding the distribution of the pixel value f and a constraint on smoothness regarding the distribution of motion of the image obtained from f are given. Hereinafter, how to give each constraint will be described.

First, as a constraint on smoothness of the RGB image f regarding the distribution of pixel values for the first line G (hereinafter referred to as Gr) and R, and the second line B and G (hereinafter referred to as Gb), The constraint equation of (Expression 15) or (Expression 16) is used.

Here, λ _CR , λ _CGr , λ _CGb , and λ _CB are weights for restraining the smoothness of R, G, and B, and when pixel values read out from all pixels by three-plate photography can be prepared in advance. Is determined so that the PSNR (Peak Signal to Noise Ratio) of the generated image is the best.

PSNR is defined by (Equation 17).

  Here, fmax is the maximum pixel value (for example, 255 for 8 bits and 1023 for 10 bits), and σ is a standard deviation of the difference between the correct image and the generated image.

On the other hand, if not, λλ _CR , λ _CGr , λ _CGb , and λ _CB may be _determined while looking at the image quality newly generated by manual operation based on the above values. As shown in (Expression 4), the constraint condition Q also constitutes a part of the evaluation function J. Therefore, in order to make the value of the evaluation function J as small as possible, it is preferable that the constraint condition Q is also small.

  Regarding (Expression 15), the smaller the change in the distribution of the pixel value f, the smaller the Q value. Regarding (Expression 16), the value of Q decreases as the change in the distribution of the pixel value f is constant. These are the smoothness constraint conditions regarding the distribution of the pixel value f.

  In (Equation 14), || represents the norm of the vector. The value of the power exponent m is preferably 1 or 2 from the viewpoint of the amount of computation, like the power exponent M in (Equation 4) and (Equation 14).

Note that the partial differential values ∂ / ∂x and ∂ ² / ∂x ² can be approximated by, for example, (Equation 18) by differential development using pixel values near the pixel of interest.

The difference development is not limited to the above (Equation 18), and other pixels in the vicinity may be referred to, for example, as shown in (Equation 19).

(Equation 19) is averaged in the vicinity of the calculated value of (Equation 18). Thereby, although spatial resolution falls, it can make it hard to receive the influence of a noise. Furthermore, as an intermediate between them, weighting may be performed with α in a range of 0 ≦ α ≦ 1, and the following formula may be employed.

  The difference expansion calculation method may be performed by predetermining α according to the noise level so that the image quality of the processing result is further improved, or in order to reduce the calculation amount and the circuit scale as much as possible ( Equation 18) may be used.

  As shown in (Equation 15) and (Equation 16), for the problem of solving (Equation 4) by introducing smoothness constraints on the distribution of pixel values of the moving image f, a known solution, for example, finite It can be calculated using a solution to a variational problem such as the element method.

  In addition, as described above, the RGB image f can be calculated independently for G (hereinafter referred to as Gr) and R on the first line, and B and G (hereinafter referred to as Gb) on the second line. Processing can be performed with only one-dimensional memory access, and the amount of computation and circuit scale can be reduced. For example, if a moving image to be generated is 2000 pixels wide and 1000 pixels long, each of Gr / R / B / Gb can be processed for every 1000 pixels in width × 1 pixel in height. Therefore, memory access can be greatly reduced.

  Next, a description will be given of the constraint on smoothness relating to the motion distribution of the image included in the RGB image f.

The following (Equation 21) or (Equation 22) is used as the smoothness constraint regarding the motion distribution of the image included in the RGB image f.

  Here, u is a vertical vector having the x-direction component of the motion vector for each pixel obtained from the moving image f as an element, and v is a y-direction component of the motion vector for each pixel obtained from the moving image f. Is a vertical vector.

The smoothness constraint relating to the motion distribution of the image obtained from f is not limited to (Equation 21) and (Equation 22). For example, the directional differentiation of the first or second floor shown in (Equation 23) or (Equation 24). It is good.

Furthermore, as shown in (Equation 25) to (Equation 28), the constraint conditions of (Equation 21) to (Equation 24) may be adaptively changed according to the gradient of the pixel value of f. As a result, the effect of smoothness constraint on the edge (border where the luminance change is discontinuous) in the image can be reduced compared to the flat part of the image, so that the edge in the image can be reproduced more clearly. Can do.

Here, w (x, y) is a weight function related to the gradient of the pixel value of f, and is defined by the sum of powers of the gradient component of the pixel value shown in (Equation 29).

The weight function w (x, y) may be defined by the magnitude of the power of the directional differentiation shown in (Equation 30) instead of the square sum of the components of the luminance gradient shown in (Equation 29).

Here, the vector n _max and the angle θ are directions in which the directional differentiation is maximized, and are given by the following (Equation 31).

  By introducing such a weight function, it is possible to prevent the motion information of f from being smoothed more than necessary, and as a result, the newly generated image f is smoothed more than necessary. Can be prevented.

  As shown in (Equation 21) to (Equation 28), with respect to the problem of solving (Equation 4) by introducing the smoothness constraint on the motion distribution obtained from the image f, the smoothness constraint on f is applied. Complicated calculation is required compared with the case of using. This is because the image f to be newly generated and the motion information (u, v) depend on each other.

  This problem can be calculated using a known solution, for example, a variational solution using an EM algorithm or the like. At that time, the image f to be newly generated and the initial value of the motion information (u, v) are necessary for repeated calculation.

  As an initial value of f, an interpolation enlarged image of the input image may be used. On the other hand, as the motion information (u, v), the motion information obtained by calculating (Equation 1) to (Equation 2) in the motion detection unit 201 is used. As a result, as described above, the image processing unit 105 introduces the constraint on smoothness regarding the motion distribution obtained from the image f as shown in (Expression 21) to (Expression 28), and By solving, the image quality of the processing result can be improved.

The processing in the image processing unit 105 is one of the smoothness constraints related to the distribution of pixel values shown in (Equation 15) and (Equation 16), and the smoothness related to the motion distribution shown in (Equation 21) to (Equation 28). Any one of the above constraints may be combined and used simultaneously as in (Expression 32).

Here, Q _f is a smoothness constraint on the gradient of the pixel value of f, Q _uv is a smoothness constraint on the distribution of image motion obtained from f, and λ ₁ and λ ₂ are related to the constraint between Q _f and Q _uv . It is a weight.

  Although a plurality of constraint condition expressions have been described here, the optimal combination of these may be determined according to the status of the apparatus such as cost, operation scale, circuit scale, and the like. For example, what has the best image quality may be selected. “The image quality is the best” can be determined based on the subjectivity of the person who confirms the video, for example.

  The problem of solving (Equation 4) by introducing both the smoothness constraint on the distribution of pixel values and the smoothness constraint on the distribution of image motion is a variational problem using a known solution (for example, an EM algorithm). Can be used.

In addition, the constraints on motion are not limited to those relating to the smoothness of the motion vector distribution shown in (Equation 21) to (Equation 28), but the residual between corresponding points (the pixel value between the start point and the end point of the motion vector). You may make it make this small by making an evaluation value into (difference). The residual between corresponding points can be expressed as (Expression 33) when f is expressed as a function f (x, y, t).

Considering the entire image with f as a vector, the residual in each pixel can be expressed as a vector as shown in the following (Equation 34).

The sum of squares of the residuals can be expressed as shown below (Equation 35).

In (Equation 34) and (Equation 35), H _m is a matrix of the number of elements of vector f (total number of pixels in space-time) × f. In H _m , in each row, only elements corresponding to the viewpoint and end point of the motion vector have non-zero values, and other elements have zero values. When the motion vector has integer precision, the elements corresponding to the viewpoint and the end point have values of −1 and 1, respectively, and the other elements are 0.

  When the motion vector has sub-pixel accuracy, a plurality of elements corresponding to a plurality of pixels near the end point have values according to the value of the sub-pixel component of the motion vector.

(Equation 35) may be set as Q _m and the constraint condition may be as shown in (Equation 36).

Here, λ ₃ is a weight related to the constraint condition Q _m .

By using the motion information extracted from the short-time exposure images of G _S , R _S, and B _S by the motion detection unit 201 by the method described above, the RGB moving image captured by the Bayer array image sensor is converted into the image processing unit. In 105, the signal-to-noise ratio (S / N) can be improved, and a new moving image with less color blur can be generated.

  Further, the image processing unit 105 improves the signal-to-noise ratio (S / N) by generating a known demosaicing process using the RGB image thus generated, and generates a new moving image with less color blur. be able to. Since the imaging processing apparatus of the present application performs processing for each line, processing can be performed only by one-dimensional memory access, and the amount of computation and circuit scale can be reduced.

  In addition, since the imaging processing apparatus of the present application generates only an image at the RGB pixel position in the Bayer array and generates a color image by demosaicing processing on the generated image, compared with a case where image generation is performed with all pixels. Therefore, the S / N can be further improved without causing color misregistration while reducing the calculation amount and the circuit scale.

  Note that the above-described calculation method of the R, G, and B pixel values in the image processing unit 105 is an example, and other calculation methods may be employed.

  As another method, for example, a desired target image may be obtained using an iterative calculation type optimization method such as the steepest gradient method.

  As described above, according to the first embodiment, the function of time addition is added to the single-plate imaging device, and a new image is generated from the input image that is time-added for each line, thereby reducing the amount of light during imaging. It is possible to estimate and generate an image with a high resolution and a high frame rate with little color blurring (an image in which all pixels are read without performing time addition) while ensuring.

(Modification of Embodiment 1)
In the first embodiment, the processing in the image processing unit 105 uses all of the degradation constraint using (Equation 10) and (Equation 12), the motion constraint using motion detection, and the smoothness constraint relating to the distribution of pixel values. Said mainly.

  Of the various constraints, the motion constraint is particularly computationally intensive and requires computer resources. Therefore, in the present modification, a process that does not use a motion constraint among these constraints will be described.

  FIG. 13 is a block diagram illustrating a configuration of the imaging processing apparatus 200 including the image processing unit 105 that does not include the motion detection unit 201. The image generation unit 111 of the image processing unit 105 generates a new image without using motion constraints.

  In FIG. 13, the same reference numerals as those in FIG. 1 and FIG. 6 are given to the components that perform the same operations as those in FIG. 1 and FIG.

  In the conventional technique, when the motion constraint is not used, the image quality is clearly deteriorated in the processing result.

  However, in the present invention, it is possible to eliminate motion restriction without causing a significant deterioration in image quality. The reason is the pixel arrangement shown in FIG. In this pixel arrangement, when generating an RGB image, the pixels of the respective color components that perform long exposure and short exposure of the single-plate color image sensor 102 are sequentially scanned by scanning in units of one line in a horizontal direction. It is mixed. For this reason, pixels captured by short exposure and pixels captured by long exposure are mixed in order, and when an image is generated without using motion constraints, the pixel value captured by short exposure is blurred. There is an effect to suppress the occurrence of. Furthermore, since a new moving image is generated without imposing a motion constraint condition, the amount of calculation and the circuit scale can be reduced.

  Hereinafter, the image quality improvement processing by the image generation unit 111 will be described.

In (Equation 4), M = 2, and (Equation 15) to (Equation 16) are used as Q, and m in these equations is 2. When any one of (Expression 18), (Expression 19), and (Expression 20) is used as the differential expansion of the first-order differentiation and the second-order differentiation, the evaluation expression J becomes a quadratic expression of f. The calculation of f that minimizes the evaluation formula results in the calculation of simultaneous equations for f by (Equation 37).

Here, the simultaneous equations to be solved are set as (Equation 38).

  In (Equation 38), since f has elements for the number of pixels to be generated (the number of pixels in one frame × the number of frames to be processed), the calculation amount of (Equation 38) is usually very large. However, as described above, the imaging processing apparatus of the present application can process each Gr / R / B / Gb only for each line (for example, width 1000 pixels × height 1 pixel × number of frames to be processed). Therefore, memory access can be greatly reduced, and the amount of computation and the circuit scale can be reduced.

  As a method for solving such simultaneous equations, a method (an iterative method) for converging the solution f by an iterative calculation such as a conjugate gradient method or a steepest descent method is generally used.

  When obtaining f without using motion constraints, the evaluation functions are only the degradation constraint term and the smoothness constraint term, so the processing does not depend on the content. By utilizing this fact, the inverse matrix of the coefficient matrix A of the simultaneous equations (Equation 38) can be calculated in advance, and by using this, image processing can be performed by a direct method.

When the smoothness constraint shown in (Equation 18) is used, the second-order partial differentiation in the x direction becomes, for example, a filter of three coefficients 1, -2, and 1 as shown in (Equation 18), and its square is It becomes a filter of five coefficients 1, -4, 6, -4, and 1. These coefficients can be diagonalized by sandwiching a coefficient matrix between horizontal Fourier transform and inverse transform. Similarly, the long-time exposure deterioration constraint can be diagonalized by sandwiching a coefficient matrix between time-wise Fourier transform and inverse Fourier transform. That is, the matrix can be set as Λ as in (Equation 39).

Thereby, the number of non-zero coefficients per row can be reduced as compared with the coefficient matrix A. As a result, the calculation of the inverse matrix Λ ⁻¹ of Λ becomes easy. Then, according to (Equation 40) and (Equation 41), f can be obtained by a direct method with less calculation amount and circuit scale without performing repeated calculation.

  Further, by using (Equation 14) as a constraint on smoothness, the G pixel Gr in the odd row, the R pixel in the odd row, the B pixel in the even row, and the G pixel Gb in the even row are calculated independently. Therefore, the S / N can be further improved without causing color misregistration while reducing memory access, reducing the amount of calculation and the circuit scale. In this case, the imaging processing apparatus of the present application can perform processing only with pixels of each color for each line (for example, 1000 pixels wide × 1 pixel high × number of frames to be processed), so that memory access is greatly reduced. In addition, the calculation amount and the circuit scale can be reduced.

  Further, the pixel arrangement in the imaging processing apparatus of the present application is not limited to that in FIG. 3, and any of FIGS. As described above, in the imaging processing apparatus of the present application, in each color pixel of the single-plate color imaging element 102, it is necessary that the pixels that perform long-time exposure and short-time exposure in the x direction are mixed in order. 5A-5G satisfy this condition.

(Embodiment 2)
In recent years, electronic viewfinders have begun to be used in many devices. However, when such an electronic viewfinder is realized by the technique described in Patent Document 1, it is difficult to display an image without color blur or delay on the viewfinder at the time of shooting. The reason is that all pixels are exposed for a long time for a certain color.

  On the other hand, in Embodiment 2 of the present invention, some or all of the pixels from the first color to the third color are photographed with short-time exposure. For this reason, in the present embodiment, information on pixels captured by short-time exposure is displayed on the viewfinder, so that an image free from delay and color blur is presented to the photographer.

  FIG. 14 is a block diagram illustrating a configuration of the imaging processing apparatus 400 according to the present embodiment. 14 that operate in the same manner as in FIG. 1 are assigned the same reference numerals as in FIG. 1 and description thereof is omitted.

  The imaging processing apparatus 400 includes an optical system 101, a single-plate color imaging element 102, a readout control unit 103, a control unit 104, an image processing unit 105, a display control unit 801, and a display unit 802. .

  A pixel signal captured by the single-plate color image sensor 102 for a short exposure time is input to the display control unit 801.

  The display control unit 801 performs processing for display on the pixel signal of the input short-time exposure image. For example, the display control unit 801 aligns the sampling positions of RGB colors and adjusts the aspect ratio by a simple interpolation process. In the imaging processing apparatus of the present application, among the RGGB4 pixels, RGB3 pixels are photographed with short-time exposure. Therefore, the display control unit 801 performs processing for display by performing interpolation processing on these short-time exposure pixels.

  As an example, FIG. 15 performs interpolation processing on each of the R, G, and B color images constituting the short-time exposure image, and then generates each color image to be displayed on the display unit 802 by reducing each color image. It is a conceptual diagram of a process. As another example, FIG. 16 generates each color image to be displayed on the display unit 802 by performing an interpolation process and an aspect ratio changing process on each of the R, G, and B color images constituting the short-time exposure image. It is a conceptual diagram of a process. The display control unit 801 performs processing as shown in FIGS. 15 and 16, and then performs alignment processing for aligning the sampling position on each color image, and displays it on the display unit 802.

  The photographer can perform composition and focus adjustment by looking at the displayed image.

  As described above, the imaging processing apparatus according to the present invention has an optical system, an imaging device, a readout control unit, and an image processing unit, and is useful as a technique for high-resolution and high-sensitivity imaging using a small imaging device.

  In the above-described embodiment, the read control unit 103, the control unit 104, the image processing unit 105, and the display control unit 801 are illustrated and described as different components. Each of these can be implemented by a separate processor. However, the configuration is an example. The functions of two or more components may be realized by a single processor. Of course, the functions of all the components may be realized by a single processor. In that case, for example, a program module for performing an operation corresponding to the operation of each component described above may be prepared, and any one or a plurality of program modules may be executed at a certain point in time. Thereby, the operation of each component described above is realized by one processor.

  According to the exemplary embodiment described above, the single-chip color imaging device of the imaging processing device has a plurality of first pixels (from which pixel signals obtained by exposure with an exposure time corresponding to a predetermined frame rate are read out) Short exposure pixels) and a plurality of second pixels (long exposure pixels) from which pixel signals obtained by exposure with an exposure time longer than the predetermined frame rate are read out. This single-plate color imaging device is provided with a plurality of pixels for detecting light of the first color component, the second color component, and the third color component by each of the first pixel and the second pixel. By using this single-plate color imaging device, a pixel signal of a moving image at a certain frame rate and a pixel signal of a moving image captured at an exposure time longer than the frame rate are obtained for each color component for each line. It is done. Thereby, it is possible to obtain an image signal in which the color blur caused by the movement of the subject is suppressed as compared with the case where the moving images of the first color component, the second color component, and the third color component are always taken by long exposure. Can do.

  According to another exemplary embodiment of the present application, the imaging processing device includes two types of pixels, two types of pixels, that is, a long-time exposure pixel and a short-time exposure pixel. Read signal from. At this time, in each color component, the pixels are arranged so that the long-time exposure pixels and the short-time exposure pixels are sequentially photographed in the readout direction. By performing an image restoration process on the readout signal in such a pixel arrangement, an image signal that suppresses color bleeding caused by the movement of the subject compared to a case where the image signal is obtained by only long-time exposure. Can be obtained.

  Note that the image processing unit 105 in this specification may not be realized as an apparatus. For example, the above-described operation of the image processing unit 105 may be performed by a general-purpose processor that is a computer executing a computer program recorded on a computer-readable recording medium. Such a computer program includes, for example, a group of instructions for causing a computer to execute processing realized by the flowchart of FIG. The computer program is recorded on a recording medium such as a CD-ROM and distributed as a product to the market, or transmitted through an electric communication line such as the Internet.

  In the above-described embodiment, it has been described that a color moving image is captured with three colors of R, G, and B, and only G is captured with both short-time exposure and long-time exposure. However, how to separate these color components is an example. For example, three color filters of cyan, magenta, and yellow may be used. Further, instead of three colors, filters of two colors or four colors or more may be used.

  The imaging apparatus and processing apparatus of the present invention are useful for high-resolution imaging at low light amounts and imaging with small pixels. It can also be applied as a program.

DESCRIPTION OF SYMBOLS 100 Image pick-up processing apparatus 101 Optical system 102 Single plate color image sensor 103 Reading control part 104 Control part 105 Image processing part

Claims (8)

An image processing apparatus including a single-plate color image sensor and an image processing unit,
In the odd-numbered line of the single-plate color image sensor, the first color pixel and the second color pixel are alternately arranged, and in the even line, the first color pixel and the third color pixel are alternately arranged,
It is possible to read pixel values from a plurality of the first color pixels provided on the same line alternately with a first exposure time and a second exposure time longer than the first exposure time according to the arrangement order. And
From a plurality of the second color pixels provided on the same line, it is possible to read pixel values alternately in the first exposure time and the second exposure time according to the arrangement order,
From the plurality of third color pixels provided on the same line, it is possible to read pixel values alternately in the first exposure time and the second exposure time according to the arrangement order,
The image processing unit generates a new moving image having the predetermined frame rate based on the pixel value read out during the first exposure time and the pixel value read out during the second exposure time. , Imaging processing device. The image processing unit
The moving image of the first color component of the subject, the second color component obtained based on the pixel value read out during the first exposure time and the pixel value read out during the second exposure time And a moving image of the third color component,
(A) The moving image of the first color component, the second color component and the third color component read out at the first exposure time and the new moving image are sampled at the first exposure time. A first difference from the moving image obtained in the above, and (b) each moving image of the first color component, the second color component and the third color component read out in the second exposure time, and An evaluation formula including a second difference from a moving image obtained by sampling the new moving image at the second exposure time is set, and the moving image satisfies the evaluation formula under a predetermined condition. The imaging processing device according to claim 1, wherein the image processing device is obtained as the new moving image.   The imaging processing apparatus according to claim 2, wherein the image processing unit obtains a moving image that minimizes the evaluation formula as the new moving image. The image processing unit
A motion detection unit that detects the motion of the subject using the moving images of the first color component, the second color component, and the third color component read out in the first exposure time;
A constraint condition term relating to the motion distribution of the new moving image is set using the detection result of the motion, and a moving image that minimizes the evaluation expression further including the constraint condition term related to the motion distribution is set as the new motion image. An image processing apparatus according to claim 3, further comprising: an image generation unit that is obtained as a simple moving image.   The image processing unit sets a constraint condition term so that a change in pixel value distribution of the new moving image is reduced or a change in pixel value of the new moving image is constant, and the constraint The imaging processing apparatus according to claim 2, wherein a moving image that minimizes the evaluation expression further including a conditional term is obtained as the new moving image.   The imaging processing apparatus according to claim 5, wherein the image processing unit sets a constraint condition relating to the distribution of the pixel values in one direction of the image.   The imaging processing according to claim 6, wherein the image processing unit acquires the pixel value for each line from the single-plate color imaging element, and sets a constraint condition regarding the distribution of the pixel value for each acquired line. apparatus. For display on the moving image of the first color component, the moving image of the second color component, and the moving image of the third color component obtained based on the pixel signal read out in the first exposure time. A display control unit for performing predetermined image processing;
The imaging processing apparatus according to claim 7, further comprising: a display unit that displays the first moving image, the second moving image, and the third moving image that have been subjected to image processing by the display control unit.

Images