JP2011015228A - Image processing device, image processing device, image processing method, and control program of image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging processing device and a method capable of generating a moving image at a high frame rate with a high resolution.SOLUTION: The image processing device is provided with: a separation part 102 which separates visible light into at least first and second color components; a first imaging part 103 which photographs a moving image of the first color component; and a second imaging part 104 which photographs a moving image of the second color component. The first imaging part 103 performs exposure in a first charge accumulation period, and photographs each image of the moving image with a first spatial resolution and a first time resolution. The second imaging part 104 performs the exposure in a second charge accumulation period longer than the first charge accumulation period, and photographs each image of the moving image with a second spatial resolution higher than the first spatial resolution and a second time resolution lower than the first time resolution. The image processing device is further provided with: a control part 120 which controls imaging conditions in the first and the second imaging parts 103, 104; and an up-converter 106 which generates the moving image of the second component raised in time resolution on the basis of information on each moving image of the first color component and the second color component.

Description

  The present invention relates to image processing of moving images. More specifically, the present invention relates to a technique for generating a moving image in which at least one of resolution and frame rate of a captured moving image is increased by image processing.

  In recent years, in the field of video input, the number of pixels and the size of pixels in mobile phone cameras and digital still cameras are increasing.

  The spatial resolution varies greatly depending on the image quality required for the imaging device. For example, the resolution of a TV phone is a relatively low number of pixels of about QCIF (Quarter Common Intermediate Format, horizontal 176 pixels, vertical 144 pixels) or QVGA (Quarter Video Graphics Array, horizontal 320 pixels, vertical 144 pixels). . On the other hand, the resolution of a digital single-lens reflex camera exceeds 10 million pixels.

  Time resolution is also properly used. For example, regarding the time resolution of the number of pixels up to HDTV, imaging at a video rate (30 frames / second) by a consumer device is realized. However, for imaging with a larger number of pixels, the frame rate is limited to a few frames per second by the continuous shooting function installed in the digital still camera.

  On the other hand, in the field of video display, flat panel displays are rapidly spreading. Along with this, it is predicted that the user will view videos by combining various resolution cameras and displays in the future.

  Comparing the spatio-temporal resolution of the input-side camera (meaning “temporal resolution and spatial resolution”; the same applies hereinafter) and the spatio-temporal resolution of the output-side display, the current consumer device has a higher output-side display. . Therefore, there is no situation in which a general user can easily input an image that maximizes the performance on the output side. The reason for this situation is that the reading speed has been a bottleneck until now. High spatial resolution shooting is limited to about 5 frames per second, and conversely, 30 frame shooting per second is limited to HDTV spatial resolution. Therefore, it is difficult to capture a high spatial resolution image at a high frame rate.

  In response to the above problems, Patent Documents 1 to 3 disclose, as a method for achieving both high spatial resolution and high frame rate imaging, images with different temporal resolution and spatial resolution are input by two cameras, and signal processing is performed. A method for generating images with high spatial resolution and high frame rate has been proposed. These patent documents describe the configuration shown in FIG.

  FIG. 20 shows a configuration of a conventional imaging apparatus. A part of the light incident on the imaging device passes through the half mirror 171 and enters the first camera 172. As a result, a moving image with a low resolution and a high frame rate is captured. On the other hand, of the light incident on the imaging device, the light reflected by the half mirror 171 enters the second camera 173. As a result, a moving image with a high resolution and a low frame rate is captured.

  The up-converter 174 receives each moving image taken by the first camera 172 and the second camera 173, performs image processing, and outputs a moving image having a high resolution and a high frame rate.

JP-A-7-143439 JP 2005-515675 A JP 2005-318548 A Japanese Patent No. 3934151

  However, since the above-described imaging apparatus is basically configured assuming processing for a monochrome image, it is applied to a color image composed of three channels of red (R), green (G), and blue (B). The amount of processing is required 3 times. Further, since there is no correlation between the R, G, and B colors in the vicinity of the edge in the processed image, a false color may occur, causing a reduction in image quality.

  The present invention solves the above-described problems, and an object of the present invention is to provide an image processing apparatus capable of reducing the amount of calculation and suppressing the generation of false colors and increasing the resolution of a color image, and imaging A processing apparatus, an image processing method, and a control program for the image processing apparatus are provided.

  The image processing apparatus according to the present invention provides information on moving images of the first color component and the third color component captured at the first spatial resolution and at the first temporal resolution, and the second higher than the first spatial resolution. An image processing apparatus for receiving and processing moving image information of a second color component captured at a second temporal resolution that is spatial resolution and lower than the first temporal resolution, wherein the first color component A first up-converter that generates a moving image of the second component with increased time resolution based on the moving image information of the second color component and the moving image information of the third color component; and the first color component and the third color component Based on the information on each color component moving image and the information on the second component moving image with increased temporal resolution, the first color component and third color component moving images with increased spatial resolution are generated. And a second up-converter that Upconverter generates a constraint determined by the imaging process, a moving image of the second component with an increased time resolution based on evaluation values including the constraints of smoothness spanning between colors.

  In a preferred embodiment, the first color component, the second color component, and the third color component are red, green, and blue, respectively.

  In a preferred embodiment, the smoothness constraint condition between the colors is expressed by an expression that defines continuity between neighboring pixel values in a color space determined by the first to third color components.

  In a preferred embodiment, each of the three coordinate components of the pixel value in the color space is a linear sum of the first to third color components.

  In a preferred embodiment, the first up-converter calculates a pixel value constituting the moving image of the second component so that the evaluation value is minimized.

  In a preferred embodiment, the evaluation value includes a constraint condition for a change in pixel value along with movement in the captured scene.

  In a preferred embodiment, the evaluation value is minimized by dividing a simultaneous equation of a sparse coefficient matrix into a filter operation in which the coefficient is fixed and a filter operation in which the coefficient changes according to the content of the image. Do.

  In a preferred embodiment, the space-time resolution enhancement of the second component moving image in the first up-converter is performed by calculating only a region where the magnitude of motion in the scene is equal to or greater than a threshold value.

  In a preferred embodiment, the space-time resolution enhancement of the second component moving image by the first up-converter is performed by increasing the temporal smoothness constraint in a region where the motion is small in the scene.

  In a preferred embodiment, the first up-converter changes the evaluation value according to the magnitude of the moving image.

  The imaging processing apparatus of the present invention includes a separation unit that separates the second component from visible light including the first color component, the second color component, and the third color component, and a moving image of the first color component and the third color component. A first imaging unit that captures an image, the first imaging unit that performs exposure in a first charge accumulation period and captures each image constituting the moving image at a first spatial resolution and at a first temporal resolution. And a second imaging unit that captures a moving image of the second color component, wherein each image constituting the moving image is exposed in a second charge accumulation period longer than the first charge accumulation period. A second imaging unit that shoots at a second spatial resolution that is higher than the first spatial resolution and at a second temporal resolution that is lower than the first temporal resolution; and imaging conditions in the first imaging unit and the second imaging unit And a control unit that controls the image processing apparatus and any one of the image processing apparatuses described above.

  Another imaging processing apparatus of the present invention receives visible light including a first color component, a second color component, and a third color component, and a first charge accumulation period for the light of the first color component and the third color component. The images constituting the moving image of the first color component and the third color component are exposed at the first spatial resolution, and the light of the second color component is lighter than the first charge accumulation period. Each image constituting the moving image of the second color component is exposed at a long second charge accumulation period with a second spatial resolution higher than the first spatial resolution and lower than the first temporal resolution. An imaging unit that captures images at a second time resolution, a control unit that controls imaging conditions in the imaging unit, and any one of the image processing apparatuses described above are provided.

  The image processing method of the present invention provides information on moving images of the first color composition and the third color component captured at the first spatial resolution and at the first temporal resolution, and the second higher than the first spatial resolution. Receiving information of a moving image of a second color component captured at a spatial resolution and a second temporal resolution lower than the first temporal resolution; and moving images of the first color component and the third color component A first up-conversion step for generating information on the moving image of the second component with an increased temporal resolution based on the information of the moving image, information on moving images of the first color component and the third color component, and time resolution And a second up-conversion step for generating information on the moving image of the first color component and the third color component with enhanced spatial resolution based on the information of the moving image of the second component with enhanced And the first up Emissions Bart step generates a constraint determined by the imaging process, the information of the moving image of the second component with an increased time resolution based on evaluation values including the constraints of smoothness spanning between colors.

  The control program of the image processing apparatus according to the present invention includes information on moving images of first color components and third color components captured at a first spatial resolution and at a first temporal resolution, and more than the first spatial resolution. Receiving information of a moving image of a second color component captured at a second temporal resolution that is higher and lower than the first temporal resolution; and the first color component and the third color component A first up-conversion step for generating information on the second component moving image with an increased temporal resolution based on the information on the moving image of the first color component, information on the moving image of the first color component and the third color component, and A second up-conversion step for generating information on the moving image of the first color component and the third color component with increased spatial resolution based on the information of the moving image of the second component with increased temporal resolution, Real for image processing equipment In the first up-conversion step, the information on the moving image of the second component whose temporal resolution is increased based on an evaluation value including a constraint condition determined in the imaging process and a constraint condition of smoothness across colors is obtained. Generate.

  According to the imaging processing apparatus of the present invention, it is possible to generate a multi-color moving image having a high resolution and a high frame rate from moving images having a plurality of color components having different resolutions and frame rates. Each of the moving images of a plurality of color components is photographed by separating incident light for each color component using, for example, a dichroic mirror without using a half mirror or the like. Therefore, the effect of obtaining a color image with higher resolution than that of HDTV as a moving image and the color camera for HDTV can be further reduced in size.

It is a figure which shows the structure of the imaging processing apparatus by Embodiment 1. FIG. It is a figure which shows the structure of the up converter 106 for G in detail. (A) And (b) is a figure which respectively shows the base frame and reference frame when performing motion detection by block matching. It is explanatory drawing of the filter mounting of the product-sum calculation in the up converter for G. It is a figure which shows the structure of the upconverter 107 for R, B in detail. It is a figure which shows the structure of the upconverter 107 for R and B by another example in detail. It is a figure which shows the example of the relationship between correlation value (rho) and weight W2. FIG. 10 is a diagram showing a modification of the configuration of the upconverter 107 for R and B. It is a figure which shows the structure of the imaging processing apparatus 11 by the modification of Embodiment 1. FIG. It is a figure which shows the structure of the imaging processing apparatus 13 which generalized the imaging processing apparatus 1. FIG. FIG. 10 is a diagram illustrating another configuration example of the G upconverter 106 in the imaging processing apparatus according to the second embodiment. An example of area division by the area division unit 172 according to the second embodiment will be described. It is a figure which shows the structural example of the time resolution up converter in Embodiment 2. FIG. FIG. 10 is a diagram illustrating an example of an image processing system including an imaging device 901, a network 902, and a processing device 903 according to a third embodiment. It is a data diagram which shows the structure of the input image of this invention. It is a figure which shows the hardware of the image processing apparatus comprised by the computer. It is a flowchart which shows the procedure of the process of this invention. It is a flowchart which shows the procedure of the detailed process of step S103 shown in FIG. It is a flowchart which shows the procedure of the detailed process of step S103 by another example. It is a figure which shows the structure of the conventional imaging device.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
FIG. 1 is a diagram illustrating a configuration of an imaging processing apparatus 1 according to the present embodiment.

  The imaging processing device 1 includes a lens system 101, a dichroic mirror 102, a first imaging unit 103, a second imaging unit 104, an image processing unit 105, and a control unit 120. Hereinafter, first, the function of each component will be outlined, and then the operation of each component will be described in detail in relation to the operation of the imaging processing apparatus 1.

  The lens system 101 focuses light incident from the outside of the imaging processing apparatus 1, that is, an image of a subject. The dichroic mirror 102 functions as a color separation unit that transmits the red (R) component and blue (B) component of light and reflects the green (G) component of light. That is, the dichroic mirror 102 separates the incident light into a “red (R) component and a blue (B) component” and a “green (G) component”. In this specification, the red component is also described as “R component”, the green component as “G component”, and the blue component as “B component”.

  The first imaging unit 103 captures moving images of the R component and the B component based on incident light (here, the R component and the B component of light) with a short exposure time, a low resolution, and a high frame rate, respectively. Then, the obtained data is output. Note that in order to acquire the R component moving image and the B component moving image, the first imaging unit 103 has two dichroic mirrors inside, and two image sensors for detecting the R component and the B component, respectively. May be provided. Another example of the first image sensor 103 may have a configuration in which light including both the R component and the B component is incident on one image sensor (single plate type). In that case, an R filter that transmits the R component and does not transmit the B component and a B filter that transmits the B component and does not transmit the R component may be alternately arranged two-dimensionally on the front surface of the imaging device. Since the use efficiency of light decreases when a color filter is used, filterless imaging that generates a color signal using the fact that the depth of light entering the photodiode differs according to the wavelength of incident light, as will be described later. An element may be used.

  The second imaging unit 104 shoots a moving image with a long exposure time, high resolution, and low frame rate based on incident light (here, the G component of light), and outputs G component moving image data. .

  The image processing unit 105 receives moving image data corresponding to light of the R component and B component and moving image data corresponding to light of the G component, and each of them receives a moving image having a high resolution and a high frame rate by image processing. Convert to image and output.

  The image processing unit 105 includes a G upconverter 106 and an RB upconverter 107.

  The G up-converter 106 generates moving image data with a high resolution and a high frame rate in which the frame rate of the G component is increased. The upconverter 107 for R and B increases the resolution of the R component and the B component, and generates high resolution and high frame rate moving image data. A moving image is displayed by switching one or more images continuously at a predetermined frame rate. The process of increasing the resolution by the R and B upconverters 107 means increasing the number of pixels of each image constituting the moving image.

  Details of the G upconverter 106 and the R and B upconverters 107 will be described later.

  The control unit 120 controls imaging conditions when a moving image is captured using the first imaging unit 103 and the second imaging unit 104. The control information is output to the G upconverter 106 and the R and B upconverters 107.

  Next, the operation of each component will be described in detail along with the operation of the imaging processing apparatus 1.

  The position of the lens system 101 is adjusted so that the image of the subject is connected to the imaging elements of the first imaging unit 103 and the second imaging unit 104. The light passing through the lens system 101 is separated into “R component and B component” and “G component” by the dichroic mirror 102.

  The moving images of the R component and the B component are photographed by the first imaging unit 103 under the imaging conditions instructed from the control unit 120, that is, short-time exposure, low spatial resolution, and high frame rate. Here, the “low spatial resolution” is, for example, the number of pixels of one frame of NTSC (horizontal 720 pixels × vertical 480 pixels) or less VGA (Video Graphics Array: horizontal 640 pixels × vertical 480 pixels). ) Spatial resolution of about the number of pixels. “High frame rate” refers to about 30 fps (frames / second) to about 60 fps. “Short-time exposure” refers to exposure for a time equal to or shorter than the upper limit (in the present embodiment, 1/30 second to 1/60 second) determined by the frame rate at the longest.

  The G component moving image is also captured by the second imaging unit 104 under the imaging conditions instructed by the control unit 120, that is, imaging conditions of long exposure, high spatial resolution, and low frame rate. Here, “high spatial resolution” refers to, for example, a spatial resolution of the number of pixels of a general digital still camera (for example, about 4000 pixels horizontally and about 3000 pixels vertically). The “low frame rate” refers to a frame rate (for example, 3 fps (frame / second)) that is about 1/10 to 1/20 of the first imaging unit 103. “Long exposure” refers to exposure with a time determined by the value of the low frame rate (for example, 1/3 second) as an upper limit.

  In the present embodiment, the first image capturing unit 103 and the second image capturing unit 104 operate under synchronous control by the control unit 120. However, it is not essential to operate synchronously.

  Note that the above-described exposure time length, spatial resolution level, and frame rate (time resolution) level mean relative imaging conditions in the first imaging unit 103 and the second imaging unit 104. The exposure time of the R component and B component of the color image may be shorter than the exposure time of the G component. The spatial resolution (corresponding to the number of pixels here) of the R component and the B component only needs to be lower than that of the G component. The frame rate of the R component and the B component only needs to be higher than that of the G component. It is not limited to the above-mentioned numerical range illustrated.

Hereinafter, in this specification, a moving image having a high spatial resolution (H), a low frame rate (L), and a G color component is denoted as G _HL . The moving images having the low spatial resolution (L), the high frame rate (H), and the R and B color components are denoted as R _LH and B _LH , respectively. In each of these codes, the first character represents the color component, the second character (first subscript) represents the spatial resolution, and the third character (second subscript) represents the frame rate (temporal resolution). .

The G up-converter 106 includes data of a G image G _HL captured at a long exposure, high spatial resolution, and a low frame rate, and an R image R _LH captured at a short exposure, a low spatial resolution, and a high frame rate. The data of B image B _LH is received. Then, the G up-converter 106 generates and outputs a G image G _{HH in} which the time resolution of the G image G _HL is increased based on the data of the G image G _HL , the R image R _LH , and the B image B _LH. . That is, the G up-converter 106 generates a synthesized moving image with the same resolution and an increased frame rate (time resolution).

In a preferred embodiment of the present invention, a local correlation between GB and GR in a low spatial resolution region (low spatial frequency region) is used as a gain coefficient, and a high frequency component (high spatial frequency) of the G image is used. Component) is superimposed on R and B. Since the correlation between colors increases as the wavelength difference between colors decreases, the G image is acquired at a high spatial resolution and a low frame rate rather than the R image or the B image at a high spatial resolution and a low frame rate. By doing so, the correlation between colors increases, and high frequency components can be superimposed efficiently. For this reason, in this embodiment, the G image _GHL is imaged with long exposure, high spatial resolution, and low frame rate.

  FIG. 2 shows the configuration of the G upconverter 106 in more detail. In FIG. 2, the same reference numerals are given to components common to the components of the imaging processing apparatus in FIG. 1, and description thereof is omitted.

  The G upconverter 106 includes a motion detection unit 108 and a time resolution upconverter 109.

The motion detection unit 108 detects motion (optical flow) from R _LH and B _LH by existing known techniques such as block matching, gradient method, and phase correlation method. Known techniques include, for example, J. L. Barron, D. J. Fleet, S. S. Beauchemin, and T. A. Burkitt. "Performance of Optical Flow Technologies", In Proc. P. 36. -242, 1992 is known.

  FIGS. 3A and 3B show a base frame and a reference frame when motion detection is performed by block matching, respectively. The motion detection unit 108 sets a window region A shown in FIG. 3A in a reference frame (image at time t when attention is given to obtain motion). Then, a pattern similar to the pattern in the window area is searched for in the reference frame. As the reference frame, for example, a frame next to the frame of interest is often used.

As shown in FIG. 3B, the search range is normally set in advance with a predetermined range (C in FIG. 3B) based on the position B where the movement amount is zero. In addition, the similarity (degree) of the pattern is determined by the residual sum of squares (SSD: Sum of Differences) shown in (Equation 1) or the absolute sum of residuals (SAD: Sum of Absorbed Differences) shown in (Equation 2). Is evaluated as an evaluation value.

  In (Equation 1) and (Equation 2), x, yεW means the coordinate value of the pixel included in the window region of the reference frame.

  The motion detection unit 108 searches for a pair (u, v) that minimizes the evaluation value by changing (u, v) within the search range, and sets this as a motion vector V between frames. . By sequentially shifting the set position of the window area, the movement is obtained for each pixel or for each block (for example, 8 pixels × 8 pixels).

Refer to FIG. 2 again. The temporal resolution upconverter 109 is detected by the R image R _LH and the B image B _LH captured by the first imaging unit 103, the G image G _HL captured by the second imaging unit 104, and the motion detection unit 108. Each data of the motion vector V is received, and the resolution of _GH is increased and output as _GHH .

Here, the resolution of G _HL is increased by _obtaining G _HH that minimizes the following (Equation 3).

The first term on the right side of (Equation 3) is a term indicating the luminance difference between the corresponding pixels of G _HH and G _HL , the second term is a term indicating “motion constraint”, and the third, fourth and Item 5 is a term indicating “smooth constraint”. Here, G _HH , G _HL , R _LH , and B _LH are vertical vectors having each pixel of the moving image as elements, H _T is a matrix that models addition of light by long exposure, λm is a weight for motion constraint, λ _sC1 , λ _sC2 , and λ _sC3 are weights for smoothness constraint, and C ₁ , C ₂ , and C ₃ are vertical vectors of three color components different from RGB defined in the three-dimensional color space (each pixel of the moving image column vector) whose elements, Q _sH, Q _sV evaluates each three axes defined in smoothness constraints in the horizontal and vertical directions (the color space, the _{C 1, C 2, C 3} ), Qm is motion constraint Respectively. Since the frame rate of the G _HH is higher than the frame rate of the G _HL, the number of pixels of G _HH (the number of elements of the column vector) is larger than the number of pixels of G _HL (the number of elements of the column vector).

The imaging conditions set in the first imaging unit 103 and the second imaging unit 104 by the control unit 120 are reflected in at least the above H _T , Qm, λ _sC1 , λ _sC2 , and λ _sC3 . Note that the left side J of Equation 3 is a scalar. The “imaging condition” is the spatial resolution or exposure time of each of the first imaging unit 103 and the second imaging unit 104, or the spatial resolution ratio or the exposure time ratio. From these, it can be said that the first term of (Equation 3) is “a constraint condition determined by the imaging process”.

G _HH that minimizes the above (Equation 3) means G _HH that best satisfies the linear sum of given constraints. The process of deriving each term on the right side will be described later.

C ₁ , C ₂ , and C ₃ are calculated from R _LH , G _HH , and B _{LH according} to the following equations, respectively.

Here, c ₁₁ to c ₃₃ are color conversion coefficients from the R, G, B color space to other color spaces C ₁ , C ₂ , C ₃ different from the R, G, B color space, and M is the spatial This is a coefficient matrix for performing large expansion (so-called interpolation expansion). That is, (Equation 4) indicates that C ₁ , C ₂ , and C ₃ are represented by a linear sum of R _LH , B _LH, and G _HH that have been interpolated and expanded.

The temporal resolution upconverter 109 obtains _GHH that minimizes (Equation 3) based on (Equation 5) below. (Equation 5) is obtained by substituting the above (Equation 4) into (Equation 3) and performing partial differentiation with _GHH .

As a result, the time resolution upconverter 109 obtains _GHH by solving the simultaneous equations shown in (Expression 6).

  (Expression 6) can be solved by using an existing numerical calculation method (solution of simultaneous equations) such as a conjugate gradient method or a steepest descent method.

The time resolution upconverter 109 obtains _{GHH in} which the time resolution of the input _GH is increased by the above-described procedure is not only a process for increasing the number of frames per unit time (that is, a process for increasing the time resolution). Also included is a process of converting a blurred image into a sharp (sharp) image. The G component moving image is captured at a high spatial resolution by the second imaging unit 104, but a part of the image may be blurred due to the effect of long exposure. For this reason, when the frame rate of an image shot at a low frame rate is increased, not only the temporal resolution but also the spatial resolution is increased (sharpening). Accordingly, when the temporal resolution up-conversion is performed, both the temporal resolution and the spatial resolution are improved. Therefore, these can be collectively expressed as “(spatiotemporal) resolution can be increased”.

  In the following, the meaning and function of each item in (Equation 3) will be described in more detail.

The first term of (Equation 3) indicates the difference between the long-exposure image (H _T G _HH ) predicted from G _HH with increased resolution and the long-exposure image G _HL obtained by actual observation. . This represents a spatio-temporal correspondence between _GHH whose resolution has been increased and the long-time exposure image _GH . Here, the number of rows in H _T is less than the number of columns. This can be understood from the fact that _GH is a long-exposure image and has a lower frame rate than _GHH (that is, the total number of pixels is small). Therefore, if the left side of (Equation 3) is only the first term, a problem to be solved (that is, simultaneous equations) will be a defective setting problem that cannot be solved uniquely.

  In order to change this failure setting problem to a good setting problem, the second term, the third term, the fourth term, and the fifth term are added to the first term of (Equation 3).

The second term of (Equation 3) is a term that prescribes constraints relating to movement in the image. Here, it is assumed that each point in the moving image moves without changing the brightness. In the matrix Qm in the second term, only elements related to the start and end pixels of the motion vector detected in the moving image are 1 and −1, respectively. Therefore, (QmG _HH ) ² is the total sum of the whole moving image of the square of the difference between the start point and the end point of the motion vector.

として表すことができる。ここで、積分範囲は動画像の占める時空間全体であり、数７の被積分関数中の||・||はベクトルのノルムを示す。（数３）の第３、４、５項は、（数７）を差分展開し、ＩをＣ₁、Ｃ₂、Ｃ₃に置き換えて、行列ＱｓとベクトルＧ_HHの積の形で表している。 The third, fourth, and fifth terms of (Expression 3) indicate characteristics generally satisfied by the image, that is, local smoothness between neighboring pixels. When a monochrome (single color) moving image is I (x, y, t), the local smoothness is

Can be expressed as Here, the integration range is the entire time space occupied by the moving image, and || · || in the integrand of Equation 7 represents the norm of the vector. The third, fourth, and fifth terms of (Equation 3) are expressed in the form of a product of the matrix Qs and the vector G _HH by differentially expanding (Equation 7) and replacing I with C ₁ , C ₂ , and C _3. Yes.

By defining the smoothness constraint in such a manner as to span between colors, it is possible to reflect the information of the R component and the B component in increasing the time resolution of the G component. In the present invention, since the R, G, and B components are captured at different spatiotemporal resolutions, it is effective to perform image restoration using such constraint conditions. Here, the advantage of decomposing into a color space (C ₁ C ₂ C ₃ ) different from RGB will be described.

  From the viewpoint of human visual characteristics, RGB signals forming a three-dimensional color space are divided into components that greatly contribute to the sense of resolution and components that do not contribute much to the sense of resolution. Therefore, an image with a sense of resolution can be restored by setting the regularization parameter λ for the former small and setting the regularization parameter λ for the latter large.

  From the viewpoint of the distribution of data to be restored, the regularization parameter λ is set to be small for the direction in which the solution variation is large, and the regularization parameter λ is set to be large for the direction in which the solution variation is small. Accuracy can be increased.

  In an image obtained by capturing a natural image, the luminance component variation tends to be large and the color component variation tends to be small in most images.

As a method of separation into three colors, for example, conversion from RGB to a luminance-type difference signal of YCbCr is performed by Equation 8.

  Further, the method of separation into three colors is not limited to the above method, and the first principal component, the second principal component, and the third principal component are obtained by principal component analysis in the RGB space for an image prepared in advance. Three axes in the color space may be determined. Usually, the first principal component obtained by principal component analysis from a natural image is substantially equal to the luminance component, and the second principal component and the third principal component are components corresponding to color signals.

As for the smoothness constraint between the colors, the same image quality can be obtained even if C ₃ is selected to be perpendicular to G in the color space and replaced with a color spanning between 3 colors to a color spanning between 2 colors. The processing result can be obtained. In that case, the following equation is used instead of (Equation 3).

In order to obtain G _HH , the following equation may be solved instead of (Equation 6).

By the method and procedure described above, G _HH is obtained as a solution of simultaneous equations of (Equation 6) or (Equation 10). The calculation for solving the simultaneous equations requires a large amount of calculation as the number of variables (the number of unknowns to be solved) increases. Various methods for efficiently solving large-scale simultaneous equations have been proposed, but optimization of calculation efficiency largely depends on the problem. Below, the efficient calculation of the problem dealt with this time is explained.

として表している。なお、後述するように、拘束条件として（数３）の右辺の全ての項を用いる必要は無い。図４の例では、（数３）の右辺第５項を省略している。 Figure 4 shows a procedure of the conjugate gradient method for calculating the G _HH, an example of a block diagram efficiently performing calculation of a product of a coefficient matrix A takes the longest amount of computation in the conjugate gradient method and the update vector pk . In FIG. 4, the simultaneous equations to be solved are

It represents as. As will be described later, it is not necessary to use all the terms on the right side of (Equation 3) as constraint conditions. In the example of FIG. 4, the fifth term on the right side of (Equation 3) is omitted.

  In the conjugate gradient method, the update amount p is calculated so that the residual r becomes small, and the update of the solution x is repeated using this. The number of iterations required for convergence depends on the condition (numerical stability) of the coefficient matrix A and the number of variables. In the case of this embodiment, the repetition is performed about 100 times.

  In this iterative calculation, calculation of the product of the coefficient matrix A and the update amount vector p requires the most amount of calculation. Since the vector p is the update amount of the solution, the number of elements (dimensions) is the number of G pixels in the moving image. For example, if processing for four frames is performed with the spatial resolution of the HDTV signal, the number of G pixels (that is, the number of elements of p) is 1920 × 1080 × 4 = 8294400. Since the coefficient matrix A is a square matrix, the coefficient matrix A has a square element number. Such an enormous memory access and product-sum operation cannot be calculated in a calculation environment that is normally available at present.

However, the coefficient matrix A, as shown in (6) or (number 10), the process H _T long exposure, motion constraint Qm, smoothness General Q _sH, 2 squared Q _sV (with its transpose Product). Since these terms are calculations of the pixel of interest and only its neighboring pixels, most of the terms are 0 for the entire coefficient matrix. Such a matrix is called a sparse matrix. The problem of which pixel in the coefficient matrix becomes 0 and how much the ratio of non-zero coefficients varies depends on the problem, and the efficient solution depends on the individual problem.

Long exposure process H _T can be considered to use indicated by the reference numeral 401 in FIG. 4, coefficients all of the time direction of the storage filter. Although the product of the transpose and H _T of H _T is as a constraint condition, the transpose of H _T represents the function of outputting the value of one pixel to pixel number storage pixel minutes. Therefore, (H _T ) ^T H _T can be interpreted as a filter that outputs a value obtained by accumulating a plurality of input frames to a pixel position corresponding to an input pixel.

  Alternatively, the long exposure process HT may be an averaging filter in the time direction in which all the coefficients are ¼ (in the case of adding four frames). The difference in processing between the case where all the coefficients of the long-time exposure process HT are 1 and the case where the coefficient is 1 / the number of added frames is as follows.

  In the former case, the motion blur included in the long-time exposure image is removed, and the light amount exposed for a long time is distributed to the image of each frame. As a result, the processing result is darker than the input image. On the other hand, the processing in the latter case only removes motion blur included in the long-time exposure image, and the processing result has the same brightness as the input image. Therefore, in the former case, before increasing the resolution of R and B using the restoration result of G, it is necessary to align the brightness (level) of G with R and B by a multiplier (not shown) (this embodiment). In the case of the example, it is necessary to multiply by 4).

Next, the smoothness constraint will be described. Qs is a second-order differential operator, and can be expressed by the following equation when the difference is expanded.

(Equation 12) assumes that G _HH is input and ∂G _HH / ∂x and ∂G _HH / ∂y are output. As a filter, the horizontal or vertical coefficient is (−1, 2, −1) 3. It can be interpreted as a tap filter. When this is viewed as a coefficient matrix, Qs is a diagonal matrix with a diagonal component of 2, one pixel on the left or right or one line on the upper and lower sides is -1, and all others are 0.

Since these are diagonal matrices, the transpose matrix is also equal to itself, and the product of the transpose and itself is a 5-tap filter with coefficients (1, -4, 6, -4, 1). In other words, it smoothness constraint ^{_{(Q sH) T Q sH,}} (Q sV) T Q sV , as indicated by 402 in FIG. 4, can be calculated as a filter and a vertical 5 tap horizontal 5 tap filter.

Finally, the motion constraint Qm is a collection of data pairs in which the element corresponding to the start point of the motion vector is 1 and the element corresponding to the end point is -1. As a matrix, all diagonal components are 1, and each row has one element of -1. Therefore, (Qm) ^T Qm has all the diagonal components of 1 and the element whose Qm is −1 and the element at the transposition position thereof have a value of −1. Such a matrix operation can be implemented as an interframe filter having a spatial size determined by a motion vector search range as a filter. For example, when the motion detection range is a range of ± n pixels in the vertical and horizontal directions on the high resolution side, it can be implemented as a filter of vertical and horizontal (1 + 2n) pixels × 2 frames including the target pixel.

  With the filter implementation described above, computation of the product of matrix A and vector p, which requires the most amount of computation in the conjugate gradient method, is necessary as a linear sum of several filter computations, omitting computation with a coefficient of 0. It is possible to calculate with a minimum amount of calculation. Among these filter operations, those other than motion constraints can be prepared in advance with fixed coefficient filters regardless of the contents of the image to be processed. On the other hand, since the motion constraint depends on the motion of the scene, it is necessary to determine the filter coefficient each time according to the content of the image to be processed (in the present embodiment, the result of motion detection).

In this embodiment, the calculation of _GHH by the conjugate gradient method has been described for all pixels. However, the motion detection result (the sum of the lengths of motion vectors for frames corresponding to the exposure time of G) is described. Accordingly, the calculation amount G _HH is calculated only for a region with motion, and for a region with little motion, the G input image is directly used as G _HH , thereby further reducing the amount of calculation.

Further, in the calculation of _GHH by the conjugate gradient method, smoothness constraint in the time direction may be strengthened in a region where the motion is small. Thus, qualitatively, with the same thing as it is directly and G _HH the input image G, it is possible to numerically stabilized, it is possible to accelerate the convergence of the iterative calculation.

としたり、

としてもよい。 In addition, regarding the above (Equation 3) and (Equation 9), it is not always necessary to use all of the constraint conditions on the right side at the same time. The following modifications may be used. That is, without using the second term on the right side of (Equation 3) and (Equation 9)

Or

It is good.

  According to these modified examples, it is possible to realize high resolution of the G component with a smaller amount of calculation than that calculated by (Equation 3) or (Equation 9) using all the constraint conditions. However, since the constraint condition is relaxed, the high-resolution feeling of the generated image may be slightly reduced.

  Further, in a scene where motion detection is difficult, by performing high resolution processing using a mathematical expression ((Equation 11) or (Equation 12)) that does not use the result of motion detection, erroneous detection of motion with respect to the image quality of the output image It is possible to suppress the occurrence of artifacts (disturbances of images, noise) due to. Whether motion detection from a scene is difficult is determined from (a) the difference in the results of motion detection in both directions in time, or (b) the motion shown in (Equation 1) and (Equation 2). It can be judged from the minimum value within the search range of the evaluation value at the time of detection.

  In the case of the former (a), if the forward motion detection result is (u, v) in time in (x, y) in the image of the reference frame, the reference for the next forward motion detection is performed. If motion detection is performed in the reverse direction with reference to the frame and the motion detection result at (x + u, y + v) is (−u, −v), the motion detection in both directions is consistent and reliable. On the other hand, when it is different from (−u, −v), for example, when there is a difference greater than a certain threshold value, it can be determined that it is difficult to detect motion.

  Similarly, in the case of the latter (b), when the minimum value within the search range of the evaluation value at the time of motion detection of SSD or SAD is equal to or greater than a predetermined threshold, for example, motion detection is difficult. It can be judged that there is.

  Refer to FIG. 1 again. The R and B up-converters 107 use the G component moving image that has been increased in resolution by the G up-converter 106, and the first imaging unit 103 performs imaging with short-time exposure, low resolution, and high frame rate. The resolution of the captured moving image of the R and B components is increased.

  Hereinafter, a process for increasing the resolution of a moving image of the R and B components will be described in detail with reference to FIG.

  FIG. 5 shows the configuration of the upconverter 107 for R and B in more detail. In FIG. 5, the same reference numerals are given to components common to the components of the imaging processing apparatus in FIG. 1, and description thereof is omitted.

  The R and B up converter 107 includes a down converter 110, a coefficient estimation unit 111, and a restoration filter 112.

The down-converter 110 spatially reduces the resolution of the G component (G _HH ) that has been increased in resolution by the G up-converter 106 and outputs G _LH . The coefficient estimation unit 111 estimates the filter coefficient of the restoration filter 112 (restoration filter H +) from _GHH and _GLH . As the restoration filter 112, a known filter such as a Wiener filter or a general inverse filter can be used. These restoration filters estimate the high-resolution signal from the low-resolution signal using the relationship between _GHH and _GLH . The restoration filter 112 restores R _HH and B _HH from R _LH and B _LH using the filter coefficient estimated by the coefficient estimation unit 111.

  Note that the R and B resolution enhancement processing in the R and B up converter 107 is not limited to the above-described processing, and may be performed by other resolution enhancement processing. The resolution enhancement process according to another example will be described below.

  FIG. 6 shows in detail the configuration of the upconverter 107 for R and B according to another example. Here, the upconverter 107 for R and B increases the resolution of the R component and the B component by superimposing the G high frequency component on the R component and the B component that have been interpolated and expanded according to the correlation between R and B. To do.

  The R and B up-converter 107 shown in FIG. 6 includes a down converter 113, an interpolation enlargement unit 114, a local correlation calculation unit 115, a weight generation unit 116, a counter 117, an image memory 118, and a normalization unit. 119.

  The R and B up-converters 107 perform processing in units of frames constituting a moving image. First, at the beginning of each frame process, the contents of the counter 117 and the contents of the image memory 118 are cleared by filling them with 0, for example.

Resolution reduction unit 113 high resolution has been G component (G _HH) spatially Teikai Zoka by G Upconverter 106, and outputs the G _LH.

Interpolation based expansion section 114, a low resolution, R _LH captured at a high frame rate, receives a B _LH and imaging conditions, the R _LH and B _LH interpolating enlargement to be the same number of pixels as G _HH.

The local correlation calculation unit 115 calculates a local correlation value between R _LH and B _{LH for} a local region of about 2 × 2 pixels or 3 × 3 pixels. When calculating the local correlation value ρ for 2 × 2 pixels, the local correlation calculation unit 115 can use, for example, (Equation 15).

  The weight generation unit 116 generates a weight according to the correlation value calculated by the local correlation calculation unit 115. FIG. 7 shows an example of the relationship between the correlation value ρ and the weight W2. The weight generation unit 116 obtains the weight W2 based on the correlation value ρ and the relationship shown in FIG.

As shown in FIG. 6, the weight generated and output by the weight generation unit 116 is multiplied by the difference between _GHH and _GLH (that is, the high frequency component of G), and the image memory 118 is updated. More specifically, after multiplication is performed, an address corresponding to the storage position of the pixel data in the image memory 118 is specified. Then, the multiplication result and the value held at the address are added, and the value at the address is rewritten to the addition result.

  At this time, the object to be written in the image memory 118 may be one pixel or a range in which a local correlation value is calculated. However, when the high frequency component is superimposed on a plurality of pixels as in the latter case, depending on the setting method of the region for calculating the local correlation (that is, the increment method in the image), the high frequency component is applied to the same pixel a plurality of times. It will be superimposed. In consideration of such a case, the R and B up converter 107 in FIG. 6 uses the counter 117. The counter 117 stores the number of times that the high frequency component is superimposed for each pixel.

The normalization unit 119 divides the high frequency component superimposed a plurality of times by the write count value stored in the counter 117 for each pixel. The normalized high frequency component is superimposed on the R and B images that have been interpolated and enlarged by the interpolation enlarging unit 114, and is output as R _HH and B _HH .

  By increasing the resolution of R and B by the method described above, it is possible to increase the resolution of R and B while maintaining local color balance, and as a result, the generation of false colors is suppressed. High resolution can be achieved.

FIG. 7 shows a linear relationship as an example of the relationship between the correlation value and the weight. However, this is an example, and a non-linear relationship may be used in consideration of the γ characteristic at the time of imaging or display. The weights may be normalized by (local average of R) / (local average of G) for R and normalized by (local average of B) / (local average of G) for B. By this normalization, it is possible to adjust the amplitude of the G high-frequency component to be superimposed on R and B according to the R, G, and B pixel values, and to reduce the uncomfortable feeling during observation due to excessive high-frequency superimposition. . Here, as the local average of R, G, and B, pixel values of R _LH and B _LH expanded by interpolation or G _LH obtained by down-converting _GHH by the down converter 113 in FIG. 6 may be used.

  If the operation is performed so that the high-frequency component is superimposed only once per pixel on the R and B images that have undergone interpolation enlargement, the image memory 118, the counter 117, and the normalization unit 119 in FIG. 6 are not necessary. The configuration shown in FIG. FIG. 8 shows a modification of the configuration of the upconverter 107 for R and B. As shown in FIG. 8, the R and B up-converters 107 can be realized with a simpler configuration than that of FIG. 6, and the same effect can be obtained.

Note that the resolution enhancement processing for R and B in the R and B up-converter 107 need not be limited to the processing using the above-described so-called restoration filter or the processing for superimposing the G high-frequency component on the R and B components. . It is also possible to learn the relationship between the high resolution G component and the low resolution (G _LH and G _HH ), and to increase the resolution of the R and B components based on the learning result. Good.

  Note that learning is not limited to being performed in the process of processing an input image, but may be performed by preparing a learning pattern in advance. In this case, it is possible not only to learn the relationship between the low resolution G component and the high resolution G component, but also to learn the relationship between the low resolution RGB component and the high resolution RGB component.

  S. Farsiu et al., “Multiframe Demonstration and Super-Resolution of Color Images”, IEEE Transactions of Image Processing, Vol. 15, no. 1,2006. Ron Kimmel, “Demosaicing: Image Reconstruction from Color CCD Samples”, IEEE Transactions of Image Processing, Vol. 8, no. 9, 1999. Daniel Keren et al., "Restoring Subsampled Color Images", Machine Vision and Applications 11: pp. 197-202, 1999. And the like use a constraint condition based on local correlation between RGB at the time of image restoration. In natural images, there is a positive correlation between RGB in most areas, so this constraint generally works effectively. However, if the color of the area is close to the primary colors or their complementary colors, at the boundary with another color, The local correlation becomes negative, and the set constraint condition does not work effectively, causing artifacts such as color bleeding and ringing in the restored image.

  On the other hand, in the smoothness constraint across colors used in this embodiment, a positive correlation based on local color correlation is not assumed, but only local smoothness is assumed. From the viewpoint of correlation, positive correlation or negative correlation may be used. Therefore, even in the region where the local correlation is negative, artifacts such as color bleeding and ringing do not occur remarkably in the restored image.

  In the above description, the output signal is assumed to be R, G, B color components. Hereinafter, an imaging processing apparatus that converts each RGB output signal into a luminance component and a color difference component for output will be described.

  FIG. 9 shows a configuration of an imaging processing apparatus 11 according to a modification of the present embodiment. In FIG. 9, the same reference numerals are assigned to the same components as those of the imaging processing apparatus described above, and the description thereof is omitted.

  The image processing unit 105 includes a color difference calculation unit 129 and a luminance calculation unit 130 in addition to the G up-converter 106, the interpolation enlargement unit 114, and the down converter 113.

The control unit 120 receives the R component and B component signals that have been interpolated and expanded by the interpolation expanding unit 114, and the G component signal that has been reduced in resolution by the resolution reducing unit 113, and calculates the color difference by the calculation of (Equation 16). It converts into a signal (Cb signal, Cr signal) and outputs.

The luminance calculation unit 130 receives the R component and B component signals interpolated and enlarged by the interpolation enlargement unit 114, and the G component signal whose resolution has been increased by the G upconverter, and the luminance is calculated by the calculation of (Equation 17). A signal (Y signal) is converted and output.

  As can be understood from the description of the color difference calculation unit 129 and the luminance calculation unit 130 described above, the low-resolution G is used for the calculation of the color difference components Cb and Cr, while the high resolution is used for the calculation of the luminance component Y. G is used. As a result, it is possible to increase the resolution of the output image while suppressing the occurrence of false colors.

  Note that a block that converts Y, Cb, and Cr signals into RGB signals may be further provided at the subsequent stage of the image processing unit 105 to output RGB component signals.

  Note that the imaging processing apparatus according to the present embodiment and the imaging processing apparatus according to the modification thereof capture the G component with high resolution, long exposure, and low frame rate, and the R component and B component with low resolution, short exposure, high Suppose you want to capture at the frame rate. However, this is an example. Other examples can be adopted as to which color component (wavelength band) is imaged at high resolution, long exposure, and low frame rate.

  For example, when it is known in advance that the B component appears strongly in the scene, such as when capturing an underwater scene such as the sea or a pool, the B component is displayed with high resolution, long exposure, and low frame. By imaging at a rate and imaging the R component and G component at a low resolution, short exposure, and a high frame rate, an image with a high resolution feeling can be presented to the observer.

  For example, FIG. 10 shows a configuration of an imaging processing device 13 that generalizes the imaging processing device 1. In FIG. 10, the same reference numerals are given to components common to the components of the imaging processing apparatus in FIG. 1, and description thereof is omitted.

  The imaging processing device 13 includes an R component imaging unit 131, a G component imaging unit 132, a B component imaging unit 133, a control unit 134, a switching unit 135, an HL up-converter 136, and an LH up-converter 137. And an output unit 138. Hereinafter, the function of each component will be described together with the operation of the imaging processing device 13.

  The visible light that has passed through the optical system 101 is wavelength-resolved by the dichroic prism, and is imaged by the R component imaging device 131, the G component imaging device 132, and the B component imaging device 133. The number of pixels read out by the imaging units 131, 132, and 133 for each of the RGB components can be individually and dynamically set by a binning readout method. The “binning readout method” is a method of reading out by adding charges accumulated in adjacent pixels. In each of the imaging units 131, 132, and 133, the exposure time and the frame rate can be set similarly. The condition setting at the time of reading is performed by the control unit 134.

  The control unit 134 controls any one of the R component imaging device 131, the G component imaging device 132, and the B component imaging device 133 according to the distribution of color components in the scene, with high resolution, long exposure, and low frame rate. (Corresponding to G in the first embodiment), and the rest are set to low resolution, short-time exposure, and high frame rate (corresponding to R and B in the first embodiment).

  Since the distribution of the color components in the scene is unknown at the start of imaging, for example, G may be set to high resolution, long exposure, and low frame rate.

  According to the setting of the imaging units 131, 132, 133 for each of the RGB components by the control unit, the switching unit 135 uses the imaging data of the components for which imaging with high resolution, long exposure, and low frame rate is set for HL. The switching operation is performed so that the data of the remaining components input to the up-converter 136 are input to the up converter 137 for LH.

  The HL up-converter 136 performs the same processing as the G up-converter 106 (for example, FIG. 1), and spatially increases the resolution of the moving image of the component captured at high resolution, long exposure, and low frame rate. .

  The LH up-converter 137 receives two lines (two color components) of moving images captured at a low resolution, a short exposure, and a high frame rate, and a moving image that has been increased in resolution by the HL up-converter 136. , R and B up-converter 107 (for example, FIG. 1) is used to spatially increase the resolution of the two systems of moving images.

  The output unit 138 receives moving images whose resolution has been increased by the HL up-converter 136 and the LH up-converter 137, respectively, and outputs RGB three-line moving images according to the settings by the control unit 134. Of course, the output unit 138 may convert the signal into another signal format such as a luminance signal (Y) and a color difference signal (Cb, Cr) signal and output the signal.

  Note that the imaging unit is not necessarily limited to the three-plate configuration shown in the present embodiment. For example, a single-plate configuration changes the charge accumulation period between R, G, and B, or changes the pixel addition method to change the space. The resolution may be changed, or line thinning readout may be performed.

(Embodiment 2)
Next, an embodiment in which the processing efficiency is increased according to the magnitude of the movement will be described.

  FIG. 11 shows a configuration of the G up-converter 106 in the present embodiment. The configuration of the other parts is basically the same as the configuration of the first embodiment described with reference to FIGS. For this reason, a different part is demonstrated here.

The G up-converter 106 according to the present embodiment performs processing for increasing the resolution only for a region with motion by dividing the processing for a region with motion and processing for a region without motion. In the present embodiment, as shown in FIG. 11, the motion detection unit 171 receives R _LH and B _LH and outputs both the motion detection result and the motion reliability. The area dividing unit 172 divides the moving image into three-dimensional areas based on the motion detection result detected by the motion detecting unit 181.

  FIG. 12 shows an example of area division by the area division unit 172. FIG. 12 shows a state in which a moving image is divided into a non-motion region (region number 0) and two motion regions (region numbers 1 and 2).

  The temporal resolution up-converter 173 for each region performs temporal resolution up-conversion processing on each of the regions that have motion among the region segmentation results that are segmented according to the motion by the region segmentation unit 172, and each region has no motion. For G, each pixel value of G of the input image is equally divided according to the number of G addition frames at the time of shooting. For example, if the number of added frames of G is 4, each pixel value of G of the input image is reduced to a quarter and output for 4 frames.

  By performing the above processing, it is possible to simplify the processing for a static region where there is no motion, and reduce the amount of calculation. In addition, by performing individual processing on the divided regions with motion, it is possible to reduce the number of variables of simultaneous equations to be solved for each region, and to reduce the operation scale.

  In the above description, the process of the temporal resolution upconverter 173 for each region has been described as a decomposition process including both the removal of motion blur included in the long-time exposure image and the distribution of the amount of light. However, the present invention is not limited to this, and the processing of the temporal resolution upconverter 173 for each region may be only removal of motion blur included in the long-time exposure image. In that case, processing in a region where there is no movement is unnecessary, and the input is simply output as it is.

  In the above embodiment, the case where super-resolution processing is performed only in a region where there is motion has been described, but not only the processing is discretely divided depending on the presence or absence of motion, but also the processing is switched continuously according to the magnitude of motion. You may do it.

FIG. 13 is a drawing corresponding to FIG. 4 and shows an example of the configuration of the time resolution upconverter. The spatial smoothness restraint unit 182 shown in FIG. 13 controls λ _sC1 and λ _sC2 according to the magnitude and reliability of the motion. The time smoothness control unit 183 controls λt according to the magnitude and reliability of the movement.

The spatial smoothness restricting unit 182 increases λ _sC1 and λ _sC2 as the motion is larger, that is, the motion vector is longer, and conversely, as the motion is smaller, that is, as the motion vector is shorter, λ _sC1 and λ _{Reduce sC2} . In addition, the spatial smoothness restraining unit 182 increases the control according to the length of the motion vector as the motion reliability is higher, and conversely, as the motion reliability is lower, the control according to the length of the motion vector is increased. Reduce control.

  The time smoothness restricting unit 183 increases λt as the motion is small, that is, as the motion vector is short, and conversely, as the motion is large, that is, as the motion vector is long, λt is decreased. In addition, the time smoothness restraining unit 183 increases the control according to the length of the motion vector as the motion reliability is higher, and conversely, as the motion reliability is lower, the time smoothness constraint unit 183 responds to the length of the motion vector. Reduce control.

Due to the control of λ _sC1 , λ _sC2 , and λt described above, the temporal smoothness constraint becomes stronger in a small motion region or a stationary region, so the processing is performed continuously or stepwise according to the magnitude of the motion. Can be switched.

(Embodiment 3)
In the first and second embodiments, the example in which the imaging process and the resolution enhancement process are performed by the same system (imaging processing apparatus) has been described. However, both processes are not necessarily performed in the same system.

  In the present embodiment, an example will be described in which imaging processing and high resolution processing are performed in different systems.

  FIG. 14 shows an example of an image processing system including an imaging device 901, a network 902, and a processing device 903 according to the present embodiment. The image processing system according to the present embodiment can be configured by using the media 906 instead of the network 902. In FIG. 14, the same reference numerals are given to the same components as those of the imaging processing apparatuses (for example, FIG. 1) of the first and second embodiments, and the description thereof will be omitted.

  The imaging device 901 includes a lens system 101, a dichroic mirror 102, a first imaging unit 103, a second imaging unit 104, and a shooting mode setting unit 904.

The first imaging unit 103 captures an R component and a B component of a color image with a short exposure time, a low resolution and a high frame rate, and outputs an R image R _LH and a B image B _LH . The second imaging unit 104 captures the G component of the color image with a long exposure time with a high resolution and a low frame rate, and outputs a G image _GHL .

  The shooting mode setting unit 904 sets shooting conditions whose settings are variable, such as the frame rate and the exposure time in the second imaging unit 104, for example, and writes information indicating the set conditions in the header of the video signal or separately. Data is output to the network 902 via the output unit 905.

  FIG. 15 shows an example of the data structure. In FIG. 15, the header portion stores shooting conditions (conditions in which settings can be changed) such as a frame rate and an exposure time (or a value obtained by converting the G exposure time into the number of R and B frames). , R, B image data, and G image data follow.

The output unit 905 outputs the G image G _HL , the R image R _LH , the B image B _LH , and information on the imaging conditions thereof captured by the imaging device 901 to the network 902 or the medium 906.

The processing device 903 has an image processing unit 105. The image processing unit 105 receives the information on the G _HL , R _LH , B _LH and the photographing conditions thereof via the network 902 or the medium 906, and these are spatially and temporally processed by the processing described in the first and second embodiments. G _HH , R _HH , and B _HH with higher resolution are output.

  With the above configuration, the imaging device and the processing device are configured separately, so that even if they are spatially separated, moving image signals and shooting condition information can be exchanged via the network 902 or the media 906. By configuring, the processing device can output a moving image with high spatiotemporal resolution.

  Note that the network 902 may be a local area network (LAN) built in a home, or a wide area network (WAN) such as the Internet. Alternatively, it may be a USB standard or IEEE 1394 standard communication line. Furthermore, it may be wireless or wired. The medium 906 includes a removable disk such as an optical disk and a removable hard disk, and a flash memory card.

  In each of the above-described embodiments, the imaging processing apparatus has been described as having various configurations illustrated in the drawings. For example, the image processing unit included in each configuration is described as a functional block. In terms of hardware, these functional blocks can also be realized by a single semiconductor chip or IC such as a digital signal processor (DSP), and can be realized by using, for example, a computer and software (computer program). You can also

  FIG. 16 illustrates an example of hardware of an imaging processing apparatus configured by a computer.

  The correspondence between each functional block of the image processing apparatus of each embodiment and the hardware shown in FIG. 16 is as follows. Hereinafter, as an example, the imaging processing apparatus 1 illustrated in FIG. 1 will be mainly described.

  The lens system 101, the dichroic mirror 102, the first imaging unit 103, and the second imaging unit 104 of the imaging processing apparatus 1 correspond to the camera 151 and the A / D converter 152 illustrated in FIG. A temporary buffer (not shown) and media 906 used by the image processing unit 105 in actual processing correspond to the frame memory 153 or the hard disk drive (HDD) 160 shown in FIG. The control unit 120 and the image processing unit 105 are realized by the CPU 154 in FIG. 16 that executes a computer program.

  A computer program for operating the computer of FIG. 16 is stored in, for example, the ROM 155. Alternatively, it may be stored on an optical disk or a magnetic disk. Further, it may be transmitted via a wired or wireless network, broadcast, or the like, and taken into the RAM 156 of the computer.

  The computer program is read and expanded in the RAM 156 by the CPU 154 which is a processor. The CPU 154 executes each coded instruction that is the actual state of the computer program. The digital image signal obtained as a result of executing the instruction is sent to the frame memory 157 and temporarily held, converted into an analog signal by the D / A converter 158, sent to the display 159, and displayed.

  The processing of the computer program that implements the image processing unit 105 is described, for example, according to the flowchart described below.

  For example, FIG. 17 is a flowchart showing the processing procedure of the present invention. Although this process is described as a process according to the first embodiment, it can also be understood as a process performed individually by the imaging device 901 and the processing device 903 of the second embodiment.

First, in step S101, the first imaging unit 103 and the second imaging unit 104 perform a long-exposure high-resolution low-frame-rate G image _GHL , a short-exposure low-resolution high-frame-rate R image R _LH , and a B image B _LH . Image. In step S102, the G up-converter 106 of the image processing unit 105 increases the resolution of the G component moving image. More specifically, it can be divided into steps S104 and S105. In step S104, the motion detection unit 108 of the G upconverter 106 performs motion detection. In step S105, the temporal resolution upconverter 109 uses the motion detection result and the like to calculate G _HH that minimizes (Equation 3) or (Equation 9) based on (Equation 4) or (Equation 10), respectively. Ask.

  In the next step S103, the R and B up-converters 107 increase the resolution of each of the R component and B component moving images. Thereafter, the control unit 120 determines whether or not imaging has been completed. If it is determined that the imaging has not been completed, the process is repeated from step S101. If it is determined that imaging has been completed, the process ends.

  FIG. 18 is a flowchart showing a detailed processing procedure of step S103 shown in FIG. This process corresponds to the process of the upconverter 107 for R and B shown in FIG.

In step S106 of FIG. 18, the down converter 110 reduces the resolution of _GHH based on the imaging conditions. In step S <b> 107, the coefficient estimation unit 111 estimates a coefficient to be applied to the restoration filter 112. In step S108, the estimated coefficient is applied to the restoration filter 112, and the restoration filter 112 increases the resolution of R _LH and B _LH and outputs R _HH and B _HH .

  FIG. 19 is a flowchart illustrating a detailed processing procedure of step S103 according to another example. This processing corresponds to the processing of the R and B up converter 107 shown in FIG.

In step S109, the down converter 113 reduces the resolution of _GHH based on the imaging conditions. In step S110, G _LH is subtracted from G _HH.

On the other hand, when the interpolation enlargement unit 114 performs interpolation enlargement of R _LH and B _LH based on the imaging condition in step S111, the local correlation calculation unit 115 calculates a local correlation value in step S112 based on the signal.

In step S113, when the weight generation unit 116 generates a weight, in step S114, the counter 117 stores the number of times the high frequency component is superimposed for each pixel. In step S115, the weight generated and output by the weight generation unit 116 is multiplied by the difference between _GHH and _GLH (that is, the high frequency component of G), and the image memory 118 is updated in step S116.

  In step S117, the normalization unit 119 normalizes the high-frequency component held in the image memory 118 by superimposing a plurality of times by dividing the high-frequency component by the write count value stored for each pixel in the counter 117.

In step S118, the normalized high frequency components are superimposed on the R and B images that have been interpolated and enlarged by the interpolation enlarging unit 114, and are output as R _HH and B _HH .

  The embodiments of the present invention have been described above. In the first to third embodiments, the case where the three components R, G, and B are separated by the dichroic mirror has been described, but the color component separation form is not limited to this. For example, a single image sensor (CCD or CMOS sensor) that captures images in three layers of R + G + B, R + G, and R in order in the depth direction may be used. Such an image sensor generates a color signal by using the fact that the depth of light entering the photodiode differs according to the wavelength of incident light, and thus no filter is required. When such an element is used, for example, R + G or R + G + B is imaged at a high resolution, long exposure, and low frame rate, and other images are received and processed at a low resolution, a short time, and a low frame rate. Similar effects can be obtained.

  INDUSTRIAL APPLICABILITY The imaging processing apparatus and image processing apparatus of the present invention are useful as a high-definition image capturing and reproducing apparatus and system using a camera with a reduced image sensor size. The imaging processing apparatus and the image processing apparatus of the present invention can be realized by a program that causes a known computer to execute the image processing method according to the present invention.

DESCRIPTION OF SYMBOLS 101 Lens system 102 Dichroic mirror 103 1st imaging part 104 2nd imaging part 105 Image processing part 106 G up-converter 107 R, B up-converter 120 Control part

Claims (14)

Information of moving images of the first color component and the third color component captured at the first spatial resolution and at the first temporal resolution, at a second spatial resolution higher than the first spatial resolution, and at the first An image processing apparatus that receives and processes information of a moving image of a second color component that is captured at a second time resolution lower than the one-hour resolution,
A first up-converter that generates a moving image of the second component with increased time resolution based on moving image information of the first color component and moving image information of the third color component;
Based on the information of each moving image of the first color component and the third color component, and the information of the moving image of the second component whose temporal resolution is increased, the first color component whose spatial resolution is increased and the A second up-converter that generates a moving image of the third color component,
The first up-converter generates an image of the second component with an increased time resolution based on an evaluation value including a constraint condition determined in an imaging process and a constraint condition of smoothness between colors. .   The image processing apparatus according to claim 1, wherein the first color component, the second color component, and the third color component are red, green, and blue, respectively.   2. The image processing apparatus according to claim 1, wherein the smoothness constraint condition between the colors is expressed by an expression that defines continuity between neighboring pixel values in a color space determined by the first to third color components. .   The image processing apparatus according to claim 2, wherein the three coordinate components of the pixel value in the color space are each a linear sum of the first to third color components.   The image processing apparatus according to claim 1, wherein the first up-converter calculates a pixel value constituting the moving image of the second component so that the evaluation value is minimized.   The image processing apparatus according to claim 1, wherein the evaluation value includes a constraint condition regarding a change in pixel value along a motion in a captured scene.   5. The evaluation value minimization is performed by decomposing a simultaneous equation of a sparse coefficient matrix into a filter operation with a fixed coefficient and a filter operation with a coefficient changing according to the content of the image. The image processing apparatus described.   2. The image processing apparatus according to claim 1, wherein the temporal and spatial resolution enhancement of the moving image of the second component in the first up-converter is performed by calculating only a region where the magnitude of motion in a scene is equal to or greater than a threshold value. .   The image processing according to claim 1, wherein the temporal and spatial resolution enhancement of the moving image of the second component in the first up-converter is performed by increasing a temporal smoothness constraint in a region with a small motion in the scene. apparatus.   The image processing apparatus according to claim 1, wherein the first up-converter changes the evaluation value according to a magnitude of motion of the moving image. A separation unit for separating the second component from visible light including the first color component, the second color component, and the third color component;
A first imaging unit that captures a moving image of the first color component and the third color component, wherein each of the images constituting the moving image is exposed at a first charge accumulation period with a first spatial resolution; A first imaging unit that shoots at a first time resolution;
A second imaging unit that captures a moving image of the second color component, wherein the first image is exposed in a second charge accumulation period longer than the first charge accumulation period, and each image constituting the moving image is the first image A second imaging unit that captures images at a second spatial resolution higher than the spatial resolution and at a second temporal resolution lower than the first temporal resolution;
A control unit that controls shooting conditions in the first imaging unit and the second imaging unit, and the image processing device according to any one of claims 1 to 10,
An imaging processing apparatus comprising: The first color component is received by receiving visible light including the first color component, the second color component, and the third color component, and exposing the light of the first color component and the third color component in a first charge accumulation period. And each image constituting the moving image of the third color component is exposed at a first spatial resolution, and the light of the second color component is exposed in a second charge accumulation period longer than the first charge accumulation period. An imaging unit that captures each image constituting the moving image of the second color component at a second spatial resolution higher than the first spatial resolution and at a second temporal resolution lower than the first temporal resolution; ,
A control unit that controls shooting conditions in the imaging unit, and the image processing device according to any one of claims 1 to 10,
An imaging processing apparatus comprising: Information of moving images of the first color composition and the third color component captured at the first spatial resolution and at the first temporal resolution, at a second spatial resolution higher than the first spatial resolution, and at the first Receiving information of a moving image of a second color component captured at a second time resolution lower than the one hour resolution;
A first up-conversion step for generating information on the moving image of the second component with an increased temporal resolution based on the moving image information of the first color component and the third color component;
The first color component and the third color with enhanced spatial resolution based on the information of the moving image of the first color component and the third color component and the information of the moving image of the second component with increased temporal resolution. A second up-conversion step for generating component moving image information, comprising:
The first up-conversion step includes
An image processing method for generating moving image information of the second component with an increased time resolution based on an evaluation value including a constraint condition determined in an imaging process and a constraint condition of smoothness between colors. A control program for an image processing apparatus,
Information of moving images of the first color composition and the third color component captured at the first spatial resolution and at the first temporal resolution, at a second spatial resolution higher than the first spatial resolution, and at the first Receiving information of a moving image of a second color component captured at a second time resolution lower than the one hour resolution;
A first up-conversion step for generating information on the moving image of the second component with an increased temporal resolution based on the moving image information of the first color component and the third color component;
The first color component and the third color with enhanced spatial resolution based on the information of the moving image of the first color component and the third color component and the information of the moving image of the second component with increased temporal resolution. Causing the image processing apparatus to execute a second up-conversion step of generating component moving image information;
The first up-conversion step includes
A control program for an image processing apparatus, which generates information of a moving image of the second component with an increased time resolution based on an evaluation value including a constraint condition determined in an imaging process and a constraint condition of smoothness between colors.

Images