JP2012008975A - Image processor, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology for improving a motion blur by performing motion correction in an HDR moving picture with high accuracy.SOLUTION: A motion vector determining part 114 outputs a motion vector from an image X to an image Z when reliability set in the image Z is higher than reliability set in an image W. When the reliability set in the image Z is lower than the reliability set in the image W, the motion vector determining part 114 outputs a motion vector from an image Y to the image W.

Description

  The present invention relates to a technique for obtaining a motion vector.

  In recent years, various devices including an image display unit such as a TV receiver and a PC monitor have been put into practical use. Various devices including a liquid crystal display device are used for these image display units.

  For example, a liquid crystal display device employs a method of dimming a backlight that is always emitting light with a liquid crystal shutter, and is said to be a hold-type display device because dimmed light is output over one frame period. ing. When a moving image is viewed in the hold-type display device (how to follow a moving part with a line of sight in moving image display), “motion blur” corresponding to the light output period is observed. In moving image display at 60 Hz, in principle, a “motion blur” of at least 16.7 ms is observed.

  On the other hand, for example, a self-luminous display device such as a CRT or an organic EL is said to be an impulse display device because it has a light emission period and a quenching period in one frame period. The impulse display device has a quenching period, so that “motion blur” due to tracking is difficult to observe. However, when the screen is enlarged, flicker is more noticeable in 60 Hz image display.

  As a technology to improve the “motion blur” caused by the above display method and flicker due to the large screen, a 120 Hz image is created by inserting and inserting an interpolated image according to the motion of the moving image with respect to the 60 Hz input image. A method of displaying is proposed. In 120 Hz moving image display, in principle, “motion blur” is half that of 60 Hz moving image display. In addition, it is difficult to feel flicker due to human visual characteristics.

  When generating an interpolation image, it is necessary to detect in which direction the display image moves between frames (= motion vector). In general, there is a so-called block matching method as a motion vector detection method. In the block matching method, a reference block having the same size as the reference block is extracted from a corresponding position in the reference frame image with respect to a reference block obtained by dividing the reference frame image to be processed into a predetermined size. Next, the sum of absolute differences (SAD) is calculated for all the pixels corresponding to the base block and the reference block. This SAD calculation is repeatedly performed while moving the reference block one pixel at a time within a predetermined search area set in the reference frame image. And the coordinate with respect to the starting point of the point where the SAD showed the smallest value is detected as a motion vector.

  On the other hand, as a technology for improving the image quality of video cameras, electronic cameras, etc., there is a technology that takes a plurality of images with different exposure amounts in the same scene, combines these image data with some calculation, and expands the dynamic range. Yes (Patent Document 1). Hereinafter, this high dynamic range technology is abbreviated as HDR (High Dynamic Range) technology.

  In the HDR technology, a long exposure image in which the exposure is set to be long so that a large amount of gradation information of the imaging target on the low luminance side is included, and a short exposure is set so that the gradation information of the imaging target on the high luminance side is included. And a short exposure image set to. By displaying this continuously on the display, it is possible to obtain a high dynamic range in which gradation is left on both the low luminance side and the high luminance side when viewed by integration with the human eye.

  When this HDR technology is applied to moving image display, a long exposure image and a short exposure image constituting one frame image are alternately displayed. Such a display image is hereinafter referred to as an HDR video. When displaying an HDR video on a hold-type display device such as a liquid crystal display device, it may be possible to improve motion blur by generating and inserting an intermediate image.

  However, in principle, an HDR moving image is likely to be crushed black (in a state in which the gradation is shifted to the low luminance side) or whiteout (in a state in which the gradation is shifted to the high luminance side). When these phenomena occur in the pixels in the reference block when obtaining the SAD, the search region is flat and the luminance change is poor, so the SAD peak is not a single point but a plurality or a plane. The reliability of the motion vector is reduced.

  As a method for dealing with this problem, there is a method of obtaining a motion vector of a reference block from a motion vector (motion vector of a peripheral block) when a block around the reference block is a reference block (Patent Document 2). Patent Document 2 discloses a technique for improving the reliability of the motion vector of the reference block by obtaining the motion vector of the reference block from the motion vectors of a plurality of peripheral blocks.

JP-A-6-273354 JP 2003-224854 A

  However, the technique of Patent Document 2 cannot obtain a correct motion vector when the peripheral block is also blacked out or overexposed. Further, in the case of an HDR video, even if one image is blacked out or white-out, gradation remains in the other image, and the movement of these two images is the same. Therefore, it is more reliable to adopt the motion vector in the reference block of the other image than to obtain the motion vector from the peripheral blocks of the blacked out or overexposed image. However, since Patent Document 2 does not assume an HDR moving image, the motion vector in the other image cannot be employed as described above.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for accurately correcting motion and improving motion blur in an HDR video.

  In order to achieve the object of the present invention, an image processing apparatus as an example of the present invention comprises the following arrangement. That is, a first acquisition unit that acquires two images captured under two different imaging conditions within the first frame period, and a second frame that is a frame period subsequent to the first frame period From a second acquisition unit that acquires two images captured under different two imaging conditions within a period, and an image X that is one of the two images acquired by the first acquisition unit, Of the two images acquired by the second acquisition means, a first calculation means for obtaining a motion vector to an image Z under the same imaging condition as the image X, and two images acquired by the first acquisition means Second calculation means for obtaining a motion vector from the image Y that is the other of the two images acquired by the second acquisition means to an image W under the same imaging condition as the image Y; The distribution of luminance values is on the low or high luminance side A prescribed reliability is set for the image Z, and the prescribed reliability is set for the image Z when the image Z is not biased to either the low luminance side or the high luminance side. A first setting unit that sets a higher reliability than the image W, and when the distribution of luminance values in the image W is biased toward a low luminance side or a high luminance side, a predetermined reliability is set for the image W. A second setting means for setting a reliability higher than the specified reliability for the image W when the setting is not biased to either the low luminance side or the high luminance side; When the motion vector calculated by the first calculation means and the motion vector calculated by the second calculation means are not the same, the reliability set for the image Z and the reliability set for the image W Means for performing magnitude comparison with degree, and as a result of the magnitude comparison, the image When the reliability set for the image W is higher than the reliability set for the image W, the motion vector calculated by the first calculation means is output and set for the image Z. And output means for outputting a motion vector calculated by the second calculation means when the set reliability is lower than the reliability set for the image W.

  According to the configuration of the present invention, motion correction can be performed with high accuracy in an HDR video, and motion blur can be improved.

FIG. 3 is a diagram illustrating an example of a functional configuration of an image processing apparatus. The figure for demonstrating block matching. The figure explaining the process which a reliability calculation part performs. The flowchart of the process which the motion vector determination part 114 performs. The figure which shows the example which expands the area | region for searching the reference | standard block 200. FIG.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
First, a configuration example of the image processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG. In the following description, each unit illustrated in FIG. 1 is assumed to be configured by hardware.

  The input terminal 100 receives an image signal of an image X (one) captured with a relatively short exposure time within the first frame period (current frame period). The input terminal 101 receives an image signal of an image Z captured with a relatively short exposure time within a second frame period that is a frame period subsequent to the first frame period. In the present embodiment, the exposure time of the image X and the exposure time of the image Z are described as being exactly the same, but may be slightly different.

  An image signal of the image Y (the other) captured with a relatively long exposure time within the first frame period (current frame period) is input to the input terminal 102. An image signal of an image W captured with a relatively long exposure time within the second frame period is input to the input terminal 103. In the present embodiment, the exposure time of the image Y and the exposure time of the image W are described as being exactly the same, but may be slightly different.

  That is, two images captured under two different imaging conditions in the first frame period and two images captured under two different imaging conditions in the second frame period by the input terminals 100 to 103. Can be acquired (first acquisition, second acquisition).

  When the motion vector search unit 110 receives the image signals of the images X and Z input via the input terminals 100 and 101, the motion vector search unit 110 obtains a motion vector 204 from the image X to the image Z (first calculation). . In addition, the motion vector search unit 110 sends the image Z (201) to the reliability calculation unit 112.

  The reliability calculation unit 112 determines whether the distribution of luminance values in the image Z is biased toward the low luminance side or the high luminance side, or whether the luminance value distribution is biased toward either the low luminance side or the high luminance side. The reliability 104 is set for Z (first setting).

  When the motion vector search unit 111 receives the respective image signals of the image Y and the image W input via the input terminals 102 and 103, the motion vector search unit 111 obtains a motion vector 205 from the image Y to the image W (second calculation). . Also, the motion vector search unit 111 sends the image W (203) to the reliability calculation unit 113.

  The reliability calculation unit 113 determines whether the distribution of luminance values in the image W is biased toward the low luminance side or the high luminance side, or whether the luminance value distribution is biased toward either the low luminance side or the high luminance side. A reliability 105 is set for W (second setting).

  The motion vector determination unit 114 determines a motion vector from the first frame period to the second frame period using the motion vectors 204 and 205 and the reliability levels 104 and 105. Then, the motion vector determination unit 114 outputs the determined motion vector to the outside via the output terminal 106.

  Next, the operation of the motion vector search unit 110 will be described in more detail. Since the motion vector search processing by the motion vector search unit 110 is a well-known technique, this will be described briefly.

  When receiving the image X and the image Z, the motion vector search unit 110 stores these as data. FIG. 2A shows a peripheral region of the reference block 200 described below in the image X. FIG. 2B shows a reference area (pixel block 291) which is an area for performing block matching with the standard block 200 in the image Z. The size of the pixel block 291 is not particularly limited, and may be the size of the image Z itself, or may be the size of one rectangle when the image Z is divided into a plurality of rectangles. In FIG. 2, the pixel value (luminance value) of each pixel is expressed by 8 bits (0 to 255).

  In FIG. 2A, a reference block 200 having a size of 3 pixels × 3 pixels is a reference block for searching for a motion vector. Then, in the pixel block 291 shown in FIG. 2B, an area that most closely matches the reference block 200 is searched.

  In the present embodiment, the motion vector search unit 110 moves the reference block 200 in units of pixels in the pixel block 291 and searches for an area that most closely matches the reference block 200 in the pixel block 291. As this method, block matching using the sum of absolute differences (SAD) between blocks is performed.

  In FIG. 2C, the center pixel position of the region that most closely matches the reference block 200 in the pixel block 291 is indicated by an asterisk. That is, the pixel position indicated by the star is the position at which the SAD peak is obtained. Then, the motion vector search unit 110 obtains a vector from the center position of the pixel block 291 to the pixel position of the star as a motion vector.

  Assume that the peripheral region of the standard block 202 is as shown in FIG. 2D and the pixel block 293 that is the reference region in the image Z is as shown in FIG. Similarly, in this case, the reference block 202 is moved in units of pixels in the pixel block 293, and an area that most closely matches the reference block 202 in the pixel block 293 is searched. At this time, as shown in FIG. 2 (f), it is assumed that there are a plurality of central pixel positions (star pixel positions) in an area that most closely matches the reference block 202 in the pixel block 293. In this case, the motion vector search unit 110 calculates a vector from the center position of the pixel block 293 to the pixel position of each star as a motion vector.

  In this way, the motion vector search unit 110 obtains all the motion vectors that can be obtained and outputs them as the motion vectors 204 to the motion vector determination unit 114. The operation of the motion vector search unit 111 is substantially the same as that of the motion vector search unit 110 except that the images to be used are the image Z and the image W. Description of is omitted.

  Next, the operation of the reliability calculation unit 112 will be described in more detail. In the case of FIGS. 2A to 2C, the reliability calculation unit 112 calculates the reliability with respect to the reference area (or the image Z (201) when the reference area is the entire image Z (201)). In the present embodiment, the reliability is defined as an index for measuring whether or not the reference region has sufficient information for searching for a motion vector.

  Hereinafter, processing performed by the reliability calculation unit 112 to calculate the reliability will be described. First, upon receiving the image Z (201), the reliability calculation unit 112 creates a histogram of luminance values in the reference area in the image Z (201). In the case of FIGS. 2A to 2C, a histogram of luminance values in the pixel block 291 is created.

  An example of the created histogram is shown in FIG. The horizontal axis represents the luminance value (0 to 255 is normalized to 0 to 1), and the vertical axis represents the appearance frequency value (cumulative frequency) of each luminance value. The motion vector search is required with high accuracy when the gradations appear evenly. However, if the reference area is blacked out or overexposed, the motion vector cannot be obtained accurately. That is, when the reference area is blacked out or overexposed, this reference area does not have information for searching for a motion vector in its male part, and its reliability is low. .

  Therefore, in the present embodiment, the reliability for the reference region is determined as follows. When a histogram curve created from the reference region passes through the region 300 (luminance value [0-0.25], cumulative frequency [50% -100%]), that is, the distribution of luminance values in the reference region is low. Suppose that it is biased toward the luminance side. In this case, it is defined that the reference area is blacked out. Further, when the histogram curve created from the reference region passes through the region 303 (luminance value [0.75-1], cumulative frequency [0% -50%]), that is, the distribution of luminance values in the reference region. Is biased toward the high luminance side. In this case, it is defined that the reference area is in an overexposure state.

  Furthermore, the area 300 is divided into two areas 301 and 302 according to the degree of black crushing. Similarly, the area 303 is divided into two areas 304 and 305 according to the degree of overexposure. Then, the reliability is defined in advance for each of the areas 301, 302, 304, 305 and the other areas 306. The reliability for each area is shown in FIG.

  When the curve of the histogram created from the reference region passes through the region 301 (region A) (in the case of the curve h1), the reliability for the reference region is “−2”. When the histogram curve created from the reference region passes through the region 302 (region B) instead of passing through the region 301 (region A), the reliability for the reference region is “−1”. When the histogram curve created from the reference region passes through the region 304 (region D) (in the case of the curve h3), the reliability with respect to the reference region is “−1”. When the histogram curve created from the reference region does not pass through the region 304 (region D) but passes through the region 305 (region E), the reliability for the reference region is “−2”. When the histogram curve created from the reference region passes through the region 306 (region C) without passing through any of the regions A, B, D, and E (in the case of the curve h2), the reliability for the reference region is “0”. "

  Of course, the reliability value is not limited to the above example. The essence of the reliability setting is that when the distribution of luminance values in the image Z is biased toward the low luminance side or the high luminance side, a specified reliability is set for the image Z, and the low luminance side and the high luminance side are set. If the image Z is not biased to any of the above, a higher reliability than the prescribed reliability is set for the image Z.

  The reliability calculation unit 113 performs the same operation as the reliability calculation unit 112. That is, when the distribution of luminance values in the image W is biased toward the low luminance side or the high luminance side, a specified reliability is set for the image W, and the luminance value is biased toward either the low luminance side or the high luminance side. If not, a reliability higher than the specified reliability is set for the image W.

  Next, processing performed by the motion vector determination unit 114 will be described with reference to FIG. 4 showing a flowchart of the processing. First, in step S <b> 400, the motion vector determination unit 114 compares the motion vector 204 from the motion vector search unit 110 with the motion vector 205 from the motion vector search unit 111. When both the motion vector search unit 110 and the motion vector search unit 11 obtain motion vectors one by one, the motion vector 204 and the motion vector 205 may be simply compared. However, when the motion vector search unit 110 obtains a plurality of motion vectors 204 and the motion vector search unit 11 obtains one motion vector 205, it is necessary to compare each motion vector 204 and the motion vector 205. This is the same when one is plural and the other is singular even when both are plural.

  Then, the motion vector determination unit 114 determines whether there is a set of motion vectors that match each other as a result of the comparison in step S400. As a result of this determination, if there is a pair that matches each other, the process proceeds to step S401; otherwise, the process proceeds to step S402.

  In step S401, the motion vector determination unit 114 determines whether there is only one set. As a result of this determination, if there is only one, the process proceeds to step S406. If there are a plurality of groups, the plurality of sets are set as output candidates, and the process proceeds to step S407.

  In step S406, the motion vector determination unit 114 selects one of the motion vector 204 and the motion vector 205, and outputs the selected one to the outside via the output terminal 106.

  In step S402, the motion vector determination unit 114 compares the reliability 104 obtained by the reliability calculation unit 112 with the reliability 105 obtained by the reliability calculation unit 113. As a result of the size comparison, if the reliability 104 is not equal to the reliability 105, the process proceeds to step S403. If reliability 104 = reliability 105 = 0, the process proceeds to step S404. If reliability 104 = reliability 105 <0, the process advances to step S405.

  In step S403, if the reliability 104> the reliability 105, the motion vector determination unit 114 selects the motion vector 204 as an output candidate. On the other hand, if reliability 104 <reliability 105, motion vector 205 is selected as an output candidate.

  In step S404, since both the motion vector 204 and the motion vector 205 have high reliability, the area for searching for the reference block 200 is expanded, and the motion vector search units 110 and 111 and the reliability calculation units 112 and 113 are set. Make it work. As in step S403, the motion vector with the higher reliability is selected as an output candidate. An example of enlarging a region for searching the reference block 200 will be described with reference to FIG. It is assumed that the reference block 200 was originally searched in the area 500, but the reliability 104 = reliability 105 = 0. In this case, the area 500 is enlarged to form an area 501, and the reference block 200 is searched in the area 501.

  In step S405, since the motion vector 204 and the motion vector 205 have low reliability, the motion vector determination unit 114 does not select any motion vector as an output candidate, and outputs a motion vector composed of 0 components as an output terminal. The data is output to the outside via 106. In the present embodiment, the maximum value of reliability is set to 0. However, as described above, other numerical values may be adopted as long as the reliability value maintains the above relationship. However, the essence of step S405 is that the reliability set for the image Z and the reliability set for the image W are the same, and if the respective reliability is lower than the specified value, 0 is set. Is to output a motion vector consisting of the components. Instead of outputting a motion vector composed of zero components, a code indicating “no motion vector” may be output.

  In step S407, the motion vector determination unit 114 outputs the motion vector selected as the output candidate to the outside via the output terminal 106. Here, when there is one motion vector selected as an output candidate, it may be output as it is. However, when there are a plurality of motion vectors selected as output candidates, it is necessary to further select one of them.

  Various methods can be considered for this selection method. For example, there are the following methods. An average value of an angle θ (cos θ may be used) formed by the target motion vector and each motion vector other than the target motion vector is obtained. Of the average values obtained for the respective motion vectors, the motion vector for which the smallest average value is obtained is set as a motion vector to be finally output. If there are two or more motion vectors having the same average value, the motion vector having the smallest vector size is set as the motion vector to be finally output. If all the vector sizes are the same, a motion vector composed of zero components is output as in step S405. Instead of outputting a motion vector composed of zero components, a code indicating “no motion vector” may be output.

  As described above, according to the present embodiment, motion correction can be performed with high accuracy in an HDR video, and motion blur can be improved. In the present embodiment, images having different exposure times are used as the image X and the image Y (the image Z and the image W). However, the images applicable to the image X and the image Y may be images acquired in other acquisition modes as long as they are two images captured under two different imaging conditions. For example, an image captured as an image for the right eye within the first frame period may be acquired as an image X, and an image captured as an image for the left eye may be acquired as an image Y. The same applies to the image Z and the image W.

[Second Embodiment]
In the first embodiment, the respective units illustrated in FIG. 1 have been described as being configured by hardware. However, some or all of them may be configured as a computer program. In this case, the computer program is installed in a device such as a PC (personal computer) and executed by a CPU or the like included in the device. As a result, this apparatus can execute the motion vector determination process described in the first embodiment.

[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (7)

First acquisition means for acquiring two images captured under two different imaging conditions within the first frame period;
Second acquisition means for acquiring two images captured under the two imaging conditions different from each other within a second frame period that is a frame period subsequent to the first frame period;
A motion vector from an image X which is one of the two images acquired by the first acquisition unit to an image Z having the same imaging condition as the image X out of the two images acquired by the second acquisition unit A first calculating means for obtaining
A motion vector from an image Y, which is the other of the two images acquired by the first acquisition unit, to an image W having the same imaging condition as the image Y among the two images acquired by the second acquisition unit A second calculating means for obtaining
When the distribution of the luminance value in the image Z is biased toward the low luminance side or the high luminance side, a specified reliability is set for the image Z, and either the low luminance side or the high luminance side is set. If there is no bias, first setting means for setting a reliability higher than the prescribed reliability for the image Z;
When the distribution of the luminance value in the image W is biased toward the low luminance side or the high luminance side, a specified reliability is set for the image W, and either the low luminance side or the high luminance side is set. 2nd setting means for setting a reliability higher than the prescribed reliability for the image W when there is no bias,
When the motion vector calculated by the first calculation means and the motion vector calculated by the second calculation means are not the same, the reliability set for the image Z and the image W are set. A means of comparing the degree of reliability with
When the reliability set for the image Z is higher than the reliability set for the image W as a result of the size comparison, the motion vector calculated by the first calculation means is output. Output means for outputting a motion vector calculated by the second calculation means when the reliability set for the image Z is lower than the reliability set for the image W; An image processing apparatus comprising: The first acquisition means acquires two images captured with different exposure times within the first frame period as the image X and the image Y,
The second acquisition unit is configured to capture the image Z captured at the same exposure time as the exposure time of the image X and the exposure time equal to the exposure time of the image Y within the second frame period. The image processing apparatus according to claim 1, wherein an image W is acquired. The first acquisition means acquires the image X captured as a right-eye image and the image Y captured as a left-eye image within the first frame period,
The second acquisition unit acquires the image Z captured as a right-eye image and the image W captured as a left-eye image within the second frame period. An image processing apparatus according to 1.   When the reliability set for the image Z and the reliability set for the image W are the same and the respective reliability is lower than a specified value, the output means is 0. The image processing apparatus according to claim 1, wherein a motion vector composed of the components is output.   The output means, when the motion vector calculated by the first calculation means and the motion vector calculated by the second calculation means are the same, the motion vector calculated by the first calculation means, 5. The image processing apparatus according to claim 1, wherein any one of the motion vectors calculated by the second calculation unit is output. An image processing method performed by an image processing apparatus,
A first acquisition step in which a first acquisition unit included in the image processing apparatus acquires two images captured under two different imaging conditions within a first frame period;
A second acquisition unit included in the image processing apparatus acquires two images captured under the two imaging conditions that are different from each other within a second frame period that is a frame period subsequent to the first frame period. A second acquisition step,
The first calculation means of the image processing apparatus uses the image X, which is one of the two images acquired in the first acquisition step, of the two images acquired in the second acquisition step. A first calculation step for obtaining a motion vector to the image Z under the same imaging condition as the image X;
The second calculation means included in the image processing apparatus uses the image Y, which is the other of the two images acquired in the first acquisition step, of the two images acquired in the second acquisition step. A second calculation step for obtaining a motion vector to the image W under the same imaging condition as the image Y;
The first setting means included in the image processing apparatus sets a specified reliability for the image Z when the distribution of luminance values in the image Z is biased toward a low luminance side or a high luminance side. A first setting step for setting a reliability higher than the prescribed reliability for the image Z when the image is not biased to either the low luminance side or the high luminance side;
The second setting means included in the image processing apparatus sets a specified reliability for the image W when the distribution of luminance values in the image W is biased toward a low luminance side or a high luminance side. A second setting step of setting a reliability higher than the specified reliability for the image W when the image is not biased to either the low luminance side or the high luminance side;
If the comparison unit included in the image processing apparatus does not have the same motion vector calculated in the first calculation step and the motion vector calculated in the second calculation step, the reliability set for the image Z is set. Performing a magnitude comparison between a degree and a reliability set for the image W;
When the reliability set for the image Z is higher than the reliability set for the image W as a result of the size comparison, the output means included in the image processing apparatus has the first When the motion vector calculated in the calculation step is output and the reliability set for the image Z is lower than the reliability set for the image W, the calculation is performed in the second calculation step. An image processing method comprising: an output step of outputting the motion vector obtained.   A computer program for causing a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 5.

Images