JP5219771B2 - Video processing apparatus and video processing apparatus control method - Google Patents

Description

  The present invention relates to a video processing apparatus that detects and suppresses motion blur in a video and a control method for the video processing apparatus.

  The video signal photographed by the photographing device having the time integration effect becomes an average image of the exposure time. For this reason, if the object to be photographed is moving, blurring (hereinafter referred to as motion blur) occurs in the image, the sharpness is lowered, and the image is deteriorated. Note that this motion blur also occurs when the photographing apparatus moves while the object to be photographed is fixed. It also occurs when the object and the imaging device are both moving (except when the object and the imaging device are moving in the same direction and at the same speed). That is, it occurs when the object and the photographing apparatus move relatively.

  Conventionally, there have been the following methods for detecting motion blur and improving images. First, Japanese Patent Laid-Open No. 2006-081150 (Patent Document 1) acquires motion vector information indicating blur characteristics and parameter values of shutter speed information from a photographing apparatus to detect blur, and according to those values. A method for performing the improvement process is described.

Japanese Patent Laid-Open No. 2005-260929 (Patent Document 2) divides an image into a foreground area and a background area in a moving image, predicts a mixture ratio of the foreground area and the background area based on motion vector information, and improves image degradation. How to do is described.
Japanese Patent Laid-Open No. 2006-081150 JP 2005-260929 A

  However, in the method of Patent Document 1 described above, image degradation due to motion blur cannot be improved without information from the imaging apparatus. For example, in a video processing apparatus that acquires a video signal from a broadcasting station such as a television, image deterioration cannot be improved because information from a photographing apparatus cannot be acquired.

  Furthermore, in the method of Patent Document 1, an area without motion blur, such as a telop synthesized after shooting, is processed in the same manner as an area with motion blur, so that the outline of the telop is excessively emphasized.

  Further, in the method described in Patent Document 2, since the mixture ratio of the foreground area and the background area is predicted from the motion vector value, it is not possible to detect the presence or absence of image degradation due to motion blur, and it is improved uniformly according to the motion vector value. Process. For this reason, the improvement process is uniformly performed on a moving image having no motion blur, and the image is uncomfortable.

  The present invention has been made in view of the above circumstances, and its object is as follows. That is, the presence / absence of motion blur and the degree of image degradation due to motion blur are detected for each pixel or each arbitrary region, and improvement processing corresponding to the degree of degradation is performed on a region with motion blur. This is to provide a clear image with improved image quality.

In order to achieve the above object, the first aspect of the present invention employs the following configuration. That is,
A video processing device that corrects motion blur generated in a captured video,
A motion vector detection unit that divides an image included in a current frame to be corrected into detection units of a predetermined size and detects a motion vector for each detection unit;
A spatial frequency division unit for obtaining a frequency component of the image region and obtaining a frequency component of the image region including the detection unit and a frequency component in a previous frame of the image region including the start point of the detected motion vector; ,
Based on the detected motion vector and the change in the magnitude and frequency component of the motion vector between the current frame and the previous frame, the magnitude of the motion vector increases at least from the previous frame to the current frame. When the frequency component changes to a low frequency, or when the magnitude of the motion vector decreases and the frequency component changes to a high frequency , a motion blur detection unit that determines that there is motion blur in the detection unit;
An image processing process comprising: an improvement processing unit that corrects motion blur based on a change in magnitude of the motion vector and a change in frequency component when the motion blur detection unit determines that motion blur exists Device.

In the second invention of the present invention, the following configuration is adopted. That is,
A control method of a video processing device for correcting motion blur that occurs in a captured video,
Dividing an image included in a current frame to be corrected into detection units of a predetermined size, and detecting a motion vector for each detection unit;
Obtaining the frequency component of the image region, and determining the frequency component of the image region including the detection unit and the frequency component in the previous frame of the image region including the start point of the detected motion vector;
Based on the detected motion vector and the change in the magnitude and frequency component of the motion vector between the current frame and the previous frame, the magnitude of the motion vector increases at least from the previous frame to the current frame. And a motion blur detection step for determining that there is motion blur in the detection unit when the frequency component changes to a low frequency, or when the magnitude of the motion vector decreases and the frequency component changes to a high frequency ,
In the motion blur detection step, when it is determined that there motion blur, based on the change in the change of the magnitude and frequency components of the motion vector, and having a improved process step of correcting the motion blur A control method for a video processing apparatus.

  According to the present invention, the presence or absence of motion blur and the degree of degradation of an image due to motion blur are detected for each pixel or arbitrary area, and an improvement process corresponding to the degree of degradation is performed on a region with motion blur, thereby It is possible to provide clear images with improved image quality.

  Exemplary embodiments of the present invention will be described in detail below with reference to the drawings.

<Embodiment 1>
In the present embodiment, a video processing apparatus that uses a change in a spatial frequency component between a previous frame and a change in a motion vector in order to detect motion blur of a target pixel will be described.

(Configuration of video processing device)
The configuration of the video processing apparatus according to the present embodiment will be described with reference to the block diagram shown in FIG. The video processing apparatus includes a frame memory unit 102, a motion vector detection unit 103, a pixel extraction unit 104, and a previous frame spatial frequency division unit 105. The video processing apparatus also includes a current frame spatial frequency division unit 106, a frequency component change amount calculation unit 107, a motion vector frame memory unit 108, a motion vector value extraction unit 109, and a motion vector change amount calculation unit 110. The video processing apparatus further includes a target pixel blur determination unit 111, an improvement processing adaptation determination unit 112, an improvement processing value frame memory unit 113, an improvement processing value extraction unit 114, and an improvement processing unit 115.

(Process flow)
Hereinafter, motion blur detection and improvement processing will be described. First, the video processing apparatus acquires an input video signal 101 having a predetermined frame rate from a broadcast signal or the like.

  The frame memory unit 102 generates and stores a frame delay signal of the input video signal 101. In this embodiment, it is assumed that the signal is one frame delayed.

The motion vector detection unit 103 detects a motion vector in the video from the input video signal 101 and the 1-frame delay signal stored in the frame memory unit 102. At this time, the frame image is divided into detection units of a predetermined size, and a motion vector is detected for each detection unit. The size (number of pixels) of the detection unit is arbitrary, and an area composed of a plurality of pixels may be used as the detection unit, or a single pixel. In this embodiment, one pixel is used as one detection unit. The direction of the motion vector is the horizontal direction.

  The motion vector detection method will be described with reference to the pixel of interest in the current frame in FIG. First, the difference between the pixel value of the pixel of interest and the pixel values of 30 pixels on the left and right (a total of 61 pixels) from the pixel of interest one frame before is obtained. A vector from a pixel having the minimum difference (hereinafter referred to as a corresponding pixel) to a target pixel is a motion vector. For example, when the difference indicated by the arrow A in FIG. 2 is the minimum value, the direction of the motion vector of the target pixel is the left direction and the size is 1 (hereinafter, the size of the motion vector is changed to “ Motion vector value ”). When the detection unit is an area composed of a plurality of pixels, the motion vector can be detected by using the above method or performing block matching on the average pixel value of the pixels included in the detection unit.

  Here, in the present embodiment, as shown in FIG. 3, it is assumed that the round moving body is moving from the left to the right, and that the motion vector having the right direction and the size N is detected for the pixel A.

  The pixel extraction unit 104 extracts a region near the start point of the motion vector detected by the motion vector detection unit 103 (hereinafter referred to as an extraction region) from the one-frame delay signal stored in the frame memory unit 102. In this embodiment, since a motion vector of N pixels in the right direction is detected, an X × Y pixel region is extracted centering on a point moved N pixels to the left from the pixel A as shown in FIG. The size of the extraction region (X × Y) is set to a size that can detect motion blur by performing spatial frequency division, and is 9 × 9 pixels here.

  The previous frame spatial frequency division unit 105 detects spatial frequency division information from the extraction region of the previous frame obtained by the pixel extraction unit 104. Here, in the present embodiment, as shown in FIG. 4, the pixels in the image region are divided into n from the low-frequency component to the high-frequency component, and the component amount in each band is detected. In the present embodiment, Hadamard transform is performed on an extraction region of 9 × 9 pixels to detect spatial frequency division information.

  The current frame spatial frequency division unit 106 detects spatial frequency division information from a 9 × 9 pixel region centered on the target pixel of the current frame of the input video signal. The detection method is the same as the method performed by the previous frame spatial frequency division unit 105 for the one-frame delayed signal.

  The frequency component change amount calculation unit 107 calculates the change amount of the frequency component from the spatial frequency division information of the previous frame and the current frame. In the present embodiment, as shown in FIG. 5, the number of spatial frequency divisions n = 8 is used, and the component amounts of the respective bands are compared. Here, the spatial frequency division information of the previous frame is shown in FIG. 5A, and the spatial frequency division information of the current frame is shown in FIG. 5B. Therefore, the frequency component change amount can be calculated by taking the difference between the component amounts for each band in FIGS. 5A and 5B. Comparing FIG. 5 (a) and FIG. 5 (b), FIG. 5 (b) has a larger amount of high-frequency components, and changes from low-frequency components to high-frequency components from the previous frame to the current frame. I understand that.

  The motion vector frame memory unit 108 delays the motion vector obtained for each pixel by the motion vector detection unit 103 by an arbitrary frame. In this embodiment, the delay is one frame.

The motion vector value extraction unit 109 extracts a motion vector value from the motion vector frame memory unit 108 according to the motion vector obtained by the motion vector detection unit 103. Here, in the present embodiment, as shown in FIG. 6, the motion vector of the right direction and the size of N pixels is detected for the target pixel. Therefore, the motion vector value in the previous frame is extracted for the corresponding pixel located at the start point of the motion vector.

  The motion vector change amount calculation unit 110 calculates a motion vector change amount from the extracted motion vector of the corresponding pixel in the previous frame and the motion vector of the target pixel in the current frame. The motion vector change amount is a difference between two motion vector values.

The comparison results are roughly divided into the following five (a) to (e). The motion vector change amount calculation unit 110 determines the tendency of the motion amount based on this result. If the motion vector value of the current frame is Va and the motion vector value of the previous frame is Vb,
(A) When Va> Vb, it is determined that the amount of motion has increased.
(B) If Va <Vb, it is determined that the amount of motion has decreased.
(C) When Va = Vb, it is determined that there is no change in the motion amount.
(D) When Va = 0, it is determined as stationary.
(E) If Vb cannot be extracted, that is, if a moving object appears from the outside of the screen, it is determined that the motion vector change amount cannot be detected.

  The target pixel blur determination unit 111 performs target pixel blur determination based on the motion vector value of the target pixel of the current frame and the spatial frequency division information. First, the motion vector value and the low frequency component amount of the current frame are plotted on a graph in which the horizontal axis shown in FIG. 7 is the motion vector value and the vertical axis is the low frequency component amount of the spatial frequency. Here, the low-frequency component amount is a sum of component amounts of 0 to 2, which are low-frequency components, when the number of divisions of the spatial frequency division information of the pixel of interest is 8 as shown in FIG.

  Subsequently, it is determined whether the plotted portion in FIG. 7 is between the predetermined inclination A and the predetermined inclination B. When the plotted portion is above the slope A, it can be determined that the motion vector value is largely blurred and is not a motion blur caused by shooting. On the other hand, when the plotted portion is below the slope B, since the blur is small, it can be determined that it is not motion blur by recognizing it as a video such as a telop. In this embodiment, as shown in FIG. 7, when the motion vector value of the pixel of interest is 3, the low-frequency component amount is between the gradient A and the gradient B, so there is motion blur of the pixel of interest. Can be determined. When the motion vector value of the pixel of interest is 4, since the low-frequency component amount does not fall between the gradient A and the gradient B, it can be determined that there is no motion blur of the pixel of interest.

The improvement process adaptation determination unit 112 determines the presence / absence of motion blur of image quality, and determines what image quality improvement process should be performed according to the determination. The determination result at this time is divided into four.
(1) A still image.
(2) Motion blur.
(3) It is not motion blur.
(4) Cannot be determined.

The determination method is summarized in FIG.
The determination of (1) is when the motion vector change amount calculation unit 110 determines that the target pixel of the current frame is stationary. At this time, the amount of change in the frequency component does not matter.

(2) is a case where it is determined that the motion is blurred. In FIG. 9, “when the motion vector change amount is increased and the frequency component change amount is changed to a low region” and “the motion vector change amount is decreased and the frequency component change amount is changed to a high region” are applicable. This uses the motion blur property that blurring increases as the moving speed of the moving object increases and blurring decreases as the moving speed decreases.

(3) is a case where it is determined that there is no motion blur. In FIG. 9, cases other than “(2)” among “increase in motion vector change amount” or “decrease in motion vector change amount” are applicable. Further, when the motion vector change amount does not change, it corresponds regardless of the frequency component change amount.

  The determination of (4) is when the motion vector change amount calculation unit 110 cannot detect the motion vector change amount. At this time, the amount of change in the frequency component does not matter.

  The improvement processing value frame memory unit 113 delays an improvement processing value output from the improvement processing unit 115 described later by an arbitrary frame. In this embodiment, the delay is one frame.

  The improvement process value extraction unit 114 extracts an improvement process value from the improvement process value frame memory unit 113 in accordance with the motion vector detected by the motion vector detection unit 103. In the present embodiment, since a motion vector having the right direction and the size of N pixels is detected, as shown in FIG. 10, the improvement processing value in the previous frame of the corresponding pixel that is the starting point of the vector is extracted. .

  The improvement processing unit 115 determines a motion blur improvement processing value according to the determination results of the improvement processing adaptation determination unit 112 and the target pixel blur determination unit 111, and performs the improvement processing. In the present embodiment, the improvement is performed by the contour emphasis correction processing, and the strength of the contour emphasis correction processing is the improvement processing value. The improvement processing value corresponds to the correction amount in the present invention. The improvement processing unit 115 corresponds to a correction amount determination unit in the present invention.

  When the improvement process adaptation determination unit 112 determines (1), it determines the strength of contour enhancement correction suitable for a still image.

  When the improvement process adaptation determination unit 112 determines (2), that is, when there is motion blur, the gain value is determined according to the frequency component change amount and the motion vector change amount. Based on the determined gain value, the improvement processing value of the previous frame extracted by the improvement processing value extraction unit 114 is corrected to determine the strength of contour enhancement correction.

  For example, when “increase in motion vector change amount and spatial frequency division information changes to low frequency” in FIG. 9, the gain value A with respect to the motion vector change amount from the graph of FIG. 11 and the frequency component from the graph of FIG. A gain value B with respect to the change amount is obtained. Then, a gain value C corresponding to the product of the gain value A and the gain value B is obtained from the graph of FIG. Finally, the strength of the contour enhancement correction is determined by multiplying the gain value C by the improvement processing value of the previous frame.

  In this way, by modifying the improvement processing value based on the change in the motion vector value, it is possible to prevent the improvement processing value between the previous frame and the current frame from changing abruptly, and to improve the contour emphasis correction with less sense of incongruity. Processing becomes possible. In the present embodiment, the frequency component change amount obtained here corresponds to the feature amount in the present invention.

  When the improvement process adaptation determination unit 112 determines (3), the improvement process value of the previous frame extracted by the improvement process value extraction unit 114 is directly determined as the strength of the contour enhancement correction.

  When the improvement process adaptation determination unit 112 determines (4), the moving object appears in the screen from the outside of the screen, and thus the improvement processing value of the previous frame cannot be used. Therefore, the strength of the contour enhancement correction is first determined according to the determination result of the target pixel blur determination unit 111. Since the target pixel blur determination is obtained from the motion vector and spatial frequency division information for the current frame, it can be used even in this case.

  When the improvement process adaptation determination unit 112 determines (4) and the target pixel blur determination unit 111 determines that there is motion blur, contour enhancement correction is performed based on the motion vector value of the target pixel of the motion vector detection unit 103. Determine strength. For example, as shown in the graph of FIG. 14, the strength of edge enhancement correction is determined according to the motion vector value of the target pixel.

  When the improvement process adaptation determination unit 112 determines (4) and the target pixel blur determination unit 111 determines that there is no motion blur, the target pixel determines that there is no motion blur, and the contour is suitable for a still image. Determine the strength of enhancement correction.

  Through the series of processes described above, the improvement processing unit 115 can perform appropriate edge enhancement correction on the target pixel. By performing this process on each pixel, an output video signal 116 is obtained.

(Summary)
The effects of the above technique will be described with reference to FIG. In the previous frame video signal of FIG. 15, both the object A and the object B are stationary, and in the current frame video signal, both the object A and the object B are moving objects. It is assumed that motion blur occurs in the object A from the previous frame to the current frame, and motion blur does not occur in the object B. Specifically, the object A is an object in which motion blur is generated by the photographing apparatus, and the object B is a texture or a telop multiplexed on the video after photographing.

  When this video is processed using the video processing apparatus of the present embodiment, the motion vector change amount of the pixel of interest 1 in FIG. 15 increases and the frequency component change amount changes to a low frequency, which is motion blur. Can be detected. Then, motion blur can be adaptively improved according to the amount of change. In addition, although the amount of change in the motion vector of the target pixel 2 in FIG. As described above, according to the video processing apparatus of the present embodiment, motion blur can be detected, and the image quality of an area with motion blur can be effectively improved without a sense of incongruity in the video.

<Embodiment 2>
In the present embodiment, a video processing device that detects a motion blur region and a blur degree using pixel inclination, calculates a correction value, and corrects the image processing device and a control method thereof will be described.

  The motion blur will be described with reference to FIG. First, the video signal 1 on the left is an example in which motion blur does not occur because the exposure time is short. A, B, and C are continuous frames, and are shot with a short exposure time and a long interval. In this case, there is no mixing of pixel values at the boundary between the moving foreground and the stationary background, and they are clearly separated.

  Next, the video signal 2 on the right is an example in which motion blur occurs due to a long exposure time. D and E are continuous frames. In this case, the boundary between the moving foreground and the stationary background becomes a pixel value in which the foreground and the background are mixed, and motion blur occurs. Here, the motion blur region is shown by shading.

Such a decrease in sharpness due to motion blur becomes more prominent when an intermediate frame is generated in order to increase the frame rate. FIG. 17 shows the relationship between frame rate conversion and blur. In the figure, A is an image with a frame rate of 60 Hz that displays the photographed image as it is, and the sharpness is reduced due to motion blur. However, in this case, the blurred region where the foreground and the background are mixed with the pixel value 11a and the pixel value 12a does not overlap. B in the figure is an image with a frame rate of 120 Hz in which the pixel value 13b of the intermediate frame is inserted by frame rate conversion. In this case, since the amount of motion of the foreground between frames is small, blur regions overlap and sharpness is significantly reduced.
Therefore, as shown in C in the figure, it is desired to reduce the motion blur region to eliminate the overlap and to improve the sharpness of the image.

(Configuration of video processing device)
The configuration of the video processing apparatus will be described with reference to the block diagram shown in the figure. FIG. 18 is a block diagram of a system to which the present invention is applied. The figure includes a frame memory unit 202, a motion vector detection unit 203, a pixel inclination detection unit 204, an inclination normalization processing unit 205, a motion blur detection unit 206, an improvement processing value selection unit 207, and an improvement processing unit 208.

  Hereinafter, the function of each block will be described. The frame memory unit 202 generates and stores a frame delay signal of the input video signal 201. In this embodiment, it is assumed that the signal is one frame delayed. The motion vector detection unit 203 detects a motion vector for each detection unit.

  The pixel inclination detection unit 204 includes a differentiation circuit, and detects a change in the value of successive pixels. Since the blurred area spreads in proportion to the motion vector value, the pixel inclination of the blurred area has an inversely proportional relationship with the motion vector value. Therefore, the inclination normalization processing unit 205 normalizes the inclination value by multiplying the inclination value detected by the pixel inclination detection unit 204 by the motion vector value detected by the motion vector detection unit 203. That is, the normalization referred to in this embodiment is to weight an inclination value with a motion vector value to obtain an evaluation value.

  The motion blur detection unit 206 detects a blur region from the normalized tilt calculated by the tilt normalization processing unit 205 and the motion vector detected by the motion vector detection unit 203. Details of the method for detecting the blurred region will be described later.

  The improvement process value selection unit 207 selects and outputs an improvement process value corresponding to the motion vector value detected by the motion vector detection unit 203 for the blur region detected by the motion blur detection unit 206. The improvement processing unit 208 improves motion blur in the image according to the improvement processing value output from the improvement processing value selection unit 207.

(Motion vector detection / motion blur improvement method)
In the following, a method for improving an image by detecting a motion vector, detecting and normalizing a pixel inclination, detecting a motion blur region, calculating a gain value for improving the motion blur will be described.

  The motion vector detection unit 203 detects a motion vector for each detection unit obtained by dividing a frame image into a predetermined size. In the present embodiment, one pixel is used as one detection unit, but the detection unit may be configured by a plurality of pixels as in the first embodiment.

  FIG. 19 is a timing chart showing an operation example of this embodiment. FIG. 19A is a one-dimensional signal obtained by cutting a two-dimensional image in a frame of the input video signal 201 in the direction of the motion vector with respect to pixels through which the motion vector detected by the motion vector detection unit 203 passes. A constant region with a high pixel value indicates the foreground, a low constant region indicates the background, and a region where the pixel value changes indicates a region where the foreground and the background are mixed due to motion blur.

  The motion vector of the pixel of interest can be detected by the following method using the block matching method. FIG. 20 shows the state of motion vector detection. Here, b in FIG. 19 shows a pixel-by-pixel motion vector detected by the motion vector detection unit 203. As the motion vector, a motion vector corresponding to the pixel motion is input to the foreground region and the blur region, and a still vector is input to the still background. In the figure, it is assumed that the foreground is moving with a vector indicated by an arrow.

In FIG. 20, frame 1 indicates an image of the input video signal 201, and frame 2 indicates an image input from the frame memory unit 202. Each image is composed of a moving foreground (21a, 21b) and a stationary background (22a, 22b), and shows a state in which the foreground has moved from frame 2 to frame 1. In block matching, a block (23a) of, for example, horizontal ± 8 pixels and vertical ± 8 pixels, for example, is extracted around the pixel of interest (24a) of frame 1, and the difference in pixel value from the same-shaped block (23b) of the image of frame 2 Calculate The absolute value of the pixel value difference is calculated and the sum is calculated over the entire block. The total value is calculated so that the surrounding area is scanned around the position of the pixel of interest, and the position where the total value is the smallest is detected as the block (23b) with good block matching.

It can be considered that the center pixel (24b) of the detection block (23b) of frame 2 has moved to the target pixel (24a) of frame 1. Therefore, the motion vector of the pixel of interest (24a) is a vector from the center pixel (24b) of the detection block (23b) toward the pixel of interest (24a). With such a method, a vector of all pixels of the image can be detected. In the case of FIG. 20, as the motion vector of frame 1, a vector having motion in the foreground area (21c) is detected, and a zero vector is detected in the background area (22c).

  From the pixel inclination detection unit 204, the pixel inclination shown in FIG. 19c is detected and output. In the present embodiment, the pixel inclination obtained here corresponds to the feature amount in the present invention.

  The inclination normalization processing unit 205 multiplies the pixel inclination detection value by the motion vector size of the pixel to obtain a normalized inclination value.

  The motion blur detection unit 206 detects a region where the normalized inclination value is within a predetermined range as a motion blur region. Since it is considered that the slope is less than a certain value in the region where motion blur occurs, the slope upper limit threshold value b is determined. On the other hand, the slope lower than a certain level can be determined as noise, so the slope lower limit threshold value a is determined. Further, assuming that the time integration time at the time of shooting is a time of one frame, it is predicted that motion blur is generated by the same number of pixels as the movement amount indicated by the motion vector. From this, the condition for determining the range is that pixels whose level is greater than or equal to the threshold value a and less than or equal to the threshold value b continue for the vector size or more, or pixels whose level is greater than or equal to the threshold value −b and less than or equal to the threshold value −a It becomes a case where it continues for the size of a vector.

  The improvement process value selection unit 207 outputs an improvement process value corresponding to the motion vector value to the area detected as the motion blur area (e in FIG. 19).

  The improvement processing unit 208 performs motion blur processing of the input video signal according to the improvement processing value, and obtains an output video signal 209 indicated by f in FIG. The improvement processing unit 208 can use, for example, contour enhancement processing, and may switch the intensity in accordance with the improvement processing value.

  When the motion vector is large, the inclination becomes small, but since the inclination normalization processing unit 205 normalizes the inclination value with the motion vector, it is possible to process from a small vector to a large vector.

  FIG. 21 shows an operation example of an image having no motion blur such as a telop synthesized after shooting. In such an input video signal, a large pixel inclination is detected, and the normalized inclination is not less than the threshold value b or not more than the threshold value −b. Therefore, the input image signal is not detected as a motion blur region and is subjected to a motion blur improvement process. There is no.

(Process flow)
The video processing flow of the present embodiment will be described with reference to the flowchart shown in FIG.

  In step S1, the motion vector detection unit 203 detects a motion vector for each pixel.

  In step S2, the pixel inclination detection unit 204 calculates the pixel inclination of the input video signal.

  In step S3, the inclination normalization processing unit 205 normalizes the inclination value calculated in step S2 with the motion vector value detected in step S1.

  In step S4, the motion blur detection unit 206 determines whether or not the slope value normalized in S3 is between the threshold value a and the threshold value b and continues in the direction of the motion vector detected in S1 for the magnitude or more. Determine.

  If the determination result is Yes, a motion blur region is detected in step S5. When the normalized pixel inclination value is negative, an area where the value is between the threshold value -a and the threshold value -b and is continuous in the vector direction by the magnitude or more is defined as a motion blur area. To detect.

  In step S6, the improvement process value selection unit 207 selects and outputs an improvement process value corresponding to the motion vector value in the motion blur region.

  In step S <b> 7, the improvement processing unit 208 improves motion blur using the input improvement process value.

  In step S8, the improvement processing unit 208 outputs a moving image having a high frame rate in which each pixel value with motion blur is corrected.

(Summary)
According to the present embodiment, a motion blur region can be detected in this way, and motion blur improvement processing can be performed with an appropriate improvement processing value in a region where motion blur occurs, and the sharpness of a video converted at a high frame rate can be improved. .

<Embodiment 3>
In the second embodiment, the motion blur detection range is a fixed constant. Therefore, a blurred region can be detected with an image signal having a large level difference between the foreground and the background and a contrast, but may not be detected when the level difference is small. Therefore, in the present embodiment, the detection range is switched according to the contrast of the image signal to improve the accuracy of motion blur region detection.

(Configuration of video processing device)
A block diagram of this embodiment is shown in FIG. Here, a detection threshold value generation unit 210 is added to the configuration of the second embodiment. The detection threshold generation unit 210 generates a blur threshold detection threshold according to the contrast level of the input video signal 201.

(Motion blur detection method)
FIG. 24 shows a timing chart showing an operation example of video processing. In FIG. 24, g represents a signal obtained by applying a maximum value filter to the input video signal 101 over a pixel range that is twice the size of the vector, and h represents a signal obtained by performing a minimum value filter over a pixel range that is twice the size of the vector. Show.

FIG. 25 shows a timing chart of the maximum value filter and the minimum value filter. The maximum value filter is a filter that selects the pixel value having the largest pixel value in the reference pixel range. For example, the pixel value of l5 of the maximum value filter output signal of b in FIG. 24 is that of i7 which is the maximum pixel value in (i3, i4, i5, i6, i7) when the reference pixel range is 5 pixels. A pixel value is selected. Similarly, the minimum value filter is a filter that selects the pixel value having the smallest pixel value in the reference pixel range. I in FIG. 24 is a signal indicating the difference between the signal g and the signal h, and is a signal indicating the magnitude of the contrast between the foreground and the background, and can be used as a threshold value for motion blur detection.

  The upper limit of the time estimated as the time integration time at the time of capturing an image is considered to be the frame period. At this time, the pixel inclination of the motion blur region is proportional to the reciprocal of the motion vector value, and the product of the pixel inclination and the motion vector value is equal to the contrast. On the other hand, the lower limit of the time integration time during shooting that requires improvement of motion blur is the reciprocal of the frame rate converted by the frame rate conversion. When the frame rate is doubled, the motion blur can be improved up to the time integration time at the time of shooting. At this time, the pixel tilt of the motion blur region is double the tilt at the above upper limit time. Correspondingly, the product of the pixel inclination and the motion vector value is equal to twice the contrast.

  As described above, when the motion blur improvement target is determined, the upper limit of the motion blur region detection threshold may be set to a value twice the contrast, and the lower limit may be set to the contrast value. That is, by setting the value of the filter output difference i in FIG. 24 and twice the value as a threshold value, motion blur to be processed can be appropriately detected.

  As described above, when a motion blur region is detected, by selecting an improvement processing value according to the magnitude of the vector, it is possible to perform motion blur improvement processing with an appropriate improvement processing value in a region where motion blur occurs. It is possible to improve the sharpness of the video converted at a high frame rate.

1 is a block diagram illustrating a configuration of a video processing apparatus according to a first embodiment. The figure which shows the detection method of a motion vector value. The figure which shows the method of calculating | requiring an extraction area | region from a front frame. The figure which shows the component amount in a spatial frequency division | segmentation. The figure which shows the change of the component amount in a spatial frequency division | segmentation. The figure which shows the motion vector value extraction of the corresponding pixel of a front frame. The figure which shows the low-pass component amount with respect to a motion vector value. The figure which shows the component amount in the spatial frequency division | segmentation divided into eight. A table used to determine the content of improvement processing. The figure which shows extraction of the improvement process value of a front frame. The figure which shows the gain value A with respect to motion vector variation | change_quantity. The figure which shows the gain value B with respect to frequency component variation | change_quantity. The figure which shows the gain value C with respect to motion vector variation | change_quantity x frequency component variation | change_quantity. The figure which shows the intensity | strength of the outline emphasis correction | amendment with respect to the motion vector value of an attention pixel. FIG. 6 is a diagram illustrating an effect of the first embodiment. The figure which shows the mode of motion blur. The figure which shows the influence of frame rate conversion and motion blur. FIG. 3 is a block diagram illustrating a configuration of a video processing apparatus according to a second embodiment. 9 is a timing chart 1 illustrating an operation example of the video processing apparatus according to the second embodiment. The figure explaining the motion vector of a motion foreground and a stationary background. 9 is a timing chart 2 illustrating an operation example of the video processing apparatus according to the second embodiment. The flowchart which shows operation | movement of a video processing apparatus. FIG. 4 is a block diagram illustrating a configuration of a video processing apparatus according to a third embodiment. 10 is a timing chart showing an operation example of the video processing apparatus according to the third embodiment. Timing chart of maximum value filter and minimum value filter.

Explanation of symbols

103, 203 Motion vector detection unit 107 Frequency component change amount calculation unit 110 Motion vector change amount calculation unit 115, 208 Improvement processing unit 206 Motion blur detection unit

Claims (10)

A video processing device that corrects motion blur generated in a captured video,
A motion vector detection unit that divides an image included in a current frame to be corrected into detection units of a predetermined size and detects a motion vector for each detection unit;
A spatial frequency division unit for obtaining a frequency component of the image region and obtaining a frequency component of the image region including the detection unit and a frequency component in a previous frame of the image region including the start point of the detected motion vector; ,
Based on the detected motion vector and the change in the magnitude and frequency component of the motion vector between the current frame and the previous frame, the magnitude of the motion vector increases at least from the previous frame to the current frame. And when the frequency component changes to a low frequency, or when the magnitude of the motion vector decreases and the frequency component changes to a high frequency , a motion blur detection unit that determines that there is motion blur in the detection unit;
An improvement processing unit that corrects motion blur based on a change in magnitude of the motion vector and a change in frequency component when the motion blur detection unit determines that there is motion blur ;
A video processing apparatus comprising: The improvement processing unit corrects the motion blur in the previous frame based on the change in the magnitude of the motion vector and the change in the frequency component, thereby correcting the motion blur for the current frame. The video processing apparatus according to claim 1 . If the motion vector in the previous frame cannot be detected,
Based on the magnitude and frequency component of the motion vector in the current frame, the motion blur detection unit has a magnitude of the motion vector in the current frame and a low-frequency component amount in the current frame within a predetermined range. If the motion vector size is as follows, it is determined that there is motion blur,
The improvement processing unit, a video processing apparatus according to claim 1 or 2, characterized in that for correcting the motion blur based on the magnitude of the motion vector in the current frame. A video processing device that corrects motion blur generated in a captured video,
A motion vector detection unit that divides an image included in a current frame to be corrected into detection units of a predetermined size and detects a motion vector for each detection unit;
A pixel inclination detecting unit which detects a pixel gradient indicating a degree of change of successive pixel values of the image area including the detection unit,
Based on the detected motion vector magnitude and pixel inclination, motion blur detection is performed to determine that there is motion blur in the detection unit when a value obtained by weighting the pixel inclination by the magnitude of the motion vector is within a predetermined threshold range. And
When the motion blur detection unit determines that there is motion blur, an improvement processing unit that corrects motion blur based on the magnitude of the motion vector;
It further comprising a Film image processing apparatus. The video processing apparatus according to claim 4 , wherein the motion blur detection unit changes the predetermined threshold according to a contrast of an image. A control method of a video processing device for correcting motion blur that occurs in a captured video,
Dividing an image included in a current frame to be corrected into detection units of a predetermined size, and detecting a motion vector for each detection unit;
Obtaining the frequency component of the image region, and determining the frequency component of the image region including the detection unit and the frequency component in the previous frame of the image region including the start point of the detected motion vector;
Based on the detected motion vector and the change in the magnitude and frequency component of the motion vector between the current frame and the previous frame, the magnitude of the motion vector increases at least from the previous frame to the current frame. And a motion blur detection step for determining that there is motion blur in the detection unit when the frequency component changes to a low frequency, or when the magnitude of the motion vector decreases and the frequency component changes to a high frequency ,
In the motion blur detection step, when it is determined that there motion blur, based on the change in the change of the magnitude and frequency components of the motion vector, and having a improved process step of correcting the motion blur Control method of video processing apparatus.   In the improvement processing step, the motion blur correction for the current frame is performed by correcting the motion blur correction in the previous frame based on the change in the magnitude of the motion vector and the change in the frequency component. A method for controlling a video processing apparatus according to claim 6.   If the motion vector in the previous frame cannot be detected,
  In the motion blur detection step, the magnitude of the motion vector in the current frame and the amount of the low frequency component in the current frame are within a predetermined range based on the magnitude and frequency component of the motion vector in the current frame. If the motion vector size is as follows, it is determined that there is motion blur,
  8. The method according to claim 6, wherein in the improvement processing step, motion blur is corrected based on a magnitude of a motion vector in a current frame.   A control method of a video processing device for correcting motion blur that occurs in a captured video,
  Dividing an image included in a current frame to be corrected into detection units of a predetermined size, and detecting a motion vector for each detection unit;
  A pixel inclination detection step for detecting a pixel inclination representing a degree of change in a continuous pixel value of an image region including the detection unit;
  Based on the detected motion vector magnitude and pixel inclination, motion blur detection is performed to determine that there is motion blur in the detection unit when a value obtained by weighting the pixel inclination by the magnitude of the motion vector is within a predetermined threshold range. Steps,
  In the motion blur detection step, when it is determined that there is motion blur, an improvement processing step of correcting motion blur based on the magnitude of the motion vector;
A control method for a video processing apparatus, comprising:   The video processing apparatus control method according to claim 9, wherein, in the motion blur detection step, the predetermined threshold is changed according to a contrast of an image.

Images