JP2010288006A - Moving picture data processor, imaging device and moving picture reproduction device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving picture data processor which generates moving pictures of a high image quality, based on moving picture data, the number of frames of which is small. <P>SOLUTION: The moving picture data processor includes: a motion detector which divides moving picture data frames into a plurality of blocks; and detects motion vectors and prediction errors for each of the blocks using a standard frame and a reference frame of the moving picture data; a reliability generator which generates the reliability that indicates the probability of motion vectors to prediction errors of each of the blocks; a reliability determination part which calculates and generates an interpolation processing flag to select an appropriate interpolation processing method based on the reliability for every block, which is generated by the reliability generator; and an interpolation frame generator which selects an appropriate interpolation processing method using the moving picture data, the motion vectors and the interpolation processing flag, and generates interpolation frames between frames of the moving picture data. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a moving image data processing device that generates a high-quality moving image from moving image data with a small number of frames, an imaging device including the moving image data processing device, and a moving image reproduction device.

  CRT TVs and plasma TVs display images while instantaneously blinking the light that represents the image (referred to as impulse display), while LCD televisions continue to hold the same image during the frame display period (referred to as hold display). Therefore, there is a problem that a blurred image (called moving image blur) in which the position of the displayed object is shifted from the viewpoint of the person following the object is visually recognized.

Therefore, in particular, when a moving image with a small number of frames is displayed on a display device such as a liquid crystal television, moving image blur occurs and a high image quality cannot be obtained.
In order to solve this problem, in the liquid crystal television, for example, the moving image blur is improved by increasing the frame rate and shortening the hold display time per frame.

As a conventional method for generating the interpolation frame, there is the following method.
(1) Interpolated frame generation method by indirect estimation:
FIG. 15 is a diagram for explaining a conventional method of generating an interpolation frame. Here, a frame positioned temporally after the interpolation frame is a reference frame, and a frame positioned temporally before is a reference frame.

  First, the base frame is divided into blocks, block matching is performed between each block and the reference frame, and a motion vector mv (solid arrow) is calculated. Such a motion vector estimation method that does not use an interpolation frame is called indirect estimation.

The value of the motion vector mv calculated here represents the displacement on the reference frame of the object of the block (target block) to be processed on the base frame. Therefore, in an interpolated frame that is located between the base frame and the reference frame in time, it can be estimated that the block is at a position displaced by mv / 2.
Therefore, motion compensation (broken arrows) can be performed by generating an interpolation frame by shifting an area corresponding to a block (reference block) on the reference frame by mv / 2.

  When the above processing is applied to all the blocks of the reference frame, an area where the blocks generated on the interpolation frame overlap (overlap area) and an area not covered by the block (uncovered area) are generated. There is. In that case, the pixel values of the overlapping blocks are averaged in the overlap region. In the uncovered area, an interpolation frame is generated by copying the pixel value of the reference frame.

(2) Interpolated frame generation method by direct estimation:
As shown in FIG. 16, there is a conventional method in which the position of the interpolated frame is set as a reference frame, and the frame positioned before in time is set as a reference frame. At this time, the interpolation frame (base frame) is divided into blocks, and the positions of two reference frames corresponding to the target block are calculated as motion vectors mv (solid arrows). A method for estimating a motion vector using an interpolation frame as a reference frame is called direct estimation. Then, motion compensation (broken arrows) can be performed by generating an interpolation frame by moving the area corresponding to the reference block on the reference frame by mv.

  In the moving image interpolation apparatus of Patent Document 1, a motion vector (high frame rate motion vector) detected using a high frame rate moving image (high frame rate video) having a large inter-frame correlation is transmitted, and the transmitted motion vector is transmitted. And an appropriate interpolated frame is interpolated from the frame image. In a high frame rate video, the possibility of approximating the motion of an object between frames by constant linear motion increases, so that the subjective image quality can be improved by this technique.

  Further, in the image signal decoding device of Patent Document 2, an interpolation frame is generated using a motion vector scheduled to be transmitted on the transmission side, and the corresponding original image is compared. Send the pixel value of the original image instead. In this method, since the pixel value of the original image is sent to a region where the assumption of constant velocity linear motion is not established, the failure of the interpolation frame can be avoided.

JP-A-10-271508 Japanese Patent Laid-Open No. 10-56644

However, since the technique of Patent Document 1 assumes the movement of an object by a uniform linear motion, correct interpolation is performed for a complex motion in which the object performs a curved motion, a circular motion, or an inconstant motion. Cannot be performed, and the subjective image quality deteriorates, such as the movement of the object becoming awkward.
Further, in the technique of Patent Document 2, since an original image is transmitted for a complicated movement of an object, there is a problem that a code amount of encoded data becomes large.

  The present invention has been made in consideration of the above situation, and includes a moving image data processing device that generates high-quality moving images from moving image data with a small number of frames, and the moving image data processing device. An object of the present invention is to provide an imaging device and a moving image reproduction device.

  In order to solve the above-described problem, a moving image data processing apparatus according to the present invention divides frame data of moving image data into a plurality of blocks, and each of the blocks from the reference frame data and the reference frame data of the moving image data. A motion detection unit that detects a motion vector and a prediction error with respect to the block, and a reliability generation unit that generates a reliability indicating the reliability of the motion vector with respect to the prediction error of each block, and the reference frame data of the moving image data And the motion vector and reliability are output.

The reliability generation unit generates the reliability by any of the following.
(1) A reliability is generated by a value obtained by quantizing the prediction error.
(2) Reliability using at least a value obtained by quantizing a prediction error in a color component different from the component used for the motion detection obtained from the reference block and the target block associated with the motion vector detected by the motion detection unit Is generated.
(3) The reliability is generated based on the ratio between the activity indicating the complexity of each block and the prediction error.

  The moving image data processing apparatus includes a frame switching unit that outputs moving image data by switching between a reference frame and a reference frame for each frame, and the motion detection unit outputs a reference that is output from the frame switching unit. A motion vector and a prediction error are generated from frame data and reference frame data, and the reliability generation unit is configured to generate the reliability based on the motion vector and the prediction error generated by the motion detection unit. Also good.

  Further, the moving image data processing device includes a reliability determination unit that generates an interpolation processing flag for selecting an appropriate interpolation processing method based on the reliability generated by the reliability generation unit, and the moving image An interpolation frame generation unit that generates an interpolation frame between frames of the moving image data by selecting an appropriate interpolation processing method using the image data, the motion vector, and the interpolation processing flag;

  Here, the interpolation processing flag value includes a first value indicating that the reliability is high, a second value indicating that the reliability is low, and a third value indicating that the reliability is extremely low. The interpolation frame generation unit applies normal frame interpolation processing when the interpolation processing flag indicates the first value, and applies when the interpolation flag indicates the second value. Applies an interpolation process that corrects pixel values in a region where image quality degradation occurs during interpolation, and applies an interpolation process that corrects the value of a motion vector if the interpolation flag indicates the third value.

  Furthermore, the imaging device may include an imaging device that outputs captured moving image data and the above-described moving image data processing device, and the moving image output by the imaging device to the imaging device. An encoder that outputs moving image encoded data obtained by encoding data; and additional information obtained by encoding and multiplexing the motion vector generated by the motion detector and the reliability generated by the reliability generator And an additional information encoding unit.

  In addition, the moving image reproduction device may include the above-described moving image data processing device, and the moving image reproduction device includes a decoder that decodes the input moving image encoded data, and the input additional information. An additional information decoding unit that performs demultiplexing and decoding and outputs a motion vector and reliability, and the reliability determination unit generates the interpolation processing flag from the reliability decoded by the additional information decoding unit Then, the interpolation frame generation unit is based on the moving image data decoded by the decoder, the motion vector decoded by the additional information decoding unit, and the interpolation processing flag generated by the reliability determination unit, An interpolation method may be generated by selecting an appropriate interpolation processing method.

According to the present invention, a high-quality moving image can be generated from moving image data with a small number of frames.
In addition, by mounting the moving image data processing device of the present invention on an imaging device and a moving image playback device, a moving image with reduced moving image blur can be displayed on the liquid crystal display device.

It is a figure for demonstrating the outline of the frame interpolation process in 1st embodiment of this invention. 1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is a block diagram which shows the detailed structure of the motion detection part which concerns on 1st embodiment of this invention. It is a figure for demonstrating each term in the frame interpolation process of this invention. It is a flowchart for demonstrating operation | movement of the motion detection part which concerns on 1st embodiment of this invention. It is a flowchart for demonstrating operation | movement of the reliability production | generation part which concerns on 1st embodiment of this invention. It is a block diagram which shows the structure of the moving image reproducing device which concerns on 1st embodiment of this invention. It is a flowchart for demonstrating operation | movement of the reliability determination part which concerns on 1st embodiment of this invention. It is a block diagram which shows the structure of the imaging device which concerns on 2nd embodiment of this invention. It is a block diagram which shows the structure of the additional information encoding part which concerns on 2nd embodiment of this invention. It is a block diagram which shows the structure of the moving image reproduction apparatus in 2nd embodiment of this invention. It is a block diagram which shows the structure of the additional information decoding part which concerns on 2nd embodiment of this invention. It is a block diagram which shows the structure of the imaging device which concerns on 3rd embodiment of this invention. It is a figure for demonstrating the outline of the frame interpolation process in 3rd embodiment of this invention. It is a figure for demonstrating the outline | summary of the frame interpolation process (indirect estimation) in a prior art. It is a figure for demonstrating the outline | summary of the frame interpolation process (direct estimation) in a prior art.

  Hereinafter, embodiments of the moving image data processing device, the imaging device, and the moving image reproducing device of the present invention will be described with reference to the drawings.

<First embodiment>
An outline of the frame interpolation processing in this embodiment will be described.
FIG. 1 is a diagram for explaining the outline of motion vector detection and frame interpolation processing, and shows each frame in the time direction.
As shown in FIG. 1A, the imaging apparatus sequentially processes captured moving image data as input frames, detects a motion vector corresponding to each input frame, and generates moving image data, a motion vector, and reliability. ,Output. The motion vector and reliability represent a set of motion vectors and reliability between frames of each block obtained by dividing the frame, and are associated with the blocks. Note that the motion vector is detected by the indirect estimation described above, but can be similarly applied even if the direct estimation is used.

  As shown in FIG. 1B, the moving image reproducing apparatus uses the input moving image data as an input frame, and is positioned in the middle of the two input frames in terms of time based on the input frame, the motion vector, and the reliability. An interpolation frame to be generated is generated.

(Configuration of imaging device)
First, the configuration of the imaging apparatus according to the present embodiment will be described.
FIG. 2 is a block diagram illustrating a configuration of the imaging apparatus according to the present embodiment. In FIG. 2, the imaging apparatus 30 includes an imaging element 301, a frame memory 302, a motion detection unit 303, and a reliability generation unit 304. ing. Note that the encoding process is performed on the moving image data, the motion vector, and the reliability output from the imaging apparatus 30. This encoding process will be described in the second embodiment.

  The image pickup element 301 is an element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) that converts input optical information into an electric signal to obtain a moving image. The color component of a moving image (input moving image) obtained from the image sensor 301 is generally composed of a plurality of color components such as RGB and YCbCr. In the following description, it is assumed that the color component is a YCbCr component. To do. Further, the image sensor 301 may perform demosaicing and color conversion processing for converting the photoelectric signals spatially thinned for each color component such as a Bayer array into necessary color components and resolution.

  The frame memory 302 can store a plurality of frames, and stores at least the number of frames (2 frames) necessary for estimating a motion vector. The frame stored in the frame memory 302 is used as a reference frame and a reference frame by the motion detection unit 303, and further used as a frame for output as moving image data.

  As will be described in detail later, the motion detection unit 303 divides the reference frame into blocks, and detects and outputs motion vectors in units of blocks. The color component used in the motion detection unit 303 is a luminance component, and a motion vector is detected by indirect estimation. Further, a residual at the time of block matching in the motion vector detected at the same time is obtained and output as a prediction error.

  As will be described in detail later, the reliability generation unit 304 inputs a reference frame, a motion vector, and a prediction error, and generates a reliability. The reliability is output as a value corresponding to each block on the reference frame. The reliability is a value obtained by quantifying the probability of the motion vector, and can be calculated, for example, by any of the following.

(Reliability A)
The reliability A is a quantized value of the residual of the luminance component used in motion detection. This luminance component residual is a prediction error input to the reliability generation unit 304.
The residual becomes small if the detected motion is reliable, and conversely increases if the detected motion is not accurate, and can be used as a reliability. Also, by performing quantization, it is possible to reduce the code amount of reliability.

(Reliability B)
The reliability B is the difference between the reference block extracted from the detected motion vector and the target block with respect to the color component different from the luminance component used for the motion search in the motion detection unit 303, in this embodiment, the color difference component. It is a quantized value.
When the interpolation frame is generated, if the residual of the color difference component is large in a certain block, there is a concern that the image quality is deteriorated due to the shift of the pixel position of each color component (color shift).
Therefore, if the residual of the color difference component is large, the reliability is low, and if the residual is small, the reliability is high.

Although there are two color difference components, Cb and Cr, only one of the color components can be used, or a combination of two components (for example, an average value) may be used.
In addition, it is preferable to use both the luminance component and the color component because it is more robust to noise. For example, the reliability obtained by adding the residual of the luminance component and the color difference component and quantizing may be used.
In general, the residual of the color difference component is smaller than the residual of the luminance component. Therefore, in order to increase the contribution of the color difference component, a weight (for example, 4 times) may be added. The reliability may be obtained by quantizing the larger value of the luminance component residual and the color difference component residual. In this case as well, it is appropriate to weight the residual color difference component.

(Reliability C)
The reliability C is a ratio between the activity of the target block and the luminance component residual (prediction error). That is, the ratio of the target blocks is expressed by the following (Formula 1).

  (Ratio) = (activity) / (luminance component residual (prediction error)) (Equation 1)

  Here, the activity of the target block is the variance (standard deviation) of the pixel values in the block, but can also be obtained from the absolute difference sum of adjacent pixels in the block, the energy of the AC coefficient when frequency-converted, and the like.

The reason why the above ratio can be reliable is as follows.
In a block including a flat region, activity (dispersion value of pixel values) is low, and there is no feature point in the block.
On the other hand, in a block in a non-flat region, activity (dispersion value of pixel values) is high, and feature points exist in the block, so that it is easy to detect an accurate motion vector.

  Therefore, the smaller the residual of the luminance component and the higher the activity, that is, the higher the reliability of the block with the higher activity relative to the residual of the luminance component, while the higher the residual of the luminance component, the higher the activity. The smaller the block, that is, the lower the activity in the luminance component residual, the lower the reliability, so the relationship (ratio) between the luminance component residual and the activity can be treated as the reliability. It becomes.

  These reliability generation methods are not limited to the above methods, and other methods may be used. Further, the types of reliability are not limited to the above three types, and any number may be used.

(Operation of Imaging Device 30)
Next, the operation of the imaging device 30 according to the present embodiment will be described.
First, an input moving image captured by the image sensor 301 is input to the motion detection unit 303 and the reliability generation unit 304 and is stored in the frame memory 302.

Next, the motion detection unit 303 reads frames corresponding to the base frame and the reference frame from the frame memory 302 and detects a motion vector using them. For example, the data is output to an encoder for encoding a motion vector for output to the transmission path and also output to the reliability generation unit 304.
In addition, when the motion vector is derived, the calculated prediction error is output to the reliability generation unit 304.

  Next, the reliability generation unit 304 generates a reliability based on the motion vector and the prediction error output from the motion detection unit 303 and, for example, to an encoder for encoding the reliability for output to the transmission path. Output.

The unit of input and output of the reliability generation unit 304 may be a block, a plurality of blocks, or a frame. In the case of a block unit, the motion detection unit 303 and the reliability generation unit 304 can use the same base frame pixel value and reference frame pixel value, thereby reducing the data transfer amount between the frame memory 302 and each unit. can do.
In the case of a plurality of block units or frame units, a motion vector storage unit and a prediction error storage unit (not shown) are provided in the imaging device 30, and the motion vector and prediction error output from the motion detection unit 303 are stored only in necessary units. deep.

  As described above, by using the imaging device 30, it is possible to generate and output a motion vector and reliability from an input moving image captured by the imaging element 301.

(Configuration of Motion Detection Unit 303)
Next, a detailed configuration of the motion detection unit 303 will be described.
FIG. 3 is a block diagram showing the configuration of the motion detection unit 303. In FIG. 3, the motion detection unit 303 includes a target block extraction unit 3031, a reference region extraction unit 3032, a residual calculation unit 3033, and a motion vector determination unit 3034. It is composed of

  The target block extraction unit 3031 sequentially outputs the target block obtained by dividing the input reference frame into blocks having a predetermined shape to the residual calculation unit 3033, and refers to the upper left pixel position (referred to as target block position information) of the target block. The data are sequentially output to the area extraction unit 3032 (see FIG. 4).

  The block size is preferably 16 × 16 pixels, which is a macroblock size in general moving picture coding. However, 8 × 8 or 4 × 4 is preferable for a video having a small moving area with respect to the image size. Further, the block size and the block shape do not have to be uniform, and different sizes and shapes may be used according to the shape of the input moving image. In the following description, it is assumed that the target block is composed only of the luminance component (Y) of the input moving image.

The reference area extraction unit 3032 reads a reference area on the reference frame corresponding to the input target block position information from the frame memory 302 and outputs the reference area to the residual calculation unit 3033.
This reference area is a reference required when searching for the input target block in the search range (search range) of the motion vector used by the motion vector determination unit 3034 with the position corresponding to the target block position information as the center. An area within a frame. A predetermined fixed value is used as the size of the search range (search range), but it may be switched adaptively for each frame or block.

  As shown in FIG. 4, when the size of the search range (search range) is 8 pixels, the search range (search range) is a region of ± 8 pixels on the reference frame centered on the target block position information, The reference area is an area surrounded by a broken line.

  The residual calculation unit 3033 calculates a residual based on the input target block and the reference region, generates a residual map that is a set of residuals, and outputs the residual map to the motion vector determination unit 3034. The residual is calculated using a sum of absolute differences SAD (Sum of Absolute Differences), such as a sum of squared differences (SSD), a sum of absolute differences after conversion SATD (Sum of Absolute Transformed Differences), and the like. Other calculation formulas may be used. The number of elements in the residual map is the size of the search range. The residual between each reference block (size and shape is the same as the target block) in the search range and the target block is recorded corresponding to the position on the search range.

The motion vector determination unit 3034 determines and outputs a motion vector from the position of the minimum residual in the input residual map and the target block position information.
Further, the minimum value in the residual map is output as a prediction error.
The motion vector detection unit 303 may calculate and output a motion vector and a prediction error in units of blocks, or associates the motion vector and the prediction error generated for each target block with the target block, and a motion vector (not shown) The data may be sequentially stored in the storage unit and the prediction error storage unit, and output in a predetermined unit such as a block unit or a frame unit.

(Operation of the motion detection unit 303)
Next, an operation in which the motion detection unit 303 derives a prediction error and a motion vector based on the input reference frame and the reference frame recorded on the frame memory will be described with reference to the flowchart of FIG.

(Step S1) The input reference frame is divided into blocks with a predetermined block size, and the process proceeds to Step S2.
(Step S2) Extract as target blocks sequentially from the upper left block on the reference frame, and simultaneously extract the coordinate position of the target block as target block position information, and proceed to Step S3. The subsequent steps S3 to S12 are executed for each target block in the reference frame.

(Step S3) Based on the target block position information and search range parameters (for example, the size of the search range), the search range is determined, the reference area on the reference frame is extracted, and the process proceeds to Step S4. The following steps S4 to S6 are executed for each search point in the search range.

(Step S4) A reference block on the reference frame corresponding to each search point in the search range is extracted, and the process proceeds to Step S5.
(Step S5) The residual between the target block and the reference block is calculated, stored in the residual map in association with the search point, and the process proceeds to Step S6.
(Step S6) If it is the last search point, the process proceeds to step S7, the process for the next target block is performed, and if not, the process returns to step S4.

(Step S7) The position of the minimum value of the residual map is extracted as a motion vector, the motion vector information is stored in a motion vector storage unit (not shown), and the process proceeds to Step S8.
(Step S8) The minimum value of the residual map is extracted as a prediction error, the prediction error is stored in a prediction error storage unit (not shown), and the process proceeds to step S9.
(Step S9) If the current block is the last block in a predetermined unit, the process proceeds to step S10. If not, the process returns to step S2, and the process for the next target block is performed.

(Step S10) The stored motion vector is output, and the process proceeds to Step S11. Note that the output unit may be a block, a plurality of blocks, or a frame.
(Step S11) The stored prediction error is output, and the process proceeds to Step S12. The output unit is the same unit as in step S10.
(Step S12) If the reference frame is the last frame, the process ends. If not, the process returns to step S1, and the process for the next reference frame is performed.

(Operation of the reliability generation unit 304)
Next, the operation of the reliability generation unit 304 for deriving the reliability based on the input prediction error and motion vector will be described with reference to the flowchart of FIG.

(Step S21) The type of reliability to be generated is selected, and the process proceeds to Step S22.
Here, the type of reliability may be determined by the processing capability of the imaging device 30 or the type of image, or may be stored in advance in the imaging device 30 in consideration of these.
For example, since the process of generating the reliability A is only quantization, the amount of processing is small, so that it can be used for an imaging apparatus with low processing capability. On the other hand, since the reliability B and the reliability C require a certain amount of processing, they can be used for an imaging apparatus with a medium processing capacity or higher. In addition, in motion estimation using only the luminance component, the reliability B is effective for an image in which a color component error is likely to occur. Further, the reliability C is effective for an image in which a flat region and a non-flat region are mixed.

(Step S22) If the reliability type is reliability A, the process proceeds to step S23. If the reliability type is reliability B, the process proceeds to step S24. If the reliability type is reliability C, Proceed to step S27.

(Step S23) The input prediction error is quantized to generate a reliability, and the process proceeds to step S30.

(Step S24) A reference block on the reference frame corresponding to the input motion vector is extracted, and the process proceeds to step S25.
(Step S25) The color difference components of the target block and the reference block are extracted, the residual between both blocks is calculated, and the process proceeds to step S26. When there are a plurality of color difference components, a plurality of color difference components are extracted and averaged.
(Step S26) Quantize the residual calculated in step S25, and proceed to step S30.

(Step S27) The luminance component of the target block is extracted, and the process proceeds to Step S28.
(Step S28) The activity is calculated from the distribution of the pixel values of the target block, and the process proceeds to Step S29.
(Step S29) Using the residual (prediction error) corresponding to the target block, the ratio of the residual to the activity is calculated using (Equation 1), and the process proceeds to Step S30.

(Step S30) The calculated reliability is stored in a reliability storage unit (not shown) in association with the target block, and the process proceeds to Step S31.
(Step S31) If the current target block is the last block, the process proceeds to step S32. If not, the process returns to step S22 to perform the process for the next target block.
(Step S32) The stored reliability is output, and the process proceeds to Step S33. The unit of output may be any of a block, a plurality of blocks, and a frame, but is the same as the unit output in step S10 of FIG.
(Step S33) If the reference frame is the last frame, the process ends. If not, the process returns to step S21 to perform the process for the next reference frame.

  The above quantization in (reliability A) and (reliability B) is uniform quantization in which the quantization step width is a fixed value, but non-linear quantization may be used. In addition, the reliability is preferably quantized so as to have a value of about 4 (2 bits) per block because of the amount of code. In this case, the step width for dividing the dynamic range of the residual into four equal parts is set. Calculate and quantize.

  When the reliability type is determined as any one of (Reliability A) to (Reliability C), the above processing may be performed only for the determined reliability.

(Configuration of video playback device)
Next, the configuration of the moving image playback apparatus according to this embodiment will be described.
FIG. 7 is a block diagram showing the configuration of the moving image playback apparatus according to the present embodiment. In FIG. 7, the moving image playback apparatus 60 includes a data synchronization unit 601, a reliability determination unit 602, and an interpolation frame generation unit 603. Composed. Note that the moving image data, motion vectors, and reliability output from the imaging device 30 are encoded, but these decoding processes will be described in the second embodiment.

  The data synchronization unit 601 synchronizes the input moving image data, the corresponding motion vector, and the reliability, and outputs the moving image data and the corresponding motion vector to the interpolation frame generation unit 603. The reliability is output to the reliability determination unit 602.

The reliability determination unit 602 determines the magnitude of the input reliability, and sets an interpolation processing flag for selecting an appropriate interpolation processing method according to the reliability from a plurality of predetermined interpolation processing method candidates. And output to the interpolation frame generation unit 603.
In this embodiment, the value of the interpolation processing flag is set to 3 when the reliability is high, 2 or 1 when the reliability is slightly low, and 0 when extremely low, but is not limited to this. Absent.

  The interpolation processing flag may be switched for each block or for each frame. When the interpolation processing flag is different for each block, the interpolation processing flag is generated for each block. If the interpolation processing method differs for each frame, an interpolation processing flag is generated for each frame.

Next, the operation of the reliability determination unit 602 will be described using the flowchart of FIG.
The threshold values shown below have a relationship of TH1 <TH2 <TH3, and these threshold values are set in advance through experiments or the like.

(Step S41) The target block is sequentially selected from the upper left block on the reference frame, the reliability corresponding to the target block is extracted, and the reliability is compared with the threshold value TH1. If not, the process proceeds to step S43.
(Step S42) It is determined that the reliability is extremely low, the interpolation processing flag 0 is assigned, and the process proceeds to Step S48.

(Step S43) The reliability of the target block is compared with the threshold value TH2, and if it is equal to or less than the threshold value TH2, the process proceeds to step S44, and if not, the process proceeds to step S45.
(Step S44) It is determined that the reliability is slightly low, the interpolation processing flag 1 is assigned, and the process proceeds to Step S48.

(Step S45) The reliability of the target block is compared with the threshold value TH3, and if it is equal to or less than the threshold value TH3, the process proceeds to step S46, and if not, the process proceeds to step S47.
(Step S46) It is determined that the reliability is normal, the interpolation processing flag 2 is assigned, and the process proceeds to Step S48.

(Step S47) It is determined that the reliability of the target block is high, the interpolation processing flag 3 is assigned, and the process proceeds to Step S48.

(Step S48) The assigned interpolation processing flag is stored in an interpolation processing flag storage unit (not shown) in association with the target block, and the process proceeds to Step S49.
(Step S49) If the target block is the last block, the process returns to step S41, and if not, the process ends.

The interpolation frame generation unit 603 generates and outputs an interpolation frame from the input moving image data, the motion vector, and the interpolation processing flag.
The method for generating the interpolation frame is changed according to the value of the interpolation processing flag input from the reliability determination unit 602. For example, an interpolation frame is generated by any of the following methods.

(1) When the interpolation processing flag is 3:
When the reliability of the block corresponding to the interpolation processing flag is high, a reference block corresponding to the motion vector of the target block is extracted from the reference frame and subjected to interpolation processing. This interpolation process is a normal process using a motion vector.

(2) When the interpolation processing flag is 1 or 2:
When the reliability of the block corresponding to this interpolation processing flag is slightly low, in the interpolation processing of (1) above, the reference block is expanded and pasted by several pixels vertically and horizontally, and the same interpolation as in (1) above is performed. To process. The size of expansion is preferably about 1/4 of the size of the target block.

  For example, when the size of the target block is 8 × 8 pixels, the upper / lower / left / right two pixels are expanded and interpolation is performed with 12 × 12 pixels. With this interpolation method, the uncovered area in which the interpolation frame is not covered by the block is reduced, so that deterioration of the interpolation frame can be suppressed. In addition, an average value of overlapping pixel values is used for an overlapping region where blocks overlap on an interpolation frame.

(3) When the interpolation processing flag is 0:
When the reliability of the block corresponding to this interpolation processing flag is extremely low, the motion vector of the target block is corrected using the motion vector of the neighboring peripheral block and interpolated. For example, the median value, average value, or weighted average of motion vectors for a total of 9 blocks (or a total of 25 blocks of 5 × 5) of 3 × 3 around the target block is taken as the motion vector of the target block.
This interpolation method is effective in removing motion vectors that are significantly different from the surrounding motion vectors.

  These interpolation methods are not limited to the above methods, and other methods may be used depending on the type of moving image to be handled.

(Operation of Moving Image Reproducing Device 60)
Next, the operation of the moving image playback device 60 will be described.
The data synchronization unit 601 stores the input moving image data, motion vector, and reliability in a buffer (not shown), outputs the reliability to the reliability determination unit 602 at a predetermined timing, and the interpolation processing flag is interpolated. The reference frame and the corresponding motion vector are output to the interpolation frame generation unit 603 at the timing output to the frame generation unit 603.

The interpolation frame generation unit 603 generates an interpolation frame from the moving image data, the motion vector, and the interpolation processing flag according to various interpolation processing flags, and outputs it as an output moving image.
If the reliability is slightly low, a relatively accurate motion vector may be detected. However, in the interpolated frame, the deterioration of the motion vector accuracy is likely to cause deterioration such as block distortion. Interpolation processing having the effect of a deblocking filter like the above processing is desirable.
On the other hand, when the reliability is extremely low, a motion vector that is completely different from the actual motion may be detected, and deterioration is likely to occur in the interpolation frame. Therefore, as in the process (3) above, the motion A method of correcting the motion vector itself on the assumption that the spatial correlation of the vector is large is desirable.

As described above, by configuring the embodiment, by performing appropriate interpolation processing according to the reliability, which is a value obtained by quantifying the probability of the motion vector, interpolation is also performed for a moving image with complicated motion. Frame failure can be suppressed, and the quality of the reproduced moving image is improved.
Furthermore, even when a moving image with complicated motion is encoded, a moving image is not included instead of a motion vector, so that the code amount can be suppressed.

<Second Embodiment>
In the second embodiment of the present invention, a data encoding portion is added to the imaging device 30 of the first embodiment, and a data decoding portion is added to the moving image reproduction device 60.

(Configuration of imaging device)
FIG. 9 is a block diagram illustrating a configuration of the imaging apparatus according to the present embodiment. The imaging apparatus 30 includes an imaging element 301, a frame memory 302, a motion detection unit 303, a reliability generation unit 304, an encoder 40, and an additional information code. It is comprised from the conversion part 50. FIG.
Among these, the image sensor 301, the frame memory 302, the motion detection unit 303, and the reliability generation unit 304 are the same as those in the first embodiment, and a description thereof will be omitted.
Hereinafter, although the encoder 40 and the additional information encoding part 50 are demonstrated, you may provide these as an additional function instead of providing in the imaging device 30. FIG.

  The encoder 40 is configured by a known technique, encodes moving image data output from the image sensor 301, and outputs moving image encoded data. It should be noted that an encoder that realizes an improvement in encoding efficiency and a reduction in processing amount by using the motion vector output from the motion detection unit 303 and the reliability output from the reliability generation unit 304 at the time of encoding. Can be used.

In the following, examples of using motion vectors and reliability as encoders will be given.
(1) The processing amount of motion search inside the encoder can be greatly reduced. That is, when the reliability is low, there is a possibility of causing an erroneous estimation of the motion vector. Therefore, the erroneous estimation can be suppressed by narrowing the search range.
Further, the search range can be changed depending on whether the reliability of the target block is relatively large or small relative to the reliability of the surrounding blocks. Furthermore, when a plurality of prediction vector candidates are set and a search is performed using each prediction vector as a search center point, when a motion vector that gives an index equal to or lower than a predetermined threshold is obtained, the threshold is set according to the reliability. By changing the search and aborting the search, unnecessary searches can be suppressed.

(2) In an encoder capable of motion compensation by switching the block size, an appropriate block size can be selected with a small amount of processing. That is, by selecting a large block for a block with high reliability and a small block for a block with low reliability, high encoding efficiency can be achieved with a small amount of processing.

(3) Switching between inter-screen prediction and intra-screen prediction can be performed with a small amount of processing. In other words, in the region with high reliability, the accuracy of inter-screen prediction is high and the possibility of selecting intra-screen prediction is low. Therefore, intra-screen prediction is performed only for the portion with low reliability, and the result of inter-screen prediction is compared. Can be switched with a small amount of processing.

  The additional information encoding unit 50 receives the motion vector output from the motion detection unit 303 and the reliability output from the reliability generation unit 304, encodes and multiplexes each of the motion vector and the reliability, Generate and output additional information.

Next, the additional information encoding unit 50 will be described in detail.
FIG. 10 is a block diagram showing a detailed configuration of the additional information encoding unit 50. In the figure, the additional information encoding unit 50 includes a motion vector encoding unit 501, a reliability encoding unit 502, a multiplexing unit. 503.

The motion vector encoding unit 501 encodes the input motion vector and outputs the motion vector encoded data to the multiplexing unit 503.
As a motion vector encoding method, a method generally used for moving image encoding such as variable length encoding is applied to a difference value between an input motion vector and a prediction vector thereof. .

  For example, variable length coding is applied to the difference value between the motion vector of the target block and the prediction vector, and motion vector coded data is output. The prediction vector is preferably, for example, the median value of neighboring blocks (left, upper, upper right blocks) of the target block using the spatial correlation of motion vectors.

  The reliability encoding unit 502 encodes the input reliability and outputs reliability encoded data to the multiplexing unit 503. As an encoding method, there is a method such as variable length encoding for the difference value between the input reliability and the prediction reliability. As the predicted value, for example, the median value of the neighboring blocks of the target block is preferable.

  The multiplexing unit 503 receives the motion vector encoded data output from the motion vector encoding unit 501 and the reliability encoded data output from the reliability encoding unit 502, multiplexes them, and outputs them as additional information. . As a multiplexing method, motion vector encoded data and reliability encoded data may be interleaved and output in units of blocks or a plurality of blocks, or one frame of motion vector encoded data and reliability encoding may be output. Data may be output in order.

  As described above, by using the imaging device 30 of the present embodiment, moving image encoded data and additional information can be generated from an input moving image and output to a transmission path, a recording medium, or the like.

(Configuration of video playback device)
FIG. 11 is a block diagram showing the configuration of the moving image playback apparatus according to the present embodiment. In FIG. 11, the moving image playback apparatus 60 includes a decoder 70, an additional information decoding unit 80, a data synchronization unit 601, a reliability determination. A unit 602 and an interpolation frame generation unit 603 are included.
Among these, the data synchronization unit 601, the reliability determination unit 602, and the interpolated frame generation unit 603 are the same as those in the first embodiment, and a description thereof will be omitted.
Hereinafter, the decoder 70 and the additional information decoding unit 80 will be described, but these may be provided as an additional function instead of being provided in the moving image reproduction device 60.

  The decoder 70 decodes the moving image encoded data received from the transmission path by a known technique, and outputs the moving image decoded data to the data synchronization unit 601.

The additional information decoding unit 80 separates and decodes the additional information received from the transmission path, and outputs the motion vector and the reliability to the data synchronization unit 601.
As shown in FIG. 12, the additional information decoding unit 80 includes a demultiplexing unit 801, a motion vector decoding unit 802, and a reliability decoding unit 803.

  The demultiplexing unit 801 demultiplexes the additional information received from the transmission path, separates it into encoded motion vector data and reliability encoded data, and outputs the motion vector encoded data to the motion vector decoding unit 802. The reliability encoded data is output to the reliability decoding unit 80.

  The motion vector decoding unit 802 receives encoded motion vector data as input, performs a decoding process, and outputs a motion vector. As a decoding method of motion vector encoded data, decoding is performed by a known decoding method corresponding to the encoding method used in the motion vector encoding unit 501.

  The reliability decoding unit 803 receives the reliability encoded data, performs a decoding process, and outputs the reliability. As a decoding method of the reliability encoded data, decoding is performed by a known decoding method corresponding to the encoding method used in the reliability encoding unit 502.

  As described above, by using the moving image reproduction device 60 of the present embodiment, a moving image can be reproduced from the encoded moving image data input from the transmission path and the additional information.

<Third embodiment>
In the third embodiment of the present invention, a frame switching unit is added to the imaging device 30 of the second embodiment, and the moving image playback device 60 is the same as that of the second embodiment.
Only the differences will be described below.

(Structure of the imaging device)
FIG. 13 is a block diagram illustrating a configuration of the imaging apparatus according to the present embodiment. In FIG. 13, the imaging apparatus 30 includes an imaging element 301, a frame switching unit 90, a frame memory 302, a motion detection unit 303, and reliability generation. Part 304, encoder 40, and additional information encoding part 50.
Among these, the image sensor 301, the frame memory 302, the motion detection unit 303, the reliability generation unit 304, the encoder 40, and the additional information encoding unit 50 are the same as those in the second embodiment, and the description thereof is omitted. The encoder 40 and the additional information encoding unit 50 may be provided as an additional function instead of being provided in the imaging device 30.
Hereinafter, the added frame switching unit 90 and the points changed thereby will be described.

The frame switching unit 90 classifies the inputted input moving image into a standard frame and a reference frame and outputs them. The frame rate of the input moving image and the frame rate of the standard frame are switched to classify the frame into the standard frame and the reference frame. The ratio between the frame rate of the input moving image and the frame rate of the reference frame may be any of 1/2, 1/3, 1/4, and the like.
In the present embodiment, it is assumed that the ratio is ½, and input video images that are sequentially input are switched between the base frame and the reference frame and output every other frame.

  The frame memory 302 stores the reference frame output from the frame switching unit 90. The accumulated reference frames are used by the motion detection unit 303, output to the encoder 40 at an appropriate timing, and encoded as moving image data.

(Operation of Imaging Device 30)
Next, the operation of the imaging device 30 according to the present embodiment will be described.
For simplicity of explanation, the frame rate of the input moving image is 60 fps, and the frame rate of the base frame and the reference frame is 30 fps.

  First, a 60 fps input moving image shot by the image sensor 301 is classified into a 30 fps reference frame and a reference frame in the frame switching unit 90, the 30 fps reference frame is stored in the frame memory 302, and the 30 fps reference frame is moved. The data is input to the detection unit 303 and the reliability generation unit 304.

Next, the motion detection unit 303 calls a reference frame corresponding to the base frame from the frame memory 302 and derives a motion vector using them.
The reliability generation unit 304 generates a reliability from the motion vector output by the motion detection unit 303 and the prediction error and outputs the reliability.

The encoder 40 encodes the reference frame using the motion vector and the reliability and outputs moving image encoded data. The additional information encoding unit 50 outputs additional information obtained by encoding and multiplexing the motion vector and the reliability.
These moving image encoded data and additional information are output to the transmission line, and on the display side, the moving image can be reproduced from the moving image encoded data and the additional information by using the moving image reproducing device 60 of the second embodiment. .

Next, generation of an interpolation frame in the moving image reproduction device 60 according to the present embodiment will be described.
FIG. 14 is a diagram for explaining an outline of processing from moving image input to frame interpolation in the present embodiment, and the horizontal axis indicates the time direction. 14A corresponds to the process of detecting a motion vector in the imaging apparatus 30, and FIG. 14B corresponds to the frame interpolation process in the moving image reproduction apparatus 60.

  The imaging apparatus 30 receives a captured 60 fps moving image. The input moving image is classified into a base frame and a reference frame every other frame by the frame switching unit 90, the reference frame is encoded and output as moving image encoded data, and between the base frame and the reference frame. A motion vector corresponding to the motion is generated and output as additional information together with the reliability.

  The moving image reproduction device 60 reproduces a 60 fps moving image by using each frame of the moving image decoded data obtained by decoding from the encoded moving image data and generating an interpolation frame using the motion vector and the reliability. To do.

As described above, by using the imaging apparatus of the present embodiment, it is possible to output moving image encoded data and additional information from an input moving image. Therefore, when a moving image having a high frame rate is input, more accurate movement is performed. It is possible to detect vectors.
Furthermore, by using the moving image reproducing apparatus of the present embodiment, an appropriate interpolation process can be performed on each block based on the reliability determined by the imaging apparatus, so that the image quality is good.

<Other embodiments>
In addition, the present invention is not limited to the above-described embodiment, and various modifications and corrections can be made without departing from the scope of the present invention.
For example, the moving image data processing apparatus may be configured by the frame memory 302, the motion detection unit 303, and the reliability generation unit 304 described above. Further, the encoding device may be composed of the above moving image data processing device, the encoder 40, and the additional information encoding unit 50.

  In this case, the imaging apparatus of the first embodiment can be configured by combining the above-described moving image data processing apparatus with the imaging element 301. In addition, the imaging apparatus of the second embodiment can be configured by combining the above-described encoding apparatus with the imaging element 301.

  Further, the frame rate conversion apparatus may be configured by a data synchronization unit 601, a reliability determination unit 602, and an interpolation frame generation unit 603. Further, the decoding device may be composed of the decoder 70, the additional information decoding unit 80, and the above frame rate conversion device.

  In this case, the moving image reproducing apparatus according to the first embodiment can be configured by combining the above-described frame rate conversion apparatus with a device that reads moving images, motion vectors, and reliability and a display device that displays the reproduced moving images. In addition, the moving image reproduction apparatus according to the second embodiment is a combination of an apparatus that reads an encoded moving image and encoded additional information, the above-described encoding apparatus, and a display device that displays the reproduced moving image. Can be configured.

  In the imaging devices of the first and second embodiments, the same moving image and motion vector as in the conventional case are output without generating reliability. Then, in the moving image reproduction apparatuses of the first and second embodiments, the interpolation processing flag may be generated after the reliability is generated before the interpolation frame is generated. Alternatively, the interpolation processing flag may be generated directly from the calculated reliability.

  Further, the moving image data processing device is composed of the frame switching unit 90, the frame memory 302, the motion detection unit 303, and the reliability generation unit 304, and the encoding device is added to the moving image data processing device with the encoder 40 and the additional information code. It is good also as a structure which combined the conversion part 50. FIG.

In this case, the imaging apparatus according to the third embodiment can be configured by combining the moving image data processing apparatus with the imaging element 301 or combining the encoding apparatus with the imaging element 301.
In addition, the moving image playback device of the third embodiment can be configured in the same manner as the moving image playback device of the first or second embodiment described above.

DESCRIPTION OF SYMBOLS 30 ... Imaging device 301 ... Image pick-up element 302 ... Frame memory 303 ... Motion detection part 304 ... Reliability generation part 3031 ... Target block extraction part 3032 ... Reference area extraction part 3033 ... Residual calculation part 3034 ... motion vector determination unit, 3034 ... motion vector determination unit, 40 ... encoder, 50 ... additional information encoding unit, 501 ... motion vector encoding unit, 502 ... reliability encoding unit, 503 ... multiplexing unit, 60 ... moving image Image reproducing device, 601... Data synchronization unit, 602... Reliability determination unit, 603... Interpolation frame generation unit, 70... Decoder, 80 .. additional information decoding unit, 801 .. demultiplexing unit, 802. ... reliability decoding unit, 90 ... frame switching unit.

Claims (9)

Dividing the frame data of the moving image data into a plurality of blocks, and detecting a motion vector and a prediction error for each block from the base frame data and the reference frame data of the moving image data; A reliability generation unit that generates a reliability indicating the certainty of the motion vector with respect to the prediction error, and
A moving image data processing apparatus that outputs reference frame data of the moving image data, and the motion vector and reliability.   The moving image data processing apparatus according to claim 1, wherein the reliability generation unit generates a reliability based on a value obtained by quantizing the prediction error.   The reliability generation unit obtains at least a value obtained by quantizing a prediction error in a color component different from the component used for the motion detection obtained from the reference block and the target block associated with the motion vector detected by the motion detection unit. The moving image data processing apparatus according to claim 1, wherein the reliability is generated by using the moving image data processing apparatus.   The moving image data processing apparatus according to claim 1, wherein the reliability generation unit generates the reliability based on a ratio between an activity indicating complexity of each block and the prediction error.   A frame switching unit configured to switch and output moving image data for each frame between a reference frame and a reference frame, and the motion detection unit predicts a motion vector based on the reference frame data and the reference frame data output from the frame switching unit. 5. The error generation unit according to claim 1, wherein the reliability generation unit generates the reliability based on a motion vector generated by the motion detection unit and a prediction error. Moving image data processing device. A reliability determination unit for calculating and generating an interpolation processing flag for selecting an appropriate interpolation processing method based on a reliability value for each block generated by the reliability generation unit;
An interpolation frame generation unit configured to select an appropriate interpolation processing method using the moving image data, the motion vector, and the interpolation processing flag to generate an interpolation frame between frames of the moving image data. A moving image data processing apparatus.   The interpolation frame generation unit applies a normal frame interpolation process when the interpolation processing flag for each block indicates the first value, and performs interpolation when the interpolation flag indicates the second value. An interpolation process for correcting pixel values in a region where image quality degradation occurs is applied, and when the interpolation flag indicates the third value, an interpolation process for correcting a motion vector value is applied. The moving image data processing apparatus according to claim 6.   6. An image sensor that outputs captured moving image data, an encoder that outputs moving image encoded data obtained by encoding the moving image data output by the image sensor, and the moving image data according to claim 1. An imaging apparatus comprising: a processing device; and an additional information encoding unit that outputs additional information obtained by encoding and multiplexing the motion vector and the reliability generated by the moving image data processing device.   8. A moving image data processing device according to claim 6 or 7, wherein the input moving image encoded data is decoded, and the input additional information is demultiplexed and decoded to obtain a motion vector and reliability. An output additional information decoding unit, wherein the reliability determination unit generates the interpolation processing flag from the reliability decoded by the additional information decoding unit, and the interpolation frame generation unit decodes by the decoder Based on the obtained moving image data, the motion vector decoded by the additional information decoding unit, and the interpolation processing flag generated by the reliability determination unit, an appropriate interpolation processing method is selected to generate an interpolation frame A moving image reproducing apparatus characterized by the above.

Images