JP2006033142A - Moving picture coder, moving picture coding method, program, recording medium, image processing apparatus, and image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving picture coder for detecting a shift amount usable by an image processing apparatus for composing images and using a motion vector on the basis of the shift amount, so as to code a moving picture. <P>SOLUTION: The moving picture coder generates coded data provided to an image processing apparatus using a plurality of frame images composing a moving picture so as to apply processing with respect to a deviation between frame images includes: a shift amount detection means that detects a deviation of a position of the noted frame images between the two frame images among the plurality of frame images; a motion vector calculation means for converting the detected shift amount into a form of vector information, so as to calculate a motion vector used for the coding; and a coding means for generating the coded data in a form capable of extracting the motion vector at the image processing apparatus side. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a moving image encoding device, a moving image encoding method, a program, a recording medium, and a moving image encoding device for encoding moving image data using a motion vector based on a shift amount between images. The present invention relates to an image processing apparatus that inputs digitized data and generates one still image, and an image processing system.

  Conventionally, there are an image processing apparatus and an image processing method for generating a single high-resolution image by combining a plurality of images. For example, a technique is known in which one frame image is determined from a plurality of frame images constituting a moving image taken by a digital video camera, and a still image having a higher resolution (higher pixel density) than the determined frame image is generated. It has been. In this technique, a shift amount between a reference frame image and a target frame image is calculated, and based on the shift amount, each frame image is synthesized to generate a still image (for example, Patent Document 1).

Japanese Patent Laid-Open No. 2000-244851

  Many of the moving images that are materials in such image composition processing are moving images that are compressed by an encoding method using a motion vector typified by, for example, MPEG (Moving Picture Expert Group). In the MPEG format, the correlation between adjacent frames is effectively used, and the difference in the motion portion is reduced using motion compensation prediction, thereby reducing the amount of encoded information and compressing the entire moving image.

  However, when synthesizing images compressed in the MPEG format, the compressed data is decoded to generate a frame image, and then the amount of shift between the frame images is detected. There was a problem. The motion vector in the MPEG format is for the purpose of compressing the amount of data, and does not represent the overall position shift between frame images. That is, although the motion vector in the MPEG format is information related to the shift between the frame images, its value may be low in reliability for use in the synthesis process as the shift amount of the entire image, and is suitable for the image synthesis process. It was not.

  In view of these problems, the present invention provides a moving image encoding device that detects a shift amount that can be used in an image processing apparatus that performs image synthesis and encodes a moving image using a motion vector based on the shift amount. It is an object of the present invention to construct an image processing apparatus and an image processing system that perform efficient image processing using data encoded by the moving image encoding apparatus.

  In view of the above problems, the moving picture encoding apparatus of the present invention employs the following technique. That is, a moving image encoding apparatus that generates encoded data to be provided to an image processing apparatus that uses a plurality of frame images constituting a moving image to perform a process related to a shift between the frame images. Among these, a shift amount detecting means for detecting a shift in the position of the entire frame image between the two frame images of interest, and converting the detected shift amount into a vector information format to obtain a motion vector used for encoding. The gist of the invention is that it comprises a motion vector calculation means for calculating, and an encoding means for generating the encoded data in a format in which the motion vector can be extracted on the image processing apparatus side.

  Also, the encoding method corresponding to the moving image encoding apparatus of the present invention provides encoded data to be provided to an image processing apparatus that performs processing relating to a shift between the frame images using a plurality of frame images constituting the moving image. A moving image encoding method to be generated, comprising: detecting a shift of a position of an entire frame image between two focused frame images out of the plurality of frame images, and converting the detected shift amount into a format of vector information The gist is to calculate the motion vector used for encoding, and to generate the encoded data in a format in which the motion vector can be extracted on the image processing apparatus side.

  According to the moving picture coding apparatus and the coding method of the present invention, a motion vector is calculated based on the amount of position shift of the entire frame image, and coded data is generated using this. That is, the motion vector included in the encoded data represents a positional shift between frame images. Therefore, on the image processing apparatus side, it is not necessary to detect the amount of shift between frame images by using various detection methods, image processing can be executed using motion vectors, and processing time spent on image processing can be reduced. It can be shortened. Resource saving can be achieved by adding a motion vector necessary for an encoding method for compressing the data amount of a moving image to encoded data in a form suitable for image processing.

  It should be noted that a frame image that is divided into several blocks of about 2 or 4 and detects a positional shift in units of blocks can be regarded as “detecting a positional shift of the entire frame image”. .

  The shift amount detection means of the moving picture coding apparatus having the above configuration may be a means for detecting a shift amount in the rotation direction in addition to the shift amount in the translation direction between the frame images.

  According to such a moving image coding apparatus, the shift amount between the translation direction and the rotation direction is detected as a shift in the position of the entire frame image, and a motion vector is obtained based on the shift amount. When this motion vector is used for image processing, it is possible to execute image processing that takes into account the rotational direction deviation. Therefore, in an image processing apparatus that performs processing related to shift, for example, an image processing apparatus that corrects shift between frame images and combines them to generate one still image, the processing time is shortened and the accuracy of image processing is improved. Can be achieved.

  The moving image encoding apparatus having the above configuration further includes means for adding an identifier for identifying that the encoded data is based on a motion vector based on a shift amount between the frame images to a predetermined area in the encoded data. It is good also as a thing provided.

  According to such a moving image encoding apparatus, an identifier is added to a predetermined area in the encoded data. This identifier indicates that the motion vector is based on a shift amount between frame images or a special motion vector. By adding such information to the encoded data, it is possible to easily recognize the difference from the motion vector according to the normal encoding method.

  The encoding format in the moving image encoding apparatus having the above configuration is the MPEG standard and H.264 standard. The encoding format may conform to the 26x standard.

  According to such a moving image encoding apparatus, typical MPEG formats and H.264 encoding formats using motion vectors. Encoded data is generated using 26x format encoding. Therefore, if a motion vector based on the shift amount is obtained, an existing encoding technique can be used.

  An image processing apparatus that handles such encoded data decodes the encoded data generated by the moving image encoding apparatus, generates a plurality of frame images constituting the moving image, and generates the frame images from the generated frame images. An image processing apparatus that obtains a predetermined number of frame images and generates a single still image by combining the predetermined number of frame images, wherein a deviation amount between the frame images is determined prior to the synthesis of the frame images. It is possible to provide a deviation amount correcting means for correcting, the deviation amount correcting means extracting the motion vector corresponding to the frame image, and executing the correction of the deviation amount using the motion vector.

  According to such an image processing apparatus, the motion vector added to the encoded data is extracted, the shift amount is corrected using the extracted motion vector, and the synthesizing process is executed. Therefore, the shift amount used for correcting the shift amount between frame images. Is easy to detect. Accordingly, it is possible to shorten the processing time of a series of image processing, particularly detection processing of the shift amount.

  The image processing apparatus according to a different aspect from the above decodes the encoded data generated by the moving image encoding apparatus to generate a plurality of frame images constituting the moving image, and the generated An image processing apparatus that acquires a predetermined number of frame images from a frame image and generates a single still image by combining the predetermined number of frame images, and before the frame images are combined, A shift amount correcting unit that detects a shift of the position of the entire frame image and corrects a shift amount between the frame images based on the detected value, and the shift amount correcting unit includes the predetermined amount of the encoded data. When the area is read and it is determined that the data in the predetermined area is the identifier, the motion vector corresponding to the frame image is used instead of detecting the position shift of the entire frame image. Extracted, it can be assumed that the amount of deviation is detected by using a motion vector.

  According to such an image processing apparatus, a predetermined area of encoded data is read prior to correction of the shift amount. As a result, when it is determined that the data in the predetermined area is an identifier, image processing is executed using a motion vector corresponding to the frame image. For example, if the encoded data is normal MPEG format, the position shift of the entire frame image is detected, and if the encoded data is MPEG format data including a motion vector based on the shift amount, the motion vector is extracted. To detect the amount of deviation. The determination of which process is performed is made by confirming the identifier. Therefore, the availability of motion vectors in image processing can be easily determined at an early stage of image processing.

  An image processing system including such a moving image encoding device and an image processing device includes a moving image encoding device that encodes moving image data to generate encoded data, and inputs the encoded data, An image processing system comprising: an image processing device that generates a single still image from a plurality of frame images that are configured, wherein the moving image encoding device includes a plurality of frame images that constitute the moving image, A shift amount detecting means for detecting a shift in the position of the entire frame image between the two frame images of interest, and converting the detected shift amount into a vector information format to calculate a motion vector used for the encoding. A motion vector calculating unit; and an encoding unit configured to generate the encoded data in a format in which the motion vector can be extracted on the image processing apparatus side. A decoding unit that decodes the encoded data and generates a plurality of frame images constituting a moving image; a frame image acquisition unit that acquires a predetermined number of frame images from the plurality of frame images; A motion vector extracting means for extracting the motion vector for the frame image, a shift amount correcting means for correcting a shift amount between the predetermined number of frame images using the extracted motion vector, and the corrected predetermined number And a synthesizing unit for synthesizing the frame images to generate one still image.

  According to such an image processing system, the moving image encoding device generates encoded data suitable for processing related to the shift amount of the image processing device, and the image processing device generates a motion vector of the encoded data suitable for image processing. Used to execute processing related to the deviation amount. Conventionally, two similar processes, ie, a motion vector detection process between frame images necessary for encoding and a shift amount detection process between frame images used for image processing, are used as a motion vector based on the shift amount. Summarize and use effectively. Therefore, it is possible to construct a system that shortens the image processing time and saves resources.

  The present invention can also be implemented as a computer program and a medium recording the computer program. As the recording medium, various computer-readable media such as a flexible disk, a CD-ROM, a DVD-ROM / RAM, a magneto-optical disk, a memory card, and a hard disk can be used.

Hereinafter, embodiments of the present invention will be described in the following order based on examples.
A1. Image processing system configuration:
A2. Encoding process:
A2-1. Motion vector calculation processing:
A3. High resolution processing:
B. Variations:

A1. Image processing system configuration:
FIG. 1 is an explanatory diagram showing an image processing system 100 as an embodiment of the present invention. As shown in the figure, the image processing system 100 includes an image database 20 including moving image data, a moving image encoding apparatus that performs encoding of a predetermined format on moving image data input from the image database 20, or A personal computer 30 as an image processing apparatus that performs image processing on moving image data, a user interface 40 for a user to instruct execution of data encoding and image processing, and a color that outputs an image subjected to image processing It comprises a printer 50 and the like.

  The image database 20 includes devices that handle moving image data such as a digital video camera 21, a DVD 23, and a hard disk 24. The moving image data handled in this embodiment is data acquired by the digital video camera 21. This moving image data is composed of a plurality of frame images constituting the moving image. It is assumed that the moving image data of the present embodiment is not subjected to irreversible compression processing.

  The user interface 40 includes a keyboard 41 and a mouse 42 for a user to perform data encoding and image processing execution operations, a display 43 for displaying still images before image processing and still images after composition processing, and the like. I have.

  A personal computer 30 (hereinafter referred to as a PC 30) communicates with external devices such as a CPU 31 that executes image processing, a ROM 32, a RAM 33, a hard disk 34 that installs various software, an image database 20, a user interface 40, and a color printer 50. I / F circuit unit 35 and the like are connected to each other by an internal bus.

  A predetermined operation system and a plurality of application programs that operate on the operation system are installed in the hard disk 34. One of these programs extracts a predetermined number of frame images from a plurality of frame images constituting a moving image, corrects a deviation between the frame images, and combines the corrected frame images to produce a single high-resolution image. Is a program for generating a still image (hereinafter referred to as high resolution processing). The other is represented by “motion vector” indicating the amount of shift between frame images used for correction of shift in the resolution enhancement processing, and using this motion vector, moving image data is encoded in a predetermined format (hereinafter referred to as code). This is a program to be referred to as a conversion process. In other words, the latter is a so-called software encoder, and the former is a software decoder and an image processing program that uses decoded data. The CPU 31 develops such a program in the RAM 33 and executes it.

  In the present embodiment, the MPEG (Moving Picture Expert Group) format (MPEG-1, MPEG-2, MPEG-4), which is a typical encoding format using motion vectors, is used as the encoding of the predetermined format. As the encoding of the moving image data, if the encoding is performed using a motion vector, H.264 is used. 261, H.M. 263, H.M. H.264 H.264. It may be compliant with various standards other than the MPEG format, such as the 26x standard.

  FIG. 2 is an explanatory diagram showing the flow of processing in the image processing system 100 of the present embodiment. As shown in the figure, in the image processing system 100 configured as described above, moving image data is input, encoding processing is performed, and encoded data in MPEG format including a motion vector based on the shift amount is generated. The generated encoded data is stored in a memory such as the hard disks 24 and 34, and the processing as the moving image encoding apparatus is completed. When the user selects one piece of encoded data from the memory, the selected encoded data is decoded to generate moving image data, and a high-resolution process is performed on a predetermined frame image of the moving image data. One still image is generated. One still image generated in this way is output, and the processing as the image processing apparatus is terminated. Hereinafter, the encoding process and the high resolution process will be described.

A2. Encoding process:
Of the two processes executed by the image processing system 100 of the present embodiment, the encoding process will be described first. FIG. 3 is a flowchart of the moving image data encoding process. This process is executed by the CPU 31 of the PC 30 according to the user's encoding instruction.

  When the process is started, the PC 30 inputs the moving image data acquired by the digital video camera 21 to be encoded, and acquires a frame image from the moving image data (step S300). The acquired frame image is stored in a frame memory which is a predetermined storage area (not shown).

  The obtained frame image to be encoded is set and controlled as to which of the three picture types in accordance with the MPEG encoding method (step S310). In the MPEG format encoding, there are three prediction directions based on the motion vectors of the forward direction, the reverse direction, and the bidirectional direction, and three picture types (image types) are prepared by combining these prediction directions. The three image types are an I picture obtained by encoding the original data without using prediction, a P picture using forward motion compensation prediction, and a B picture using bidirectional motion compensation prediction. These pictures are handled in group units called GOP (Group of Pictures), and at least one I picture is provided in one GOP. The I picture serves as a reference for the picture in the GOP, and is encoded first in the GOP. Therefore, in the first stage of the encoding process that is repeatedly executed, generation of an I picture is set.

  Subsequently, based on the set picture type, it is determined whether or not the frame image to be encoded requires motion compensation prediction using a motion vector (step S320). As described above, P and B pictures are encoded using motion vectors based on past and future images that are predicted images. Therefore, in step S320, it is determined whether the picture type is P, B picture, or I picture.

  In step S320, if the frame image to be encoded is a P picture or a B picture, a motion vector calculation process is executed (step S330). This motion vector calculation process is a process for detecting a position shift of the entire frame image between frame images and calculating a motion vector of the frame image to be encoded based on the shift amount. In this processing, a motion vector is calculated for each macro block unit of 16 × 16 pixels based on the shift amount.

  Specifically, when encoding P and B pictures, a frame image (hereinafter referred to as an encoding reference frame image) serving as a reference of a predicted image in the frame memory is read, and a frame image (hereinafter referred to as an encoding reference frame image) to be encoded is read out. A shift between frame images with a frame image to be encoded) is detected. Note that the details of the motion vector calculation processing based on the amount of shift between frame images are the main part of the present invention, and will be described in detail later.

  Subsequently, in step S330, motion compensation prediction is performed using the motion vector obtained for each macroblock (step S340). Motion-compensated prediction is performed by general MPEG encoding, from one macroblock (processing macroblock) of the encoding target frame image and a position corresponding to the processing macroblock on the encoding reference frame image. This is a process for obtaining a difference (prediction error) from a predicted macroblock at a position shifted by a motion vector. In the MPEG format, the amount of data is compressed by such motion compensation prediction. When performing motion compensation prediction, a local decoder in the frame memory is used to execute the same processing as in decoding such as inverse quantization and inverse DCT (inverse discrete cosine transform) transformation, and the data is The difference is calculated using this.

  The PC 30 executes DCT (discrete cosine transform) processing, which is processing for separating the data into horizontal and vertical frequency components, for the difference data obtained by motion compensation prediction (step S350), and performs the following processing Migrate to

  On the other hand, when the encoding target frame image is an I picture in step S320, DCT processing is executed without performing motion compensation prediction (step S350), and the process proceeds to the next step. As described above, since an I picture is a picture that is encoded without using prediction, motion compensation prediction using a motion vector is not executed.

  A quantization process is performed on the data thus DCT processed (step S360). The quantization process is a process of compressing using DCT coefficients of DCT-processed data arranged in the order of frequency components. Specifically, a process (quantization) is performed in which the DCT coefficient is divided by a predetermined unit value, and the result of which the result becomes a certain value or less is made zero.

  Subsequently, a VLC (variable length coding) process using a combination of a run-length code and a Huffman code is executed on the quantized value read while performing zigzag scanning (step S370). As a result of quantization, the high-frequency component is generally zero, so the data is compressed by the VLC process.

  After performing the processing executed in the general MPEG format encoding such as motion compensation, DCT, quantization, and VLC, the PC 30 writes the generated VLC data (step S380). The writing of the VLC data itself is the same as the processing in the normal MPEG format, but in this step, the processing of writing “motion vector based on deviation amount” in the header of a group of data as a GOP is performed. Running. That is, when a series of encoding processes are repeated to form one GOP, an identifier ID indicating that the MPEG format is compatible with a high-resolution process described later is stored in the user area in the GOP header. Yes.

  The PC 30 determines whether or not a series of encoding processing conforming to the MPEG format has been completed for all the frame images constituting the input moving image data (step S390). If it is determined in step S390 that the processing has not been completed for all frame images, the process returns to step S300, and a series of encoding processes is repeated. On the other hand, if it is determined in step S390 that the processing has been completed for all the frame images, the file is closed and the series of encoding processing is terminated. In this way, MPEG-format encoded data having a motion vector based on the deviation amount is generated. The generated encoded data is stored in the DVD 23 and the hard disk 24 as the image database 20.

  The encoded data in the MPEG format thus generated has a 6-layer hierarchical structure. FIG. 4 is a conceptual diagram showing a data structure of encoded data in the MPEG format. As shown in the figure, the encoded data is composed of six layers of sequence, GOP, picture, slice, macro block, and block. The motion vector is written into this macroblock layer.

  In the above-described encoding process, the processing content of the general MPEG format encoding process (step S340 and subsequent steps) has been briefly described. In practice, however, the motion compensation prediction process is executed in units of macroblocks, and the DCT process is performed. Etc. are executed in block units. That is, in forming the data structure shown in FIG. 4, the process up to the motion vector calculation process shown in step S330 of the encoding process is executed for each picture, and the motion vector obtained for each macroblock constituting the picture is obtained. Once stored in the frame memory. Subsequent motion compensation, DCT, quantization, and VLC processes, which are general MPEG processes, are executed in units of macroblocks and blocks. In this MPEG processing, a motion vector corresponding to one macroblock is read from the frame memory and subjected to processing such as motion compensation, and is repeatedly executed for all macroblocks constituting one picture until the processing is completed. . Therefore, in the data writing process in step S380, the motion vector is written in the processed macroblock unit, the process is repeated up to the picture unit, and the identifier ID is written in the GOP unit processing stage including a plurality of pictures.

A2-1. Motion vector calculation processing:
FIG. 5 is a flowchart of a motion vector calculation process for obtaining a motion vector based on the shift amount in the encoding process. This process is executed at the time of encoding a P picture or a B picture, as shown in step S330 of FIG.

  When the process is started, the PC 30 executes a process for detecting a shift amount between frame images (step S500). Specifically, a positional shift amount between the frame images of the encoding reference frame image extracted from the frame memory and the encoding target frame image is detected.

  The “deviation amount” detected here means a deviation in position in the translation direction and the rotation direction between the frame images due to camera work such as panning and tilting or camera shake. Hereinafter, the shift amount in the translation direction is referred to as a translation shift amount (u, v), and the shift amount in the rotation direction is referred to as a rotation shift amount δ.

  The detection of the shift amount is based on a gradient method in which the pixel position is estimated in units smaller than one pixel using the luminance of each pixel between frame images. FIG. 6 is an explanatory diagram showing a method of calculating the translational deviation (u, v) by this gradient method. FIG. 6A shows the luminance of the pixels on each image, and FIG. 6B shows the principle of the gradient method. Here, (x1i, y1i) represents the coordinates of one pixel on the encoding reference frame image Fr, and Z1 (x1i, y1i) represents the luminance of the pixel. Here, it is assumed that the pixel of the coordinate (x2i, y2i) on the encoding target frame image Ft is between the coordinates (x1i to x1i + 1, y1i to y1i + 1) on the encoding reference frame image Fr. Is (x1i + Δx, y1i + Δy).

As shown in FIG. 6B, it is assumed that the pixel at the coordinates (x2i, y2i) in the encoding target frame image Ft is at the coordinates (x1i + Δx, y1i) on the encoding reference frame image Fr.
Px = Z1 (x1i + 1, y1i) −Z1 (x1i, y1i) (1)
Then,
Px · Δx = Z2 (x2i, y2i) −Z1 (x1i, y1i) (2)
Holds. In this case, if Z1 (x1i, y1i) and Z2 (x2i, y2i) are simply placed as Z1, Z2,
{Px · Δx− (Z2−Z1)} ² = 0 (3)
If Δx that satisfies the above is obtained, the translational deviation amount in the x-axis direction of the encoding target frame image F2 can be obtained. Actually, Δx is calculated for each pixel and averaged.

Similarly, it is assumed that the pixel at the coordinates (x2i, y2i) in the encoding target frame image Ft is at the coordinates (x1i, y1i + Δy) on the encoding reference frame image Fr,
Py = Z1 (x1i, y1i + 1) −Z1 (x1i, y1i) (4)
Then,
Py · Δy = Z2 (x2i, y2i) −Z1 (x1i, y1i) (5)
Holds. In this case, if Z1 (x1i, y1i) and Z2 (x2i, y2i) are simply placed as Z1, Z2,
{Py · Δy− (Z2−Z1)} ² = 0 (6)
If Δy that satisfies the above is obtained, the translational deviation amount in the y-axis direction of the encoding target frame image Ft can be obtained. In practice, Δy is calculated for each pixel and averaged.

The above formula (3) is a case where only the x-axis direction is considered, and the above formula (6) is a case where only the y-axis direction is considered. Therefore, when this is expanded in both the x-axis direction and the y-axis direction,
S ² = Σ {Px · Δx + Py · Δy− (Z2−Z1)} ² (7)
.DELTA.x and .DELTA.y that minimize the .sigma. Can be obtained by the method of least squares. Δx and Δy thus obtained correspond to the translational deviation amounts u and v.

The above is a calculation for the case where there is only a translational deviation between frame images. In addition, a method for calculating a deviation amount in consideration of the rotational deviation amount δ will be described. FIG. 7 is an explanatory diagram schematically showing the amount of pixel rotation deviation. As shown in the figure, assuming that the distance from the origin O of a certain coordinate (x, y) of the encoded reference frame image Fr is r and the rotation angle from the x-axis is θ, r and θ are obtained by the following general expressions. .
r = (x ² + y ² ) ^1/2 (8)
θ = tan ⁻¹ (x / y) (9)

Here, assuming that the translational deviation is corrected, the origin O of the encoding reference frame image Fr and the encoding target frame image Ft are matched, and the coordinates (x1, x1) of the encoding target frame image Ft are centered on the origin O. It is assumed that when y1) is rotated by an angle δ, it coincides with the coordinates (x2, y2) of the encoding target frame image Ft. The amount of movement Δx in the x-axis direction and the amount of movement Δy in the y-axis direction due to this rotation can be obtained by the following equations. In addition, assuming that the rotational deviation amount δ is a minute amount, an approximate expression of cos δ≈1 and sin δ≈δ is used.
Δx = x2−x1≈−r · δ · sinδ = −δ · y1 (10)
Δy = y2−y1≈r · δ · cosδ = δ · x1 (11)

Therefore, when Δx and Δy in the above equation (7) are expressed by adding the translational deviation amounts u and v to the rotational deviation amount δ, the following equations are obtained.
Δx = u−δ · y1 (12)
Δy = v + δ · x1 (13)
Substituting these into equation (7) gives the following general equation:
S ² = Σ {Px · (u−δ · y) + Py · (v + δ · x) − (Z2−Z1)} ² (14)

That is, by obtaining u, v, and δ that minimizes S ² in the above equation (14) by the method of least squares, it is possible to accurately detect a shift amount of less than one pixel between frame images.

  Prior to the detection of such a shift amount, a rough shift amount detection (for example, detection by a pattern matching method) in units of pixels may be executed for only the translational shift amount (u, v). By roughly detecting and correcting the amount of translational deviation (u, v) between the frame images, it is possible to obtain a more accurate detected value.

  Returning to FIG. 5, a motion vector is calculated using the shift amount (u, v, δ) between the frame images thus detected (step S510). Specifically, the motion vector is calculated from the geometric relationship between the shift amount (u, v, δ), the encoding reference frame image, and the encoding target frame image.

FIG. 8 is an explanatory diagram showing a geometric relationship for deriving a motion vector. 8 shows the encoding reference frame image (x1, y1) and the encoding target frame image (x2, y2), and the middle part of FIG. 8 shows the encoding target frame image as the encoding reference frame image. By shifting (u, v, δ), the images match. That is, the following equation is established between frame images.
x1 = cosδ · x2−sinδ · y2 + u (15)
y1 = sinδ · x2 + cosδ · y2 + v (16)

As described in the detection of the deviation amounts (u, v, δ), assuming that the rotational deviation amount δ is a minute amount, the following equations are obtained by using approximate expressions of cos δ≈1 and sin δ≈δ.
x1 = x2-δ · y2 + u (17)
y1 = δ · x2 + y2 + v (18)

Here, the coordinate point (xt, yt) on the encoding target frame image is set as the center point coordinate of one macroblock constituting the encoding target frame image, and the coordinate point (xr, yr) on the encoding reference frame image is set. And the corresponding point coordinates corresponding to the center point coordinates,
xr = xt−δ · yt + u (19)
yr = δ · xt + yt + v (20)
The motion vector MV from the center point coordinates to the corresponding point coordinates is expressed by the following equation.
MV = (xr−xt, yr−yt) = (− δ · yt + u, δ · xt + v) (21)

  The motion vector MV is calculated for all the macroblocks of the encoding target frame image by substituting the shift amount (u, v, δ) between the frame images obtained in step S500 of FIG. 5 into the equation (21). . Thus, after calculating the motion vector for the encoding target frame image and storing it in the frame memory, the series of processing ends, and the process proceeds to the motion compensation prediction processing shown in step S340 of FIG.

  Note that the size of the motion vector MV in the case of assuming decoding (for example, reproduction of a moving image) by a general MPEG format decoder is up to a half pixel unit in the MPEG-1 and MPEG-2 formats, and in the MPEG-4 format. Then, it is possible to 1/4 pixel unit. In the present embodiment, the calculated motion vector MV is corrected in units of half pixels and used as data stored in the macroblock.

  In addition, when encoding is performed in the MPEG-1 and MPEG-2 formats, the validity of the calculated motion vector MV for each macroblock is confirmed. In the MPEG-1 and MPEG-2 formats, reference to macroblocks outside the frame of the encoding standard frame image is prohibited. Specifically, as in the case of the macro block MB4 shown in FIG. 9, when the position of the macro block (slashed line) moved by the calculated motion vector is outside the frame of the encoded reference frame image, the motion Vectors cannot be used. A block corresponding to such a macro block is an intra block having no motion vector (a block that performs intra-frame coding).

  Furthermore, since the range in which the motion vector MV itself can be expressed is set in advance, even when the x component and y component of the motion vector MV do not fall within the set range, the macro block is set as an intra block.

  The motion vector obtained in this way is calculated by detecting a shift of the entire frame image between frame images. Therefore, in various apparatuses that perform processing related to the shift amount between frame images, for example, an image processing apparatus that generates a still image by combining a plurality of frame images that have been corrected for shift amounts, encoded data including a motion vector Can be used, and the processing time relating to the amount of deviation can be shortened.

  Since the motion vector in the general MPEG format is calculated so as to minimize the difference data between the frame images for each block of a predetermined size, it does not represent the total shift between the frame images. That is, it does not necessarily represent information that captures the motion between frame images, and it is unreliable to use the information as it is as motion information between frame images.

  On the other hand, since the motion vector of the present embodiment is based on the shift amount of the entire frame image, it is suitable for image processing involving processing related to the shift amount, and the motion vector for encoding moving images is effectively used. can do.

  Since the encoded data of this embodiment is a motion vector based on the shift of the entire frame image, shifting one frame image almost overlaps with another frame image, and the difference data between the frame images decreases. Therefore, as compared with general MPEG encoded data, even if the amount of information such as motion vectors increases, the overall data amount does not increase. That is, it is possible to generate encoded data to which a motion vector suitable for image processing is added while maintaining a data amount substantially equivalent to general MPEG encoded data.

  In this embodiment, the shift of the frame image “whole” is obtained. However, even if the frame image is divided into several blocks and the shift is obtained for each block, it is divided into two, four, If it is about a division, it can be considered that the frame image “whole” is seeking a shift.

  Further, in this embodiment, in order to explain the principle of obtaining the motion vector of each macroblock from the shift amount of the entire frame image, it is assumed that there is no moving object region having a large motion in the frame image to be encoded. However, in practice, there may be a moving object region in the frame image to be encoded. In this case, a position shift is detected in units of macro blocks, and a macro block that is determined to have a large motion with respect to the shift of the entire frame image may be set as an intra block.

A3. High resolution processing:
Next, among the two processes executed by the image processing system 100 of the present embodiment, the resolution enhancement process will be described. FIG. 10 is a flowchart of the resolution enhancement process. The encoded data generated by the above-described encoding process is converted into time-series continuous frame images via an MPEG decoder which is a software decoder. The PC 30 reproduces a moving image composed of continuous time-series frame images on the display 43 in accordance with a user's reproduction operation.

  The user selects one scene desired to be output as a still image from the reproduced moving image by a predetermined operation, and designates the scene (frame image). When the user instructs execution of a resolution enhancement process for outputting the frame image as a high resolution still image, the flowchart shown in FIG.

  The PC 30 extracts frame images used for image processing from the designated frame images in time series order (step S700). In this embodiment, it is assumed that four frame images that are continuous in time series are extracted from the operation timing for designating the frame image. These four frame images include one I picture and P picture, the I picture and P picture are called reference frame images, and the other three pictures are called target frame images. Note that the number of frame images to be extracted may be arbitrarily set by the user.

  The PC 30 determines whether or not the extracted frame image is MPEG-format encoded data having a motion vector for high resolution processing (that is, a motion vector based on a shift amount) (step S710). Specifically, the user area of the GOP header to which the frame image belongs is referred to, and if it has the above-described identifier ID, it is determined that it is for high resolution processing.

  If it is determined in step S710 that the data is in MPEG format for high resolution processing (Yes), a motion vector at the time of encoding the extracted target frame image is extracted (step S720). In this step, for each target frame image, two arbitrary macro blocks other than the intra block are selected from the macro blocks constituting the frame image, and a motion vector included in the macro block is extracted.

  From the extracted motion vectors of the two macroblocks, processing for detecting the translational deviation amount (u, v) of the target frame image is performed (step S730). This process is a process of inverting the sign of the value of one of the two extracted motion vectors and setting it as the translational deviation (u, v).

  Subsequently, in step S730, a process of detecting the rotational deviation amount δ from the extracted motion vector MV2 of the other macroblock with the center point of the macroblock used as a reference for detecting the translational deviation amount (u, v) as the origin. This is performed (step S740). In this process, the motion vector MV1 of the reference macroblock is subtracted from the motion vector MV2 of the other macroblock to detect a rotation component.

  The process of detecting the translational deviation amount (u, v) and the rotational deviation amount δ in steps S730 and S740 will be specifically described with reference to FIGS. FIG. 11 is a conceptual diagram showing a motion vector at the time of encoding one target frame image. As shown in the figure, each macroblock of the target frame image has a motion vector indicated by an arrow. Here, it is assumed that arbitrary macroblocks MB15 and MB18 are selected and a motion vector MV1 (a, 0) of the macroblock MB15 is extracted. In step S730, the central point of the macro block MB15 is considered as the origin, and (−a, 0) obtained by inverting the sign of the motion vector MV1 (a, 0) is set as the translational deviation amount (u, v).

  FIG. 12 is a conceptual diagram showing the rotation component of the motion vector. FIG. 12 is obtained by subtracting the motion vector MV1 of the macroblock MB15 from the motion vector of each macroblock shown in FIG. As shown in the figure, the motion vector is expressed only by the rotation component having the origin at the center point of the macroblock MB15. The motion vector MV2 of the macro block MB18 represented by only the rotation component is referred to as a motion vector MV2δ.

In step S740, a distance L (referred to as a line segment L) from the origin (the center point of the macroblock MB15) to the center point of the other macroblock MB18 is obtained, and a vertical component P with respect to the line segment L of the motion vector MV2δ is obtained. . From the relationship between the obtained line segment L and the vertical component P, the rotational component α is calculated by the following equation, and −α, which is the sign of the rotational component α inverted, is set as the rotational deviation amount δ.
α = arctan (P / L) (22)

  Note that the inversion of the sign of the rotation component α is due to the difference in the expression method between the rotation component of the motion vector and the rotation shift amount δ in the resolution enhancement processing. FIG. 13 is an explanatory diagram showing the relationship between the rotational deviation amount and the rotational component of the motion vector in the resolution enhancement processing. As shown in the figure, the amount of rotational deviation in the resolution enhancement process represents the amount of deviation for rotating the target frame image with respect to the reference frame image to match the image contents. On the other hand, the rotation component of the motion vector expresses a rotation amount for rotating the image content on the reference frame image to match the image content on the target frame image after a predetermined time. Therefore, in order to replace the rotational displacement amount in the resolution enhancement processing using the rotational component of the motion vector, the sign is inverted. The same applies to the inversion of the sign in the detection of the translational deviation amount.

  Through the above processing, the shift amount (u, v, δ) of the target frame image is detected as (−a, 0, −α). Similarly, for other target frame images, the shift amount (u, v, δ) is detected from the motion vector. By using the motion vector at the time of encoding, the shift amount (u, v, δ) can be easily detected.

  On the other hand, if it is determined in step S710 of FIG. 10 that the data is not MPEG format data for high resolution processing (No), the shift amount between the target frame image Ft and the reference frame image Fr is detected (step S750). . This shift amount detection process is a process for detecting a shift of the entire frame image using a gradient method, a pattern match method, or the like, similar to the process shown in step S500 of FIG.

  Subsequently, the target frame image is corrected based on the detected shift amount (u, v, δ) (step S760). Specifically, the target frame image is converted into an image in a state where the target frame image is moved in the translation direction by (u, v) and by δ in the rotation direction. Such correction of the shift amounts (u, v, δ) is performed on the three target frame images, and the target frame image after shift correction is superimposed on the reference frame image.

  One high pixel density still image is generated by a predetermined interpolation method using pixels of a plurality of superimposed frame images (step S770). As the interpolation method, an existing method such as a bi-linear method, a bi-cubic method, or a nearest neighbor method can be used. In the image processing of this embodiment, interpolation processing is executed using the bi-linear method, and a still image that has been subjected to high resolution conversion 1.5 times densely with respect to the reference frame image is generated. The PC 30 displays the still image synthesized in this way at a predetermined position on the display 43 and ends this process. The user performs an operation of outputting the still image to the color printer 50, the hard disk 34, or the like according to his / her preference. In image processing using such interpolation processing, frame images are shifted and synthesized, so that block distortion of an image compressed in the MPEG format or the like can be reduced and a clear still image can be generated.

  In the above-described high resolution processing, it is determined whether or not the motion vector can be used for detecting the shift amount with reference to the identifier ID of the encoded data in the MPEG format. When it is determined that a motion vector can be used, the amount of shift between frame images is detected using the motion vector. Since the detection of the shift amount is a relatively easy calculation, it is possible to shorten the time for the shift amount detection process in the image processing.

  In this embodiment, the translational deviation amount (u, v) and the rotational deviation amount δ are detected from two arbitrary macroblocks. However, for example, a plurality of motion vectors may be used.

  For example, in step S730 in FIG. 10, the motion vectors in all the macroblocks are extracted, the average values of the X and Y components of the extracted motion vectors are calculated, the sign is inverted, and the translational deviation amount (u, v ) May be set. By doing so, it is possible to reduce the quantization error at the time of encoding which the motion vector has, and to obtain the translational deviation amount (u, v) with sub-pixel accuracy.

  In step S740 in FIG. 10, the center of the target frame image is set as the origin, macroblocks at positions (four places) farthest from the origin in the X direction and the Y direction are selected, and the average value component is selected from the motion vector components. Are subtracted and four rotational components are obtained using equation (22) (see FIG. 14). An average value of the calculated rotational components of the motion vector may be obtained, and the rotational deviation amount δ may be set by inverting the sign.

  In the present embodiment, the principle of the method of obtaining the shift amount from the motion vector by setting the reference frame image as an I picture or P picture has been described. However, one frame image (for example, a B picture) selected by the user is used. It may be set as a basic frame image as it is.

  In such a setting method, depending on the combination of the basic frame image to be extracted and the target frame image (the combination of picture types), the motion vector of the target frame image does not indicate a deviation from the basic frame image (that is, two pictures May not have a direct correlation). In this case, another picture that is correlated in common with two pictures that are not directly correlated is extracted, and the correlation, that is, a shift between frame images is obtained indirectly using this picture. It ’s fine.

B. Variations:
In the encoding process of the present embodiment, the motion vector is recorded in units of half pixels. However, when the accuracy is insufficient when performing the process related to the shift amount, the user data area of the encoded data is used. It is good as a thing.

  For example, by detecting the shift amount in the motion vector calculation process of FIG. 5, the shift amount is obtained in units of 1/4 pixel, and the shift amount (u, v, δ) is (3.25, −2.25, 0.5). °). In this case, (3, -2, 0.5 °) is used to calculate the motion vector, and the frame No. In addition, (0.25, −0.25, 0) is stored. When the resolution enhancement processing is executed using such encoded data, a true shift amount obtained by correcting the shift amount obtained from the motion vector with the stored value of the user data area may be used. By doing so, the accuracy of image processing can be improved. Further, the shift amount (u, v, δ) itself in units of 1/4 pixel and 1/8 pixel may be stored in the user data area.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to such embodiment at all, Of course, it can implement with various forms within the range which does not deviate from the meaning of this invention. is there. In this embodiment, the personal computer in which the encoding processing and high resolution processing programs are installed has been described as a moving image encoding device and an image processing device. However, for example, this function is applied to various devices such as a printer and a digital video camera. And a moving picture encoding apparatus and an image processing apparatus according to the present invention.

  In particular, when the digital video camera is provided with the encoding function of the present invention, it is possible to detect the shift amount of the entire frame image with respect to the captured frame image closest to the raw data. By executing the resolution enhancement process using the encoded data of the digital video camera, a still image with high accuracy can be generated.

It is explanatory drawing which shows the image processing system as one Example of this invention. It is explanatory drawing which showed the flow of the process in the image processing system of a present Example. It is a flowchart of the encoding process of moving image data. It is a conceptual diagram which shows the data structure of the encoding data of an MPEG format. It is a flowchart of the motion vector calculation process which calculates | requires the motion vector based on deviation | shift amount. It is explanatory drawing which shows the calculation method of the translational deviation amount by the gradient method. It is explanatory drawing which shows typically the rotation shift amount of a pixel. It is explanatory drawing which showed the geometric relationship of derivation | leading-out of a motion vector. It is explanatory drawing explaining the expression range of a motion vector. It is a flowchart of high resolution processing. It is a conceptual diagram which shows the motion vector at the time of the encoding of the object frame image. It is the conceptual diagram shown by the rotation component of the motion vector. It is explanatory drawing which shows the relationship between the rotation shift amount and the rotation component of a motion vector in high resolution processing. It is explanatory drawing of calculation of the amount of rotation gaps in high resolution processing.

Explanation of symbols

20 ... Image database 21 ... Digital video camera 23 ... DVD
24 ... Hard disk 30 ... Personal computer 31 ... CPU
32 ... ROM
33 ... RAM
34 ... Hard disk 35 ... I / F circuit unit 40 ... User interface 41 ... Keyboard 42 ... Mouse 43 ... Display 50 ... Color printer 100 ... Image processing system

Claims (10)

A moving image encoding device that generates encoded data to be provided to an image processing device that performs processing relating to a shift between the frame images using a plurality of frame images constituting a moving image,
A deviation amount detecting means for detecting a deviation of the position of the entire frame image between two frame images of interest among the plurality of frame images;
Motion vector calculating means for converting the detected deviation amount into a vector information format and calculating a motion vector used for encoding;
A video encoding apparatus comprising: encoding means for generating the encoded data in a format in which the motion vector can be extracted on the image processing apparatus side. The moving image encoding device according to claim 1,
The displacement amount detection means is a moving image encoding device that is a means for detecting a displacement amount in the rotation direction in addition to the displacement amount in the translation direction between the frame images. The moving image encoding device according to claim 1, further comprising:
A moving picture coding apparatus comprising means for adding an identifier for identifying encoded data based on a motion vector based on a shift amount between the frame images to a predetermined area in the encoded data. The moving image encoding device according to any one of claims 1 to 3,
The encoding format includes MPEG standards and H.264 standards. A moving image encoding apparatus which is an encoding format compliant with the 26x standard. A plurality of frame images constituting a moving image are generated by decoding the encoded data generated by the moving image encoding apparatus according to any one of claims 1 to 4, and a predetermined number is generated from the generated frame images. An image processing device that generates a single still image by synthesizing the predetermined number of frame images,
Prior to the synthesis of the frame images, a shift amount correction means for correcting a shift amount between the frame images is provided.
The deviation correction means is
An image processing apparatus that extracts the motion vector corresponding to the frame image and corrects the shift amount using the motion vector. The encoded data generated by the moving image encoding device according to claim 3 is decoded to generate a plurality of frame images constituting the moving image, and a predetermined number of frame images are obtained from the generated frame images And an image processing device that generates a single still image by combining the predetermined number of frame images,
Prior to the synthesis of the frame images, a displacement amount correction unit that detects a displacement of the position of the entire frame image between the frame images and corrects the displacement amount between the frame images based on the detection value,
The deviation correction means is
When the predetermined area of the encoded data is read and it is determined that the data in the predetermined area is the identifier, instead of detecting the displacement of the entire frame image, the frame image corresponding to the frame image is detected. An image processing apparatus that extracts a motion vector and detects a shift amount using the motion vector. A moving image encoding device that encodes moving image data to generate encoded data, and an image processing device that inputs the encoded data and generates one still image from a plurality of frame images constituting the moving image; , An image processing system comprising:
The moving image encoding device is:
A shift amount detecting means for detecting a shift in the position of the entire frame image between two focused frame images among a plurality of frame images constituting the moving image;
Motion vector calculation means for converting the detected deviation amount into a vector information format and calculating a motion vector used for the encoding;
Encoding means for generating the encoded data in a format in which the motion vector can be extracted on the image processing apparatus side;
The image processing apparatus includes:
Decoding means for decoding the encoded data and generating a plurality of frame images constituting a moving image;
Frame image acquisition means for acquiring a predetermined number of frame images from the plurality of frame images;
Motion vector extraction means for extracting the motion vector for the acquired frame image;
Using the extracted motion vector, a deviation amount correcting means for correcting a deviation amount between the predetermined number of frame images;
An image processing system comprising: a combining unit configured to combine the corrected predetermined number of frame images to generate one still image. A moving image encoding method for generating encoded data to be provided to an image processing apparatus that performs processing relating to a shift between frame images using a plurality of frame images constituting a moving image,
Detecting a shift in the position of the entire frame image between the two frame images of interest among the plurality of frame images;
Converting the detected shift amount into a vector information format, and calculating a motion vector used for encoding;
A moving image encoding method for generating the encoded data in a format in which the motion vector can be extracted on the image processing apparatus side. A computer program for controlling a moving image encoding device that generates encoded data to be provided to an image processing device that performs processing relating to a shift between the frame images using a plurality of frame images constituting a moving image,
A function of detecting a shift of the position of the entire frame image between two frame images of interest among the plurality of frame images;
A function of converting the detected shift amount into a vector information format and calculating a motion vector used for encoding;
A computer program that causes a computer to realize the function of generating the encoded data in a format in which the motion vector can be extracted on the image processing apparatus side.   A recording medium in which the computer program according to claim 9 is recorded in a computer-readable manner.

Images