JP4412062B2 - Moving picture conversion apparatus, moving picture conversion method, and computer program - Google Patents

Description

  The present invention relates to a moving image conversion apparatus, a moving image conversion method, and a computer program. In particular, it enables high-quality image data restoration from compressed data with pixel thinning, and eliminates image quality degradation such as jaggies (folding of spatial frequency collection) that are conspicuous in slow-motion display of restored images. The present invention relates to a moving image conversion apparatus, a moving image conversion method, and a computer program that realizes accurate image restoration.

  The moving image data is subjected to data conversion for reducing the amount of data, that is, compression processing, when it is stored in a storage medium such as a hard disk or DVD or distributed via a network. In particular, in recent years, the quality of moving image data has been improved, for example, HD (High Definition) data has been improved, and the amount of data has increased rapidly with the improvement of the quality of this data. Under such circumstances, many studies and studies have been made on techniques for improving the compression efficiency in the compression and decompression processing of moving image data and preventing the quality degradation of the decompressed data.

  As a moving image compression processing method, for example, thinning processing of pixels constituting an image frame constituting moving image data, that is, thinning processing in a spatial direction, thinning processing of a frame rate, that is, thinning processing in a time direction, and the like are known. Yes.

  By reducing the amount of data by such data conversion, there is an advantage that saving to a storage medium and data transfer via a network are efficiently performed. However, when the compressed data is restored and reproduced, there is a problem that the image quality deteriorates. In particular, when the original data is a high-definition image, the quality deterioration becomes more remarkable.

Various studies have been made on how to reduce such image quality deterioration. For example, Patent Document 1 discloses an image compression processing configuration in which a parameter based on image brightness information is set and the compression mode is changed according to the image brightness. Patent Document 2 discloses a compression processing configuration in which a screen is divided into a plurality of areas and the compression mode is changed for each area.
JP 2003-169284 A Japanese Patent Laid-Open No. 2002-27466

  As described above, in some conventional techniques, a configuration is disclosed in which the compression mode is changed based on various characteristics obtained from the processing target image and the data quality is improved. In the method, the compressed image data is restored and reproduced, and image quality deterioration is not sufficiently suppressed. In particular, in slow motion display of an image obtained by restoring compressed data accompanied by pixel thinning, there is a problem that jaggy (folding of spatial frequency collection) based on pixel data lost by the thinning process becomes noticeable.

  The present invention has been made in view of such problems, and has made it possible to restore high-quality image data from compressed data with pixel thinning, and particularly in slow-motion display of images. An object of the present invention is to provide a moving image conversion apparatus, a moving image conversion method, and a computer program that can eliminate image quality degradation such as conspicuous jaggy and enable high-quality image restoration.

The first aspect of the present invention is:
A moving image conversion apparatus that executes a moving image compressed data restoration process,
An input unit for inputting compressed data in units of blocks as an area included in a frame constituting a moving image;
A block position in a different frame on the time axis is calculated based on motion information included in the compressed data or motion information calculated from the compressed data, and an address corresponding to the calculated position is used for frame buffer access corresponding to the frame. An address conversion unit that calculates an address;
A plurality of frame buffers corresponding to frames for storing pixel values included in block compressed data corresponding to each frame or calculated values based on the pixel values at a memory location specified by an address corresponding to the frame calculated by the address conversion unit;
An interpolation unit that generates a restoration block based on sample point data of a plurality of different frames based on data stored in the frame buffer;
An output unit that outputs moving image frame data constituted by the restoration block generated by the interpolation unit ;
The plurality of frame buffers include a restoration processing target frame, a pixel value over at least one of a frame preceding or following the restoration processing target frame on a time axis, or an overlap of calculated values based on the pixel value The moving image conversion apparatus is characterized in that writing is performed.

  Furthermore, in one embodiment of the moving image conversion device of the present invention, the moving image conversion device further includes pixel value data already written in a memory location of a frame buffer designated by a frame corresponding address calculated by the address conversion unit. Is present, an addition processing unit that calculates an average value of the pixel value data and a pixel value included in the block compression data corresponding to the processing frame, and the addition average value calculated by the addition processing unit The present invention is characterized in that a process for writing to a memory location specified by a frame correspondence address is executed.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the address conversion unit inputs a motion vector indicating block movement information in units of frames as the motion information of the block, and on the time axis based on the motion vector. The block positions in different frames are calculated, and an address corresponding to the calculated position is calculated as an address corresponding to each frame.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the interpolation unit may select a pixel value missing from the sample point data in the block configuration data in a direction parallel to the motion direction included in the motion information corresponding to the block. It is the structure which performs the process which interpolates based on this pixel value.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the interpolation unit performs a process of interpolating a pixel value missing from the sample point data in the block configuration data based on a surrounding pixel value of the missing pixel. It is the structure to perform.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the moving image conversion device further performs a process of sequentially switching a frame buffer storing output data for the interpolation unit as a process of switching output data for the interpolation unit. It is the structure which has the selection part to perform.

Furthermore, the second aspect of the present invention provides
It is a moving image conversion method for executing a moving image compressed data restoration process,
An input step for inputting compressed data in units of blocks as an area included in a frame constituting a moving image;
A block position in a different frame on the time axis is calculated based on motion information included in the compressed data or motion information calculated from the compressed data, and an address corresponding to the calculated position is used for frame buffer access corresponding to the frame. An address conversion step to calculate as an address;
A data storage step for each of a plurality of frame buffers, wherein the pixel value included in the block compressed data corresponding to each frame or the memory location of each frame buffer accessed by the address corresponding to the frame calculated in the address conversion step A data storage step for storing a calculated value based on the pixel value;
An interpolation step for generating a restoration block based on sample point data of a plurality of different frames based on the data stored in the frame buffer;
An output step of outputting moving image frame data constituted by the restoration block generated in the interpolation step ;
The data storing step includes
For the plurality of frame buffers, a pixel value over a plurality of frames of at least one of a restoration processing target frame and a frame preceding and following the restoration processing target frame on a time axis, or a calculated value based on the pixel value There is a moving image conversion method characterized in that it is a step of executing the redundant writing .

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the moving image conversion method further includes pixel value data already written in a memory location of a frame buffer designated by a frame corresponding address calculated in the address conversion step. Is present, an addition processing step of calculating an average value of the pixel value data and a pixel value included in the block compressed data corresponding to the processing frame is included, and the data storage step is calculated in the addition processing step. It is a step of executing a process of writing the added average value to a memory location designated by the frame corresponding address.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the address conversion step inputs a motion vector indicating block movement information in units of frames as block motion information, and on the time axis based on the motion vector. This is a step of calculating a block position in different frames and calculating an address corresponding to the calculated position as an address corresponding to each frame.

  Furthermore, in an embodiment of the moving image conversion method of the present invention, the interpolation step may include a pixel value missing from the sample point data in the block configuration data in a direction parallel to the motion direction included in the motion information corresponding to the block. The method includes a step of executing a process of performing interpolation based on the pixel value.

  Furthermore, in an embodiment of the moving image conversion method of the present invention, the interpolation step includes a process of interpolating a pixel value missing from the sample point data in block configuration data based on a surrounding pixel value of the missing pixel. Including a step of executing.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the moving image conversion method further includes frame buffer selection for sequentially switching frame buffers as output data switching processing for an interpolation unit that performs the pixel value interpolation processing. It has a step.

Furthermore, the third aspect of the present invention provides
A computer program for executing a decompression process of compressed moving image data,
An input step for inputting compressed data in units of blocks as an area included in a frame constituting a moving image;
A block position in a different frame on the time axis is calculated based on motion information included in the compressed data or motion information calculated from the compressed data, and an address corresponding to the calculated position is used for frame buffer access corresponding to the frame. An address conversion step to calculate as an address;
A data storing step for each of a plurality of frame buffers, wherein a pixel value included in the block compressed data corresponding to each frame or the memory location of each frame buffer accessed by the address corresponding to the frame calculated in the address converting step A data storage step for storing a calculated value based on the pixel value;
An interpolation step for generating a restoration block based on sample point data of a plurality of different frames based on the data stored in the frame buffer;
An output step of outputting moving image frame data constituted by the restoration block generated in the interpolation step ;
The data storing step includes
For the plurality of frame buffers, a pixel value over a plurality of frames of at least one of a restoration processing target frame and a frame preceding or following the restoration processing target frame on a time axis, or a calculated value based on the pixel value There is a computer program characterized in that it is a step of executing the redundant writing .

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, a block calculated based on motion information such as a motion vector of a block in decompression processing of compressed data in which pixel data is thinned out by a thinning process in a spatial direction or a thinning process in a time direction. Since the data of the same block over a plurality of frames is accumulated in the frame buffer reflecting the moving position of the data, and the data restoration is executed based on the accumulated sample point data of a plurality of different frames, missing in one frame It is possible to acquire the pixel values that are being processed based on the data of different frames, and the pixel value reproduction rate of each frame can be improved. As a result, high-quality moving image restoration closer to the original image is realized.

  In the conventional configuration, there is a problem that jaggies (folding of spatial collection frequency) based on the pixel data lost by the thinning-out process is conspicuous at the time of reproduction of the restored data. However, according to the present invention, the pixel value of each frame Since the reproducibility is improved, high-quality data close to the original data can be suppressed even in playback processing at a rate different from the playback rate of the original moving image, such as slow motion playback. Playback is realized.

Hereinafter, the configuration of a moving image conversion apparatus, a moving image conversion method, and a computer program according to the present invention will be described with reference to the drawings. The description will be made according to the following items.
(1) Basic configuration of moving picture conversion apparatus using super-resolution effect (2) Moving picture conversion apparatus and moving picture conversion method for generating high-quality restored image

[(1) Basic configuration of moving image conversion apparatus using super-resolution effect]
First, a basic configuration of a moving image conversion apparatus that performs moving image compression using the super-resolution effect as a base of the present invention will be described. This basic configuration is described in detail in Japanese Patent Application No. 2003-412501 filed earlier by the present applicant, and the image is divided into small areas (blocks), and according to the moving speed of each area. Thus, the amount of data is compressed by adaptively performing the thinning of the number of pixels and the thinning of the frame rate.

  FIG. 1 shows a configuration example of the moving image conversion apparatus 10 described in Japanese Patent Application No. 2003-412501. The moving image conversion apparatus 10 can reduce the amount of data so that an observer does not perceive image quality deterioration due to the reduction of the data amount by performing a moving image conversion process using the super-resolution effect. It is a thing.

  Note that the super-resolution effect is a visual effect in which the observer perceives a resolution higher than the number of display pixels when following the moving subject discretely sampled in the spatial direction. This is because a human has a visual characteristic of perceiving an addition of a plurality of images presented within a certain time. This characteristic is attributed to a visual temporal integration function known as a block law, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp. 219-220”. There is a report that the integration time for which the block law is established varies depending on the presentation conditions such as the intensity of the background light, but is approximately 25 ms to 100 ms.

  The moving image conversion apparatus 10 shown in FIG. 1 performs compression for reducing data so that the observer does not perceive image quality degradation by performing moving image conversion processing using the super-resolution effect caused by the temporal integration function. It is set as the structure to perform. A configuration of the moving image conversion apparatus 10 in FIG. 1 will be described.

  The block dividing unit 11 divides each frame of the input moving image into blocks as divided areas of predetermined pixels and supplies them to the movement amount detecting unit 12. The movement amount detection unit 12 detects the movement amount for each block supplied from the block division unit 11, and transmits the block and the movement amount to the block processing unit 13. The block processing unit 13 performs a moving image conversion process corresponding to the movement amount, that is, a compression process, on the block supplied from the movement amount detection unit 12 to reduce the data amount. The block processing unit 13 supplies the output unit 14 with data about the block with the reduced data amount obtained as a result of the processing. The output unit 14 collectively outputs, as stream data, the data regarding the blocks with the reduced data amount supplied from the block processing unit 13.

  Next, details of each unit will be described with reference to FIG. The frame of the moving image supplied to the moving image conversion apparatus 10 is input to the image storage unit 21 of the block dividing unit 11. The image storage unit 21 stores the input frame. Each time the number of accumulated frames reaches N (N is a positive integer), the image accumulating unit 21 supplies the N frames to the block dividing unit 22, and M in the N frames The second stored frame (hereinafter referred to as the Mth frame) is supplied to the movement amount detection unit 12 (movement amount detection unit 31). For example, N = 4.

  The block dividing unit 22 divides each of the N frames (consecutive N frames) supplied from the image storage unit 21 into blocks of a certain size (for example, 8 × 8, 16 × 16) and moves them. It outputs to the quantity detection part 12 (block distribution part 32). The block dividing unit 22 also converts each block of the Pth frame stored in the image storage unit 21 (hereinafter referred to as the Pth frame) among the N frames to the movement amount detection unit 12 (movement amount detection). Part 31). The Pth frame is a different frame from the Mth frame.

  Next, the movement amount detection unit 12 will be described. The movement amount detection unit 31 of the movement amount detection unit 12 uses the motion vector of each block of the Pth frame supplied from the block division unit 22 of the block division unit 11 as the Mth frame supplied from the image storage unit 21. For example, the block matching process between the frames is detected and detected, and the detected motion vector is supplied to the block distributor 32. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and in the vertical direction (Y-axis direction). In addition, the movement amount detection unit 31 may be configured to enlarge the image in order to improve the detection accuracy of the movement amount and detect the movement amount using the enlarged image.

  The block distribution unit 32 of the movement amount detection unit 12 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 22 in units of N, and the movement amount detection unit From 31, the amount of movement of the block of the P-th frame among the N blocks is supplied. The block distribution unit 32 supplies the supplied N blocks and the movement amount to any of the block processing units 51 to 53 that perform processing corresponding to the movement amount of the block processing unit 13.

  Specifically, the block distribution unit 32 is supplied from the movement amount detection unit 31 and has a movement amount in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) between one frame of 2 pixels (pixels) or more. In some cases, the N blocks supplied from the block dividing unit 22 and the movement amount supplied from the movement amount detection unit 31 are output to the block processing unit 51. If the amount of movement in the horizontal direction and the vertical direction between one frame is less than 2 pixels and more than 1 pixel, the block distribution unit 32 outputs N blocks and the amount of movement to the block processing unit 53. . When the movement amount is other than that, the block distribution unit 32 supplies the N blocks and the movement amount to the block processing unit 52.

  That is, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs block processing for performing processing for converting image data according to the frame rate and spatial resolution. The block images are distributed to the units 51 to 53.

  Next, details of the block processing unit 13 will be described. The block processing unit 13 is composed of the three block processing units 51 to 53 as described above. The block processing unit 51 supplies a total of N blocks (horizontal or vertical) at the same position in each of consecutive N (for example, N = 4) frames supplied from the block distribution unit 32 of the movement amount detection unit 12. Similarly, a process of thinning out the number of pixels according to the amount of movement supplied from the block distributor 32 (spatial direction thinning process) is performed on N blocks in the case where the amount of movement in the direction is 2 pixels or more. .

Specifically, when the amount of horizontal movement between one frame is 2 pixels (pixels) or more, the block processing unit 51 determines that the block to be processed is 4 × 4 pixels as shown in FIG. When configured, only one pixel value among the four pixels in the horizontal direction is selected as a representative value. In the example of FIG. 3 _(b), to enable only _{P 10} of four pixels _{P 00} through _{P 30} as a representative value (sample point). Other pixel values are invalid. Similarly, for four pixels P _{01 to} P ₃₁ , P ₁₁ is a representative value (sample point), for four pixels P _{02 to} P ₃₂ , P ₁₂ is a representative value (sample point), and P representative value _{P 13} are selected from four pixels ₀₃ through _{P 33} as (sample point).

When the amount of movement in the vertical direction between one frame is 2 pixels or more, the block processing unit 51 has four pixels in the vertical direction when the block is composed of 4 × 4 pixels as shown in FIG. One pixel value is valid as a sample point. In the example of FIG. 3 _(c), to enable as a sample point only _{P 01} of four pixels _{P 00} to _{P 03.} Other pixel values are invalid. _Similarly, the sample point _{P 11} are selected from four pixels _{P 10} to _{P 13,} and the sampling point _{P 21} are selected from four pixels _{P 20} to _{P 23,} the four pixels _{P 30} through _{P 33} It is for the sampling point _{P 31.}

  The block processing unit 51 performs such spatial thinning processing on a total of N = 4 blocks at the same position in each of the supplied consecutive N (for example, N = 4) frames. The data amount of each block is reduced to ¼, and the data amount of all four blocks is reduced to ¼. The block processing unit 51 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  Next, processing executed by the block processing unit 52 shown in FIG. 2 will be described. The block processing unit 52 shown in FIG. 2 has a total of N blocks (the amount of movement in both the horizontal direction and the vertical direction is the same) in each of consecutive N frames supplied from the block distribution unit 32 of the movement amount detection unit 12. A process of thinning out the number of frames (time direction thinning process) is performed on N blocks in the case of less than one pixel.

  Specifically, as shown in FIG. 4, the block processing unit 52 converts four blocks Bi at the same position in four consecutive frames F1 to F4 into one block (in this example, Then, the number of frames to be converted into the block Bi) of the frame F1 is thinned out (thinning of the number of frames between four frames). The block processing unit 52 supplies the output unit 14 with data (one block) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing. Pixel data of one selected block becomes sample point data corresponding to 4 frames.

  The block processing unit 53 is supplied from the block distribution unit 32 of the movement amount detection unit 12 and has a total of N blocks (the movement amount in the horizontal direction and the vertical direction is 1 in each of the consecutive N frames). The number of pixels is thinned out (spatial direction thinning process) and the number of frames is thinned out (time direction thinning process) for each of N blocks in the case of more than pixels and less than 2 pixels.

Unlike the thinning-out process in the block processing unit 51, the block processing unit 53, when the amount of horizontal movement between one frame is 1 pixel or more and less than 2 pixels as shown in FIG. When the block to be processed is composed of 4 × 4 pixels as shown in FIG. 5A, only two pixel values of the four pixels in the horizontal direction are selected as representative values. In the example of FIG. 5B, only P ₀₀ and P ₂₀ out of the four pixels P _{00 to} P ₃₀ are validated as representative values (sample points). Other pixel values are invalid. Similarly, only P ₀₁ and P ₂₁ are representative values (sample points) for the four pixels P _{01 to} P ₃₁ , and only P ₀₂ and P ₂₂ are the representative values for the four pixels P _{02 to} P _32. (Sample points), and for the four pixels P _{03 to} P ₃₃ , only P ₀₃ and P ₂₃ are representative values (sample points).

When the amount of vertical movement between one frame is 1 pixel or more and less than 2 pixels, the block processing unit 53 includes 4 × 4 pixels as a processing target block as shown in FIG. At this time, two pixel values of the four pixels in the vertical direction are validated. In the example of FIG. 5C, P ₀₀ and P ₀₂ out of the four pixels P _{00 to} P ₀₃ are validated as representative values (sample points). Other pixel values are invalid. Similarly, P ₁₀ and P ₁₂ are representative values (sample points) for the four pixels P _{10 to} P ₁₃ , and P ₂₀ and P ₂₂ are representative values (sample points) for the four pixels P _{20 to} P _23. and sample _point), with respect to the four pixels _{P 30} to _{P 33} to _{P 30} and _{P 32} as representative values (sample points).

  Furthermore, the block processing unit 53 performs a frame number thinning process. Specifically, the block processing unit 53 performs a thinning process for the number of frames that enable two of the four blocks F1 to F4 in the same position on the four blocks at the same position. To do. In the thinning-out process for the number of frames, the block processing unit 53 is different from the thinning-out process in the block processing unit 52, as shown in FIG. 6, at the same position in each of the four consecutive frames F1 to F4. Decreasing the number of frames to make a total of four blocks Bi any one of them (two blocks of frames F1 and F3 in the example in the figure) (decimating the number of frames between two frames) Do. The pixel data of the two selected blocks becomes sample point data corresponding to 4 frames. In this case, in sampling in the spatial direction described with reference to FIG. 5, eight sampling points have already been selected for one block, and a total of 16 sampling points are selected from two blocks, which correspond to four frames. Is set as sample point data.

  The block processing unit 53 performs the spatial direction decimation process for reducing the data amount described with reference to FIG. 5 to 1/2 and the time direction decimation process for reducing the data amount described with reference to FIG. Since it is applied to the four supplied blocks, the data amount of the four blocks is reduced to (1/2) × (1/2) = 1/4 as a result. The block processing unit 53 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  The output unit 14 generates stream data from the data for the N blocks with the reduced data amount supplied from each of the block processing units 51 to 53 of the block processing unit 13.

  A specific processing example in the moving image conversion apparatus 10 will be described with reference to FIG. FIG. 7 shows five consecutive frames F1 to F5 processed in the moving image conversion apparatus 10, and represents the arrangement of valid data selected from each frame. The valid data selected in the moving image conversion apparatus 10 is a pixel portion other than the blank shown in FIG. 7, and the blank portion indicates an invalid pixel portion not selected in the moving image conversion apparatus 10.

  Each of the frames F1 to F5 is a frame at the same position in time series, and has 4 × 4 pixel blocks 101 to 116, respectively.

  Among the blocks 101 to 116, the blocks 101 and 102 are blocks that are subjected to spatial direction thinning processing in the horizontal direction by the block processing unit 51. Blocks 111 and 112 are blocks on which spatial direction thinning processing is performed in the vertical direction by the block processing unit 51. Blocks 103 and 104 are blocks that are subjected to spatial direction thinning processing and time direction thinning processing in the horizontal direction by the block processing unit 53. Blocks 113 and 114 are blocks that are subjected to spatial direction thinning processing and time direction thinning processing in the vertical direction by the block processing unit 53. Blocks 105, 106, 115, and 116 are blocks that are subjected to time direction thinning processing by the block processing unit 52.

  In the first frame F1, the blocks 101 and 102 are thinned by 1/4 in the horizontal direction by the block processing unit 51, and the effective data becomes 4 pixels each. The blocks 111 and 112 are thinned out to ¼ in the vertical direction by the block processing unit 51, and the effective data becomes 4 pixels each. The blocks 103 and 104 are thinned by half in the horizontal direction by the block processing unit 53, and the effective data becomes 8 pixels each. The blocks 113 and 114 are thinned by half in the vertical direction by the block processing unit 53, and the effective data becomes 8 pixels each. The blocks 105, 106, 115, and 116 are thinned out in the time direction by the block processing unit 52, but all pixels become effective data in the first frame.

  In the second frame F2, the blocks 101 and 102 and the blocks 111 and 112 each have four pixels as valid data by the same processing as the frame F1. The blocks 103 and 104 and the blocks 113 and 114 are thinned in the time direction by the block processing unit 53, and all the pixels become invalid. Blocks 105, 106, 115, and 116 are thinned out in the time direction by the block processing unit 52, and all pixels become invalid.

  In the third frame F3, the blocks 101 and 102 and the blocks 111 and 112 each have four pixels as effective data by the same processing as the frame F1. The blocks 103 and 104 and the blocks 113 and 114 become effective frames, and 8 pixels become effective data by the same processing as the frame F1. Blocks 105, 106, 115, and 116 are thinned out in the time direction by the block processing unit 52, and all pixels become invalid.

  In the fourth frame F4, the blocks 101 and 102 and the blocks 111 and 112 each have four pixels as valid data by the same processing as the frame F1. The blocks 103 and 104 and the blocks 113 and 114 are thinned in the time direction by the block processing unit 53, and all the pixels become invalid. Blocks 105, 106, 115, and 116 are thinned out in the time direction by the block processing unit 52, and all pixels become invalid.

  Since N = 4 here, the fifth frame F5 has the same data arrangement as the first frame F1. In this way, the data of each block is subjected to the optimum thinning process according to the moving speed information, and finally a data amount reduction of 1/4 is realized.

  As described above, the moving image conversion apparatus 10 shown in FIG. 1 performs processing for converting an input moving image into a moving image (compressed data) with a reduced data amount. This is a device that prevents the observer from perceiving image quality degradation due to a reduction in the amount of data by performing moving image conversion processing using the super-resolution effect realized based on the visual characteristics of the image.

  Specifically, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs image data conversion according to the optimal frame rate and spatial resolution. It supplies to the block processing parts 51-53 to perform, and it is set as the structure which performs the data conversion process of a different aspect in each block processing part 51-53, and makes an observer perceive image quality degradation by this structure. It realizes a moving image conversion process without any problem. The super-resolution effect is a visual effect in which the observer perceives a resolution higher than the number of display pixels when the observer follows the moving subject discretely sampled in the spatial direction. . This is due to the temporal integration function in human vision, and the moving image conversion apparatus 10 shown in FIG. 1 is configured to perform a moving image conversion process using the super-resolution effect caused by the temporal integration function. Have.

  The principles and explanations regarding human visual characteristics and super-resolution effect are described in detail in Japanese Patent Application No. 2003-412501. An outline of conditions for generating the super-resolution effect described in Japanese Patent Application No. 2003-412501 will be described below.

In order for the super-resolution effect to occur when the number of pixels of the thinning amount m (pixel) is thinned, it is necessary to cancel all the primary to m−1 order folding components due to the thinning. k (= 1, 2,..., m−1) The condition that the next folding component is canceled is to satisfy the following expressions (Expression 1) and (Expression 2).

In the above equation, φ _t is a sampling position shift amount in thinning out the number of pixels, time t (= 0, 1T, 2T,...), Signal moving speed v, time interval (reciprocal of frame rate) T , Is a value defined by the following formula (formula 3).

  If the above equations (Equation 1) and (Equation 2) are satisfied under the conditions of the thinning amount m and the small region movement amount v, the super-resolution effect occurs, and the deterioration of the image quality is hardly perceived by the observer.

  This is because when the display is performed at the same frame rate as the frame rate of the input moving image, the block thinned out in the spatial direction is added to the previous and next frame images by the time integration function of human vision. This is to perceive an image equivalent to the original image.

  However, when displaying at a frame rate lower than the frame rate of the input moving image (slow motion display) or continuing to display the same frame for a certain period of time (pause), the addition with the previous and next frame images is established. There was a problem that jaggy (return of the spatial frequency) due to the reduction of the data amount could be seen.

  In order to solve this, a method of interpolating the compressed image subjected to the spatial thinning process with a linear or non-linear filter can be considered. However, this method has a problem that high-frequency components contained in the original moving image cannot be restored, resulting in a blurred image (deterioration of spatial frequency). Hereinafter, a configuration that enables the same moving image display as the original moving image even when the problem is solved and displayed at a frame rate lower than the frame rate of the input moving image (slow motion display) will be described.

[(2) Moving Image Conversion Device and Moving Image Conversion Method for Generating High Quality Restored Image]
As described above, image quality degradation may be conspicuous in restoring a compressed image that has undergone spatial thinning processing. In particular, in slow motion display in which the frame rate is lower than the frame rate of the input moving image, there is a problem that jaggy (return of spatial frequency) due to data amount reduction is visible. Below, the structure which solves this problem is demonstrated.

  First, the principle of restoring an original image from a moving image in which information is lost by the spatial direction thinning process will be described. FIG. 8 shows images S1 to S4 obtained by performing 1/4 spatial thinning processing in the horizontal direction on the original images I1 to I4 obtained by photographing the subject (car) moving in the right direction. In the frame F1, the original image I1 is an image obtained by photographing a subject moving in the right direction.

  A condition in which the moving speed of the subject (car) is distributed to the block processing unit 51 (spatial direction thinning process) in the block distribution unit 32 of the moving image conversion apparatus shown in FIG. 2, that is, the movement amount per frame is predetermined. When the condition is met, the block processing unit 51 executes a spatial direction thinning process. As a result, the original image I1 is thinned out to 1/4 in the horizontal direction at the sampling point indicated by the arrow in FIG. Here, the position of the sampling point (corresponding to the representative pixel (sample point) in FIG. 3) in the thinning-out process is fixed with respect to the frame. The thinned image S1 has a data amount of ¼ that of the original image I1.

  In the frame F2, in the original image I2, the subject moves to the right by the moving speed v (pixel / frame) with respect to the original image I1. In this state, the thinning process is performed in the same manner as the frame F1 to obtain a thinned image S2. At this time, since the position of the sampling point (sample point) is the same as that of the frame F1, the thinned image S2 is different from the thinned image S1.

  Similarly, thinning processing is performed on the original images I3 and I4 in the frames F3 and F4 to obtain thinned images S3 and S4. In this example, it is assumed that v = 1 (pixel / frame), and a super-resolution effect can be obtained in the 1/4 thinning process during normal display.

  The thinned images S1 to S4 are images that have been thinned at sampling points (sample points) at the same position. If the moving speed v of the subject is not an integral multiple of the thinning amount (4 in this embodiment), The images are sampled at different positions. Accordingly, the thinned images S1 to S4 lack information from the original image, and each has different information. The original image can be restored by adding the plurality of images together while aligning the positions based on the moving speed information.

  FIG. 9 shows an image obtained by moving the thinned images S1 to S4 corresponding to each frame according to the moving speed v. In this example, since v = 1 (rightward), the thinned image S2 is one pixel to the right, the thinned image S3 is two pixels to the right, and the thinned image S4 is right to the thinned image S1. Are shifted by 3 pixels. In order to align them, the thinned image S2 is moved 1 pixel to the left, the thinned image S3 is moved 2 pixels to the left, and the thinned image S4 is moved 3 pixels to the left (FIG. 9). As a result, the subject positions of the thinned images S1 to S4 are the same, and the sampling positions are different. When these images are added together, a restored image R1 is obtained.

In this example, since the moving speed v = 1 and the spatial thinning process of 1/4 is performed in the horizontal direction, it is possible to completely restore the original image by adding four frames of images. At this time, in order for the original image to be completely restored, the following conditions must be satisfied.
(Condition 1) The shape and color of the subject do not change over the integrated frame.
(Condition 2) The moving speed of the subject is known.

  Further, in addition to the above conditions 1 and 2, the condition 3 satisfies the condition that the aliasing of the spatial frequency caused by the thinning process is completely canceled by the integration process due to the relationship between the moving speed v, the thinning amount, and the number of integration frames. It is necessary. That is, it is necessary to satisfy the above-described requirements for generating the super-resolution effect. In order to generate the super-resolution effect when the number of pixels of the thinning-out amount m (pixel) is thinned, it is necessary to cancel all the primary to m−1-order folding components due to the thinning-out. k (= 1, 2,..., m−1) The conditions for canceling the next folding component must satisfy the above-described expressions (Expression 1) and (Expression 2).

  Actually, it is very difficult to satisfy the above conditions in the input moving image. In particular, simultaneous equations must be solved in order to derive a condition in which the aliasing component is completely canceled. Even if a condition for canceling the aliasing component is obtained, the number of frames that can be integrated is limited in implementation.

  The image conversion apparatus according to the present invention enables high-quality data restoration regardless of whether or not the above conditions are satisfied, and performs image restoration using a realistic circuit scale, and cannot be restored. Realizes image generation without failure by performing interpolation processing.

  Hereinafter, a moving image conversion apparatus according to the present invention, that is, an apparatus that restores compressed image data that has undergone pixel thinning processing, for example, in slow motion display that displays at a frame rate lower than the frame rate of an input moving image. An apparatus capable of reducing the occurrence of jaggy (spatial frequency aliasing) due to data amount reduction and restoring a high-quality moving image will be described. FIG. 10 shows a configuration of the moving image conversion apparatus 120 in the present embodiment.

  The input unit 121 of the moving image conversion apparatus 120 inputs a data stream including compressed data generated by, for example, the moving image conversion apparatus 10 of FIG. The input unit 121 outputs converted image data corresponding to blocks as compressed data included in the input data stream to the addition processing units 122-1 to 122-n. This compressed data is compressed data including sample point data selected by thinning in the spatial direction or the time direction. In addition, the block attribute data (decimation method, thinning direction, etc.) and motion vector data included in the data stream are output to the address conversion unit 126. When no motion vector data exists in the data stream generated by the moving image conversion apparatus 10, the input unit 121 or the address conversion unit 126 performs motion vector detection using the image data included in the data stream, A process using the detected motion vector is executed. When motion vector detection is performed in the address conversion unit 126, compressed image data is input to the address conversion unit 126, and motion vector detection based on the compressed image data is executed.

  The addition processing units 122-1 to 122-n receive image data of the current frame including compressed data corresponding to a plurality of blocks given from the input unit 121, and further, in the preceding process, the frame buffers 1 to n (123-1). Through 123-n), the data stored in the current frame is read and the pixel value of the image data of the current frame and the buffer read value are added and averaged. The data stored in the frame buffers 1 to n (123-1 to 123-n) is the pixel value of the sample point in the preceding frame or the addition average value of the pixel values of the sample points in a plurality of frames. Details of these processes will be described later.

  Frame buffers 1 to n (123-1 to 123-n) are memories that temporarily store image data in past frames after alignment, and are twice as long as the original image (image before thinning processing). It has a large number of pixels. The frame buffer number n corresponds to the number of frames used for image restoration. When n is small, the number of areas that cannot be restored increases. When n is large, the areas that cannot be restored decrease, but there is a problem that errors due to changes in the moving speed of the subject and changes in form increase. For example, n = 8.

  The address conversion unit 126 receives block attribute data (a thinning processing method, a thinning direction, etc.) and motion vector data given from the input unit 121, and obtains addresses for the frame buffers 123-1 to 123-n for each block.

  The selection unit 127 selects any one of the outputs of the frame buffers FB1 to n (123-1 to n) and outputs the selected buffer output to the interpolation unit 124. The frame buffer selected at this time is a frame buffer selected corresponding to the restoration processing frame.

  During the restoration process of the first frame (F1), the selection unit 127 selects the frame buffer FB1 (123-1) as an output. At this time, the frame buffers FB2 (123-2) to FBn (123-n) correspond to +1 frame (second frame) to + (n-1) frames.

  During the subsequent restoration process of the second frame (F2), the selection unit 127 is switched to output the frame buffer FB2 (123-2). At this time, the frame buffers FB3 (123-3) to FBn (123-n) sequentially shift to correspond to +1 frame to (third frame) + (n-2) frames. The frame buffer FB1 (123-1) is set to correspond to + (n-1) frames.

  Similarly, the selection unit 127 switches to output the frame buffer FB3 during the restoration process of the third frame (F3), and the frame buffer FB4 in the fourth frame (F4) and the frame buffer in the fifth frame (F5). FB5 is switched to output each.

  Thus, the selection unit 127 cyclically switches the frame buffers FB1 to n (123-1 to n) to the interpolation unit 124 and outputs them.

Table 1 shows the correspondence between the frame output to the interpolation unit 124 and restored, and the frame data stored in the frame buffers FB1 to n (1233-1 to n) at that time.

  In Table 1 above, for example, when the processing frame is the first frame (F1), the current frame, that is, the data output to the interpolation unit 124 is the data of the frame buffer FB1 (123-1). The frame buffer FB2 (123-2) to the frame buffer FBn (123-n) indicate that the data of the second to (n-1) th frames are stored. When the processing frame is the second frame (F2), the current frame, that is, the data output to the interpolation unit 124 is the data of the frame buffer FB2 (123-2). At this time, the frame buffer FB3 (123-123) 3) to the frame buffer FB1 (123-1) indicate that each data from the third frame is stored.

  In this manner, the selection unit 127 cyclically switches and outputs the frame buffers FB1 to n (123-1 to n) to the interpolation unit 124, and the interpolation unit 124 performs interpolation processing of each frame. Perform frame data restoration.

  In this example, the selection unit 127 is provided, and the selection unit cyclically switches the frame buffers FB1 to n (123-1 to n) and outputs them to the interpolation unit 124. However, the frame buffers FB1 to n The stored data of (123-1 to n) may be cyclically shifted, and only a specific frame buffer may be positioned as an output buffer.

  The interpolation unit 124 generates a restoration block based on the sample point data of a plurality of different frames based on the data written in the frame buffer selected by the selection unit 127. Further, a pixel portion (pixel missing) that cannot be restored only by the sample point data is interpolated from surrounding pixels. The interpolation method will be described later. Further, the interpolation unit 124 converts the image data twice as long as the length and width of the original image stored in the frame buffer into the same number of pixels as the original image by the reduction process, and outputs the same.

  The output unit 125 converts the image data output from the interpolation unit 124 into an image signal format that can be received by the display device, and outputs the image signal format. Furthermore, there is a control unit (not shown), and the control unit controls each unit in accordance with a computer program that records a processing sequence, for example.

  The operation of the moving image conversion apparatus 120 in the present embodiment will be described in detail with reference to the flowchart of FIG. 11 and FIGS.

  In step S1, the moving image conversion apparatus 120 resets, that is, initializes the frame buffers 1 to n (123-1 to 123-n). At the same time, the selection unit 127 is initialized, and the frame buffer as the output frame is set to FB1 (123-1).

  In step S2, the control unit initializes the block number representing the processing target block of the frame image to zero. The block number is a sequence number of 0, 1, 2,... Set for the blocks constituting one frame.

  In step S3, the input unit 121 outputs converted image data corresponding to one block of the processing frame to the addition processing units 122-1 to 122-n. In addition, block attribute data (a thinning processing method, a thinning direction, etc.) and motion vector data are output to the address conversion unit 126.

  In step S4, the address conversion unit 126 receives block-corresponding attribute data and motion vector data, and includes attribute data including decimation processing mode information executed for the block, and the movement amount and movement direction of the block per frame. Addresses for the frame buffers 1 to n (123-1 to 123-n) are obtained based on the motion vector data having information.

  The address obtained by the address conversion unit 126 is an address indicating a writing destination in each frame buffer 1 to n (123-1 to 123-n) of the pixel value of the effective pixel of the processing target block, and the effective pixel of the processing target block The pixel position corresponds to the position in the current frame and the pixel position of the effective pixel of the processing target block in the (n−1) future frames.

  It is assumed that the block moves corresponding to the motion vector v every time one frame advances, and the address conversion unit 126 sets the write address of the pixel position of the effective pixel of the processing target block for each buffer as the motion vector. It is calculated as a position where only pixels corresponding to v are sequentially moved.

  For example, the motion vector v indicates the number of motion pixels per frame. That is, it has a unit of (pixel / frame). The address conversion unit 126 uses the current address (initially the frame buffer 1 (123-1)) set corresponding to the processing target frame as the write destination address of the pixel value of the effective pixel of the processing target block. The position of the frame buffer 1 (123-1) corresponding to the position is calculated.

  Furthermore, the write address for the current frame (initially frame buffer 1 (123-1)) is used as the write address of the frame buffer (initially frame buffer 2 (123-2)) for writing frame data that has progressed by one frame. To which the motion vector v is added. Furthermore, the motion vector v is further added to the write address of the frame buffer (initially frame buffer 3 (123-3)) for writing the frame data progressed by two frames to the address for the frame buffer 2 (123-2). To calculate. Thereafter, the same address calculation is performed to obtain addresses for all the frame buffers 1 to n (123-1 to 123-n). A specific processing example will be described later.

  In step S5, the addition processing units 122-1 to 122-n access the buffers based on the addresses corresponding to the frame buffers 1 to n (123-1 to 123-n) calculated by the address conversion unit 126. Then, the value written in the address designation position of the frame buffers 123-1 to 123-n is read. In step S <b> 6, pixel value addition averaging processing is performed between the pixel values read from the frame buffers 123-1 to 123-n and the effective pixel value given from the input unit 121. The value written in the address designation position of the frame buffers 123-1 to 123-n is the effective pixel value data set in the corresponding block of the past frame written at the time of processing of the already processed frame or the average value thereof. is there. If the data written at the address given from the address conversion unit 126 does not exist in the buffer, the processing in steps S5 and S6 may be omitted.

  In step S7, under the control of the control unit, the pixel values subjected to the addition averaging process by the addition processing units 122-1 to 122-n are given from the address conversion unit 126 of the frame buffers 123-1 to 123-n. The process of writing to the location accessed by is executed. If there is no written data at the address given from the address conversion unit 126, only the effective pixel value data given from the input unit 121 is written, and the written data is written to the address given from the address conversion unit 126. If data exists, writing of the average value calculated between the value and the effective pixel value given from the input unit 121 is executed.

  In step S8, it is determined whether or not the block to be processed is the last block of the current frame image. If it is not the last block, the process proceeds to step S9.

  In step S9, the block number is incremented by 1, and the process proceeds to data input in step S3.

  The processing from step S3 to step S8 is repeatedly executed, and if it is determined in step S8 that the block is the last block, the process proceeds to step S10.

  In step S10, the frame buffer storage data applied to the restoration of the restoration target frame is output to the interpolation unit 124 via the selection unit 127, and the interpolation unit 124 includes the block configuration data included in the frame buffer storage data. A process of interpolating the missing pixel values based on the surrounding pixel values is executed. Further, the interpolation unit 124 generates an image having the same number of pixels as that of the original image by the reduction process because the frame buffer has twice as many pixels as the original image.

  In step S <b> 11, the output unit 125 converts the frame image data interpolated and reduced by the processing of the interpolation unit 124 into a predetermined output image signal format and outputs it.

  In step S12, it is determined whether the current frame is the last frame. If it is not the last frame, the process proceeds to step S13. In step S13, the selection unit 127 is controlled to switch the output frame buffer. In step S14, the frame buffer selected in the selection unit 127 at the time of the previous frame is initialized to zero.

  For example, referring to Table 1, when the processing frame is switched from the frame F1 to the frame F2, in step S13, the selection unit 127 uses the frame buffer FB2 (123) as the frame buffer (current frame) to be output to the interpolation unit 124. -2) is performed, and in step S14, the frame buffer selected by the selection unit 127 at the time of the previous frame, that is, the frame buffer FB1 (123-1) is reset and initialized. The frame buffer FB1 (123-1) is reset to a state in which no write data exists at the start of processing of the frame F2.

  Returning to step S12, if the current frame is the last frame, the process ends.

  Here, the operation of the address conversion unit 126 and the frame buffer writing process will be described in detail with reference to FIGS. In the example described below, a processing example corresponding to a motion vector only in the horizontal direction will be described for ease of understanding. However, in actual moving image processing, the following processing is also performed for a motion vector in the vertical direction. The same processing as described in (1) is possible. Further, by executing two-dimensional processing by applying both horizontal and vertical motion vectors, processing corresponding to all motion vectors in the two-dimensional direction is possible.

First, referring to FIGS. 12 to 16, the motion vector _{v 0} is the block data which is 2.5 (pixel / frame) in the right direction, illustrating an example of processing for restoring the image conversion device 120 shown in FIG. 10 . 12 to 16 are diagrams illustrating the transition of data stored in the frame buffer FB1 (123-1) to the frame buffer FBn (123-n) during the restoration process.

  12 to 16 are processing examples when the restoration target block is a block that has been subjected to the spatial direction thinning process in the horizontal direction (corresponding to the blocks 101 and 102 in FIG. 7). Here, it is assumed that each frame buffer is associated as shown in Table 1.

12 to 16, two adjacent blocks of each frame in which the data shown in FIG. 12A is given from the input unit 121, first block = (x0, y0) to (x3, y3), second block = Compressed image data consisting of sample point data of (x4, y0) to (x7, y7),
FIG. 12 = first frame,
FIG. 13 = second frame,
FIG. 14 = third frame,
FIG. 15 = fourth frame,
FIG. 16 = Fifth frame
Is shown.
Each is subjected to spatial direction thinning processing, and is set as compressed data in which the number of pixels in the horizontal direction is ¼ of the 4 × 4 pixel block in the same manner as the blocks 101 and 102 in FIG. 7. Yes. In each frame, the effective pixel (sample point) is (x1, y0), (x1, y1), (x1, y2), (x1, y3) in the first block, and the effective pixel is (x5) in the second block. , Y0), (x5, y1), (x5, y2), (x5, y3).

  FIG. 12 shows processing in the first frame (referred to as the first frame) after the frame buffer is initialized in step S1. FIG. 12A shows compressed image data composed of sample point data of two adjacent blocks of the first frame given from the input unit 121, and these blocks are subjected to spatial direction thinning processing. In the same manner as the blocks 101 and 102 in FIG. 7, the number of pixels in the horizontal direction is ¼ of the 4 × 4 pixel block, and only four pixels are valid in each of the 16 pixels in one block. ing.

  FIG. 12B shows frame buffer data for the current frame (first frame). In the moving image conversion apparatus according to the present invention, alignment is performed for each frame based on the motion vector data, and the effective pixel value is written in the frame buffer. Here, the frame buffer has twice as many pixels as the original image as described above. Since the block position is the same as the position of the input image in the current frame, the pixel value is written at an address obtained by doubling the effective pixel address of the input image data both vertically and horizontally.

  That is, the effective pixel addresses of the input image data shown in FIG. 12A are (x1, y0), (x1, y1), (x1, y2), (x1, y3), and (x5, y0), ( x5, y1), (x5, y2), and (x5, y3), these addresses are doubled to (2 × x1, 2 × y0), (2 × x1, 2 × y1), ( 2 × x1, 2 × y2), (2 × x1, 2 × y3), and (2 × x5, 2 × y0), (2 × x5, 2 × y1), (2 × x5, 2 × y2), The corresponding pixel value of the effective pixel of the input image data shown in FIG. 12A is written in (2 × x5, 2 × y3).

FIG. 12C shows data corresponding to [+1 frame (second frame)] calculated by moving the motion data v ₀ for the write data in the frame buffer of FIG. The data written in the frame buffer FB2 (123-2) is shown.

In the example of FIG. 12, the motion vector _{v 0} is the right direction 2.5 (pixel / frame). Therefore, the movement of the motion vector is performed on the data of the first frame, and the corresponding pixel value of the effective pixel of the input image data shown in FIG. 12A is written at this position.

The position of the block in the +1 frame is the position moved from the block position in the current frame to the right of the magnitude of the motion vector v ₀ = 2.5 pixels, but each frame buffer is set to a size that is twice as long as the length and width. Therefore, the corresponding pixel value of the effective pixel of the input image data shown in FIG. 12A is written at the position moved to the right by 2.5 × 2 = 5 pixels. Address conversion unit 126 determines on the basis of the magnitude and direction of the vector v ₀ motion address of the write position, and writes the corresponding pixel value of the effective pixels of the input image data shown in FIG. 12 (a) to determine position.

  The data write address of the frame buffer FB2 (123-2) in FIG. 12C is set as an address (+5) in the x component of the write address of the frame buffer FB1 (123-1).

FIG. 12D shows data in the frame buffer FB3 (123-3) for +2 frame (third frame). Position of the block after the 2 frames, the block position in the current frame, the motion vector v ₀ × 2, i.e. a position shifted by 2v ₀ in the right direction. In the address buffer 126, the data write address in the frame buffer FB3 (123-3) is moved to the right by 2v ₀ from the address of the frame buffer in the current frame, that is, in the frame buffer FB2 (123-2) in the +1 frame. It is calculated as a position further shifted by 5 pixels in the right direction from the write address, and according to this address, the corresponding pixel value of the effective pixel of the input image data shown in FIG. 12A is written into the frame buffer FB3 (123-3). .

FIG. 12E shows data in the frame buffer FBn (123-n) for + (n-1) frames (nth frame). The position of the block after n−1 frames is a position moved from the block position in the current frame by a motion vector v ₀ × (n−1), that is, (n−1) v _{0 in the} right direction. The data write address in the frame buffer FBn (123-n) is calculated by the address conversion unit 126 as a position moved (n−1) × 5 pixels rightward from the address of the frame buffer in the current frame. The corresponding pixel value of the effective pixel of the input image data shown in FIG. 12A is written into the frame buffer FBn (123-n).

  FIG. 13 shows processing in the second frame. FIG. 13A shows the image data of two adjacent blocks of the second frame given from the input unit 121, and is the spatial thinning data in the horizontal direction as in FIG. Each of the four pixels is effective.

  FIG. 13B shows frame buffer data at the time of processing the current frame (second frame). At the time of inputting the second frame, as described above with reference to Table 1, the storage buffers of the current frame to + (n-1) frames are sequentially shifted. Therefore, FIG. 13B shows the data contents of the current frame buffer FB2 (123-2) at the time of processing the current frame (second frame). In the frame buffers FB2 to FBn, the data corresponding to the first frame written in the frame buffers FB2 to FBn in the process of FIG. 12 is already written.

  In the frame buffer FB2 (123-2) shown in FIG. 13B, the data written in the frame buffer FB2 in FIG. 12C has been written. That is, the data corresponding to the first frame written to the frame buffer FB2 based on the motion vector information at the time of processing the first frame is written, and the data writing of the current frame (second frame) is performed as an additional process. Is executed.

  In step S4 in the flow of FIG. 11, the address conversion unit 126 obtains a write address for the current block in the current frame, and in step S5, the data at the address in the frame buffer is read. The effective pixel storage position is written at a position different from the write address, and there is no valid data at the write address position. Therefore, in this case, the averaging process is not executed, and the effective pixel value of the processing block of the current frame (second frame) is written at the address obtained by the address conversion unit 126.

  That is, in the example shown in FIG. 13 (b), the write address for the block of the current frame (second frame) by the address converter 126 is (2x1, 2y0), (2x1, 2y1), (2x1, 2y2),. However, since the effective pixel storage position of the first frame is different from these positions, in step S5, (2x1, 2y0), (2x1, 2y1), (2x1, 2y2) of the frame buffer. )... Is not read out, and the addresses (2 × 1, 2y0), (2 × 1, 2y1), (2 × 1, 2y2),... This is executed as a process in which the effective pixel value of the block is simply written to the block.

FIG. 13C shows data in the frame buffer FB3 (123-3) for +1 frame (third frame). In the frame buffer FB3 (123-3), data after the first frame processing (corresponding to FIG. 12D) is written. Then, as in the case of the first frame, the effective pixel value is written to the address moved to the right by the motion vector v ₀ from the block position of the current frame.

FIG. 13D shows frame buffer data for +2 frames (fourth frame). Similarly, the data after the first frame processing is written here. Then, as in the case of the first frame, and writes the valid pixel value from the block position of the current frame address moved rightward motion vector 2v _0.

  FIG. 13E shows frame buffer data for + (n−1) frames (n + 1th frame). The moving image conversion apparatus has n frame buffers, the frame buffer is shifted after the first frame processing, and the frame buffer for the selected frame at the previous frame is initialized in step S14. That is, no data in the previous frame exists in the nth frame buffer, and only effective pixels in the current frame are always written. In this example, the nth frame buffer is the frame buffer FB1 (123-1).

  FIG. 14 shows processing in the third frame. The third frame is processed in the same manner as the second frame. However, at the time of inputting the image data of the third frame, the first frame and the first frame are stored in the frame buffer other than the reset frame buffer FB2 (123-2). The difference is that two frames of effective pixel data are written.

  For example, the first and second frame-corresponding data corresponding to FIG. 13C have been written in the frame buffer FB3 (123-3) of FIG. 14B, and the frame buffer FB4 of FIG. In (123-4), the first and second frame correspondence data corresponding to FIG. 13D is already written.

  The effective pixel value of the block of the third frame to be processed is written into each frame buffer in which the data corresponding to the first and second frames has been written. Address calculation for each buffer similar to that described with reference to FIG. 12 is performed, and write processing is performed. If write data already exists at the write destination (address designation destination), the addition processing units 122-1 to 12-n corresponding to each buffer add pixel values to and from the effective pixel value provided from the input unit 121. Averaging processing is performed and the average value is written.

  FIG. 15 shows processing for the fourth frame. At the time of inputting the image data of the fourth frame, the effective pixel data of the first frame, the second frame, and the third frame are written in the frame buffers other than the reset frame buffer FB3 (123-3).

  For example, the first, second, and third frame correspondence data corresponding to FIG. 14C has been written in the frame buffer FB4 (123-4) of FIG. 15B, and FIG. In the frame buffer FB5 (123-5), the first, second, and third frame correspondence data corresponding to FIG. 14D has been written.

  After address calculation processing for each frame buffer FB1 to n (123-1 to n) and pixel value addition averaging processing as necessary, the fourth processing is performed on each frame buffer FB1 to n (1233-1 to n). The writing process of the pixel value of the effective pixel set to the block of the frame or the addition average value is executed.

  FIG. 16 shows processing in the fifth frame. At the time of inputting the image data of the fifth frame, the effective pixel data of the first to fourth frames are written in the frame buffers other than the reset frame buffer FB4 (123-4).

  For example, the first, second, third, and fourth frame correspondence data corresponding to FIG. 15C have been written in the frame buffer FB5 (123-5) of FIG. 16B, and FIG. In the frame buffer FB6 (123-6) of c), the first, second, third, and fourth frame correspondence data corresponding to FIG.

  After address calculation processing for each frame buffer FB1 to n (123-1 to n) and pixel value addition averaging processing as necessary, the fifth processing is performed for each frame buffer FB1 to n (1233-1 to n). The writing process of the pixel value of the effective pixel set to the block of the frame or the addition average value is executed.

  As shown in FIG. 16 (b), after the processing of the fifth frame, effective pixel data from the first frame to the fifth frame is written in the frame buffer FB5 (123-5) for the current frame, and compression is performed. It can be seen that the missing pixel data is restored.

Next, with reference to FIG. 17 to FIG. 21, processing for restoring block data whose motion vector v ₁ is 1.5 (pixel / frame) in the right direction in the image conversion apparatus 120 shown in FIG. 10 will be described. . FIGS. 17 to 21 show transitions of data stored in the frame buffer FB1 (123-1) to the frame buffer FBn (123-n).

  FIGS. 17 to 21 show processing when the restoration target block is a block that has been subjected to spatial direction thinning and time direction thinning processing in the horizontal direction (corresponding to blocks 103 and 104 in FIG. 7). Here, it is assumed that each frame buffer is associated as shown in Table 1.

In FIG. 17 to FIG. 21, two adjacent blocks of each frame to which the data shown in each figure (a) is given from the input unit 121, first block = (x8, y0) to (x11, y3), second block = Compressed image data consisting of sample point data of (x12, y0) to (x15, y3),
FIG. 17 = first frame,
FIG. 18 = second frame,
FIG. 19 = third frame,
FIG. 20 = fourth frame,
FIG. 21 = fifth frame,
Is shown.
Each is subjected to a spatial direction thinning process and a time direction thinning process. As a result of thinning out in the time direction, there is no effective pixel (sample point) in the second frame (FIG. 18A) and the fourth frame (FIG. 20A). Further, as a result of the spatial thinning, the first frame (FIG. 17A), the third frame (FIG. 19A), and the fifth frame (FIG. 21A) are the same as the blocks 103 and 104 in FIG. Of the 4 × 4 pixel block, it is set as compressed data in which the number of pixels is halved in the horizontal direction. In each frame, in the first block, effective pixels (sample points) are (x8, y0), (X8, y1), (x8, y2), (x8, y3), (x10, y0), (x10, y1), (x10, y2), (x10, y3), in the second block, the effective pixel is (X12, y0), (x12, y1), (x12, y2), (x12, y3), (x14, y0), (x14, y1), (x14, y2), (x14, y3) Yes.

  FIG. 17 shows processing in the first frame (referred to as the first frame) after the frame buffer is initialized in step S1. FIG. 17A shows compressed image data composed of sample point data of two adjacent blocks given from the input unit 121, and these blocks are subjected to spatial direction thinning and time direction thinning processing. In this case, as shown in FIG. 7, in the first, third, and fifth frames, among the 4 × 4 pixel blocks, the number of pixels in the horizontal direction is halved. Only 8 pixels are valid.

  FIG. 17B shows data in the frame buffer FB1 (FB123-1) at the time of processing the current frame (first frame). In the moving image conversion apparatus according to the present invention, alignment is performed for each frame based on the motion vector data, and the effective pixel value is written in the frame buffer. Here, the frame buffer has twice as many pixels as the original image as described above. Since the block position is the same as the position of the input image in the current frame, the pixel value is written at an address obtained by doubling the effective pixel address of the input image data both vertically and horizontally.

  That is, the effective pixel addresses of the input image data shown in FIG. 17A are (x8, y0), (x8, y1), (x8, y2), (x8, y3), (x10, y0), (x10 , Y1), (x10, y2), (x10, y3), and (x12, y0), (x12, y1), (x12, y2), (x12, y3), (x14, y0), (x14, Since y1), (x14, y2), and (x14, y3), these addresses are doubled to (2 × x8, 2 × y0), (2 × x8, 2 × y1), (2 × x8, 2 × y2), (2 × x8, 2 × y3), (2 × x10, 2 × y0), (2 × x10, 2 × y1), (2 × x10, 2 × y2), (2 × x10, 2 × y3), and (2 × x12, 2 × y0), (2 × x12, 2 × y1), (2 × x12, 2 × y2), (2 × x1) 2, 2 × y3), (2 × x14, 2 × y0), (2 × x14, 2 × y1), (2 × x14, 2 × y2), (2 × x14, 2 × y3) in FIG. The corresponding pixel value of the effective pixel of the input image data shown in a) is written.

FIG. 17 (c) in the frame buffer FB1 [+1 frame (second frame) is calculated by performing a movement of the motion vector _{v 1} minute for write data (123-1) shown in FIG. 17 (b) The corresponding data is data written to the frame buffer FB2 (123-2). In the example of FIG. 17, the motion vector _{v 1} is 1.5 in the right direction (pixel / frame). Therefore, the position of the block in the +1 frame is a position moved by 1.5 pixels in the right direction from the block position in the current frame, and the address in the frame buffer is a position moved by 3 times twice.

FIG. 17D shows frame buffer data for +2 frames (third frame). The position of the block after two frames is a position moved from the block position in the current frame by a motion vector v ₁ × 2, that is, 2v _{1 in the} right direction. Data write address in the frame buffer FB3 (123-3), in the address conversion unit 126, the movement from the address of the frame buffer in the current frame to the right by 2v _1, i.e., the frame buffer FB2 (123-2) in + 1 frame It is calculated as a position shifted three pixels further to the right from the write address, and according to this address, the corresponding pixel value of the effective pixel of the input image data shown in FIG. 17A is written into the frame buffer FB3 (123-3). .

FIG. 17E shows data in the frame buffer FBn (123-n) for + (n-1) frames (nth frame). The position of the block after n-1 frames is a position moved from the block position in the current frame by motion vector v ₁ × (n−1), that is, (n−1) v _{1 in the} right direction. The data write address in the frame buffer FBn (123-n) is calculated by the address conversion unit 126 as a position moved (n−1) × 3 pixels in the right direction from the address of the frame buffer in the current frame. The corresponding pixel value of the effective pixel of the input image data shown in FIG. 17A is written into the frame buffer FBn (123-n).

  FIG. 18 shows processing in the second frame. The example of FIGS. 17 to 21 corresponds to the blocks 103 and 104 in FIG. 7 because the block has been subjected to spatial direction decimation and time direction decimation processing. In the second frame, time direction decimation processing is performed. Therefore, there is no effective pixel (FIG. 18A).

  FIG. 18B shows frame buffer data for the current frame (second frame). At the time of inputting the second frame, as described above with reference to Table 1, the storage buffers of the current frame to + (n-1) frames are sequentially shifted. Accordingly, FIG. 18B shows the data contents of the current frame buffer FB2 (123-2) at the time of processing the current frame (second frame). In the frame buffers FB2 to FBn, the data corresponding to the first frame written in the frame buffers FB2 to FBn in the process of FIG. 17 is already written. Here, since no effective pixel exists in the second frame, no new data is written in the frame buffer.

  FIG. 18C shows data in the frame buffer FB3 (123-3) for +1 frame (third frame). Data after the first frame processing (corresponding to FIG. 17D) is written in the frame buffer FB3 (123-3). Since no effective pixel exists in the second frame, new data is not written.

  FIG. 18D shows data in the frame buffer FB4 (123-4) for +2 frame (fourth frame). Data after the first frame processing is written in the frame buffer FB4 (123-4). Since no effective pixel exists in the second frame, new data is not written.

  FIG. 18E shows data in the frame buffer FB2 (123-2) for + (n-1) frames (n + 1th frame). As described above, since initialization is performed in step S14, no data is written in the frame buffer. Further, since no effective pixel exists in the second frame, new data is not written.

  FIG. 19 shows processing in the third frame. In the third frame, the same input data as in the first frame is given, and the number of pixels in the 4 × 4 pixel block is halved in the horizontal direction, and only 8 pixels are valid. (FIG. 19 (a)).

  In the process of the third frame, the effective pixel value in the third frame is written into the frame buffer in which the data after the second frame process is written. At this time, as described above, since there is no effective pixel in the second frame, the effective pixel value in the first frame and the effective pixel value in the third frame are written in the frame buffer.

  For example, the first frame corresponding data corresponding to FIG. 18C has been written in the frame buffer FB3 (123-3) of FIG. 19B, and the frame buffer FB4 (123-123) of FIG. In 4), the first and second frame correspondence data corresponding to FIG. 18D has been written.

  The effective pixel value of the block of the third frame to be processed is written to each frame buffer in which the first frame correspondence data has been written. Address calculation for each buffer similar to that described with reference to FIG. 12 is performed, and write processing is performed. If write data already exists at the write destination (address designation destination), the addition processing units 122-1 to 12-n corresponding to each buffer add pixel values to and from the effective pixel value provided from the input unit 121. Averaging processing is performed and the average value is written.

  FIG. 20 shows processing in the fourth frame. In the fourth frame, as in the second frame, there is no effective pixel by the time direction thinning process. Therefore, the same processing as in FIG. 18 is performed. That is, since no effective pixel exists in the fourth frame, new data is not written.

  FIG. 21 shows processing in the fifth frame. At the time of inputting the image data of the fifth frame, the effective pixel data of the first and third frames are written in the frame buffer.

  For example, data corresponding to the first and third frames corresponding to FIG. 20C has been written in the frame buffer FB5 (123-5) of FIG. 21B, and the frame buffer FB6 of FIG. In (123-6), the first and third frame correspondence data corresponding to FIG. 20D is already written.

  After address calculation processing for each frame buffer FB1 to n (123-1 to n) and pixel value addition averaging processing as necessary, the fifth processing is performed for each frame buffer FB1 to n (1233-1 to n). The writing process of the pixel value of the effective pixel set to the block of the frame or the addition average value is executed.

FIG. 21B shows frame buffer data for the current frame (fifth frame). As described above, since the data after the fourth frame processing is written in the frame buffer, the effective pixel values of the first frame and the third frame already exist. The address conversion unit 126 obtains the address of the effective pixel of the current frame with respect to the frame buffer, and the control unit reads data from the frame buffer. At this point, the address of x = _{2x 14} there is effective pixel value (4 pixels) of the first frame (Fig. 20 (c)).

Accordingly, the control unit reads the data from the address of x = _{2x 14,} sends the data to the addition unit 122. Adding processing unit 122, data read out from the address of x = 2x ₁₄ received from the control unit, i.e., the effective pixel values in the first frame, and averaging the effective pixel values of the fifth frame received from the input unit, Send it to the controller again. Control unit writes the data received from the addition unit 122 to the address of the frame buffer x = _{2x 14.} In this way, the weighting of the data of the past frame is reduced by performing the averaging process.

Specifically, data read from the address of x = _{2x 14} framebuffer 5 (123-5), i.e., the effective pixel values in the first frame,
Write value of address (2x ₁₄ , 2y ₀ ) = α1,
Write value of address (2x ₁₄ , 2y ₁ ) = α2,
Write value of address (2x ₁₄ , 2y ₂ ) = α3,
Write value of address (2x ₁₄ , 2y ₃ ) = α4,
And the effective pixel value (FIG. 21A) of the fifth frame at the corresponding position received from the input unit is the value of (x ₁₄ , y ₀ ) = β1,
Value of (x ₁₄ , y ₁ ) = β2,
Value of (x ₁₄ , y ₂ ) = β3,
Value of (x ₁₄ , y ₃ ) = β4,
If it was in, the data to be written into the address position x = _{2x 14} framebuffer 5 (123-5),
Write value of the address _{(2x 14, 2y 0) =} (α1 + β1) / 2,
Write value of address (2x ₁₄ , 2y ₁ ) = (α2 + β2) / 2,
Write value of address (2x ₁₄ , 2y ₂ ) = (α3 + β3) / 2,
Write value of address (2x ₁₄ , 2y ₃ ) = (α4 + β4) / 2,
As, in the addition processing unit 122-5, the arithmetic mean value is calculated, x = _{2x 14} address location to the value of the frame buffer FB5 (123-5): (α1 + β1) / 2~ the (α4 + β4) / 2 Write processing is executed.

  In this embodiment, the data written in the frame buffer and the input data are subjected to the simple addition averaging process. However, for example, the weighted addition averaging process may be performed, or the input data is simply overwritten. It does not matter.

  FIG. 21C shows data in the frame buffer FB6 (123-6) for +1 frame (sixth frame), and FIG. 21D shows the frame buffer FB7 (123 in FIG. 21 for +2 frame (seventh frame). The data of -7) is shown. The data corresponding to the first and third frames has already been written in each of these buffers, and data reading according to the address is performed in the same manner as the processing for the frame buffer FB5 (123-5) in FIG. Processing and addition average calculation processing are executed, and data writing is performed.

  FIG. 21E shows frame buffer data of + (n−1) frames (n + 4th frame), and only the fifth frame data is written.

Subsequently, with reference to FIGS. 22 to 26, the motion vector _{v 2} is the block data which is 0.5 (pixel / frame) to the right, a description will be given of a process of restoring the image conversion device 120 shown in FIG. 10 . FIGS. 17 to 21 show transitions of data stored in the frame buffer FB1 (123-1) to the frame buffer FBn (123-n).

  FIGS. 22 to 26 show processing for restoring block data in the image conversion apparatus 120 shown in FIG. 10 when the block is a block that has been subjected to thinning processing in the time direction (corresponding to the blocks 105 and 106 in FIG. 7). explain. 22 to 26 show transitions of data stored in the frame buffer FB1 (123-1) to the frame buffer FBn (123-n). Here, it is assumed that each frame buffer is associated as shown in Table 1.

22 to 26, two adjacent blocks of each frame in which the data shown in each figure (a) is given from the input unit 121, first block = (x16, y0) to (x19, y3), second block = Compressed image data consisting of sample point data of (x20, y0) to (x23, y3),
FIG. 22 = first frame,
FIG. 23 = second frame,
FIG. 24 = third frame,
FIG. 25 = fourth frame,
FIG. 26 = fifth frame,
Is shown.
Each is thinned in the time direction. As a result of thinning out in the time direction, there is no effective pixel (sample point) in the second frame (FIG. 23A) to the fourth frame (FIG. 25A). In the first frame (FIG. 22A) and the fifth frame (FIG. 26A), all the pixels in the block are set as effective pixels.

  FIG. 22 shows processing in the first frame (referred to as the first frame) after the frame buffer is initialized in step S1. FIG. 22A shows compressed image data composed of sample point data of two adjacent blocks given from the input unit 121, and when these blocks have been subjected to time direction decimation processing, FIG. As shown in FIG. 7, all the pixels of the 4 × 4 pixel block in the first frame are valid.

  FIG. 22B shows data in the frame buffer FB1 (FB123-1) at the time of processing the current frame (first frame). As described above, the frame buffer has twice as many pixels as the original image. Since the block position is the same as the position of the input image in the current frame, the pixel value is written at an address obtained by doubling the effective pixel address of the input image data both vertically and horizontally.

FIG. 22C shows data written to the frame buffer FB2 (123-2) for +1 frame (second frame). In the example of FIG. 22, the motion vector _{v 2} is 0.5 (pixel / frame) in the right direction. Therefore, the position of the block in the next frame is a position shifted by 0.5 pixels in the right direction from the block position in the current frame, and the motion vector v ₂ minutes with respect to the write data in the frame buffer FB1 (123-1). The corresponding pixel value of the effective pixel of the input image data shown in FIG. 17A is written in the frame buffer FB2 (123-2) according to the address calculated by performing the (0.5 × 2) pixel movement.

FIG. 22D shows frame buffer data for +2 frames (third frame). The position of the block after two frames is a position moved by 2v _{2 in} the right direction from the block position in the current frame. The address in the frame buffer is a position moved by one pixel further to the right from the address of the frame buffer in the +1 frame. According to this calculated address, the input shown in FIG. 22A is input to the frame buffer FB3 (123-3). The corresponding pixel value of the effective pixel of the image data is written.

FIG. 22 (e) shows frame buffer data for + (n-1) frames (nth frame). The position of the block after n−1 frames is a position moved by (n−1) v _{2 in} the right direction from the block position in the current frame. The address in the frame buffer is a position shifted by (n−1) × 1 pixel in the right direction from the address of the frame buffer in the current frame. According to this calculated address, the frame buffer FBn (123-n) is moved as shown in FIG. The corresponding pixel value of the effective pixel of the input image data shown in a) is written.

  FIG. 23 shows processing in the second frame. FIG. 24 shows processing in the third frame, and FIG. 25 shows processing in the fourth frame. Since the example of FIGS. 22 to 26 is a case where the block is subjected to the time direction thinning process, there is no effective pixel in the second frame to the fourth frame by the time direction thinning process (FIG. 23A). , FIG. 24 (a), FIG. 25 (a)). Accordingly, in these frames, only the current frame shift processing (see Table 1) of the frame buffer is performed as in FIG. 18 described above, and new data is not written.

  FIG. 26 shows processing in the fifth frame. In the fifth frame, as in the first frame, all the pixels of the block provided from the input unit 121 are valid.

FIG. 26B shows data in the frame buffer FB5 (123-5) for the current frame (fifth frame). Here, data after the fourth frame processing is written in the frame buffer. Since there are no effective pixels in the second to fourth frames, only the effective pixel data of the first frame is written. Since the shift process (1 pixel / frame) for 4 frames is executed, the data of the first frame is the address x = 2x ₁₈ , 2x ₁₉ , 2x ₂₀ , 2x ₂₁ , of the frame buffer FB1 (123-1), 2x ₂₂ , 2x ₂₃ , 2x ₂₄ and 2x ₂₅ .

On the other hand, the write address for the frame buffer FB5 (123-5) of the current frame (fifth _frame), each of _{x = 2x 16, 2x 17,} 2x 18, 2x 19, 2x 20, 2x 21, 2x 22, 2x 23 It becomes a column.

Therefore, when data is written to each column of x = 2x ₁₈ , 2x ₁₉ , 2x ₂₀ , 2x ₂₁ , 2x ₂₂ , 2x ₂₃ , the pixel value of the first frame data that has already been written in the addition processing unit 122-1. Then, after the addition average processing with the pixel value of the current frame (fifth frame) is executed, the addition average value is written. The pixel value of the current frame (fifth frame) is written in each column of x = 2x ₁₆ and 2x ₁₇ .

  FIG. 26C shows the frame buffer data for the +1 frame (sixth frame), and FIG. 26D shows the frame buffer data for the +2 frame (seventh frame). These are the same processes as the frame buffer of FIG. 26B, that is, when written data exists at the position of the write address, the averaging process of the value and the pixel value of the current frame (fifth frame) After executing, write the addition average value. When there is no written data at the position of the write address, the pixel value is written in the current frame (fifth frame).

FIG. 26 (e) shows frame buffer data of + (n−1) frames (n + 4th frame). Since there is no written data, the address calculated based on the motion vector v ₂ is shown here. The pixel value of the current frame (fifth frame) is written at the position.

  Next, the operation of the interpolation unit 124 will be described with reference to FIGS. FIG. 27 shows an interpolation process for restoring the image of the fifth frame in the block subjected to the spatial direction thinning process (the embodiment in FIGS. 12 to 16). FIG. 27A shows the input image data in the fifth frame (the same as FIG. 16A), and FIG. 27B shows the selection by the selection unit 127 after the processing of the fifth frame (after step S9). The frame buffer data (same as FIG. 16B) is shown.

  In the data of the frame buffer selected by the selection unit 127 after processing the fifth frame (after step S9) shown in FIG. 27B (same as FIG. 16B), the first to fifth frames are valid. Data recorded based on the pixel value data is stored. Although the amount of data stored in the frame buffer differs depending on the number of applied buffers n, here, description will be made assuming that data for five frames is stored in the buffer.

  The interpolation unit 124 executes a process of interpolating the pixel value lacking the sample point data based on the surrounding pixel values based on the data stored in the frame buffer. FIG. 27C shows the frame buffer shown in FIG. As described above, the frame buffers 1 to n (121-1 to n) have a storage area having twice as many pixels as the input pixel data, and the interpolation unit 124 converts the pixel data of the storage area into the original pixel data. A process of reducing to a size corresponding to the input pixel data is executed. The 4-pixel area shown in FIG. 27B is reduced to the 1-pixel area shown in FIG.

  In the example shown in FIG. 27, there is no portion where a plurality of valid data exists in four pixels of the frame buffer corresponding to one pixel of the reduced image shown in (c), but a plurality of valid data in four pixels. If there is, the added average value is used as the pixel value of the reduced image.

In FIG. 27C, pixel loss occurs at each address of x = x ₀ , x ₂ , x ₄ , x ₉ , x ₁₃ , x ₁₄ . The interpolation unit 124 performs interpolation processing on these missing pixels according to the direction of the motion vector. Specifically, since the motion vector of the block is the horizontal direction, for example x = x ₂ columns, performs interpolation processing in the horizontal direction only by using each pixel of x = x ₁ and x = x ₃ .

Specifically, in FIG. 27C, the pixel values of each pixel of x = x ₁ and x = x ₃ are respectively
Pixel value of (x ₁ , y ₀ ) = α1,
Pixel value of (x ₁ , y ₁ ) = α2,
Pixel value of (x ₁ , y ₂ ) = α3,
Pixel value of (x ₁ , y ₃ ) = α4,
Pixel value of (x ₃ , y ₀ ) = β1,
Pixel value of (x ₃ , y ₁ ) = β2,
Pixel value of (x ₃ , y ₂ ) = β3,
Pixel value of (x ₃ , y ₃ ) = β4,
When it is, the pixel values of x = 2x ₂ is
Pixel value of (x ₁ , y ₀ ) = (α1 + β1) / 2,
Pixel value of (x ₁ , y ₁ ) = (α2 + β2) / 2,
_(X 1, _{y 2)} of the pixel value = (α3 + β3) / 2 ,
Pixel value of (x ₁ , y ₃ ) = (α4 + β4) / 2,
As, in the interpolation section 124, arithmetic mean value is calculated, the pixel value of a x = 2x _2.

Further, if the missing pixels are continuous two pixels or more (e.g., columns of _{x = x 13, x 14)} , linear interpolation using the two ends of the pixels of the motion direction. Further, if there is no pixel value only on one side of the missing pixel (e.g. x = column of x _0), fill all the missing pixel in the pixel values of the edge with the pixel value of the motion direction. FIG. 27D shows data after interpolation processing, and an image in which all pixel values are aligned is generated.

In FIG. 27 (d),
The pixel value of the column of x = x ₀ is determined by the pixel value of the column of x = x ₁
pixel values of a column of x = _{x 2} is determined by x = _{x 1} and x = pixel value of a column of _{x 3,}
pixel values of a column of x = _{x 4} is determined by x = _{x 3} and x = pixel value of a column of _{x 5,}
pixel values of a column of x _{= x 13} and x _{= x 14} is determined by the pixel values of a column of x _{= x 12} and x _{= x 15.}

  In this embodiment, only two blocks adjacent to each other are described. However, since there are actually different blocks around them, the effective pixel values of those blocks are also written in the frame buffer. It is.

  FIG. 28 shows an interpolation process for restoring an image in the fifth frame in the block subjected to the spatial direction thinning process and the time direction thinning process (the embodiment in FIGS. 17 to 21). FIG. 28A shows the input image data in the fifth frame (same as FIG. 21A), and FIG. 27B shows the selection by the selection unit 127 after the processing of the fifth frame (after step S9). The frame buffer data (same as FIG. 21B) is shown.

  As in the case of FIG. 27, FIG. 28C shows data obtained by reducing the frame buffer data to ½ in the vertical and horizontal directions, and FIG. 28D shows an interpolation process in the horizontal direction for the missing pixels in FIG. Shows the result of interpolation.

In FIG. 28C, pixel loss occurs at each address of x = x ₉ , x ₁₉ , x ₂₁ , x ₂₂ , x ₂₃ . The interpolation unit 124 performs interpolation processing on these missing pixels according to the direction of the motion vector. Specifically, since the motion vector of the block is in the horizontal direction, for example, for the column of x = x ₉ , interpolation processing only in the horizontal direction is performed using each pixel of x = x ₈ and x = x _30. . For the column of x = x ₁₉ , interpolation processing only in the horizontal direction is performed using each pixel of x = x ₁₈ and x = x ₂₀ . For the column of x = x _{21 to} x ₂₃ , interpolation processing only in the horizontal direction is performed using pixels of x = x ₂₀ .

  FIG. 29 shows an interpolation process for restoring an image in the fifth frame in the block subjected to the time direction thinning process (the embodiment in FIGS. 22 to 26). FIG. 29A shows input image data in the fifth frame (same as FIG. 26A), and FIG. 29B shows selection by the selection unit 127 after processing the fifth frame (after step S9). The frame buffer data (same as FIG. 26B) is shown. As in the case of FIG. 27, FIG. 29 (c) shows data obtained by reducing the frame buffer data to 1/2 in the vertical and horizontal directions, and FIG. 29 (d) shows a horizontal interpolation process for the missing pixels in FIG. Shows the result of interpolation.

In FIG. 29 _(c), pixel missing occurs in the address of _x = _x 26 _{~x 31.} The interpolation unit 124 performs interpolation processing on these missing pixels according to the direction of the motion vector. Specifically, since the motion vector of the block is the horizontal direction, the columns of x = x ₂₆ ~x _31, performs an interpolation process only in the horizontal direction by using the pixels of x = x _25.

  In the above-described example, the interpolation unit 124 has shown an example of interpolation processing in which pixel values in a direction parallel to the motion direction are applied. However, effective pixels in four pixels above, below, left, and right of the missing pixel, or in the surrounding seven pixels. A configuration may be adopted in which interpolation processing using values is performed.

FIG. 30 is a diagram for explaining the details of the interpolation processing in the interpolation unit 124. FIG. 30A shows interpolation processing of missing pixels when the processing block has a horizontal motion vector. In this case, the interpolation unit 124 is an example in which linear interpolation processing is performed using the nearest pixels on the left and right of the missing pixel. The pixel values of the columns x = x ₁ and x ₂ are linearly interpolated based on the pixel values of the columns x = x ₀ and x ₃ .

FIG. 30B shows a missing pixel interpolation process when the processing block has a vertical motion vector. In this case, the interpolation unit 124 is an example in which linear interpolation processing is performed using the nearest neighboring pixels above and below the missing pixel. The pixel values in the row of y = y ₁ , y ₂ are linearly interpolated based on the pixel values in the column of y = y ₀ , y ₃ .

  FIG. 30C shows the interpolation processing of missing pixels when the processing block is not moving at all. In this case, the interpolation unit 124 performs linear interpolation in each direction using the nearest pixels above, below, left, and right of the missing pixel, and further averages the two results.

FIG. 30D shows the interpolation processing when the processing block has a horizontal motion vector and there is no effective pixel on the left side of the missing pixel. In this case, the interpolation unit 124 uses the value of the nearest pixel on the right side of the missing pixel as the missing pixel value as it is. The pixel value of the column of x = x _{0 to} x ₂ is set to the pixel value of the column of x = x ₃ . When there is no effective pixel on the right side of the missing pixel, the nearest pixel value on the left side is set as the missing pixel value. If the block has a vertical motion vector and there is no effective pixel above the missing pixel, the lower nearest pixel is set as the missing pixel value. Similarly, when there is no effective pixel on the lower side, the uppermost nearest pixel is set as the missing pixel value.

  In the above-described embodiment, an example in which the effective pixel values of the past frame are written in the frame buffers FB1 to n (123-1 to n) of the moving image conversion apparatus 120 and the pixel value calculation process of the current frame is executed will be described. However, the effective pixel value in the future frame of the current frame may be applied.

Of the number n of frame buffers in the moving image conversion apparatus 120 shown in FIG. 10, b represents the number of frame buffers for storing data for past frames, f represents the number of frame buffers for storing data for future frames, and corresponds to the current processing frame. The number of frame buffers is set as 1. That is,
n = b + f + 1
It becomes.

In such a setting example, the address conversion unit 126 obtains an effective pixel address for the future frame by adding the motion vector corresponding to the processing target block, and at the same time, subtracts the motion vector of the block. The effective pixel address for the frame is obtained. Table 2 shown below is a table explaining the switching operation of the frame buffer.

  During the processing of the first frame, the pixel value in the current frame is written into the frame buffer FB (b + 1). Pixel values for future frames are written in the frame buffer FB (b + 2) to the frame buffer FB (f + b + 1). On the other hand, pixel values for past frames are written in the frame buffers FB1 to FB (b). At this time, the writing operation from the future +1 frame to the + f frame is the same as in FIGS. On the other hand, the writing from the past -1 frame to the -b frame is the same as the case where the motion vector is reversed in both the x direction and the y direction in the operation of FIGS.

  The selection unit 127 is controlled to select the frame buffer for the oldest frame without fail. In the example of Table 2, the −b frame is the selected frame. Since the data in the frame buffer selected by the selection unit 127 is interpolated by the interpolation unit 124 and output from the output unit 125, the output image is delayed by b frames with respect to the input image in this embodiment. It will be.

  In the processing example described above, the processing example corresponding to the motion vector only in the horizontal direction has been described for easy understanding. However, as described above, in the actual moving image processing, the motion vector in the vertical direction is converted to the motion vector in the vertical direction. In contrast, the same processing as described above can be performed. Further, by executing two-dimensional processing by applying both horizontal and vertical motion vectors, processing corresponding to all motion vectors in the two-dimensional direction is possible.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the present invention, in decompression processing of compressed data in which pixel data is thinned out by a thinning process in a spatial direction or a thinning process in a time direction, motion information such as a motion vector of a block The data of the same block over a plurality of frames is accumulated in the frame buffer reflecting the movement position of the block calculated based on the data, and the data recovery is executed based on the accumulated sample point data of a plurality of different frames. Therefore, it is possible to acquire pixel values missing in one frame based on data of different frames, and it is possible to improve the pixel value reproduction rate of each frame. As a result, high-quality moving image restoration closer to the original image is realized.

  In the conventional configuration, there is a problem that jaggies (folding of the spatial frequency collection) based on the reproduced data of the restored data and the pixel data lost by the thinning process are conspicuous. However, according to the present invention, the pixel value of each frame Since the reproducibility is improved, high-quality data close to the original data can be suppressed even in playback processing at a rate different from the playback rate of the original moving image, such as slow motion playback. Playback is realized.

It is a figure which shows the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure which shows the detail of the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining an example of the process result of the block process part in a moving image converter. It is a figure explaining the process example of the horizontal space thinning process with respect to a moving subject, and the decompression | restoration process of an original image. It is a figure explaining the process example which changed the thinning position for every frame, and the restoration process of an original image in the horizontal space thinning process with respect to a moving subject. It is a figure which shows the structure of the image converter which performs the decompression | restoration process from moving image compression data. It is a flowchart explaining the image conversion process sequence which performs the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the transition of the frame buffer storage data in the decompression | restoration process from moving image compression data. It is a figure explaining the example of the interpolation process in the decompression | restoration process from moving image compression data. It is a figure explaining the example of the interpolation process in the decompression | restoration process from moving image compression data. It is a figure explaining the example of the interpolation process in the decompression | restoration process from moving image compression data. It is a figure explaining the aspect of the interpolation process in the decompression | restoration process from moving image compression data.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image converter 11 Block division part 12 Movement amount detection part 13 Block processing part 14 Output part 21 Image storage part 22 Block division part 31 Movement amount detection part 32 Block distribution part 51-53 Block processing part 101-116 Block 120 Image conversion Device 121 Input unit 122 Addition processing unit 123 Frame buffer 124 Interpolation unit 125 Output unit 126 Address conversion unit 127 Selection unit

Claims (13)

A moving image conversion apparatus that executes a moving image compressed data restoration process,
An input unit for inputting compressed data in units of blocks as an area included in a frame constituting a moving image;
A block position in a different frame on the time axis is calculated based on motion information included in the compressed data or motion information calculated from the compressed data, and an address corresponding to the calculated position is used for frame buffer access corresponding to the frame. An address conversion unit that calculates an address;
A plurality of frame buffers corresponding to frames for storing pixel values included in block compressed data corresponding to each frame or calculated values based on the pixel values at a memory location specified by an address corresponding to the frame calculated by the address conversion unit;
An interpolation unit that generates a restoration block based on sample point data of a plurality of different frames based on data stored in the frame buffer;
An output unit that outputs moving image frame data constituted by the restoration block generated by the interpolation unit ;
The plurality of frame buffers include a restoration processing target frame, a pixel value over at least one of a frame preceding or following the restoration processing target frame on a time axis, or an overlap of calculated values based on the pixel value A moving image conversion apparatus having a configuration in which writing is performed . The moving image conversion device further includes:
When there is already written pixel value data at the memory location of the frame buffer specified by the frame corresponding address calculated by the address conversion unit, the pixel value data and the pixel value included in the block compressed data corresponding to the processing frame An addition processing unit for calculating the addition average value of
The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to execute a process of writing the addition average value calculated by the addition processing unit into a memory location specified by the frame corresponding address. The address conversion unit
A motion vector indicating block movement information in units of frames is input as the motion information of the block, the block position in a different frame on the time axis is calculated based on the motion vector, and the address corresponding to the calculated position corresponds to each frame The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to calculate the address of the moving image. The interpolation unit
The block configuration data is configured to execute a process of interpolating a pixel value missing from the sample point data based on a pixel value in a direction parallel to the motion direction included in the motion information corresponding to the block. The moving image conversion apparatus according to claim 1. The interpolation unit
A configuration for executing a process of interpolating a missing pixel value of the sample point data in the block configuration data and a missing pixel value in the block configuration data in the data stored in the frame buffer based on a surrounding pixel value of the missing pixel in the block configuration data The moving image conversion apparatus according to claim 1, wherein: The moving image conversion device further includes:
2. The moving image conversion apparatus according to claim 1, further comprising a selection unit that executes a process of sequentially switching a frame buffer storing output data for the interpolation unit as the output data switching process for the interpolation unit. . It is a moving image conversion method for executing a moving image compressed data restoration process,
An input step for inputting compressed data in units of blocks as an area included in a frame constituting a moving image;
A block position in a different frame on the time axis is calculated based on motion information included in the compressed data or motion information calculated from the compressed data, and an address corresponding to the calculated position is used for frame buffer access corresponding to the frame. An address conversion step to calculate as an address;
A data storing step for each of a plurality of frame buffers, wherein a pixel value included in the block compressed data corresponding to each frame or the memory location of each frame buffer accessed by the address corresponding to the frame calculated in the address converting step A data storage step for storing a calculated value based on the pixel value;
An interpolation step for generating a restoration block based on sample point data of a plurality of different frames based on the data stored in the frame buffer;
An output step of outputting moving image frame data constituted by the restoration block generated in the interpolation step ;
The data storing step includes
For the plurality of frame buffers, a pixel value over a plurality of frames of at least one of a restoration processing target frame and a frame preceding or following the restoration processing target frame on a time axis, or a calculated value based on the pixel value A moving image conversion method characterized in that it is a step of executing redundant writing . The moving image conversion method further includes:
When written pixel value data exists in the memory location of the frame buffer specified by the frame corresponding address calculated in the address conversion step, the pixel value data and the pixel value included in the block compressed data corresponding to the processing frame An addition processing step for calculating an addition average value of
The data storing step includes
8. The moving image conversion method according to claim 7 , wherein the moving image conversion method is a step of executing a process of writing the addition average value calculated in the addition processing step into a memory location specified by the frame corresponding address. The address conversion step includes:
A motion vector indicating block movement information in units of frames is input as the motion information of the block, the block position in a different frame on the time axis is calculated based on the motion vector, and the address corresponding to the calculated position corresponds to each frame The moving image conversion method according to claim 7 , wherein the moving image conversion method is a step of calculating the address of the moving image. The interpolation step includes
A step of interpolating a pixel value missing from the sample point data in the block configuration data based on a pixel value in a direction parallel to a motion direction included in the motion information corresponding to the block. The moving image conversion method according to claim 7 . The interpolation step includes
The moving image conversion method according to claim 7 , further comprising a step of interpolating a missing pixel value of the sample point data in the block configuration data based on a surrounding pixel value of the missing pixel. . The moving image conversion method further includes:
The moving image conversion method according to claim 7 , further comprising: a frame buffer selection step of sequentially switching frame buffers as output data switching processing for an interpolation unit that performs the pixel value interpolation processing. A computer program for executing a decompression process of compressed moving image data,
An input step for inputting compressed data in units of blocks as an area included in a frame constituting a moving image;
A block position in a different frame on the time axis is calculated based on motion information included in the compressed data or motion information calculated from the compressed data, and an address corresponding to the calculated position is used for frame buffer access corresponding to the frame. An address conversion step to calculate as an address;
A data storage step for each of a plurality of frame buffers, wherein the pixel value included in the block compressed data corresponding to each frame or the memory location of each frame buffer accessed by the address corresponding to the frame calculated in the address conversion step A data storage step for storing a calculated value based on the pixel value;
An interpolation step for generating a restoration block based on sample point data of a plurality of different frames based on the data stored in the frame buffer;
An output step of outputting moving image frame data constituted by the restoration block generated in the interpolation step ;
The data storing step includes
For the plurality of frame buffers, a pixel value over a plurality of frames of at least one of a restoration processing target frame and a frame preceding and following the restoration processing target frame on a time axis, or a calculated value based on the pixel value A computer program characterized in that it is a step of executing redundant writing .

Images