JP4779424B2 - Moving picture conversion apparatus, moving picture conversion method, and computer program - Google Patents

Description

  The present invention relates to a moving image conversion apparatus, a moving image conversion method, and a computer program. In particular, the present invention relates to a moving image conversion apparatus, a moving image conversion method, and a computer program that can suppress deterioration in image quality and enable high-quality data conversion in data conversion executed as data compression processing of moving image data.

  The moving image data is subjected to data conversion for reducing the amount of data, that is, compression processing, when it is stored in a storage medium such as a hard disk or DVD or distributed via a network. In particular, in recent years, the quality of moving image data has been improved, for example, HD (High Definition) data has been improved, and the amount of data has increased rapidly with the improvement of the quality of this data. Under such circumstances, many studies and studies have been made on techniques for improving the compression efficiency in the compression and decompression processing of moving image data and preventing the quality degradation of the decompressed data.

  As a moving image compression processing method, for example, thinning processing of pixels constituting an image frame constituting moving image data, that is, thinning processing in a spatial direction, thinning processing of a frame rate, that is, thinning processing in a time direction, and the like are known. Yes.

  By reducing the amount of data by such data conversion, there is an advantage that saving to a storage medium and data transfer via a network are efficiently performed. However, when the compressed data is restored and reproduced, there is a problem that the image quality deteriorates. In particular, when the original data is a high-definition image, the quality deterioration becomes more remarkable.

Various studies have been made on how to reduce such image quality deterioration. For example, Patent Document 1 discloses an image compression processing configuration in which a parameter based on image brightness information is set and the compression mode is changed according to the image brightness. Patent Document 2 discloses a compression processing configuration in which a screen is divided into a plurality of areas and the compression mode is changed for each area.
JP 2003-169284 A Japanese Patent Laid-Open No. 2002-27466

  As described above, in some conventional techniques, a configuration is disclosed in which the compression mode is changed based on various characteristics obtained from the processing target image and the data quality is improved. In the method, the compressed image data is restored and reproduced, and image quality deterioration is not sufficiently suppressed.

  In particular, moving image data is data including various types of motion data such as a portion where the subject's motion is large, a small portion, a portion where the subject is stationary, etc., and it is appropriate to correspond to these different motions without losing data quality. There is a problem that it is difficult to prevent the deterioration of quality when compression processing is performed and data restoration and reproduction are performed.

  The present invention has been made in view of such problems, and determines the optimum compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject, and optimizes each region according to the determined mode. It is an object of the present invention to provide a moving image conversion apparatus, a moving image conversion method, and a computer program that can perform compression and decompression with extremely little deterioration in quality by performing a data conversion process in such a manner.

The first aspect of the present invention is:
A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit for detecting a subject movement amount in each block divided by the block division unit;
The movement amount information detected by the movement amount detection unit is input, and based on the movement amount information, the spatial direction thinning process, the time direction thinning process, or the spatial direction, A block distribution unit that determines which one of the time direction thinning processes is to be executed, and executes a process of supplying the block processing target block data to the block processing unit that performs the thinning process;
A block processing unit that inputs block data and executes either a spatial direction thinning process, a temporal direction thinning process, or a spatial direction and a temporal direction thinning process according to a movement amount corresponding to the input block;
The block processing unit
Corresponding to the past frame detected by the movement amount detection unit based on the block corresponding to the processing target block and the pixel position and belonging to the previous thinning unit frame of the processing target when executing the temporal direction thinning process on the input block A frame determination unit that inputs a movement amount , compares the movement amount corresponding to the past frame with a predetermined threshold U, and determines a block-corresponding frame to be left as data in time direction thinning according to the comparison result as a different frame ;
A time direction decimation processing unit that executes a time direction decimation process that leaves block data corresponding to the frame determined by the frame determination unit;
The moving image converting apparatus is characterized by having a configuration including:

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the frame determination unit is configured such that when the past frame corresponding movement amount is equal to or greater than a predetermined threshold U, that is, the past frame corresponding movement amount ≧ the threshold value U. In some cases, the last frame (n + N-1) of the frames (n to n + N-1) to which the time direction thinning process target block belongs is determined as a block corresponding frame to be left as data in the time direction thinning, and the past frame corresponding movement amount < In the case of the threshold value U, a process of determining the first frame (n) of the frame (n to n + N−1) to which the time direction decimation processing target block belongs as a block corresponding frame to be left as data in the time direction decimation is executed. The time direction thinning-out processing unit is a block corresponding to the frame determined by the frame determination unit. Characterized in that it is configured to perform temporal decimation processing to leave over data.

  Furthermore, in an embodiment of the moving image conversion apparatus of the present invention, the movement amount detection unit includes N blocks corresponding to pixel positions in a plurality of continuous frames (n to n + N−1) serving as a thinning process execution unit. From the above, a process for detecting the movement amount based on the block belonging to the last frame (n + N−1) is executed, and the frame determination unit is configured to detect the movement amount detection unit in a block belonging to the last frame (n + N−1). The movement amount detected based on the above is input as the movement amount corresponding to the past frame, and based on the input movement amount corresponding to the past frame, the process of determining the block correspondence frame to be left as data in the time direction decimation is executed. Features.

Furthermore, in an embodiment of the moving image conversion apparatus of the present invention, the movement amount detection unit includes N blocks corresponding to pixel positions in a plurality of continuous frames (n to n + N−1) serving as a thinning process execution unit. And a process of detecting the movement amount A based on the block belonging to the last frame (n + N−1), and the movement amount B based on the block belonging to the frame other than the last frame (n + N−1). It is configured to execute a process of detecting and calculating an output movement amount to be output to the block processing unit by averaging the movement amount A and the movement amount B.

Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the block distribution unit is configured to determine a thinning processing mode based on movement amount information detected by the movement amount detection unit, and In the setting of Vt, two predetermined thresholds Va and Vb, where Va> Vb, when the movement amount is Vt ≧ Va, the execution processing for a plurality of consecutive frames serving as a thinning processing execution unit is performed in the spatial direction thinning processing. When the movement amount is Vt <Vb,
When execution processing for a plurality of continuous frames that are thinning process execution units is determined as thinning processing in the time direction and the movement amount Vt is Vb ≦ Vt <Va, execution for a plurality of continuous frames that are thinning process execution units The present invention is characterized in that the process is determined as a thinning process in the spatial direction and the time direction.

Furthermore, the second aspect of the present invention provides
A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
The movement amount information detected in the movement amount detection step is input, and on the basis of the movement amount information, a spatial direction thinning process, a time direction thinning process, or a spatial direction is performed for a plurality of consecutive frames that are thinning process execution units. And a block distribution step for determining which of the time direction thinning processing is to be executed, and executing processing for supplying the thinning target block data to the block processing unit that performs the thinning processing;
A block processing step for inputting block data and performing either a spatial direction thinning process, a temporal direction thinning process, or a spatial direction and a temporal direction thinning process according to a movement amount corresponding to the input block;
The block processing step includes:
When executing the temporal direction decimation process for the input block, the past frame corresponding movement amount detected based on the block corresponding to the process target block and the pixel position and belonging to the previous decimation process unit frame of the process target is input. A frame determination step of comparing the movement amount corresponding to the past frame with a predetermined threshold U, and determining a block-corresponding frame to be left as data in the time direction thinning according to the comparison result as a different frame ;
There is provided a moving image conversion method comprising: a time direction thinning process step for executing a time direction thinning process for leaving block data corresponding to the frame determined in the frame determining step.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the frame determination step is performed when the past frame correspondence movement amount is equal to or greater than a predetermined threshold U, that is, the past frame correspondence movement amount ≧ threshold value U. In some cases, the last frame (n + N-1) of the frames (n to n + N-1) to which the time direction thinning process target block belongs is determined as a block corresponding frame to be left as data in the time direction thinning, and the past frame corresponding movement amount < If it is the threshold value U, a process of determining the top frame (n) of the frame (n to n + N−1) to which the time direction decimation processing target block belongs as a block corresponding frame to be left as data in the time direction decimation is executed, The time direction decimation processing step includes the frame pair determined in the frame determination step. And executes a leaving temporal decimation processing block data.

  Furthermore, in an embodiment of the moving image conversion method of the present invention, the movement amount detection step includes N blocks corresponding to pixel positions in a plurality of consecutive frames (n to n + N−1) serving as a thinning process execution unit. From the above, the processing for detecting the movement amount is executed based on the block belonging to the last frame (n + N−1), and the frame determination step determines whether the block belonging to the last frame (n + N−1) in the movement amount detection step. The movement amount detected based on this is input as the movement amount corresponding to the past frame, and the process of determining the block corresponding frame to be left as data in the time direction decimation is executed based on the input movement amount corresponding to the past frame.

Furthermore, in an embodiment of the moving image conversion method of the present invention, the movement amount detection step includes N blocks corresponding to pixel positions in a plurality of consecutive frames (n to n + N−1) serving as a thinning process execution unit. And a process of detecting the movement amount A based on the block belonging to the last frame (n + N−1), and the movement amount B based on the block belonging to the frame other than the last frame (n + N−1). A process of detecting and calculating an output movement amount to be output to the block processing unit by an averaging process of the movement amount A and the movement amount B is performed.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the block distribution step includes a step of determining a thinning processing mode based on the movement amount information detected in the movement amount detection step, and the movement amount is calculated. In the setting of Vt, two predetermined thresholds Va and Vb, where Va> Vb, when the movement amount is Vt ≧ Va, the execution processing for a plurality of consecutive frames serving as a thinning processing execution unit is performed in the spatial direction thinning processing. When the movement amount: Vt <Vb, the execution process for a plurality of consecutive frames as a thinning process execution unit is determined as the time direction thinning process, and the movement amount Vt is Vb ≦ Vt <Va The execution processing for a plurality of consecutive frames as a thinning processing execution unit is determined as the thinning processing in the spatial direction and the time direction. The features.

Furthermore, the third aspect of the present invention provides
A computer program for executing data conversion processing of moving image data on a computer,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
The movement amount information detected in the movement amount detection step is input, and on the basis of the movement amount information, a spatial direction thinning process, a time direction thinning process, or a spatial direction is performed for a plurality of consecutive frames that are thinning process execution units. And a block distribution step for determining which of the time direction thinning processing is to be executed, and executing processing for supplying the thinning target block data to the block processing unit that performs the thinning processing;
A block processing step for inputting block data and performing either a spatial direction thinning process, a temporal direction thinning process, or a spatial direction and a temporal direction thinning process according to a movement amount corresponding to the input block;
The block processing step includes:
When executing the temporal direction decimation process for the input block, the past frame corresponding movement amount detected based on the block corresponding to the process target block and the pixel position and belonging to the previous decimation process unit frame of the process target is input. A frame determination step of comparing the movement amount corresponding to the past frame with a predetermined threshold U, and determining a block-corresponding frame to be left as data in the time direction thinning according to the comparison result as a different frame ;
And a time direction decimation process step for executing a time direction decimation process that leaves the block data corresponding to the frame determined in the frame determination step.

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, the optimum compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject is determined, and the data conversion processing is performed in the optimal mode for each region according to the determined mode. By doing so, it becomes possible to perform compression and decompression with very little quality deterioration. In particular, in the configuration of the present invention, when the time direction decimation is executed, the past frame correspondence movement amount detected based on the block belonging to the previous decimation unit frame to be processed is input, and the past frame Since the block-corresponding frame to be left as data in the time direction decimation is determined based on the corresponding movement amount, and the time-direction decimation process is performed to leave the block data corresponding to the determined frame, it is not originally present in the restored image. Image deterioration such as displaying a moving subject can be reduced, and data conversion that enables high-quality image compression and restoration is realized.

  In particular, according to the configuration of the present invention, based on the movement amount corresponding to the past frame, the block correspondence frame to be left as data in the time direction thinning is determined, and the time direction thinning process for leaving the determined frame correspondence block data is executed. Because of the configuration, when there are very few moving subjects in the time direction decimation processing target block, there is no display of moving subjects that do not exist in the restored image, enabling high-quality image compression and restoration. Data conversion is realized.

Hereinafter, the configuration of a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program according to the present invention will be described with reference to the drawings. The description will be made according to the following items.
(1) Basic configuration of moving image conversion apparatus using super-resolution effect (2) Configuration of moving image conversion apparatus for executing improved thinning process

[(1) Basic configuration of moving image conversion apparatus using super-resolution effect]
First, a basic configuration of a moving image conversion apparatus that performs moving image compression using the super-resolution effect as a base of the present invention will be described. This basic configuration is described in detail in Japanese Patent Application No. 2003-412501 filed earlier by the present applicant. The image is divided into small areas, and the number of pixels is determined according to the moving speed of each area. This is a configuration in which compression of the data amount is realized by adaptively performing decimation and frame rate decimation.

  FIG. 1 shows a configuration example of the moving image conversion apparatus 10 described in Japanese Patent Application No. 2003-412501. The moving image conversion apparatus 10 can reduce the amount of data so that an observer does not perceive image quality deterioration due to the reduction of the data amount by performing a moving image conversion process using the super-resolution effect. It is a thing.

  Note that the super-resolution effect is a visual effect realized based on a visual characteristic that an observer perceives an image obtained by adding a plurality of images within a certain period of time. When human vision perceives a stimulus, it has a function of storing the stimulus for a certain period of time after the presentation of the stimulus ends (referred to as a sensory memory function). There are many reports that this time is 10 ms to 200 ms. This function is also called iconic memory or visual persistence, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp.229-230”. The super-resolution effect is considered to be caused by a complicated relationship between the temporal integration function and the sensory memory function in the human visual function.

  The moving image conversion apparatus 10 shown in FIG. 1 performs compression for reducing data so that the observer does not perceive image quality degradation by performing moving image conversion processing using the super-resolution effect caused by the temporal integration function. It is set as the structure to perform. A configuration of the moving image conversion apparatus 10 in FIG. 1 will be described.

  The block dividing unit 11 divides each frame of the input moving image into blocks as divided areas of predetermined pixels and supplies them to the movement amount detecting unit 12. The movement amount detection unit 12 detects the movement amount for each block supplied from the block division unit 11, and transmits the block and the movement amount to the block processing unit 13. The block processing unit 13 performs a moving image conversion process corresponding to the movement amount, that is, a compression process, on the block supplied from the movement amount detection unit 12 to reduce the data amount. The block processing unit 13 supplies the output unit 14 with data about the block with the reduced data amount obtained as a result of the processing. The output unit 14 collectively outputs, as stream data, the data regarding the blocks with the reduced data amount supplied from the block processing unit 13. The restoration unit 15 receives the compressed data output from the output unit 14, executes restoration processing by decompression processing, and generates reproducible moving image data.

  Next, with reference to FIG. 2, details of each unit excluding the restoration unit 15 will be described. The frame of the moving image supplied to the moving image conversion apparatus 10 is input to the image storage unit 21 of the block dividing unit 11. The image storage unit 21 stores the input frame. Each time the number of accumulated frames reaches N (N is a positive integer), the image accumulating unit 21 supplies the N frames to the block dividing unit 22, and M in the N frames The second stored frame (hereinafter referred to as the Mth frame) is supplied to the movement amount detection unit 12 (movement amount detection unit 31). For example, N = 4.

  The block dividing unit 22 divides each of the N frames (consecutive N frames) supplied from the image storage unit 21 into blocks of a certain size (for example, 8 × 8, 16 × 16) and moves them. It outputs to the quantity detection part 12 (block distribution part 32). The block dividing unit 22 also converts each block of the Pth frame stored in the image storage unit 21 (hereinafter referred to as the Pth frame) among the N frames to the movement amount detection unit 12 (movement amount detection). Part 31). The Pth frame is a different frame from the Mth frame.

  Next, the movement amount detection unit 12 will be described. The movement amount detection unit 31 of the movement amount detection unit 12 uses the motion vector of each block of the Pth frame supplied from the block division unit 22 of the block division unit 11 as the Mth frame supplied from the image storage unit 21. For example, the block matching process between the frames is detected and detected, and the detected motion vector is supplied to the block distributor 32. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and in the vertical direction (Y-axis direction). In addition, the movement amount detection unit 31 may be configured to enlarge the image in order to improve the detection accuracy of the movement amount and detect the movement amount using the enlarged image.

  The block distribution unit 32 of the movement amount detection unit 12 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 22 in units of N, and the movement amount detection unit From 31, the amount of movement of the block of the P-th frame among the N blocks is supplied. The block distribution unit 32 supplies the supplied N blocks and the movement amount to any of the block processing units 51 to 53 that perform processing corresponding to the movement amount of the block processing unit 13.

  Specifically, the block distribution unit 32 is supplied from the movement amount detection unit 31 and has a movement amount in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) between one frame of 2 pixels (pixels) or more. In some cases, the N blocks supplied from the block dividing unit 22 and the movement amount supplied from the movement amount detection unit 31 are output to the block processing unit 51. If the amount of movement in the horizontal direction and the vertical direction between one frame is less than 2 pixels and more than 1 pixel, the block distribution unit 32 outputs N blocks and the amount of movement to the block processing unit 53. . When the movement amount is other than that, the block distribution unit 32 supplies the N blocks and the movement amount to the block processing unit 52.

  That is, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs block processing for performing processing for converting image data according to the frame rate and spatial resolution. The block images are distributed to the units 51 to 53.

  This condition for determining the distribution destination is merely an example, and the distribution destination may be determined under other conditions.

  Next, details of the block processing unit 13 will be described. The block processing unit 13 is composed of the three block processing units 51 to 53 as described above. The block processing unit 51 supplies a total of N blocks (horizontal or vertical) at the same position in each of consecutive N (for example, N = 4) frames supplied from the block distribution unit 32 of the movement amount detection unit 12. Similarly, a process of thinning out the number of pixels according to the amount of movement supplied from the block distributor 32 (spatial direction thinning process) is performed on N blocks in the case where the amount of movement in the direction is 2 pixels or more. .

  Specifically, when the amount of movement in the horizontal direction between one frame is 2 pixels (pixels) or more, the block processing unit 51, when the block is configured by 8 × 8 pixels, as shown in FIG. The pixels in the block are divided into sets of 1 × 4 pixel units. Further, as shown in FIG. 4, the block processing unit 51 sets the pixel values p1 to p4 of each set of 1 × 4 pixels to one pixel value (p1 in this example) of the pixel values. Thinning (thinning of the number of pixels between four pixels) (thinning of the thinning amount 4) is performed.

  When the vertical movement amount between one frame is 2 pixels or more, the block processing unit 51 divides the pixels in the block into a set of 4 × 1 pixel units as shown in FIG. As shown, the pixel values p1 to p4 of each set are thinned out to be one pixel value (p1 in this example).

  When the vertical and horizontal movement amounts between one frame are both 2 pixels or more, the block processing unit 51 divides the pixels in the block into sets of 2 × 2 pixel units as shown in FIG. As shown in FIG. 8, the pixel number p1 to p4 of each set is thinned out to be one pixel value (p1 in this example).

  Since the block processing unit 51 performs such spatial thinning processing on each of the four supplied blocks, the data amount of each block is reduced to one pixel data amount for every four adjacent pixels. The data amount of the entire four blocks is reduced to ¼. The block processing unit 51 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  In the example of FIG. 4, the leftmost pixel value p1 in the 1 × 4 pixel in the example of FIG. 4, the uppermost pixel value p1 in the 4 × 1 pixel in the example of FIG. In the example, the pixel value p1 in the upper left corner of 2 × 2 pixels is set, but the pixel value may be set to any pixel value from p1 to p4. The pixel value may be set to a calculation using the pixel values p1 to p4, for example, an average value.

  Next, processing executed by the block processing unit 52 shown in FIG. 2 will be described. The block processing unit 52 shown in FIG. 2 has a total of N blocks (moving in the horizontal and vertical directions) at the same position in each of the consecutive N frames supplied from the block distribution unit 32 of the movement amount detection unit 12. A process of thinning out the number of frames (a temporal direction thinning process) is performed on N blocks in which both amounts are less than one pixel.

  Specifically, as shown in FIG. 9, the block processing unit 52 converts four blocks Bi at the same position in four consecutive frames F1 to F4 into one block (in this example, Then, the number of frames to be converted into the block Bi) of the frame F1 is thinned (the number of frames between the four frames is thinned). The block processing unit 52 supplies the output unit 14 with data (one block) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing.

  The block processing unit 53 is supplied from the block distribution unit 32 of the movement amount detection unit 12 and has a total of N blocks (the movement amount in the horizontal direction and the vertical direction is 1 in each of the consecutive N frames). The number of pixels is thinned out (spatial direction thinning process) and the number of frames is thinned out (time direction thinning process) for each of N blocks in the case of more than pixels and less than 2 pixels.

  Unlike the thinning-out process in the block processing unit 51, the block processing unit 53 converts each set of pixel values p1 to p4 into any two pixel values (in this example, as shown in FIGS. 10 and 11). In this case, the number of pixels to be reduced to p1, p3) is thinned out (thinning amount 2 is thinned out).

  That is, the block processing unit 51 thins out three types of pixels of 1 × 4, 4 × 1, or 2 × 2, while the block processing unit 53 performs two types of pixels of 1 × 2 or 2 × 1. Decimation of numbers is performed.

  In the frame number thinning process, the block processing unit 53 is different from the thinning process in the block processing unit 52, as shown in FIG. 12, at the same position in each of the four consecutive frames F1 to F4. Decreasing the number of frames to make a total of four blocks Bi any one of them (two blocks of frames F1 and F3 in the example in the figure) (decimating the number of frames between two frames) Do.

  Since the block processing unit 53 performs the spatial direction thinning process for reducing the data amount by 1/2 and the time direction thinning process for reducing the data amount by 1/2 to the four supplied blocks. As a result, the data amount of the four blocks is reduced to (1/2) × (1/2) = 1/4. The block processing unit 53 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  The output unit 14 configures and outputs, as stream data, data regarding the blocks obtained from the block processing units 51 to 53 and information indicating what processing has been performed on each block.

  Next, the configuration and processing of the restoration unit 15 illustrated in FIG. 1 will be described with reference to FIG. As shown in FIG. 13, the restoration unit 15 includes a block distribution unit 61, block expansion units 62 to 64, and a synthesis unit 65.

  The stream data obtained from the output unit 14 is input to the block distribution unit 61 of the restoration unit 15. This stream data includes converted (compressed) data generated by the processing up to immediately before and attribute data necessary for restoration. The attribute data includes information related to each block and information related to the specific contents of processing performed on each block. The specific contents of the processing applied to each block include the spatial direction thinning process, the time direction thinning process, the process including both the spatial direction thinning process and the time direction thinning process, and the spatial thinning process. Information about whether the thinning process executed is in the horizontal direction or in the vertical direction.

  Based on the processing contents of each block included in the input attribute information, the block distribution unit 61 sends the conversion data to be restored and the processing applied to each block to any of the block expansion units 62 to 64. Attribute information indicating specific contents is sent.

  Specifically, when the block is data processed by the block processing unit 51 in the moving image conversion apparatus 10 shown in FIGS. 1 and 2, the distribution unit 61 uses the block data thinned out to the block expansion unit 62. And send information such as the direction of spatial thinning.

  If the data is processed by the block processing unit 52 of the moving image conversion apparatus 10, similar information is sent to the block extension unit 63. If processed by the block processing unit 53, the same information is sent to the block extension unit 64. The information related to the processing applied to each block and the information related to each block may be any information as long as it is sufficient information for restoring a moving image.

  The block extension unit 62 performs an extension process on the data that has been subjected to the thinning process in the spatial direction in the block processing unit 51 of the moving image conversion apparatus 10. Based on the information regarding the direction of spatial thinning input from the distribution unit 61, the data expansion processing in the mode shown in FIGS. 14 to 16 is executed to reconfigure the block. Further, the reconstructed block is output to the synthesis unit 65.

  The processing shown in FIGS. 14 to 16 will be described. FIG. 14 shows processing performed by the block extension unit 62 of the restoration unit 15 when the block processing unit 51 of the moving image conversion apparatus 10 has been subjected to horizontal thinning processing (see FIGS. 3 and 4). For example, when the size of the original block is 8 × 8, data corresponding to ¼ of the 16 pixels is sent to the block extension unit 62. With respect to the data of each pixel, as shown in FIG. 14A, a process of expanding one pixel data into four pixels in the horizontal direction is performed to expand one pixel to 1 × 4 pixels. Note that the processing example shown in FIG. 14A is an example in which expansion is performed by arranging the pixel values of the input pixels as they are as pixel values for four pixels, but the values of other input pixels are included. You may arrange | position the result by the calculation based on a some pixel value.

  Subsequently, when the thinning is performed in the horizontal direction, the block extension unit 62 performs the process shown in FIG. That is, the process of expanding one pixel to 1 × 4 pixels is executed for 16 pixels input to the block expansion unit 62, and a set of 16 1 × 4 pixels executed for each of the 16 pixels is shown in FIG. ). By this processing, the block extension unit 62 restores an 8 × 8 block from the input 16 pixels, and outputs the result to the synthesis unit 65.

  FIG. 15 shows processing performed by the block extension unit 62 of the restoration unit 15 when the block processing unit 51 of the moving image conversion apparatus 10 has been subjected to vertical thinning processing (see FIGS. 5 and 6). For example, when the size of the original block is 8 × 8, data corresponding to ¼ of the 16 pixels is sent to the block extension unit 62. With respect to the data of each pixel, 1 pixel is expanded to 4 × 1 pixels by performing the processing of FIG. That is, as shown in FIG. 15A, one pixel is expanded to 1 × 4 pixels by performing processing for expanding one pixel data into four pixels in the vertical direction.

  Subsequently, when the thinning is performed in the vertical direction, the block extension unit 62 performs the process shown in FIG. That is, the process of expanding one pixel to 1 × 4 pixels is performed for 16 pixels input to the block expansion unit 62, and a set of 16 1 × 4 pixels executed for each of the 16 pixels is shown in FIG. ). By this processing, the block extension unit 62 restores an 8 × 8 block from the input 16 pixels, and outputs the result to the synthesis unit 65.

  FIG. 16 shows processing performed by the block extension unit 62 of the restoration unit 15 when the block processing unit 51 of the moving image conversion apparatus 10 has been subjected to thinning processing in the horizontal and vertical directions (see FIGS. 7 and 8). For example, when the size of the original block is 8 × 8, data corresponding to ¼ of the 16 pixels is sent to the block extension unit 62. With respect to the data of each pixel, 1 pixel is expanded to 2 × 2 pixels by performing the processing of FIG. That is, as shown in FIG. 16A, one pixel is expanded to 2 × 2 pixels by performing processing for expanding one pixel data into 2 × 2 pixels in the horizontal and vertical directions.

  Subsequently, when thinning is performed in the horizontal and vertical directions, the block extension unit 62 performs the processing shown in FIG. That is, the process of expanding one pixel to 2 × 2 pixels is executed for 16 pixels input to the block expansion unit 62, and the 16 sets of 2 × 2 pixels executed for each of the 16 pixels are shown in FIG. ). By this processing, the block extension unit 62 restores an 8 × 8 block from the input 16 pixels, and outputs the result to the synthesis unit 65.

  Next, processing of the block extension unit 63 will be described. The block expansion unit 63 performs an expansion process on the data that has been subjected to the thinning process in the time direction in the block processing unit 52 of the moving image conversion apparatus 10.

  As shown in FIG. 17, the block extension unit 63 generates block data of a plurality of frames continuous in the time axis direction based on the block data of one frame. Specifically, the block extension unit 63 has a counter with an initial value of 0, and this counter is incremented by 1 every time one frame is restored, and the value is 4 (= N (however, when the thinning amount m = 4) )) Is reset to 0 when it is reached.

  When data to be expanded is input from the data distribution unit 61 to the block expansion unit 63, the block expansion unit 63 stores the block in the memory of the data expansion unit 63, and the counter upper limit set according to the thinning amount Until reaching the value, block data corresponding to a plurality of frame data is generated as a copy of the block data stored in the memory, and output to the combining unit 65. In the example illustrated in FIG. 17, the thinning-out amount m = 4, and block data corresponding to four frames is generated from one block data and output to the combining unit 65.

  Next, processing of the block extension unit 64 will be described. The block extension unit 64 performs an extension process on the data that has been subjected to the thinning process in the spatial direction and the time direction in the block processing unit 53 of the moving image conversion apparatus 10. Either the spatial direction or temporal direction expansion processing may be executed first.

  The time direction expansion process executed in the block expansion unit 64 performs the process shown in FIG. Specifically, the block extension unit 64 has a counter with an initial value of 0, and this counter is incremented by 1 every time restoration of one frame is completed, and the value is 2 (= N / 2 (however, the amount of thinning in the time direction) When m = 2)), it is reset to 0.

  When the data to be expanded is input from the data distribution unit 61 to the block expansion unit 64, the block expansion unit 64 stores the block in the memory of the data expansion unit 64, and the counter upper limit set according to the thinning amount Until the value is reached, block data corresponding to a plurality of frame data is generated as a copy of the block data stored in the memory and output to the combining unit 65. In the example shown in FIG. 18, the time direction thinning-out amount m = 2, and block data corresponding to two frames is generated from one block data.

  Next, the spatial direction expansion processing executed in the block expansion unit 64 will be described. Based on information such as the direction of spatial thinning input from the distribution unit 61, the block expansion unit 64 performs data expansion processing in the manner shown in FIGS. 19 and 20, and reconfigures the block.

  The processing of FIGS. 19 and 20 will be described. FIG. 19 shows processing by the block extension unit 64 of the restoration unit 15 when the horizontal thinning processing is performed in the block processing unit 53 of the moving image conversion apparatus 10. For example, when the size of the original block is 8 × 8, data corresponding to ½ of 32 pixels is sent to the block extension unit 64. Corresponding to the data of each of the two pixels, the process of FIG. 19A is performed to expand the two pixels to 1 × 4 pixels. That is, as shown in FIG. 19 (a), two pixels are expanded to 1 × 4 pixels by performing processing for expanding one pixel data into two pixels in the horizontal direction.

  Subsequently, when the thinning is performed in the horizontal direction, the block extension unit 64 performs the process shown in FIG. That is, the process of extending 2 pixels to 1 × 4 pixels is executed for 32 pixels input to the block extension unit 64, and the 16 sets of 1 × 4 pixels executed for each of the 32 pixels are shown in FIG. ). By this processing, the block extension unit 64 restores an 8 × 8 block from the input 32 pixels, and outputs the result to the synthesis unit 65.

  FIG. 20 shows a process performed by the block extension unit 64 of the restoration unit 15 when the block processing unit 53 of the moving image conversion apparatus 10 performs the thinning process in the vertical direction. For example, when the size of the original block is 8 × 8, data corresponding to ½ of 32 pixels is sent to the block extension unit 64. Corresponding to the data of each of the two pixels, the process of FIG. 20A is performed to expand the two pixels to 4 × 1 pixels.

  Subsequently, when the thinning is performed in the vertical direction, the block extension unit 64 performs the process shown in FIG. That is, the process of expanding 2 pixels to 4 × 1 pixels is executed for 32 pixels input to the block expanding unit 64, and a set of 16 4 × 1 pixels executed for each 32 pixels is shown in FIG. ). By this processing, the block extension unit 64 restores an 8 × 8 block from the input 32 pixels, and outputs the result to the synthesis unit 65.

  When the block input from the block expansion units 62 to 64 reaches an amount that can represent one entire frame, the combining unit 65 restores one frame by appropriately arranging the blocks, and outputs the restored one frame. .

  As described above, the moving image conversion apparatus 10 shown in FIG. 1 performs processing for converting an input moving image into a moving image (compressed data) with a reduced data amount. This is a device that prevents the observer from perceiving image quality degradation due to a reduction in the amount of data by performing moving image conversion processing using the super-resolution effect realized based on the visual characteristics of the image.

  Specifically, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs image data conversion according to the optimal frame rate and spatial resolution. It supplies to the block processing parts 51-53 to perform, and it is set as the structure which performs the data conversion process of a different aspect in each block processing part 51-53, and makes an observer perceive image quality degradation by this structure. It realizes a moving image conversion process without any problem. Note that the super-resolution effect is a visual effect realized based on the visual characteristic that an observer perceives a sum of a plurality of images within a certain time as described above. It is considered that the temporal integration function and the sensory memory function in the function are caused in a complicated manner, and the moving image conversion apparatus 10 shown in FIG. 1 has a super-resolution effect caused by the temporal integration function. The moving image conversion process is used.

  The principles and explanations regarding human visual characteristics and super-resolution effect are described in detail in Japanese Patent Application No. 2003-412501. The conditions for generating the super-resolution effect described in Japanese Patent Application No. 2003-412501 will be described below.

In order for the super-resolution effect to occur when the number of pixels of the thinning amount m (pixel) is thinned, it is necessary to cancel all the primary to m−1 order folding components due to the thinning. k (= 1, 2,..., m−1) The condition that the next folding component is canceled is to satisfy the following expressions (Expression 1) and (Expression 2).

In the above equation, φ _t is a sampling position shift amount in thinning out the number of pixels, time t (= 0, 1T, 2T,...), Signal moving speed v, time interval (reciprocal of frame rate) T , Is a value defined by the following formula (formula 3).

  In the above equation, the case where the sampling position is shifted to the right is positive (this condition is different from the setting disclosed in Japanese Patent Application No. 2003-412501).

  If the above equations (Equation 1) and (Equation 2) are satisfied under the conditions of the thinning amount m and the small region movement amount v, the super-resolution effect occurs, and the deterioration of the image quality is hardly perceived by the observer.

[(2) Configuration of Moving Image Conversion Device that Performs Improved Thinning Process]
Similar to the moving image conversion apparatus described in Japanese Patent Application No. 2003-412501, the moving image conversion apparatus of the present invention described above is connected to the block distribution unit 32 of the movement amount detection unit 12 shown in FIG. Blocks (a total of N blocks at the same position in each of N frames) are supplied in units, and the movement amount of the P-th frame in the N blocks is transferred from the movement amount detection unit 31. Is supplied. The block distribution unit 32 supplies the supplied N blocks and the movement amount to any of the block processing units 51 to 53 that perform processing corresponding to the movement amount of the block processing unit 13.

  In the moving image conversion apparatus described in Japanese Patent Application No. 2003-412501, the block to be thinned out is supplied to one of the block processing units 51 to 53 and the block is thinned out. The same decimation processing, that is, decimation processing in one direction of space direction decimation, time direction decimation, or space and time direction decimation, is performed for a predetermined number of decimation unit frames: N frames It was applied uniformly.

  However, the movement of the subject in the block is not always uniform. That is, there are cases where a single block includes a plurality of movements, that is, subjects moving in different directions at different moving speeds, and cases where a stationary subject and a moving subject are mixed. However, in the moving image conversion apparatus described in Japanese Patent Application No. 2003-412501, the same thinning process is performed on all the pixels in the block according to the processing mode determined for a predetermined number of thinning process unit frames: N frames. Therefore, when the image elements in the block are taken into consideration, there may be a portion that is subjected to a thinning process that is not suitable for the original movement of the subject.

  When thinning processing that is not suitable for the original movement of the subject is performed, a great deterioration in image quality occurs. When one block includes both a part of a subject that moves at high speed and a background that hardly moves, particularly noticeable image quality degradation is observed.

  A specific example of image quality deterioration will be described with reference to FIG. FIG. 21 shows an image obtained by cutting a part of a certain moving image by four frames of frames 0 to 3 that are temporally continuous. The subject 71 indicated by diagonal lines has moved to the right, and the other part is the background. The four left frames are in the state before (A) thinning-out processing. The result of the time thinning performed on the block 72 indicated by the dotted lines of these four frames shows the four frames of the restored data after (B) thinning processing shown on the right side. In this time decimation, processing for copying data in frame 0 to other frames 1 to 3 was executed. It is assumed that portions other than the block 72 have been appropriately processed. As is apparent from the figure, when the four frames of the restored data after the thinning process (B) on the right side are seen, data of a part of the moving subject 71 that should not exist originally is located at the position of the block 72 over the four frames. Stays. As described above, as a result of performing the uniform thinning process, there is a case where image quality deterioration occurs such that the image configuration itself changes.

  Thus, when the amount of movement within a block changes between frames, the thinning process should be performed according to the amount of movement of each block in each frame. However, the moving image conversion apparatus 10 having the above-described configuration is configured to perform the same thinning process on all pixels in the block in accordance with a processing mode determined for a predetermined number of thinning-out processing unit frames: N frames. Therefore, as described with reference to FIG. 21, image quality deterioration may occur in which the image configuration itself changes.

  Although it is possible to change the type of thinning process to be applied according to the amount of movement of each frame, if it is configured to change the thinning processing mode for each frame, the compression efficiency is significantly reduced, as in the configuration described above. The feature that the amount of data can be reduced to 1 / N is lost.

  For example, in the frame structure composed of moving images shown in FIG. 21, the block 72 should be subjected to space thinning rather than time thinning over four frames. In the above-described thinning mode determination process of the moving image conversion apparatus, for example, when the amount of movement in one frame in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) is 2 pixels (pixels) or more, the spatial direction The configuration is such that thinning is performed, and the block that does not satisfy such a condition is configured to perform temporal direction thinning or combined processing of spatial direction and temporal direction thinning, but the horizontal direction between one frame (X-axis direction) ) Or the vertical direction (Y-axis direction) movement amount is less than 2 pixels (pixels), and if predetermined conditions are satisfied, the spatial direction thinning is configured to suppress image quality deterioration. Thinning processing and decoding processing are possible.

  In other words, when compared with the situation in which the image quality degradation that occurs when time decimation is performed in a block including a moving object, that is, a subject that should not be originally in the position as described with reference to FIG. The degree of deterioration is much less when space thinning is applied.

  In order to realize the application of the spatial thinning process to the dotted block 72 in the situation of FIG. 21, for example, there are blocks in which it is desirable to sequentially examine the thinning process to be performed on each block of 4 frames and apply the spatial thinning process. If it appears, a method of deciding to apply spatial thinning processing to the block of the four frames can be considered.

  However, in the example as shown in FIG. 22, there is a high possibility that time decimation is performed even if the above method is used. FIG. 22 shows a situation similar to FIG. That is, the subject 73 is moving in the right direction. Compared to FIG. 21, the ratio of the block 74 surrounded by the dotted line including the moving subject (hatched line) is small. In general, a motion vector detected by block matching is used as the movement amount. When there are a plurality of motions in the block, the motion vector detection result by block matching is often the motion vector of the subject having the largest area (number of pixels) among the plurality of subjects included in the block. .

  That is, in the example shown in FIG. 22, in the dotted line block 74, the magnitude of the motion vector obtained as a result of the detection of the movement amount in the frame 1 as well as the frame 1 to the frame 3 is almost 0 and is moving. Therefore, it is highly possible that the temporal thinning process is performed. However, even in the case of FIG. 22, if the time thinning process is performed on a block including a part of the moving subject, as in FIG. 21, (B) the restored data after the thinning process shown on the right side of FIG. As shown, the moving subject is fixedly displayed in the block 74. In other words, a part of the moving subject exists for several frames at a place where the moving subject should not exist, and the image quality degradation is very conspicuous.

  As described above, when the thinning process determined under a uniform condition is performed as in the above-described moving image conversion apparatus, time thinning is performed on blocks having a small moving speed or moving subject area. As a result, there is a possibility that the image quality is deteriorated such that the moving subject as shown in FIGS. 21 and 22 is fixedly displayed. The configuration and processing of the image conversion apparatus that solves such problems will be described below.

  In the image conversion apparatus of the present invention, for example, data reduction (compression) processing that suppresses image quality degradation that occurs at the boundary between regions that move at different speeds is realized. The configuration of the moving image conversion apparatus 100 will be described with reference to FIG. The moving image conversion apparatus 100 shown in FIG. 23 is similar to the moving image conversion apparatus 10 described above with reference to FIG. 1, and uses the super-resolution effect due to human visual characteristics, thereby reducing the image quality by reducing the data amount. The amount of data can be reduced so that the observer does not perceive the deterioration, and further, the data can be reduced while suppressing the image quality deterioration that occurs at the boundary between regions moving at different speeds. It is.

  A configuration of the moving image conversion apparatus 100 in FIG. 23 will be described. The moving image conversion apparatus 100 shown in FIG. 23 is similar to the moving image conversion apparatus 10 described above with reference to FIG. 1, and uses the super-resolution effect due to human visual characteristics, thereby reducing the image quality by reducing the data amount. The amount of data can be reduced so that the observer does not perceive the deterioration, and further, the data can be reduced while suppressing the image quality deterioration that occurs at the boundary between regions moving at different speeds. It is.

  The basic configuration of the moving image conversion apparatus 100 shown in FIG. 23 is substantially the same as the configuration described above with reference to FIG. The block dividing unit 110 divides each frame of the input moving image into blocks and supplies the blocks to the movement amount detecting unit 120. The movement amount detection unit 120 detects the movement amount of the block supplied from the block division unit 110 and transmits the block and the movement amount to the block processing unit 130. The block processing unit 130 performs a moving image conversion process corresponding to the movement amount on the block supplied from the movement amount detection unit 120 to reduce the data amount. The block processing unit 130 supplies the output unit 140 with data regarding the block whose data amount is reduced, which is obtained as a result of the processing. The output unit 140 collectively outputs the data regarding the blocks with the reduced data amount supplied from the block processing unit 130, for example, as stream data.

  Next, with reference to FIG. 24, details of each unit will be described. First, the block dividing unit 110 will be described. The frame of the moving image supplied to the moving image conversion apparatus 100 is input to the image storage unit 111 of the block dividing unit 110. The image storage unit 111 stores the input frame. The image storage unit 111 supplies the N frames to the block division unit 112 and the movement amount detection unit 121 every time the number of stored frames reaches N (N is a positive integer).

  The block division unit 112 divides each of N frames (consecutive N frames) supplied from the image storage unit 111 into blocks of a certain size (for example, 4 × 4, 8 × 8), and the movement amount detection unit 120. To the block distribution unit 122 and the movement amount detection unit 121.

  Next, the movement amount detection unit 120 will be described. The movement amount detection unit 121 of the movement amount detection unit 120 is supplied with a motion vector for the future of the N blocks supplied from the block division unit 112 of the block division unit 110 in the past from the image storage unit 111. , M (M is an integer) frames are referred to, for example, a block matching process between frames is executed and detected. A movement amount corresponding to the magnitude of the detected motion vector is defined as a movement amount A. The movement amount A is supplied to the block distribution unit 122 and the block processing units 131 and 133.

  With reference to FIG. 25, the process which the movement amount detection part 121 performs is demonstrated. The movement amount detection unit 121 detects a motion vector for the most future one among the N blocks supplied from the block division unit 112 of the block division unit 110, that is, N blocks corresponding to the number N of thinning-out processing unit frames. To do. In the example shown in FIG. 25, N = 4, and when processing is performed on four blocks at the same position corresponding to frames n to n + 3, the motion vector is detected based on the most future block, that is, the block belonging to frame n + 3. .

  The frame matching process between the frame n + 3 and the frame n + 3-M is executed by referring to the frame (frame n + 3-M) before M (M is an integer) stored in the image storage unit 111. A vector is obtained, and a movement amount corresponding to the magnitude of the motion vector is calculated. This movement amount A is supplied to the block distribution unit 122 and the block processing units 131 and 133. Further, the movement amount A is stored in a movement amount detection unit memory 123 which is a memory in the movement amount detection unit 121. The movement amount A stored in the memory is used in the time thinning process in the block processing unit 132. This process will be described later.

  The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction). The means for detecting the movement amount may be other than block matching.

  In the processing example described above, the movement amount detection unit 121 calculates the movement amount A by applying only the temporally final frame in the N frames as a thinning processing unit, and executes the spatial direction thinning process. This is a configuration provided to the block processing unit 131 and the block processing unit 133 that executes the spatial direction and temporal direction thinning processing, and the amount of movement obtained by applying the last frame in time among the N frames as the thinning processing unit In addition to A, the second movement amount (movement amount B) is calculated based on another frame, and the output movement amount provided to the block processing unit 131 and the block processing unit 133 is determined from the plurality of movement amounts. It is good also as a structure.

  This processing example will be described with reference to FIG. In the example illustrated in FIG. 26, the movement amount detection unit 121 is the most future among N blocks supplied from the block division unit 112 of the block division unit 110, that is, N blocks corresponding to the number N of thinning-out processing unit frames. Frame matching process between frame n + 3 and frame n + 3-M with reference to frame n + 3 of frame N and a frame (frame n + 3-M) before M (M is an integer) stored in image storage unit 111 Is executed to calculate the movement amount A, and the second movement amount B is calculated based on the blocks of other frames included in the N blocks corresponding to the thinning-out processing unit frame number N. In the example shown in the figure, the frame n + 1 is selected, the block matching process between the frames n + 1 and n + 1-M is executed, and the movement amount B is calculated.

  In this way, when the movement amount B is obtained in addition to the movement amount A, one output movement amount is generated from the movement amount A and the movement amount B by, for example, averaging processing, and the output movement amount is converted into the block distribution unit. 122, supplied to the block processing units 131 and 133. Also in this case, the movement amount A calculated based on the last frame is stored in the movement amount detection unit memory 123. The movement amount A stored in the memory is used in the time thinning process in the block processing unit 132.

  The block distribution unit 122 of the movement amount detection unit 120 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 112 in units of N, and the movement amount detection unit From 121, movement amount information calculated based on the movement amount detected by the above-described processing, that is, movement amount A or movement amount A and other movement amounts B is supplied. The block distribution unit 122 includes the N blocks supplied and the movement amount input from the movement amount detection unit 121 in the block processing units 131 to 133 that perform processing corresponding to the movement amount of the block processing unit 130. Supply to either.

  Specifically, the block distribution unit 122, based on the movement amount information supplied from the movement amount detection unit 121, the N blocks supplied from the block division unit 112 and the movement amount supplied from the movement amount detection unit 121. Determine where to enter information. That is, when determining to any one of the block processing units 131 to 133, inputting N blocks as the determination destination, and inputting the block to the block processing unit 131 or the block processing unit 133, the movement amount information is also input. To do.

  Specifically, when the amount of movement in the horizontal direction (X-axis direction) or the vertical direction (Y-axis direction) between one frame is 2 pixels (pixels) or more, N pieces of data supplied from the block division unit 112 The movement amount supplied from the block and movement amount detection unit 121 is output to the block processing unit 131 that executes the spatial direction thinning process.

  When the amount of movement in the horizontal direction and the vertical direction between one frame is less than 2 pixels and more than 1 pixel, the block distribution unit 122 outputs N blocks and the movement amount to the block processing unit 133. . When the amount of movement is other than that, the block distribution unit 122 supplies N blocks to the block processing unit 132.

  That is, the block distribution unit 122 determines an optimum frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 121, and performs block processing for performing processing for converting image data according to the frame rate and spatial resolution. The block images are distributed to the units 131 to 133. This condition for determining the distribution destination is merely an example, and the distribution destination may be determined under other conditions.

The generalization of the thinning mode determination process performed by the block distribution unit 122 is as follows. In the setting where the movement amount input from the movement amount detection unit 121 is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Movement amount: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Movement amount: When Vt <Vb,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the movement amount Vt is Vb ≦ Vt <Va,
Execution processes for a plurality of consecutive frames serving as a thinning process execution unit are determined as thinning processes in the spatial direction and the time direction.

  Next, details of the block processing unit 130 will be described. The block processing unit 130 is composed of the three block processing units 131 to 133 as described above. The block processing unit 131 executes the same processing as the block processing unit 51 of the moving image conversion device 10, and the block processing unit 133 executes the same processing as the block processing unit 53 of the moving image conversion device 10.

  That is, the block processing unit 131 supplies a total of N blocks (horizontal direction) at the same position in each of consecutive N (for example, N = 4) frames supplied from the block distribution unit 122 of the movement amount detection unit 120. (N blocks in the case where the vertical movement amount is 2 pixels or more), similarly, a process of thinning out the number of pixels according to the movement amount supplied from the block distribution unit 122 (spatial direction thinning process) I do.

  Specifically, when the amount of movement in the horizontal direction between one frame is larger than the amount of movement in the vertical direction and is 2 pixels (pixels) or more, the block processing unit 131 includes a block composed of, for example, 8 × 8 pixels. As shown in FIG. 3, the pixels in the block are divided into sets of 1 × 4 pixel units. Further, as shown in FIG. 4, the block processing unit 131 sets the pixel values p1 to p4 of each set of 1 × 4 pixels to one of the pixel values (in this example, p1) of the number of pixels set. Thinning (thinning of the number of pixels between four pixels) (thinning of the thinning amount 4) is performed.

  When the amount of movement in the vertical direction between one frame is larger than the amount of movement in the horizontal direction and is 2 pixels or more, the block processing unit 131 converts the pixels in the block into units of 4 × 1 pixels as shown in FIG. As shown in FIG. 6, the pixel values p1 to p4 of each set are thinned out to one pixel value (p1 in this example).

  Since the block processing unit 131 performs such spatial thinning processing on each of the four supplied blocks, the data amount of each block is reduced to one pixel data amount for every four adjacent pixels. The data amount of the entire four blocks is reduced to ¼. The block processing unit 131 supplies data about four blocks whose data amount has been reduced to ¼ to the output unit 140.

  The block processing unit 133 also supplies a total of N blocks (the amount of movement in the horizontal and vertical directions) at the same position in each of the consecutive N frames supplied from the block distribution unit 122 of the movement amount detection unit 120. The number of pixels is thinned out (spatial direction thinning process) and the number of frames is thinned out (time direction thinning process) for N blocks in the case where the number of pixels is 1 pixel or more and less than 2 pixels.

  Unlike the thinning-out process in the block processing unit 131, the block processing unit 133 converts each set of pixel values p1 to p4 into any two pixel values (in this example, as shown in FIGS. 10 and 11). In this case, the number of pixels to be reduced to p1, p3) is thinned out (thinning amount 2 is thinned out).

  That is, the block processing unit 51 thins out three types of pixels of 1 × 4, 4 × 1, or 2 × 2, but the block processing unit 133 has two types of pixels of 1 × 2 or 2 × 1. Decimation of numbers is performed.

  In the thinning-out processing for the number of frames, the block processing unit 133 includes a total of four blocks Bi at the same position in each of the four consecutive frames F1 to F4, as shown in FIG. (2 in the example in the figure, two blocks of frames F1 and F3) is thinned out (the number of frames between two frames is thinned out).

  Since the block processing unit 133 performs the spatial direction thinning process for reducing the data amount by 1/2 and the time direction thinning process for reducing the data amount by 1/2 to the supplied four blocks. As a result, the data amount of the four blocks is reduced to (1/2) × (1/2) = 1/4. The block processing unit 133 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 140.

  As described above, the block processing unit 131 that performs spatial direction thinning performs the same processing as the block processing unit 51 of the moving image conversion apparatus 10, and the block processing unit 133 that performs spatial direction and temporal direction thinning is a moving image. The same processing as that of the block processing unit 53 of the conversion device 10 is executed.

  The block processing unit 132 that executes the time direction thinning-out process executes processing different from that of the block processing unit 52 of the moving image conversion apparatus 10 described above. Hereinafter, the configuration and processing details of the block processing unit 132 will be described.

  The block processing unit 132 has a total of N blocks (the amount of movement in both the horizontal direction and the vertical direction is 1 in each of the consecutive N frames supplied from the block distribution unit 122 of the movement amount detection unit 120). A process of thinning out the number of frames (time direction thinning process) is performed on N blocks in the case of less than pixels.

  With reference to FIG. 27, a detailed configuration and process of the block processing unit 132 that performs the time direction thinning process will be described. The block processing unit 132 includes a time direction decimation processing frame determination unit 151 and a time direction decimation processing unit 152.

  When the number of thinning-out processing unit frames: N is set to N = 4, the block time direction thinning-out processing unit 152 selects any one of the four blocks Bi at the same position in each of the four consecutive frames F1 to F4. Thinning out the number of frames in which only the image data of the block is left (thinning out the number of frames between four frames) is performed. A specific example of the time direction thinning process will be described with reference to FIGS. FIG. 28 shows an example of time direction thinning processing in which only the frame F1 is left out of four consecutive frames F1 to F4 included in one processing unit when N is N = 4. 29 shows an example of time direction thinning processing in which only the frame F2 is left, FIG. 30 shows an example of time direction thinning processing in which only the frame F3 is left, and FIG. 31 shows an example of time direction thinning processing in which only the frame F4 is left. It is. Which frame is left is determined for each set of blocks in the thinning-out processing unit frame by the time direction thinning-out processing frame determination unit 151 shown in FIG.

  The time direction thinning-out frame determination unit 151 determines a frame to be left in the time direction thinning-out processing as a selection frame for a set of N-frame blocks currently being processed, and determines the selected selection frame to the time direction thinning-out processing unit 152 Output information.

  The time direction thinning process frame determination unit 151 inputs movement amount information corresponding to a past frame at the same position as the block currently being processed, that is, a movement amount corresponding to the past frame (movement amount A) from the movement amount detection unit 121. To do. As described with reference to FIGS. 25 and 26, the movement amount detection unit 121 calculates the movement amount A for each block set of each thinning-out processing unit frame, and stores this in the movement amount detection unit memory 123. Storing. The time direction thinning process frame determination unit 151 acquires, from the movement amount detection unit memory 123, a past frame corresponding movement amount that is movement amount information (movement amount A) corresponding to a past frame at the same position as the block currently being processed. . Note that the past frame indicates the frame n−1 when the N frame currently being processed is a frame n, a frame n + 1,..., A frame n + N−1.

  That is, in the movement amount detection unit memory 123 of the movement amount detection unit 121, the block processing unit 132 that executes the temporal direction thinning process has a thinning processing unit immediately before the thinning processing unit frame group to which the block currently being processed belongs. The movement amount A of the block corresponding to the last frame of the frame group is stored as the movement amount corresponding to the past frame, and the time direction thinning processing frame determination unit 151 of the block processing unit 132 corresponds to the previous processing unit. The movement amount corresponding to the past frame (movement amount A) is acquired, and the selection frame to be left as data is determined in the time direction thinning process.

Based on the past frame correspondence movement amount information acquired from the movement amount detection unit memory 123 of the movement amount detection unit 121, the time direction thinning processing frame determination unit 151 determines the frames to be left in the time direction thinning process as follows.
(1) If the absolute value V of the movement amount (movement amount A) calculated based on the previous frame included in the previous processing unit is less than the reference speed U = 1 pixel / frame, leave the frame n. Frame.
(2) In cases other than the above, it is assumed that the frame n + N−1 is left.

  That is, when the movement amount corresponding to the past frame (movement amount A) is a movement amount smaller than a predetermined threshold value [reference speed U = 1 pixel / frame], the oldest frame among the N frames being processed is data. If the past movement amount A is a movement amount equal to or greater than a predetermined threshold value [reference speed U = 1 pixel / frame], the most future frame of N frames is used as data. The selection frame to be left is assumed. The reference speed U is 1 pixel / frame in this example, but may be other speeds.

  In this way, the block processing unit 132 dynamically changes the frame to be left in the time direction thinning based on the movement amount of the past block, and executes the time thinning process in a different mode.

  The effect of the process of changing the mode of the time direction thinning process according to the magnitude of the movement amount detected based on the past frame will be described with reference to FIGS. 32 to 34. In FIGS. 32 to 33, as in FIGS. 21 and 22 described above, the hatched portion is the moving subject 91 and the other portions are the background.

  FIG. 32 shows images of each frame of the restored data (A) before the thinning process and (B) after the thinning process. The thinning process executed here is a thinning process in the time direction. When N = 4, one frame data is left as a selected frame from four frames, and the data of the selected frame is transferred to another frame in the restoration process. The process to copy is executed. This process is executed individually for each block included in the frame.

  FIG. 32 shows (A) frames 0 to 3 belonging to the processing unit (N = 4) and a frame 1 immediately before frame 0 and frame-1 as frames before the thinning process. (B) Frames 0 to 3 belonging to the processing unit (N = 4) are shown as frames after the thinning process.

  As described above with reference to FIG. 27 and others, in the moving image conversion apparatus 100 according to the present invention, when a time direction thinning process is performed on a certain block, it is detected based on the last frame belonging to the previous thinning process unit. The movement amount A is acquired from the movement amount detection unit memory 123, and based on the movement amount A, the selection frame to be left in the time direction thinning is determined.

  In the case of the image data example shown in FIG. 32, (A) the moving subject occupies most of the block 92 of frame-1 before the thinning process. In such a configuration, it can be expected that the movement amount calculated from the block 92 will be a large value (1 pixel / frame or more).

  When the block processing unit 132 that executes the temporal direction thinning process determines which block data of the four frame data 200 to 203 of frame 0 to frame 3 to be thinned out is to be left as a selected frame The determination process based on the movement amount calculated from the block 92 is performed.

As described above, the time direction decimation processing frame determination unit 151
(1) When the absolute value V of the movement amount (movement amount A) corresponding to the past frame calculated based on the past frame included in the previous processing unit is less than the reference speed U = 1 pixel / frame, Let n be a frame that leaves n.
(2) In cases other than the above, it is assumed that the frame n + N−1 is left.
According to these processes, the selected frame is determined.

  In the example of FIG. 32, the amount of movement calculated from the block 92 is a large value (1 pixel / frame or more). Therefore, the block processing unit 132 that performs the temporal direction thinning process performs the thinning process target frame 0 to frame 3. Among the four frames 200 to 203, time thinning is performed to leave the block 203 belonging to the frame 3 which is the most future frame.

  The result of restoring this thinning-out processing data is (B) restored data after thinning-out processing shown in the right part of FIG. In all of the blocks 210 to 213 in each of the frames 0 to 3, (A) data similar to the block 203 of the frame 3 before the thinning process, that is, data composed of image data including only background data is restored. . Although a part of the moving subject 91 (shaded line) is missing in frame 0, there is only one piece of moving subject over four frames as in the restoration data after the thinning process in FIG. 22 described above. It does not stay in one place, image degradation is suppressed, and good moving image data restoration close to the original image is realized.

  Further, a processing example for moving image data including a moving subject having different motion will be described with reference to FIGS. 33 and 34. In FIGS. 33 and 34, frames 0 to 3 of the moving image in which the moving subject 93 indicated by diagonal lines is moving leftward are shown. FIG. 33 shows a case where the conventional time direction thinning process, that is, the process of selecting the frame to be left based on the movement amount of the past frame is not executed, and FIG. 34 shows the process of selecting the frame to be left based on the movement amount of the past frame. Image data when executed is shown.

  In the processing example of FIG. 33, if time thinning is performed with the blocks 223 of the frame 3 left in the time direction thinning for the blocks 220 to 223 of the frames 0 to 3, (B) thinning shown in the right part of FIG. 33. In the blocks 230 to 233 included in the frames 0 to 3 of the post-processing restoration data, the image data including a part of the moving subject is displayed in the same manner as the data of the block 223 of the frame 3. As a result, for example, in frame 0 or the like, it is displayed inside the partial block 230 of the moving subject 93 that should be suddenly away.

  On the other hand, when the above-described process, that is, the process of selecting a frame to be left in the time direction thinning process based on the movement amount of the past frame, the result shown in FIG. 34 is obtained.

  In the case of the image data example shown in FIG. 34, (A) the moving subject is not included in the block 95 of frame-1 before the thinning process. In such a configuration, the movement amount calculated from the block 95 can be expected to be a small value (less than 1 pixel / frame).

  When the block processing unit 132 that executes the temporal direction thinning process determines which block data of the four frame data 200 to 203 of frame 0 to frame 3 to be thinned out is to be left as a selected frame Then, the determination process based on the movement amount calculated from the block 95 is performed.

As described above, the time direction decimation processing frame determination unit 151
(1) If the absolute value V of the movement amount (movement amount A) calculated based on the previous frame included in the previous processing unit is less than the reference speed U = 1 pixel / frame, leave the frame n. Frame.
(2) In cases other than the above, it is assumed that the frame n + N−1 is left.
According to these processes, the selected frame is determined.

  In the example of FIG. 34, the amount of movement calculated from the block 95 is a small value (less than 1 pixel / frame). Among the four frames of 240 to 243, the time direction decimation is performed to leave the block 240 belonging to the frame 0 which is the oldest frame in time.

  The result of restoring this thinned-out processing data is (B) restored data after the thinning-out processing shown in the right part of FIG. In all of the blocks 250 to 253 of each of the frames 0 to 3, (A) data similar to the block 240 of the frame 0 before the thinning process, that is, data constituted by image data including only background data is restored. . Although a part of the moving subject 93 (shaded line) is missing in the frame 3, there is one moving subject fragment over four frames as in the restored data after the thinning process in FIG. 22 described above. It does not stay in one place, image deterioration is suppressed, and good moving image data restoration close to the original image is realized.

  The block processing unit 132 supplies data (one block) for N blocks whose data amount is reduced to 1 / N by such time direction thinning processing to the output unit 140. Since the output unit 140 and the restoration unit 150 are configured to execute the same processing as the output unit 14 and the restoration unit 15 of the moving image conversion apparatus 10 described above, description thereof is omitted.

  FIG. 35 is a flowchart for explaining the processing sequence of the block processing unit 132 that executes the time direction thinning processing. The processing procedure of the block processing unit 132 will be described according to this flowchart.

First, in step S <b> 101, the block processing unit 132 inputs block data to be subjected to a time direction thinning process from the movement amount detection unit 120. Basically, the blocks input to the block processing unit 132 are blocks for which the movement amount detection unit 120 has determined that the time direction thinning processing is optimal. Specifically, for example, the movement amount detection unit 120 sets the movement amount to Vt, two predetermined thresholds Va and Vb, where Va> Vb.
Movement amount: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Movement amount: When Vt <Vb,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the movement amount Vt is Vb ≦ Vt <Va,
The execution processing for a plurality of consecutive frames that are thinning processing execution units is executed as processing for determining the thinning processing in the spatial direction and the temporal direction,
In the block processing unit 132 that executes the time direction thinning process,
Movement amount: A block included in the thinning-out processing unit frame determined to satisfy Vt <Vb is input. Here, it is assumed that the number of processing unit frames is N, and N blocks belonging to frames n to n + N are input.

  In step S102, the movement amount A detected based on the corresponding block of the thinning-out processing unit before one processing unit is input. As described with reference to FIGS. 25 to 27, the movement amount detection unit 120 calculates the movement amount A based on the block belonging to the last frame of each processing unit, and stores this in the movement amount detection unit memory 123. Storing. The block processing unit 132 inputs the movement amount A of the previous processing unit stored in the memory 123.

In step S103, the movement amount A is compared with a predetermined threshold value U.
As described above, the time direction decimation processing frame determination unit 151 of the block processing unit 132
(1) When the absolute value V of the movement amount (movement amount A) corresponding to the past frame calculated based on the past frame included in the previous processing unit is less than the reference speed U = 1 pixel / frame, Let n be a frame that leaves n.
(2) In cases other than the above, it is assumed that the frame n + N−1 is left.
According to these processes, the selected frame is determined.

  If it is determined in step S103 that the past frame correspondence movement amount A ≧ the threshold value U, the process proceeds to step S104. This process corresponds to the process (1). That is, in step S104, the time direction thinning process frame determination unit 151 sets the block corresponding to this frame as the selected block as the last frame of the process frame, that is, the frame that leaves frame n + N-1.

  On the other hand, if it is determined in step S103 that the past frame correspondence movement amount A ≧ threshold value U is not satisfied, the process proceeds to step S105. This process corresponds to the process (2) above. That is, in step S105, the time direction thinning-out processing frame determination unit 151 sets a block corresponding to this frame as a selected block as a head frame of the processing frame, that is, a frame that leaves the frame n.

  In step S106, the time direction decimation processing unit 152 in the block processing unit 132 executes time direction decimation processing that leaves only block data corresponding to the selected frame, and ends the process.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  According to the configuration of the present invention, the optimum compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject is determined, and the data conversion processing is performed in the optimal mode for each region according to the determined mode. By doing so, it becomes possible to perform compression and decompression with very little quality deterioration. In particular, in the configuration of the present invention, when the time direction decimation is executed, the past frame correspondence movement amount detected based on the block belonging to the previous decimation unit frame to be processed is input, and the past frame Since the block-corresponding frame to be left as data in the time direction decimation is determined based on the corresponding movement amount, and the time-direction decimation process is performed to leave the block data corresponding to the determined frame, it is not originally present in the restored image. Image deterioration such as displaying a moving subject can be reduced, and data conversion that enables high-quality image compression and restoration is realized.

  In particular, according to the configuration of the present invention, based on the movement amount corresponding to the past frame, the block correspondence frame to be left as data in the time direction thinning is determined, and the time direction thinning process for leaving the determined frame correspondence block data is executed. Because of the configuration, when there are very few moving subjects in the time direction decimation processing target block, there is no display of moving subjects that do not exist in the restored image, enabling high-quality image compression and restoration. Data conversion is realized.

It is a figure which shows the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure which shows the detail of the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a block diagram which shows the structure of the decompression | restoration part which performs decompression | restoration of the compressed data produced | generated by the thinning process. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the problem at the time of performing a time direction thinning-out process. It is a figure explaining the problem at the time of performing a time direction thinning-out process. It is a figure explaining the structural example of the image conversion apparatus which performs the data conversion using the super-resolution effect of this invention. It is a figure which shows the detail of the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining an example of the movement amount detection process which a movement amount detection part performs. It is a figure explaining an example of the movement amount detection process which a movement amount detection part performs. It is a figure explaining the structure and process of a block process part which performs a time direction thinning-out process. It is a figure explaining an example of the execution aspect of a time direction thinning-out process. It is a figure explaining an example of the execution aspect of a time direction thinning-out process. It is a figure explaining an example of the execution aspect of a time direction thinning-out process. It is a figure explaining an example of the execution aspect of a time direction thinning-out process. It is a figure explaining the process example of the time direction thinning-out process according to the process of this invention. It is a figure explaining the process example of a time direction thinning-out process. It is a figure explaining the process example of the time direction thinning-out process according to the process of this invention. It is a figure which shows the flowchart explaining the process sequence of the block process part which performs a time direction thinning-out process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image converter 11 Block division part 12 Movement amount detection part 13 Block processing part 14 Output part 15 Restoration part 21 Image storage part 22 Block division part 31 Movement amount detection part 32 Block distribution part 51-53 Block processing part 61 Block distribution part 62 to 64 Block expansion unit 65 Composition unit 71 Moving subject 72 Block 73 Moving subject 74 Block 91 Moving subject 92 Block 93 Moving subject 95 Block 100 Image conversion device 110 Block dividing unit 111 Image accumulating unit 112 Block dividing unit 120 Movement amount detecting unit 121: Movement amount detection unit 122: Block distribution unit 123: Movement amount detection unit memory 130: Block processing unit 131-133: Block processing unit 140: Output unit 150: Restoration unit 151: Time direction decimation processing frame determination unit 152: Time direction decimation Processing section 200 to 203 block 210-213 block 220 to 223 block 230 to 233 block 240-243 block 250 to 253 block

Claims (9)

A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit for detecting a subject movement amount in each block divided by the block division unit;
The movement amount information detected by the movement amount detection unit is input, and based on the movement amount information, the spatial direction thinning process, the time direction thinning process, or the spatial direction, A block distribution unit that determines which one of the time direction thinning processes is to be executed, and executes a process of supplying the block processing target block data to the block processing unit that performs the thinning process;
A block processing unit that inputs block data and executes either a spatial direction thinning process, a temporal direction thinning process, or a spatial direction and a temporal direction thinning process according to a movement amount corresponding to the input block;
The block processing unit
Corresponding to the past frame detected by the movement amount detection unit based on the block corresponding to the processing target block and the pixel position and belonging to the previous thinning unit frame of the processing target when executing the temporal direction thinning process on the input block A frame determination unit that inputs a movement amount, compares the movement amount corresponding to the past frame with a predetermined threshold U, and determines a block-corresponding frame to be left as data in time direction thinning according to the comparison result as a different frame;
A time direction decimation processing unit that executes a time direction decimation process that leaves block data corresponding to the frame determined by the frame determination unit;
I have a,
The frame determination unit
When the movement amount corresponding to the past frame is equal to or greater than a predetermined threshold value U, that is,
Movement amount corresponding to past frame ≧ threshold value U
If
Determining the last frame (n + N−1) of the frames (n to n + N−1) to which the time direction thinning process target block belongs as a block corresponding frame to be left as data in the time direction thinning;
Movement amount corresponding to past frame <threshold value U
If
It is a configuration that executes a process of determining the first frame (n) of the frame (n to n + N-1) to which the time direction thinning process target block belongs as a block corresponding frame to be left as data in the time direction thinning.
The time direction decimation processing unit
A moving image conversion apparatus having a configuration in which a time direction thinning process is performed to leave block data corresponding to a frame determined by the frame determination unit. The movement amount detector
Processing for detecting a movement amount based on a block belonging to the last frame (n + N−1) from a plurality of N blocks corresponding to pixel positions in a plurality of continuous frames (n to n + N−1) serving as a thinning process execution unit. Run
The frame determination unit
The movement amount detection unit inputs the movement amount detected based on the block belonging to the last frame (n + N−1) as the movement amount corresponding to the past frame, and based on the input movement amount corresponding to the past frame, The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to execute a process of determining a block corresponding frame to be left as data. The movement amount detector
Based on a block belonging to the last frame (n + N−1), a movement amount A is detected from a plurality of N blocks corresponding to pixel positions in a plurality of consecutive frames (n to n + N−1) serving as a thinning process execution unit. As processing is performed,
A movement amount B is detected based on a block belonging to a frame other than the last frame (n + N−1), and an output movement amount to be output to the block processing unit is calculated by averaging the movement amount A and the movement amount B. The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to execute the processing to be performed. The block distributor is
Based on the movement amount information detected by the movement amount detection unit, it is configured to determine a thinning processing mode,
In the setting where the amount of movement is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Movement amount: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Movement amount: When Vt <Vb,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the movement amount Vt is Vb ≦ Vt <Va,
2. The moving image conversion apparatus according to claim 1, wherein execution processing for a plurality of continuous frames serving as a thinning processing execution unit is determined as thinning processing in a spatial direction and a time direction. A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
The movement amount information detected in the movement amount detection step is input, and on the basis of the movement amount information, a spatial direction thinning process, a time direction thinning process, or a spatial direction is performed for a plurality of consecutive frames that are thinning process execution units. And a block distribution step for determining which of the time direction thinning processing is to be executed, and executing processing for supplying the thinning target block data to the block processing unit that performs the thinning processing;
A block processing step for inputting block data and performing either a spatial direction thinning process, a temporal direction thinning process, or a spatial direction and a temporal direction thinning process according to a movement amount corresponding to the input block;
The block processing step includes:
When executing the temporal direction decimation process for the input block, the past frame corresponding movement amount detected based on the block corresponding to the process target block and the pixel position and belonging to the previous decimation process unit frame of the process target is input. A frame determination step of comparing the movement amount corresponding to the past frame with a predetermined threshold U, and determining a block-corresponding frame to be left as data in the time direction thinning according to the comparison result as a different frame;
Possess a temporal decimation processing step of executing the frame leaving the frame corresponding block data determined at decision step temporal decimation processing,
In the frame determination step,
When the movement amount corresponding to the past frame is equal to or greater than a predetermined threshold value U, that is,
Movement amount corresponding to past frame ≧ threshold value U
If
Determining the last frame (n + N−1) of the frames (n to n + N−1) to which the time direction thinning process target block belongs as a block corresponding frame to be left as data in the time direction thinning;
Movement amount corresponding to past frame <threshold value U
If
Executing the process of determining the first frame (n) of the frame (n to n + N−1) to which the time direction decimation processing target block belongs as a block corresponding frame to be left as data in the time direction decimation;
In the time direction thinning-out processing step,
A moving image conversion method characterized by executing a time direction thinning process for leaving block data corresponding to the frame determined in the frame determining step . The movement amount detection step includes:
Processing for detecting a movement amount based on a block belonging to the last frame (n + N−1) from a plurality of N blocks corresponding to pixel positions in a plurality of continuous frames (n to n + N−1) serving as a thinning process execution unit. Run
The frame determining step includes:
The movement amount detected based on the block belonging to the last frame (n + N−1) in the movement amount detection step is input as the movement amount corresponding to the past frame, and the data in the time direction decimation is based on the input movement amount corresponding to the past frame. 6. The moving image conversion method according to claim 5 , wherein a process for determining a block corresponding frame to be left as is executed. The movement amount detection step includes:
Based on a block belonging to the last frame (n + N−1), a movement amount A is detected from a plurality of N blocks corresponding to pixel positions in a plurality of consecutive frames (n to n + N−1) serving as a thinning process execution unit. As processing is performed,
A movement amount B is detected based on a block belonging to a frame other than the last frame (n + N−1), and an output movement amount to be output to the block processing unit is calculated by averaging the movement amount A and the movement amount B. The moving image conversion method according to claim 5 , wherein the processing is performed. The block distribution step includes:
A step of determining a thinning processing mode based on the movement amount information detected in the movement amount detection step;
In the setting where the amount of movement is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Movement amount: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Movement amount: When Vt <Vb,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the movement amount Vt is Vb ≦ Vt <Va,
6. The moving image conversion method according to claim 5 , wherein execution processing for a plurality of continuous frames serving as a thinning processing execution unit is determined as thinning processing in a spatial direction and a time direction. A computer program for executing data conversion processing of moving image data on a computer,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
The movement amount information detected in the movement amount detection step is input, and on the basis of the movement amount information, a spatial direction thinning process, a time direction thinning process, or a spatial direction is performed for a plurality of consecutive frames that are thinning process execution units. And a block distribution step for determining which of the time direction thinning processing is to be executed, and executing processing for supplying the thinning target block data to the block processing unit that performs the thinning processing;
A block processing step for inputting block data and performing either a spatial direction thinning process, a temporal direction thinning process, or a spatial direction and a temporal direction thinning process according to a movement amount corresponding to the input block;
The block processing step includes:
When executing the temporal direction decimation process for the input block, the past frame corresponding movement amount detected based on the block corresponding to the process target block and the pixel position and belonging to the previous decimation process unit frame of the process target is input. A frame determination step of comparing the movement amount corresponding to the past frame with a predetermined threshold U, and determining a block-corresponding frame to be left as data in the time direction thinning according to the comparison result as a different frame;
Possess a temporal decimation processing step of executing the frame leaving the frame corresponding block data determined at decision step temporal decimation processing,
In the frame determination step,
When the movement amount corresponding to the past frame is equal to or greater than a predetermined threshold value U, that is,
Movement amount corresponding to past frame ≧ threshold value U
If
Determining the last frame (n + N−1) of the frames (n to n + N−1) to which the time direction thinning process target block belongs as a block corresponding frame to be left as data in the time direction thinning;
Movement amount corresponding to past frame <threshold value U
If
Executing the process of determining the first frame (n) of the frame (n to n + N−1) to which the time direction decimation processing target block belongs as a block corresponding frame to be left as data in the time direction decimation;
In the time direction thinning-out processing step,
A computer program for executing a time direction thinning process for leaving block data corresponding to the frame determined in the frame determining step .

Images