JP2006005904A - Dynamic image converting apparatus and method, dynamic image restoring apparatus and method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and method for preventing reduction of image quality in compression and restoration of dynamic image data. <P>SOLUTION: The amount of movement of an object in a block constituting the dynamic image data is detected, and sample point fixed type spatial decimation is performed in a mode or a sample point moving type spatial decimation is performed, on the basis of the detected amount of movement. When the evaluation value of image quality corresponding to the moving speed of the object equivalent to information on the amount of movement obtained by analyzing blocks is lower than a predetermined evaluation value, the sample point moving type spatial decimation is performed. Consequently, the virtual moving speed of the object can be set and the spatial decimation is made possible, by which generation of super resolution effect is made possible, thereby achieving data conversion without causing significant degradation in image quality. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program. In particular, in data conversion executed as data compression processing of moving image data, high-quality data conversion is possible while suppressing image quality deterioration, and high-quality image data restoration from compressed converted data is possible. The present invention relates to a conversion device, a moving image restoration device and method, and a computer program.

  The moving image data is subjected to data conversion for reducing the amount of data, that is, compression processing, when it is stored in a storage medium such as a hard disk or DVD or distributed via a network. In particular, in recent years, the quality of moving image data has been improved, for example, HD (High Definition) data has been improved, and the amount of data has increased rapidly with the improvement of the quality of this data. Under such circumstances, many studies and studies have been made on techniques for improving the compression efficiency in the compression and decompression processing of moving image data and preventing the quality degradation of the decompressed data.

  As a moving image compression processing method, for example, thinning processing of pixels constituting an image frame constituting moving image data, that is, thinning processing in a spatial direction, thinning processing of a frame rate, that is, thinning processing in a time direction, and the like are known. Yes.

  By reducing the amount of data by such data conversion, there is an advantage that saving to a storage medium and data transfer via a network are efficiently performed. However, when the compressed data is restored and reproduced, there is a problem that the image quality deteriorates. In particular, when the original data is a high-definition image, the quality deterioration becomes more remarkable.

Various studies have been made on how to reduce such image quality deterioration. For example, Patent Document 1 discloses an image compression processing configuration in which a parameter based on image brightness information is set and the compression mode is changed according to the image brightness. Patent Document 2 discloses a compression processing configuration in which a screen is divided into a plurality of areas and the compression mode is changed for each area.
JP 2003-169284 A Japanese Patent Laid-Open No. 2002-27466

  As described above, in some conventional techniques, a configuration is disclosed in which the compression mode is changed based on various characteristics obtained from the processing target image and the data quality is improved. In the method, the compressed image data is restored and reproduced, and image quality deterioration is not sufficiently suppressed.

  In particular, moving image data is data including various types of motion data such as a portion where the subject's motion is large, a small portion, a portion where the subject is stationary, etc., and it is appropriate to correspond to these different motions without losing data quality. There is a problem that it is difficult to prevent the deterioration of quality when compression processing is performed and data restoration and reproduction are performed.

  The present invention has been made in view of such problems, and determines the optimum compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject, and optimizes each region according to the determined mode. An object of the present invention is to provide a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program that enable compression and restoration with extremely little quality deterioration by adopting a configuration for performing data conversion processing in various modes. .

The first aspect of the present invention is:
A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit for detecting a subject movement amount in each block divided by the block division unit;
A block processing unit that inputs the block data divided by the block dividing unit and the movement amount information detected by the movement amount detection unit, and executes block data thinning processing;
The block processing unit
Based on the movement amount information, a decimation mode determination unit that determines a decimation mode of whether to perform sampling point fixed type spatial direction decimation or sample point movement type decimation in the spatial direction;
In accordance with the determination of the thinning mode determination unit, a thinning process execution unit that performs either a sample point fixed type spatial direction thinning process or a sample point movement type spatial direction thinning process;
The moving image converting apparatus is characterized by having a configuration including:

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning mode determination unit further determines a sample point moving direction according to the progress of the frame as a processing mode in the sampling point movement type spatial direction thinning. The thinning process execution unit is configured to execute a sample point movement type spatial direction thinning process with a sample point movement process in the direction determined by the thinning mode determination unit.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning mode determination unit corresponds to the subject moving speed generated based on the processing data when the sampling point fixing thinning processing is executed and the image quality evaluation value. If the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit is greater than or equal to a predetermined reference evaluation value If execution of the fixed point thinning process is determined and the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit is less than a predetermined reference evaluation value, the sample point is moved The present invention is characterized in that the execution of the thinning process of the mold is determined.

  Furthermore, in an embodiment of the moving image conversion apparatus of the present invention, the thinning mode determination unit further includes a memory for holding a thinning mode determination result for a past frame by the thinning mode determination unit, and the sampling point is fixed. This is a configuration for performing the thinning mode determination process according to the correspondence data between the subject moving speed and the image quality evaluation value generated based on the processing data when the thinning process is executed, and the thinning mode determination result for the past frame. The thinning mode determination result for the frame is a sampling point fixed thinning process, and the image quality evaluation value corresponding to the subject moving speed corresponding to the moving amount information detected by the moving amount detecting unit is equal to or higher than a predetermined reference evaluation value T1. Is determined to execute the thinning process of the fixed sampling point type, and the thinning mode determination result for the past frame is the sampling point. If the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit is less than the predetermined reference evaluation value T1, the sampling point movement type thinning is performed. The execution of the process is determined, and the image quality evaluation corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit, in which the thinning mode determination result for the past frame is a sampling point movement type thinning process If the value is less than a predetermined reference evaluation value T2, execution of the sampling point moving type thinning process is determined, and the thinning mode determination result for the past frame is the sampling point moving type thinning process, and the movement When the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the amount detection unit is equal to or higher than a predetermined reference evaluation value T2, the sampling point fixed thinning process is performed. Characterized in that it is configured to determine the line.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the block processing unit is a correspondence data between the subject moving speed and the image quality evaluation value generated based on the processing data when the thinning process for fixing the sample points is executed. The sample point movement mode is set so that the moving speed of the virtual subject generated by the sampling point moving type thinning process is set in the area of the moving speed where the image quality evaluation value is equal to or higher than the predetermined reference evaluation value in FIG. It is characterized by being configured to execute spatial thinning processing.

  Furthermore, in an embodiment of the moving image conversion apparatus of the present invention, the block processing unit further includes a memory that holds a movement mode of sample points when spatial thinning processing is performed on a past frame by the block processing unit. In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed, the sampling point corresponding to the movement mode of the sampling point at the time of spatial thinning with respect to the past frame If the image quality evaluation value corresponding to the moving speed of the virtual subject is equal to or higher than a predetermined reference evaluation value T3, a spatial thinning process is performed in which the movement mode of the sample points at the time of spatial thinning for the past frame is set. In the correspondence data between the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed, When the image quality evaluation value corresponding to the moving speed of the virtual subject corresponding to the movement mode of the sample point at the time of spatial thinning with respect to the past frame is lower than the predetermined reference evaluation value T3, the thinning process for fixing the sample point Sample point moving thinning-out processing is performed in a moving speed region in which the image quality evaluation value is equal to or higher than a predetermined reference evaluation value T4 in the correspondence data of the object moving speed and the image quality evaluation value generated based on the processing data when It is characterized in that the spatial thinning process is performed in which the movement manner of the sample points is set so that the moving speed of the virtual subject generated by the above is set.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the block processing unit may perform (a) only spatial direction thinning processing or (b) spatial direction thinning processing and time direction based on the movement amount information. Only the thinning-out process or (c) the time-direction thinning-out process is configured to execute the thinning-out process in any of the above-described aspects (a) to (c).

  Furthermore, in an embodiment of the moving image conversion apparatus of the present invention, the movement amount detection unit performs a process of detecting a motion vector based on a corresponding block of a different frame, and the block processing unit is based on the motion vector. The thinning mode is determined, and the thinning process according to the determined mode is executed.

Furthermore, the second aspect of the present invention provides
A moving image restoration apparatus that performs a restoration process of moving image conversion data,
A block extension unit that inputs block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executes an extension process of the block correspondence conversion data based on the conversion mode information;
A synthesis unit that synthesizes the blocks restored by the block extension process in the block extension unit and generates frame data;
The block extension is
Block expansion processing corresponding to at least one of the sampling point fixed type spatial direction thinning processing, sample point movement type spatial direction thinning processing, and time direction thinning processing executed in the generation of the moving image conversion data To restore the block,
The synthesis unit is
The moving image restoration apparatus is characterized in that the block restored by the block extension process is combined to generate frame data.

  Furthermore, in an embodiment of the moving image restoration apparatus of the present invention, the synthesis unit may be configured such that when the block restored by the block extension unit is a block that has undergone sampling direction moving type spatial direction thinning processing, The present invention is characterized in that a process of shifting the block positions according to the progress is executed.

  Furthermore, in an embodiment of the moving image restoration apparatus of the present invention, the synthesis unit may be configured such that when the block restored by the block extension unit is a block that has undergone sampling direction moving type spatial direction thinning processing, The present invention is characterized in that a process for shifting the block positions according to the progress is executed, and a pixel value correction process for determining pixel values of pixel gaps or overlapping pixels generated by the block arrangement is executed.

Furthermore, the third aspect of the present invention provides
A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data generated in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Based on the movement amount information, a thinning mode determination step for determining a thinning mode for performing sampling point fixed type spatial direction thinning or sampling point movement type spatial direction thinning;
In accordance with the determination information in the thinning mode determination step, a thinning process execution step for executing either a sample point fixed type spatial direction thinning process or a sample point movement type spatial direction thinning process;
A moving image conversion method characterized by comprising:

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning mode determining step includes a step of determining a sample point moving direction according to the progress of the frame as a processing mode in the sampling point moving type spatial direction thinning. And the thinning process execution step executes a sample point movement type spatial direction thinning process with a sample point movement process in the direction determined in the thinning mode determination step.

  Furthermore, in an embodiment of the moving image conversion method of the present invention, the thinning mode determination step includes a correspondence between the subject moving speed generated based on the processing data when the sampling point fixing thinning process is executed and the image quality evaluation value. A sampling point determination process according to data is executed, and when the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected in the movement amount detection step is equal to or higher than a predetermined reference evaluation value, a sampling point If execution of the fixed thinning process is determined and the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected in the movement amount detection step is less than a predetermined reference evaluation value, the sample point is moved It is characterized in that execution of thinning-out processing of the mold is determined.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning mode determination step further holds a thinning mode determination result for a past frame by the thinning mode determination step, and executes a thinning process for fixing a sample point. If the subject movement speed and the image quality evaluation value are generated based on the processing data in this case, the thinning mode determination process is executed according to the thinning mode determination result for the past frame and the corresponding thinning mode determination result for the past frame. Is a sample point fixed thinning process, and the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection step is equal to or higher than a predetermined reference evaluation value T1. Execution of fixed thinning processing is determined, and the thinning mode determination result for the past frame is fixed at the sampling point If the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection step is less than a predetermined reference evaluation value T1, the sampling point movement type thinning processing is performed. The image quality evaluation value corresponding to the subject moving speed corresponding to the moving amount information detected by the moving amount detecting step is the sampling point moving type thinning process whose thinning mode determination result for the past frame is determined. Is less than a predetermined reference evaluation value T2, the execution of the sampling point moving type thinning process is determined, the thinning mode determination result for the past frame is the sampling point moving type thinning process, and the movement amount When the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the detection step is equal to or higher than a predetermined reference evaluation value T2, the sample point fixed type And determining the execution of the thinning process.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the block processing step includes correspondence data between the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning processing is executed. The sample point movement mode is set so that the moving speed of the virtual subject generated by the sampling point moving type thinning process is set in the area of the moving speed where the image quality evaluation value is equal to or higher than the predetermined reference evaluation value in FIG. A spatial thinning process is performed.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the block processing step further holds a specimen point movement mode during spatial thinning processing for a past frame by the block processing step, and fixes the specimen point. In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the thinning process is executed, a virtual subject corresponding to the moving mode of the sample point at the time of spatial thinning for the past frame If the image quality evaluation value corresponding to the moving speed of the image is equal to or greater than a predetermined reference evaluation value T3, a spatial thinning process is performed in which the movement mode of the sample points during spatial thinning for the past frame is set. In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the fixed thinning processing is executed, When the image quality evaluation value corresponding to the moving speed of the virtual subject corresponding to the movement mode of the sample point at the time of spatial thinning with respect to the past frame is below the predetermined reference evaluation value T3, the sampling point fixing thinning process is performed. In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data in the case of the execution, the sampling point moving type thinning process is performed in the moving speed area where the image quality evaluation value is equal to or higher than a predetermined reference evaluation value T4. It is characterized in that a spatial thinning process is performed in which the movement mode of the sample points is set so that the generated moving speed of the virtual subject is set.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the block processing step is based on the movement amount information, (a) only spatial direction thinning processing, or (b) spatial direction thinning processing and time direction. Only the thinning-out process or (c) the time-direction thinning-out process executes the thinning-out process in any one of the above-described aspects (a) to (c).

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the movement amount detection step executes a process of detecting a motion vector based on a corresponding block of a different frame, and the block processing step is based on the motion vector. And determining a thinning mode, and performing a thinning process according to the determined mode.

Furthermore, the fourth aspect of the present invention provides
It is a moving image restoration method for executing moving image conversion data restoration processing,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block expansion processing corresponding to at least one of the sampling point fixed type spatial direction thinning processing, sample point movement type spatial direction thinning processing, and time direction thinning processing executed in the generation of the moving image conversion data To restore the block,
The synthesis step includes
The moving image restoration method is a step of generating frame data by synthesizing blocks restored by the block extension processing.

  Furthermore, in an embodiment of the moving image restoration method of the present invention, the synthesis step includes: when the block restored in the block extension step is a block that has been subjected to sampling point movement type spatial direction thinning processing; It is characterized by executing a process of shifting the block position according to the progress.

  Furthermore, in an embodiment of the moving image restoration method of the present invention, the synthesis step includes: when the block restored in the block extension step is a block that has been subjected to sampling point movement type spatial direction thinning processing; The present invention includes a step of executing a process of shifting the block positions according to the progress and executing a pixel value correction process for determining pixel values of pixel gaps or overlapping pixels generated by the block arrangement.

Furthermore, the fifth aspect of the present invention provides
A computer program for executing data conversion processing of moving image data;
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data generated in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Based on the movement amount information, a thinning mode determination step for determining a thinning mode for performing sampling point fixed type spatial direction thinning or sampling point movement type spatial direction thinning;
In accordance with the determination information in the thinning mode determination step, a thinning process execution step for executing either a sample point fixed type spatial direction thinning process or a sample point movement type spatial direction thinning process;
There is a computer program characterized by comprising:

Furthermore, the sixth aspect of the present invention provides
A computer program for executing data conversion processing of moving image data;
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data generated in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Based on the movement amount information and the past thinning mode, a thinning mode determination step for determining a thinning mode to execute the sampling point fixed type spatial direction thinning or the sampling point movement type spatial direction thinning;
In accordance with the decision information and the past decision information in the thinning mode decision step, a thinning process execution step for executing either the sample point fixed type spatial direction thinning process or the sample point movement type spatial direction thinning process;
A computer program characterized by comprising:

Furthermore, the seventh aspect of the present invention provides
A computer program for executing a restoration process of moving image conversion data,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block expansion processing corresponding to at least one of the sampling point fixed type spatial direction thinning processing, sample point movement type spatial direction thinning processing, and time direction thinning processing executed in the generation of the moving image conversion data To restore the block,
The synthesis step includes
The computer program is a step of generating frame data by synthesizing blocks restored by the block expansion process.

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, the optimum compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject is determined, and the data conversion processing is performed in the optimal mode for each region according to the determined mode. By doing so, it becomes possible to perform compression and decompression with very little quality deterioration.

  In the configuration of the present invention, a subject movement amount in a block constituting moving image data is detected, and based on the detected movement amount, a sampling point fixed type spatial direction thinning process or a sampling point movement type thinning in the spatial direction is performed. Execute the process. The sampling point movement type thinning out in the spatial direction makes it possible to generate a super-resolution effect based on the virtual subject speed setting, thereby realizing data conversion with suppressed image quality degradation.

  According to the configuration of the present invention, the thinning mode determination process according to the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed is executed, and the processing target If the image quality evaluation value corresponding to the subject movement speed corresponding to the block movement amount information is greater than or equal to a predetermined reference evaluation value, a fixed sampling point type thinning process is executed to obtain the subject movement speed corresponding to the movement amount information. If the corresponding image quality evaluation value is less than a predetermined reference evaluation value, it is possible to set a virtual subject speed by executing a sampling point moving type thinning process, and various movements of the processing target block Data conversion with reduced image quality deterioration that causes a super-resolution effect with respect to the amount becomes possible.

  Further, according to the configuration of the present invention, by the block expansion process corresponding to at least one of the sampling point fixed type spatial direction thinning process, the sample point moving type spatial direction thinning process, and the time direction thinning process. Configures a moving image restoration device that executes block restoration, combines the blocks restored by the block expansion process, and generates frame data. Since the arrangement is changed according to the movement mode, a high-quality image close to the original image can be restored.

Hereinafter, the configuration of a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program according to the present invention will be described with reference to the drawings. The description will be made according to the following items.
(1) Basic configuration of moving image conversion device using super-resolution effect (2) Configuration of moving image conversion device for executing improved thinning process (3) Moving image restoration device and moving image restoration method

[(1) Basic configuration of moving image conversion apparatus using super-resolution effect]
First, a basic configuration of a moving image conversion apparatus that performs moving image compression using the super-resolution effect as a base of the present invention will be described. This basic configuration is described in detail in Japanese Patent Application No. 2003-412501 filed earlier by the present applicant. The image is divided into small areas, and the number of pixels is determined according to the moving speed of each area. This is a configuration in which compression of the data amount is realized by adaptively performing decimation and frame rate decimation.

  FIG. 1 shows a configuration example of the moving image conversion apparatus 10 described in Japanese Patent Application No. 2003-412501. The moving image conversion apparatus 10 can reduce the amount of data so that an observer does not perceive image quality deterioration due to the reduction of the data amount by performing a moving image conversion process using the super-resolution effect. It is a thing.

  Note that the super-resolution effect is a visual effect realized based on a visual characteristic that an observer perceives an image obtained by adding a plurality of images within a certain period of time. When human vision perceives a stimulus, it has a function of storing the stimulus for a certain period of time after the presentation of the stimulus ends (referred to as a sensory memory function). There are many reports that this time is 10 ms to 200 ms. This function is also called iconic memory or visual persistence, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp.229-230”. The super-resolution effect is considered to be caused by a complicated relationship between the temporal integration function and the sensory memory function in the human visual function.

  The moving image conversion apparatus 10 shown in FIG. 1 performs compression for reducing data so that the observer does not perceive image quality degradation by performing moving image conversion processing using the super-resolution effect caused by the temporal integration function. It is set as the structure to perform. A configuration of the moving image conversion apparatus 10 in FIG. 1 will be described.

  The block dividing unit 11 divides each frame of the input moving image into blocks as divided areas of predetermined pixels and supplies them to the movement amount detecting unit 12. The movement amount detection unit 12 detects the movement amount for each block supplied from the block division unit 11, and transmits the block and the movement amount to the block processing unit 13. The block processing unit 13 performs a moving image conversion process corresponding to the movement amount, that is, a compression process, on the block supplied from the movement amount detection unit 12 to reduce the data amount. The block processing unit 13 supplies the output unit 14 with data about the block with the reduced data amount obtained as a result of the processing. The output unit 14 collectively outputs, as stream data, the data regarding the blocks with the reduced data amount supplied from the block processing unit 13.

  Next, details of each unit will be described with reference to FIG. The frame of the moving image supplied to the moving image conversion apparatus 10 is input to the image storage unit 21 of the block dividing unit 11. The image storage unit 21 stores the input frame. Each time the number of accumulated frames reaches N (N is a positive integer), the image accumulating unit 21 supplies the N frames to the block dividing unit 22, and M in the N frames The second stored frame (hereinafter referred to as the Mth frame) is supplied to the movement amount detection unit 12 (movement amount detection unit 31). For example, N = 4.

  The block dividing unit 22 divides each of the N frames (consecutive N frames) supplied from the image storage unit 21 into blocks of a certain size (for example, 8 × 8, 16 × 16) and moves them. It outputs to the quantity detection part 12 (block distribution part 32). The block dividing unit 22 also converts each block of the Pth frame stored in the image storage unit 21 (hereinafter referred to as the Pth frame) among the N frames to the movement amount detection unit 12 (movement amount detection). Part 31). The Pth frame is a different frame from the Mth frame.

  Next, the movement amount detection unit 12 will be described. The movement amount detection unit 31 of the movement amount detection unit 12 uses the motion vector of each block of the Pth frame supplied from the block division unit 22 of the block division unit 11 as the Mth frame supplied from the image storage unit 21. For example, the block matching process between the frames is detected and detected, and the detected motion vector is supplied to the block distributor 32. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and in the vertical direction (Y-axis direction). In addition, the movement amount detection unit 31 may be configured to enlarge the image in order to improve the detection accuracy of the movement amount and detect the movement amount using the enlarged image.

  The block distribution unit 32 of the movement amount detection unit 12 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 22 in units of N, and the movement amount detection unit From 31, the amount of movement of the block of the P-th frame among the N blocks is supplied. The block distribution unit 32 supplies the supplied N blocks and the movement amount to any of the block processing units 51 to 53 that perform processing corresponding to the movement amount of the block processing unit 13.

  Specifically, the block distribution unit 32 is supplied from the movement amount detection unit 31 and has a movement amount in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) between one frame of 2 pixels (pixels) or more. In some cases, the N blocks supplied from the block dividing unit 22 and the movement amount supplied from the movement amount detection unit 31 are output to the block processing unit 51. If the amount of movement in the horizontal direction and the vertical direction between one frame is less than 2 pixels and more than 1 pixel, the block distribution unit 32 outputs N blocks and the amount of movement to the block processing unit 53. . When the movement amount is other than that, the block distribution unit 32 supplies the N blocks and the movement amount to the block processing unit 52.

  That is, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs block processing for performing processing for converting image data according to the frame rate and spatial resolution. The block images are distributed to the units 51 to 53.

  This condition for determining the distribution destination is merely an example, and the distribution destination may be determined under other conditions.

  Next, details of the block processing unit 13 will be described. The block processing unit 13 is composed of the three block processing units 51 to 53 as described above. The block processing unit 51 supplies a total of N blocks (horizontal or vertical) at the same position in each of consecutive N (for example, N = 4) frames supplied from the block distribution unit 32 of the movement amount detection unit 12. Similarly, a process of thinning out the number of pixels according to the amount of movement supplied from the block distributor 32 (spatial direction thinning process) is performed on N blocks in the case where the amount of movement in the direction is 2 pixels or more. .

  Specifically, when the amount of movement in the horizontal direction between one frame is 2 pixels (pixels) or more, the block processing unit 51, when the block is configured by 8 × 8 pixels, as shown in FIG. The pixels in the block are divided into sets of 1 × 4 pixel units. Further, as shown in FIG. 4, the block processing unit 51 sets the pixel values p1 to p4 of each set of 1 × 4 pixels to one pixel value (p1 in this example) of the pixel values. Thinning (thinning of the number of pixels between four pixels) (thinning of the thinning amount 4) is performed.

  When the vertical movement amount between one frame is 2 pixels or more, the block processing unit 51 divides the pixels in the block into a set of 4 × 1 pixel units as shown in FIG. As shown, the pixel values p1 to p4 of each set are thinned out to be one pixel value (p1 in this example).

  When the vertical and horizontal movement amounts between one frame are both 2 pixels or more, the block processing unit 51 divides the pixels in the block into sets of 2 × 2 pixel units as shown in FIG. As shown in FIG. 8, the pixel number p1 to p4 of each set is thinned out to be one pixel value (p1 in this example).

  Since the block processing unit 51 performs such spatial thinning processing on each of the four supplied blocks, the data amount of each block is reduced to one pixel data amount for every four adjacent pixels. The data amount of the entire four blocks is reduced to ¼. The block processing unit 51 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  In the example of FIG. 4, the leftmost pixel value p1 in the 1 × 4 pixel in the example of FIG. 4, the uppermost pixel value p1 in the 4 × 1 pixel in the example of FIG. In the example, the pixel value p1 in the upper left corner of 2 × 2 pixels is set, but the pixel value may be set to any pixel value from p1 to p4. The pixel value may be set to a calculation using the pixel values p1 to p4, for example, an average value.

  Next, processing executed by the block processing unit 52 shown in FIG. 2 will be described. The block processing unit 52 shown in FIG. 2 has a total of N blocks (the amount of movement in both the horizontal direction and the vertical direction is the same) in each of consecutive N frames supplied from the block distribution unit 32 of the movement amount detection unit 12. A process of thinning out the number of frames (time direction thinning process) is performed on N blocks in the case of less than one pixel.

  Specifically, as shown in FIG. 9, the block processing unit 52 converts four blocks Bi at the same position in four consecutive frames F1 to F4 into one block (in this example, Then, the number of frames to be converted into the block Bi) of the frame F1 is thinned out (thinning of the number of frames between four frames). The block processing unit 52 supplies the output unit 14 with data (one block) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing.

  The block processing unit 53 is supplied from the block distribution unit 32 of the movement amount detection unit 12 and has a total of N blocks (the movement amount in the horizontal direction and the vertical direction is 1 in each of the consecutive N frames). The number of pixels is thinned out (spatial direction thinning process) and the number of frames is thinned out (time direction thinning process) for each of N blocks in the case of more than pixels and less than 2 pixels.

  Unlike the thinning-out process in the block processing unit 51, the block processing unit 53 converts each set of pixel values p1 to p4 into any two pixel values (in this example, as shown in FIGS. 10 and 11). In this case, the number of pixels to be reduced to p1, p3) is thinned out (thinning amount 2 is thinned out).

  That is, the block processing unit 51 thins out three types of pixels of 1 × 4, 4 × 1, or 2 × 2, while the block processing unit 53 performs two types of pixels of 1 × 2 or 2 × 1. Decimation of numbers is performed.

  In the frame number thinning process, the block processing unit 53 is different from the thinning process in the block processing unit 52, as shown in FIG. 12, at the same position in each of the four consecutive frames F1 to F4. Decreasing the number of frames to make a total of four blocks Bi any one of them (two blocks of frames F1 and F3 in the example in the figure) (decimating the number of frames between two frames) Do.

  Since the block processing unit 53 performs the spatial direction thinning process for reducing the data amount by 1/2 and the time direction thinning process for reducing the data amount by 1/2 to the four supplied blocks. As a result, the data amount of the four blocks is reduced to (1/2) × (1/2) = 1/4. The block processing unit 53 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  As described above, the moving image conversion apparatus 10 shown in FIG. 1 performs processing for converting an input moving image into a moving image (compressed data) with a reduced data amount. This is a device that prevents the observer from perceiving image quality degradation due to a reduction in the amount of data by performing moving image conversion processing using the super-resolution effect realized based on the visual characteristics of the image.

  Specifically, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs image data conversion according to the optimal frame rate and spatial resolution. It supplies to the block processing parts 51-53 to perform, and it is set as the structure which performs the data conversion process of a different aspect in each block processing part 51-53, and makes an observer perceive image quality degradation by this structure. It realizes a moving image conversion process without any problem. Note that the super-resolution effect is a visual effect realized based on the visual characteristic that an observer perceives a sum of a plurality of images within a certain time as described above. It is considered that the temporal integration function and the sensory memory function in the function are caused in a complicated manner, and the moving image conversion apparatus 10 shown in FIG. 1 has a super-resolution effect caused by the temporal integration function. The moving image conversion process is used.

  The principles and explanations regarding human visual characteristics and super-resolution effect are described in detail in Japanese Patent Application No. 2003-412501. The conditions for generating the super-resolution effect described in Japanese Patent Application No. 2003-412501 will be described below.

In order for the super-resolution effect to occur when the number of pixels of the thinning amount m (pixel) is thinned, it is necessary to cancel all the primary to m−1 order folding components due to the thinning. k (= 1, 2,..., m−1) The condition that the next folding component is canceled is to satisfy the following expressions (Expression 1) and (Expression 2).

In the above equation, φ _t is a sampling position shift amount in thinning out the number of pixels, time t (= 0, 1T, 2T,...), Signal moving speed v, time interval (reciprocal of frame rate) T , Is a value defined by the following formula (formula 3).

  In the above equation, the case where the sampling position is shifted to the right is positive (this condition is different from the setting disclosed in Japanese Patent Application No. 2003-412501).

  If the above equations (Equation 1) and (Equation 2) are satisfied under the conditions of the thinning amount m and the small region movement amount v, the super-resolution effect occurs, and the deterioration of the image quality is hardly perceived by the observer.

The moving image conversion apparatus 10 described with reference to FIG. 1 to FIG. 12 sets the moving speed condition between one frame in a small area when thinning out the number of pixels with a thinning amount of 4 as two or more pixels in the horizontal or vertical direction. Was set. However, for example, when the moving speed is 4 pixels per frame, that is, when the moving amount v of the small region is v = 4, if the thinning amount is m = 4, the deviation amount of the sampling position of the above equation (Equation 3): φ _t Can only take an integer greater than or equal to 0.

That is, the above formula (Formula 1) is not satisfied in all k cases, and the super-resolution effect does not occur. Similarly,
When the thinning amount m = 2 and the small region moving amount v = 1,
When the thinning amount m = 4 and the small region moving amount v = 2,
Even in such a setting, the above equation (Equation 1) is not satisfied in all k cases, the super-resolution effect cannot be obtained, and the image quality is greatly deteriorated. That is, the moving image conversion apparatus 10 described with reference to FIGS. 1 to 12 exhibits a super-resolution effect when each value of the thinning amount m and the small region movement amount v satisfies a specific condition. It is possible to prevent the observer from perceiving image quality degradation due to the reduction in the data amount. However, if the values of the thinning amount m and the small region movement amount v do not satisfy a specific condition, the super-resolution effect is obtained. It does not occur, leading to image quality degradation. A configuration that solves this problem will be described below.

[(2) Configuration of Moving Image Conversion Device that Performs Improved Thinning Process]
In the following, by solving the problems of the above-mentioned moving image conversion apparatus and executing the improved thinning process, the case where the super-resolution effect does not occur is reduced, and the data reduction (compression process) that suppresses the image quality deterioration is realized. A moving image conversion apparatus will be described.

  The configuration of the moving image conversion apparatus 100 will be described with reference to FIG. Similar to the moving image conversion apparatus 10 described above with reference to FIG. 1, the moving image conversion apparatus 100 shown in FIG. 13 uses the super-resolution effect due to human visual characteristics, thereby reducing the image quality by reducing the data amount. It is possible to reduce the amount of data so that the observer does not perceive deterioration, and whether or not each value of the thinning-out amount m and the small region moving amount v satisfies a specific condition. Therefore, it is possible to reduce data without causing a super-resolution effect and causing image quality degradation.

  A configuration of the moving image conversion apparatus 100 in FIG. 13 will be described. The block dividing unit 110 divides each frame of the input moving image into blocks and supplies the blocks to the movement amount detecting unit 120. The movement amount detection unit 120 detects the movement amount for each block supplied from the block division unit 110, and transmits the block and the movement amount to the block processing unit 130. The block processing unit 130 performs a moving image conversion process corresponding to the movement amount on the block supplied from the movement amount detection unit 120 to reduce the data amount. The block processing unit 130 supplies the output unit 140 with data regarding the block whose data amount is reduced, which is obtained as a result of the processing. The output unit 140 collectively outputs the data regarding the blocks with the reduced data amount supplied from the block processing unit 130, for example, as stream data.

  Next, details of each unit will be described with reference to FIG. First, the block dividing unit 110 will be described. The frame of the moving image supplied to the moving image conversion apparatus 100 is input to the image storage unit 111 of the block dividing unit 110. The image storage unit 111 stores the input frame. Each time the number of accumulated frames reaches N (N is a positive integer), the image accumulation unit 111 supplies the N frames to the block division unit 112 and M in the N frames. The second stored frame (hereinafter referred to as the Mth frame) is supplied to the movement amount detection unit 121 of the movement amount detection unit 120. For example, N = 4.

  The block dividing unit 112 divides each of the N frames (consecutive N frames) supplied from the image storage unit 111 into blocks of a certain size (for example, 8 × 8, 16 × 16) and moves them. The data is output to the block distribution unit 122 of the amount detection unit 120. The block division unit 112 also detects the movement amount of the P-th frame stored in the image storage unit 111 (hereinafter referred to as the P-th frame) among the N frames by the movement amount detection unit 120. To the unit 121. The Pth frame is a different frame from the Mth frame.

  Next, the movement amount detection unit 120 will be described. The movement amount detection unit 121 of the movement amount detection unit 120 uses the motion vector of each block of the Pth frame supplied from the block division unit 112 of the block division unit 110 as the Mth frame supplied from the image storage unit 111. For example, the block matching process between frames is executed and detected, and the detected motion vector is supplied to the block distributor 122 and the block processors 131 and 132. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and in the vertical direction (Y-axis direction). For example, when M = 2 and P = 3, the motion vector represents the amount of movement in the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction) between one frame. The motion vector may be detected not only by block matching but also by another method.

  The block distribution unit 122 of the movement amount detection unit 120 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 112 in units of N, and the movement amount detection unit From 121, the movement amount of the block of the P-th frame among the N blocks is supplied. The block distribution unit 122 performs block processing units 131 to 133 that perform processing corresponding to the movement amount of the block processing unit 130 for the supplied N blocks. The block processing unit 130).

  Specifically, the block distribution unit 122 is supplied from the movement amount detection unit 121 and has a larger movement amount of 2 pixels between the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction) between one frame. In the case above, the N blocks supplied from the block dividing unit 112 are output to the block processing unit 131. When the movement amount between the horizontal direction and the vertical direction between one frame is less than 2 pixels and 1 pixel or more, the block distribution unit 122 outputs N blocks to the block processing unit 132. To do. When the movement amount is other than that, the block distribution unit 122 supplies N blocks to the block processing unit 133.

That is,
(A) Movement amount ≧ 2 pixels / frame: block processing unit 131 (spatial direction thinning process)
(B) 2 pixels / frame> movement amount ≧ 1 pixel / frame: block processing unit 132 (time / space direction thinning process)
(C) 1 pixel / frame> movement amount: block processing unit 133 (time direction thinning process)
The processing block is output to the above (a) to (c).

  When the moving amount ≧ 2 pixels, the block processing unit 131 executes the spatial direction thinning process. When 2 pixels> the moving amount ≧ 1 pixel, the block processing unit 132 executes the time / spatial direction thinning process. When pixel> movement amount, the block processing unit 133 performs the time direction thinning process.

  As described above, the block distribution unit 122 determines the optimal frame rate and spatial resolution for conversion based on the movement amount supplied from the movement amount detection unit 121, and converts the image data according to the frame rate and spatial resolution. Block processing units 131 to 133 that perform processing A process of distributing block images is executed. This condition for determining the distribution destination is merely an example, and the distribution destination may be determined under other conditions.

  For example, when the movement amount in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction), which is supplied from the movement amount detection unit 121, whichever is larger is 2 pixels or more, the block division unit The N blocks supplied from 112 are output to the block processing unit 131. If the movement amount between the horizontal direction and the vertical direction between one frame, whichever is larger, is less than 2 pixels, the N blocks are output. By setting a condition for outputting to the block processing unit 133, it is possible to perform no processing of the block processing unit 132.

  Next, the block processing unit 130 will be described. In this example, the block processing unit 130 includes three block processing units 131 to 133.

  The block processing unit 131 is supplied from the block distribution unit 122 of the movement amount detection unit 120 and has a total of N blocks (the movement amount in the horizontal direction or the vertical direction is 2 at the same position in each of consecutive N frames). Processing for thinning out the number of pixels in accordance with the larger moving amount (horizontal direction or vertical direction supplied from the moving amount detecting unit 131) for the N blocks in the case of pixels / frame or more (spatial direction) (Thinning process). That is, the spatial direction thinning process is executed for N blocks whose movement amount ≧ 2 pixels.

  When the amount of movement in the horizontal direction between 1 frame is 2 pixels or more, the block processing unit 131, for example, when the block is composed of 8 × 8 pixels, as shown in FIG. Divide the pixels into sets of 1 × 4 pixel units.

  Further, the block processing unit 131 sets the pixel values p1 to p4 of each set of 1 × 4 pixels to one of the pixel values in any of the modes of FIG. 15B, FIG. 16 and FIG. Thinning out the number of pixels (thinning out the number of pixels between four pixels) (thinning out the thinning amount 4) is performed.

  The thinning-out process shown in FIG. 15B is the same process as the thinning-out process described with reference to FIG. 4 as a process example in the basic configuration of the moving image conversion apparatus, and includes N consecutive frames (k This is a thinning process in which the pixel value at the same pixel position is set as the representative pixel value, that is, the sampling point (sampling point = p1 in the example in the figure) as the pixel value of four pixels in all frames of .about.k + 3).

  The thinning-out process shown in FIGS. 16 and 17 is not set so that the same pixel position is used as a sample point in N consecutive frames (k to k + 3), but pixels at different pixel positions are used as sample points in each frame. This is a thinning-out process in which pixel values at different positions for each frame are set as the pixel values of four pixels in each frame.

The thinning-out process shown in FIG. 16 is performed for pixel values p1 to p4 of four horizontal pixels at the same position in the four frames from k to k + 3.
k-th frame: sample point = p1
k + 1 frame: sample point = p2
k + 2nd frame: sample point = p3
k + 3rd frame: sample point = p4
As described above, the thinning process is performed by changing the pixel position of the sample point in the right direction one pixel at a time according to the progress of the frame.

The thinning-out process shown in FIG. 17 is performed for pixel values p1 to p4 of four horizontal pixels at the same position in the four frames k to k + 3.
k-th frame: sample point = p4
k + 1 frame: sample point = p3
k + 2nd frame: sample point = p2
k + 3rd frame: sample point = p1
In this manner, the thinning process is performed by changing the pixel position of the sample point in the left direction one pixel at a time according to the progress of the frame.

  As described above, the thinning-out processing shown in FIGS. 16 and 17 is performed in such a manner that the pixel values at different pixel positions are not the same pixel position but the pixel values of the four pixels in each frame in N consecutive frames (k to k + 3). Is a thinning process set as

  The block processing unit 131 selects and executes one of the three thinning processes shown in FIGS. 15 to 17 according to the movement amount information supplied from the movement amount detection unit 121 of the movement amount detection unit 120.

  A detailed configuration of the block processing unit 131 is shown in FIG. The block processing unit 131 includes a thinning processing mode determination unit 151 and a thinning processing execution unit 152. The thinning processing mode determination unit 151 inputs movement amount information from the movement amount detection unit 121. As described above, the block input to the block processing unit 131 is a block of moving amount ≧ 2 pixels / frame, and the moving amount information input to the decimation processing mode determining unit 151 of the block processing unit 131 is: The moving amount information is 2 pixels / frame or more.

  The thinning process mode determination unit 151 determines whether to perform the thinning process in any of the previously described FIGS. 15, 16 and 17 based on the value of the movement amount. Details of this determination method will be described later. The thinning process execution unit 152 executes the thinning process in any of the modes of FIGS. 15, 16, and 17 according to the determination of the thinning process mode determination unit 151.

  The effect of performing the thinning process by changing the sample point position according to each frame as shown in FIG. 16 or FIG. 17 will be described. Hereinafter, the thinning process executed by changing the sample point position in accordance with each frame in this manner is referred to as “sample point moving thinning process” or “decimation position changing process”. In the [sample point moving thinning process], the process of shifting the sample point position to the right as the frame progresses as shown in FIG. 16 and the process of changing the thinning position to the right direction as shown in FIG. There are two modes of changing the thinning position in which the process of shifting the sample point position to the left according to the progress of the frame is shifted to the left.

In the thinning position changing process, the sample point shift amount φ _t shown in the mathematical formula (formula 3) described in the basic configuration of the moving image conversion apparatus is added by 1 / m for each frame (when shifted to the right (FIG. 16). )), Or 1 / m subtraction (when shifting to the left (FIG. 17)). Note that m is a thinning amount (m pixels).

A new sample point shift amount φ ′ _t when the thinning position changing process is performed is expressed by the following mathematical formula (Formula 4).

  From the above equation (Equation 4), shifting the position of the sample point in the traveling direction of the subject decelerates the moving speed of the subject, and shifting the position opposite to the traveling direction accelerates the moving speed and signal processing. It can be seen that the theoretical meaning of is equivalent. Of course, this does not mean that the moving speed of the subject changes on the actual video. Hereinafter, a specific processing example will be described with reference to FIG.

  FIG. 19 shows a situation in which the subject (5 pixels between p1 and p5 of the kth frame) 201 has not moved (A) during 4 frames (frames k to k + 3), and (B) the velocity v = in the horizontal direction. It shows the situation of moving at 1 pixel / frame.

  Since the block processing unit 131 of the moving image conversion apparatus 100 shown in FIGS. 13 and 14 is originally input with a block whose horizontal movement amount between one frame is 2 pixels or more, it is shown in FIG. Although no frame data is input as in (A) and (B), in the case where the amount of movement in the horizontal direction between one frame is 2 pixels or more, FIGS. Any one of the thinning processes shown in FIG. That is, there are cases where [decimation position change processing] is performed and not performed depending on the amount of movement.

  With reference to FIG. 19, processing when the [decimation position changing process] is performed and when it is not performed will be described. For example, in the situation where (A) it is not moving, the thinning process of the fixed sample points shown in FIG. 15 is executed, and (B) in the situation where it is moving in the horizontal direction at a speed v = 1 pixel / frame. It is assumed that the sampling point shown in FIG. 16 is moved every frame [decimation position changing process], that is, the sampling point moving type thinning process is performed.

As shown in FIG. 19A, in the situation where the subject is not moving, if the thinning amount = 4, that is, if the pixel end thinning between four pixels is performed, the fixed sampling point thinning described with reference to FIG. 15 is thinned out. When the process is executed, the output values for every four pixels are the pixels of p1 and p5 in all the frames, and are the pixel values at fixed positions. On the other hand, as shown in FIG. 19B, the sample point described with reference to FIG. 16 is moved for each frame in a situation where the subject is moving in the horizontal direction at a speed v = 1 pixel / frame [ If the thinning position change process] is performed, the output value of each frame corresponding to each 4 pixel is as shown in FIG.
k-th frame: p1, p5
k + 1 frame: p2, p6
k + 2nd frame: p3, p7
k + 3rd frame: p4, p8
It becomes.

  As a result, the output value of each frame is identical between the moving subject of 1 pixel / frame in FIG. 19B and the non-moving subject in FIG. 19A. That is, an output obtained by performing [decimation position change processing] for moving the sample point described with reference to FIG. 16 for each frame with respect to a moving image including a subject with a moving amount v = 1 pixel / frame, and a moving speed. It can be seen that the output shown in FIG. 15, that is, the output obtained by performing the thinning process using a fixed sample point is matched with the subject (A) of v−1 = 0. That is, it can be confirmed that performing the process of FIG. 16 on the moving subject is equivalent to performing the process of FIG. 15 on the subject whose moving speed is decelerated by one.

  Eventually, the moving speed of the subject can be virtually accelerated or decelerated by the thinning position changing process. The moving image conversion apparatus 10 described with reference to FIGS. 1 to 12 exhibits a super-resolution effect only when each value of the thinning amount m and the small region movement amount v satisfies a specific condition, and the thinning amount is obtained. m and the moving amount v of the small area do not satisfy the specific condition and cause the image quality degradation. However, the moving image conversion apparatus 100 described in the present embodiment in FIG. The unit 130 is configured to execute the thinning process by changing the thinning processing mode based on the movement amount and virtually changing the moving speed of the subject to a speed at which the super-resolution effect occurs. In the moving image conversion apparatus 10 described with reference to FIG. 12, even the data for which the super-resolution effect cannot be obtained can obtain the super-resolution effect, and image conversion (compression) with little image quality degradation is realized. The

  Next, it is confirmed at what speed the super-resolution effect does not occur when the subject is moving. FIG. 20 shows a result obtained by performing the fixed sample point thinning-out process (four-pixel thinning-out process) of FIG. 15 on moving image data including a subject having a movement speed: v as a movement amount per various frames. The quality data of the image quality obtained by restoring (expanding) the converted (compressed) image again is shown. The horizontal axis represents the moving speed: v (pixel / frame) as the moving amount of the subject per frame, and the vertical axis represents the image quality evaluation value. That is, it is a graph showing the tendency of the result of the subjective evaluation experiment of the image quality when the subject is moved rightward by the movement amount shown on the horizontal axis, and the curve shown in the graph is the image quality evaluation curve 205. A reference evaluation value indicated by T in FIG. 20 is set as an image quality determination criterion, and a case where an evaluation equal to or higher than the reference evaluation value T is obtained is a good image quality.

  Fixed sample point thinning processing (4 pixels) described with reference to FIG. 15 for various subjects having a moving speed as a moving amount per frame: v = P (pixel / frame) to Q (pixel / frame). The area where the thinning process is performed and the evaluation equal to or higher than the reference evaluation value T is the area A in FIG. A region B other than the region A is a region that obtained an evaluation less than the reference evaluation value T.

  An area obtained with an evaluation equal to or higher than the reference evaluation value T, that is, an area A in FIG. 20, is a case where the moving speed: v as a moving amount of the subject per frame is in the range of ab, cd, e. In the case of having these movement amounts, even if the fixed sample point thinning process described with reference to FIG. 15 is applied, the thinning is performed so that the image quality is good and the super-resolution effect is not impaired. Is determined to be realized. On the other hand, when the moving speed: v as the moving amount per frame of the subject, that is, the region B in FIG. Therefore, it is considered that the super-resolution effect does not occur or is not sufficient.

  Based on the evaluation result, when the moving speed of the subject corresponds to the speed of the region B in FIG. 20, the moving speed of the subject is conceptually accelerated or decelerated by the thinning position changing process. That is, for example, the thinning position changing process as shown in FIGS. 16 and 17 is executed. As a result, as described with reference to FIG. 19, an output similar to that when moving at the speed of the region A is obtained, and a good image quality with a super-resolution effect is obtained.

  Regions B-1 and B-2 shown in FIG. 21 are part of the region B shown in FIG. That is, it is an image quality region that is less than the reference evaluation value T, and is a region that is considered to be a region that does not generate a super-resolution effect or has an insufficient subject moving speed: v. When the subject is moving at the speed of the areas B-1 and B-2, the subject is virtually accelerated (+ Δv1, + Δv2), that is, in the direction opposite to the traveling direction of the subject. By performing the process of changing the thinning position (for example, if the subject has moved to the right, the process of moving the sample point position to the left as the frame progresses as shown in FIG. 17), the region of FIG. The same effect as when B-1 is moved to the area C-1 in FIG. 21 and the area B-2 in FIG. 21 is moved to the area C-2 in FIG. 21 can be obtained. It is possible to set the moving speed: v as the moving amount per frame to an image quality region (C-1, C-2 in FIG. 21) that is equal to or higher than the reference evaluation value T. That is, as shown in FIG. 17, image conversion (compression) with little image quality degradation is realized by executing thinning processing for moving the sample point position to the left as the frame progresses.

  Similarly, when the subject is moving at the speed of the area B-3 in FIG. 22, the thinning position is changed in the same direction as the traveling direction by performing a process of virtually decelerating the subject (−Δv3). (For example, if it has moved to the right, the processing in FIG. 16 is performed), thereby moving the region B-3 in FIG. 22 to the region C-3 in FIG. 22, and as a result, the amount of movement of the subject per virtual frame Can be set to an image quality region (C-3 in FIG. 22) that is equal to or higher than the reference evaluation value T. That is, as shown in FIG. 16, image conversion (compression) with little image quality deterioration is realized by executing thinning processing for moving the sample point position to the right as the frame progresses.

  In the moving image conversion apparatus 100 in the present configuration example shown in FIGS. 13 and 14, the block processing unit 131 thins out four pixels in the spatial direction only when the movement amount per frame in the horizontal or vertical direction is 2 pixels or more. Since the processing is performed, in the graphs shown in FIGS. 20 to 22, the portion where the moving speed: v is 2 pixels / frame or less is not a processing target. However, as described above, the block distribution unit 122 is the block processing unit. The conditions for sending blocks to the 130 block processing units 131 to 133 are merely examples, and the block distribution unit 122 sends blocks to the block processing unit 131 even when the movement amount is less than 2 pixels. Since any one of the thinning modes described with reference to FIG. 17 can be selected and space thinning can be performed. It was added to the consideration.

  Note that the image quality evaluation curves shown in FIGS. 20 to 22 show an evaluation curve set based on the result of one experiment based on subjective evaluation, and various image quality evaluation methods can be applied. A configuration may be adopted in which the thinning processing mode is determined based on the result.

The block processing unit 131 shown in FIG. 14 that executes processing of block data with a movement amount of 2 pixels / frame or more is based on the movement amount information from the movement amount detection unit in the thinning processing mode determination unit 151 shown in FIG. ,
(A) Thinning process for fixing sample points (see FIG. 15)
(B) Thinning processing for shifting the sample point position to the right as the frame progresses (see FIG. 16)
(C) Thinning processing for shifting the sample point position to the left as the frame progresses (see FIG. 17)
Is executed, and the thinning process of the processing mode determined by the thinning process execution unit 152 shown in FIG. 18 is executed.

  In determining the thinning processing mode in the thinning processing mode determination unit 151 illustrated in FIG. 18, for example, determination processing based on the image quality evaluation curves illustrated in FIGS. 20 to 22 is performed. That is, when the image quality evaluation curve is a moving speed region in which the image quality equal to or higher than the reference evaluation value T is maintained, (a) the sampling point fixing thinning process (see FIG. 15) is executed, and the image quality evaluation curve becomes the reference evaluation value. In the moving speed region where the image quality is less than T, (b) thinning processing for shifting the sample point position to the right according to the progress of the frame (see FIG. 16), or (c) the sample point position to the left according to the progress of the frame As a configuration for performing any of the thinning-out thinning processing (see FIG. 17), the virtual movement amount is set to enter the movement amount region where the image quality evaluation curve maintains the image quality equal to or higher than the reference evaluation value T.

  FIG. 23 shows a determination example of the thinning processing mode to be determined based on the movement amount information from the movement amount detection unit in the thinning processing mode determination unit 151 shown in FIG. The determination example in FIG. 23 illustrates an example in which the processing mode is determined based on the image quality evaluation curves illustrated in FIGS.

23 (a) and 23 (b)
Movement speed: 2 pixels / frame ≦ movement amount <c pixels / frame,
Movement speed: f pixel / frame ≦ movement amount <e pixel / frame, and
Direction of movement: right, in this case,
This process of executing the thinning process (see FIG. 17) for shifting the sample point position to the left according to the progress of the frame is a virtual acceleration process, and the process of moving from the area B-1 to C-1 in FIG. This corresponds to the process of shifting from B-2 to C-2.

23 (c) and (d)
Movement speed: 2 pixels / frame ≦ movement amount <c pixels / frame,
Movement speed: f pixel / frame ≦ movement amount <e pixel / frame, and
Movement direction: Left, in this case,
This process of executing the thinning process (see FIG. 16) for shifting the sample point position to the right according to the progress of the frame is also a virtual acceleration process, or the process of moving from the area B-1 to C-1 in FIG. This corresponds to the process of shifting from B-2 to C-2.

FIG. 23 (e)
Movement speed: d pixel / frame ≦ movement amount <f pixel / frame, and
Direction of movement: right, in this case,
A thinning process (see FIG. 16) for shifting the sample point position to the right as the frame progresses is executed. This process is a virtual deceleration process and corresponds to the process of moving from the region B-3 to C-3 in FIG.

FIG. 23 (f)
Movement speed: d pixel / frame ≦ movement amount <f pixel / frame, and
Movement direction: Left, in this case,
A thinning process (see FIG. 17) for shifting the sample point position to the left as the frame progresses is executed. This process is a virtual deceleration process and corresponds to the process of moving from the region B-3 to C-3 in FIG.

FIG. 23 (g)
Movement speed other than the above, i.e.
Movement speed: c pixels / frame ≦ movement amount <d pixels / frame,
Movement speed: either e pixel / frame ≦ movement amount,
Direction of movement: left or right, in this case,
A thinning process (see FIG. 15) is performed with a fixed sample point instead of a thinning process in which the sample point position is shifted according to the progress of the frame. This processing corresponds to processing of a block having a subject having a movement amount corresponding to the area A in FIG.

The above processing example is a processing example of a block having a horizontal movement amount. However, the same processing is performed in the processing of a block having a vertical movement amount, that is, the block processing unit 131 shown in FIG. In the thinning processing mode determination unit 151 shown in FIG. 18, based on the movement amount information from the movement amount detection unit,
(A) Thinning process for fixing sample points (see FIG. 24B)
(B) Thinning processing for shifting the sample point position downward as the frame progresses (see FIG. 25)
(C) Thinning processing for shifting the sample point position upward in accordance with the progress of the frame (see FIG. 26)
Is executed, and the thinning process of the processing mode determined by the thinning process execution unit 152 shown in FIG. 18 is executed.

  When the amount of movement in the vertical direction between one frame is 2 pixels or more, the block processing unit 131, for example, when the block is composed of 8 × 8 pixels, as shown in FIG. The pixels are divided into sets of 1 × 4 pixel units, and the block processing unit 131 performs pixel values p1 to p1 of each set of 1 × 4 pixels in any of the modes of FIG. 24B, FIG. 25, and FIG. Thinning out the number of pixels to make p4 one of the pixel values (thinning out the number of pixels between the four pixels) (thinning out the thinning amount 4) is performed.

  In determining the thinning processing mode in the thinning processing mode determination unit 151 shown in FIG. 18 at this time, determination based on the image quality evaluation curve assuming the horizontal axis of the graphs shown in FIGS. 20 to 22 as the amount of movement in the vertical direction. Perform processing. That is, when the image quality evaluation curve is a moving speed region in which the image quality equal to or higher than the reference evaluation value T is maintained, (a) thinning processing for fixing sample points (see FIG. 24) is executed, and the image quality evaluation curve becomes the reference evaluation value. In the moving speed region where the image quality is less than T, (b) thinning processing (see FIG. 25) for shifting the sample point position downward in accordance with the progress of the frame, or (c) the sample point position upward in accordance with the progress of the frame. As a configuration for performing either of the thinning-out thinning processing (see FIG. 26), the virtual movement amount is set to enter the movement amount region in which the image quality evaluation curve maintains the image quality equal to or higher than the reference evaluation value T.

  FIG. 27 is a table summarizing the movement speed: v and the processing mode as the amount of movement of the subject included in the processing block when the thinning processing mode is determined by applying the image quality evaluation curves shown in FIGS. Shown in (a). FIG. 27B is a table for explaining the processes 1 to 3 in FIG.

The contents of FIG. 27A will be described.
If 2 ≦ v <c,
Process 3: The thinning position is changed in the direction opposite to the traveling direction of the subject.
If c ≦ v <d,
Process 1: The thinning position is not changed.
If d ≦ v <f,
Process 2: The thinning position is changed in the same direction as the moving direction of the subject.
If f ≦ v <e,
Process 3: The thinning position is changed in the direction opposite to the traveling direction of the subject.
If e ≦ v,
Process 1: The thinning position is not changed.

  In this way, by changing the thinning processing mode according to the moving speed: v as the moving amount of the subject included in the processing block, the virtual moving speed of the subject is all changed to the region A shown in FIG. Therefore, image conversion that guarantees an image quality equal to or higher than the reference image quality evaluation value T is executed.

  In this way, the block processing unit 131 executes different thinning processing depending on the moving speed: v as the moving amount of the subject included in the processing block per frame, and outputs it to the output unit 140 (see FIGS. 13 and 14). Output. When the thinning amount is 4, the block processing unit 131 performs the thinning process in the spatial direction for each of the supplied four blocks (one pixel is selected for every four adjacent pixels). The amount is reduced to ¼, and the data amount of the entire four blocks is reduced to ¼. The block processing unit 131 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  In the above-described embodiment, the parameter setting example in which the thinning amount m = 4 and the number of frames N = 4 executed in the block processing unit 131 has been described. However, other values may be applied to the values of these parameters. Is also possible. However, the relationship between the moving speed of the subject and the image quality described with reference to FIGS. 20 to 22 and the image quality evaluation curve 205 is an example in the case where the thinning amount m = 4. The evaluation curves are different, and it is necessary to obtain an image quality evaluation curve according to the parameters and perform a thinning mode determination process based on the obtained image quality evaluation curve.

  Further, regarding the thinning process when the sample point is fixed, in the example of the horizontal movement described with reference to FIG. 15, the leftmost pixel value p1 in 1 × 4 pixels is selected, and FIG. In the example of the vertical movement described with reference to the above, the pixel value p1 at the upper end of the 1 × 4 pixels is selected, but it may be a pixel at another position. Also, in the example of FIG. 16 and FIG. 17 described as the thinning-out process example for moving the sample point, the sample pixel is moved to the right or left starting from the left or right pixel value p1 in 1 × 4 pixels. However, it does not matter if it starts from another position. The same applies to the case of the vertical direction in FIGS.

  Further, in the above-described example, whether one reference evaluation value T is determined and whether the sample point fixed type spatial thinning process is performed or the sample point moving type spatial thinning process is performed depending on whether or not the reference evaluation value T is exceeded. In the case of performing the moving type space thinning process, it has been determined how to move the thinning position. However, the configuration of FIG. 18 is changed as shown in FIG. 28 and the allowable speed change amounts Δv1 to Δv9 (≧ 0). And determining the spatial thinning method with reference to FIG. 29, the change in the spatial thinning method between frames can be stabilized. The fact that the space thinning method changes frequently over time means that the processing applied to the block frequently fluctuates, and in such a situation, there is a possibility that image quality degradation may occur. By using the configuration as described above (FIG. 28) and the allowable speed change amounts (Δv1 to Δv9), it is possible to suppress the deterioration of the image quality as described above. The flow of determination of the space thinning method when the configuration of FIG. 28 is used will be described below.

  28 shows a configuration in which a memory 153 is newly connected between the movement amount detection unit 121 and the thinning processing mode determination unit 151 in FIG. By using this memory 153, the thinning-out processing mode determination unit 151 can input the movement amount p of the past frame in addition to the movement amount v of the currently processed frame. Here, the processing of the thinning-out processing mode determination unit 151 will be described with reference to FIG. The thinning processing mode determination unit 151 determines the thinning processing mode as follows.

First, the thinning processing mode determination unit 151 refers to the movement amount p regarding the block of the immediately preceding frame that is an input from the memory 153.
The movement amount p is
(1) When 2 ≦ p <c,
If the movement amount v of the block of the current frame input from the movement amount detection unit 121 is 2−Δv1 ≦ v <c + Δv2, the thinning processing mode determination unit 151 performs processing 3 (subject of object) as shown in FIG. It is decided to change the thinning position in the direction opposite to the traveling direction. When the movement amount p does not satisfy the above condition, the processing is determined with reference to the reference shown in FIG.

Also, the movement amount p is
(2) When c ≦ p <d,
If the movement amount v of the block of the current frame input from the movement amount detection unit 121 is c−Δv3 ≦ v <d + Δv4, the thinning processing mode determination unit 151 performs processing 1 (decimation position) as shown in FIG. Do not make changes). When the movement amount p does not satisfy the above condition, the processing is determined with reference to the reference shown in FIG.

Also, the movement amount p is
(3) When d ≦ p <f,
If the movement amount v of the block of the current frame input from the movement amount detection unit 121 is d−Δv5 ≦ v <f + Δv6, the thinning processing mode determination unit 151 performs processing 2 (subject of object) as shown in FIG. (Decimation position is changed in the same direction as the traveling direction). When the movement amount p does not satisfy the above condition, the processing is determined with reference to the reference shown in FIG.

Also, the movement amount p is
(4) When f ≦ p <e,
If the movement amount v of the block of the current frame input from the movement amount detection unit 121 is f−Δv7 ≦ v <e + Δv8, the thinning processing mode determination unit 151 performs processing 3 (subject of object) as shown in FIG. It is decided to change the thinning position in the direction opposite to the traveling direction. When the movement amount p does not satisfy the above condition, the processing is determined with reference to the reference shown in FIG.

Also, the movement amount p is
(5) When e ≦ p,
If the movement amount v of the block of the current frame input from the movement amount detection unit 121 is e−Δv9 ≦ v, the thinning processing mode determination unit 151 performs processing 1 (change of the thinning position) as shown in FIG. Do not do). When the movement amount p does not satisfy the above condition, the processing is determined with reference to the reference shown in FIG.

  As described above, when the thinning processing mode determination unit 151 determines the thinning processing method, regarding the same processing as the past processing method, the thinning method is increased with time by expanding the range of the speed to which the processing method is applied. It is also possible to suppress frequent fluctuations.

  The allowable speed change amounts Δv1 to Δv9 can be determined by an arbitrary method. For example, a new reference evaluation value T1 other than the reference evaluation value T is set as shown in FIG. There is a method of setting the amount of change. Specifically, when determining Δv3 and Δv4, the velocities in the vicinity of c and d at which the evaluation values are T1 are c ′ and d ′, respectively, and the absolute values of the differences between c and c ′ and d and d ′ are determined. They are set as Δv3 and Δv4, respectively.

  With reference to FIG. 28, the configuration and processing of the block processing unit 131 that executes the above-described processing will be described. The memory 153 holds a thinning mode determination result for past frames by the thinning mode determination unit 151. The thinning mode determination unit 151 performs the thinning mode according to the subject moving speed and the image quality evaluation value correspondence data generated based on the processing data when the sampling point fixed thinning processing is executed, and the thinning mode determination result for the past frame. Perform decision processing.

  In other words, the thinning mode determination unit 151 has an image quality corresponding to the subject moving speed corresponding to the moving amount information detected by the moving amount detection unit 121, in which the thinning mode determination result for the past frame is the sampling point fixed thinning process. When the evaluation value is equal to or greater than a predetermined reference evaluation value T1, execution of the sampling point fixed thinning process is determined, the thinning mode determination result for the past frame is the sampling point fixed thinning process, and the movement amount When the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the detection unit 121 is less than the predetermined reference evaluation value T1, execution of the sampling point movement type thinning process is determined, and a past frame is determined. Corresponds to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit 121. If the image quality evaluation value is less than a predetermined reference evaluation value T2, the execution of the sampling point movement type thinning process is determined, the thinning mode determination result for the past frame is the sampling point movement type thinning process, and When the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit 121 is equal to or higher than a predetermined reference evaluation value T2, execution of the sampling point fixed thinning process is determined.

  In addition, the memory 153 holds the movement mode of the sample points when the block processing unit 131 performs the spatial thinning process on the past frame. The block processing unit 131 moves sample points at the time of spatial decimation with respect to past frames in the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed. When the image quality evaluation value corresponding to the virtual subject movement speed corresponding to the aspect is equal to or higher than a predetermined reference evaluation value T3, spatial thinning in which the movement mode of the sample points at the time of spatial thinning for the past frame is set. In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the processing is executed and the sampling point fixing thinning processing is executed, the movement mode of the sampling points at the time of spatial thinning with respect to the past frame If the image quality evaluation value corresponding to the moving speed of the virtual subject in accordance with is lower than the predetermined reference evaluation value T3, the sampling point fixing thinning process is performed. In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data in the case of the image processing, the sample speed is thinned out in the moving speed area where the image quality evaluation value is equal to or higher than a predetermined reference evaluation value T4. Spatial thinning processing is performed in which the movement mode of the sample points is set so that the resulting moving speed of the virtual subject is set.

  Subsequently, the operation of the block processing unit 133 will be specifically described before the description of the block processing unit 132.

  The block processing unit 133 is supplied from the block distribution unit 122 of the movement amount detection unit 120 and has a total of N blocks at the same position in each of consecutive N frames (both movement amounts in the horizontal direction and the vertical direction are one pixel). (N blocks in the case where the number is less), a process of thinning out the number of frames (time direction thinning process) is performed.

  Specifically, as shown in FIG. 31, the block processing unit 133 converts four blocks Bi at the same position in four consecutive frames F1 to F4 into one block (in this example, Then, the number of frames to be converted into the block Bi) of the frame F1 is thinned out (thinning of the number of frames between four frames).

  The block processing unit 133 supplies the output unit 140 with data (four blocks) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing. Although the case where N = 4 has been described as an example here, similar processing is performed even if N is another value.

  In the case of this example, as shown in FIG. 31, the example in which the block Bi of the frame F1 is output among the blocks Bi of the four frames F1 to F4 is shown. However, blocks of other frames are output. You may make it do. In addition, a block newly obtained by calculation using the frames F1 to F4 may be output.

  Next, the operation of the block processing unit 132 will be specifically described. The block processing unit 132 is supplied from the block distribution unit 122 of the movement amount detection unit 120 and has a total of N blocks (the movement amount in the horizontal direction and the vertical direction is 1 pixel) at the same position in each of the consecutive N frames. As described above, the number-of-pixels thinning process (spatial direction thinning process) and the number of frames thinning-out process (time direction thinning process) are respectively performed on N blocks in the case of less than 2 pixels.

  The moving speed: v = 1 to 2 as the moving amount per frame of the block supplied to the block processing unit 132 satisfies Expressions (Expression 1) and (Expression 2). That is, the conditions for obtaining the super-resolution effect are satisfied. However, it has been found that image quality degradation may be perceived if thinning processing is performed with sample points fixed as described above with reference to FIGS.

  Therefore, the block processing unit 132 executes the thinning process with the thinning amount m = 2 instead of the thinning amount m = 4, and furthermore, the sampling point fixed thinning process similar to the thinning process in the block processing unit 131 is performed. The sampling point moving type thinning process is appropriately selected and executed according to the subject moving speed.

A mode of the sampling point fixed thinning process and the sampling point moving type thinning process when the thinning amount m = 2 is described with reference to FIGS.
FIGS. 32 to 34 are thinning-out processing examples of the block processing unit 132 when the moving speed as the moving amount per frame in the horizontal direction: v = 1 to 2 pixels / frame,
FIG. 32 shows a sampling point fixed thinning process, in which pixel values p1 to p4 of each frame (k to k + 3) are converted into any two pixel values (in this example, p1 and p3). This is an example of thinning out the number of pixels to be thinned out (thinning out the number of pixels between two pixels), that is, the thinning-out amount m = 2.

  FIG. 33 shows a sampling point moving thinning process in which the pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p1 and p3 for the preceding frame (k, k + 1), and the succeeding frame (K + 2, k + 3) is a thinning process for setting the pixel values p2 and p4. FIG. 34 is also a sampling point moving thinning process, in which the pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p2 and p4 for the preceding frame (k, k + 1), and the subsequent frame is set. (K + 2, k + 3) is a thinning process for setting the pixel values p1 and p3.

FIG. 35 to FIG. 37 are examples of decimation processing of the block processing unit 132 when the moving speed as the moving amount per frame in the vertical direction: v = 1 to 2 pixels / frame.
FIG. 35 is a sampling point fixed thinning process, and pixel values p1 to p4 of each frame (k to k + 3) are converted into any two pixel values (in this example, p1 and p3). This is an example of thinning out the number of pixels to be thinned out (thinning out the number of pixels between two pixels), that is, the thinning-out amount m = 2.

  FIG. 36 shows a sampling point moving thinning process, in which the pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p1 and p3 for the preceding frame (k, k + 1), and the subsequent frame is set. (K + 2, k + 3) is a thinning process for setting the pixel values p2 and p4. FIG. 37 is also a sampling point moving thinning process, and the pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p2 and p4 for the preceding frame (k, k + 1), and the subsequent frame is set. (K + 2, k + 3) is a thinning process for setting the pixel values p1 and p3.

  Similarly to the block processing unit 131, the block processing unit 132 has the configuration illustrated in FIG. That is, the block processing unit 132 includes a thinning processing mode determination unit 151 and a thinning processing execution unit 152. The thinning processing mode determination unit 151 inputs movement amount information from the movement amount detection unit 121. As described above, the block input to the block processing unit 132 is a block having a moving speed: v = 1 to 2 pixels / frame as a moving amount per frame. The moving amount information input to the determining unit 151 is moving amount information of moving speed v = 1 to 2 pixels / frame.

  The thinning process mode determination unit 151 determines which mode in FIGS. 32 to 37 performs the thinning process based on the value of the movement amount, and the thinning process execution unit 152 performs the thinning process mode determination unit 151. According to the determination, the thinning process is executed in any of the modes of FIGS.

  Details of the thinning processing mode determination executed by the thinning processing mode determination unit 151 included in the block processing unit 132 will be described with reference to FIGS. 38 to 40. FIGS. 38 to 40 are similar to FIGS. 22 to 24, and are generated by subjecting the subject to thinning processing with a fixed sample point, that is, performing thinning processing (space / time thinning processing) in FIG. 32 or FIG. This is a graph generated by evaluating the image quality when image data is restored and played back, and the tendency of the result of the subjective evaluation experiment when the subject is moved to the right by the amount of movement shown on the horizontal axis is the image quality evaluation curve 215. The vertical axis represents image quality evaluation (evaluation that the higher the image quality, the better).

  As described above with reference to FIG. 20, the reference evaluation value indicated by T in FIG. 38 is set, and when the evaluation equal to or higher than the reference evaluation value T is obtained, the image quality is good. As shown. The area B other than the area A is an area having an image quality less than the reference evaluation value T. In the case of the subject speed included in this area, the super-resolution effect is obtained when the sampling point fixed thinning process is performed. It does not occur or the effect is considered to be insufficient.

  Therefore, when the moving speed is the speed of the region B, the moving speed of the subject is conceptually accelerated or decelerated by the thinning position changing process. That is, in the case of horizontal movement, the sampling point movement type thinning process of FIG. 33 or 34 is executed, and in the case of vertical movement, the sampling point movement type thinning process of FIG. 36 or FIG. Execute. By this processing, the same image quality as that when moving at the speed of the region A is obtained.

  Note that the block processing unit 132 also performs a thinning process in the time direction. The thinning process in the time direction is the same process as described above with reference to FIG. 12, and is executed, for example, as a process of selecting a 2 frame block from a continuous 4 frame block. When the temporal direction thinning process is performed prior to the spatial direction thinning process, the frame rate is reduced to ½ at the stage of executing the spatial direction thinning process. The conceptual change in subject speed is ± 0.5 pixels / frame.

  Regions B-1 and B-2 in FIG. 39 are a part of region B shown in FIG. That is, it is an image quality region that is less than the reference evaluation value T, and is a region that is considered to be a region that does not generate a super-resolution effect or has an insufficient subject moving speed: v. When the subject is moving at the speed of the areas B-1 and B-2, the subject is virtually accelerated (+ Δv1, + Δv2), that is, in the direction opposite to the traveling direction of the subject. By performing the process of changing the thinning position, it is possible to obtain the same effect as when the region B-1 in FIG. 39 is moved to the region C-1 and the region B-2 is moved to the region C-2. As described above, it is possible to set the moving speed: v as the moving amount of the subject per virtual frame to the image quality region (C-1, C-2 in FIG. 39) that is equal to or higher than the reference evaluation value T. That is, image conversion (compression) with little deterioration in image quality is realized by executing a thinning process in which the sample point position is moved as the frame of FIG. 33, FIG. 34, FIG. 36, or FIG.

  Similarly, when the subject is moving at the speed of the area B-3 in FIG. 40, a process of virtually decelerating the subject (−Δv3) is performed, that is, the thinning position is changed in the same direction as the traveling direction. (For example, if moving to the right, the sampling point moving type thinning process in FIG. 33 is performed), thereby moving the area B-3 in FIG. 40 to the area C-3. Can be set in an image quality region (C-3 in FIG. 40) that is equal to or greater than the reference evaluation value T, and image conversion (compression) with little image quality degradation is realized.

  In the moving image processing apparatus 100 in this configuration example, the block processing unit 132 performs the two-pixel thinning process in the spatial direction only when the amount of movement per frame in the horizontal or vertical direction is not less than 1 pixel and less than 2 pixels. Therefore, in the graphs shown in FIG. 38 to FIG. 40, the portion where the moving speed: v is 1 pixel / frame or less is not a processing target. However, as described above, the block distributor 122 uses each block of the block processor 130. The conditions for sending blocks to the processing units 131 to 133 are merely examples, and the block distribution unit 122 sends blocks to the block processing unit 132 even if the movement amount is less than 1 pixel. Considering that it is possible to select one of the thinning modes described with reference to the spatial thinning configuration I was painting.

  Note that the image quality evaluation curves shown in FIGS. 38 to 40 show an evaluation curve set based on the result of one experiment based on subjective evaluation, and various image quality evaluation methods can be applied. A configuration may be adopted in which the thinning processing mode is determined based on the result.

The block processing unit 132 shown in FIG. 14 that executes processing of block data whose moving amount per frame is 1 pixel or more and less than 2 pixels is the moving amount information from the moving amount detection unit in the thinning processing mode determination unit 151 shown in FIG. On the basis of the,
(A) Thinning process for fixing sample points (see FIGS. 32 and 35)
(B) Thinning processing for shifting the sample point position in the subject moving direction according to the frame progress (see FIGS. 33, 34, 36, and 37)
(C) Thinning processing for shifting the sample point position in the direction opposite to the subject moving direction according to the progress of the frame (see FIGS. 33, 34, 36, and 37)
Is executed, and the thinning process of the processing mode determined by the thinning process execution unit 152 shown in FIG. 18 is executed.

  In determining the thinning processing mode in the thinning processing mode determination unit 151 illustrated in FIG. 18, for example, determination processing based on the image quality evaluation curves illustrated in FIGS. 38 to 40 is performed. That is, when the image quality evaluation curve is a moving speed region in which the image quality equal to or higher than the reference evaluation value T is maintained, (a) the sampling point fixing thinning process is executed, and the image quality evaluation curve becomes an image quality lower than the reference evaluation value T. In the moving speed region, the sampling point moving type thinning process is executed, and the virtual moving amount is set to enter the moving amount region where the image quality evaluation curve maintains the image quality equal to or higher than the reference evaluation value T.

  The block processing unit 132 performs different thinning processing according to the horizontal movement amount received from the movement amount detection unit 121, that is, the movement speed: v as the movement amount per frame. The correspondence between the moving speed: v and the thinning processing mode is shown in FIG. FIG. 41B is a table for explaining the processes 1 to 3 shown in FIG. The symbols i, j, m, and k in FIG. 41A correspond to the subject speed shown on the horizontal axis in FIGS.

The contents of FIG. 41 (a) will be described.
If 1 ≦ v <i,
Process 3: The thinning position is changed in the direction opposite to the traveling direction of the subject.
If i ≦ v <j,
Process 1: The thinning position is not changed.
If j ≦ v <m,
Process 2: The thinning position is changed in the same direction as the moving direction of the subject.
If m ≦ v <k,
Process 3: The thinning position is changed in the direction opposite to the traveling direction of the subject.
If k ≦ v,
Process 1: The thinning position is not changed.

  As described above, by changing the thinning processing mode according to the moving speed: v as the moving amount of the subject included in the processing block, the virtual moving speed of the subject can be changed to the area A shown in FIG. Therefore, image conversion that guarantees an image quality equal to or higher than the reference image quality evaluation value T is executed.

  In this way, the block processing unit 132 executes a thinning process that differs depending on the moving speed: v as the moving amount of the subject included in the processing block, and outputs it to the output unit 140 (see FIGS. 13 and 14). Output. Since the block processing unit 132 performs the thinning-out amount 2 in the spatial direction thinning process and also performs the time direction thinning-out (4 frames → 2 frames) shown in FIG. 12, the data amount of each block is reduced to ¼, The data amount of all four blocks is reduced to ¼. The block processing unit 131 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  Note that the same result can be obtained for either the spatial direction thinning or the time direction thinning, whichever is performed first. Further, since this processing can be performed on N / 2 frames and N / 2 adjacent pixels when N is a positive even number, it is not necessarily 2 (= 4/2) frames, It need not be performed in units of two pixels.

  In the above-described embodiment, the parameter setting example in which the thinning amount m = 4 and the number of frames N = 4 executed in the block processing unit 131 has been described. However, other values may be applied to the values of these parameters. Is also possible. However, the relationship between the moving speed of the subject and the image quality described with reference to FIGS. 38 to 40 and the image quality evaluation curve 215 are different when the parameter is changed, and an image quality evaluation curve corresponding to the parameter is obtained and obtained. It is necessary to perform a thinning mode determination process based on the image quality evaluation curve.

  Next, the output unit 140 will be described. The output unit 140 indicates the data about N blocks supplied from each of the block processing units 131 to 133 of the block processing unit 130 and the amount of data reduced, and how each block is processed. Output information. Information on the processing contents is information indicating whether thinning is performed by spatial thinning, temporal thinning, or spatial / temporal thinning. Information that indicates what kind of thinning position change processing has been performed, information about the frame rate of the original video, spatial resolution, etc., but other information is added. It doesn't matter.

  Next, the operation sequence of the movement amount detection unit 120 and the block processing unit 130 of the moving image conversion apparatus 100 will be described with reference to the flowchart of FIG.

  In step S <b> 1, the movement amount detection unit 121 of the movement amount detection unit 120 includes the P-th frame among the four frames continuously input from the block division unit 112 of the block division unit 110 to the moving image conversion apparatus 100. The supply of each block of the frame (search target block) and the supply of the Mth frame (reference frame) from the image storage unit 111 are received and input.

  Next, in step S2, the movement amount detection unit 121 sets one of the blocks of the Pth frame as a search target block and detects a motion vector of the search target block with reference to the reference frame. . The movement amount detection unit 121 supplies the detected motion vector to the block distribution unit 122.

  In step S <b> 3, the block distribution unit 122 determines the magnitude of the motion vector supplied from the movement amount detection unit 121 in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) (the X-axis direction or Y-axis between one frame). It is determined whether or not the larger movement amount in the axial direction is 2 pixels or more. If it is determined that at least one movement amount is 2 pixels or more, the process proceeds to step S4.

Step S4 is a determination process executed as a thinning mode determination process in the block processing unit 131 of the block processing unit 130, and is a process in the thinning process mode determination unit 151 shown in FIG. That is, the thinning processing mode determination unit 151 performs the above-described FIGS. 15, 16, and 17 (in the case of horizontal movement) or FIGS. 24, 25, and 26 (vertical) based on the value of the movement amount. In the case of direction movement), it is determined whether the thinning process is executed. The thinning processing mode determination unit 151 is based on the movement amount information from the movement amount detection unit 121.
(A) Sample point fixed thinning process (see FIGS. 15 and 24)
(B) Sample point moving thinning process (see FIGS. 16, 17, 25, and 26)
Decide which one to execute.

  In determining the thinning processing mode in the thinning processing mode determination unit 151, for example, determination processing based on the image quality evaluation curves shown in FIGS. That is, the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) of the motion vector supplied from the movement amount detection unit 121 (the movement amount in the X-axis direction or Y-axis direction between one frame) is When in the moving speed region where the image quality equal to or higher than the reference evaluation value T is maintained, (a) the sampling point fixed thinning process (see FIGS. 15 and 24) is executed and supplied from the moving amount detection unit 121. When the magnitude of the motion vector is in an area where the image quality evaluation curve has an image quality less than the reference evaluation value T, (b) sampling point movement type thinning processing (see FIGS. 16, 17, 25, and 26) The virtual movement amount is set to be in the movement amount region where the image quality evaluation curve maintains the image quality equal to or higher than the reference evaluation value T.

  In step S <b> 4, the thinning processing mode determination unit 151 determines the magnitude (horizontal direction (X-axis direction) or vertical direction (Y-axis direction) of the motion vector supplied from the movement amount detection unit 121 (X-axis direction between one frame). If it is determined that the movement amount in the Y-axis direction is in the movement speed region where the image quality equal to or higher than the reference evaluation value T is maintained, it is determined that the movement amount is not to be changed, and the process proceeds to step S6. . In step S6, the thinning process execution unit 152 shown in FIG. 18 executes the sampling point fixing thinning process (see FIGS. 15 and 24).

  On the other hand, the thinning-out processing mode determination unit 151 performs the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) size of the motion vector supplied from the movement amount detection unit 121 (the X-axis direction or Y-axis between one frame). If it is determined that the moving amount in the axial direction is not in the moving speed region in which the image quality equal to or higher than the reference evaluation value T is maintained, it is determined that the moving amount should be changed, and the process proceeds to step S5. In step S5, the thinning process execution unit 152 shown in FIG. 18 executes a sampling point moving thinning process (see FIGS. 16, 17, 25, and 26). Note that (b) the thinning processing mode determining unit 151 also determines whether to move the sample point position in the moving direction of the subject or in the reverse direction according to the progress of the frame, and the thinning processing according to this determination. Is executed by the thinning process execution unit 152 (see FIG. 18).

  As a result of the thinning process executed by the block processing unit 131 in step S5 or step S6, for example, four blocks supplied from the block distribution unit 122 are shown in FIG. 15 to FIG. 17 or FIG. 24 to FIG. Data for four blocks in which a thinning process (thinning process for the number of pixels between four pixels) is performed to reduce the number of pixels shown to ¼, and the resulting data amount is reduced to ¼, This is supplied to the output unit 140.

  The above-described data example is an example of the data amount when parameters are set such that the thinning amount m = 4 and the number of processing unit frames N = 4. Other values can be applied to the values of each parameter, and conversion data with different compression rates is generated according to the applied parameters. However, as described above, the relationship between the moving speed of the subject and the image quality described with reference to FIGS. 20 to 22 and the image quality evaluation curve depend on parameters, and when different parameters are applied, It is necessary to obtain an image quality evaluation curve corresponding to the applied parameter and perform a thinning mode determination process based on the obtained image quality evaluation curve.

  In step S <b> 3, the block distribution unit 122 determines the magnitude of the motion vector supplied from the movement amount detection unit 121 in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) (the X-axis direction or Y-axis between one frame). If it is determined that none of the movement amounts in the axial direction is 2 pixels or more, the process proceeds to step S7. Further, in step S7, the block distribution unit 122 determines the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) of the motion vector supplied from the movement amount detection unit 121 (X-axis direction between one frame). If it is determined that any of the movement amounts in the Y-axis direction) is less than one pixel, the process proceeds to step S8.

  Step S8 is processing of the block processing unit 133 of the block processing unit 130, and executes only thinning processing in the time direction. The block processing unit 133 includes a total of N blocks (in the horizontal and vertical directions) at the same position in each of consecutive N frames supplied from the block distribution unit 122 of the movement amount detection unit 120. A process of thinning out the number of frames (time direction thinning process) is performed on N blocks when the movement amounts are both less than one pixel.

  Specifically, as described above with reference to FIG. 31, the block processing unit 133 converts four blocks Bi at the same position in each of the four consecutive frames F1 to F4 into one of them. Decimation of the number of frames (decimation of the number of frames between four frames) is performed for the block (in this example, the block Bi of the frame F1).

  The block processing unit 133 supplies the output unit 140 with data (four blocks) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing. Although the case where N = 4 has been described as an example here, similar processing is performed even if N is another value.

  In the case of this example, as shown in FIG. 31, the example in which the block Bi of the frame F1 is output among the blocks Bi of the four frames F1 to F4 is shown. However, blocks of other frames are output. You may make it do. In addition, a block newly obtained by calculation using the frames F1 to F4 may be output.

  In step S7, the block distribution unit 122 determines the magnitude of the motion vector supplied from the movement amount detection unit 121 in the horizontal direction (X-axis direction) or the vertical direction (Y-axis direction) (the X-axis direction or Y-axis between one frame). If it is determined that any of the movement amounts in the axial direction is not less than one pixel, the process proceeds to step S9.

Step S9 is a determination process executed as a thinning mode determination process in the block processing unit 132 of the block processing unit 130, and is a process in the thinning process mode determination unit 151 shown in FIG. That is, the thinning processing mode determination unit 151 performs the above-described FIGS. 32, 33, and 34 (in the case of horizontal movement), or FIGS. 35, 36, and 37 (vertical) based on the value of the movement amount. In the case of direction movement), it is determined whether the thinning process is executed. The thinning processing mode determination unit 151 is based on the movement amount information from the movement amount detection unit 121.
(A) Sample point fixed thinning process (see FIGS. 32 and 35)
(B) Sample point moving thinning process (see FIGS. 33, 34, 36, and 37)
Decide which one to execute.

  In determining the thinning processing mode in the thinning processing mode determination unit 151, for example, determination processing based on the image quality evaluation curve shown in FIGS. 38 to 40 is performed. That is, the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) of the motion vector supplied from the movement amount detection unit 121 (the movement amount in the X-axis direction or Y-axis direction between one frame) is When in the moving speed region where the image quality equal to or higher than the reference evaluation value T is maintained, (a) a sampling point fixed thinning process (see FIGS. 32 and 35) is executed and supplied from the moving amount detection unit 121. When the magnitude of the motion vector is in an area where the image quality evaluation curve has an image quality less than the reference evaluation value T, (b) sampling point movement type thinning processing (see FIGS. 33, 34, 36, and 37) The virtual movement amount is set to enter the movement amount region where the image quality evaluation curve maintains the image quality equal to or higher than the reference evaluation value T.

  In step S <b> 9, the thinning processing mode determination unit 151 determines the magnitude (horizontal direction (X-axis direction) or vertical direction (Y-axis direction) of the motion vector supplied from the movement amount detection unit 121 (X-axis direction between one frame). If it is determined that the movement amount in the Y-axis direction is in the movement speed region where the image quality equal to or higher than the reference evaluation value T is maintained, it is determined that the movement amount is not to be changed, and the process proceeds to step S11. . In step S11, the thinning process execution unit 152 shown in FIG. 18 executes a sampling direction fixed spatial direction thinning process (see FIGS. 32 and 35). Further, the thinning process in the time direction described above with reference to FIG. 12 is executed. Note that either the spatial direction thinning process or the time direction thinning process may be executed first.

  On the other hand, the thinning-out processing mode determination unit 151 performs the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) size of the motion vector supplied from the movement amount detection unit 121 (the X-axis direction or Y-axis between one frame). If it is determined that the moving amount in the axial direction is not in the moving speed region in which the image quality equal to or higher than the reference evaluation value T is maintained, it is determined that the moving amount should be changed, and the process proceeds to step S10. In step S10, a sampling point moving type thinning process (see FIGS. 33, 34, 36, and 37) is executed in the thinning process execution unit 152 shown in FIG. Further, the thinning process in the time direction described above with reference to FIG. 12 is executed. Note that either the spatial direction thinning process or the time direction thinning process may be executed first.

  Note that (b) the thinning processing mode determining unit 151 also determines whether to move the sample point position in the moving direction of the subject or in the reverse direction according to the progress of the frame, and the thinning processing according to this determination. Is executed by the thinning process execution unit 152 (see FIG. 18).

  In step S10 or step S11, as a result of the thinning process executed by the block processing unit 132, for example, for four blocks supplied from the block distribution unit 122, a spatial direction thinning process with a thinning amount m = 2 and a time The direction thinning process is executed, and the data for the four blocks whose data amount obtained as a result is reduced to ¼ is supplied to the output unit 140.

  As described above, various values such as the thinning-out amount m and the number of processing unit frames can be applied, and conversion data having different compression rates is generated according to the applied parameters. It will be. However, as described above, the relationship between the moving speed of the subject and the image quality described with reference to FIGS. 38 to 40 and the image quality evaluation curve depend on the parameters. When different parameters are applied, It is necessary to obtain an image quality evaluation curve corresponding to the applied parameter and perform a thinning mode determination process based on the obtained image quality evaluation curve.

  The above processing is performed every time each block of N (for example, N = 4) frames is supplied from the block dividing unit 110.

  13 is connected to a storage medium such as a hard disk or a DVD, or network output means, for example, for storing data compressed by data conversion or network output. Is executed.

[(3) Moving Image Restoration Apparatus and Moving Image Restoration Method]
Next, a moving image restoration apparatus and a moving image restoration method for executing restoration processing of converted (compressed) data generated by the above-described moving image conversion apparatus 100 will be described.

  The configuration of the moving image restoration apparatus 300 is shown in FIG. As shown in FIG. 43, the moving image restoration apparatus 300 includes a distribution unit 310, block expansion units 321 to 323, and a synthesis unit 330.

  The distribution unit 310 receives conversion (compression) data conversion data generated by the above-described moving image conversion apparatus 100 and attribute data necessary for restoration. The attribute data includes information related to each block and information related to the specific contents of processing performed on each block. The specific contents of the processing applied to each block are either spatial direction thinning processing or temporal direction thinning processing executed, and the thinning processing executed in the spatial thinning processing is a sample point fixed type or a sample point This is information including sample point movement direction information in the case of the moving type or the sample point moving type.

  Based on the processing contents of each block included in the input attribute information, the distribution unit 310 sends the conversion data to be restored and the processing performed on each block to any of the block expansion units 321 to 323. Attribute information indicating the target content is sent.

  Specifically, when the block is data processed by the block processing unit 131 in the moving image conversion apparatus 100 illustrated in FIGS. 13 and 14, the distribution unit 310 decimates each block that has been thinned into the block expansion unit 321. Data and information such as the direction of the thinning position, whether or not the thinning position is changed, and the direction of the thinning position change are sent.

  In addition, when the data is processed by the block processing unit 132 of the moving image conversion apparatus 100, the same information is sent to the block expansion unit 322. If processed by the block processing unit 133, the same information is sent to the block extension unit 323. The information related to the processing applied to each block and the information related to each block may be any information as long as it is sufficient information for restoring a moving image.

  The block extension unit 321 executes an extension process of data that has been subjected to the thinning process in the spatial direction in the block processing unit 131 of the moving image conversion apparatus 100. Based on information such as the spatial thinning direction input from the distribution unit 310, whether or not the thinning position is changed, and the direction of the thinning position change, the data expansion processing in the form shown in FIGS. 44 (a) and (b) is performed. Run and reconstruct the block. Further, the information indicating the reconfigured block and the content of the process is output to the synthesis unit 330.

  The processing of FIGS. 44A and 44B will be described. FIG. 44A shows processing by the block expansion unit 321 of the moving image restoration apparatus 300 when the horizontal thinning process is performed in the block processing unit 131 of the moving image conversion apparatus 100. For example, when the size of the original block is 8 × 8, data corresponding to 1/4 of the 16 pixels is sent to the block extension unit 321. With respect to the data of each pixel, 1 pixel is expanded to 1 × 4 pixels by performing the processing of FIG. The processing example shown in FIG. 44 (a-1) is an example in which expansion is performed by arranging the pixel values of input pixels as pixel values for four pixels as they are. The result of the calculation based on the pixel value including may be arranged.

  Subsequently, when the thinning is performed in the horizontal direction, the block expanding unit 321 performs the process shown in FIG. That is, by arranging the set of 1 × 4 pixels obtained by the processing of (a-1) as shown in (a-2), an 8 × 8 block is restored from the input 16 pixels, and the result is output to the combining unit 330. To do. Information indicating the contents of the thinning process is also output.

  FIG. 44B shows processing performed by the block extension unit 321 when the block processing unit 131 of the moving image conversion apparatus 100 performs thinning processing in the vertical direction. For example, when the size of the original block is 8 × 8, data corresponding to 1/4 of the 16 pixels is sent to the block extension unit 321. For the data of each pixel, 1 pixel is expanded to 4 × 1 pixels by performing the process of FIG. 44 (b-1). Note that the processing example shown in FIG. 44 (b-1) is an example in which expansion is performed by arranging the pixel values of the input pixels as pixel values for four pixels as they are, but the values of the other input pixels The result of the calculation based on the pixel value including may be arranged.

  Subsequently, when the thinning is performed in the vertical direction, the block extension unit 321 performs the process shown in FIG. That is, by arranging the 4x1 pixel set obtained by the process of Fig. 44 (b-1) as shown in Fig. 44 (b-2), the 8x8 block is restored from the input 16 pixels and the result is synthesized. To the unit 330. Information indicating the contents of the thinning process is also output.

  Next, the process of the block extension unit 323 will be described. The block extension unit 323 executes the extension process of data that has been subjected to the thinning process in the time direction in the block processing unit 133 of the moving image conversion apparatus 100.

  As illustrated in FIG. 45, the block extension unit 323 generates block data of a plurality of frames based on block data of one frame. Specifically, the block extension unit 323 has a counter with an initial value of 0, and is incremented by 1 every time restoration of one frame is completed, and the value reaches 4 (= N (when the thinning amount m = 4)). Reset to 0.

  When data to be expanded is input from the data distribution unit 310, the block is stored in the memory of the data expansion unit 323, and the block data stored in the memory until reaching the counter upper limit value set according to the thinning amount Is generated as block data corresponding to a plurality of frame data, and is output to the synthesis unit 330. In the example shown in the figure, the thinning-out amount m = 4, and block data corresponding to four frames is generated from one block data and output to the combining unit 330.

  Next, the processing of the block extension unit 322 will be described. The block extension unit 322 executes extension processing of data that has been subjected to thinning processing in the spatial direction and time direction in the block processing unit 132 of the moving image conversion apparatus 100. . Any of the expansion processing in the spatial direction and the temporal direction may be executed first.

  The time direction expansion processing executed in the block expansion unit 322 performs processing as shown in FIG. Specifically, the block extension unit 322 has a counter with an initial value of 0, and is incremented by 1 every time one frame is restored, and the value is 2 (= N / 2 (however, the time direction decimation amount m = 2) In case))) is reset to 0.

  When data to be expanded is input from the data distribution unit 310, the block is stored in the memory of the data expansion unit 322, and the block data stored in the memory until reaching the counter upper limit value set according to the thinning amount Is generated as block data corresponding to a plurality of frame data, and is output to the synthesis unit 330. In the example shown in the figure, the time direction thinning-out amount m = 2, and block data corresponding to two frames is generated from one block data.

  Next, spatial direction expansion processing executed by the block expansion unit 322 will be described. The block expansion unit 322 is based on the information input from the distribution unit 310, such as the spatial thinning direction, whether or not the thinning position is changed, and the direction of the thinning position change, as shown in FIGS. 47 (a) and 47 (b). The data expansion process is executed in the above and the block is reconfigured.

  The processing of FIGS. 47A and 47B will be described. FIG. 47A shows processing performed by the block expansion unit 322 of the moving image restoration apparatus 300 when the block processing unit 132 of the moving image conversion apparatus 100 performs horizontal thinning processing. For example, when the size of the original block is 8 × 8, data corresponding to half of 32 pixels is sent to the block extension unit 322. In correspondence with the data of each two pixels, the processing of FIG. 47A-1 is performed to expand the two pixels to 1 × 4 pixels. The processing example shown in FIG. 47 (a-1) is an example in which expansion is performed by arranging the input pixel values of the two pixels as they are as the pixel values for four pixels. You may arrange | position the result by the calculation based on the pixel value containing a value.

  Subsequently, when the thinning is performed in the horizontal direction, the block extension unit 322 performs the process shown in FIG. That is, by arranging the set of 1 × 4 pixels obtained by the processing of (a-1) as shown in (a-2), an 8 × 8 block is restored from the input 32 pixels, and the result is output to the combining unit 330. To do. Information indicating the contents of the thinning process is also output.

  FIG. 47B shows processing performed by the block extension unit 322 when the block processing unit 132 of the moving image conversion apparatus 100 performs thinning processing in the vertical direction. For example, when the size of the original block is 8 × 8, data corresponding to half of 32 pixels is sent to the block extension unit 322. In correspondence with the data of each of the two pixels, the processing of FIG. 47B-1 is performed to expand the two pixels to 4 × 1 pixels. Note that the processing example shown in FIG. 47 (b-1) is an example in which the input pixel values of the two pixels are arranged as they are as the pixel values for four pixels, but expansion is performed. You may arrange | position the result by the calculation based on the pixel value containing a value.

  Subsequently, when the thinning is performed in the vertical direction, the block extension unit 322 performs the process shown in FIG. 47 (b-2). That is, by arranging the set of 4x1 pixels obtained by the process of Fig. 47 (b-1) as shown in Fig. 47 (b-2), an 8x8 block is restored from the input 32 pixels and the result is synthesized. To the unit 330. Information indicating the contents of the thinning process is also output.

  When the blocks input from the block expansion units 321 to 323 reach an amount that can represent the entire frame, the combining unit 330 appropriately arranges the blocks according to the information indicating the contents of the thinning process that is input simultaneously. Restore the frame and output the entire restored frame. The block arrangement process will be described below with reference to FIGS.

  48 to 51 show an example of data of one frame composed of a plurality of blocks. 48 to 51, one frame data is composed of 9 blocks of 3 × 3. In each figure, each of the squares divided by the broken line is a block, and a square painted in gray, for example, in FIG. 48, the block 401 is input by the combining unit 330 from the block expansion units 321 to 323, The block to be placed.

  FIG. 48 shows the block arrangement position when the synthesis unit 330 inputs from the block expansion units 321 to 323 and the block to be arranged is a block that has not been subjected to the thinning position changing process. In this case, the synthesizing unit 330 arranges the restored block at the same position as when it was first divided in the block dividing unit 110 in the moving image conversion apparatus 100 shown in FIGS.

  FIG. 49 shows an example of restored block placement processing when a block input from the block expansion units 321 to 323 has been subjected to horizontal thinning position change processing.

  FIG. 49A shows a restoration data block when the restoration block executes the sampling point moving thinning process in which the sampling point position is moved in the right direction according to the progress of the frame as described with reference to FIG. The block of the first frame when moving images are processed in units of 4 (= N) frames, or the sample point position is moved to the left according to the frame progression as described with reference to FIG. This is a restored data block when the sample point moving thinning process is executed, and shows the block arrangement position when the moving image is processed in units of 4 (= N) frames in the fourth frame. Yes. In this case, the block is arranged at a position shifted to the left by one pixel from the original block position.

  FIG. 49B shows a restoration data block when the restoration block executes a sampling point moving thinning process in which the sampling point position is moved in the right direction according to the frame progression as described with reference to FIG. The block of the second frame when moving images are processed in units of 4 (= N) frames, or the sample point position is moved to the left as the frame progresses, for example, as described with reference to FIG. This is a restored data block when the sampling point moving thinning process is executed, and shows the arrangement position of the block when it is a block of the third frame when a moving image is processed in units of 4 (= N) frames. Yes. In this case, the block is arranged at the same position as the original block position.

  FIG. 49C shows a restored data block when the restored block is subjected to the sample point moving thinning process in which the sample point position is moved in the right direction according to the frame progression as described with reference to FIG. The block of the third frame when the moving image is processed in units of 4 (= N) frames, or the sample point position is moved to the left according to the frame progression as described with reference to FIG. This is a restored data block when the sampling point moving thinning process is executed, and shows the arrangement position of the block when it is a block of the second frame when a moving image is processed in units of 4 (= N) frames. Yes. In this case, the block is arranged at a position shifted to the right by one pixel from the original block position.

  FIG. 49D shows a restoration data block when the restoration block executes a sampling point moving thinning process in which the sampling point position is moved in the right direction according to the progress of the frame as described with reference to FIG. The block of the fourth frame when the moving image is processed in units of 4 (= N) frames, or the sample point position is moved to the left according to the frame progression as described with reference to FIG. This is a restored data block when the sampling point moving thinning process is executed, and shows the arrangement position of the block when it is a block of the first frame when a moving image is processed in units of 4 (= N) frames. Yes. In this case, the block is arranged at a position shifted to the right by 2 pixels from the original block position.

  As described above, the synthesizer 330 performs a process of moving the restoration block in the horizontal direction for each frame and arranging it when restoring the thinning process data obtained by executing the process of moving the sample point in the horizontal direction. . The reason for this processing will be described with reference to FIG.

  FIGS. 50A to 50D show partial pixel data (4 × 4) data constituting the block of frame data corresponding to FIGS. 49A to 49D. As shown in FIGS. 50A to 50D, the converted data is generated with the sample point positions shifted by one pixel in the right direction according to the frame.

  When the thinning amount m = 4, for example, all four pixels in the horizontal direction are set to the same pixel value. However, the position of the pixel having the pixel value is originally shifted by one pixel in each frame as shown in FIGS. 50 (a) → (d). When the position of the pixel having the original pixel value is set so as to be substantially at the center of the display area, as shown in FIGS. 50A to 50D, the original data display area indicated by the dotted line frame According to 50 (a) → (d), an arrangement such as [position shifted by 1 pixel to the left] → [position shifted by 2 pixels to the right] is performed. By this display processing, an image closer to the original image data can be reproduced.

  FIG. 51 shows an example of restored block placement processing when a block input from the block expansion units 321 to 323 has been subjected to vertical thinning position change processing.

  FIG. 51A shows a restoration data block when the restoration block executes a sampling point moving thinning process in which the sampling point position is moved downward as the frame progresses as described with reference to FIG. 25, for example. The block of the first frame when the moving image is processed in units of 4 (= N) frames, or the sample point position is moved upward according to the progress of the frame as described with reference to FIG. This is a restored data block when the sample point moving thinning process is executed, and shows the block arrangement position when the moving image is processed in units of 4 (= N) frames in the fourth frame. Yes. In this case, the block is arranged at a position shifted by one pixel from the original block position.

  FIG. 51B is a restored data block when the restoration block executes the sampling point moving thinning process in which the sampling point position is moved downward according to the frame progression as described with reference to FIG. 25, for example. The block of the second frame when the moving image is processed in units of 4 (= N) frames, or the sample point position is moved upward according to the progress of the frame as described with reference to FIG. This is a restored data block when the sampling point moving thinning process is executed, and shows the arrangement position of the block when it is a block of the third frame when a moving image is processed in units of 4 (= N) frames. Yes. In this case, the block is arranged at the same position as the original block position.

  FIG. 51 (c) is a restored data block when the restoration block executes the sampling point moving thinning process in which the sampling point position is moved downward as the frame progresses as described with reference to FIG. 25, for example. The block of the third frame when moving images are processed in units of 4 (= N) frames, or the sample point position is moved upward according to the progress of the frame as described with reference to FIG. This is a restored data block when the sampling point moving thinning process is executed, and shows the arrangement position of the block when it is a block of the second frame when a moving image is processed in units of 4 (= N) frames. Yes. In this case, the block is arranged at a position shifted by one pixel below the original block position.

  FIG. 51D shows a restoration data block when the restoration block executes a sampling point moving thinning process in which the sampling point position is moved downward according to the progress of the frame as described with reference to FIG. The block of the fourth frame when moving images are processed in units of 4 (= N) frames, or the sample point position is moved upward according to the progress of the frame as described with reference to FIG. This is a restored data block when the sampling point moving thinning process is executed, and shows the arrangement position of the block when it is a block of the first frame when a moving image is processed in units of 4 (= N) frames. Yes. In this case, the block is arranged at a position shifted by 2 pixels below the original block position.

  As described above, when performing the restoration of the thinning process data obtained by executing the process of moving the sample point in the vertical direction, the synthesis unit 330 executes the process of moving the restored block in the vertical direction for each frame and arranging the restored block. . The reason for this processing is the same as the horizontal movement processing described with reference to FIG. 50, and is for realizing reproduction of an image closer to the original image data.

  By the way, as described above, when a block is arranged at a position deviated from the original block arrangement, depending on the contents of the processing applied to the adjacent block, for example, adjacent blocks as shown in FIG. May overlap, or as shown in FIG. 52 (b), a gap may be opened between the blocks. In such a case, for example, when overlapping, the average of the overlapping portions of both blocks is used as the pixel value of the overlapping portion, or when the gap is opened, linear interpolation is performed using the pixel values on both sides of the gap. Execute correction processing such as The synthesizing unit 330 performs such pixel value correction processing, and completes and outputs frame data composed of block data.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and executed.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  The various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the present invention, the optimal compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject is determined, and the optimal mode is determined for each region according to the determined mode. By adopting a configuration for performing data conversion processing, compression and decompression with very little quality degradation can be performed.

  In the configuration of the present invention, a subject movement amount in a block constituting moving image data is detected, and based on the detected movement amount, a sampling point fixed type spatial direction thinning process or a sampling point movement type thinning in the spatial direction is performed. Execute the process. The sampling point movement type thinning out in the spatial direction makes it possible to generate a super-resolution effect based on the virtual subject speed setting, thereby realizing data conversion with suppressed image quality degradation.

  According to the configuration of the present invention, the thinning mode determination process according to the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed is executed, and the processing target If the image quality evaluation value corresponding to the subject movement speed corresponding to the block movement amount information is greater than or equal to a predetermined reference evaluation value, a fixed sampling point type thinning process is executed to obtain the subject movement speed corresponding to the movement amount information. If the corresponding image quality evaluation value is less than a predetermined reference evaluation value, it is possible to set a virtual subject speed by executing a sampling point moving type thinning process, and various movements of the processing target block Data conversion with reduced image quality deterioration that causes a super-resolution effect with respect to the amount becomes possible.

  Further, according to the configuration of the present invention, by the block expansion process corresponding to at least one of the sampling point fixed type spatial direction thinning process, the sample point moving type spatial direction thinning process, and the time direction thinning process. Configures a moving image restoration device that executes block restoration, combines the blocks restored by the block expansion process, and generates frame data. Since the arrangement is changed according to the movement mode, a high-quality image close to the original image can be restored.

It is a figure which shows the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure which shows the detail of the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure which shows the structure of the moving image converter of this invention which performs the data conversion using a super-resolution effect. It is a figure which shows the detailed structure of the moving image converter of this invention. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the detailed structure of the block process part in a moving image converter. It is a figure explaining the outline | summary of the thinning process of the sample point fixed type performed in the block processing part in a moving image converter, and the thinning process of a sample point movement type. It is a figure which shows the correlation with a subject moving amount explaining the effect by the thinning process of the sample point movement type performed in the block processing part in a moving image converter, and image quality. It is a figure explaining the setting of the virtual moving speed in the thinning process of the sample point movement type performed in the block process part in a moving image converter. It is a figure explaining the setting of the virtual moving speed in the thinning process of the sample point movement type performed in the block process part in a moving image converter. It is a figure explaining the response | compatibility of the moving speed performed in the block process part in a moving image converter, and the aspect of a thinning process. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the response | compatibility of the moving speed performed in the block process part in a moving image converter, and the aspect of a thinning process. FIG. 3 is a diagram illustrating a configuration example of a block processing unit having a memory, which is a configuration of a block processing unit in a moving image conversion apparatus. It is a figure explaining the process of a thinning-out process mode determination part. It is an example of a method for setting an allowable speed change amount, and is a diagram for explaining a method for setting a new reference evaluation value other than the reference evaluation value and setting an allowable speed change amount. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure which shows the correlation with a subject moving amount explaining the effect by the thinning process of the sample point movement type performed in the block processing part in a moving image converter, and image quality. It is a figure explaining the setting of the virtual moving speed in the thinning process of the sample point movement type performed in the block process part in a moving image converter. It is a figure explaining the setting of the virtual moving speed in the thinning process of the sample point movement type performed in the block process part in a moving image converter. It is a figure explaining the response | compatibility of the moving speed performed in the block process part in a moving image converter, and the aspect of a thinning process. It is a flowchart explaining the process sequence of a moving image converter. It is a figure explaining the structure of a moving image decompression | restoration apparatus. It is a figure explaining the process of the block expansion part of a moving image decompression | restoration apparatus. It is a figure explaining the process of the block expansion part of a moving image decompression | restoration apparatus. It is a figure explaining the process of the block expansion part of a moving image decompression | restoration apparatus. It is a figure explaining the process of the block expansion part of a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process and correction | amendment process which are performed with a moving image decompression | restoration apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image converter 11 Block division part 12 Movement amount detection part 13 Block processing part 14 Output part 21 Image storage part 22 Block division part 31 Movement amount detection part 32 Block distribution part 51-53 Block processing part 100 Image conversion apparatus 110 Block division Unit 120 movement amount detection unit 130 block processing unit 140 output unit 111 image storage unit 112 block division unit 121 movement amount detection unit 122 block distribution unit 131 to 133 block processing unit 151 decimation processing mode determination unit 152 decimation processing execution unit 153 memory 201 Subject area 205 Image quality evaluation curve 215 Image quality evaluation curve 300 Image restoration device 310 Distribution unit 321-323 Block expansion unit 330 Composition unit 401 block

Claims (25)

A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit for detecting a subject movement amount in each block divided by the block division unit;
A block processing unit that inputs the block data divided by the block dividing unit and the movement amount information detected by the movement amount detection unit, and executes block data thinning processing;
The block processing unit
Based on the movement amount information, a decimation mode determination unit that determines a decimation mode of whether to perform sampling point fixed type spatial direction decimation or sample point movement type decimation in the spatial direction;
In accordance with the determination of the thinning mode determination unit, a thinning process execution unit that performs either a sample point fixed type spatial direction thinning process or a sample point movement type spatial direction thinning process;
A moving image conversion apparatus characterized by comprising: The thinning mode determination unit further includes:
As a processing mode in sampling direction moving type spatial direction decimation, it is a configuration for determining the sampling point moving direction according to the progress of the frame,
The thinning process execution unit
The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to execute a sample point movement type spatial direction thinning process with a sample point movement process in a direction determined by the thinning mode determination unit. The thinning mode determination unit
It is a configuration for performing a thinning mode determination process according to the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the thinning process for fixing the sample points is executed,
If the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit is greater than or equal to a predetermined reference evaluation value, the execution of the sampling point fixed thinning process is determined,
When the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected by the movement amount detection unit is less than a predetermined reference evaluation value, execution of the sampling point movement type thinning process is determined. The moving image conversion apparatus according to claim 1. The thinning mode determination unit further includes:
A memory for holding a thinning mode determination result for a past frame by the thinning mode determination unit;
This is a configuration for performing a thinning mode determination process according to data corresponding to a subject moving speed and an image quality evaluation value generated based on processing data when performing thinning processing with fixed sample points, and a thinning mode determination result for a past frame. ,
A reference evaluation in which an image quality evaluation value corresponding to a subject moving speed corresponding to the moving amount information detected by the moving amount detection unit is determined in advance as a result of determining the thinning mode for the past frame is a sampling point fixed thinning process. When the value is equal to or greater than T1, the execution of the thinning process of the fixed sampling point type is determined,
A reference evaluation in which an image quality evaluation value corresponding to a subject moving speed corresponding to the moving amount information detected by the moving amount detection unit is determined in advance as a result of determining the thinning mode for the past frame is a sampling point fixed thinning process. If the value is less than T1, the execution of the sampling point movement type thinning process is determined,
A reference evaluation in which an image quality evaluation value corresponding to a subject moving speed corresponding to the moving amount information detected by the moving amount detection unit is determined in advance is a sampling point moving type thinning process for the past frame. If the value is less than T2, the execution of the sampling point moving type thinning process is determined,
A reference evaluation in which an image quality evaluation value corresponding to a subject moving speed corresponding to the moving amount information detected by the moving amount detection unit is determined in advance is a sampling point moving type thinning process for the past frame. 2. The moving image conversion apparatus according to claim 1, wherein when the value is equal to or greater than T2, the execution of the sampling point fixed thinning process is determined. The block processing unit
In the region of the moving speed at which the image quality evaluation value is equal to or higher than a predetermined reference evaluation value in the correspondence data of the object moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed,
2. The configuration according to claim 1, wherein a space thinning process is performed in which a movement mode of the sample points is set so that a virtual subject moving speed generated by the thinning process of the sample points is set. Video conversion device. The block processing unit further includes:
A memory for holding a movement mode of a sample point at the time of spatial thinning processing for a past frame by the block processing unit;
In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed, the virtual corresponding to the movement mode of the sampling point at the time of spatial thinning for the past frame When the image quality evaluation value corresponding to a specific moving speed of the subject is equal to or greater than a predetermined reference evaluation value T3, a spatial thinning process is performed in which the movement mode of the sample points at the time of spatial thinning for the past frame is set. ,
In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed, the virtual corresponding to the movement mode of the sampling point at the time of spatial thinning for the past frame If the image quality evaluation value corresponding to the specific object moving speed is lower than the predetermined reference evaluation value T3, the object moving speed and image quality evaluation generated based on the processing data when the sampling point fixing thinning process is executed The movement of the sample point is set such that the moving speed of the virtual subject generated by the sampling point moving type thinning process is set in a moving speed region in which the image quality evaluation value is equal to or greater than a predetermined reference evaluation value T4 in the corresponding data of the values. The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to execute a spatial thinning process in which a mode is set. The block processing unit
Based on the movement amount information,
(A) Only spatial direction thinning processing, or
(B) Spatial direction decimation process and time direction decimation process, or
(C) Only time direction decimation processing,
The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to perform a thinning process in any of the above aspects (a) to (c). The movement amount detector
Executes a process for detecting a motion vector based on a corresponding block of a different frame,
The block processing unit
The moving image conversion apparatus according to claim 1, wherein a thinning mode is determined based on the motion vector, and a thinning process is performed according to the determined mode. A moving image restoration apparatus that performs a restoration process of moving image conversion data,
A block extension unit that inputs block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executes an extension process of the block correspondence conversion data based on the conversion mode information;
A synthesis unit that synthesizes the blocks restored by the block extension process in the block extension unit and generates frame data;
The block extension is
Block expansion processing corresponding to at least one of the sampling point fixed type spatial direction thinning processing, sample point movement type spatial direction thinning processing, and time direction thinning processing executed in the generation of the moving image conversion data To restore the block,
The synthesis unit is
A moving image restoration apparatus characterized in that it is configured to generate a frame data by synthesizing blocks restored by a block expansion process. The synthesis unit is
When the block restored by the block extension unit is a block that has been subjected to a sampling point movement type thinning process in the spatial direction, it is configured to execute a process of shifting the block position according to the progress of the frame. The moving image restoration apparatus according to claim 9. The synthesis unit is
When the block restored in the block extension unit is a block that has been subjected to the sampling point movement type thinning process in the spatial direction, a process of shifting the block position according to the progress of the frame is executed and generated by the block arrangement The moving image restoration apparatus according to claim 9, wherein a pixel value correction process for determining a pixel value of a pixel gap or an overlapping pixel is executed. A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data generated in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Based on the movement amount information, a thinning mode determination step for determining a thinning mode for performing sampling point fixed type spatial direction thinning or sampling point movement type spatial direction thinning;
In accordance with the determination information in the thinning mode determination step, a thinning process execution step for executing either a sample point fixed type spatial direction thinning process or a sample point movement type spatial direction thinning process;
A moving image conversion method characterized by comprising: The thinning mode determination step includes:
As a processing mode in the sampling direction moving-type spatial direction thinning, the step of determining the sampling point movement direction according to the progress of the frame,
The thinning process execution step includes:
13. The moving image conversion method according to claim 12, wherein a sampling point movement type spatial direction thinning process accompanied by a sampling point movement process in the direction determined in the thinning mode determination step is executed. The thinning mode determination step includes:
Execute the thinning mode determination process according to the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed,
If the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected in the movement amount detection step is equal to or higher than a predetermined reference evaluation value, the execution of the sampling point fixed thinning process is determined,
If the image quality evaluation value corresponding to the subject movement speed corresponding to the movement amount information detected in the movement amount detection step is less than a predetermined reference evaluation value, execution of the sampling point movement type thinning process is determined. The moving image conversion method according to claim 12, wherein The thinning mode determination step further includes:
Holding a decimation mode determination result for a past frame in the decimation mode determination step;
Execute the thinning mode determination processing according to the subject movement speed and the image quality evaluation value correspondence data generated based on the processing data when the sampling point fixed thinning processing is executed, and the thinning mode determination result for the past frame,
A reference evaluation in which an image quality evaluation value corresponding to a subject moving speed corresponding to the moving amount information detected by the moving amount detecting step is a reference evaluation in which the thinning mode determination result for the past frame is a sampling point fixed thinning process. When the value is equal to or greater than T1, the execution of the thinning process of the fixed sampling point type is determined,
A reference evaluation in which an image quality evaluation value corresponding to a subject moving speed corresponding to the moving amount information detected by the moving amount detecting step is a reference evaluation in which the thinning mode determination result for the past frame is a sampling point fixed thinning process. If the value is less than T1, the execution of the sampling point movement type thinning process is determined,
A reference evaluation in which an image quality evaluation value corresponding to the subject moving speed corresponding to the moving amount information detected by the moving amount detecting step is a reference evaluation in which the thinning mode determination result for the past frame is a sampling point moving thinning process. If the value is less than T2, the execution of the sampling point moving type thinning process is determined,
A reference evaluation in which an image quality evaluation value corresponding to the subject moving speed corresponding to the moving amount information detected by the moving amount detecting step is a reference evaluation in which the thinning mode determination result for the past frame is a sampling point moving thinning process. 13. The moving image conversion method according to claim 12, wherein if the value is equal to or greater than T2, execution of thinning processing of a fixed sample point type is determined. The block processing step includes:
In the region of the moving speed at which the image quality evaluation value is equal to or higher than a predetermined reference evaluation value in the correspondence data of the object moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed,
13. The moving image conversion according to claim 12, wherein spatial thinning processing is performed in which a movement mode of the sample points is set so that a virtual subject moving speed generated by the thinning processing of the sample points is set. Method. The block processing step further includes:
The movement mode of the sample point during the spatial thinning process for the past frame by the block processing step is retained,
In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed, the virtual corresponding to the movement mode of the sampling point at the time of spatial thinning for the past frame When the image quality evaluation value corresponding to a specific moving speed of the subject is equal to or greater than a predetermined reference evaluation value T3, a spatial thinning process is performed in which the movement mode of the sample points at the time of spatial thinning for the past frame is set. ,
In the correspondence data of the subject moving speed and the image quality evaluation value generated based on the processing data when the sampling point fixing thinning process is executed, the virtual corresponding to the movement mode of the sampling point at the time of spatial thinning for the past frame If the image quality evaluation value corresponding to the specific object moving speed is lower than the predetermined reference evaluation value T3, the object moving speed and image quality evaluation generated based on the processing data when the sampling point fixing thinning process is executed The movement of the sample point is set such that the moving speed of the virtual subject generated by the sampling point moving type thinning process is set in a moving speed region in which the image quality evaluation value is equal to or greater than a predetermined reference evaluation value T4 in the corresponding data of the values. The moving image conversion method according to claim 12, wherein spatial thinning processing in which a mode is set is executed. The block processing step includes:
Based on the movement amount information,
(A) Only spatial direction thinning processing, or
(B) Spatial direction decimation process and time direction decimation process, or
(C) Only time direction decimation processing,
The moving image conversion method according to claim 12, wherein the thinning-out process is performed in any one of the modes (a) to (c). The movement amount detection step includes:
Executes a process for detecting a motion vector based on a corresponding block of a different frame,
The block processing step includes:
13. The moving image conversion method according to claim 12, wherein a thinning mode is determined based on the motion vector, and a thinning process according to the determined mode is executed. It is a moving image restoration method for executing moving image conversion data restoration processing,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block expansion processing corresponding to at least one of the sampling point fixed type spatial direction thinning processing, sample point movement type spatial direction thinning processing, and time direction thinning processing executed in the generation of the moving image conversion data To restore the block,
The synthesis step includes
A moving image restoration method, comprising the step of synthesizing blocks restored by block expansion processing to generate frame data. The synthesis step includes
When the block restored in the block expansion step is a block that has undergone a sampling point movement type thinning process in the spatial direction, a process of shifting the block positions according to the progress of the frame is executed. 20. The moving image restoration method according to 20. The synthesis step includes
When the block restored in the block expansion step is a block that has been subjected to the sampling point movement type thinning process in the spatial direction, the block position is shifted according to the progress of the frame, and the block arrangement is performed. 21. The moving image restoration method according to claim 20, further comprising a step of executing a pixel value correction process for determining a pixel value of a pixel gap or an overlapping pixel. A computer program for executing data conversion processing of moving image data;
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data generated in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Based on the movement amount information, a thinning mode determination step for determining a thinning mode for performing sampling point fixed type spatial direction thinning or sampling point movement type spatial direction thinning;
In accordance with the determination information in the thinning mode determination step, a thinning process execution step for executing either a sample point fixed type spatial direction thinning process or a sample point movement type spatial direction thinning process;
A computer program characterized by comprising: A computer program for executing data conversion processing of moving image data;
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data generated in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Based on the movement amount information and the past thinning mode, a thinning mode determination step for determining a thinning mode to execute the sampling point fixed type spatial direction thinning or the sampling point movement type spatial direction thinning;
In accordance with the decision information and the past decision information in the thinning mode decision step, a thinning process execution step for executing either the sample point fixed type spatial direction thinning process or the sample point movement type spatial direction thinning process;
A computer program characterized by comprising: A computer program for executing a restoration process of moving image conversion data,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block expansion processing corresponding to at least one of the sampling point fixed type spatial direction thinning processing, sample point movement type spatial direction thinning processing, and time direction thinning processing executed in the generation of the moving image conversion data To restore the block,
The synthesis step includes
A computer program characterized in that it is a step of generating frame data by synthesizing blocks restored by block expansion processing.

Images