JP2008017061A - Moving picture converting device, moving picture restoring device, method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device and a method for converting and restoring data while preventing the image quality of moving picture data from deteriorating. <P>SOLUTION: An optimum compression processing embodiment is determined, where the embodiment corresponds to the configuration of an object included in a block for composing the moving picture data, and the movement of the block. Then, data conversion processing is performed in the optimum embodiment for each block region according to the decided embodiment. Concretely, the configuration of the object included in the block for composing the moving picture data and the movement of the block are analyzed, and no thin-out processing is set when a plurality of different moving objects are included in the block and when the displacement of the block varies sharply for a short time, or the like, thus achieving the data conversion and restoration processing, where deterioration in the image quality is suppressed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program. In particular, the present invention relates to a moving image conversion apparatus, a moving image restoration apparatus, a method, and a computer program that can suppress high-quality data conversion in data conversion executed as data compression processing of moving image data.

  Furthermore, more specifically, the present invention reduces image quality even when a plurality of different moving objects exist in a block as a segmented area of an image, or in a moving image in which an object whose abrupt movement speed changes occurs. The present invention relates to a moving image conversion apparatus, a moving image conversion method, and a computer program that can perform high-quality data conversion while suppressing.

  The moving image data is subjected to data conversion for reducing the amount of data, that is, compression processing, when it is stored in a storage medium such as a hard disk or DVD or distributed via a network. In particular, in recent years, the quality of moving image data has been improved, for example, HD (High Definition) data has been improved, and the amount of data has increased rapidly with the improvement of the quality of this data. Under such circumstances, many studies and studies have been made on techniques for improving the compression efficiency in the compression and decompression processing of moving image data and preventing the quality degradation of the decompressed data.

  As a moving image compression processing method, for example, thinning processing of pixels constituting an image frame constituting moving image data, that is, thinning processing in a spatial direction, thinning processing of a frame rate, that is, thinning processing in a time direction, and the like are known. Yes.

  By reducing the amount of data by such data conversion, there is an advantage that saving to a storage medium and data transfer via a network are efficiently performed. However, when the compressed data is restored and reproduced, there is a problem that the image quality deteriorates. In particular, when the original data is a high-definition image, the quality deterioration becomes more remarkable.

Various studies have been made on how to reduce such image quality deterioration. For example, Patent Document 1 discloses an image compression processing configuration that switches between a pixel count thinning process in the spatial direction and a pixel count thinning process in the time direction according to the moving speed of the subject. Further, Patent Document 2 discloses a compression processing configuration in which processing for changing the phase of a sample point in accordance with the moving speed of a subject in thinning out the number of pixels in the spatial direction is added to the processing of Patent Document 1. .
JP 2005-198268 A JP 2006-5904 A

  As described above, in some conventional technologies, a configuration is disclosed in which image quality deterioration due to compression processing due to thinning processing is suppressed and data quality is improved by performing different processing for each region of an image according to the moving speed of a subject. Has been. For example, in Patent Documents 1 and 2 described above, a frame constituting a moving image is divided into blocks of a predetermined area, and after the movement amount for each block is detected, the blocks are uniformly distributed according to the movement amount of each block. A configuration for performing decimation processing is disclosed.

  However, most of the moving image data includes a plurality of objects, and includes various motion data such as an object having a large motion, a small object, and a stationary object. Therefore, an object included in a certain block is not necessarily an object that moves at a single moving speed, and an object having a plurality of different movement amounts may be included in one block. For such a block, there is a case where high-quality compressed data cannot be obtained when the above-described conventional technique is applied.

  The present invention has been made in view of such a situation, and generates control information for thinning processing indicating whether it is appropriate to perform thinning on a block by executing analysis of each block of the image. In addition, an optimal data conversion process is performed for each block, and if the thinning process is not appropriate in accordance with the control information, a moving image conversion that enables compression with extremely low quality deterioration by adopting a configuration in which the thinning process is not performed. It is an object to provide an apparatus, a moving image restoration apparatus and method, and a computer program.

The first aspect of the present invention is:
A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit that detects a movement amount of a subject included in each block divided by the block division unit as a block movement amount;
Based on the block movement amount detected by the movement amount detection unit, a thinning processing mode information generation unit that generates thinning processing mode information indicating a thinning processing mode to be applied to the block;
Based on the block movement amount detected by the movement amount detection unit, a thinning process control information generation unit that generates thinning process control information indicating whether or not to perform a thinning process on a block;
Each block divided by the block dividing unit, the thinning processing mode information generated by the thinning processing mode information generation unit, and the thinning processing control information generated by the thinning processing control information generation unit are input to the input information. A thinning process execution unit that determines whether or not to perform a thinning process for each input block and a thinning process mode, and performs a process according to the determination;
The moving image conversion apparatus is characterized by comprising:

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning-out process execution unit indicates that the thinning-out process control information generated by the thinning-out process control information generation unit should execute a thinning process on a block. The thinning-out processing mode information generated by the thinning-out processing mode information generation unit executes the thinning-out processing according to the thinning-out processing mode information generated by the thinning-out processing control information generation unit. If it is not to be executed, the thinning process is not executed, and the original block data is output.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning-out process execution unit indicates that the thinning-out process control information generated by the thinning-out process control information generation unit should execute a thinning process on a block. The thinning-out processing mode information generated by the thinning-out processing mode information generation unit executes the thinning-out processing according to the thinning-out processing mode information generated by the thinning-out processing control information generation unit. Even in the case where it should not be executed, the thinning process according to the thinning processing mode information generated by the thinning processing mode information generation unit is executed, and the thinning process result and the original block which does not execute the thinning process It is the structure which outputs data.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning processing mode information generation unit is configured to use a spatial direction as a thinning processing mode to be applied to the block based on the block movement amount detected by the movement amount detection unit. It is characterized in that at least one of thinning-out processing, thinning-out processing in the time direction, or complex thinning-out processing is determined.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning processing mode information generation unit is configured to use a spatial direction as a thinning processing mode to be applied to the block based on the block movement amount detected by the movement amount detection unit. In the case of determining to perform the thinning process, it is further characterized in that a process for determining a sample point phase change amount indicating a selected position of the sample point in the spatial direction thinning process is executed.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning-out process control information generating unit determines whether or not to perform a thinning-out process on the block based on the block movement amount detected by the movement amount detection unit. In the generation of the thinning-out process control information to be shown, the thinning-out process control information indicating whether or not the thinning-out process should be executed for all the blocks corresponding to the specific frame, regardless of the block movement amount, can be thinned out or cannot be thinned out. The present invention is characterized in that a process for generating uniform control information is performed.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning-out process control information generation unit verifies whether or not a plurality of objects having different moving speeds are included in one block, and the plurality of objects are It is a configuration that generates control information that disables the thinning-out processing for the included blocks.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning-out process control information generation unit includes a selected block of a current frame including a block selected as a control information generation target, and a reference frame different from the current frame. The variance of the difference between the selected block and the reference block determined as the destination of the selection block is calculated, and the decimation processing control information is generated based on the comparison result between the calculated difference variance and a predetermined threshold value It is the structure.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning-out process control information generation unit includes a movement amount of a block at a corresponding position in the current frame including the block selected as a control information generation target and the reference frame. The thinning process control information is generated on the basis of the difference.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the thinning-out process control information generation unit may be configured to detect a difference in movement amount of blocks around the block in a current frame including a block selected as a control information generation target. Based on this, the thinning process control information is generated.

Furthermore, the second aspect of the present invention provides
A moving image restoration apparatus that performs a restoration process of moving image conversion data,
Based on the input information, block correspondence data constituting moving image conversion data, decimation processing mode information indicating a decimation processing mode for the block, and decimation processing control information indicating whether or not to perform decimation processing for the block are input. A block extension unit that executes an extension process of block-compatible conversion data; and
A synthesis unit that synthesizes the blocks restored by the block extension process in the block extension unit and generates frame data;
The block extension is
Based on the thinning-out process control information, it is determined whether or not to perform the thinning-out process data expansion process for the block, and the thinning-out processing mode indicated by the thinning-out processing mode information for the block corresponding conversion data determined to be expanded. The moving image restoration apparatus is configured to execute an expansion process corresponding to the above.

  Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the thinning process mode information is either a thinning process in a spatial direction, a thinning process in a time direction, or a thinning process of at least one of a complex thinning process. It is information indicating a mode, and the block extension unit performs either a thinning process in a spatial direction indicated by the thinning process mode information, a thinning process in a time direction, or a thinning process in at least one of a complex thinning process. The configuration is such that the corresponding extension processing is executed.

  Furthermore, in an embodiment of the moving image restoration apparatus of the present invention, the block extension unit determines whether or not to perform the extension process of the thinning process data for the block based on the thinning process control information, and performs the extension process. A block that is determined to be non-existent is configured to execute a process of outputting block-corresponding original data included as block-corresponding data to the combining unit.

  Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the synthesizing unit selects a sample point selection position in the spatial direction decimation process obtained from the decimation process mode information for the block restored in the block extension unit. Based on the sample point phase change amount shown, the block position is shifted and arranged according to the progress of the frame.

  Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the synthesizing unit selects a sample point selection position in the spatial direction decimation process obtained from the decimation process mode information for the block restored in the block extension unit. A configuration in which a block position is shifted and arranged according to the progress of the frame based on the sample point phase change amount shown, and a pixel value correction process for determining a pixel gap or an overlapped pixel value generated by the block arrangement is executed. It is characterized by being.

Furthermore, the third aspect of the present invention provides
A moving image conversion method for executing data conversion processing of moving image data in a moving image conversion apparatus,
In the block dividing unit, a block dividing step for executing a block dividing process for each frame constituting the moving image data;
In a movement amount detection unit, a movement amount detection step of detecting a movement amount of a subject included in each block divided in the block division step as a block movement amount;
In the thinning processing mode information generation unit, based on the block movement amount detected in the movement amount detection step, a thinning processing mode information generation step that generates thinning processing mode information indicating a thinning processing mode to be applied to the block;
In the thinning process control information generating unit, a thinning process control information generating step for generating thinning process control information indicating whether or not to perform the thinning process on the block based on the block moving amount detected in the moving amount detecting step; ,
In the thinning processing execution unit, each block divided in the block dividing step, the thinning processing mode information generated in the thinning processing mode information generation step, and the thinning processing control information generated in the thinning processing control information generation step Input, based on the input information, determine whether or not to perform a thinning process for each input block and a thinning process mode, and perform a thinning process execution step for performing processing according to the determination;
A moving image conversion method characterized by comprising:

Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning process execution step indicates that the thinning process control information generated in the thinning process control information generation step should execute a thinning process on a block. In the case of showing, execute the thinning process according to the thinning processing mode information generated by the thinning processing mode information generation unit,
When the thinning process control information generated in the thinning process control information generation step indicates that the thinning process should not be performed on the block, the original block data is output without performing the thinning process. And

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning process execution step indicates that the thinning process control information generated in the thinning process control information generation step should execute a thinning process on a block. The thinning processing mode information generated by the thinning processing mode information generation unit executes the thinning processing according to the thinning processing mode information generated by the thinning processing mode information generation unit. Even in the case where it should not be executed, the thinning process according to the thinning processing mode information generated by the thinning processing mode information generation unit is executed, and the thinning process result and the original block which does not execute the thinning process And a step of outputting data.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning processing mode information generation step includes a spatial direction as a thinning processing mode to be performed on the block based on the block movement amount detected in the movement amount detection step. It is characterized by determining a thinning processing mode of at least one of thinning processing, thinning processing in the time direction, or complex thinning processing.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning processing mode information generation step includes a spatial direction as a thinning processing mode to be performed on the block based on the block movement amount detected in the movement amount detection step. When it is determined to perform the thinning process, a process for determining a sample point phase change amount indicating a selected position of the sample point in the spatial direction thinning process is further performed.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning process control information generation step determines whether or not to perform a thinning process on the block based on the block movement amount detected in the movement amount detection step. In the generation of the thinning-out process control information to be shown, the thinning-out process control information indicating whether or not the thinning-out process should be executed for all the blocks corresponding to the specific frame, regardless of the block movement amount, can be thinned out or cannot be thinned out. A process for generating uniform control information is performed.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning process control information generation step verifies whether or not a plurality of objects having different moving speeds are included in one block, and the plurality of objects are It is a step of generating control information that disables thinning processing for included blocks.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning-out process control information generation step includes a selection block of a current frame including a block selected as a control information generation target, and a reference frame different from the current frame. The variance of the difference between the selected block and the reference block determined as the destination of the selection block is calculated, and the decimation processing control information is generated based on the comparison result between the calculated difference variance and a predetermined threshold value It is a step.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning-out process control information generation step includes the movement amount of the block at the corresponding position in the current frame including the block selected as the control information generation target and the reference frame. This is a step of generating the thinning process control information based on the difference.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the thinning-out process control information generation step may be performed based on a difference in movement amount of blocks surrounding the block in a current frame including a block selected as a control information generation target. Based on this, the thinning process control information is generated.

Furthermore, the fourth aspect of the present invention provides
In the moving image restoration apparatus, a moving image restoration method for executing restoration processing of moving image conversion data,
The block extension unit inputs block correspondence data constituting moving image conversion data, decimation processing mode information indicating a decimation processing mode for the block, and decimation processing control information indicating whether or not to perform decimation processing for the block. A block expansion step for performing an expansion process of the block-compatible conversion data based on the input information;
A combining unit that combines the blocks restored by the block extension process in the block extension unit to generate frame data;
The block expansion step includes:
Based on the thinning-out process control information, it is determined whether or not to perform the thinning-out process data expansion process for the block, and the thinning-out processing mode indicated by the thinning-out processing mode information for the block corresponding conversion data determined to be expanded. The moving image restoration method is characterized in that it is a step of executing an expansion process corresponding to.

  Furthermore, in one embodiment of the moving image restoration method of the present invention, the thinning process mode information is either a thinning process in a spatial direction, a thinning process in a time direction, or a thinning process of at least one of a complex thinning process. Information indicating an aspect, and the block expansion step includes either a thinning process in a spatial direction or a thinning process in a time direction indicated by the thinning process aspect information, or a thinning process of at least one of complex thinning processes. It is a step for executing a corresponding extension process.

  Furthermore, in an embodiment of the moving image restoration method of the present invention, the block expansion step determines whether or not to perform the thinning process data expansion process for the block based on the thinning process control information, and performs the expansion process. For a block determined not to exist, this is a step of executing a process of outputting block corresponding original data included as block corresponding data to the synthesis unit.

  Furthermore, in one embodiment of the moving image restoration method of the present invention, the synthesis step selects the sample point selection position in the spatial direction thinning processing obtained from the thinning processing mode information for the block restored in the block extension unit. It is a step of executing a process of shifting the block position according to the progress of the frame based on the sample point phase change amount shown.

  Furthermore, in one embodiment of the moving image restoration method of the present invention, the synthesis step selects the sample point selection position in the spatial direction thinning processing obtained from the thinning processing mode information for the block restored in the block extension unit. A step of executing a process of shifting the block position according to the progress of the frame and performing a pixel value correction process for determining a pixel value of a pixel gap or an overlapping pixel generated by the block arrangement based on the indicated sample point phase change amount It is characterized by being.

Furthermore, the fifth aspect of the present invention provides
In the moving image conversion apparatus, a computer program for executing data conversion processing of moving image data,
In the block dividing unit, a block dividing step for executing a block dividing process for each frame constituting the moving image data;
In a movement amount detection unit, a movement amount detection step for detecting a movement amount of a subject included in each block divided in the block division step as a block movement amount;
In the thinning processing mode information generation unit, a thinning processing mode information generation step for generating thinning processing mode information indicating a thinning processing mode to be applied to the block based on the block movement amount detected in the movement amount detection step;
A decimation process control information generation unit for generating decimation process control information indicating whether or not to perform a decimation process on a block based on the block movement amount detected in the movement amount detection step; ,
In the thinning processing execution unit, each block divided in the block dividing step, the thinning processing mode information generated in the thinning processing mode information generation step, and the thinning processing control information generated in the thinning processing control information generation step A decimation process execution step for determining whether to perform decimation processing for each input block and a decimation process mode based on the input information, and performing a process according to the determination;
In a computer program characterized by causing

Furthermore, the sixth aspect of the present invention provides
In the moving image restoration device, a computer program for executing restoration processing of moving image conversion data,
The block extension unit inputs block correspondence data constituting moving image conversion data, decimation processing mode information indicating a decimation processing mode for the block, and decimation processing control information indicating whether or not to perform decimation processing for the block. , A block expansion step for executing the expansion processing of the block corresponding conversion data based on the input information,
A combining unit that combines the blocks restored by the block extension process in the block extension unit to generate frame data;
The block expansion step includes:
Based on the thinning-out process control information, it is determined whether or not to perform the thinning-out process data expansion process for the block, and the thinning-out processing mode indicated by the thinning-out processing mode information for the block corresponding conversion data determined to be expanded. The present invention resides in a computer program characterized in that it is a step of executing an expansion process corresponding to the above.

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the embodiment of the present invention, the optimal compression processing mode corresponding to the configuration of the object included in the block constituting the moving image data and the motion of the block is determined, and the block area is determined according to the determined mode. Since the data conversion process is performed in an optimal manner, compression and decompression with very little quality deterioration can be realized.

  In addition, according to the configuration of the embodiment of the present invention, the configuration of the object included in the block constituting the moving image data and the motion of the block are analyzed. For example, when the block includes a plurality of different moving objects, thinning is performed. Since the setting is made so that the processing is not performed, the data conversion / restoration processing in which the deterioration of the image quality is suppressed is realized. Also, when the block movement amount fluctuates greatly in a short period of time, since the setting is made so that the thinning process is not performed, the data conversion and restoration process with suppressed image quality deterioration is realized.

  Furthermore, according to the configuration of the embodiment of the present invention, verification of the variance of the difference between the selected block of the current frame including the selected block as the verification target and the reference block determined as the movement destination of the selected block in the reference frame Or, based on the verification of the difference in the movement amount of the block at the corresponding position in the current frame including the selected block and the reference frame, or the verification of the difference in the movement amount of the surrounding blocks in the current frame including the selection block, etc. Since it is configured to perform the thinning process according to the determination by determining whether or not the process is executed, the data conversion and restoration process that suppresses the deterioration of the image quality is realized.

Hereinafter, the configuration of a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program according to the present invention will be described with reference to the drawings. The description will be made according to the following items.
(1) Basic configuration of moving image conversion device using super-resolution effect (2) Configuration of moving image conversion device for executing improved thinning process (3) Configuration of moving image restoration device (4) Improved thinning processing Configuration of moving image conversion device to be executed and modified example of moving image restoration device

[(1) Basic configuration of moving image conversion apparatus using super-resolution effect]
First, a basic configuration of a moving image conversion apparatus that performs moving image compression using the super-resolution effect as a base of the present invention will be described. This basic configuration is described in detail in Japanese Patent Application Laid-Open No. 2006-5904 filed earlier by the present applicant. The image is divided into small areas, and the number of pixels is determined according to the moving speed of each area. This is a configuration in which compression of the data amount is realized by adaptively performing decimation and frame rate decimation.

  FIG. 1 shows a configuration example of a moving image conversion apparatus 10 described in JP-A-2006-5904. The moving image conversion apparatus 10 can reduce the amount of data so that an observer does not perceive image quality deterioration due to the reduction of the data amount by performing a moving image conversion process using the super-resolution effect. It is a thing.

  Note that the super-resolution effect is a visual effect realized based on a visual characteristic that an observer perceives an image obtained by adding a plurality of images within a certain period of time. When human vision perceives a stimulus, it has a function of storing the stimulus for a certain period of time after the presentation of the stimulus ends (referred to as a sensory memory function). There are many reports that this time is 10 ms to 200 ms. This function is also called iconic memory or visual persistence, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp.229-230”. The super-resolution effect is considered to be caused by a complicated relationship between the temporal integration function and the sensory memory function in the human visual function.

  FIG. 1 shows a configuration example of a moving image conversion apparatus 100 described in Japanese Patent Application Laid-Open No. 2006-5904. The moving image conversion apparatus 100 has a configuration capable of reducing the amount of data so that the observer does not perceive image quality degradation due to the reduction in the amount of data by performing moving image conversion processing using the super-resolution effect. It is a thing.

  Note that the super-resolution effect is a visual effect realized based on a visual characteristic that an observer perceives an image obtained by adding a plurality of images within a certain period of time. Human vision has a function of perceiving light when the sum of received light stimuli reaches a certain threshold (hereinafter referred to as a temporal integration function). That is, the perception of light follows the sum of the light integrated over time regardless of the distribution state of the light stimulus within the presentation time. Further, the stimulus (threshold) that can perceive light becomes smaller as the presentation time of the stimulus becomes longer, and becomes larger as the presentation time becomes shorter. Details of the temporal integration function are described in, for example, “Visual Information Handbook, Japanese Visual Society, pp.229-230”.

  The moving image conversion apparatus 100 shown in FIG. 1 performs compression that reduces data so that an observer does not perceive image quality deterioration by performing a moving image conversion process using a super-resolution effect caused by a temporal integration function. The configuration is to be performed.

  A configuration of the moving image conversion apparatus 100 in FIG. 1 will be described. The block dividing unit 110 decomposes each frame of the input moving image into blocks and supplies the blocks to the movement amount detection unit 120. The movement amount detection unit 120 detects the movement amount for each block supplied from the block division unit 110, and transmits the block and the movement amount to the block processing unit 130. The block processing unit 130 performs a moving image conversion process corresponding to the movement amount on the block supplied from the movement amount detection unit 120 to reduce the data amount. The block processing unit 130 supplies the output unit 140 with data regarding the block whose data amount is reduced, which is obtained as a result of the processing. The output unit 140 collectively outputs the data regarding the blocks with the reduced data amount supplied from the block processing unit 130, for example, as stream data.

  Next, details of each unit will be described with reference to FIG. First, the block dividing unit 110 will be described. The frame of the moving image supplied to the moving image conversion apparatus 100 is input to the image storage unit 111 of the block dividing unit 110. The image storage unit 111 stores the input frame. Each time the number of accumulated frames reaches N (N is a positive integer), the image accumulation unit 111 supplies the N frames to the block division unit 112 and M in the N frames. The second stored frame (hereinafter referred to as the Mth frame) is supplied to the movement amount detection unit 121 of the movement amount detection unit 120. For example, N = 4.

  The block dividing unit 112 divides each of the N frames (consecutive N frames) supplied from the image storage unit 111 into blocks of a certain size (for example, 8 × 8, 16 × 16) and moves them. The data is output to the block distribution unit 122 of the amount detection unit 120. The block division unit 112 also detects the movement amount of the P-th frame stored in the image storage unit 111 (hereinafter referred to as the P-th frame) among the N frames by the movement amount detection unit 120. To the unit 121. The Pth frame is a different frame from the Mth frame.

  Next, the movement amount detection unit 120 will be described. The movement amount detection unit 121 of the movement amount detection unit 120 uses the motion vector of each block of the Pth frame supplied from the block division unit 112 of the block division unit 110 as the Mth frame supplied from the image storage unit 111. For example, the block matching process between frames is executed and detected, and the detected motion vector is supplied to the block distributor 122 and the block processors 131 and 132. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and in the vertical direction (Y-axis direction). For example, when the reference frame: M = 2 and the supply frame: P = 3, the motion vector represents the amount of movement in the horizontal direction (X-axis direction) and vertical direction (Y-axis direction) between one frame. Become. The motion vector may be detected not only by block matching but also by another method.

  The block distribution unit 122 of the movement amount detection unit 120 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 112 in units of N, and the movement amount detection unit From 121, the movement amount of the block of the P-th frame among the N blocks is supplied. The block distribution unit 122 performs block processing units 131 to 133 that perform processing corresponding to the movement amount of the block processing unit 130 for the supplied N blocks. The block processing unit 130).

  Specifically, the block distribution unit 122 is supplied from the movement amount detection unit 121 and has a larger movement amount of 2 pixels between the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction) between one frame. In the case above, the N blocks supplied from the block dividing unit 112 are output to the block processing unit 131. When the movement amount between the horizontal direction and the vertical direction between one frame is less than 2 pixels and 1 pixel or more, the block distribution unit 122 outputs N blocks to the block processing unit 132. To do. When the movement amount is other than that, the block distribution unit 122 supplies N blocks to the block processing unit 133.

That is,
(A) Movement amount ≧ 2 pixels / frame: block processing unit 131 (spatial direction thinning process)
(B) 2 pixels / frame> movement amount ≧ 1 pixel / frame: block processing unit 132 (time / space direction thinning process)
(C) 1 pixel / frame> movement amount: block processing unit 133 (time direction thinning process)
The processing block is output to the above (a) to (c).

  When the moving amount ≧ 2 pixels, the block processing unit 131 executes the spatial direction thinning process. When 2 pixels> the moving amount ≧ 1 pixel, the block processing unit 132 executes the time / spatial direction thinning process. When pixel> movement amount, the block processing unit 133 performs the time direction thinning process.

  As described above, the block distribution unit 122 determines the optimal frame rate and spatial resolution for conversion based on the movement amount supplied from the movement amount detection unit 121, and converts the image data according to the frame rate and spatial resolution. Block processing units 131 to 133 that perform processing A process of distributing block images is executed. This condition for determining the distribution destination is merely an example, and the distribution destination may be determined under other conditions.

Next, the block processing unit 130 will be described. In this example, the block processing unit 130 includes three block processing units 131 to 133.
The block processing unit 131 is supplied from the block distribution unit 122 of the movement amount detection unit 120 and has a total of N blocks (the movement amount in the horizontal direction or the vertical direction is 2 at the same position in each of consecutive N frames). Processing for thinning out the number of pixels in accordance with the larger moving amount (horizontal direction or vertical direction supplied from the moving amount detecting unit 131) for the N blocks in the case of pixels / frame or more (spatial direction) (Thinning process). That is, the spatial direction thinning process is executed for N blocks whose movement amount ≧ 2 pixels.

  When the amount of movement in the horizontal direction between one frame is 2 pixels or more, the block processing unit 131, for example, when the block is composed of 8 × 8 pixels, as shown in FIG. Divide the pixels into sets of 1 × 4 pixel units.

  Further, the block processing unit 131 sets the pixel values p1 to p4 of each set of 1 × 4 pixels to one of the pixel values in any of the modes of FIG. 3B, FIG. 4 and FIG. Thinning out the number of pixels (thinning out the number of pixels between four pixels) (thinning out the thinning amount 4) is performed.

  In the thinning-out process shown in FIG. 3B, the pixel values at the same pixel position are represented as representative pixel values, that is, sample points (sample points in the example in the figure) in all frames of N consecutive frames (k to k + 3). = P1), the thinning process is set as the pixel values of the four pixels.

  The thinning-out process shown in FIG. 4 and FIG. 5 does not set the same pixel position as a sample point in consecutive N frames (k to k + 3), but uses pixels at different pixel positions for each frame as sample points. This is a thinning process in which pixel values at different positions for each frame are set as the pixel values of four pixels in each frame.

The thinning process shown in FIG. 4 is performed for pixel values p1 to p4 of four horizontal pixels at the same position in the 4th frame from k to k + 3.
k-th frame: sample point = p1
k + 1 frame: sample point = p2
k + 2nd frame: sample point = p3
k + 3rd frame: sample point = p4
As described above, the thinning process is performed by changing the pixel position of the sample point in the right direction one pixel at a time according to the progress of the frame.

The thinning-out process shown in FIG. 5 is performed for pixel values p1 to p4 of four horizontal pixels at the same position in the 4th frame from k to k + 3.
k-th frame: sample point = p4
k + 1 frame: sample point = p3
k + 2nd frame: sample point = p2
k + 3rd frame: sample point = p1
In this manner, the thinning process is performed by changing the pixel position of the sample point in the left direction one pixel at a time according to the progress of the frame.

  As described above, the thinning-out processing shown in FIGS. 4 and 5 performs pixel values of four pixels of each frame, not the same pixel position, but the pixel values of the four pixels of each frame in consecutive N frames (k to k + 3). Is a thinning process set as

  The block processing unit 131 selects and executes one of the three thinning processes shown in FIGS. 3 to 5 according to the movement amount information supplied from the movement amount detection unit 121 of the movement amount detection unit 120.

  A detailed configuration of the block processing unit 131 is shown in FIG. The block processing unit 131 includes a thinning processing mode determination unit 151 and a thinning processing execution unit 152. The thinning processing mode determination unit 151 inputs movement amount information from the movement amount detection unit 121. As described above, the block input to the block processing unit 131 is a block of moving amount ≧ 2 pixels / frame, and the moving amount information input to the decimation processing mode determining unit 151 of the block processing unit 131 is: The moving amount information is 2 pixels / frame or more.

  The thinning process mode determination unit 151 determines whether to perform the thinning process in any of the above-described FIGS. 3, 4, and 5 based on the value of the movement amount. Details of this determination method will be described later. The thinning process execution unit 152 executes the thinning process in any of the modes of FIGS. 3, 4, and 5 in accordance with the determination of the thinning process mode determination unit 151.

  As described above, the moving image conversion apparatus 100 shown in FIG. 1 performs processing for converting an input moving image into a moving image (compressed data) with a reduced data amount. This is a device that prevents the observer from perceiving image quality degradation due to a reduction in the amount of data by performing a moving image conversion process using a super-resolution effect realized based on the visual characteristics of the image.

  Note that the super-resolution effect is a visual effect realized based on the visual characteristic that an observer perceives a sum of a plurality of images within a certain time as described above. It is considered that the temporal integration function and the sensory memory function in the function are caused in a complicated manner, and the moving image conversion apparatus 100 shown in FIG. 1 has a super-resolution effect caused by the temporal integration function. The moving image conversion process is used.

  The principle and explanation regarding human visual characteristics and super-resolution effect are described in detail in Japanese Patent Laid-Open No. 2005-298268. The conditions for generating the super-resolution effect described in JP-A-2005-298268 will be described below.

In order for the super-resolution effect to occur when the number of pixels of the thinning amount m (pixel) is thinned, it is necessary to cancel all the primary to m−1 order folding components due to the thinning. k (= 1, 2,..., m−1) The condition that the next folding component is canceled is to satisfy the following expressions (Expression 1) and (Expression 2).

In the above equation, φt is a sampling position shift amount in thinning out the number of pixels, time t (= 0, 1T, 2T,...), Signal moving speed v, time interval (reciprocal of frame rate) T, Is a value defined by the following formula (formula 3).

  In the above formula, the case where the sampling position is shifted to the right is positive (this condition is different from the setting disclosed in JP-A-2005-198268).

  If the above equations (Equation 1) and (Equation 2) are satisfied under the conditions of the thinning amount m and the small region movement amount v, the super-resolution effect occurs, and the deterioration of the image quality is hardly perceived by the observer.

  Next, the reason for using the three types of spatial thinning processing (b1) to (b3) shown in FIGS. 3 to 5 will be described. The moving image conversion apparatus 100 described with reference to FIG. 1 to FIG. 6 sets the condition of the moving speed between one frame in a small area when thinning the number of pixels with a thinning amount of 4 as two or more pixels in the horizontal or vertical direction. Was set. Here, a configuration in which usable space thinning-out processing is only processing shown in FIG. 3 will be considered. In such a configuration, for example, when the moving speed is 4 pixels per frame, that is, when the moving amount v of the small region is v = 4, if the thinning amount m = 4, the sampling position shift of the above equation (Equation 3) Amount: The value that φt can take is only an integer of 0 or more.

  That is, in this case, the above formula (Formula 1) is not satisfied in all k cases, and the super-resolution effect does not occur. Similarly, when the thinning amount m = 2, the small region moving amount v = 1, the thinning amount m = 4, and the small region moving amount v = 2, all the above formulas (Formula 1) are also set. In the case of k, it is not satisfied, the super-resolution effect is not obtained, and the image quality is greatly deteriorated. That is, the moving image conversion apparatus 100 described with reference to FIGS. 1 to 6 uses the three types of spatial thinning processes (b1) to (b3) shown in FIGS. The super-resolution effect can be obtained regardless of the moving speed.

  The effect of performing the thinning process by changing the sample point position according to each frame as shown in FIG. 4 or 5 will be described. Hereinafter, the thinning process that is executed by changing the sample point position in accordance with each frame in this way may be referred to as “sample point moving thinning process” or “decimation position changing process”. In the [sample point moving thinning process], the thinning position changing process for shifting the sample point position to the right as the frame progresses as shown in FIG. 4 and the sample point position as shown in FIG. Since various modes other than the two modes of changing the thinning position shifted to the left according to the progress are conceivable, different thinning position changing processes are distinguished by the amount of change of the sample point coordinates for each frame. This amount of change is called a sample point phase change amount.

In the thinning position change process,
The sampling point phase change amount of [0] is the thinning process in FIG.
The sampling point phase change amount is [+1] thinning-out processing is the processing in FIG.
When the sampling point phase change amount is [−1], the processing of FIG.
It corresponds to. The sample point phase change amount is distinguished by adding a +-sign to distinguish the direction. For example, the right direction and the downward direction are [+], and the left direction and the upward direction are [−].

  In the thinning position changing process, the sample point deviation φt shown in the equation (Equation 3) described in the basic configuration of the moving image conversion apparatus is added by 1 / m for each frame (when shifted to the right (FIG. 4)). ), Or 1 / m subtraction (when shifting to the left (FIG. 5)). Note that m is a thinning amount (m pixels).

A new sample point shift amount φ′t when the thinning position changing process is performed is expressed by the following mathematical formula (Formula 4).

  From the above equation (Equation 4), shifting the position of the sample point in the traveling direction of the subject decelerates the moving speed of the subject, and shifting the position opposite to the traveling direction accelerates the moving speed and signal processing. It can be seen that the theoretical meaning of is equivalent. Of course, this does not mean that the moving speed of the subject changes on the actual video. Hereinafter, a specific processing example will be described with reference to FIG.

  FIG. 7 shows a situation where the subject (5 pixels between p1 and p5 of the kth frame) 201 has not moved (A) during 4 frames (frames k to k + 3), and (B) the velocity v = horizontal direction. It shows the situation of moving at 1 pixel / frame.

  The block processing unit 131 of the moving image conversion apparatus 100 shown in FIG. 1 and FIG. 2 originally receives a block whose horizontal movement amount between one frame is 2 pixels or more. Although frame data is not input as in (A) and (B), in data in which the horizontal movement amount between one frame is 2 pixels or more, FIG. 3, FIG. 4, FIG. Any one of the thinning processes shown in FIG. That is, there are cases where [decimation position change processing] is performed and not performed depending on the amount of movement.

  With reference to FIG. 7, a description will be given of processing when [decimation position changing processing] is performed and when it is not performed. For example, in the situation where (A) it is not moving, the thinning process of the fixed sample points shown in FIG. 3 is executed, and (B) in the situation where it is moving in the horizontal direction at a speed v = 1 pixel / frame. It is assumed that the sampling point shown in 4 is moved every frame [decimation position changing process], that is, the sampling point moving type thinning process is performed.

As shown in FIG. 7A, in the situation where the subject is not moving, if the thinning amount = 4, that is, if the pixel end thinning between four pixels is performed, the fixed sampling point thinning described with reference to FIG. When the process is executed, the output values for every four pixels are the pixels of p1 and p5 in all the frames, and are the pixel values at fixed positions. On the other hand, as shown in FIG. 7B, in the situation where the subject is moving in the horizontal direction at a speed v = 1 pixel / frame, the sample points described with reference to FIG. If the thinning position change process] is performed, the output value of each frame corresponding to each 4 pixel is as shown in FIG.
k-th frame: p1, p5
k + 1 frame: p2, p6
k + 2nd frame: p3, p7
k + 3rd frame: p4, p8
It becomes.

  As a result, the output value of each frame is identical between the moving subject of 1 pixel / frame in FIG. 7B and the non-moving subject in FIG. 7A. That is, an output obtained by performing [decimation position change processing] for moving the sample point described with reference to FIG. 4 for each frame with respect to a moving image including a subject with a moving amount v = 1 pixel / frame, and a moving speed. It can be seen that the processing of FIG. 3, that is, the output obtained by performing the thinning processing using a fixed sample point coincides with the subject (A) of v−1 = 0. That is, it can be confirmed that performing the process of FIG. 4 on the moving subject is equivalent to performing the process of FIG. 3 on the subject whose moving speed is decelerated by one.

  Eventually, the moving speed of the subject can be virtually accelerated or decelerated by the thinning position changing process.

  Next, it is confirmed at what speed the super-resolution effect does not occur when the subject is moving. FIG. 8 shows a result obtained by performing the fixed sample point thinning process (four pixel thinning process) of FIG. 3 on moving image data including a subject having a moving speed: v as various moving amounts per frame. The quality data of the image quality obtained by restoring (expanding) the converted (compressed) image again is shown. The horizontal axis represents the moving speed: v (pixel / frame) as the moving amount of the subject per frame, and the vertical axis represents the image quality evaluation value. That is, it is a graph showing the tendency of the result of the subjective evaluation experiment of the image quality when the subject is moved rightward by the movement amount shown on the horizontal axis, and the curve shown in the graph is the image quality evaluation curve 205. A reference evaluation value indicated by T in FIG. 8 is set as an image quality determination criterion, and a case where an evaluation equal to or higher than the reference evaluation value T is obtained as a good image quality.

  Fixed sample point thinning processing (4 pixels) described with reference to FIG. 3 for various subjects having a moving speed as a moving amount per frame: v = P (pixel / frame) to Q (pixel / frame). The region where the thinning process is performed and the evaluation equal to or higher than the reference evaluation value T is the region A in FIG. A region B other than the region A is a region that obtained an evaluation less than the reference evaluation value T.

  A region that has been evaluated to be equal to or higher than the reference evaluation value T, that is, a region A in FIG. In the case of having these movement amounts, even if the fixed sample point thinning process described with reference to FIG. 3 is applied, the thinning is performed with good image quality and without losing the super-resolution effect. Is determined to be realized. On the other hand, when the movement speed: v as the movement amount per frame of the subject, that is, the region B in FIG. Therefore, it is considered that the super-resolution effect does not occur or is not sufficient.

  Based on the evaluation result, when the moving speed of the subject corresponds to the speed of the region B in FIG. 8, the moving speed of the subject is conceptually accelerated or decelerated by the thinning position changing process. That is, for example, the thinning position changing process as shown in FIGS. 4 and 5 is executed. As a result, as described with reference to FIG. 7, an output similar to that when moving at the speed of the region A in FIG. 8 is obtained, and a good image quality with a super-resolution effect is obtained.

  Regions B1 and B2 shown in FIG. 9 are part of the region B shown in FIG. That is, it is an image quality region that is less than the reference evaluation value T, and is a region that is considered to be a region that does not generate a super-resolution effect or has an insufficient subject moving speed: v. When the subject is moving at the speed of the areas B1 and B2, the subject is virtually accelerated (+ Δv1, + Δv2), that is, the thinning position is changed in the direction opposite to the subject's traveling direction. 9 (for example, if the subject has moved to the right, the process of moving the sample point position to the left as the frame progresses as shown in FIG. 5) causes the region B1 in FIG. 9 and moving the region B2 in FIG. 9 to the region C2 in FIG. 9 can result in the same effect. As a result, the moving speed of the subject as a moving amount per virtual frame: v Can be set to an image quality region (C1, C2 in FIG. 9) that is equal to or higher than the reference evaluation value T. That is, as shown in FIG. 5, image conversion (compression) with little image quality deterioration is realized by executing thinning processing for moving the sample point position to the left as the frame progresses.

  Similarly, when the subject is moving at the speed of the area B3 in FIG. 10, a process of virtually decelerating the subject (-Δv3) is performed, that is, the thinning position is changed in the same direction as the traveling direction (for example, 4 moves the region B3 in FIG. 10 to the region C3 in FIG. 10, and as a result, the moving speed of the subject as a moving amount per virtual frame: v Can be set in an image quality region (C3 in FIG. 10) that is equal to or higher than the reference evaluation value T. That is, as shown in FIG. 4, image conversion (compression) with little image quality degradation is realized by executing thinning processing for moving the sample point position to the right as the frame progresses.

  Note that the image quality evaluation curves shown in FIGS. 8 to 10 are merely evaluation curves set based on the results of one experiment based on subjective evaluation, and various image quality evaluation methods can be applied. A configuration may be adopted in which the thinning processing mode is determined based on the result.

The block processing unit 131 shown in FIG. 2 that executes processing of block data with a movement amount of 2 pixels / frame or more is based on the movement amount information from the movement amount detection unit in the thinning processing mode determination unit 151 shown in FIG. ,
(A) Thinning process for fixing sample points (see FIG. 3)
(B) Thinning processing for shifting the sample point position to the right as the frame progresses (see FIG. 4)
(C) Thinning processing for shifting the sample point position to the left as the frame progresses (see FIG. 5)
It is determined which of the thinning processes (a) to (c) is executed, and the thinning process of the processing mode determined by the thinning process execution unit 152 shown in FIG. 6 is executed.

  In determining the thinning processing mode in the thinning processing mode determination unit 151 illustrated in FIG. 6, for example, determination processing based on the image quality evaluation curves illustrated in FIGS. 8 to 10 is performed. That is, when the image quality evaluation curve is a moving speed region in which the image quality equal to or higher than the reference evaluation value T is maintained, (a) the sampling point fixing thinning process (see FIG. 3) is executed, and the image quality evaluation curve becomes the reference evaluation value. In the moving speed region where the image quality is less than T, (b) thinning processing (see FIG. 4) for shifting the sample point position to the right according to the progress of the frame, or (c) the sample point position to the left according to the progress of the frame As a configuration for performing any of the thinning-out thinning processing (see FIG. 5), the virtual movement amount is set to enter the movement amount region in which the image quality evaluation curve maintains the image quality equal to or higher than the reference evaluation value T.

FIG. 11 shows a determination example of the thinning processing mode to be determined based on the movement amount information from the movement amount detection unit in the thinning processing mode determination unit 151 shown in FIG. The determination example in FIG. 11 illustrates an example in which the processing mode is determined based on the image quality evaluation curves illustrated in FIGS.
11 (a) and 11 (b)
Movement speed: 2 pixels / frame ≦ movement amount <c pixels / frame,
Movement speed: f pixel / frame ≦ movement amount <e pixel / frame,
And,
Movement direction: right,
In this case, a thinning process (see FIG. 5) for shifting the sample point position to the left as the frame progresses, that is, a thinning process with a sample point phase change amount = [+ n] is executed. This process is a virtual acceleration process and corresponds to the process of moving from the area B1 to C1 in FIG. 9 or the process of moving from the area B2 to C2.

11 (c) and 11 (d)
Movement speed: 2 pixels / frame ≦ movement amount <c pixels / frame,
Movement speed: f pixel / frame ≦ movement amount <e pixel / frame,
And,
In this case, a thinning process (see FIG. 4) for shifting the sample point position to the right as the frame progresses, that is, a thinning process with a sample point phase change amount = [− n] is performed. Execute. This process is also a virtual acceleration process and corresponds to the process of moving from the area B1 to C1 in FIG. 9 or the process of moving from the area B2 to C2.

FIG. 11 (e)
Movement speed: d pixel / frame ≦ movement amount <f pixel / frame, and
In this case, a thinning process (see FIG. 4) for shifting the sample point position to the right according to the progress of the frame, that is, a thinning process with the sample point phase change amount = [+ n] is executed. To do. This process is a virtual deceleration process and corresponds to the process of moving from the region B3 to C3 in FIG.

FIG. 11 (f)
Movement speed: d pixel / frame ≦ movement amount <f pixel / frame, and
In this case, a thinning process (see FIG. 5) for shifting the sample point position to the left according to the progress of the frame, that is, a thinning process with a sample point phase change amount = [− n] is performed. Execute. This process is a virtual deceleration process and corresponds to the process of moving from the region B3 to C3 in FIG.

FIG. 11 (g)
Movement speed other than the above, i.e.
Movement speed: c pixels / frame ≦ movement amount <d pixels / frame,
Movement speed: either e pixel / frame ≦ movement amount,
Moving direction: left or right. In this case, thinning processing (see FIG. 3) using fixed sampling points instead of thinning processing that shifts the sampling point position as the frame progresses, that is, sampling point phase change amount = [ 0] is executed. This processing corresponds to processing of a block having a subject having a movement amount corresponding to the area A in FIG.

The above-described processing example is a processing example of a block having a horizontal movement amount, but the same processing, that is, the block processing unit 131 shown in FIG. In the thinning processing mode determination unit 151 shown in FIG. 6, based on the movement amount information from the movement amount detection unit,
(1) Sample point fixed thinning process (see FIG. 12 (b1)) [sample point phase change amount = [0]]
(2) Thinning process for shifting the sample point position downward as the frame progresses (see FIG. 13B2) [Sample point phase change amount = [+ n]]
(3) Thinning process for shifting the sample point position upward in accordance with the progress of the frame (see FIG. 14B3) [sample point phase change amount = [− n]]
Is executed, and the thinning process of the processing mode determined by the thinning process execution unit 152 shown in FIG. 6 is executed.

  In this way, the block processing unit 131 executes different thinning processing according to the moving speed: v as the moving amount of the subject included in the processing block per frame, and outputs the thinned out process to the output unit 140. When the thinning amount is 4, the block processing unit 131 performs the thinning process in the spatial direction for each of the supplied four blocks (one pixel is selected for every four adjacent pixels). The amount is reduced to ¼, and the data amount of the entire four blocks is reduced to ¼. The block processing unit 131 supplies data about four blocks whose data amount has been reduced to ¼ to the output unit 140.

  Subsequently, the operation of the block processing unit 133 will be specifically described before the description of the block processing unit 132.

  The block processing unit 133 is supplied from the block distribution unit 122 of the movement amount detection unit 120 and has a total of N blocks at the same position in each of consecutive N frames (both movement amounts in the horizontal direction and the vertical direction are one pixel). (N blocks in the case where the number is less), a process of thinning out the number of frames (time direction thinning process) is performed.

  Specifically, as shown in FIG. 15, the block processing unit 133 converts four blocks Bi at the same position in each of four consecutive frames F1 to F4 into one block (in this example, Then, the number of frames to be converted into the block Bi) of the frame F1 is thinned out (thinning of the number of frames between four frames).

  The block processing unit 133 supplies the output unit 140 with data (four blocks) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing. Although the case where N = 4 has been described as an example here, similar processing is performed even if N is another value.

  Next, the operation of the block processing unit 132 will be specifically described. The block processing unit 132 is supplied from the block distribution unit 122 of the movement amount detection unit 120 and has a total of N blocks (the movement amount in the horizontal and vertical directions is 1 in each of the consecutive N frames). The number of pixels is thinned out (spatial thinning out processing) and the number of frames is thinned out (time direction thinning out processing) for N blocks when the number of pixels is greater than or equal to 2 and less than 2 pixels.

  The moving speed: v = 1 to 2 as the moving amount per frame of the block supplied to the block processing unit 132 satisfies Expressions (Expression 1) and (Expression 2). That is, the conditions for obtaining the super-resolution effect are satisfied. However, it has been found that if a thinning process with a fixed sample point similar to that described above with reference to FIG. 3 is executed, image quality degradation may be perceived.

  Therefore, the block processing unit 132 executes the thinning process with the thinning amount m = 2 instead of the thinning amount m = 4, and furthermore, the sampling point fixed thinning process similar to the thinning process in the block processing unit 131 is performed. The sampling point moving type thinning process is appropriately selected and executed according to the subject moving speed.

  A mode of the sampling point fixed thinning process and the sampling point movement type thinning process when the thinning amount m = 2 is described with reference to FIGS. FIGS. 16 to 18 are thinning-out processing examples of the block processing unit 132 when the moving speed as the moving amount per frame in the horizontal direction is v = 1 to 2 pixels / frame.

  FIG. 16 shows a sampling point fixed thinning process, that is, a thinning process in which the sampling point phase change amount, which is a change amount of each sampling point coordinate, is set to [0], and the pixel value of each frame (k to k + 3). Thinning out the number of pixels (thinning out the number of pixels between two pixels) that makes p1 to p4 any two of them (in this example, p1, p3), that is, the thinning amount m = 2 It is an example of a thinning process.

  FIG. 17 is a sampling point moving thinning process, in which the pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p1 and p3 for the preceding frame (k, k + 1), and the subsequent frame (K + 2, k + 3) is a thinning process for setting the pixel values p2 and p4. This processing example corresponds to a thinning process in which the sample point phase change amount = [0.5], which is the change amount of the sample point coordinates for each frame.

  FIG. 18 is also a sampling point moving thinning process, in which the pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p2 and p4 for the preceding frame (k, k + 1), and the subsequent frame (K + 2, k + 3) is a thinning process for setting the pixel values p1 and p3. This processing example corresponds to a thinning process in which the sample point phase change amount = [− 0.5], which is the change amount of the sample point coordinates for each frame.

  FIGS. 19 to 21 are thinning-out processing examples of the block processing unit 132 when the moving speed as the moving amount per frame in the vertical direction is v = 1 to 2 pixels / frame.

  FIG. 19 shows a sampling point fixed thinning process, that is, a thinning process in which the sampling point phase change amount, which is the amount of change of the sampling point coordinates for each frame = [0], and the pixel value of each frame (k to k + 3). Thinning out the number of pixels (thinning out the number of pixels between two pixels) that makes p1 to p4 any two of them (in this example, p1, p3), that is, the thinning amount m = 2 It is an example of a thinning process.

  FIG. 20 shows a sampling point moving thinning process, in which the pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p1 and p3 for the preceding frame (k, k + 1), and the succeeding frame (K + 2, k + 3) is a thinning process for setting the pixel values p2 and p4. This processing example corresponds to a thinning process in which the sample point phase change amount = [0.5], which is the change amount of the sample point coordinates for each frame.

  FIG. 21 is also a sampling point moving thinning process. The pixel values p1 to p4 of each frame (k to k + 3) are set to the pixel values p2 and p4 for the preceding frame (k, k + 1), and the subsequent frame is set. (K + 2, k + 3) is a thinning process for setting the pixel values p1 and p3. This processing example corresponds to a thinning process in which the sample point phase change amount = [− 0.5], which is the change amount of the sample point coordinates for each frame.

  Similarly to the block processing unit 131, the block processing unit 132 has the configuration illustrated in FIG. That is, the block processing unit 132 includes a thinning processing mode determination unit 151 and a thinning processing execution unit 152. The thinning processing mode determination unit 151 inputs movement amount information from the movement amount detection unit 121. As described above, the block input to the block processing unit 132 is a block having a moving speed: v = 1 to 2 pixels / frame as a moving amount per frame. The moving amount information input to the determining unit 151 is moving amount information of moving speed v = 1 to 2 pixels / frame.

  The thinning process mode determination unit 151 determines which mode of the thinning process in FIGS. 16 to 21 is executed based on the value of the movement amount, and the thinning process execution unit 152 performs the thinning process mode determination unit 151. In accordance with this determination, the thinning process is executed in any of the modes of FIGS.

  In this way, the block processing unit 132 executes different thinning processing depending on the moving speed: v as the moving amount of the subject included in the processing block per frame, and outputs it to the output unit 140 (see FIGS. 1 and 2). Output. Since the block processing unit 132 executes the thinning-out amount 2 in the spatial direction thinning process and also performs the temporal direction thinning-out (4 frames → 2 frames) shown in FIG. 22, the data amount of each block is reduced to ¼, The data amount of all four blocks is reduced to ¼. The block processing unit 131 supplies data about four blocks whose data amount has been reduced to ¼ to the output unit 140.

  Next, the output unit 140 will be described. The output unit 140 indicates the data about N blocks supplied from each of the block processing units 131 to 133 of the block processing unit 130 and the amount of data reduced, and how each block is processed. Output information. Information on the processing content is information indicating whether thinning is performed in the spatial direction, thinning out in the time direction, or thinning out in the spatial / temporal direction, and if thinning out is performed in the space, whether the thinning position change processing has been performed. Information that indicates what kind of thinning position change processing has been performed, information about the frame rate of the original video, spatial resolution, etc., but other information is added. It doesn't matter.

  As described above, the moving image conversion apparatus 100 described with reference to FIGS. 1 to 22 performs compression in which image quality deterioration is suppressed by performing spatial direction thinning of different thinning phase change amounts depending on the moving speed of the subject. Therefore, the data can be reduced.

  Specifically, data compression (reduction) is performed to perform spatial thinning processing of different thinning phase change amounts depending on the moving amount of the moving object in the image. When the compressed data is restored and reproduced by this thinning process, the observer follows the moving object (subject) in the image so that the thinning process corresponding to the speed of each moving object (subject) is performed. Based on this, a super-resolution phenomenon occurs, and the effect of preventing the observer from perceiving image quality degradation due to the thinning of the number of pixels is generated.

  Note that “following sight” or “observer's line of sight follows the subject” means that the moving speed of the subject: V (pixel / frame) and the sight line speed of the observer: Vo (pixel / frame) are the same value or almost the same value. It is to become. The observer's line-of-sight speed is the amount of change in the coordinate position in the image of interest of the observer as the frame progresses.

  By the way, the above-described moving image conversion apparatus 100 applies the same thinning process to blocks at the same position between N frames. That is, the thinning-out process is performed on the block in the only thinning-out processing mode determined based on the single block movement amount calculated in units of blocks at the same position between N frames.

  However, in moving image data, the amount of movement of a block at the same position may change significantly between N frames. For example, at the edge of a moving object on the image, as shown in FIG. 23, the moving speed corresponding to the block may change greatly between N frames.

  FIG. 23 shows a state of the object 232 moving at a speed V in the right direction on the object 231 as the background in the continuous frames 0 and 1. Consider the processing applied to block 233. When frames 0 and 1 are included in a frame to which one thinning processing mode is applied, the thinning processing mode of the block 233 shown in FIG.

  When the movement amount detection corresponding to the block is performed with reference to the frame 1, for example, when the frame 1 and the frame 2 are applied, the detected movement amount is included in the block 233 as an object as a background. Since it is an area 231, it becomes [0]. In this case, the block 233 is time-decimated. When N frames to be subjected to the same thinning process are four frames (−1 to 2), since the block 233 includes the background in the frame 1, the corresponding movement amount is 0, which is an appropriate process. However, in frame 0, since the speed corresponding to block 233 is V, the time reduction is not an appropriate process for the speed.

  On the other hand, when the movement amount detection is performed with reference to frame 0, for example, when frame-1 and frame 0 are applied, the detected movement amount is included in the area of moving object 232 in block 233. Since there is [V]. In this case, the block 233 is subjected to space thinning. In frame 0, the movement amount of the block 233 is V, which is an appropriate process. However, in frame 1, since the speed corresponding to block 233 is 0, spatial decimation is not an appropriate process for the speed.

  Thus, for example, when an end of an object to be moved is included in a block, a block of a certain frame can be obtained by applying the same processing for several consecutive frames, regardless of whether spatial thinning or time thinning is applied. Even if the process is appropriate for, it is not appropriate for a block of another frame. As described above, when thinning processing that is not appropriate for the block movement amount is performed, image quality degradation is perceived.

  Furthermore, it is common for a plurality of objects to exist in an image, and an overlap often occurs between a plurality of objects. For example, as shown in FIG. 24, there are a plurality of objects 241 and 242 in the image, the object 241 is stopped, and the object 242 is at a speed V at which spatial thinning can be applied in the direction indicated by the arrow in the figure. Suppose you are moving. At this time, a thinning process applied to the block 243 indicated by a dotted line will be considered. As in the block 243, the moving speed corresponding to the block located in the overlapping portion of the plurality of objects cannot normally be defined correctly. However, when motion detection is performed by block matching, the speed of any object included in the block often becomes the movement speed corresponding to the block, and therefore the movement speed corresponding to the block 243 is 0 (here, It is assumed that the speed is calculated as V speed of the object 241) or V (speed of the object 242).

  When the moving speed corresponding to the block 243 is determined to be 0, the moving image conversion apparatus 100 performs frame number thinning in the time direction. In this case, in the portion corresponding to a part of the object 241 in the block 243, an appropriate process is applied to the speed, so that there is no problem in image quality. However, in the portion corresponding to a part of the object 242 in the block 2503, the number of frames in the time direction is originally thinned out where the number of pixels in the spatial direction is thinned, and the image quality deteriorates. appear.

  Similarly, when the moving speed corresponding to the block 243 is determined to be V, according to the moving image conversion apparatus 100, the number of pixels in the spatial direction is thinned out. In the part corresponding to a part of the object 242 in the block 243, an appropriate process is applied to the speed, so that no image quality problem occurs. However, in the part corresponding to a part of the object 241 in the block 243, the number of pixels in the spatial direction is originally thinned out to the place where the number of frames in the time direction is thinned, and the image quality deteriorates. appear.

In this way, when a plurality of objects having different movement amounts are mixed in one block, a problem in image quality occurs even if thinning processing is performed according to any moving speed.
Further, even if the block does not include a plurality of objects as described above, an incorrect amount of movement may be detected if the block is close to the overlapping portion (edge, object boundary) of the plurality of objects. In particular, motion detection by block matching may use an evaluation function that inputs a wider range than the block itself that is the target of motion detection, and even for blocks that do not include object boundaries, An error may occur. This is shown in FIG.

  FIG. 25 is an example in which an area larger than a block whose motion is to be detected is used for motion vector evaluation in block matching. One image frame shown in FIG. 25 includes a background region 251 in which the movement is zero and an object 252 that moves at a speed V in the right direction.

  When the movement amount (motion vector) corresponding to the block 254 is detected by block matching, a rectangular area 253 indicates a range of pixels to be referred to. That is, as an evaluation function used in block matching, the sum of absolute differences of pixels of the current frame and reference frame blocks is often used, but in the example shown in FIG. 25, a rectangular area 253 is used instead of the block 254. Calculate the sum of absolute differences.

  As a result, although depending on the amount of texture of the actual image, etc., since many of the pixels belonging to the rectangular area 253 used when detecting the movement amount of the block 254 are the background, the motion amount corresponding to the block 254 Can be zero. In the example of FIG. 25, although the block 254 completely includes the object 252 moving with the moving amount V to be subjected to the pixel number thinning process in the spatial direction, the moving amount is detected as 0, and the time direction There is a possibility that the number of frames will be reduced. In this case, appropriate space thinning corresponding to the block 254 is not performed, and the possibility of significant image quality degradation increases.

  As described above, when an appropriate movement amount corresponding to the block is not calculated, image conversion that leads to image quality deterioration is performed. A configuration that solves this problem will be described below.

[(2) Configuration of Moving Image Conversion Device that Performs Improved Thinning Process]
Hereinafter, data reduction (compression processing) that suppresses image quality degradation that occurs in an area where inappropriate thinning processing is performed by solving the above-described problem of the moving image conversion apparatus 100 and executing improved thinning processing. A moving image conversion apparatus to be realized will be described.

  A configuration of a moving image conversion apparatus 300 according to an embodiment of the present invention will be described with reference to FIG. Similar to the moving image conversion apparatus 100 described above with reference to FIG. 1, the moving image conversion apparatus 300 illustrated in FIG. 26 uses the super-resolution effect due to human visual characteristics, thereby reducing the image quality due to the reduction in the amount of data. This makes it possible to reduce the amount of data so that the observer does not perceive deterioration, and minimizes image quality deterioration when appropriate thinning processing cannot be performed for the amount of movement of the subject. It is possible to suppress data reduction.

  A configuration of the moving image conversion apparatus 300 illustrated in FIG. 26 will be described. Each frame of the input moving image is temporarily stored in the frame buffer 301. The accumulated frames are output to the block dividing unit 302 and the movement amount detecting unit 303. The block dividing unit 302 divides the input frame into blocks, and outputs them to the movement amount detection unit 303 and the thinning process execution unit 306. The movement amount detection unit 303 detects the movement amount relating to each block using the frame supplied from the frame buffer 301 and the block supplied from the block division unit, and generates the thinning processing mode information generation unit 304 and the thinning processing control information generation. The data is output to the unit 305.

  The thinning process mode information generation unit 304 refers to the movement speed (block movement amount) corresponding to each block supplied from the movement amount detection unit 303, and the appropriate thinning process to be performed is the number of pixels in the spatial direction. It is determined whether there is a frame count in the time direction. Further, in the case of the spatial direction pixel count thinning process, an optimal sampling point phase change amount in the spatial thinning is determined, and information on whether the temporal direction thinning process or the spatial direction thinning process is performed, and the thinning process execution unit 306 Output to.

  The thinning-out process control information generation unit 305 refers to the movement amount information corresponding to each block supplied from the movement amount detection unit 303 and the block data supplied from the block division unit 302, and each block constituting the frame Whether or not to perform the thinning process is output, and the result is output to the thinning process execution unit 306 as the thinning process control information.

  The thinning process execution unit 306 performs a thinning process on each block based on the thinning process mode information supplied from the thinning process mode information generation unit 304. However, at this time, the thinning process is not performed on the blocks for which the thinning process is not appropriate based on the control information corresponding to each block supplied from the thinning process control information generation unit 305. The thinning process execution unit 306 further outputs the processing result to the output unit 307. The output unit 307 outputs the data regarding the blocks with the reduced data amount supplied from the thinning process execution unit 306, for example, as stream data.

  Next, details of processing of each component of the moving image conversion apparatus 300 that executes the improved thinning process will be described.

  First, the frame buffer 301 will be described. The frame buffer 301 accumulates frame data constituting input moving image data. The number of frames stored in the frame buffer 301 depends on the frame image applied to the movement amount detection process in the movement amount detection unit 303.

  For example, in the movement amount detection process in the movement amount detection unit 303, when the movement amount in the current frame F is detected using the current frame F and the past frame Fk (k is a positive integer), the frame The buffer 301 is configured to store at least the past k frames when the frame F is input.

  In the movement amount detection process in the movement amount detection unit 303, when the movement amount in the current frame F is detected using the current frame F and the future frame F + k, the frame buffer 301 receives the frame F + k. The storage state of the frame F is maintained until the frame is stored.

  In addition, in the movement amount detection process in the movement amount detection unit 303, when detecting the movement amount of the frame F, the frame buffer 301 is used when the past frame F-k and the future frame F + k are used. It is configured to store at least frames from frame F-k to frame F + k.

  Hereinafter, as an example, a processing example in the case where the movement amount detection unit 303 is configured to refer to the past frame F-1 when detecting the movement amount of the frame F will be described. That is, an example of processing for detecting the movement amount of frame F based on two consecutive frames will be described. The frame buffer 301 accumulates frames necessary for the movement amount detection unit 303 to detect the movement amount, and outputs the current frame N to the block division unit 302. Further, the frame F-1 necessary for detecting the movement amount is output to the movement amount detection unit 303. As described above, various settings can be made for the frame applied to the movement amount detection process in the movement amount detection unit 303, and the present invention can be applied to each setting.

  Next, processing performed by the block dividing unit 302 will be described. The block dividing unit 302 divides each frame of the input moving image into blocks having a predetermined size, for example, 8 pixels × 8 pixels, 4 pixels × 4 pixels, etc. The information is supplied to the information generation unit 304 and the thinning process control information generation unit 305.

  The movement amount detection unit 303 detects the movement amount for each block of the frame F supplied from the block division unit 302. That is, block matching is performed with reference to a reference frame (for example, a past frame) input from the frame buffer 301 to detect a movement amount in units of blocks. The movement amount detection unit 303 supplies the detected block movement amount for each block to the thinning processing mode information generation unit 304 and the thinning processing control information generation unit 305.

  Note that the movement amount detection unit 303 can supply movement amount information corresponding to an arbitrary frame required by the thinning-out process control information generation unit 305. That is, for example, when the thinning-out process control information generation unit 305 requires movement amount information for Q frames before and after the movement amount information corresponding to the current frame, the movement amount detection unit 303 sets the movement amount information. Can be supplied. Note that the movement amount detection method used may be a method other than the block matching method. The amount of movement is expressed as a motion vector corresponding to each block. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction).

Next, processing of the thinning processing mode information generation unit 304 will be described. The thinning processing mode information generation unit 304 determines the most appropriate thinning processing mode to be applied to each block based on the movement amount information (block movement amount) corresponding to each block supplied from the movement amount detection unit 303. That is,
(A) Whether to perform time thinning processing or space thinning processing;
(B) When performing the spatial thinning process, it is determined whether the spatial thinning process in the horizontal direction or the vertical direction is performed, and further, the sampling point phase change amount at the time of spatial thinning is determined.

  To determine which time or space thinning processing is performed, first, the X direction component: Vx and the Y direction component: Vy of the block movement amount supplied from the movement amount detection unit 303 are compared. Among the components, the value of the component having the larger absolute value: When V exceeds the threshold value Vt, the number of pixels in the spatial direction is thinned out. When V is lower than the threshold value Vt, the number of frames in the time direction is thinned out. When performing spatial direction thinning, vertical space thinning is performed if | Vy |> | Vx |, and horizontal spatial thinning is performed otherwise. Further, in the case of spatial decimation, the sample point phase change amount when performing the spatial decimation process is also determined. The method for determining the sample point phase change amount when the thinning amount M = 4 will be described in detail below.

The determination process of the sample point phase change amount is performed based on the image quality evaluation curves shown in FIGS. 27 to 30 and FIG.
In the graph shown in FIG. 27, the horizontal axis is the amount of movement of the subject,
Sample point phase change = 0
FIG. 9 is a graph in which image quality evaluation is recorded in the case where the converted image data generated by performing the process of thinning out the number of pixels in the spatial direction is restored / reproduced and the follow-up view is performed, and is the same as the graph shown in FIG. The tendency of the result of the subjective evaluation experiment when the subject is moved to the right by the movement amount shown on the horizontal axis is shown as an image quality evaluation curve 401 (similar to 205 in FIG. 8). The vertical axis represents image quality evaluation (evaluation that the higher the image quality, the better).

  A reference evaluation value 402 indicated by T in FIG. 27 is set, and a region of speed at which an evaluation equal to or higher than the reference evaluation value T is indicated by high image quality movement amount regions 403 to 405. From FIG. 27, it can be seen that when the subject is moving at a speed in the region from 403 to 405, sufficient image quality can be obtained even if the sampling point phase change amount = 0 is thinned out.

FIG. 28 to FIG. 30 are image quality evaluation curves created by shifting the curve 401 of FIG. 27 in the right direction.
FIG. 28: When the sampling point phase change amount is +1,
FIG. 29: When the sampling point phase change amount is +2,
FIG. 30: When the sampling point phase change amount is +3,
Correspond as above.

  The reason why an image quality evaluation curve for different sample point phase change amounts can be obtained by shifting the curve 401 in FIG. 27 to the right will be described below.

As already shown in (Expression 4), the thinning position change processing (processing in which the sampling point phase change amount ≠ 0) is performed by changing the sampling point shift amount φt shown in (Expression 3) by 1 for each frame. This is equivalent to / m addition (when shifting to the right) or 1 / m subtraction (when shifting to the left). Note that m is a thinning amount (m pixels). More generally, the thinning process when the sampling point phase change amount = P is equivalent to adding P / m to the sampling point deviation amount φt shown in (Equation 3) for each frame. A new sample point shift amount φ′t in the case where the number of pixels is thinned out under the sample point phase change amount = P is expressed by the following formula (formula 5).

  After all, it can be seen from (Equation 5) that performing the thinning process with the sample point phase change amount = P is equivalent to conceptually reducing the moving speed v of the subject. This indicates that, for example, when P = 1, the image quality evaluation value at v = V1 is obtained from the image quality evaluation value at v = V1-1 when P = 0. Therefore, it can be seen that the image quality evaluation curve for P = 1 (curve 411 in FIG. 28) is obtained by shifting the curve 401 in FIG. 27 (curve indicated by the dotted line in FIG. 28) to the right by one pixel. A region of speed at which an evaluation equal to or higher than the reference evaluation value T is indicated by high image quality movement amount regions 413 to 416.

  Similarly, the image quality evaluation value at v = Vp in the case of the sampling point phase change amount P is obtained from the image quality evaluation value in the case of the velocity v = Vp−P in the case of the sampling point phase change amount = 0. The image quality evaluation curve 421 in the case of P = 2 is obtained by shifting the curve 401 (curve indicated by the dotted line in FIG. 29) to the right by 2 pixels. A region of a speed at which an evaluation equal to or higher than the reference evaluation value T is indicated by high image quality movement amount regions 423 to 425.

  Further, it can be seen that the image quality evaluation curve 431 when P = 3 in FIG. 30 is obtained by shifting the curve 401 (curve indicated by the dotted line in FIG. 30) to the right by 3 pixels. A region of a speed at which an evaluation equal to or higher than the reference evaluation value T is indicated by high image quality movement amount regions 433 to 436.

  As described above, the image quality evaluation curve other than the case where the sample point phase change amount P = 0 is created by simulation. However, the case of all sample point phase change amounts is the same as the case where P = 0 is actually obtained through experiments or the like. You may create a graph.

  FIG. 31 is a diagram showing all the image quality evaluation curves 401, 411, 421, and 431 of FIGS. As can be understood with reference to FIG. 31, at any speed, there are a plurality of sample point phase change amounts where the image quality evaluation value exceeds the threshold value T (a sufficient image quality can be obtained).

  For example, in FIGS. 31A and 31B, the curves 401 and 421 exceed the threshold value T at the position of the movement amount v = 3 pixels / frame (on the dotted line 441). That is, in the case of v = 3, it can be seen that when the sampling point phase change amount P = 0 or P = 2 is thinned, it is possible to provide an image with sufficient quality that the observer does not feel image quality deterioration.

  In this way, the thinning-out processing mode information determination unit 304 has sufficient image quality with the movement amount corresponding to each block supplied from the movement amount detection unit 303 based on the image quality evaluation curve corresponding to each sample point phase change amount. A sample point phase change amount that can obtain (image quality exceeding the threshold value T) is determined. In the case where there are a plurality of such sample point phase change amounts, it may be determined that the highest evaluation value can be obtained. However, it may be selected by other methods.

  In the above example, the converted image data is moved in the right direction to create a graph. However, the image quality evaluation curves shown in FIGS. 27 to 31 do not depend on the moving direction (horizontal / vertical) of the image data. Candidates for the sample point phase change amount in thinning out the number of pixels can also be determined using FIGS.

  The above example is the image quality evaluation characteristic when the thinning amount M = 4. However, the image quality evaluation characteristic can be generated by the same means even when the thinning amount is other values. That is, first, converted image data when the sample point phase change amount is set to 0 is generated, displayed at different moving speeds, measured for image quality, and then the obtained image quality evaluation curve is shifted. Thus, it is possible to obtain evaluation curves corresponding to all sample point phase change amounts corresponding to the thinning amount M. However, an image quality evaluation curve corresponding to all sample point phase change amounts may be obtained by experiments. That is, regarding the sample point phase change amount other than 0, it is also possible to actually create converted image data, display it at a different speed, and obtain an evaluation curve.

  As described above, the thinning processing mode information determination unit 304 determines the thinning processing mode to be applied to each block and supplies it to the thinning processing execution unit 306.

  Next, the thinning process control information generation unit 305 will be described. The thinning-out process control information generation unit 305 schedules execution of the thinning-out process based on the movement amount information corresponding to each block supplied from the movement amount detection unit 303 and the pixel data of each block supplied from the block division unit 302. For each block constituting one current frame selected as a frame, it is determined whether it is appropriate to perform the thinning process, and the result is output to the thinning process execution unit 306 as the thinning process control information.

  There are many possible methods for determining blocks that should not be thinned out, but as described above, the boundary between multiple objects that move at different speeds (the background may also be considered as one object) In addition, since the thinning process impossible area should be set as the surrounding area, any method may be used as long as it can be determined that the thinning process is impossible. That is, for example, the thinning-out process control information generation unit 305 verifies whether or not a plurality of objects having different moving speeds are included in one block, and performs control that disables thinning-out processing for a block including a plurality of objects. It has a configuration for generating information.

  Hereinafter, three different determination methods A to C for determining whether or not to perform a thinning process on a block will be described. The thinning-out process control information generation unit 305 may determine a block that does not perform the thinning-out process, that is, disables the thinning-out process by all methods of determination methods A, B, and C described below. It may be determined by only one method.

(Method A) Method Using Variance Between Block Differences First, a method for determining whether or not to perform a thinning process on a block using variance between block differences will be described. In this method, in order to determine thinning-out process control information regarding a certain block B, a frame other than the current frame F1 to which the block B belongs is used as a reference frame. Here, it is assumed that a frame F0 that is one frame past the current frame F1 is used as a reference frame. The reference frame may be any frame as long as it is a frame other than the current frame.

  When determining whether or not to perform the thinning process on the block using the method A, that is, the distribution of the difference between blocks, the thinning process control information generation unit 305 can refer to the immediately preceding frame F0 in addition to the current frame F1. Suppose that it is a configuration. The thinning process control information generation unit 305 is supplied with movement amount information (motion vector) corresponding to each block of the current frame from the movement amount detection unit 303. Let the motion vector corresponding to block B be (Vx, Vy). Also, the pixel value of the pixel (x, y) in the frame F1 is I1 (x, y), and the pixel value of the pixel (x, y) in the frame F0 is I0 (x, y). In method A, the variance of the difference between each pixel at the position where block B was present in the immediately preceding frame and each pixel constituting block B is calculated. The position where the block B was present in the previous frame can be obtained by subtracting the motion vector (Vx, Vy) corresponding to the block B from the current coordinates.

The variance [var] of the difference between corresponding blocks in the current frame and the reference frame is calculated by the following equation. That is,
Here, I _ave is the average value of the differences of each pixel,
It is.
M is the block size.

The thinning process control information generation unit 305 compares the difference [var] of the difference between the corresponding blocks in the current frame and the reference frame calculated by the above formula with a predetermined threshold [Tv].
var> Tv
In other words, if the difference variance [var] exceeds the threshold [Tv], the block is determined as a block that cannot be thinned out.
on the other hand,
var> Tv
Is not satisfied, that is, when the difference variance [var] is equal to or smaller than the threshold [Tv], the block is determined as a block that can be thinned.

  This is a method for solving the problem of the moving image conversion apparatus 100 as shown in FIG. 24, for example. That is, the movement amount corresponding to the block 243 shown in FIG. 24 is likely to be the movement amount of either the object 241 or the object 242. Here, it is assumed that the movement amount corresponding to the block 243 becomes the movement amount V of the object 242. In this case, when each pixel of the block 243 is compared with each pixel at the position where the block 243 existed in the immediately preceding frame, the difference value of the left portion of the block 243 decreases, but the pixel value of the right portion increases.

  A specific example will be described with reference to FIG. FIG. 32 shows the position of the object 242 shown in FIG. 24 in the current frame and the past frame. The object 241 is omitted. Since the movement amount corresponding to the block 243 is V, the difference between the pixels of the block 243 and the region 244 is calculated. At this time, the pixel value is clearly determined on the left side of the block, that is, the portion including the object 242. Are consistent. Or since it is very close, a difference value becomes small and the difference value of a part other than that becomes large. That is, the value of the variance var increases. On the other hand, when the block contains only one object, the difference value becomes smaller as a whole, so the variance becomes smaller. Accordingly, a block whose variance exceeds a certain threshold value Tv is likely to include a plurality of objects, and thus thinning processing is disabled.

  The control information that indicates whether or not the thinning process is possible is output to the thinning process execution unit 306 in FIG. The thinning process execution unit 306 performs the thinning process on the blocks for which the thinning process is enabled, but does not perform the thinning process on the blocks for which the thinning process is disabled.

  This control prevents thinning out of blocks including a plurality of different objects. As a result, image quality deterioration due to uniform thinning processing based on one movement amount corresponding to a block as described with reference to FIG. Can be suppressed. By appropriately adjusting the threshold value Tv, the total number of blocks that cannot be thinned out can be adjusted, so that the data reduction amount can also be adjusted.

(Method B) Method for Detecting Variation in Thinning Process of Frame Next, a method for determining whether or not thinning is performed based on detection of variation in thinning process between frames will be described. In this method, in order to determine thinning-out process control information regarding a certain block B, movement amount information corresponding to a frame other than the current frame F1 to which the block B belongs is used. That is, when Method B is used, the thinning-out process control information generation unit 305 uses movement amount information corresponding to the previous and subsequent N frames in addition to the movement amount information corresponding to the current frame. The value of N may be any value, but here it is assumed that N = 2. That is, the moving amount information of the block at the same position as the block B of the two frames before and after the current frame is referred to.

  In this method B, that is, the thinning-out possibility determination processing method by detecting the thinning-out processing variation between frames, when determining the thinning-out processing control information for a certain block B, in addition to the movement amount information corresponding to the block B in the current frame Then, referring to the movement amount information of the block at the same position as block B in the two frames before and after, if the optimum thinning-out processing form determined from these five types of block movement amount information is all the same, block B The corresponding thinning process control information is set to “Thinning process is possible”. In other cases, the thinning process control information is set to “thinning process impossible”.

  A specific example will be described with reference to FIG. FIG. 33 shows the fluctuation between 5 frames of 8 blocks arranged in the horizontal direction. The horizontal axis is the block position (x coordinate), the vertical axis is time (t), the time progresses from top to bottom, the current frame is the middle frame, and the top two frames In the lower row, a total of 5 frames, 2 subsequent frames, are shown.

  The shaded portion shown in the figure is the moving object (subject). One block is 4 × 4 pixels, and the object (subject) moves in the right direction at 4 pixels / frame. The white part is the background and does not move.

  Here, paying attention to the block B0 of the current frame, all the blocks at the same position in the previous and subsequent frames are moving objects in the hatched area, and move at 4 pixels / frame. That is, the movement amount corresponding to all the blocks between the five frames centering on the current frame is the movement amount that makes space thinning an appropriate thinning process, and in this case, the block B0 is a “thinning process is possible” block.

  Similarly, in block B2, since the movement amounts of all five blocks including the blocks in the preceding and succeeding frames are 0 and the movement amount is set to appropriate thinning processing, the thinning processing control corresponding to block B2 is performed. The information is also “Thinning is possible”.

  On the other hand, referring to the movement amount of the previous and subsequent frames of the block B1, the blocks of the past frames (the two blocks immediately above the block B1 in the figure) are blocks for which the time thinning is appropriately thinned, and the block B1 and the future frame These blocks (two blocks immediately below the block B1) are blocks for which spatial thinning is performed as appropriate thinning processing. In this case, since the optimum thinning form is not the same in the five blocks, the thinning process control information corresponding to the block B1 is set to “impossible thinning process”.

  The control information that indicates whether or not the thinning process is possible is output to the thinning process execution unit 306 in FIG. The thinning process execution unit 306 performs the thinning process on the blocks for which the thinning process is enabled, but does not perform the thinning process on the blocks for which the thinning process is disabled.

  This control is a method for solving the problems of the moving image conversion apparatus 100 as mainly shown in FIGS. That is, as shown in FIG. 23, in the moving image conversion apparatus 100 described above, at the boundary portion between the area where space thinning is performed and the area where time thinning is performed, in units of N frames (for example, 4 frames). Although the same thinning process is performed, if this determination method (B) is applied, the thinning process is not executed in such a block, and there are objects that move at different moving speeds in continuous frames. Inappropriate thinning processing can be prevented.

  Also, as shown in FIG. 25, an error in detecting the amount of movement is likely to occur at the boundary portion of objects with different amounts of movement. Therefore, thinning processing should be performed in consideration of the possibility that an error has occurred in advance. Can be suppressed. As in the case of Method A, the total number of blocks that cannot be thinned out can be adjusted by appropriately adjusting the number of frames before and after reference = N, so that the data reduction amount can be adjusted. Can be done.

(Method C) Method for Detecting Variation in Thinning Process of Surrounding Blocks Next, a method for determining whether or not thinning is possible based on detection of variation in thinning processing of surrounding blocks will be described. In this method, in order to determine the thinning-out process control information related to a certain block B, movement amount information corresponding to a frame other than the current frame F1 to which the block B belongs and block data of the current frame are not required. That is, unlike the method A and the method B, it is not necessary to refer to another frame different from the current frame.

  In the method C, when determining the thinning-out process control information of a certain block B, the optimum thinning-out determined from the movement amount information corresponding to the block B and the surrounding blocks is referred to by referring to the movement amount of the surrounding blocks of the block B in the current frame When all the processing forms are the same, the thinning process control information corresponding to the block B is set to “thinning process is possible”. In other cases, the thinning process control information is set to “thinning process impossible”.

  The surrounding blocks are blocks included in an area larger than the block B including the block B. For example, adjacent blocks on the top, bottom, left, and right of the block B are set as the surrounding blocks. A specific processing example will be described with reference to FIG. FIG. 34 shows the movement amount of each block at a certain position in the current frame.

  Each rectangle in the figure is a block, the shaded block is a block with a corresponding movement amount corresponding to a movement amount that makes space thinning processing the optimum thinning process, and a white block is a corresponding movement The amount is a block corresponding to the movement amount that makes the time thinning process the optimum thinning process.

  Focusing on the block B3 in the figure, the upper and left blocks are the movement amounts that optimize the time decimation, while the block B3 and the right and lower blocks are the movement amounts that optimize the space decimation. . Therefore, since the thinning process corresponding to the movement amount of all the blocks is not common, the thinning process control information corresponding to the block B3 is set to “thinning process impossible”.

  On the other hand, since the block B4 corresponds to the block B4 and the movement amount that optimizes the space thinning to all the upper, lower, left, and right blocks, the thinning process control information corresponding to the block B4 is “thinning process is possible”. To do.

  The control information that indicates whether or not the thinning process is possible is output to the thinning process execution unit 306 in FIG. The thinning process execution unit 306 performs the thinning process on the blocks for which the thinning process is enabled, but does not perform the thinning process on the blocks for which the thinning process is disabled.

  The above processing is a method for solving the problems of the moving image conversion apparatus 100 as mainly shown in FIGS. That is, as shown in FIG. 23, the boundary between the area where space thinning is performed and the area where time thinning is performed is different from the moving image conversion apparatus 10, but N frames like the moving image conversion apparatus 100. When the same thinning process is performed in units of (for example, 4 frames), it is possible to prevent the thinning process that is not appropriate for the movement amount from being performed. Also, as shown in FIG. 25, an error in detecting the amount of movement is likely to occur at the boundary portion of objects with different amounts of movement. Therefore, thinning processing should be performed in consideration of the possibility that an error has occurred in advance. Can be suppressed. As in the case of Method A, the total number of blocks that cannot be thinned out can be adjusted by appropriately adjusting the range of the surrounding blocks to be referenced, so that the data reduction amount can also be adjusted. Can do.

  As described above, the thinning-out process control information generating unit 305 determines control information of “Thinning processing is possible” or “Thinning processing is impossible” for all the blocks constituting the current frame, and this is thinned out. The data is output to the process execution unit 306. The thinning-out process control information generation unit 305 applies all the above-described determination methods A, B, and C, and disables the thinning-out process for a block that has been determined to be non-thinning process in any of the determination methods. Alternatively, it may be configured to determine whether block thinning is possible by only one of the above-described determination methods A, B, and C.

  Note that the thinning-out process control information generation unit 305 depends on the block data divided by the block division unit 302 and the movement amount information detected by the movement amount detection unit 303 as control information corresponding to all blocks corresponding to the specific frame. Instead, a configuration may be adopted in which “decimation processing is possible” or “decimation processing is not possible” is uniformly set and output to the thinning processing execution unit 306 is enabled.

  For example, when the storage means that is the output destination has a sufficient storage capacity, or when the original data is required in the receiving unit via the network as the output destination, for all blocks corresponding to the specific frame Set the control information to “Thinning is not possible” and output the original data, or vice versa, if the storage means that is the output destination does not have sufficient storage capacity, or the network as the output destination Is congested and it is difficult to secure sufficient bandwidth, the control information for all blocks corresponding to the specific frame is set to “Thinning is possible” and data is thinned out for all blocks. It may be configured to perform processing such as reducing the amount.

  Next, processing of the thinning process execution unit 306 will be described. The decimation processing execution unit 306 is supplied from the block dividing unit 302 based on the decimation processing mode information supplied from the decimation processing mode information generation unit 304 and the decimation processing control information supplied from the decimation processing control information generation unit 305. Data reduction processing by thinning is performed on each block.

  As described above in “(1) Basic configuration of moving image conversion apparatus using super-resolution effect”, the type of decimation performed by the decimation processing execution unit 306 is the same as described in “(1) Basic configuration of moving image conversion apparatus using super-resolution effect”. Processing), a frame number thinning process in the time direction (time thinning process), and a combination of selective combination of space thinning process and time thinning process can be considered. The thinning process of the moving image conversion apparatus 300 of the embodiment described below It is assumed that the execution unit 306 performs either the pixel count thinning process in the spatial direction or the frame number thinning process in the time direction. In the moving image conversion apparatus 100 shown in (1), the thinning process is performed in units of 4 frames. However, in the moving image conversion apparatus 300 of this embodiment, an example in which the thinning process is performed in units of 1 frame will be described. . However, the processing may be performed in units of several frames like the moving image conversion apparatus 100. Further, thinning processing combining time thinning and space thinning may be performed. First, the details of each thinning process will be described below.

As described above, the thinning processing mode information generation unit 304 performs the most appropriate thinning processing to be performed on each block based on the movement amount information (block movement amount) corresponding to each block supplied from the movement amount detection unit 303. Define aspects. That is,
(A) Whether to perform time thinning processing or space thinning processing;
(B) When performing the spatial thinning process, it is determined whether the spatial thinning process in the horizontal direction or the vertical direction is performed, and further, the sampling point phase change amount at the time of spatial thinning is determined.

  The thinning process execution unit 306 changes the process according to the thinning process mode information input from the thinning process mode information generation unit 304.

  Further, the thinning process execution unit 306 receives block-corresponding control information for which the thinning process is enabled or disabled from the thinning process control information generation unit 305, and performs the thinning process for blocks that are enabled for the thinning process. For blocks that cannot be thinned out, the thinned-out processing is not performed.

  That is, when the control information input from the thinning process control information generation unit 305 is not thinned, the thinning process execution unit 306 stops the thinning process and the control information input from the thinning process control information generation unit 305 can be thinned out. If it is, the thinning process according to the information input from the thinning process mode information generation unit 304 is executed.

  Hereinafter, a specific example of the thinning process executed by the thinning process execution unit 306 when the control information input from the thinning process control information generation unit 305 can be thinned will be described.

  (1) When the decimation process mode information supplied from the decimation process mode information generation unit 304 is a time decimation process

  When the decimation process mode information corresponding to a certain block is the decimation process for the number of frames in the time direction, the decimation process execution unit 306 responds to the frame number F (= 0, 1, 2,...) Currently being processed. The number of frames is thinned out. If the thinning amount is M (integer), if the frame number F is a multiple of M, the data of that block is output as it is. If the frame number F is not a multiple of M, all the data in that block is deleted. The time thinning process when the thinning amount M = 4 corresponds to the process shown in FIG.

  (2) When the thinning processing mode information supplied from the thinning processing mode information generation unit 304 is a spatial thinning process

  When the thinning processing mode information corresponding to a certain block is a pixel number thinning process in the spatial direction, the thinning process execution unit 306 performs a pixel number thinning process according to the frame number F currently being processed. The thinning amount is M. When the spatial thinning direction indicated by the thinning processing mode information is the horizontal direction, the thinning processing execution unit 306 first divides the M × M block into a set of vertical 1 pixel × horizontal M pixel units as shown in FIG. .

  The sampling point phase change amount P obtained from the thinning processing mode information supplied from the thinning processing mode information generation unit 304 can take a value from 0 to M-1. At this time, if the frame number F currently being processed is represented by M × N + k and the coordinates of the leftmost pixel among the M pixels arranged in the horizontal direction are represented by (X, Y), the thinning process execution unit 306 A pixel value at a position of (X + (P × k + 1 MOD M), Y) is set as a representative pixel value, that is, a sampling point, as pixel values of M pixels arranged in the horizontal direction.

  36 to 40 show horizontal spatial thinning processing in the case of M = 4. The thinning processing execution unit 306 first converts a 4 × 4 block into horizontal 1 × vertical 4 pixel units as shown in FIG. The pixels are divided into sets, and the pixel number thinning process illustrated in FIGS. 37 to 40 is performed according to the sampling point phase change amount P indicated by the thinning mode information. The possible value of P is 0, 1, 2, or 3 when M = 4. FIG. 37 shows the process when P = 0, and FIG. 38 shows the process when P = 1. The processing in the case of 2 corresponds to FIG. 39, and the processing in the case of P = 3 corresponds to FIG.

  When the spatial thinning direction indicated by the thinning processing mode information is the vertical direction, the thinning processing execution unit 306 first divides the M × M block into a set of vertical M pixels × horizontal 1 pixel units as shown in FIG. .

  The sampling point phase change amount P obtained from the thinning processing mode information supplied from the thinning processing mode information generation unit 304 can take a value from 0 to M-1. At this time, if the frame number F currently being processed is represented by M × N + k, and the coordinates of the top pixel among the M pixels arranged in the vertical direction are represented by (X, Y), the thinning process execution unit 306 A pixel value at a position of (X, Y + (P × k + 1 MOD M)) is set as a representative pixel value, that is, a sampling point, as pixel values of M pixels arranged in the vertical direction.

  42 to 46 show the spatial thinning process in the vertical direction when M = 4, and the thinning process execution unit 306 first converts a 4 × 4 block into a horizontal 4 × vertical pixel unit as shown in FIG. The pixels are divided into sets, and the pixel number thinning process shown in FIGS. 43 to 46 is performed according to the sampling point phase change amount P indicated by the thinning mode information. The process when P = 0 corresponds to FIG. 43, the process when P = 1 corresponds to FIG. 44, the process when P = 2 corresponds to FIG. 45, and the process when P = 3 corresponds to FIG.

  A more detailed flow of the process performed by the thinning process execution unit 306 will be described with reference to the flowchart shown in FIG. FIG. 47 shows a flow until the thinning process execution unit 306 finally applies some processing to a certain block.

  In step S101, decimation mode information corresponding to the block: R and decimation control information corresponding to the block: S are input from the decimation processing mode information generation unit 304 and the decimation processing control information generation unit 305, respectively.

  In step S102, by checking the control information S for the block, it is checked whether or not thinning processing is possible for the block. If S indicates that the thinning process is not possible, no thinning process is performed for the block, the block data is output as it is, and the process for the block is terminated (S108).

  Control information for the block: If S indicates that the thinning process is possible, the process proceeds to S103, and the thinning mode information R for the block is referred to. If R indicates that the processing to be performed on the block is spatial thinning, spatial thinning is performed on the block (S104), the block data resulting from the spatial thinning is output, and the processing ends (S107). . If R indicates that the processing to be performed on the block is not spatial thinning, the process proceeds to S105, time thinning processing is performed on the block, block data as a result of the time thinning is output, and the processing is terminated (S106). .

  The thinning process execution unit 306 executes such processing according to the flowchart shown in FIG. 47 for all the blocks constituting the frame, and outputs the result to the output unit 307.

  Finally, the output unit 307 will be described. The output unit 307 outputs data about a block in which the data amount supplied from the thinning process execution unit 306 is reduced or an original block (when the thinning process control information is “thinning process is not possible”), Output information about processing. Information regarding the processing contents includes “information indicating whether or not thinning processing has been performed (thinning processing control information)”, “information indicating whether spatial or temporal thinning processing has been performed, and spatial direction thinning processing. If applied, information indicating which direction (horizontal / vertical) thinning processing has been performed, information indicating sample point phase change amount in thinning processing in the spatial direction (thinning processing mode information), original video Although it is composed of information on the frame rate of the image, spatial resolution, etc., other information may be added.

  Note that a storage medium such as a hard disk or a DVD, or network output means is connected to the subsequent stage of the output unit 307, and data compressed by data conversion or network output is executed.

[(3) Configuration of moving image restoration apparatus]
Next, a moving image restoration apparatus that performs restoration processing of converted (compressed) data generated by the above-described moving image conversion apparatus 300 will be described.

The configuration of the moving image restoration apparatus 500 is shown in FIG. As shown in FIG. 48, the moving image restoration apparatus 500 includes a data expansion unit 501 and a synthesis unit 502.
The data expansion unit 501 receives the conversion (compression) data generated by the above-described moving image conversion apparatus 300 and attribute data necessary for restoration. The attribute data includes decimation process control information regarding each block and decimation process mode information regarding the specific contents of the process applied to each block. The specific contents of the processing applied to each block include information on whether the spatial direction thinning process or the time direction thinning process has been executed, the information on the sampling point phase change amount in the thinning process executed in the spatial thinning process, etc. It is information to include.

  The data extension unit 501 performs either the spatial extension process or the time extension process on the basis of the thinning process mode information and the thinning process control information included in the input attribute information. Or block data is sent directly to the combining unit 502.

  Specifically, when the process performed by the thinning process execution unit 306 on the block in the moving image conversion apparatus 300 is a spatial thinning process, the data expansion unit 501 applies space to each block data that has been thinned. Based on the thinning processing mode information indicating the thinning direction and the sampling point phase change amount, the spatial expansion processing is performed and output to the combining unit 502.

  If the process performed by the thinning process execution unit 306 in the moving image conversion apparatus 300 is a time decimation process, a time extension process is performed and output to the synthesis unit 502. If the thinning process control information corresponding to the block is “cannot be thinned”, since the thinning process is not performed on the block, the data is directly output to the combining unit 502.

  In the space expansion process executed by the data expansion unit 501, the data expansion process that has been thinned in the spatial direction by the thinning process execution unit 306 of the moving image conversion apparatus 300 is executed. Based on the inputted thinning-out processing mode information, the data expansion processing in the mode shown in FIGS. 49 and 50 is executed to reconfigure the block. Further, the information indicating the reconstructed block and the content of the processing is output to the synthesis unit 502.

  The processing of FIGS. 49 and 50 will be described. FIG. 49 shows processing performed by the data expansion unit 501 of the moving image restoration apparatus 500 when the thinning process execution unit 306 of the moving image conversion apparatus 300 performs spatial thinning processing in the horizontal direction. For example, when the size of the original block is 4 × 4, data corresponding to ¼ of the four pixels is sent to the data expansion unit 501. With respect to the data of each pixel, one pixel is expanded to 1 × 4 pixels by performing a pixel copy expansion process as shown in FIG. 49 (1). Note that the processing example shown in FIG. 49 (1) is an example in which expansion is performed by arranging the pixel values of the input pixels as the pixel values for four pixels as they are, but includes the values of the other input pixels. You may arrange | position the result by the calculation based on a pixel value.

  Subsequently, when the spatial thinning in the horizontal direction is performed, the data expansion unit 501 performs the process shown in FIG. That is, by arranging the set of 1 × 4 pixels obtained by the processing of FIG. 49 (1) as shown in FIG. 49 (2), a 4 × 4 block is restored from the four input pixels, and the result is output to the synthesis unit 502 To do. Information indicating the contents of the thinning process is also output.

  FIG. 50 illustrates the spatial expansion process when the vertical thinning process is performed in the thinning process execution unit 306 of the moving image conversion apparatus 300. For example, when the size of the original block is 4 × 4, data corresponding to ¼ of the four pixels is sent to the data expansion unit 501. With respect to the data of each pixel, by performing the process of FIG. 50 (1), one pixel is expanded to 4 × 1 pixels. The processing example shown in FIG. 50 (1) is an example in which expansion is performed by arranging the pixel values of the input pixels as the pixel values for four pixels as they are, but includes the values of the other input pixels. You may arrange | position the result by the calculation based on a pixel value.

  Subsequently, when space thinning in the vertical direction is performed, the data expansion unit 501 performs the process shown in FIG. That is, by arranging a set of 4 × 1 pixels obtained by the processing of FIG. 50 (1) as shown in FIG. 50 (2), a 4 × 4 block is restored from the four input pixels, and the result is output to the synthesis unit 502 To do. Information indicating the contents of the thinning process is also output.

  Next, the time extension process executed by the data extension unit 501 will be described. In the time extension process, the moving image conversion apparatus 300 performs a process of extending the data subjected to the time reduction process in the thinning process execution unit 306. To do.

  In the time extension processing, as shown in FIG. 51, block data of a plurality of frames is generated based on block data of one frame. Specifically, the data expansion unit 501 has a counter with an initial value of 0, and is incremented by 1 every time one frame is restored, and the value reaches 4 (= N (when the thinning amount m = 4)). Reset to 0.

  When data subject to time expansion is input, the block is stored in the memory, and multiple frame data are supported as a copy of the block data stored in the memory until the counter upper limit value set according to the thinning amount is reached. Block data is generated and output to the synthesis unit 502. In the example shown in the figure, the thinning-out amount m = 4, and block data corresponding to four frames is generated from one block data and output to the combining unit 502.

  In the data expansion unit 501, if the thinning process control information corresponding to the block is “thinning is not possible”, the block data of the block is not thinned, and is output to the combining unit 502 as it is. .

  When the block input from the data expansion unit 501 reaches an amount capable of expressing the entire frame, the synthesizer 502 appropriately arranges the block according to the information indicating the contents of the thinning process input at the same time, and arranges one frame. Restore and output the entire restored frame. The block arrangement process will be described below with reference to FIGS.

  52 to 56 show an example of data of one frame composed of a plurality of blocks. In the example of FIGS. 52 to 56, one frame data is composed of nine 3 × 3 blocks. In each figure, each of the squares divided by the broken lines is a block and is a square painted in gray. For example, in FIG. 52, the block 601 is input from the data expansion unit 501 by the combining unit 502 and arranged. Is a block.

  FIG. 52 is a block that is input from the data expansion unit 501 by the combining unit 502 and to be placed is a block that has been subjected to thinning processing in the time direction, or a block that has not been subjected to any thinning processing. The block arrangement position in the case is shown. In this case, the synthesizing unit 502 arranges the restored block at the same position as when it was first divided in the block dividing unit 302 in the moving image conversion apparatus 300.

  FIG. 53 shows an example of restored block placement processing when a block to be placed that has been input from the data expansion unit 501 has been subjected to horizontal spatial thinning processing.

  FIG. 53A shows a restored data block when the horizontal spatial thinning process is executed when the sampling point offset of the restored block is zero. The sample point offset indicates w when the sample point coordinate is (X + w, Y) when the upper left coordinate of the block is (X, Y). That is, as explained in the description of the processing of the thinning processing execution unit 306, the thinning amount is M, the sampling point phase change amount is P, and the current frame number is represented by F = M × N + k (N and k are integers). Sometimes, the sample point coordinates are determined by (X + (P × k + 1 MOD M), Y), so w is determined by the current frame number and the sample point phase change amount. For example, if the sample point phase change amount = 0, the sample point offset is 1 regardless of the frame number. Therefore, if the sample point phase change amount = 0, a block is arranged at the position shown in FIG. The If the sample point phase change amount is 1, when the value of k of the frame number F = M × N + k is 0 (b), 1 is (c), 2 is (d), 3 In this case, the block is arranged at the position shown in (a).

  As described above, when restoring the thinning-out process data in the horizontal direction, the combining unit 502 executes a process of moving the restored block in the horizontal direction for each frame and arranging it. The reason for this processing will be described with reference to FIG.

  54A to 54D show partial pixel data (4 × 4) data constituting the block of frame data corresponding to FIGS. 53A to 53D. As shown in FIGS. 54A to 54D, conversion data is generated by shifting the sample point position by one pixel in the right direction in accordance with the sample point offset. When the thinning amount M = 4, for example, all four pixels in the horizontal direction are set to the same pixel value. However, the position of the pixel having the pixel value is originally shifted by one pixel in each frame as shown in FIGS. 54 (a) → (d). If the position of the pixel having the original pixel value is set so as to be substantially at the center of the display area, an image closer to the original image data can be reproduced.

  FIG. 55 shows an example of restored block arrangement processing when a block input from the data expansion unit 501 and subjected to arrangement has been subjected to vertical thinning processing.

  FIG. 55A shows a restored data block when the vertical spatial thinning process is executed when the sampling point offset of the restored block is zero. The sample point offset in the case of vertical processing refers to w when the sample point coordinate is (X, Y + w) when the upper left coordinate of the block is (X, Y). That is, as explained in detail in the thinning processing execution unit 306, when the thinning amount is M, the sampling point phase change amount is P, and the current frame number is represented by F = M × N + k (N and k are integers). Since the sample point coordinates are determined by (X, Y + (P × k + 1 MOD M)), w is determined by the current frame number and the sample point phase change amount. For example, if the sample point phase change amount = 0, the sample point offset is 1 regardless of the frame number. Therefore, if the sample point phase change amount = 0, a block is arranged at the position shown in FIG. The If the sample point phase change amount is 1, when the value of k of the frame number F = M × N + k is 0 (b), 1 is (c), 2 is (d), 3 In this case, the block is arranged at the position shown in (a).

  As described above, when restoring the thinning process data in the vertical direction, the synthesis unit 502 executes a process of moving the restored block in the vertical direction for each frame and arranging it. The reason for this processing is the same as the horizontal movement processing described with reference to FIG. 54, and is for realizing reproduction of an image closer to the original image data.

  By the way, as described above, when a block is arranged at a position deviated from the original block arrangement, depending on the contents of the processing to which the adjacent block is applied, for example, as shown in FIG. May overlap, or as shown in FIG. 56 (b), a gap may be opened between the blocks. In such a case, for example, when overlapping, the average of the overlapping portions of both blocks is used as the pixel value of the overlapping portion, or when the gap is opened, linear interpolation is performed using the pixel values on both sides of the gap. Execute correction processing such as The synthesizing unit 502 also executes such pixel value correction processing, and completes and outputs frame data composed of block data.

  As described above, the synthesizing unit 502 generates a frame based on the sample point phase change amount indicating the selected position of the sample point in the spatial direction decimation processing obtained from the decimation processing mode information for the block restored by the block expansion unit 501. In this configuration, the block positions are shifted and arranged, and a pixel value correction process for determining pixel gaps or pixel values of overlapping pixels generated by the block arrangement is executed.

  Finally, the processing of the data expansion unit 501 will be described again using the flowchart of FIG. FIG. 57 shows a flow from when the data expansion unit 501 performs an expansion process on a certain block and outputs the block to the synthesis unit 502.

  In step S201, thinning mode information R corresponding to the block to be extended and thinning control information S corresponding to the block are input. In step S202, by confirming the control information S for the block, the setting of the thinning process availability information for the block is confirmed. That is, it is checked whether or not the thinning process has been performed on the block. If S indicates that the thinning process is not possible, no thinning process is performed for the block, and the block is output to the synthesis unit 502 as it is (S208).

  If S indicates that thinning processing is possible, the process proceeds to S203, and the thinning mode information R for the block is referred to. If R indicates that the process to be performed on the block is space thinning, the process proceeds to step S204, where the space expansion process is performed on the block, and the expanded block is transferred to the synthesis unit 502 in step S207. Output. If R indicates that the processing applied to the block is not spatial decimation, the process proceeds to step S205, where time expansion processing is performed on the block, and the expanded block is output to the synthesis unit 502 in step S206. To do.

  In this way, the moving image restoration apparatus that executes the data expansion (decompression) process has thinning mode information: R given as attribute information corresponding to each block in the moving image conversion apparatus, and thinning control information corresponding to the block: Based on S, the presence / absence of execution of the extension process and the mode of the extension process are determined, and the data extension is executed.

[(4) Configuration of Moving Image Conversion Device that Performs Improved Thinning Process and Modified Example of Moving Image Restoration Device]
Next, the embodiment described above, ie,
(2) Configuration of Moving Image Conversion Device that Performs Improved Thinning Process (3) Configuration of Moving Image Restoration Device An embodiment in which the moving image conversion device 300 and the moving image restoration device 500 described in each of these items are modified will be described. To do.

  FIG. 58 shows a basic configuration of a moving image conversion apparatus 700 according to the present embodiment. The basic configuration of the moving image conversion apparatus 700 is the same as that of the moving image conversion apparatus 300 described above with reference to FIG. However, the processing of the thinning process execution unit 706 and the output unit 707 is different from that of the moving image conversion apparatus 300 described above. The processing contents of the other components 701 to 705 are the same as the corresponding components of the moving image conversion apparatus 300 described with reference to FIG.

  First, the process of the thinning process execution unit 706 will be described. The decimation process execution unit 706 outputs an original block that is not subjected to the decimation process for the block whose control information from the decimation process control information generation unit 705 is set to be non-thinning process, and the decimation process mode information generation unit 704. The thinning process according to the thinning process mode information is also executed, and the thinning process result is also output.

  The processing flow of the thinning process execution unit 706 will be described with reference to the flowchart shown in FIG. FIG. 59 shows a flow until the thinning process execution unit 706 finally applies some processing to a certain block. First, in step S301, thinning mode information R corresponding to a block to be extended and thinning control information S corresponding to a block are input. In step S302, by confirming the control information S for the block, it is checked whether the block is a block that can be thinned or not.

  If S indicates that thinning processing is not possible, block data that is not subjected to any thinning processing is output to the block (S308). Further, after that, the process proceeds to step S303, and the thinning process according to the thinning mode information: R is also executed. Even when S indicates that the thinning process is possible, the process proceeds to step S303, and the thinning mode information R for the block is referred to. That is, the thinning process after step S303 executes the thinning process according to the thinning mode information: R regardless of whether the thinning control information: S is acceptable or not.

  In step S303, the thinning mode information R for the block verifies whether or not the processing applied to the block is time thinning. If the thinning mode information R for the block indicates that it is spatial thinning, the block is subjected to spatial thinning in step S304, the result is output in step S307, and the process ends. If R indicates that the processing to be performed on the block is not spatial thinning, the process proceeds to step S305, the time thinning process is performed on the block, the result is output in step S306, and the processing is terminated.

  In this processing example, regardless of whether or not the thinning control information: S can be thinned out, thinning processing according to the thinning mode information: R is executed, and the thinning control information: S is thinned out. For blocks that cannot be processed, both original block data that has not been subjected to thinning processing and thinning processing result data that has been subjected to thinning processing in accordance with thinning mode information: R are generated.

Next, the output unit 707 will be described. The output unit 707 outputs the data supplied from the thinning process execution unit 706.
That is, for a block whose thinning process control information can be thinned, a thinning process result data with a reduced data amount and information regarding what processing has been performed on each block are output.
For blocks whose decimation processing control information cannot be decimation processing, the original block that has not been decimation processed, the block whose data amount after decimation processing has been reduced, and what processing has been performed on each block Output information.

  The information regarding the processing contents includes “information indicating whether or not thinning processing has been performed (thinning processing control information)”, “information indicating whether thinning processing has been performed in space or time, and thinning in the spatial direction. When processing is performed, information indicating which direction (horizontal / vertical) thinning processing has been performed, information indicating the sampling point phase change amount in thinning processing in the spatial direction (thinning processing mode information), However, other information may be added.

The difference between the moving image conversion apparatus 700 of the present embodiment and the moving image conversion apparatus 300 described above is that the output data when the thinning process control information corresponding to the block indicates “thinning process is impossible” It is in. The thinning process execution unit 306 of the moving image conversion apparatus 300 outputs only “the block data as it is without performing the thinning process” as a result of the process for the “thinning process impossible” block. On the other hand, in the thinning process execution unit 706 of the moving image conversion apparatus 700 of the present embodiment, when the thinning process control information corresponding to the block is “thinning process impossible”,
In addition to “block data as it is without decimation” (called the original block)
“Block data subjected to thinning processing” (referred to as thinning block)
Is also output.

  As described above, in the “thinning-incapable processing” block, both the original block and the thinning block are output, so that even if the original block does not exist, restoration by the moving image restoration apparatus 800 described below is performed. Since processing is possible, flexible data rate control is possible.

  Note that the output unit 707 selects only the original block that has not been subjected to the thinning process or only the block whose data amount has been reduced after the thinning process control information cannot be thinned. It is good also as a structure which performs control which selects and outputs. For example, when data is output to the network, if the available bandwidth is not enough, either the original block that has not been thinned or only the block whose data amount has been reduced after thinning By selecting and outputting, data output with reduced output data can be performed.

  The configuration of the moving image restoration apparatus 800 of this embodiment is shown in FIG. The basic configuration of the moving image restoration apparatus 800 has the same configuration as that of the moving image restoration apparatus 500 described above with reference to FIG. 58, but the processing in the data expansion unit 801 is different. The composition unit 802 is completely the same as the composition unit 502 of the moving image restoration apparatus 500.

  The data expansion unit 801 receives conversion (compression) data generated by the above-described moving image conversion apparatus 700 and attribute data necessary for restoration. The attribute data includes decimation process control information regarding each block and decimation process mode information regarding the specific contents of the process applied to each block. The specific contents of the processing applied to each block include information on whether the spatial direction thinning process or the time direction thinning process has been executed, the information on the sampling point phase change amount in the thinning process executed in the spatial thinning process, etc. It is information to include. Furthermore, the converted data corresponding to each block always includes “decimated block” data, and sometimes includes “original block” data at the same time.

  The data extension unit 801 converts either the space extension process or the time extension process into the conversion data to be restored based on the thinning process mode information and the thinning process control information included in the input attribute information. Or block data is sent directly to the combining unit 802.

  Specifically, when the thinning process control information corresponding to the block is “impossible to thin out” and the original data is included in the conversion data corresponding to the block, the block data that has not been subjected to the thinning process Therefore, the original data is directly output to the synthesis unit 802. If the decimation processing control information corresponding to the block is “thinning is possible” or “no thinning is not possible”, but the original block is not included in the conversion data corresponding to the block, the time direction or space direction Perform extension processing.

  When the processing performed on the block in the moving image conversion apparatus 700 by the thinning process execution unit 706 is the spatial thinning process, the data expansion unit 801 uses each block data thinned, the direction of spatial thinning, the sample Based on the decimation processing mode information indicating the point phase change amount, the space expansion process is performed and output to the synthesis unit 802. If the process performed by the thinning process execution unit 706 of the moving image conversion apparatus 700 is a time decimation process, a time extension process is performed and output to the synthesis unit 802.

  FIG. 61 shows a flow from when the data expansion unit 801 performs an expansion process to a certain block and outputs the block to the synthesis unit 802.

  In step S401, thinning mode information R corresponding to the block to be extended is input, and thinning control information S corresponding to the block are input. In step S402, by confirming the control information S for the block, the setting of the thinning process availability information for the block is confirmed. In the case where the thinning process availability information for the block is a block that is set not to be thinned, it is further verified in step S408 whether or not the original data is included in the conversion data corresponding to the block. If the original block is included, the process proceeds to step S410, the original block is output, and the process ends.

  If it is determined in step S408 that the original data is not included, the process proceeds to step S403, and the extension process according to the thinning mode information: R corresponding to the block is executed.

  If the control information: S for the block can be thinned, and if the control information: S cannot be thinned and it is determined in step S408 that the original data is not included, the process proceeds to step S403, and the block Refer to the thinning mode information R for. If R indicates that the processing to be performed on the block is space thinning, the process proceeds to step S404, where the block is subjected to space expansion processing, and the expanded block is transferred to the synthesis unit 802 in step S407. Output. If R indicates that the processing applied to the block is not spatial thinning, the process proceeds to step S405, where time expansion processing is performed on the block, and the expanded block is output to the synthesis unit 802 in step S406. To do.

  In this way, the moving image restoration apparatus that executes the data expansion (decompression) process has thinning mode information: R given as attribute information corresponding to each block in the moving image conversion apparatus, and thinning control information corresponding to the block: Based on S, the presence / absence of execution of the extension process and the mode of the extension process are determined, and the data extension is executed.

  By using the moving image conversion apparatus 700 and the moving image restoration apparatus 800 as described above, it becomes possible to control the amount of transmission data flexibly according to the bandwidth of the network, for example, by streaming moving images on the network. . Specifically, the moving image data whose data amount has been reduced by the moving image conversion apparatus 700 is stored in advance on a server on the network. When the client acquires a moving image from the server, if the network bandwidth is narrow, it notifies the server to that effect, and the server does not transmit or transmits the “original block” data when transmitting the moving image. Control such as limiting the amount of original blocks to be performed becomes possible.

  Even if the original block does not exist, the converted data by the moving image conversion device 700 can be restored by the moving image restoration device 800, so that the client can obtain complete frame data. If all the original blocks are transmitted when there is a margin in the network bandwidth, the client can obtain high-quality moving image data without image quality degradation as described with reference to FIGS. .

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the embodiment of the present invention, the configuration of the object included in the block configuring the moving image data and the optimum compression processing mode corresponding to the motion of the block are determined and determined. Since the data conversion process is performed in an optimum manner for each block area in accordance with the above-described manner, compression and decompression with very little quality deterioration can be realized.

  In addition, according to the configuration of the embodiment of the present invention, the configuration of the object included in the block constituting the moving image data and the motion of the block are analyzed. For example, when the block includes a plurality of different moving objects, thinning is performed. Since the setting is made so that the processing is not performed, the data conversion / restoration processing in which the deterioration of the image quality is suppressed is realized. Also, when the block movement amount fluctuates greatly in a short period of time, since the setting is made so that the thinning process is not performed, the data conversion and restoration process with suppressed image quality deterioration is realized.

  Furthermore, according to the configuration of the embodiment of the present invention, verification of the variance of the difference between the selected block of the current frame including the selected block as the verification target and the reference block determined as the movement destination of the selected block in the reference frame Or, based on the verification of the difference in the movement amount of the block at the corresponding position in the current frame including the selected block and the reference frame, or the verification of the difference in the movement amount of the surrounding blocks in the current frame including the selection block, etc. Since it is configured to perform the thinning process according to the determination by determining whether or not the process is executed, the data conversion and restoration process that suppresses the deterioration of the image quality is realized.

It is a figure which shows the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure which shows the detail of a detailed structure of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the detailed structure of the block process part in a moving image converter. It is a figure explaining the condition where the to-be-photographed object is not moving and the condition where it is moving with the speed v = 1 pixel / frame in the horizontal direction. It is a figure which shows the image quality evaluation curve as a graph which shows the tendency of a subject moving amount and the result of the subjective evaluation experiment of image quality. It is a figure explaining the response | compatibility with the setting of the virtual moving speed and image quality evaluation curve in the thinning process of a sample point movement type. It is a figure explaining the response | compatibility with the setting of the virtual moving speed and image quality evaluation curve in the thinning process of a sample point movement type. It is a figure explaining the example of determination of the thinning-out process mode determined based on the movement amount information from a movement amount detection part. It is a figure explaining the example of thinning-out processing of sample point fixation [sample point phase change amount = [0]]. It is a figure explaining the thinning-out process [sample point phase variation | change_quantity = [+ n]] which shifts a sample point position to a downward direction according to progress of a flame | frame. It is a figure explaining the thinning-out process [sample point phase variation | change_quantity = [-n]] which shifts a sample point position to an upward direction according to progress of a flame | frame. It is a figure explaining the example of a time direction thinning-out process. It is a figure explaining the aspect of the thinning process of the sample point fixed type and the thinning process of the sample point type when the thinning amount is m = 2. It is a figure explaining the aspect of the thinning process of the sample point fixed type and the thinning process of the sample point type when the thinning amount is m = 2. It is a figure explaining the aspect of the thinning process of the sample point fixed type and the thinning process of the sample point type when the thinning amount is m = 2. It is a figure explaining the aspect of the thinning process of the sample point fixed type and the thinning process of the sample point type when the thinning amount is m = 2. It is a figure explaining the aspect of the thinning process of the sample point fixed type and the thinning process of the sample point type when the thinning amount is m = 2. It is a figure explaining the aspect of the thinning process of the sample point fixed type and the thinning process of the sample point type when the thinning amount is m = 2. It is a figure explaining the example of a time direction thinning-out (4 frame-> 2 frame) process. It is a figure explaining the example in case a several different object with a different moving speed and moving direction exists. It is a figure explaining the example in case a several different object with a different moving speed and moving direction exists. It is a figure explaining the example in case a several different object with a different moving speed and moving direction exists. It is a figure explaining the structural example of the moving image converter according to one Example of this invention. It is a figure which shows the image quality evaluation curve at the time of carrying out a visual observation after decompress | restoring and reproducing | regenerating and reproducing | regenerating and reproducing | regenerating the conversion image data produced | generated by performing the thinning process of the number of pixels of the spatial direction which set sample point phase change amount = 0. It is a figure which shows the image quality evaluation curve in case sample point phase variation | change_quantity = + 1. It is a figure which shows the image quality evaluation curve in case sampling point phase variation | change_quantity = + 2. It is a figure which shows the image quality evaluation curve in case sampling point phase variation | change_quantity = + 3. It is the figure which showed all the image quality evaluation curves of FIG. 27 thru | or FIG. It is a figure explaining the method of determining whether the thinning-out process with respect to a block is performed using dispersion | distribution of the difference between blocks. It is a figure explaining the method of determining whether the thinning-out process with respect to a block is performed based on the detection of the thinning-out process fluctuation | variation of a frame. It is a figure explaining the method of determining whether the thinning-out process with respect to a block is performed based on the detection of the thinning-out process fluctuation | variation of a surrounding block. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure explaining the process example of the pixel count thinning of the spatial direction which a thinning process execution part performs. It is a figure which shows the flowchart explaining the process which a thinning process execution part performs. It is a figure which shows the structural example of a moving image decompression | restoration apparatus. It is a figure explaining the process by the data expansion part of a moving image decompression | restoration apparatus at the time of performing the space thinning process of a horizontal direction. It is a figure explaining the process by the data expansion part of a moving image decompression | restoration apparatus at the time of performing the space thinning process of a perpendicular direction. It is a figure explaining the process by the data expansion part of a moving image decompression | restoration apparatus at the time of performing the thinning process of a time direction. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process performed with a moving image decompression | restoration apparatus. It is a figure explaining the block arrangement | positioning process and correction | amendment process which are performed with a moving image decompression | restoration apparatus. It is a figure which shows the flowchart explaining the process of a data expansion part. It is a figure which shows the structural example of a moving image converter. It is a figure which shows the flowchart explaining the process which a thinning process execution part performs. It is a figure which shows the structural example of a moving image decompression | restoration apparatus. It is a figure which shows the flowchart explaining the process of a data expansion part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Moving image converter 110 Block division part 120 Movement amount detection part 130 Block processing part 140 Output part 111 Image storage part 112 Block division part 121 Movement amount detection part 122 Block distribution part 131-133 Block processing part 151 Thinning process mode determination part 152 Thinning Process Execution Unit 201 Subject Area 205 Image Quality Evaluation Curve 231, 232 Object 233 Block 241, 242 Object 243 Block 251, 252 Object 253 Rectangular Area 254 Block 300 Moving Image Converter 301 Frame Buffer 302 Block Dividing Unit 303 Movement Amount Detection Unit 303 Thinning Process Execution Unit 304 Thinning Process Mode Information Generation Unit 305 Thinning Process Control Information Generation Unit 306 Thinning Process Execution Unit 307 Output Unit 401 Image Quality Evaluation Curve 4 2 reference evaluation value 403 to 405 high image quality movement amount region 411 image quality evaluation curve 412 reference evaluation value 413 to 416 high image quality movement amount region 421 image quality evaluation curve 422 reference evaluation value 423 to 425 high image quality movement amount region 431 image quality evaluation curve 432 reference Evaluation value 433 to 436 High image quality moving amount area 500 Moving image restoration device 501 Data expansion unit 502 Composition unit 601 Block 700 Moving image conversion device 701 Frame buffer 702 Block division unit 703 Movement amount detection unit 703 Thinning processing execution unit 704 Thinning processing mode Information generation unit 705 Thinning-out process control information generation unit 706 Thinning-out processing execution unit 707 Output unit 800 Moving image restoration apparatus 801 Data expansion unit 802 Composition unit

Claims (32)

A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit that detects a movement amount of a subject included in each block divided by the block division unit as a block movement amount;
Based on the block movement amount detected by the movement amount detection unit, a thinning processing mode information generation unit that generates thinning processing mode information indicating a thinning processing mode to be applied to the block;
Based on the block movement amount detected by the movement amount detection unit, a thinning process control information generation unit that generates thinning process control information indicating whether or not to perform a thinning process on a block;
Each block divided by the block dividing unit, the thinning processing mode information generated by the thinning processing mode information generation unit, and the thinning processing control information generated by the thinning processing control information generation unit are input to the input information. A thinning process execution unit that determines whether or not to perform a thinning process for each input block and a thinning process mode, and performs a process according to the determination;
A moving image conversion apparatus comprising: The thinning process execution unit
When the thinning process control information generated by the thinning process control information generation unit indicates that the thinning process should be executed on the block, the thinning process mode information generated by the thinning process mode information generation unit is followed. Execute the thinning process,
When the thinning process control information generated by the thinning process control information generating unit indicates that the thinning process should not be performed on the block, the original block data is output without performing the thinning process. The moving image conversion apparatus according to claim 1. The thinning process execution unit
When the thinning process control information generated by the thinning process control information generation unit indicates that the thinning process should be executed on the block, the thinning process mode information generated by the thinning process mode information generation unit is followed. Execute the thinning process,
Even when the decimation process control information generated by the decimation process control information generation unit indicates that the decimation process should not be performed on the block, the decimation process mode information generated by the decimation process mode information generation unit is followed. The moving image conversion apparatus according to claim 1, wherein the thinning process is executed, and the result of the thinning process and original block data that is not subjected to the thinning process are output. The thinning processing mode information generation unit
Based on the block movement amount detected by the movement amount detection unit, as a thinning processing mode to be applied to the block,
2. The moving image conversion apparatus according to claim 1, wherein a thinning processing mode of at least one of a thinning process in a spatial direction, a thinning process in a time direction, or a composite thinning process is determined. . The thinning processing mode information generation unit
When it is determined to perform the thinning process in the spatial direction as a thinning process mode to be applied to the block based on the block movement amount detected by the movement amount detection unit, the selection position of the sample point in the spatial direction thinning process is further determined. The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to execute a process of determining a sample point phase change amount to be indicated. The thinning process control information generation unit
Based on the block movement amount detected by the movement amount detection unit, all blocks corresponding to a specific frame are generated regardless of the block movement amount in the generation of the thinning process control information indicating whether or not to perform the thinning process on the block. 2. The configuration according to claim 1, wherein the thinning-out process control information indicating whether or not the thinning-out process is to be executed is generated as uniform control information indicating that the thinning-out process is possible or not. Video conversion device. The thinning process control information generation unit
It is configured to verify whether or not a plurality of objects having different moving speeds are included in one block, and to generate control information that disables thinning processing for a block including a plurality of objects. Item 4. The moving image conversion device according to Item 1. The thinning process control information generation unit
Calculating a variance of a difference between a selected block of a current frame including a block selected as a generation target of control information and a reference block determined as a movement destination of the selected block in a reference frame different from the current frame; The moving image conversion apparatus according to claim 1, wherein the thinning-out process control information is generated based on a comparison result between a calculated value of the variance and a predetermined threshold value. The thinning process control information generation unit
The thinning-out process control information is generated based on a difference in movement amount between a current frame including a block selected as a control information generation target and a block at a corresponding position in a reference frame. 2. The moving image conversion apparatus according to 1. The thinning process control information generation unit
2. The configuration according to claim 1, wherein the thinning-out process control information is generated based on a difference in movement amount of blocks around the block in a current frame including a block selected as a control information generation target. Video conversion device. A moving image restoration apparatus that performs a restoration process of moving image conversion data,
Based on the input information, block correspondence data constituting moving image conversion data, decimation processing mode information indicating a decimation processing mode for the block, and decimation processing control information indicating whether or not to perform decimation processing for the block are input. A block extension unit that executes an extension process of block-compatible conversion data; and
A synthesis unit that synthesizes the blocks restored by the block extension process in the block extension unit and generates frame data;
The block extension is
Based on the thinning-out process control information, it is determined whether or not to perform the thinning-out process data expansion process for the block, and the thinning-out processing mode indicated by the thinning-out processing mode information for the block corresponding conversion data determined to be expanded. A moving image restoration apparatus characterized in that an extension process corresponding to the above is executed. The thinning processing mode information is
Information indicating a thinning processing mode of at least one of thinning processing in the spatial direction, thinning processing in the time direction, or complex thinning processing,
The block extension is
One of the thinning-out processing in the spatial direction indicated by the thinning-out processing mode information, the thinning-out processing in the time direction, or an extended processing corresponding to at least one of the thinning-out processing in combination is executed. The moving image restoration apparatus according to claim 11. The block extension is
Based on the thinning-out process control information, it is determined whether or not to extend the thinning-out process data for the block, and for the block that is determined not to be extended, the block corresponding original data included as the block corresponding data is combined. The moving image restoration apparatus according to claim 11, wherein the moving image restoration apparatus is configured to execute processing to be output to a unit. The synthesis unit is
The blocks restored by the block extension unit are arranged by shifting the block position according to the progress of the frame based on the sample point phase change amount indicating the selected position of the sample point in the spatial direction thinning process obtained from the thinning process mode information. The moving image restoration apparatus according to claim 11, wherein the moving image restoration apparatus is configured to execute processing. The synthesis unit is
The blocks restored by the block extension unit are arranged by shifting the block position according to the progress of the frame based on the sample point phase change amount indicating the selected position of the sample point in the spatial direction thinning process obtained from the thinning process mode information. The moving image restoration apparatus according to claim 11, wherein the moving image restoration apparatus is configured to execute a process and to execute a pixel value correction process for determining a pixel value of a pixel gap or an overlapping pixel generated by the block arrangement. A moving image conversion method for executing data conversion processing of moving image data in a moving image conversion apparatus,
In the block dividing unit, a block dividing step for executing a block dividing process for each frame constituting the moving image data;
In a movement amount detection unit, a movement amount detection step of detecting a movement amount of a subject included in each block divided in the block division step as a block movement amount;
In the thinning processing mode information generation unit, based on the block movement amount detected in the movement amount detection step, a thinning processing mode information generation step that generates thinning processing mode information indicating a thinning processing mode to be applied to the block;
In the thinning process control information generating unit, a thinning process control information generating step for generating thinning process control information indicating whether or not to perform the thinning process on the block based on the block moving amount detected in the moving amount detecting step; ,
In the thinning processing execution unit, each block divided in the block dividing step, the thinning processing mode information generated in the thinning processing mode information generation step, and the thinning processing control information generated in the thinning processing control information generation step Input, based on the input information, determine whether or not to perform a thinning process for each input block and a thinning process mode, and perform a thinning process execution step for performing processing according to the determination;
A moving image conversion method characterized by comprising: The thinning process execution step includes:
If the decimation process control information generated in the decimation process control information generation step indicates that the decimation process for the block should be executed, the decimation process mode information generated by the decimation process mode information generation unit is followed. Execute the thinning process,
When the thinning process control information generated in the thinning process control information generation step indicates that the thinning process should not be performed on the block, the original block data is output without performing the thinning process. The moving image conversion method according to claim 16. The thinning process execution step includes:
If the decimation process control information generated in the decimation process control information generation step indicates that the decimation process for the block should be executed, the decimation process mode information generated by the decimation process mode information generation unit is followed. Execute the thinning process,
Even when the decimation process control information generated in the decimation process control information generation step indicates that the decimation process for the block should not be executed, it follows the decimation process mode information generated by the decimation process mode information generation unit. 17. The moving image conversion method according to claim 16, wherein the thinning process is executed and the result of the thinning process and the original block data not subjected to the thinning process are output. The thinning processing mode information generation step includes:
Based on the block movement amount detected in the movement amount detection step, as a thinning processing mode to be applied to the block,
17. The moving image conversion method according to claim 16, wherein a thinning processing mode of at least one of a thinning process in a spatial direction, a thinning process in a time direction, or a complex thinning process is determined. The thinning processing mode information generation step includes:
Based on the block movement amount detected in the movement amount detection step, as a thinning process mode to be applied to the block, when determining to perform the thinning process in the spatial direction, the selection position of the sample point in the spatial direction thinning process is further determined. The moving image conversion method according to claim 16, wherein a process of determining a sample point phase change amount to be indicated is executed. The thinning process control information generation step includes:
Based on the block movement amount detected in the movement amount detection step, all the blocks corresponding to the specific frame are generated regardless of the block movement amount in the generation of the thinning process control information indicating whether or not to perform the thinning process on the block. 17. The moving image conversion according to claim 16, wherein a process of generating the thinning-out process control information indicating whether or not to perform the thinning-out process on the video as uniform control information is performed so that the thinning-out process is possible or not. Method. The thinning process control information generation step includes:
It is a step of verifying whether or not a plurality of objects having different moving speeds are included in one block, and generating control information that disables thinning processing for blocks including the plurality of objects. Item 17. The moving image conversion method according to Item 16. The thinning process control information generation step includes:
Calculating a variance of a difference between a selected block of a current frame including a block selected as a generation target of control information and a reference block determined as a movement destination of the selected block in a reference frame different from the current frame; The moving image conversion method according to claim 16, wherein the thinning process control information is generated based on a comparison result between a calculated value of the variance and a predetermined threshold value. The thinning process control information generation step includes:
The step of generating the thinning-out process control information based on a difference in movement amount of a block at a corresponding position in a current frame including a block selected as a control information generation target and a reference frame. 16. The moving image conversion method according to 16. The thinning process control information generation step includes:
17. The step of generating the thinning-out process control information based on a difference in movement amount of blocks surrounding the block in a current frame including a block selected as a control information generation target. Video conversion method. In the moving image restoration apparatus, a moving image restoration method for executing restoration processing of moving image conversion data,
The block extension unit inputs block correspondence data constituting moving image conversion data, decimation processing mode information indicating a decimation processing mode for the block, and decimation processing control information indicating whether or not to perform decimation processing for the block. A block expansion step for performing an expansion process of the block-compatible conversion data based on the input information;
A combining unit that combines the blocks restored by the block extension process in the block extension unit to generate frame data;
The block expansion step includes:
Based on the thinning-out process control information, it is determined whether or not to perform the thinning-out process data expansion process for the block, and the thinning-out processing mode indicated by the thinning-out processing mode information for the block corresponding conversion data determined to be expanded. A moving image restoration method, which is a step of executing an expansion process corresponding to the above. The thinning processing mode information is
Information indicating a thinning processing mode of at least one of thinning processing in the spatial direction, thinning processing in the time direction, or complex thinning processing,
The block expansion step includes:
It is a step of executing an expansion process corresponding to at least one of the thinning process in the spatial direction indicated by the thinning process mode information, the thinning process in the time direction, or at least one of the complex thinning processes. The moving image restoration method according to claim 26. The block expansion step includes:
Based on the thinning-out process control information, it is determined whether or not to extend the thinning-out process data for the block, and for the block that is determined not to be extended, the block corresponding original data included as the block corresponding data is combined. 27. The moving image restoration method according to claim 26, wherein the moving image restoration process is a step of executing a process to be output to the unit. The synthesis step includes
The blocks restored by the block extension unit are arranged by shifting the block position according to the progress of the frame based on the sample point phase change amount indicating the selected position of the sample point in the spatial direction thinning process obtained from the thinning process mode information. 27. The moving image restoration method according to claim 26, which is a step of executing processing. The synthesis step includes
The blocks restored by the block extension unit are arranged by shifting the block position according to the progress of the frame based on the sample point phase change amount indicating the selected position of the sample point in the spatial direction thinning process obtained from the thinning process mode information. 27. The moving image restoration method according to claim 26, wherein the moving image restoration method is a step of executing a pixel value correction process for determining a pixel value of a pixel gap generated by block arrangement or an overlapping pixel while executing the process. In the moving image conversion apparatus, a computer program for executing data conversion processing of moving image data,
In the block dividing unit, a block dividing step for executing a block dividing process for each frame constituting the moving image data;
In a movement amount detection unit, a movement amount detection step for detecting a movement amount of a subject included in each block divided in the block division step as a block movement amount;
In the thinning processing mode information generation unit, a thinning processing mode information generation step for generating thinning processing mode information indicating a thinning processing mode to be applied to the block based on the block movement amount detected in the movement amount detection step;
A decimation process control information generation unit for generating decimation process control information indicating whether or not to perform a decimation process on a block based on the block movement amount detected in the movement amount detection step; ,
In the thinning processing execution unit, each block divided in the block dividing step, the thinning processing mode information generated in the thinning processing mode information generation step, and the thinning processing control information generated in the thinning processing control information generation step A decimation process execution step for determining whether to perform decimation processing for each input block and a decimation process mode based on the input information, and performing a process according to the determination;
A computer program for executing In the moving image restoration device, a computer program for executing restoration processing of moving image conversion data,
The block extension unit inputs block correspondence data constituting moving image conversion data, decimation processing mode information indicating a decimation processing mode for the block, and decimation processing control information indicating whether or not to perform decimation processing for the block. , A block expansion step for executing the expansion processing of the block corresponding conversion data based on the input information,
A combining unit that combines the blocks restored by the block extension process in the block extension unit to generate frame data;
The block expansion step includes:
Based on the thinning-out process control information, it is determined whether or not to perform the thinning-out process data expansion process for the block, and the thinning-out processing mode indicated by the thinning-out processing mode information for the block corresponding conversion data determined to be expanded. A computer program characterized in that it is a step of executing an expansion process corresponding to.