JP4590975B2 - Moving picture conversion apparatus, moving picture restoration apparatus and method, and computer program - Google Patents

Description

  The present invention relates to a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program. In particular, in a configuration that performs compression and decompression processing of moving images, by appropriately compressing and decompressing using human visual characteristics, it is possible to efficiently reduce the amount of data and minimize image quality degradation. The present invention relates to a moving picture conversion apparatus, a moving picture restoration apparatus and method, and a computer program that can realize high-quality image restoration.

  The moving image data is subjected to data conversion, that is, compression processing for reducing the amount of data when stored in a storage medium such as a flash memory, a hard disk, or a DVD, or distributed via a network. In particular, in recent years, the quality of moving image data has been improved, for example, HD (High Definition) data has been improved, and the amount of data has increased rapidly with the improvement of the quality of this data. Under such circumstances, many studies and studies have been made on techniques for improving the compression efficiency in the compression and decompression processing of moving image data and preventing the quality degradation of the decompressed data.

  A typical method for compressing the data amount of moving images is moving image encoding that combines inter-frame encoding such as MPEG and intra-frame encoding. By the way, in order to improve the coding efficiency of moving picture coding by MPEG, many techniques for pre-processing moving pictures have been proposed.

  For example, Patent Document 1 can improve coding efficiency while suppressing deterioration in image quality by compressing after converting into a moving image of an interlaced scanning method by thinning out half of the pixels of a progressive scanning method moving image. The structure which was made is disclosed. Further, Patent Document 2 discloses a configuration in which generation of block distortion after encoding is suppressed by applying a low-pass filter to an image as preprocessing.

However, the data compression methods disclosed in these conventional techniques have the following problems.
(1) In the method disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2000-32470), the amount of data can be halved at the end of preprocessing, and a significant improvement in final coding efficiency can be expected. Since uniform thinning is performed without using the local characteristics of the moving image, that is, the features of the moving area and the non-moving area, the entire image is deteriorated uniformly.
(2) The method of Patent Document 2 (Japanese Patent Laid-Open No. 2001-352546) is intended to suppress block distortion that occurs particularly when encoding is performed at a low bit rate, and this method is used when encoding at a high bit rate. By applying this, it is not always possible to improve the encoding efficiency and suppress the deterioration of the image quality.
JP 2000-32470 A JP 2001-352546 A

  As described above, some conventional techniques disclose configurations that improve the encoding efficiency while suppressing image degradation. However, the compression methods disclosed so far restore compressed image data. However, image quality deterioration when reproduced is not sufficiently suppressed.

  The present invention has been made in view of the above problems, and by performing appropriate compression and decompression utilizing human visual characteristics, it is possible to efficiently reduce the amount of data and minimize image quality degradation. It is an object of the present invention to provide a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program that can suppress and realize high-quality image restoration.

ただしｔは時間、Ｔはフレーム間隔、
上記式（式ａ），（式ｂ）で示す超解像効果発生条件を満足する間引き態様を前記移動量情報に基づいて決定し、決定した間引き態様に一致するように間引き対象フレームを変更して間引き処理を行なう構成であることを特徴とする動画像変換装置にある。 The first aspect of the present invention is:
A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit for detecting a subject movement amount in each block divided by the block division unit;
A block processing unit that inputs the block data divided by the block dividing unit and the movement amount information detected by the movement amount detection unit, and executes block data thinning processing;
The block processing unit
The block to be thinned out is compared with the movement amount detected by the movement amount detection unit and the first threshold value and the second threshold value that are defined in advance to determine the processing mode.
If the detected amount of movement is greater than or equal to the first threshold, the amount of movement is large,
If the detected amount of movement is less than the first threshold and greater than or equal to the second threshold, the amount of movement is medium,
If the detected amount of movement is less than the second threshold, the amount of movement is small,
And change the thinning mode to be executed according to whether the movement amount belongs to the large, medium, or small classification,
When the amount of movement is large, spatial thinning processing,
When the amount of movement is small, thinning process in the time direction,
When the amount of movement is medium, execute a space thinning process and a time direction thinning process,
In the time direction thinning process when the movement amount is medium, the thinning process is performed by changing the thinning target frame based on the movement amount information.
The block processing unit
In the execution of the time direction thinning process, the thinning process is performed by changing the thinning target frame based on the movement amount information.
Formula defining the pre-defined super-resolution effect generation condition , that is, the following formula specifying the super-resolution effect generation condition when an object moving at a speed of v pixels / frame is thinned out in the spatial direction by m pixels,

Where t is time, T is frame interval,
A thinning mode that satisfies the super-resolution effect generation condition shown in the above formulas (formula a) and (formula b) is determined based on the movement amount information, and the thinning target frame is changed so as to match the determined thinning mode. The moving image conversion apparatus is characterized in that the thinning process is performed.

Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the block processing unit performs the time-direction thinning process that is executed when the movement amount whose movement amount is less than a first threshold is medium or small . A third threshold that is a movement amount threshold that is a thinning mode change point set according to a correspondence relationship between a movement amount that satisfies the super-resolution effect generation condition and a thinning mode, and a third threshold that is a threshold value less than the first threshold and the movement amount When the movement amount information is equal to or greater than the third threshold value , the thinning target frames that are continuous on the time axis are compared with the case where the movement amount information is less than the third threshold value. When the movement amount information is less than the third threshold value , the number of thinning target frames continuous on the time axis is set larger than when the movement amount information is greater than or equal to the third threshold value. Axial Characterized in that it is configured to perform a decimation process.

Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, when the block processing unit is configured to execute a temporal direction thinning process with a compression rate of 1/2 in units of four consecutive frames, the movement amount information Is equal to or greater than the third threshold value , the first frame and the third frame are left, the second and fourth frames are thinned out, and a time-axis direction thinning process is performed . If it is less than the threshold value , the first frame and the fourth frame are left, and the thinning process in the time axis direction for thinning out the second and third frames is performed.

ただしｔは時間、Ｔはフレーム間隔、
上記式（式ａ），（式ｂ）で示す超解像効果発生条件を満足するように前記移動量情報に基づいて間引き対象フレームを変更して間引き処理がなされたブロックであり、
前記ブロック補間部は、
前記移動量情報に基づく間引き処理によって間引かれたフレームを復元するように、復元対象として入力されたブロックの時間方向の間引き処理の態様に応じて、フレーム間での復元ブロックの移動方向を、より時間的に近いフレーム間移動となるように選択して、動画像データ復元を行なう構成であることを特徴とする動画像復元装置にある。 Furthermore, the second aspect of the present invention provides
A moving image restoration device that executes restoration processing of moving image conversion data including thinning processing data in the time direction,
Block interpolation that inputs moving image conversion data that has been thinned in the time direction in units of blocks, calculates pixel values of multiple blocks corresponding to frames in different time axis directions, and generates a restored block of multiple frames Part
The block interpolation unit
The input block of frames not decimated time, to generate a copy and restore block frame decimation more temporally close in time, moves according to the movement amount information input block corresponding, the only movement amount restored block between frames And moving image data restoration,
The input block to be restored is a block that is subjected to thinning processing by changing the thinning target frame based on the movement amount information, and is an expression that defines the super-resolution effect generation condition, that is, the speed v pixel / frame. The following formula specifying the super-resolution effect generation condition when moving objects are thinned out in the spatial direction with m pixels,

Where t is time, T is frame interval,
It is a block that has been subjected to thinning processing by changing the thinning target frame based on the movement amount information so as to satisfy the super-resolution effect generation condition represented by the above formulas (formula a) and (formula b),
The block interpolation unit
In order to restore the frame thinned out by the thinning process based on the movement amount information, the moving direction of the restored block between the frames is changed according to the mode of the thinning process in the time direction of the block input as the restoration target. A moving image restoration apparatus is characterized in that moving image data restoration is performed by selecting so that movement between frames is closer in time.

  Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the block interpolation unit has a configuration in which a band limiting process by a filter process is performed on a block on which an interpolation with a movement is performed.

Furthermore, in an embodiment of the moving image restoration apparatus of the present invention, the moving image restoration device determines whether or not the block is data that has been thinned out in the time direction based on the movement amount of the restoration target block. The block interpolating unit moves the restoration block by the amount of movement between the frames according to the movement amount information for the block that has been thinned out in the time direction selected based on the determination, thereby restoring moving image data. It is the structure to perform.

Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the moving image restoration apparatus is configured to reduce the block to be restored in the time direction based on a flag indicating whether or not the data has been thinned in the time direction. The block interpolation unit reconstructs the block that has been thinned in the time direction selected based on the determination by the amount of movement between frames according to the amount of movement information. The feature is that the moving image data is restored by moving the block>

Furthermore, in an embodiment of the moving image restoration apparatus of the present invention, the moving image restoration device has a movement amount detection unit, and the block is thinned out in the time direction according to the movement amount detected by the movement amount detection unit. The block interpolation unit determines whether or not the data is a restored block by the amount of movement between frames according to the amount of movement of the block that has been thinned in the time direction selected based on the determination. The moving image data is restored by moving the image data.

Furthermore, in an embodiment of the moving image restoration apparatus of the present invention, the movement amount detection unit performs a process of detecting a motion vector based on a corresponding block of a different frame, and the block interpolation unit is based on the motion vector. Thus, the moving image data is restored by moving the restoration block by the amount of movement between frames.

ただしｔは時間、Ｔはフレーム間隔、
上記式（式ａ），（式ｂ）で示す超解像効果発生条件を満足する間引き態様を前記移動量情報に基づいて決定し、決定した間引き態様に一致するように間引き対象フレームを変更して間引き処理を行なうことを特徴とする動画像変換方法にある。 Furthermore, the third aspect of the present invention provides
A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data divided in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Compare the movement amount detected in the movement amount detection step with the first threshold value and the second threshold value defined in advance for determining the processing mode, as the block to be thinned out,
If the detected amount of movement is greater than or equal to the first threshold, the amount of movement is large,
If the detected amount of movement is less than the first threshold and greater than or equal to the second threshold, the amount of movement is medium,
If the detected amount of movement is less than the second threshold, the amount of movement is small,
And change the thinning mode to be executed according to whether the movement amount belongs to the large, medium, or small classification,
When the amount of movement is large, spatial thinning processing,
When the amount of movement is small, thinning process in the time direction,
When the amount of movement is medium, execute a space thinning process and a time direction thinning process,
In the time direction decimation process when the movement amount is medium, the decimation process is performed by changing the decimation target frame based on the movement amount information,
In the execution of the time direction thinning process, as a thinning process in which the thinning target frame is changed based on the movement amount information,
Formula defining the pre-defined super-resolution effect generation condition, that is, the following formula specifying the super-resolution effect generation condition when an object moving at a speed of v pixels / frame is thinned out in the spatial direction by m pixels,

Where t is time, T is frame interval,
A thinning mode that satisfies the super-resolution effect generation condition shown in the above formulas (formula a) and (formula b) is determined based on the movement amount information, and the thinning target frame is changed so as to match the determined thinning mode. The moving image conversion method is characterized by performing a thinning process.

Furthermore, in one embodiment of the moving image conversion method of the present invention, the block processing step includes performing a time direction thinning process that is executed when the movement amount is less than or equal to a first threshold and the movement amount is medium or small . A third threshold that is a movement amount threshold that is a thinning mode change point set according to a correspondence relationship between a movement amount that satisfies the super-resolution effect generation condition and a thinning mode, and a third threshold that is a threshold value less than the first threshold and the movement amount When the movement amount information is equal to or greater than the third threshold value , the thinning target frames that are continuous on the time axis are compared with the case where the movement amount information is less than the third threshold value. less set, in case the moving amount information is less than the third threshold, the thinning-out object consecutive frames in the time axis, the movement amount information is set larger as compared with the case where the third threshold value or more values And executes thinning processing between axis.

Furthermore, in an embodiment of the moving image conversion method of the present invention, in the block processing step, when the time direction thinning process with a compression rate of 1/2 is executed in units of four consecutive frames, the movement amount information is the first value . If it is 3 thresholds or more, the first frame and the third frame are left, the time axis direction thinning process is performed to thin out the second and fourth frames, and the movement amount information is less than the third threshold . In some cases, the first frame and the fourth frame are left and the thinning process in the time axis direction is performed to thin out the second and third frames.

ただしｔは時間、Ｔはフレーム間隔、
上記式（式ａ），（式ｂ）で示す超解像効果発生条件を満足するように前記移動量情報に基づいて間引き対象フレームを変更して間引き処理がなされたブロックであり、
前記ブロック補間ステップは、
前記移動量情報に基づく間引き処理によって間引かれたフレームを復元するように、復元対象として入力されたブロックの時間方向の間引き処理の態様に応じて、フレーム間での復元ブロックの移動方向を、より時間的に近いフレーム間移動となるように選択して、動画像データ復元を行なうステップであることを特徴とする動画像復元方法にある。 Furthermore, the fourth aspect of the present invention provides
A moving image restoration method for executing restoration processing of moving image conversion data including thinning processing data in the time direction,
Block interpolation that inputs moving image conversion data that has been thinned in the time direction in units of blocks, calculates pixel values of multiple blocks corresponding to frames in different time axis directions, and generates a restored block of multiple frames Has steps,
The block interpolation step includes:
The input block of frames not decimated time, to generate a copy and restore block frame decimation more temporally close in time, moves according to the movement amount information input block corresponding, the only movement amount restored block between frames And restoring the moving image data,
The input block to be restored is a block that is subjected to thinning processing by changing the thinning target frame based on the movement amount information, and is an expression that defines the super-resolution effect generation condition, that is, the speed v pixel / frame. The following formula specifying the super-resolution effect generation condition when moving objects are thinned out in the spatial direction with m pixels,

Where t is time, T is frame interval,
It is a block that has been subjected to thinning processing by changing the thinning target frame based on the movement amount information so as to satisfy the super-resolution effect generation condition represented by the above formulas (formula a) and (formula b),
The block interpolation step includes:
In order to restore the frame thinned out by the thinning process based on the movement amount information, the moving direction of the restored block between the frames is changed according to the mode of the thinning process in the time direction of the block input as the restoration target. In the moving image restoration method, the moving image data restoration is performed by selecting the movement between frames closer to each other in time.

  Furthermore, in an embodiment of the moving image restoration method of the present invention, the block interpolation step includes a step of executing a band limiting process by a filter process for a block on which an interpolation with a movement has been executed.

Furthermore, in an embodiment of the moving image restoration method of the present invention, the moving image restoration method further determines whether or not the block is data that has been thinned out in the time direction based on the movement amount of the restoration target block. The block interpolation step moves the restoration block by the amount of movement between frames according to the amount of movement information for the block that has been thinned out in the time direction selected based on the determination. , Moving image data restoration is performed.

Furthermore, in an embodiment of the moving image restoration method of the present invention, the moving image restoration method further includes: the restoration target block is set in the time direction based on a flag indicating whether or not the data is thinned out in the time direction. Determining whether or not the data has been thinned out, and the block interpolation step is performed between the frames according to the movement amount information for the block that has been thinned in the time direction selected based on the determination. The moving image data is restored by moving the restoration block by the movement amount .

Furthermore, in one embodiment of the moving image restoration method of the present invention, the moving image restoration method further includes a movement amount detection step, and the block is moved in the time direction according to the movement amount detected in the movement amount detection step. it is determined whether the thinning is made data, the block interpolation step, for a block in which the thinning of the selected time direction based on the determination has been made, only the amount of movement between frames in accordance with the moving amount Moving image data is restored by moving the restoration block.

Furthermore, in one embodiment of the moving image restoration method of the present invention, the movement amount detecting step is a step of executing a process of detecting a motion vector based on a corresponding block of a different frame, and the block interpolation step is the motion interpolation step. Based on the vector, moving image data is restored by moving the restoration block by the movement amount between frames.

ただしｔは時間、Ｔはフレーム間隔、
上記式（式ａ），（式ｂ）で示す超解像効果発生条件を満足する間引き態様を前記移動量情報に基づいて決定し、決定した間引き態様に一致するように間引き対象フレームを変更して間引き処理を行なわせることを特徴とするコンピュータ・プログラムにある。 Furthermore, the fifth aspect of the present invention provides
In the moving image conversion device, a computer program for executing a data conversion process of moving image data, the data conversion process ,
A block dividing step for causing the block dividing unit to execute a block dividing process for each frame constituting the moving image data;
A movement amount detection step for causing a movement amount detection unit to detect a subject movement amount in each block divided in the block division step;
The block processor, the block data obtained by dividing in the block dividing step, the moving amount detecting inputs the movement amount information detected in step, the moving image conversion apparatus and a block processing step for executing the decimation process of the block data To run
In the block processing step ,
The block to be thinned out is compared with the movement amount detected in the movement amount detection step with the first threshold value and the second threshold value that are defined in advance to determine the processing mode,
If the detected amount of movement is greater than or equal to the first threshold, the amount of movement is large,
If the detected amount of movement is less than the first threshold and greater than or equal to the second threshold, the amount of movement is medium,
If the detected amount of movement is less than the second threshold, the amount of movement is small,
And changing the thinning mode to be executed depending on whether the movement amount belongs to the large, medium, or small classification,
When the amount of movement is large, spatial thinning processing,
When the amount of movement is small, thinning process in the time direction,
When the movement amount is medium, the space thinning process and the time direction thinning process are executed,
In the time direction thinning process when the movement amount is medium, the thinning target frame is changed based on the movement amount information, and the thinning process is performed.
In the execution of the time direction thinning process, as a thinning process in which the thinning target frame is changed based on the movement amount information,
Formula defining the pre-defined super-resolution effect generation condition, that is, the following formula specifying the super-resolution effect generation condition when an object moving at a speed of v pixels / frame is thinned out in the spatial direction by m pixels,

Where t is time, T is frame interval,
A thinning mode that satisfies the super-resolution effect generation condition shown in the above formulas (formula a) and (formula b) is determined based on the movement amount information, and the thinning target frame is changed so as to match the determined thinning mode. in a computer program, characterized in that to perform thinning processing Te.

ただしｔは時間、Ｔはフレーム間隔、
上記式（式ａ），（式ｂ）で示す超解像効果発生条件を満足するように前記移動量情報に基づいて間引き対象フレームを変更して間引き処理がなされたブロックであり、
前記ブロック補間ステップにおいては、
前記移動量情報に基づく間引き処理によって間引かれたフレームを復元するように、復元対象として入力されたブロックの時間方向の間引き処理の態様に応じて、フレーム間での復元ブロックの移動方向を、より時間的に近いフレーム間移動となるように選択して、動画像データ復元を行なわせることを特徴とするコンピュータ・プログラムにある。 Furthermore, the sixth aspect of the present invention provides
In the moving image restoration apparatus, the computer program for executing the restoration processing of the moving image conversion data including the thinning processing data in the time direction, the restoration processing of the moving image data,
The moving image conversion data that has been thinned out in the time direction is input to the block interpolator in units of blocks, and pixel values of a plurality of blocks corresponding to different frames in the time axis direction are calculated by interpolation processing, and a plurality of frames are restored. The moving image restoration apparatus executes a block interpolation step for generating
In the block interpolation step ,
The input block of frames not decimated time, to generate a copy and restore block frame decimation more temporally close in time, moves according to the movement amount information input block corresponding, the only movement amount restored block between frames Let the video data be restored ,
The input block to be restored is a block that is subjected to thinning processing by changing the thinning target frame based on the movement amount information, and is an expression that defines the super-resolution effect generation condition, that is, the speed v pixel / frame. The following formula specifying the super-resolution effect generation condition when moving objects are thinned out in the spatial direction with m pixels,

Where t is time, T is frame interval,
It is a block that has been subjected to thinning processing by changing the thinning target frame based on the movement amount information so as to satisfy the super-resolution effect generation condition represented by the above formulas (formula a) and (formula b),
In the block interpolation step ,
In order to restore the frame thinned out by the thinning process based on the movement amount information, the moving direction of the restored block between the frames is changed according to the mode of the thinning process in the time direction of the block input as the restoration target. selected and so as to be more temporally move between near frames, in the computer program characterized by causing a moving image data reconstruction.

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the moving image conversion apparatus and method of the present invention, it is possible to generate data having a super-resolution effect by changing the mode of thinning in the time direction according to the movement amount, and the data amount is reduced. Nevertheless, it is possible to generate high-quality data in which image degradation during decoding is suppressed.

  According to the moving image conversion apparatus and method of the present invention, block division is performed for each frame constituting moving image data, a subject movement amount in each block is detected, and spatial thinning processing in a mode determined based on the movement amount or Since at least one of the time direction thinning-out processes is executed, and in the execution of the time direction thinning-out process, the thinning process is performed by changing the thinning target frame based on the movement amount information. It is possible to generate data with the optimal time-direction decimation, that is, the super-resolution effect, and generate high-quality data that suppresses image degradation during decoding, even though the amount of data is reduced can do.

  Furthermore, according to the moving image restoration apparatus and method of the present invention, moving image conversion data that has been subjected to thinning processing in the time direction is input in units of blocks, and pixel values of a plurality of blocks corresponding to different frames in the time axis direction are interpolated. In the block interpolation unit that calculates by processing and generates a restoration block of a plurality of frames, the movement amount information corresponding to the block to be interpolated is input, and the restoration block is moved between frames according to the movement amount information. Since the restoration is configured, restoration data having a super-resolution effect can be generated, and high-quality decoded data in which image degradation is suppressed can be generated.

Hereinafter, the configuration of a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program according to the present invention will be described with reference to the drawings. The description will be made according to the following items.
(1) Basic configuration of moving image conversion apparatus using super-resolution effect (2) Apparatus and method for executing moving image compression processing according to the present invention (3) Executing moving image expansion processing according to the present invention Apparatus and method

[(1) Basic configuration of moving image conversion apparatus using super-resolution effect]
First, a basic configuration of a moving image conversion apparatus that performs moving image compression using the super-resolution effect as a base of the present invention will be described. This basic configuration is described in detail in Japanese Patent Application No. 2003-412501 filed earlier by the present applicant, and the image is divided into small areas (blocks), and according to the moving speed of each area. Thus, the amount of data is compressed by adaptively performing the thinning of the number of pixels and the thinning of the frame rate.

  FIG. 1 shows a configuration example of the moving image conversion apparatus 10 described in Japanese Patent Application No. 2003-412501. The moving image conversion apparatus 10 can reduce the amount of data so that an observer does not perceive image quality deterioration due to the reduction of the data amount by performing a moving image conversion process using the super-resolution effect. It is a thing.

  Note that the super-resolution effect is a visual effect in which the observer perceives a resolution higher than the number of display pixels when following the moving subject discretely sampled in the spatial direction. This is because a human has a visual characteristic of perceiving an addition of a plurality of images presented within a certain time. This characteristic is attributed to a visual temporal integration function known as a block law, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp. 219-220”. There is a report that the integration time for which the block law is established varies depending on the presentation conditions such as the intensity of the background light, but is approximately 25 ms to 100 ms.

  The moving image conversion apparatus 10 shown in FIG. 1 performs compression for reducing data so that the observer does not perceive image quality degradation by performing moving image conversion processing using the super-resolution effect caused by the temporal integration function. It is set as the structure to perform. A configuration of the moving image conversion apparatus 10 in FIG. 1 will be described.

  The block dividing unit 11 divides each frame of the input moving image into blocks as divided areas of predetermined pixels and supplies them to the movement amount detecting unit 12. The movement amount detection unit 12 detects the movement amount for each block supplied from the block division unit 11, and transmits the block and the movement amount to the block processing unit 13. The block processing unit 13 performs a moving image conversion process corresponding to the movement amount, that is, a compression process, on the block supplied from the movement amount detection unit 12 to reduce the data amount. The block processing unit 13 supplies the output unit 14 with data about the block with the reduced data amount obtained as a result of the processing. The output unit 14 collectively outputs, as stream data, the data regarding the blocks with the reduced data amount supplied from the block processing unit 13.

  Next, details of each unit will be described with reference to FIG. The frame of the moving image supplied to the moving image conversion apparatus 10 is input to the image storage unit 21 of the block dividing unit 11. The image storage unit 21 stores the input frame. Each time the number of accumulated frames reaches N (N is a positive integer), the image accumulating unit 21 supplies the N frames to the block dividing unit 22, and M in the N frames The second stored frame (hereinafter referred to as the Mth frame) is supplied to the movement amount detection unit 12 (movement amount detection unit 31). For example, N = 4.

  The block dividing unit 22 divides each of the N frames (consecutive N frames) supplied from the image storage unit 21 into blocks of a certain size (for example, 8 × 8, 16 × 16) and moves them. It outputs to the quantity detection part 12 (block distribution part 32). The block dividing unit 22 also converts each block of the Pth frame stored in the image storage unit 21 (hereinafter referred to as the Pth frame) among the N frames to the movement amount detection unit 12 (movement amount detection). Part 31). The Pth frame is a different frame from the Mth frame.

  Next, the movement amount detection unit 12 will be described. The movement amount detection unit 31 of the movement amount detection unit 12 uses the motion vector of each block of the Pth frame supplied from the block division unit 22 of the block division unit 11 as the Mth frame supplied from the image storage unit 21. For example, the block matching process between the frames is detected and detected, and the detected motion vector is supplied to the block distributor 32. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and in the vertical direction (Y-axis direction). In addition, the movement amount detection unit 31 may be configured to enlarge the image in order to improve the detection accuracy of the movement amount and detect the movement amount using the enlarged image.

  The block distribution unit 32 of the movement amount detection unit 12 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 22 in units of N, and the movement amount detection unit From 31, the amount of movement of the block of the P-th frame among the N blocks is supplied. The block distribution unit 32 supplies the supplied N blocks and the movement amount to any of the block processing units 51 to 53 that perform processing corresponding to the movement amount of the block processing unit 13.

  Specifically, the block distribution unit 32 is supplied from the movement amount detection unit 31 and has a movement amount in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) between one frame of 2 pixels (pixels) or more. In some cases, the N blocks supplied from the block dividing unit 22 and the movement amount supplied from the movement amount detection unit 31 are output to the block processing unit 51. If the amount of movement in the horizontal direction and the vertical direction between one frame is less than 2 pixels and more than 1 pixel, the block distribution unit 32 outputs N blocks and the amount of movement to the block processing unit 53. . When the movement amount is other than that, the block distribution unit 32 supplies the N blocks and the movement amount to the block processing unit 52.

  That is, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs block processing for performing processing for converting image data according to the frame rate and spatial resolution. The block images are distributed to the units 51 to 53.

  Next, details of the block processing unit 13 will be described. The block processing unit 13 is composed of the three block processing units 51 to 53 as described above. The block processing unit 51 supplies a total of N blocks (horizontal or vertical) at the same position in each of consecutive N (for example, N = 4) frames supplied from the block distribution unit 32 of the movement amount detection unit 12. Similarly, a process of thinning out the number of pixels according to the amount of movement supplied from the block distributor 32 (spatial direction thinning process) is performed on N blocks in the case where the amount of movement in the direction is 2 pixels or more. .

Specifically, when the amount of horizontal movement between one frame is 2 pixels (pixels) or more, the block processing unit 51 determines that the block to be processed is 4 × 4 pixels as shown in FIG. When configured, only one pixel value among the four pixels in the horizontal direction is selected as a representative value. In the example of FIG. 3 _(b), to enable only _{P 10} of four pixels _{P 00} through _{P 30} as a representative value (sample point). Other pixel values are invalid. Similarly, for four pixels P _{01 to} P ₃₁ , P ₁₁ is a representative value (sample point), for four pixels P _{02 to} P ₃₂ , P ₁₂ is a representative value (sample point), and P representative value _{P 13} are selected from four pixels ₀₃ through _{P 33} as (sample point).

When the amount of movement in the vertical direction between one frame is 2 pixels or more, the block processing unit 51 has four pixels in the vertical direction when the block is composed of 4 × 4 pixels as shown in FIG. One pixel value is valid as a sample point. In the example of FIG. 3 _(c), to enable as a sample point only _{P 01} of four pixels _{P 00} to _{P 03.} Other pixel values are invalid. _Similarly, the sample point _{P 11} are selected from four pixels _{P 10} to _{P 13,} and the sampling point _{P 21} are selected from four pixels _{P 20} to _{P 23,} the four pixels _{P 30} through _{P 33} It is for the sampling point _{P 31.}

  The block processing unit 51 performs such spatial thinning processing on a total of N = 4 blocks at the same position in each of the supplied consecutive N (for example, N = 4) frames. The data amount of each block is reduced to ¼, and the data amount of all four blocks is reduced to ¼. The block processing unit 51 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  Next, processing executed by the block processing unit 52 shown in FIG. 2 will be described. The block processing unit 52 shown in FIG. 2 has a total of N blocks (the amount of movement in both the horizontal direction and the vertical direction is the same) in each of consecutive N frames supplied from the block distribution unit 32 of the movement amount detection unit 12. A process of thinning out the number of frames (time direction thinning process) is performed on N blocks in the case of less than one pixel.

  Specifically, as shown in FIG. 4, the block processing unit 52 converts four blocks Bi at the same position in four consecutive frames F1 to F4 into one block (in this example, Then, the number of frames to be converted into the block Bi) of the frame F1 is thinned out (thinning of the number of frames between four frames). The block processing unit 52 supplies the output unit 14 with data (one block) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing. Pixel data of one selected block becomes sample point data corresponding to 4 frames.

  The block processing unit 53 is supplied from the block distribution unit 32 of the movement amount detection unit 12 and has a total of N blocks (the movement amount in the horizontal direction and the vertical direction is 1 in each of the consecutive N frames). The number of pixels is thinned out (spatial direction thinning process) and the number of frames is thinned out (time direction thinning process) for each of N blocks in the case of more than pixels and less than 2 pixels.

Unlike the thinning-out process in the block processing unit 51, the block processing unit 53, when the amount of horizontal movement between one frame is 1 pixel or more and less than 2 pixels as shown in FIG. When the block to be processed is composed of 4 × 4 pixels as shown in FIG. 5A, only two pixel values of the four pixels in the horizontal direction are selected as representative values. In the example of FIG. 5B, only P ₀₀ and P ₂₀ out of the four pixels P _{00 to} P ₃₀ are validated as representative values (sample points). Other pixel values are invalid. Similarly, only P ₀₁ and P ₂₁ are representative values (sample points) for the four pixels P _{01 to} P ₃₁ , and only P ₀₂ and P ₂₂ are the representative values for the four pixels P _{02 to} P _32. (Sample points), and for the four pixels P _{03 to} P ₃₃ , only P ₀₃ and P ₂₃ are representative values (sample points).

When the amount of vertical movement between one frame is 1 pixel or more and less than 2 pixels, the block processing unit 53 includes 4 × 4 pixels as a processing target block as shown in FIG. At this time, two pixel values of the four pixels in the vertical direction are validated. In the example of FIG. 5C, P ₀₀ and P ₀₂ out of the four pixels P _{00 to} P ₀₃ are validated as representative values (sample points). Other pixel values are invalid. Similarly, P ₁₀ and P ₁₂ are representative values (sample points) for the four pixels P _{10 to} P ₁₃ , and P ₂₀ and P ₂₂ are representative values (sample points) for the four pixels P _{20 to} P _23. and sample _point), with respect to the four pixels _{P 30} to _{P 33} to _{P 30} and _{P 32} as representative values (sample points).

  Furthermore, the block processing unit 53 performs a frame number thinning process. Specifically, the block processing unit 53 performs a thinning process for the number of frames that enable two of the four blocks F1 to F4 in the same position on the four blocks at the same position. To do. In the thinning-out process for the number of frames, the block processing unit 53 is different from the thinning-out process in the block processing unit 52, as shown in FIG. 6, at the same position in each of the four consecutive frames F1 to F4. The total number of blocks Bi is reduced to any two of them (two blocks of frames F1 and F3 in the example in the figure), and the number of frames is reduced (the number of frames between two frames is reduced). The pixel data of the two selected blocks becomes sample point data corresponding to 4 frames. In this case, in sampling in the spatial direction described with reference to FIG. 5, eight sampling points have already been selected for one block, and a total of 16 sampling points are selected from two blocks, which correspond to four frames. Is set as sample point data.

  The block processing unit 53 performs the spatial direction decimation process for reducing the data amount described with reference to FIG. 5 to 1/2 and the time direction decimation process for reducing the data amount described with reference to FIG. Since it is applied to the four supplied blocks, the data amount of the four blocks is reduced to (1/2) × (1/2) = 1/4 as a result. The block processing unit 53 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  The output unit 14 generates stream data from the data for the N blocks with the reduced data amount supplied from each of the block processing units 51 to 53 of the block processing unit 13.

  The present applicant has proposed the above-described moving image conversion device in Japanese Patent Application No. 2003-412501 filed earlier, and this moving image conversion device moves for each minute region of the moving image in view of human visual characteristics. This is a configuration that compresses the amount of data by determining the optimal frame rate and spatial resolution according to the speed information, and uses the time integration function in the human visual system to generate super-resolution when chasing a moving subject with the eyes Based on the effect, the data amount is reduced by performing spatial thinning processing in the moving direction according to the moving amount.

[(2) Apparatus and method for executing moving image compression processing according to the present invention]
In the data compression method in the above-described moving image conversion apparatus, the following problems may occur when time direction thinning is performed.
(1) The movement of the blocks thinned out in the time direction and the blocks not thinned out become unnatural.
(2) Flicker-like noise is generated when a block thinned out in the time direction has a certain high-frequency component.
Hereinafter, an apparatus and a method for executing a moving image compression process according to the present invention that realizes data compression that solves the above-described problems will be described.

  A schematic configuration of the moving image conversion apparatus according to the present embodiment will be described. FIG. 7 is a block diagram showing a configuration of a moving image conversion apparatus as an embodiment of the present invention. As shown in FIG. 7, the moving image conversion apparatus according to the present embodiment inputs moving image data to be processed, and the block dividing unit 100 divides the moving image into frames of a predetermined area for each frame constituting the moving image. The block data is output to n moving image compression means 102-104. The block synthesizing unit 105 synthesizes the compressed data of each block divided into blocks and processed by the moving image compressing unit in units of frames, and outputs the result to the encoder 106 as compressed moving image data in units of frames. The encoder 106 is an existing encoder capable of compressing moving images such as MPEG compression, and further performs data compression such as MPEG on the compressed frame data to output to a network or a storage medium. Output for storage in

  Configurations of the block dividing unit 100 and the moving image compression units 102 to 104 in the present embodiment will be described with reference to FIG. FIG. 8 is a block diagram showing the configuration of the block dividing unit 100 and the moving image compression units 102 to 104 in the present embodiment.

  The moving images to be processed are accumulated for N frames in the image accumulating unit 200 in the block dividing unit 100. In this embodiment, N = 4. The accumulated frames are decomposed into blocks of a certain size in the block dividing unit 201, for example, blocks of 8 pixels × 8 pixels, 16 pixels × 16 pixels, etc., and sent to the block distributing unit 203. Further, the block size of the divided frame at this time is sent to the movement amount detection unit 202.

  The movement amount detection unit 202 acquires the Mth and M + 1th frames of the frames stored in the image storage unit 200, and in units of blocks according to the size of the blocks obtained by dividing the frame from the block division unit 201, A motion vector relating to each block of the frame is detected using an existing method such as a block matching method. In this embodiment, M = 2.

  Furthermore, since the detected movement amount is used for decoding, the movement amount information detected by the movement amount detection unit 202 is output to the encoder 106 at the output 207 and is encoded by the encoder 106. Encode with image. In the present embodiment, the movement amount detection unit 202 is configured to calculate the movement amount by block matching using two consecutive frames in N frames. However, the movement amount detection process executed by the movement amount detection unit 202 is described. Other methods may be applied. Further, the movement amount may be calculated based on any combination of frames including selection of all the frames in N frames, instead of the movement amount detection configuration based on two frames.

  The block distribution unit 203 executes a process of allocating the blocks to the block processing units 2-4 to 206 that perform different compression processes on each block. The input of the block distribution unit 203 is the movement amount of each block output from the movement amount detection unit 202 and the image data for N frames divided into blocks output from the block division unit 201.

  The block distribution unit 203 outputs the input blocks for N frames to any of the block processing units 204 to 206 according to the movement amount of each block. Further, when the block distribution condition is used for decoding, an output 209 set as the block distribution condition, for example, as a block distribution flag is output to the encoder 106 and encoded together with the moving image to be encoded.

  The block distribution unit 203 applies the movement amount information detected by the movement amount detection unit 202 as a block distribution condition. In this embodiment, when the larger one of the horizontal or vertical movement amounts between one frame exceeds 2 pixels, the block processing unit a204 is a value among the horizontal or vertical movement amounts between one frame. When the larger one is less than 2 pixels and 1 pixel or more, it is output to the block processing unit c206, and in other cases, it is output to the block processing unit b205.

  Note that the above conditions for determining the output destination in the block distribution unit 203 are merely examples for the present embodiment, and other conditions may be applied. Each of the block processing units 204 to 206 in the block processing unit 208 receives movement amount information corresponding to block data to be processed from the block distribution unit 203, and executes a thinning process as a data compression process.

  The block processing unit a204 performs a spatial thinning process on the number of pixels of the input block according to the corresponding movement amount input simultaneously with the processing target block data. In this embodiment, when the amount of movement in the horizontal direction is 2 pixels or more, the pixels in the block are divided into sets of 1 × 4 pixel units, and the pixel value of each set is one of the four pixels included in each set. In this way, the number of pixels is reduced. When the amount of movement in the vertical direction is 2 pixels or more, the pixels in the block are divided into sets of 4 × 1 pixels, and the pixel value of each set is set to one of the four pixels included in each set. These processes are similar to the processes described above with reference to FIG. Furthermore, when the amount of movement in the vertical and horizontal directions is 2 pixels or more, the pixels in the block are divided into sets of 2 × 2 pixels, and the pixel values of each set are any one of the four pixels included in each set. To.

  Originally, making the values of all four pixels the same causes a significant deterioration in image quality, but the block processing unit a204 has a larger value of more than two pixels in the horizontal or vertical movement amount between one frame. This is a configuration that performs these spatial decimation only for blocks, and due to the temporal temporal integration function known as the block law described above, a super-resolution effect is brought about, and the observer is discrete in the spatial direction. When following a moving sample sampled in a visual manner, a visual effect of perceiving a resolution higher than the number of display pixels occurs, and the observer hardly perceives image quality degradation. That is, if the block is moving at a sufficiently high speed and the frame rate of the moving image is sufficiently large, human being cannot perceive the deterioration of the image quality.

  The block processing unit b205 executes a thinning process for the number of frames of the input block, that is, a thinning process in the time direction. The input blocks are N blocks at the same position in the N frame image. In this embodiment, when the movement amount of a certain block in both the horizontal direction and the vertical direction is less than one pixel, all the pixel values of the four input blocks are the values of any one of the four blocks. To. This process corresponds to the process described above with reference to FIG.

  The block processing unit c206 thins out the number of pixels and the number of frames of the input block. That is, both spatial thinning processing and temporal thinning processing are executed. The basic operation is a combination of the spatial thinning process of the block processing unit a204 and the time direction thinning process of the block processing unit b205.

  In the block processing unit c206, the thinning-out amount of the number of pixels and the thinning-out amount of the frame number are different from the spatial thinning processing of the block processing unit a204 and the temporal thinning processing of the block processing unit b205. First, the amount of thinning out of pixels as space thinning is not performed in units of 4 pixels as in the block processing unit a204 but in units of 2 pixels. That is, in the block processing unit a204, there are three thinning methods 1x4, 4x1, and 2x2, but there are only two thinning methods 1x2 and 2x1 performed by the block processing unit c206. That is, when the amount of movement in the horizontal direction is 1 to 2 pixels, the pixels in the block are divided into sets of 1 × 2 pixel units, and the pixel value of each set is set to one of the two pixels included in each set. By doing so, the number of pixels is reduced. When the vertical movement amount is 1 to 2 pixels, the pixels in the block are divided into sets of 2 × 1 pixels, and the pixel value of each set is set to one of the two pixels included in each set. . This process corresponds to the process described above with reference to FIG.

  In addition, the block processing unit c206 also performs temporal direction decimation processing, that is, decimation of the number of frames, but the difference is that it is performed in units of 2 frames instead of in units of 4 frames as in the block processing unit b205. is there. This process corresponds to the process described above with reference to FIG.

  Further, the block processing unit c206 according to the present embodiment executes the temporal thinning process, that is, the aspect of the frame number thinning process, depending on the movement amount of the block. FIG. 9 shows a block diagram of a configuration of the block processing unit c206 in the present embodiment.

  The process of the block processing unit c206 will be described with reference to FIG. The blocks input to the block processing unit c206 are stored in the block storage unit 301 for N frames. It is desirable that the number of blocks stored here is equal to the number of frames stored in the image storage unit 200 (see FIG. 8).

  The blocks accumulated in the block accumulation unit 301 are subjected to spatial decimation processing, that is, pixel number decimation processing, in the block processing unit c (1) 305. The space thinning process performed here is the above-described two kinds of space thinning of 1 × 2 and 2 × 1. That is, when the amount of movement in the horizontal direction is 1 to 2 pixels, the pixels in the block are divided into sets of 1 × 2 pixel units, and the pixel value of each set is set to one of the two pixels included in each set. By doing so, the number of pixels is reduced. When the vertical movement amount is 1 to 2 pixels, the pixels in the block are divided into sets of 2 × 1 pixels, and the pixel value of each set is set to one of the two pixels included in each set. . This process corresponds to the process described above with reference to FIG.

  The blocks subjected to the spatial decimation processing are collectively output to the block distribution unit 302 for N frames, and the block distribution unit 302 converts the blocks for N frames into the block processing unit c (2) 303 or the block according to the block movement amount. The data is output to the processing unit c (3) 304.

  As a distribution condition in the block distribution unit 302, in this embodiment, the block processing unit c (2) 303, one frame if the larger one of the horizontal or vertical movement amounts between the frames is less than 1.25 pixels. If the larger one of the horizontal or vertical movement amounts between them is 1.25 pixels or more, it is output to the block processing unit c (3) 304.

Since the block processing unit c206 distributes only blocks with a larger value of less than 2 pixels and more than 1 pixel in the horizontal or vertical movement amount between one frame, finally,
The larger the amount of horizontal or vertical movement between one frame,
If it is 1 pixel or more and less than 1.25 pixel, the block processing unit c (2) 303,
If it is 1.25 pixels or more and less than 2 pixels, the block processing unit c (3) 304,
Each block is output, and the time direction thinning process is executed in a different manner in each block processing unit.

  A specific time direction thinning process executed by the block processing unit c (2) 303 and the block processing unit c (3) 304 will be described with reference to FIGS.

  First, with reference to FIG. 10, the time direction thinning process performed by the block processing unit c (2) 303 will be described. The block processing unit c (2) 303 performs temporal direction thinning processing on a block having a larger value among horizontal and vertical movement amounts between one frame and one pixel or more and less than 1.25 pixels.

  The input to the block processing unit c (2) 303 is four blocks 321 to 324 obtained from the same position of four consecutive frames. The blocks 321 to 324 are obtained from the block 321 from the first frame, the block 322 from the second frame, the block 323 from the third frame, and the block 324 from the fourth frame. They are in chronological order.

  The blocks 321 to 324 input to the block processing unit c (2) 303 have already been subjected to spatial thinning processing in the block processing unit c (1) 305 (see FIG. 9). FIG. 10 shows pixels 326 to 329 left as representative pixels (sample points) in each block as a result of spatial thinning in the horizontal direction 2 × 1 in blocks 321 to 324. As shown in FIG. 10, the pixels 326 to 329 left as the front pixels (sample points) of each block as a result of the spatial thinning have the same value.

  The block processing unit c (2) 303 performs a process of thinning out the input blocks 321 to 324 in half in the time direction. The block processing unit c (2) 303 performs temporal direction thinning processing on a block having a larger value among horizontal and vertical movement amounts between one frame and one pixel or more and less than 1.25 pixels. In the present embodiment, the block processing unit c (2) 303 executes the thinning process in the time direction by leaving the first frame and the fourth frame and thinning out the second and third frames. The time direction thinning processing results are blocks 331 to 334.

  As shown in FIG. 10, as a result of thinning out in the time axis direction, blocks 331 and 334 are left, and blocks 332 and 333 are thinned out. By this spatial decimation, only representative pixels (sample points) 336 and 337 remain in the block for four frames, and the data amount is reduced to ½ by decimation in the time direction. This space decimation reduces the amount of data by half. Since this block has been subjected to spatial thinning processing that reduces the data to 1/2 in the block processing unit c (1) 305 (see FIG. 9), the data amount is reduced to 1/4 as a result.

  Next, the time direction thinning process executed by the block processing unit c (3) 304 will be described. The block processing unit c (3) 304 performs temporal direction thinning processing on a block having a larger value in the horizontal or vertical movement amount between one frame and having a value of 1.25 pixels or more and less than 2 pixels.

  The input to the block processing unit c (3) 304 is four blocks 351 to 354 obtained from the same position of four consecutive frames shown in FIG. The blocks 351 to 354 are arranged in time series in the order of the blocks 351 to 354.

  The blocks 351 to 354 input to the block processing unit c (3) 304 have already been subjected to spatial thinning processing in the block processing unit c (1) 305 (see FIG. 9). FIG. 11 shows pixels 356 to 359 left as representative pixels (sample points) in each block as a result of spatial thinning in the horizontal direction 2 × 1 in blocks 351 to 354. As shown in FIG. 11, the pixels 356 to 359 left as the front pixels (sample points) of each block as a result of the spatial thinning have the same value.

  The block processing unit c (3) 304 performs a process of thinning out the input blocks 351 to 354 by half in the time direction. In this embodiment, the block processing unit c (3) 304 executes the thinning process in the time direction by leaving the first frame and the third frame and thinning out the second and fourth frames. The time direction thinning processing results are blocks 361 to 364.

  As shown in FIG. 11, as a result of thinning out in the time axis direction, blocks 361 and 363 are left, and blocks 362 and 364 are thinned out. By this spatial decimation, only representative pixels (sample points) 366 and 367 remain in the block for four frames, and the data amount is reduced to ½ by decimation in the time direction. Since this block has been subjected to spatial thinning processing that reduces the data to 1/2 in the block processing unit c (1) 305 (see FIG. 9), the data amount is reduced to 1/4 as a result.

  As described above, for example, when the block processing unit is configured to perform the temporal direction thinning process with a compression ratio of 1/2 as processing units of four consecutive frames, when the movement amount is large, the first frame and the third frame If the movement amount is small, the first and fourth frames are left, and the second and third frames are left out. The thinning process in the time axis direction is executed. More generally, in the execution of the time direction thinning process, when the movement amount is large, the block processing unit sets a small number of frames to be thinned consecutive to the time axis, and when the movement amount is small, the block processing unit sets the time axis. A thinning process in the time axis direction in which a large number of frames to be thinned out is set is executed.

  Next, processing of the block processing unit c206 will be described according to the flowchart shown in FIG. In step S101, the block processing unit c206 accumulates the input blocks for N frames. It is desirable that the number of blocks stored here is equal to the number of frames stored in the image storage unit 200 (see FIG. 8). The accumulated block is subjected to thinning-out processing in step S102. The spatial thinning process performed here is the above-described two types of spatial thinning of 1 × 2 and 2 × 1, and the data amount is reduced to ½.

In the block subjected to the spatial thinning process, in step S103, the time thinning process mode based on the movement speed of the block is executed. As this distribution condition, in this embodiment, the larger one of the horizontal or vertical movement amounts between one frame is as follows.
If it is 1 pixel or more and less than 1.25 pixel, the block processing unit c (2) 303,
If it is 1.25 pixels or more and less than 2 pixels, the block processing unit c (3) 304,
The time direction thinning process is executed in a different manner in each block processing unit.

  In step S104, in the block processing unit c (2) 303, refer to FIG. 10 for blocks in which the larger value of the horizontal or vertical movement amount between one frame is 1 pixel or more and less than 1.25 pixels. As described above, the time direction thinning process is performed, that is, the pixel data of the first and fourth frames are left and the pixel data of the second and third frames are deleted in four consecutive frames. The

  In step S105, the block processing unit c (3) 304 applies a block having a larger value of 1.25 pixels or more and less than 2 pixels to the horizontal or vertical movement amount between one frame. The time direction thinning process described with reference to the above, that is, the time direction thinning process in which the pixel data of the first and third frames is left and the pixel data of the second and fourth frames is deleted in four consecutive frames. Executed.

As described above, in the moving image conversion apparatus of the present invention, the larger one of the horizontal or vertical movement amounts between one frame is as follows.
(A) A block having a movement amount of 2 pixels or more: spatial thinning processing (refer to FIG. 3) to reduce the data amount to 1/4
(B1) A block having a movement amount of 1.25 pixels or more and less than 2 pixels: a space thinning process (see FIG. 5) that reduces the data amount to 1/2 and a time direction thinning process that reduces the data amount to 1/2. In the four frames, the pixel data of the first and third frames are left, and the time direction thinning process (see FIG. 11) of the mode of deleting the pixel data of the second and fourth frames,
(B2) A block having a movement amount of 1 pixel or more and less than 1.25 pixels: a space thinning process (see FIG. 5) for reducing the data amount to 1/2 and a time direction thinning process for reducing the data amount to 1/2. In the four frames, the pixel data of the first and fourth frames are left, and the pixel data of the second and third frames are deleted in the time direction thinning process (see FIG. 10).
(C) Block with movement amount less than 1 pixel: time direction decimation processing (refer to FIG. 4) to reduce the data amount to 1/4.
The thinning process is performed in the above four different modes.

A difference from the processing mode in “(1) Basic configuration of moving image conversion apparatus using super-resolution effect” described above is that a block processing unit c206 (FIG. 8), and this block processing unit c206 performs time direction decimation (see FIGS. 10 and 11) in a different manner depending on the movement amount. As described above, by executing the thinning mode in the time direction in different modes depending on the movement amount of the block, the problem in “(1) Basic configuration of moving image conversion apparatus using super-resolution effect” described above. That is,
(1) The movement of the blocks thinned out in the time direction and the blocks not thinned out become unnatural.
(2) Flicker-like noise is generated when a block thinned out in the time direction has a certain high-frequency component.
Thus, it is possible to generate high quality encoded data.

  The reason why the block processing unit c206 can generate high-quality encoded data by performing temporal direction decimation (see FIGS. 10 and 11) in different modes according to the movement amount of the block. Here, high-quality encoded data is data that exhibits a super-resolution effect due to the visual temporal integration function known as the above-mentioned block law. If the data exhibits the super-resolution effect, the observer will not perceive deterioration in image quality when following the moving subject discretely sampled in the spatial direction. In the block processing unit c206, it is possible to generate data that exhibits the super-resolution effect by performing temporal direction decimation (see FIGS. 10 and 11) in a different manner depending on the movement amount of the block.

The harmonics generated when thinning out in the spatial direction cancel each other out by the harmonics generated in other frames, so that a super-resolution effect appears. The condition at this time is a case where the following equation (Equation 1) is satisfied when an object moving at a speed v pixel / frame is thinned out in a spatial direction with m pixel.

  It is known that if the condition of the above (Equation 1) is satisfied, even if thinning out of m pixels is performed in the spatial direction, harmonics are canceled by the super-resolution effect, and humans cannot perceive deterioration. Here, since there is an integration function in the visual time characteristic, if there are multiple sets of frames that satisfy the expression (Equation 1) that cancel the aliasing components within the integration period, only the combination frames that cancel each other. It is possible to thin out the frames in the time axis direction by leaving other frames and deleting other frames.

  However, if the frame to be thinned out in the time axis direction is fixed, the combination of frames that cancel each other satisfying the equation (Equation 1) changes depending on the moving speed of the block. Cannot be canceled out, resulting in subjective image deterioration such as flicker noise.

  Therefore, as shown in the embodiment, by changing the frame to be thinned out according to the moving speed of the block, a combination of frames satisfying the equation (Equation 1) is selected and thinned out. As a result, subjective image deterioration such as flicker noise can be suppressed. The moving image that has been preprocessed and decoded by the moving image processing apparatus according to the present embodiment as described above is decoded even though the data amount is reduced by the compression processing performed at the time of encoding. It is possible to suppress image degradation at the time.

  Further, the reason why the super-resolution effect can be obtained by the thinning-out mode of the present invention will be described with reference to FIGS.

The condition for obtaining the super-resolution effect when the object moving at the speed v pixel / frame is thinned out in the spatial direction by m pixel is the case where the above-described equation (Equation 1) is satisfied. When the above formula (Formula 1) is expanded by Euler's formula, the following formula (Formula 2) is obtained.

  If the above formulas (Formula 2a) and (Formula 2b) are satisfied, it is shown that a super-resolution effect can be obtained when thinning out in the spatial direction with m pixels, and humans cannot perceive image degradation. . FIG. 13 shows changes in the super-resolution condition when the spatial thinning amount is m = 2 pixels and the movement amount is v = 1 to 2 pixels / frame in a condition where time thinning is not performed. Here, the number of frames integrated by the visual time characteristic is set to 8.

  In FIG. 13, the solid line corresponds to the above formula (Formula 2a), and the dotted line corresponds to the above formula (Formula 2b). Basically, it is an ideal condition that both Equation 2a and Equation 2b are = 0. However, in the range of −1 to +1, humans almost do not feel the deterioration of the image and almost achieve the super-resolution effect. It is possible to obtain the same effect.

  In FIG. 13, the solid line and dotted line graphs are in the range of −1 to +1 when the movement amount is v = 1 to 1.6 pixels / frame. Movement amount: When v = 1.6-2 pixels / frame, the solid line and dotted line graphs deviate from −1 to +1, and the super-resolution effect cannot be obtained.

FIG. 14 shows 8 consecutive frames: t = 0 to 8T,
leaving t = 0, 2T, 4T, 6T frames,
delete frames of t = T, 3T, 5T, 7T,
FIG. 6 is a graph when the temporal thinning is performed in the above-described manner, and shows a change in super-resolution conditions of a spatial thinning amount: m = 2 pixels and a movement amount: v = 1-2 pixels / frame. The solid line corresponds to the above formula (Formula 2a), and the dotted line corresponds to the above formula (Formula 2b). This thinning mode corresponds to the time direction thinning mode described above with reference to FIG.

  From FIG. 14, the solid line and dotted line graphs are in the range of −1 to +1 when the movement amount is v = 1.2 to 1.8 pixels / frame. Movement amount: In the vicinity of v = 1.0 or v = 2.0 pixels / frame, the solid line and dotted line graphs deviate from −1 to +1, and the super-resolution effect cannot be obtained.

FIG. 15 shows 8 consecutive frames: t = 0 to 8T.
leaving t = 0, 3T, 4T, and 7T frames,
delete frames of t = T, 2T, 5T, 8T,
FIG. 6 is a graph when the temporal thinning is performed in the above-described manner, and shows a change in super-resolution conditions of a spatial thinning amount: m = 2 pixels and a movement amount: v = 1-2 pixels / frame. The solid line corresponds to the above formula (Formula 2a), and the dotted line corresponds to the above formula (Formula 2b). This thinning mode corresponds to the time direction thinning mode described above with reference to FIG.

  From FIG. 15, the solid line and dotted line graphs are in the range of −1 to +1 because there is a movement amount: v = 1 to 1.15 pixels / frame, and a movement amount: v = 1.6. This is the case when there is a movement amount of ˜1.8 pixels / frame.

  As can be understood from FIGS. 14 and 15, the range of the amount of movement in which the super-resolution effect can be obtained differs depending on the thinning mode in the time direction. Therefore, data having a super-resolution effect can be generated by performing the thinning out in the time direction that is measured according to the measured movement amount. FIG. 16 is a graph in the case where the process of changing the thinning mode in the time direction is performed before and after the movement amount: v = 1.25 pixels. The spatial thinning amount: m = 2 pixels, the movement amount: v = 1. The change in super-resolution condition of ~ 2 pixels / frame is shown. The solid line corresponds to the above formula (Formula 2a), and the dotted line corresponds to the above formula (Formula 2b).

The graph shown in FIG.
Movement amount: When v = 1.0 to 1.25, 0 and 3T in 4 frames t = 0 to 3T are left, and time direction thinning processing is performed to thin out T and 2T (corresponding to FIG. 10).
In the movement amount: v = 1.25 to 2.0, 0 and 2T in 4 frames t = 0 to 3T are left, and T and 3T are thinned out (corresponding to FIG. 11).
It is a graph at the time of performing.

  In this graph, in the range of v = 1.0 to 1.8 pixels / frame, the solid line and dotted line graphs are in the range of −1 to +1, and it is shown that the super-resolution effect is obtained in this range.

  In this way, it is possible to generate data with a super-resolution effect by changing the mode of thinning out in the time direction according to the amount of movement. It is possible to generate high quality data that suppresses image degradation.

[(3) Apparatus and method for executing moving image expansion processing according to the present invention]
Next, an apparatus and method for executing a moving image expansion process according to the present invention will be described. First, a description will be given of compressed moving image data to be restored in a moving image restoration apparatus that performs moving image decompression described below.

  The restoration target data of the moving image restoration apparatus described below is data obtained by performing different thinning processing, that is, space thinning and time thinning based on the above-described movement amount for each block. That is, according to the above-described “(1) Basic configuration of moving image conversion apparatus using super-resolution effect” and [(2) Apparatus and method for executing moving image compression processing according to the present invention] Decoding / decompression processing of compression-encoded data that has been subjected to spatial thinning and time-direction thinning processing according to the amount of movement is executed.

  That is, in the moving image frame data divided into blocks, when the amount of movement between frames is large, a space thinning process is executed, and when the amount of movement between frames is small, a time thinning process is executed, If it is in the middle, decoding the data with compression of the amount of data by determining the optimal frame rate and spatial resolution according to the moving speed information for each minute area of the moving image to perform spatial thinning and time thinning, Execute decompression processing.

  The moving image restoration apparatus described below uses a time integration function in the human visual system, and based on the super-resolution effect that occurs when a moving subject is chased with eyes, the moving image restoration device spatially moves in the moving direction according to the moving amount. Decryption / decompression processing of the thinned data is executed.

  First, with reference to FIG. 17, a generation process of data to be restored, that is, a thinning process data will be described. FIG. 17 shows a configuration corresponding to the moving image conversion apparatus described above with reference to FIG. 8 in the section [(2) Apparatus and method for executing moving image compression processing according to the present invention]. The block dividing unit 100, the block processing unit 208, the block synthesizing unit 105, and the encoder 106 are configured to execute the same processing as described with reference to FIGS. Omitted.

  That is, in the moving image frame data divided into blocks, when the movement amount between frames is large (for example, 2 pixels / frame or more), the spatial thinning process is executed in the block processing unit a204, and the movement amount between frames is small. If it is (for example, less than 1 pixel / frame), a time thinning process is executed in the block processing unit b205, and if it is in the middle (for example, 1 to 2 pixels / frame), spatial thinning and time are performed in the block processing unit c206. Perform thinning.

  After thinning processing is performed in a different manner depending on each moving amount, pack processing is performed in the pack processing units 371 to 373, and then block synthesis is performed in the block synthesis unit 105. The block synthesizing unit 105 performs a process of synthesizing blocks that have been subjected to different thinning processes into an image for N frames. The synthesized image becomes image frame data that has undergone thinning processing as different compression processing for each block, and the image is sent to the encoder 106 and encoded.

  The role of the block synthesizing unit 105 is to synthesize the blocks processed by the block processing units 204 to 206 as moving images that can be encoded by the encoder 106. In this embodiment, the processed block is reconstructed into an original frame and encoded as a moving image of 240 fps. In another embodiment, when the frame rate that can be encoded by the encoder is not 240 fps, in this embodiment, the frame rate of the input moving image is 240 fps, and the encoder 106 can be encoded by the block synthesizing unit 105. To 60 fps moving image. However, these frame rate values are merely examples for this embodiment, and any frame rate may be used in practice.

  The pack processing unit a371, the pack processing unit b372, and the pack processing unit c373 perform different operations. First, the processing of the first pack processing unit a371 will be described with reference to FIG. The input of the pack processing unit a371 is the output of the block processing unit a204. The output of the block processing unit a204 is four blocks obtained from each of four consecutive frames that have undergone pixel spatial thinning processing. That is, there are four blocks that have been subjected to space thinning in the manner described with reference to FIG.

  These blocks are represented as blocks 375 to 378 in FIG. In 1 × 4 pixels inside these blocks, for example, the pixels 380 to 383 in FIG. 18, spatial thinning is performed by the block processing unit a <b> 204, and all are converted to the same value. Specifically, the pixel values of 1 × 4 pixels in the block 375 are all p1, and the same applies to the other blocks 376 to 378. The first pack processing unit a371 performs a process of reducing the number of frames to ¼ by forming one 1 × 4 pixel area by the four pixel values p1 to p4.

  That is, a block 379 in FIG. 18 is an output of the pack processing unit a371, and a pixel value of a pixel region 384 composed of a certain 1 × 4 pixel is constituted by p1 to p4. The above is the case where the block processing apparatus performs the thinning process in the horizontal direction, that is, the thinning process for 1 × 4 pixels. In the case of 4 × 1 pixels and 2 × 2 pixels, the same pack processing is performed by the pack processing unit a371. In the example illustrated in FIG. 18, the pixel region 384 is configured by arranging the pixel values p1 to p4 in order from the left, but various methods can be applied to the order in which the pixel values are arranged. It is only necessary that the pack algorithm and the algorithm of the unpacking process executed in the decoding and pixel value interpolation processes are consistent.

  Next, the processing of the second pack processing unit b372 will be described. The input to the pack processing unit b372 is four blocks obtained from four consecutive frames in the same manner as the pack processing unit a371. Unlike the pack processing unit a371, these blocks correspond to frames that have been thinned out in the time direction. The selected representative pixel value is common to the frames and is a block having completely the same pixel value. Accordingly, a pack process is executed in which an appropriate one of the four blocks is output from the pack processing unit b372.

  Next, the process of the third pack processing unit c373 will be described with reference to FIG. The input to the pack processing unit c373 is four blocks obtained from four consecutive frames as in the other pack processing apparatuses. These are blocks subjected to space thinning and time thinning. As shown in FIG. 19, the blocks obtained from the first and second frames, that is, the block 385 and the block 386, have the same pixel values (P1) 391 and 392. The blocks obtained from the third and fourth frames, that is, the block 387 and the block 388 also have the same pixel value (P3) 393, 394.

  The pack processing unit c373 executes the pack process using two blocks, a block 386 corresponding to the second frame and a block 387 obtained from the third frame. The pack processing unit a371 described with reference to FIG. 18 performs processing in units of 4 pixels, but the pack processing unit c373 performs processing in units of 2 pixels. That is, as shown in pixels 391 to 394 in FIG. 19, two adjacent pixels have the same value, and these pixels are arranged like a pixel 395 to form the output block 389, thereby reducing the frame rate to ¼. Can be. Note that FIG. 19 shows the thinning process in units of 1 × 2 pixels, but the same applies to the case of performing processing in units of 2 × 1 pixels. Further, in the pixel 395 of the output block 389 in FIG. 19, p2 is set to the left and p3 is set to the right. However, various methods can be applied to the order in which the pixel values are arranged. It is only necessary that the pack algorithm and the algorithm of the unpacking process executed in the decoding and pixel value interpolation processes are consistent.

  Next, the configuration and processing of a moving image restoration apparatus that performs moving image expansion processing will be described with reference to FIG. FIG. 20 is a block diagram illustrating a configuration of the moving image restoration apparatus.

  The moving image restoration apparatus according to the present embodiment executes “(1) basic configuration of moving image conversion apparatus using super-resolution effect” or “(2) moving image compression processing according to the present invention” described above. In accordance with the apparatus and method, the apparatus performs decoding and decompression processing of compression-encoded data that has been subjected to spatial thinning and time-direction thinning processing according to the amount of block movement. The moving image restoration apparatus shown in FIG. 20 receives encoded moving image data and decodes it by the decoder 400. The decoding process in the decoder 400 is, for example, a decoding process of MPEG encoded data. The block movement amount acquisition unit 402 acquires the block movement amount decoded by the decoder 400 as a motion vector, and uses the acquired motion vector as the block movement amount for time thinning block determination unit 403 and space thinning block determination unit 404. Send to.

  The time-decimated block determination means 403 determines whether or not each block in the frame is thinned out in the time direction from the block movement amount sent from the block movement amount acquisition means 402, and if it is thinned out, It is determined whether the frame is thinned out, and the determination result and the block movement amount are sent to the block interpolation unit 401.

  The spatial thinning block determination unit 404 determines whether or not each block in the frame is thinned out in the spatial direction from the block movement amount sent from the block movement amount acquisition unit 402, and how much is thinned out. It is determined whether thinning is performed, and the determination result and the block movement amount are sent to the block interpolation means 401.

  The block interpolating unit 401 sends the block thinned out in the time axis direction in the decoded moving image sent from the decoder 400 from the time thinning block determining unit 403 and the space thinning block determining unit 404. Judgment is made based on the determination result, and interpolation is performed from the intra-frame block that is not thinned out to the intra-frame block that is determined to be thinned out in the time axis direction.

  FIG. 21 shows another configuration of the moving image restoration apparatus. The moving image restoration apparatus shown in FIG. 21 is different from the configuration shown in FIG. 20 in that it includes a block thinning information acquisition unit 411. In the moving image restoration apparatus shown in FIG. 21, encoded moving image data is input and decoded by the decoder 400. This decoding processing is, for example, decoding processing of MPEG encoded data. The block movement amount acquisition unit 402 detects a motion vector from the decoded moving image sent from the decoder 400 and sends the detected motion vector as a block movement amount to the block interpolation unit 401. The block decimation information acquisition unit 411 acquires the block decimation information decoded by the decoder 400 and sends it to the time decimation block determination unit 403 and the space decimation block determination unit 404.

  The time decimation block determination unit 403 determines whether each block in the frame is decimation in the time direction from the block decimation information sent from the block decimation information acquisition unit 411, and if it is decimation It is determined which frames are thinned out, and the determination result and the block movement amount are sent to the block interpolation means 401.

  The spatial thinning block determination unit 404 determines whether or not each block in the frame is thinned in the spatial direction from the block thinning information sent from the block thinning information acquisition unit 411, and if it is thinned It is determined how much is thinned out, and the determination result and the block movement amount are sent to the block interpolation means 401.

  The block interpolating unit 401 sends the block thinned out in the time axis direction in the decoded moving image sent from the decoder 400 from the time thinning block determining unit 403 and the space thinning block determining unit 404. Judgment is made based on the determination result, and interpolation is performed from the intra-frame block that is not thinned out to the intra-frame block that is determined to be thinned out in the time axis direction.

  Next, the configuration and processing of the block interpolation unit 401 will be described with reference to FIG. FIG. 22 is a block diagram showing the configuration of the block interpolation means 401 and the peripheral configuration.

  The decoder 400 performs decoding of the moving image data encoded in the moving image conversion apparatus described with reference to FIG. For example, an MPEG decoding process is executed. The data obtained as a result of this decoding is the thinned data subjected to the pack processing described above with reference to FIGS.

  When a moving image encoded as a 240 fps moving image, which is one of the embodiments, is input to the decoder 400, the decoding device 400 decodes the encoded data, and a moving image at the time of thinning processing in the moving image conversion apparatus Output as a moving image with the same frame rate as the image. In another embodiment, since the frame rate of the input moving image is 240 fps and the frame rate at the time of encoding is 60 fps, it is decoded into a 60 fps moving image and then output as a 240 fps moving image. However, these frame rate values are merely examples for this embodiment, and any frame rate may be used in practice.

  FIG. 23 is a block diagram showing a configuration of the decoder 400. The operation of the decoder in this embodiment will be described below. The moving image encoded by the encoder 106 in the moving image conversion apparatus shown in FIG. 17 is decoded by the moving image decoding unit 521 shown in FIG. At the same time, if there is block movement amount information obtained by encoding the flag 207 indicating the movement amount output from the movement amount detection unit 202 in the moving image conversion apparatus shown in FIG. Decoded by the flag decoding unit 522 of the decoder 400 shown in FIG.

  The decoded moving image is decomposed into a certain size (8 pixels × 8 pixels, 16 pixels × 16 pixels, etc.) in the block dividing unit 523 for each frame, and sent to the unpack processing unit a 524, the unpack processing unit b 525, and the unpack processing unit c 526. At this time, the divided blocks are distributed to the unpack processing units a524 to c526 based on the decoded block movement amount or the block distribution condition flag.

  For example, the block dividing unit 523 distributes each block to the unpack processing units a524 to c526 using the block distribution condition flag. If the value of the flag is not decimation in the time direction, it is distributed to the unpack processing unit a 524, if it is decimation in the time direction 1/2, it is distributed to the unpack processing unit c 526, and if it is decimation in the time direction 1/4, it is distributed to the unpack processing unit b 525.

  Further, when performing distribution using the block movement amount, the distribution condition must match the thinning condition in the block processing units 204 to 206 in the moving image conversion apparatus shown in FIG. In this embodiment, when both the horizontal and vertical movement amounts of the block are 2 pixels or more, it is determined that the block is a block in which only spatial thinning has been executed, and unpacking that performs unpacking processing corresponding to spatial thinning is executed. When the block is output to the processing unit a 524 and the amount of movement in both the horizontal direction and the vertical direction is less than 1 pixel, it is determined that the block is only subjected to thinning in the time direction, and unpacking corresponding to time thinning is performed. When the block is output to the unpack processing unit b525 that executes processing, and either the horizontal or vertical movement amount is less than 2 pixels or more than 1 pixel, both spatial thinning and time thinning are performed. And the block is output to the unpack processing unit c526 that executes the unpacking processing corresponding to the thinning-out.

  The unpack processing units a 524 to c 526 correspond to the pack processing units a 371 to c 373 in the moving image conversion apparatus shown in FIG. 17, respectively, and unpack the blocks that have been packed by the pack processing units a 371 to c 373. Perform the return operation.

  First, processing of the unpack processing unit a 524 corresponding to the pack processing unit a 371 in the moving image conversion apparatus shown in FIG. 17 will be described with reference to FIG. The input of the unpack processing unit a524 is the block output from the block dividing unit 523, that is, the four blocks obtained from four frames in the pack processing unit a371 in the moving image conversion apparatus shown in FIG. It is a block. The packed block is block 379 shown in FIG.

  The 1 × 4 pixels inside this block, for example, the pixel 384 of the block 379 in FIG. 18, are the pixels p1 to p4 for four frames packed in the pack processing unit a371 in the moving image conversion apparatus shown in FIG. The unpack processing unit a524 expands the four pixels p1 to p4 to a block for four frames, thereby quadrupling the number of frames.

  That is, a block 379 in FIG. 18 is an input of the unpack processing unit a 524, and the unpack processing unit a 524 expands the block 379 to four frames to constitute and output four blocks 375 to 378. The above is a case where the block processing apparatus performs horizontal thinning processing, that is, thinning processing for 1 × 4 pixels, and the same unpacking processing is performed in a block in which thinning processing has been performed for 4 × 1 pixels and 2 × 2 pixels. a524.

  Note that the order of the pixels p1 to p4 of the packed block assumed by the unpack processing unit a524 is required to match the pixel order when processed in the pack processing unit a371 in the moving image conversion apparatus shown in FIG. It is.

  Next, the process of the unpack processing unit b525 that executes the unpacking process corresponding to the second pack processing unit b372 in the moving image conversion apparatus illustrated in FIG. 17 will be described. Since the second pack processing unit b372 in the moving image conversion apparatus shown in FIG. 17 outputs an appropriate one of the four blocks obtained from the four frames formed by the time direction thinning, the unpacking process is performed. In the part b525, the obtained one block is simply copied to four blocks, and the number of frames is quadrupled by extending it to four frames.

  Next, the process of the unpack processing unit c526 that executes the unpacking process corresponding to the packing process of the third pack processing unit c373 in the moving image conversion apparatus shown in FIG. 17 will be described with reference to FIG. The input of the unpack processing unit c526 is a block output from the block dividing unit 523, and corresponds to a block in which both spatial thinning and time thinning are executed as thinning processing. This is a block obtained by packing four blocks obtained from four frames in the third pack processing unit c373 in the moving image conversion apparatus shown in FIG. The packed block is block 389 shown in FIG.

The 1 × 2 pixels inside this block, for example, the pixel 495 of the block 389 shown in FIG. 19, are four frames of pixels p1 to p3 packed in the third pack processing unit c373 in the moving image conversion apparatus shown in FIG. p2. The unpack processing unit c526 doubles the number of frames by expanding the two pixels p1 and p2 to a block of 4 frames.
That is, a block 389 in FIG. 19 is an input of the unpack processing unit c526, and the unpack processing unit c526 expands the block 389 to 4 frames to form and output four blocks 385 to 388. In the figure, the thinning process in units of 1 × 2 pixels is shown, but the same applies to the case where the process is performed in units of 2 × 1 pixels. However, when the packed blocks assumed by the unpack processing unit c526 are copied to the frames of the elements p1 and p2 and the order thereof is processed by the third pack processing unit c373 in the moving image conversion apparatus shown in FIG. It is necessary to match the pixel position with the pixel order.

  As described above, the unpack processing is performed in the unpack processing units a524 to c526, and the moving image whose frame rate has been converted to four times is input to the block synthesis unit 527 and synthesized for each frame. Is output as a moving image having the same frame rate as the moving image input to the moving image conversion apparatus shown in FIG.

  The moving image decoded by the decoder 400 in FIG. 22 is stored for L frames in the image storage unit 500 in the block interpolation unit 401. In addition, when the movement amount detection unit 508 described later detects the block movement amount from the accumulated frames, it is necessary to accumulate the frames necessary for the movement amount detection. In this embodiment, L = 5. is there. Further, when the movement amount decoded by the decoder is used as the movement amount, the same number of frames as the image storage unit 200 used at the time of encoding, that is, the image storage unit 200 of the moving image conversion apparatus shown in FIG. It is desirable to accumulate, and in this embodiment L = 4.

  The frames accumulated in the image accumulation unit 500 in the block interpolation unit 401 shown in FIG. 22 are decomposed into a certain size (8 pixels × 8 pixels, 16 pixels × 16 pixels, etc.) in the block dividing unit 501 and sent to the block distribution unit 502. Note that this processing block must be the same as the block size divided by the block dividing unit 201 used at the time of encoding in the moving image conversion apparatus shown in FIG.

  The frame accumulated in the image accumulation unit 500 and the size of the block divided by the block division unit 501 are sent to the block movement amount acquisition unit 402. The block movement amount acquisition unit 402 detects the block movement amount decoded by the decoder 400 as a motion vector in the movement amount detection unit 508, and uses the motion vector as the block movement amount as an output 407, and the time-decimated block determination unit 403. Send to.

  The time-decimation block determination means 403 determines the blocks that are thinned out in the time direction from the input block movement amount, and outputs the thinning-out amount in the time direction of each block and the input block movement amount to the block distribution unit 502. To do. This determination must match the thinning-out condition executed in any of the block processing units a204 to c206 among the distribution conditions of the block distribution unit 203 in the moving image conversion apparatus shown in FIG. For this reason, in this embodiment, when the block movement amount obtained from the block movement amount acquisition unit 402 is less than one pixel in both the horizontal direction and the vertical direction, the block movement amount is thinned out to 1/4 in the time direction. When either the direction or vertical movement amount is less than 2 pixels and 1 pixel or more, it is determined that the time direction decimation has been executed.

  In the moving picture restoration apparatus having the configuration shown in FIG. 21, the decoder 400 decodes the time-axis thinning block determination flag included in the input data, and outputs it to the block thinning information acquisition unit 411. The block decimation information acquisition unit 411 outputs the acquired flag to the block movement amount acquisition unit 402 and the time decimation block determination unit 403. The block movement amount acquisition unit 402 obtains the block movement amount from the time axis thinning block determination flag.

  In the moving image restoration apparatus having the configuration shown in FIG. 21, three flags of no time direction decimation, 1/2 decimation in time direction, and decimation in 1/4 in time direction are used as flags, and the values are up, down, left, and right. The block movement direction flag is used, and the block movement amount is acquired from the determination flag, the block movement direction flag, and the distribution condition of the block distribution unit 203 (see FIG. 17) used at the time of encoding. The movement amount of the block is made to correspond to the distribution condition of the block distribution unit 203. If the time direction is decimated by 1/2, the movement amount is 1 pixel movement direction as the direction of the movement direction flag. The amount was 0 pixels.

  As another configuration example, in the movement amount detection unit 508 in the block movement amount acquisition unit 402 shown in FIG. 22, the frames accumulated in the image accumulation unit 500 are spaced in the R-th and S-th time axis directions. A motion vector related to each block of the frame is obtained using an existing method, for example, a block matching method, in a block unit that matches the size of the block from which the frame from the block dividing unit 501 is obtained by acquiring an undrawn frame It is good also as a structure which detects. In this configuration example, the decoded time-axis thinning block determination flag output as the output 507 from the decoder 400 is used to determine whether or not the time-axis is thinned. = 2, S = 3, and R = 1 and S = 3 when decimating in the time direction 1/2, and R = 1 and S = 5 when decimating in the time direction ¼. This value must be the same as the value of the frame that is not thinned out in the time axis direction on the encoding side.

  In the time decimation block determination unit 403 in FIG. 22, the decoded time axis decimation block determination flag output as the output 407 from the decoder 400 or the time axis decimation block determination flag output from the block decimation information acquisition unit 411. Is input to the block distribution unit 502 based on the flag. In this embodiment, three flags are used as flags: no time direction decimation, time direction 1/2 decimation, and time direction 1/4 decimation.

  In the block distribution unit 502, block data that has been subjected to thinning processing based on the thinning-out amount in the time direction obtained from the thinning-out block determination unit 403 for the input N frames of blocks is selected from any of the block interpolation units 504 to 505. In addition, the temporal thinning amount and the block movement amount obtained from the temporal thinning block determination means 403 are also output together. In addition, a block without thinning out in the time direction is output to the block synthesis unit 506 as an output 503 without performing block interpolation processing.

  In this embodiment, when the block movement amount detected by the block movement amount acquisition unit 402 is less than one pixel in both the horizontal direction and the vertical direction of the block, the block interpolation unit b505, the horizontal direction, and the vertical direction When any of the movement amounts is less than 2 pixels and 1 pixel or more, the output to the block interpolation unit a504 is performed. That is, the block that is thinned by ¼ in the time direction is output to the block interpolation unit b505, and the block that is thinned by ½ in the time direction is output to the block interpolation unit a504. The block interpolation unit performs an interpolation process according to the thinning mode.

  A block synthesizing unit 506 performs a process of synthesizing blocks with or without different interpolation processing into images for N frames. The synthesized image becomes a decoded moving image in which different interpolation processing is performed for each block.

  Next, block interpolation processing executed by the block interpolation unit a504 and the block interpolation unit b505 in this embodiment will be described. 24 is a block diagram showing the configuration of the block interpolation unit b505, and FIG. 25 is a block diagram showing the configuration of the block interpolation unit a504.

  First, the configuration and processing of the block interpolation unit b505 shown in FIG. 24 will be described. Decoded blocks are stored in the block storage unit 541 for N frames. The blocks accumulated in the block accumulation unit 541 are moved in the block movement processing unit 542 according to the block movement amount. The block movement processing unit 542 acquires the blocks thinned out in the time direction from the block storage unit 541, and based on the thinning amount in the time direction and the block movement amount output from the block distribution unit 502 (see FIG. 22). Move on the frame in the block movement direction.

  FIG. 26 shows a specific example of block movement processing in the block movement processing unit 542. In FIG. 26, the Pth frame is denoted by 801, the next P + 1th frame in the time direction is denoted by 802, the block 803 is a block that has not been thinned out, and the block 804 is unblocked by the unpack processor b525 and unpack processor c526. Indicates the block that was copied.

The block movement processing unit 542 moves such a copied block on the frame by the block movement amount acquired from the block distribution unit 502 (see FIG. 22). In FIG. 26, assuming that the moving direction of Q pixels / frame is acquired in the right direction, the block is moved in the right direction of Q pixels, and the moved block is a block 806. The gap 809 generated when moving may leave a pixel when the block is copied, or may be interpolated from surrounding pixels.
By this processing, the block whose time is thinned out is in the same position as the block before encoding, and it is expected that the original image can be restored.

  The block synthesizing unit 543 performs a process of synthesizing the block that has been subjected to the block movement process and the block that has not been performed into an image of N frames. The synthesized blocks of N frames become decoded moving images after the band limiting unit 544 performs filtering in the time direction only for the blocks that have undergone interpolation accompanying movement by the block movement processing unit 542. .

  In the band limiting unit 544, a low-pass filter is mainly used, but in this embodiment, blur processing which is a special case of the low-pass filter, that is, blurring processing is performed. In this embodiment, the movement amount of the block input to the block interpolation unit c505 (see FIG. 22) is a block in which the movement amount in the horizontal direction and the vertical direction are both less than one pixel, so the movement amount of the block over N frames. Is small. For this reason, when the movement process is performed in units of pixels, blocks that are not subjected to the movement process are generated. Therefore, the blur process is performed in order to further improve the unnaturalness of the movement. The effect of side effects caused by blur processing that blurs the image is considered to be negligible due to the small amount of movement.

The blur process in the present embodiment is executed as a process for calculating an output pixel value from an input pixel value according to the following expression (Expression 3).

  In the above equation, in () is the input pixel of the band limiting unit 544 output from the block synthesizing unit 543, out () is the output pixel, f is the frame number x, y is the in-frame pixel coordinate value, and mod is the remainder operator. It is.

  Next, the block interpolation unit a504 will be described with reference to FIG. Since the processing flow is almost the same as that of the block interpolation unit b505, only the differences will be described. In the block interpolation unit a504, unlike the block interpolation unit b505, in the block movement device 602, the block movement selection unit 606 selects two types of block movement processing units 607 and 608 according to the block movement amount, and block movement processing is performed. I do. This is because the frame thinned out in the block in which the time direction ½ thinning is performed may be changed depending on the block moving speed.

  For example, in the above-described [(2) Apparatus and method for executing moving image compression processing according to the present invention], as described with reference to FIGS. In the conversion device, for example, when the time direction thinning process with a compression ratio of 1/2 is executed in units of four consecutive frames, if the movement amount is large, the first frame and the third frame are left, and the second Performs thinning processing in the time axis direction to thin out the fourth frame, and if the amount of movement is small, the thinning processing in the time axis direction to thin out the second and third frames while leaving the first frame and the fourth frame Corresponds to the case of executing.

  As described above, in the four frames of P to P + 3 frames, if the block movement amount is 1 pixel / frame or more and less than 1.25 pixels / frame, the blocks of P + 1 and P + 2 frames are thinned out, and 1.25 pixels / frame or more are obtained. Since restoration processing corresponding to a configuration in which blocks of P + 1 and P + 3 frames are thinned out if it is less than 2 pixels / frame, the moving direction of the restoration blocks can be changed to correspond to thinning processing.

  Here, the given block movement amount is given as a block movement amount in the direction in which the frame increases in the time axis direction. When P + 2 frame blocks are thinned out, P + 2 frame is closer in time to P + 3 frame. It is preferable to interpolate from these blocks. Therefore, when the blocks of P + 1 and P + 2 frames are thinned out, the backward block movement processing unit 608 is used for the P + 2 frame, and the block movement amount is reversed and the interpolation is performed from the P + 3 frame.

  Similar to the block interpolation unit b505, the remaining blocks are interpolated using the block movement amount from the block of the previous frame using the forward block movement processing unit 607. Subsequently, blocks that have not been thinned out and blocks that have been thinned out by time and interpolated by block movement are combined into frames by the image storage unit 604 and then stored for N frames.

  The image once stored in the image storage unit 604 is subjected to interpolation processing in the gap generated by the block movement by the block boundary interpolation unit 605. In the gap interpolation processing, the gap 809 generated by the block movement processing of the block movement processing unit described with reference to FIG. 26 is detected from the thinning amount in the time direction to the block thinned in the time direction, and the gap is further blocked. The gap 809 is obtained by calculating from the movement amount, and the gap 809 is filled with pixels interpolated from surrounding pixels. In this embodiment, linear interpolation from surrounding pixels is performed as this interpolation processing.

  With this process, natural block interpolation is performed without band limitation on blocks that are blurred due to temporal band limitation performed to improve the unnaturalness of motion, that is, blocks that have a relatively high block movement speed. It can be performed.

  In the above block interpolation processing, in this embodiment, the interpolation processing is performed in the block interpolator b505 with a band limiting device at the time direction 1/4 decimation, and the block interpolation without the band limiting device is performed at the time direction 1/2 decimation. Although the interpolation processing is performed in the part a504, these distribution methods are only examples. That is, both may perform the interpolation processing in the block interpolation unit b505 with the band limiting unit, or both may perform the interpolation processing in the block interpolation unit a504 without the band limiting unit. Further, the interpolation process may be performed in the block interpolation unit a504 without the band limiting unit at the time direction 1/4 decimation, and the interpolation process may be performed in the block interpolation unit b505 with the band limiting unit at the time direction 1/2 decimation. .

  As described above, according to the moving image restoration apparatus and method of this embodiment, moving image conversion data that has been subjected to thinning processing in the time direction is input in units of blocks, and a plurality of blocks corresponding to different frames in the time axis direction are input. A block interpolation unit that calculates pixel values by interpolation processing and generates a restoration block of a plurality of frames, inputs movement amount information corresponding to the block to be interpolated, and moves the restoration block between frames according to the movement amount information. Since the moving image data is restored, it is possible to generate restored data having a super-resolution effect, and it is possible to generate high-quality decoded data that suppresses image degradation.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the moving image conversion apparatus and method of the present invention, it is possible to generate data having a super-resolution effect by changing the mode of thinning in the time direction according to the amount of movement. Even though the amount of data is reduced, it is possible to generate high-quality data that suppresses image degradation during decoding.

  According to the moving image conversion apparatus and method of the present invention, block division is performed for each frame constituting moving image data, a subject movement amount in each block is detected, and spatial thinning processing in a mode determined based on the movement amount or Since at least one of the time direction thinning-out processes is executed, and in the execution of the time direction thinning-out process, the thinning process is performed by changing the thinning target frame based on the movement amount information. It is possible to generate data with the optimal time-direction decimation, that is, the super-resolution effect, and generate high-quality data that suppresses image degradation during decoding, even though the amount of data is reduced can do.

  Furthermore, according to the moving image restoration apparatus and method of the present invention, moving image conversion data that has been subjected to thinning processing in the time direction is input in units of blocks, and pixel values of a plurality of blocks corresponding to different frames in the time axis direction are interpolated. In the block interpolation unit that calculates by processing and generates a restoration block of a plurality of frames, the movement amount information corresponding to the block to be interpolated is input, and the restoration block is moved between frames according to the movement amount information. Since the restoration is configured, restoration data having a super-resolution effect can be generated, and high-quality decoded data in which image degradation is suppressed can be generated.

It is a figure which shows the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure which shows the detail of the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure which shows the structure of the moving image converter of this invention. It is a figure which shows the structure of the block division | segmentation means and block processing part in the moving image converter of this invention. It is a figure which shows the structure of the block process part which performs space thinning and time thinning in the moving image converter of this invention. It is a figure explaining the specific example of the time decimation process of the block process part which performs the space decimation and the time decimation in the moving image converter of this invention. It is a figure explaining the specific example of the time decimation process of the block process part which performs the space decimation and the time decimation in the moving image converter of this invention. It is a flowchart explaining the process sequence of the block process part which performs space thinning and time thinning in the moving image converter of this invention. It is a figure explaining the reason a super-resolution effect is brought about by changing a time thinning-out processing mode according to a movement amount in a block processing part which performs space thinning and time thinning-out in a moving picture conversion device of the present invention. It is a figure explaining the reason a super-resolution effect is brought about by changing a time thinning-out processing mode according to a movement amount in a block processing part which performs space thinning and time thinning-out in a moving picture conversion device of the present invention. It is a figure explaining the reason a super-resolution effect is brought about by changing a time thinning-out processing mode according to a movement amount in a block processing part which performs space thinning and time thinning-out in a moving picture conversion device of the present invention. It is a figure explaining the reason a super-resolution effect is brought about by changing a time thinning-out processing mode according to a movement amount in a block processing part which performs space thinning and time thinning-out in a moving picture conversion device of the present invention. It is a figure which shows the structure of the moving image converter which produces | generates the encoding data which the moving image decompression | restoration apparatus of this invention performs decompression | restoration. It is a figure explaining the specific example of a process of the pack process part of the moving image converter shown in FIG. It is a figure explaining the specific example of a process of the pack process part of the moving image converter shown in FIG. It is a figure which shows the structure (example 1 of a structure) of the moving image decompression | restoration apparatus of this invention. It is a figure which shows the structure (example 2 of a structure) of the moving image decompression | restoration apparatus of this invention. It is a figure which shows the block interpolation means and peripheral structure of the moving image decompression | restoration apparatus of this invention. It is a figure which shows the structure of the decoder of the moving image decompression | restoration apparatus of this invention. It is a figure which shows the structure of the block interpolation part b of the moving image decompression | restoration apparatus of this invention. It is a figure which shows the structure of the block interpolation part a of the moving image decompression | restoration apparatus of this invention. It is a figure explaining the interpolation process which the block interpolation part of the moving image decompression | restoration apparatus of this invention performs.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image converter 11 Block division part 12 Movement amount detection part 13 Block processing part 14 Output part 21 Image storage part 22 Block division part 31 Movement amount detection part 32 Block distribution part 51-53 Block processing part 100 Block division means 102-104 Moving picture compression means 105 Block composition means 106 Encoder 200 Image storage section 201 Block division section 202 Movement amount detection section 203 Block distribution section 204 to 206 Block processing section 207 Output (movement amount information)
208 Block processor 209 Output (Block distribution flag)
301 Block Accumulation Unit 302 Block Distribution Unit 303 to 305 Block Processing Unit 321 to 324 Block 326 to 329 Pixel 331 to 334 Block 336 and 337 Pixel 351 to 354 Block 356 to 359 Pixel 361 to 364 Block 366 and 367 Pixel 371 to 373 Pack Processing unit 375 to 379 block 380 to 384 pixel 385 to 389 block 391 to 395 pixel 400 decoder 401 block interpolation means 402 block movement amount acquisition means 403 time decimation block determination means 404 spatial decimation block determination means 411 block decimation information acquisition means 500 Image storage unit 501 Block division unit 502 Block distribution unit 503 Output 504, 505 Block interpolation unit 506 Block composition unit 507 Time axis decimation block Determination flag 508 movement amount detection unit 521 video decoding unit 522 flag decoding unit 523 block division unit 524 to 526 unpack processing unit 527 block synthesis unit 541 block storage unit 542 block movement processing unit 543 block synthesis unit 544 bandwidth limitation Section 601 Block accumulation section 602 Block movement processing section 603 Block composition section 604 Image accumulation section 605 Block boundary interpolation section 606 Block movement selection section 607 Forward block movement processing section 608 Reverse block movement processing section 801, 802 Frame 803, 804 blocks 805, 806 block 807, 808 frame 809 gap

Claims (20)

A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit for detecting a subject movement amount in each block divided by the block division unit;
A block processing unit that inputs the block data divided by the block dividing unit and the movement amount information detected by the movement amount detection unit, and executes block data thinning processing;
The block processing unit
The block to be thinned out is compared with the movement amount detected by the movement amount detection unit and the first threshold value and the second threshold value that are defined in advance to determine the processing mode.
If the detected amount of movement is greater than or equal to the first threshold, the amount of movement is large,
If the detected amount of movement is less than the first threshold and greater than or equal to the second threshold, the amount of movement is medium,
If the detected amount of movement is less than the second threshold, the amount of movement is small,
And change the thinning mode to be executed according to whether the movement amount belongs to the large, medium, or small classification,
When the amount of movement is large, spatial thinning processing,
When the amount of movement is small, thinning process in the time direction,
When the amount of movement is medium, execute a space thinning process and a time direction thinning process,
In the time direction thinning process when the movement amount is medium, the thinning process is performed by changing the thinning target frame based on the movement amount information.
The block processing unit
In the execution of the time direction thinning process, the thinning process is performed by changing the thinning target frame based on the movement amount information.
Formula defining the pre-defined super-resolution effect generation condition, that is, the following formula specifying the super-resolution effect generation condition when an object moving at a speed of v pixels / frame is thinned out in the spatial direction by m pixels,

Where t is time, T is frame interval,
A thinning mode that satisfies the super-resolution effect generation condition shown in the above formulas (formula a) and (formula b) is determined based on the movement amount information, and the thinning target frame is changed so as to match the determined thinning mode. A moving image conversion apparatus characterized in that the thinning process is performed. The block processing unit
In the execution of the thinning process in the time direction that is executed when the movement amount that is less than the first threshold is medium or small ,
A third threshold that is a movement amount threshold that is a thinning mode change point set according to a correspondence relationship between a movement amount that satisfies the super-resolution effect generation condition and a thinning mode, and a third threshold that is a threshold value less than the first threshold and the movement amount Perform a comparison with the information,
If the movement amount information is equal to or greater than the third threshold , set the thinning target frames that are continuous on the time axis to be smaller than when the movement amount information is less than the third threshold ;
When the movement amount information is less than the third threshold value , thinning target frames that are continuous on the time axis are thinned out in the time axis direction that is set more than when the movement amount information is equal to or more than the third threshold value. The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to execute processing. The block processing unit
When the configuration is such that the time direction decimation processing with a compression rate of 1/2 is executed with four consecutive frames as processing units,
When the movement amount information is equal to or greater than the third threshold , the first frame and the third frame are left, the second and fourth frames are thinned out, and a time-axis direction thinning process is performed.
If the movement amount information is less than the third threshold value , the first frame and the fourth frame remain, and the time-axis direction thinning process is performed to thin out the second and third frames. The moving image conversion apparatus according to claim 2, wherein: A moving image restoration device that executes restoration processing of moving image conversion data including thinning processing data in the time direction,
Block interpolation that inputs moving image conversion data that has been thinned in the time direction in units of blocks, calculates pixel values of multiple blocks corresponding to frames in different time axis directions, and generates a restored block of multiple frames Part
The block interpolation unit
The input block of frames not decimated time, to generate a copy and restore block frame decimation more temporally close in time, moves according to the movement amount information input block corresponding, the only movement amount restored block between frames And moving image data restoration,
The input block to be restored is a block that is subjected to thinning processing by changing the thinning target frame based on the movement amount information, and is an expression that defines the super-resolution effect generation condition, that is, the speed v pixel / frame. The following formula specifying the super-resolution effect generation condition when moving objects are thinned out in the spatial direction with m pixels,

Where t is time, T is frame interval,
It is a block that has been subjected to thinning processing by changing the thinning target frame based on the movement amount information so as to satisfy the super-resolution effect generation condition represented by the above formulas (formula a) and (formula b),
The block interpolation unit
In order to restore the frame thinned out by the thinning process based on the movement amount information, the moving direction of the restored block between the frames is changed according to the mode of the thinning process in the time direction of the block input as the restoration target. A moving image restoration apparatus characterized in that moving image data restoration is performed by selecting so that movement between frames is closer in time. The block interpolation unit
5. The moving image restoration apparatus according to claim 4, further comprising: a band limiting process based on a filter process for a block that has undergone interpolation with movement. The moving image restoration device includes:
Based on the movement amount of the restoration target block, it is determined whether or not the block is data that has been thinned in the time direction,
The block interpolation unit
The block that has been thinned out in the time direction selected based on the determination is configured to perform moving image data restoration by moving the restoration block by the amount of movement between frames according to movement amount information. The moving image restoration apparatus according to claim 4. The moving image restoration device includes:
Based on a flag indicating whether or not the data is thinned out in the time direction, it is determined whether or not the restoration target block is data that has been thinned out in the time direction.
The block interpolation unit
The block that has been thinned out in the time direction selected based on the determination is configured to perform moving image data restoration by moving the restoration block by the amount of movement between frames according to movement amount information. The moving image restoration apparatus according to claim 4. The moving image restoration device includes:
It has a movement amount detection unit, and determines whether the block is data thinned out in the time direction according to the movement amount detected by the movement amount detection unit,
The block interpolation unit
The block that has been thinned out in the time direction selected based on the determination is configured to perform moving image data restoration by moving the restoration block by the amount of movement between frames according to the amount of movement. The moving image restoration apparatus according to claim 4. The movement amount detector
Executes a process for detecting a motion vector based on a corresponding block of a different frame,
The block interpolation unit
9. The moving image restoration apparatus according to claim 8, wherein the moving image data is restored by moving the restoration block by an amount corresponding to the movement amount between frames based on the motion vector. A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
A movement amount detection step of detecting a subject movement amount in each block divided in the block division step;
A block processing step for inputting the block data divided in the block division step and the movement amount information detected in the movement amount detection step, and executing a block data thinning process;
The block processing step includes:
Compare the movement amount detected in the movement amount detection step with the first threshold value and the second threshold value defined in advance for determining the processing mode, as the block to be thinned out,
If the detected amount of movement is greater than or equal to the first threshold, the amount of movement is large,
If the detected amount of movement is less than the first threshold and greater than or equal to the second threshold, the amount of movement is medium,
If the detected amount of movement is less than the second threshold, the amount of movement is small,
And change the thinning mode to be executed according to whether the movement amount belongs to the large, medium, or small classification,
When the amount of movement is large, spatial thinning processing,
When the amount of movement is small, thinning process in the time direction,
When the amount of movement is medium, execute a space thinning process and a time direction thinning process,
In the time direction decimation process when the movement amount is medium, the decimation process is performed by changing the decimation target frame based on the movement amount information,
In the execution of the time direction thinning process, as a thinning process in which the thinning target frame is changed based on the movement amount information,
Formula defining the pre-defined super-resolution effect generation condition, that is, the following formula specifying the super-resolution effect generation condition when an object moving at a speed of v pixels / frame is thinned out in the spatial direction by m pixels,

Where t is time, T is frame interval,
A thinning mode that satisfies the super-resolution effect generation condition shown in the above formulas (formula a) and (formula b) is determined based on the movement amount information, and the thinning target frame is changed so as to match the determined thinning mode. A moving image conversion method characterized by performing a thinning process. The block processing step includes:
In the execution of the thinning process in the time direction that is executed when the movement amount that is less than the first threshold is medium or small ,
A third threshold that is a movement amount threshold that is a thinning mode change point set according to a correspondence relationship between a movement amount that satisfies the super-resolution effect generation condition and a thinning mode, and a third threshold that is a threshold value less than the first threshold and the movement amount Perform a comparison with the information,
If the movement amount information is equal to or greater than the third threshold , set the thinning target frames that are continuous on the time axis to be smaller than when the movement amount information is less than the third threshold ;
When the movement amount information is less than the third threshold value , the thinning target frames that are continuous on the time axis are set in a time axis direction that is set more than when the movement amount information is greater than or equal to the third threshold value. The moving image conversion method according to claim 10, wherein thinning processing is executed. The block processing step includes:
When the time direction thinning process with a compression ratio of 1/2 is executed in units of four consecutive frames,
When the movement amount information is equal to or greater than the third threshold , the first frame and the third frame are left, the second and fourth frames are thinned out, and a time-axis direction thinning process is performed.
When the movement amount information is less than the third threshold value , the first frame and the fourth frame are left, and a thinning process in a time axis direction for thinning out the second and third frames is performed. The moving image conversion method according to claim 11. A moving image restoration method for executing restoration processing of moving image conversion data including thinning processing data in the time direction,
Block interpolation that inputs moving image conversion data that has been thinned in the time direction in units of blocks, calculates pixel values of multiple blocks corresponding to frames in different time axis directions, and generates a restored block of multiple frames Has steps,
The block interpolation step includes:
The input block of frames not decimated time, to generate a copy and restore block frame decimation more temporally close in time, moves according to the movement amount information input block corresponding, the only movement amount restored block between frames And restoring the moving image data,
The input block to be restored is a block that is subjected to thinning processing by changing the thinning target frame based on the movement amount information, and is an expression that defines the super-resolution effect generation condition, that is, the speed v pixel / frame. The following formula specifying the super-resolution effect generation condition when moving objects are thinned out in the spatial direction with m pixels,

Where t is time, T is frame interval,
It is a block that has been subjected to thinning processing by changing the thinning target frame based on the movement amount information so as to satisfy the super-resolution effect generation condition represented by the above formulas (formula a) and (formula b),
The block interpolation step includes:
In order to restore the frame thinned out by the thinning process based on the movement amount information, the moving direction of the restored block between the frames is changed according to the mode of the thinning process in the time direction of the block input as the restoration target. A method for restoring a moving image, the step of restoring moving image data by selecting so that the movement between frames is closer in time. The block interpolation step includes:
The moving image restoration method according to claim 13, further comprising a step of performing band limitation processing by filter processing on a block on which interpolation with movement is performed. The moving image restoration method further includes:
Determining whether the block is data that has been thinned in the time direction based on the amount of movement of the restoration target block;
The block interpolation step includes:
The moving image data is restored by moving the restoration block by the amount of movement between frames according to movement amount information for the block that has been thinned in the time direction selected based on the determination. 14. The moving image restoration method according to 13. The moving image restoration method further includes:
Determining whether or not the restoration target block is data that has been thinned in the time direction, based on a flag that indicates whether or not the data has been thinned in the time direction;
The block interpolation step includes:
The moving image data is restored by moving the restoration block by the amount of movement between frames according to movement amount information for the block that has been thinned in the time direction selected based on the determination. 14. The moving image restoration method according to 13. The moving image restoration method further includes:
A movement amount detection step, and according to the movement amount detected in the movement amount detection step, determine whether the block is data that has been thinned in the time direction;
The block interpolation step includes:
The moving image data is restored by moving the restoration block by the amount of movement between frames according to the amount of movement of the block that has been thinned out in the time direction selected based on the determination. 14. The moving image restoration method according to 13. The movement amount detection step includes:
Performing a process of detecting motion vectors based on corresponding blocks of different frames;
The block interpolation step includes:
18. The moving image restoration method according to claim 17, wherein the moving image data restoration is performed by moving the restoration block by the amount of movement between frames based on the motion vector. In the moving image conversion device, a computer program for executing a data conversion process of moving image data, the data conversion process ,
A block dividing step for causing the block dividing unit to execute a block dividing process for each frame constituting the moving image data;
A movement amount detection step for causing a movement amount detection unit to detect a subject movement amount in each block divided in the block division step;
The block processor, the block data obtained by dividing in the block dividing step, the moving amount detecting inputs the movement amount information detected in step, the moving image conversion apparatus and a block processing step for executing the decimation process of the block data To run
In the block processing step ,
The block to be thinned out is compared with the movement amount detected in the movement amount detection step with the first threshold value and the second threshold value that are defined in advance to determine the processing mode,
If the detected amount of movement is greater than or equal to the first threshold, the amount of movement is large,
If the detected amount of movement is less than the first threshold and greater than or equal to the second threshold, the amount of movement is medium,
If the detected amount of movement is less than the second threshold, the amount of movement is small,
And changing the thinning mode to be executed depending on whether the movement amount belongs to the large, medium, or small classification,
When the amount of movement is large, spatial thinning processing,
When the amount of movement is small, thinning process in the time direction,
When the movement amount is medium, the space thinning process and the time direction thinning process are executed,
In the time direction thinning process when the movement amount is medium, the thinning target frame is changed based on the movement amount information, and the thinning process is performed.
In the execution of the time direction thinning process, as a thinning process in which the thinning target frame is changed based on the movement amount information,
Formula defining the pre-defined super-resolution effect generation condition, that is, the following formula specifying the super-resolution effect generation condition when an object moving at a speed of v pixels / frame is thinned out in the spatial direction by m pixels,

Where t is time, T is frame interval,
A thinning mode that satisfies the super-resolution effect generation condition shown in the above formulas (formula a) and (formula b) is determined based on the movement amount information, and the thinning target frame is changed so as to match the determined thinning mode. A computer program characterized in that thinning processing is performed . In the moving image restoration apparatus, the computer program for executing the restoration processing of the moving image conversion data including the thinning processing data in the time direction, the restoration processing of the moving image data,
The moving image conversion data that has been thinned out in the time direction is input to the block interpolator in units of blocks, and pixel values of a plurality of blocks corresponding to different frames in the time axis direction are calculated by interpolation processing, and a plurality of frames are restored. The moving image restoration apparatus executes a block interpolation step for generating
In the block interpolation step ,
The input block of frames not decimated time, to generate a copy and restore block frame decimation more temporally close in time, moves according to the movement amount information input block corresponding, the only movement amount restored block between frames Let the video data be restored ,
The input block to be restored is a block that is subjected to thinning processing by changing the thinning target frame based on the movement amount information, and is an expression that defines the super-resolution effect generation condition, that is, the speed v pixel / frame. The following formula specifying the super-resolution effect generation condition when moving objects are thinned out in the spatial direction with m pixels,

Where t is time, T is frame interval,
It is a block that has been subjected to thinning processing by changing the thinning target frame based on the movement amount information so as to satisfy the super-resolution effect generation condition represented by the above formulas (formula a) and (formula b),
In the block interpolation step ,
In order to restore the frame thinned out by the thinning process based on the movement amount information, the moving direction of the restored block between the frames is changed according to the mode of the thinning process in the time direction of the block input as the restoration target. selected and so as to be more temporally move between near frames, computer program characterized by causing a moving image data reconstruction.

Images