JP4915468B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

JP 2001-352546 A

Since the amount of moving image data is very large, conventionally, when storing / transmitting a moving image, it is common to compress the amount of moving image data.
A typical method for compressing the data amount of a moving image is a moving image coding that combines inter-frame coding such as MPEG (Moving Picture Experts Group) and intra-frame coding.
By the way, in order to improve the coding efficiency of moving picture coding by the MPEG system, many methods for performing preprocessing on a moving picture have been proposed.
For example, in Patent Document 1 described above, the occurrence of block distortion after encoding is suppressed by applying a low-pass filter to the image as preprocessing.
However, the method described in Patent Document 1 is intended to suppress block distortion that occurs when encoding, and detail loss due to the application of a low-pass filter is inevitable.

In the present invention, adaptively using motion information in the moving image, applying a low-pass filter simulating the motion blur caused by the exposure period of time of imaging the image, and performs motion blur addition, by image processing, which Thus, it is assumed that the encoding efficiency is improved while presenting a natural image quality for the observer of the moving image .
Correspondingly, an object of the decoding process is to suppress image quality deterioration due to the influence of the motion blur .

The image processing apparatus according to the present invention includes a motion vector as motion information indicating the motion of an image between unit images constituting the moving image data from the encoded moving image data subjected to the motion blur addition process. Motion vector generating means for generating.
Further, the coded video data in terms of the motion blur addition process has been performed is input is decoded, with respect to the entered the moving image data, generated by the motion vector generating means The image processing apparatus includes a motion blur reduction unit that performs motion blur reduction processing by performing the filter processing using the motion vector .
Then, the motion blur reduction means selects whether or not to perform the motion blur reduction process based on the motion blur addition parameter used when the motion blur addition process is performed, or This is to adjust the degree of blur reduction processing.

The image processing method of the present invention is a motion vector as motion information indicating the motion of an image between unit images constituting the moving image data from the moving image data encoded after the motion blur addition processing. Has a motion vector generation step.
In addition, the image processing apparatus includes a decoding step for decoding the moving image data that has been subjected to the motion blur addition process .
In addition, a motion blur reduction process is performed by performing a filtering process using the motion vector generated in the motion vector generation step on the decoded moving image data obtained in the decoding step. And a motion blur reduction step.
In the motion blur reduction step, whether to execute the motion blur reduction processing is selected based on the motion blur addition parameter used when the motion blur addition processing is performed, or This adjusts the degree of motion blur reduction processing.
The program of the present invention is a program that causes an information processing apparatus to execute each step of the above-described image processing method.

That is, in the present invention, on the encoding side, a low-pass filter that simulates motion blur caused by the exposure period at the time of imaging is adaptively applied to the image using motion information in the moving image. This is based on the assumption that “blur addition” is executed. That is, it is premised on that the encoding efficiency is improved while presenting a natural image quality for the observer of the moving image, as compared with a case where a spatially uniform low-pass filter is executed.
Under this assumption, the present invention refers to motion information and performs adaptive filtering (motion blur reduction processing: high-pass filter processing) for each region. Therefore, compared to the case of executing a spatially uniform low-pass filter at the time of encoding (that is, a spatially uniform high-pass filter at the time of decoding), it is adapted for each region according to the magnitude of motion. reduction of movement blur, including the adjustment of the motion blur quantity of the post-treatment if necessary, is capable of executing, Ru can present a natural image quality to the viewer of the moving image.

In general, when moving images or animations captured with a high-speed shutter are displayed using a display device such as a projector or display, the movement of moving objects included in the images is displayed discontinuously, and observation is performed to observe the images. One perceives multiple images. This is called motion jerkiness (reference: ANSI T1.801.02-1996), and image degradation occurs frequently.
On the other hand, when a moving image captured at a low shutter speed, such as an open shutter, is displayed, the loss of details and edges of the subject often become blurred due to the influence of motion blur. This phenomenon is a phenomenon of image quality deterioration called blur (motion blur).
On the encoding side on which the present invention is premised , motion blur is added to improve the compression efficiency of encoding. This makes it possible to reduce the jerkiness deterioration while improving the coding efficiency.
And in this onset bright, improve image quality by executing blur reduction processing motion in the decoded image data that has been motion blur addition. This makes it possible to reduce both the jerkiness degradation and the blur degradation, to obtain image data before motion blur addition after decoding, and the like.

In accordance with the present invention, when the encoding side performs adaptive filtering processing as motion blur addition processing using motion information in a moving image to improve encoding efficiency and reduce jerkiness degradation. In addition, by referring to the motion information for the decoded moving image data and performing filtering processing as motion blur reduction processing adaptively for each region, jerkiness degradation and blurring that occur in the moving image data are performed. It is possible to realize natural image quality presentation with suppressed deterioration.

It is a block diagram of the 1st image processing device of an embodiment of the invention. It is explanatory drawing of the encoding efficiency improvement by motion blur addition in embodiment. It is a block diagram of other examples of composition of the 1st image processing device of an embodiment. It is a block diagram of the example of further composition of the 1st image processing device of an embodiment. It is a block diagram of the motion vector generation processing unit of the embodiment. It is a flowchart of a process of the motion vector generation process part of embodiment. It is a block diagram of the motion blur addition process part of an embodiment. It is a flowchart of the process of the motion vector mask process part of embodiment. It is a flowchart of the process of the optimal shutter speed calculation / discrimination part and motion vector correction | amendment part of embodiment. It is explanatory drawing regarding the optimal shutter speed of embodiment. It is explanatory drawing of the process of the filter parameter calculation part of embodiment. It is another block diagram of the motion blur addition process part of an embodiment. It is a block diagram of the 2nd image processing device of an embodiment. It is a block diagram of the other example of composition of the 2nd image processing device of an embodiment. It is a block diagram of the further another structural example of the 2nd image processing device of an embodiment. It is a block diagram of the motion blur reduction process part of an embodiment. It is explanatory drawing of a moving average filter. It is another block diagram of the motion blur reduction process part of an embodiment. It is explanatory drawing of transmission of the filter parameter of embodiment.

Hereinafter, embodiments of the present invention will be described in the following order. Note that the first image processing device is an image processing device that performs motion blur addition processing in the previous stage of encoding, and the second image processing device is a moving image transmitted from the first image processing device. An image processing apparatus that performs decoding and motion blur reduction processing on data.
[1. First image processing apparatus]
[1-1: Configuration Example of Image Processing Device]
[1-2: Motion vector generation processing unit]
[1-3: Motion blur addition processing unit]
[2. Second image processing apparatus]
[2-1: Configuration Example of Image Processing Device]
[2-2: Motion blur reduction processing unit]
[3. Information transmission between the first and second image processing apparatuses]
[4. program]

[1. First image processing apparatus]
[1-1: Configuration Example of Image Processing Device]

A configuration example of the first image processing apparatus, that is, an image processing apparatus that performs motion blur addition processing in the previous stage of encoding will be described with reference to FIGS.
Note that the processing blocks in each drawing are shown as logical components, and each processing block may be in the same casing or in another casing. In each figure, the image processing device 1 surrounded by a broken line or the image processing device 100 also surrounded by a broken line can be considered as the configuration of the embodiment of the first image processing device of the present invention.

  FIG. 1 shows a configuration including a motion blur addition processing unit 11 as the image processing apparatus 1. Or the structure provided with the motion blur addition process part 11 and the encoding process part 2 as the image processing apparatus 100 is shown.

The motion blur addition processing unit 11 adds motion blur to the input moving image data ID, and outputs the motion blur to the subsequent encoding processing unit 2.
The motion blur addition processing unit 11 receives the moving image data ID and the motion information MI of each region of each frame of the moving image data ID (the region is a pixel unit or a pixel block unit of a plurality of pixels). Then, using the motion information MI, the moving image data ID (each frame and each divided region) is adaptively filtered to add motion blur.

The motion blur image data BD subjected to the motion blur addition process is input to the encoding processing unit 2 and is encoded.
In the encoding processing unit 2, for example, compression processing such as encoding conforming to the MPEG (Moving Picture Experts Group) standard is executed.
The encoded data ED obtained by compressing the motion blurred image data BD has improved compression efficiency as compared with data encoded by directly inputting the moving image data ID to the encoding processing unit 2. .

The reason why compression efficiency is improved by adding motion blur to moving image data will be described with reference to FIG.
Here, the compression efficiency is improved particularly by applying a low-pass filter, but adding motion blur generated by the exposure period during imaging is a kind of execution of the low-pass filter. It can be said that there is.

ここで、共分散はＣｆｇ、画像信号はｆ（ｘ），ｇ（ｘ）、μｆ，μｇはμｆ＝Ｅｘ［ｆ（ｘ）］，μｇ＝Ｅｘ［ｇ（ｘ）］によって与えられる画素値の空間平均値である。 In general, image data can be compressed because it has the property that the correlation between neighboring pixels is high. To increase the compression rate, the neighboring pixel correlation may be increased.
Here, the image is regarded as a one-dimensional signal, and covariance is introduced.

Here, the covariance is Cfg, the image signals are f (x) and g (x), μf and μg are pixel values given by μf = Ex [f (x)] and μg = Ex [g (x)]. Spatial average value.

  This (Equation 1) has the property that if f (x) and g (x) are close, it will be close to the dispersion value of f (x) or g (x) and close to 0 if it is far away. Cfg quantifies the proximity of two waveforms.

このＣｆｆ（ｕ）を自己共分関数と呼ぶ。 When compressing an image, the image signal is only f (x), and it is necessary to quantify the proximity of neighboring pixels.
Therefore, a signal obtained by shifting the image signal f (x) by the displacement u (integer) is set as g (x) = f (x + u) (equation 1).
At this time, Ex [f (x)] and Ex [g (x)] = Ex [f (x + u)] are equal to each other and expressed as μ (Equation 2).

This Cff (u) is called a self-shared function.

Based on (Expression 2), the auto-covariance in the actual image signal f (x) is plotted.
It should be noted that Cff (u) is not the magnitude of the value, but how fast it attenuates as it moves away from the origin u = 0.
First, FIG. 2A shows a case where a signal having no change is added. In this case, it can be easily understood that Cff (u) becomes 0 from the definition of (Equation 2).
Next, FIG. 2B shows the autocovariance in the image signal f (x) that changes drastically.
When there is a drastic change in the signal, if the signal is shifted by the displacement u, the positive and negative values cancel each other, so that the autocovariance converges immediately from u = 0 to 0.
Subsequently, the self-covariance in the image signal f (x) that changes gradually is shown in FIG.
Since the signal is gentle, the auto-covariance gradually attenuates from u = 0 because there is a correlation between the pixels even if the signal is shifted by the displacement u.
That is, the auto-covariance is close to a discrete delta function that quickly converges from u = 0 to 0 when the correlation between pixels is low.

When the low-pass filter is applied, it is clear that the pixels become gentle, and the gentleness means that the correlation between the pixels is further increased.
As illustrated in FIGS. 2B and 2C, the increase in the correlation between pixels is due to the fact that the correlation between the pixels in FIG. That is, it is converted into a pixel signal having a large correlation, and it can be said that the compression rate is further increased.
The above is the explanation that the compression efficiency is improved when the low-pass filter processing is performed on the moving image data, but the same can be said for the motion blur addition processing.

Next, FIG. 3 illustrates a configuration including a motion blur addition processing unit 11 and a motion vector generation processing unit 12 as the image processing apparatus 1. Or the structure provided with the motion blur addition process part 11, the motion vector production | generation process part 12, and the encoding process part 2 as the image processing apparatus 100 is shown.
FIG. 3 shows a configuration example using a motion vector VD as an example of the motion information MI in FIG.

The motion vector generation processing unit 12 generates a motion vector VD for each region (a region is a pixel unit or a pixel block unit including a plurality of pixels) of each frame of the input moving image data ID.
The motion blur addition processing unit 11 adaptively performs a filtering process on the input video image data ID with respect to the video image data ID (each frame and each divided region) using the motion vector VD. Perform appending.
The motion blur image data BD subjected to the motion blur addition process is input to the encoding processing unit 2 and subjected to an encoding process such as MPEG compression.
Also in this case, the encoded data ED obtained by compressing the motion blurred image data BD has improved compression efficiency compared to the data encoded by directly inputting the moving image data ID to the encoding processing unit 2. It will be what.

Next, FIG. 4 is a configuration example in which a motion vector used in the encoding processing unit 2 is used for motion blur addition processing.
FIG. 4 shows a basic MPEG encoding processing block as a configuration in the encoding processing unit 2.

The moving image data ID input to the motion blur addition processing unit 11 is also input to the motion vector generation processing unit 210 in the encoding processing unit 2.
The motion vector generation processing unit 210 detects a motion between two adjacent pictures, generates a motion vector, and outputs the motion vector to the motion compensation unit 209 and the entropy encoding unit 204.
The subtraction unit 201 subtracts the image data compensated for motion by the motion compensation unit 209 from the moving image data input from the motion blur addition processing unit 11, and outputs the result to the DCT processing unit 202.
The DCT unit 202 performs DCT processing on the moving image data input from the subtraction unit 201, generates DCT coefficients, and outputs the DCT coefficients to the quantization unit 203.

The quantization unit 203 quantizes the DCT coefficient input from the DCT unit 202 to generate quantized data, and outputs the quantized data to the entropy encoding unit 204 and the inverse quantization unit 205.
The inverse quantization unit 205 performs inverse quantization on the quantized data input from the quantization unit 203 to generate a DCT coefficient, and outputs the DCT coefficient to the inverse DCT unit 206.
The inverse DCT unit 206 performs inverse DCT processing on the DCT coefficient input from the inverse quantization unit 205 to generate image data, and outputs the image data to the addition unit 207.
The adder 207 adds the image data subjected to motion compensation by the motion compensator 209 and the image data input from the inverse DCT unit 206 and outputs the result to the frame memory 208. The image data temporarily stored in the frame memory 208 is output to the motion compensation unit 209 and supplied to the motion vector generation processing unit 210 as image data of a past frame used for motion vector generation.

Based on the motion vector input from the motion vector generation processing unit 210, the motion compensation unit 209 performs motion compensation processing on the image data input from the addition unit 207 via the frame memory 208, and adds the addition unit 207 and the subtraction unit 201. Output for.
The entropy encoding unit 204 performs variable-length encoding processing on the quantized data input from the quantization unit 203, generates compressed image data (encoded data ED), and outputs the compressed image data.

In the configuration of FIG. 4, the motion vector generated by the motion vector generation processing unit 210, that is, the motion vector used for MPEG compression processing is supplied to the motion blur addition processing unit 11. The motion blur addition processing unit 11 performs motion blur addition processing using the supplied motion vector.
As in this example, in predictive encoding based on motion information of an image like the MPEG system, it is also conceivable to use a motion vector generated in the encoding process.
Although illustration is omitted, for example, as shown in FIG. 3, the motion vector VD generated by the motion blur addition processing unit 11 for the motion blur addition processing is supplied to the encoding processing unit 2, and the encoding processing unit 2 may perform the predictive encoding process using the motion vector VD.

These first image processing apparatuses 1 (or 100) adaptively apply a low-pass filter that simulates motion blur caused by an exposure period during imaging to an image using motion information in a moving image. Motion blur addition by processing is executed. Thereby, it is possible to improve the encoding efficiency while presenting a natural image quality for the observer of the moving image.
By applying a low-pass filter that simulates motion blur, compared to the case where a spatially uniform low-pass filter is executed, the image quality that is natural for the observer of the moving image is presented and the encoding efficiency is improved. It is possible to perform.
In addition, because it refers to motion information and performs appropriate filtering for each area, the image quality is maintained even when a wide range of high-intensity low-pass filters are used, especially in areas where the subject moves at high speed. However, an improvement in encoding efficiency can be expected as compared with a case where a spatially uniform low-pass filter is executed.
In addition, the first image processing apparatus 1 (or 100) enables reduction of jerkiness deterioration.

[1-2: Motion vector generation processing unit]

As described above, various configuration examples of the first image processing apparatus 1 (or 100) are conceivable. In the following, a detailed configuration example of the motion vector generation processing unit 12 and the motion blur addition processing unit 11 will be described based on the configuration example shown in FIG.

First, the configuration and operation of the motion vector generation processing unit 12 will be described with reference to FIGS.
The motion vector generation processing unit 12 is a part that accurately generates a motion vector for each region as a pixel or a pixel block unit. Specifically, as shown in FIG. 5, it has a motion vector detection unit 121, a pixel block identification processing unit 122, a motion vector estimation processing unit 123, a motion vector smoothing processing unit 124, and delay units 121a and 122a.

The motion vector detection unit 121 detects a motion vector from the processing target frame and the previous frame.
The pixel block identification processing unit 122 compares the motion vector of the processing target frame and the motion vector of the immediately preceding frame for each pixel block, and identifies a pixel block having a high correlation.
The motion vector estimation processing unit 123 estimates the motion vectors of the other pixel blocks from the motion vector of the pixel block specified by the pixel block specification processing unit 122.
The motion vector smoothing processing unit 124 performs a smoothing process on the motion vector.

The moving image data ID input as shown in FIG. 3 is supplied to the motion vector detection unit 121 and the delay unit 121a that delays the moving image data ID by one frame.
The motion vector detection unit 121 sets the supplied moving image data ID as a processing target frame. Then, the motion vector of the processing target frame is detected, for example, in units of pixel blocks from the processing target frame and the immediately preceding frame delayed by one frame by the delay unit 121a.
When the processing related to the motion vector detection unit 121 is implemented by software, a motion vector may be detected in units of pixel blocks using a general block matching method.

The motion vector detected by the motion vector detection unit 121 is supplied to the pixel block identification processing unit 122 and the delay unit 122a. The delay unit 122a delays the input motion vector by one frame.
The pixel block identification processing unit 122 compares the motion vector of the processing target frame supplied from the motion vector detection unit 121 and the motion vector of the immediately preceding frame delayed by the delay unit 122a in units of pixel blocks as shown below. Then, a pixel block having a high correlation is specified from the comparison result.

  Specifically, the pixel block identification processing unit 122 sets (x, y) as the motion vector of one pixel block of the processing target frame, and (x ′, y ′) as the motion vector of the pixel block in the immediately preceding frame corresponding thereto. ), And arbitrarily determined correlation determination coefficient α, the vector correlation coefficient σ of this pixel block is calculated by the following (Equation 3).

なお、相関判定係数αは、その定義域を０＜α＜１とし、αの値が大きいほど、ベクトル相関係数σの値が１として算出される係数である。

The correlation determination coefficient α is a coefficient that is calculated such that the domain is 0 <α <1 and the value of the vector correlation coefficient σ is 1 as the value of α increases.

  Then, the pixel block identification processing unit 122 calculates the vector correlation coefficient σ of each pixel block from (Equation 3) described above, and the pixel block having the vector correlation coefficient σ of 1 has a highly correlated motion vector As specified.

The motion vector estimation processing unit 123 calculates a pixel block having a vector correlation coefficient σ of 0 from the motion vector of the pixel block identified by the pixel block identification processing unit 122 as having a vector correlation coefficient σ of 1. Estimate motion vectors.
In other words, the motion vector estimation processing unit 123 assumes that the pixel block specifying processing unit 122 in the previous stage has a valid motion vector for the pixel block having the value of the vector correlation coefficient σ of 1, and other pixels. The motion vector of a block, that is, a pixel block having a motion vector that is not valid with the value of the vector correlation coefficient σ being 0 is updated.

Specific processing steps of the motion vector estimation processing unit 123 will be described in detail with reference to FIG.
In step S <b> 1, the motion vector estimation processing unit 123 determines whether the vector correlation coefficient σ of the current pixel block to be processed (hereinafter referred to as the target pixel block) in the processing target frame is 1 or 0. That is, the motion vector estimation processing unit 123 determines whether or not the motion vector of this pixel block is valid. Then, the motion vector estimation processing unit 123 ends this processing step without updating the value of the motion vector when the motion vector of the pixel block is valid, and proceeds to step S2 when the motion vector of the pixel block is not valid. move on.

  In step S <b> 2, the motion vector estimation processing unit 123 determines whether there is a peripheral pixel block having a valid vector around the target pixel block for the target pixel block. Specifically, the motion vector estimation processing unit 123 determines whether there are valid motion vectors for a total of eight pixel blocks adjacent to the pixel block of interest as peripheral pixel blocks, and performs effective motion. When the vector exists, the process proceeds to step S3, and when the effective motion vector does not exist, this processing step is terminated without updating the motion vector of the pixel block of interest.

Here, the reason why the estimation process is not performed using the neighboring pixel blocks located in a wider range with respect to the target pixel block having no effective motion vector is as follows.
As a first reason, although it is possible to perform estimation processing using pixel blocks located in a wider range, even if it is realized, in order to end this processing step in fixed time processing, peripheral pixels This is because the storage area for temporarily storing image data handled as blocks increases.
As a second reason, by performing the smoothing process on the motion vector of the pixel block of interest using a wider range of peripheral pixel blocks than the total of the adjacent eight pixel blocks described above, in the subsequent stage of this processing step, This is because an invalid motion vector can be corrected appropriately.

  In step S <b> 3, the motion vector estimation processing unit 123 estimates and updates the motion vector of this pixel block of interest from only the motion vectors of the surrounding pixel blocks having an effective motion vector, and ends this processing. As an example of the estimation process, the motion vector estimation processing unit 123 outputs and smoothes the motion vector of the pixel block of interest using a median filter that receives only the motion vectors of the neighboring pixel blocks having an effective motion vector.

  The motion vector estimation processing unit 123 estimates the motion vector of the processing target frame in units of pixel blocks as described above. Then, the motion vector estimation processing unit 123 supplies a motion vector including the motion vector specified by the pixel block specifying processing unit 122 to the motion vector smoothing processing unit 124.

  The motion vector smoothing processing unit 124 performs a smoothing process on the motion vector of each pixel block constituting the processing target image. Specifically, the motion vector smoothing processing unit 124 uses, as an input I (x + i, y + j), the motion vector of the target pixel block before the smoothing process and the motion vector of a peripheral pixel block having a wider range than the adjacent pixel block described above. The motion vector J (x, y) of the pixel block of interest after the smoothing process is output by a Gaussian function as shown in (Equation 4) shown below.

ここで、ｒは注目画素ブロックと各周辺画素ブロックとの２次元空間上の距離を示し、σ２はこの距離ｒについての分散を示し、ｔ２は動きベクトルについての分散を示している。すなわち、σ２及びｔ２は、平滑化の度合いを表す値として任意に設定されるパラメータとなっている。

Here, r indicates the distance in the two-dimensional space between the pixel block of interest and each peripheral pixel block, σ2 indicates the variance for this distance r, and t2 indicates the variance for the motion vector. That is, σ2 and t2 are parameters that are arbitrarily set as values representing the degree of smoothing.

  The motion vector smoothing processing unit 124 performs the above-described smoothing processing on each pixel block constituting the processing target frame, and supplies the motion vector VD to the motion blur addition processing unit 11.

As described above, the motion vector smoothing processing unit 124 identifies a pixel block having an effective motion vector from each pixel block constituting the processing target frame, and estimates other motion vectors from the effective motion vector. . Therefore, it is possible to generate a motion vector according to the actual motion of the moving object with high accuracy.
The motion vector generation processing unit 12 directly supplies the motion vector detected by the motion vector detection unit 121 to the motion vector smoothing processing unit 124 without going through the pixel block specifying processing unit 122 and the motion vector estimation processing unit 123. Then, a smoothing process may be performed. Even when such processing is performed, it is possible to generate a motion vector with higher accuracy corresponding to the actual motion of the moving object than the motion vector as the encoded information described above.

[1-3: Motion blur addition processing unit]

Next, a configuration example of the motion blur addition processing unit 11 will be described.
As the motion blur addition processing unit 11, two examples of the configuration example of FIG. 7 and the configuration example of FIG. 12 will be described.

First, the motion blur addition processing unit 11 as an example of FIG. 7 will be described.
As illustrated in FIG. 7, the motion blur addition processing unit 11 includes a motion vector mask processing unit 113 that generates motion vector mask information for specifying an image region to which motion blur is added.
In addition, the motion blur addition processing unit 11 calculates a shutter speed suitable for the motion vector (hereinafter referred to as optimum shutter speed information), and the shutter speed when the optimum shutter speed information and a moving image are actually captured. An optimum shutter speed calculation / discrimination unit 114 is provided that performs discrimination processing described later by comparing the information.
The motion blur addition processing unit 11 includes a motion vector correction unit 115 that corrects a motion vector based on the determination result of the optimum shutter speed calculation / discrimination unit 114.
The motion blur addition processing unit 11 includes a filter parameter calculation unit 111 that calculates a filter parameter for motion blur addition corresponding to each pixel of the processing target frame.
The motion blur addition processing unit 11 includes a motion blur addition filter 112 that performs motion blur filter processing on the pixel value of each pixel of the processing target frame.

Here, it is possible to perform all processing in units of pixels. However, in order to reduce the burden of calculation processing, the motion blur addition processing unit 11 includes a motion vector mask processing unit 113 and an optimum shutter speed calculation / discrimination unit 114. , And the processing related to the motion vector correction unit 115 is performed in units of pixel blocks.
In addition, the filter parameter calculation unit 111 and the motion blur addition filter 112 perform a filter process for adding motion blur to the moving image data ID.
The motion vector mask processing unit 113 specifies a pixel block-based motion vector VD supplied from the motion vector generation processing unit 12 in order to identify an image region to which motion blur is to be added in the processing target frame. A mask process as shown is applied. Then, the motion vector for each pixel block after the mask processing is supplied to the optimum shutter speed calculation / discrimination unit 114 and the motion vector correction unit 115.

The image area where jerkiness is likely to occur, which needs to add motion blur, is concentrated particularly on the moving body image area and the image area around the edge in the screen.
Therefore, the motion vector mask processing unit 113 outputs only the motion vectors of the pixel blocks in the vicinity of the edge with high spatial contrast, in which jerkiness is likely to occur, by the processing shown in FIG.
In other words, in step S11, the motion vector mask processing unit 113 uses the edge of the image as processing for specifying a region having a high spatial contrast in the processing target frame in units of pixel blocks for the input moving image data ID. To detect.
In parallel with the process in step S11, in step S12, the motion vector mask processing unit 113 performs a process for specifying a moving object region in the processing target frame. That is, the moving body image area is detected by calculating the difference between frames in units of pixel blocks.

  In step S13, the motion vector mask processing unit 113 determines in pixel block units whether or not a region in which jerkiness degradation is likely to occur is detected in the processing related to one or both of step S11 and step S12 described above. To do. Then, the motion vector mask processing unit 113 sets the mask processing flag to “1” for the pixel block that is determined to be an area where jerkiness is likely to occur. Further, the motion vector mask processing unit 113 sets a mask processing flag to “0” for a pixel block that is not determined to be an area where jerkiness is likely to occur.

In step S <b> 14, the motion vector mask processing unit 113 is the motion vector VD of the pixel block whose flag is set to “1” with respect to the motion vector VD supplied from the motion vector generation processing unit 12. Judge whether or not.
The motion vector mask processing unit 113 does not change the value of the motion vector of the pixel block for which the flag is set to “1”, and does not change the value of the optimal shutter speed calculation / determination unit 114 and the motion vector correction unit at the subsequent stage. 115.
In addition, the motion vector mask processing unit 113 performs a mask process for making the value of the motion vector 0 or invalid in step S15 for the motion vector of the pixel block whose flag is set to “0”. This is output to the optimum shutter speed calculation / discrimination unit 114 and the motion vector correction unit 115 in the subsequent stage.

Next, processing steps relating to the optimum shutter speed calculation / discrimination unit 114 and the motion vector correction unit 115 will be described with reference to FIG.
As step S31, the optimum shutter speed calculation / discrimination unit 114 calculates the optimum shutter speed according to the motion vector of each pixel block of the processing target frame, for example, based on an evaluation index as shown in FIG.

Here, FIG. 10 is a diagram showing a subject speed indicating a moving speed of a moving object detected as a motion vector and an optimum shutter speed curve corresponding to the subject speed. The optimal shutter speed is a blur that makes it difficult to perceive deterioration of jerkiness in terms of visual characteristics according to the moving speed of the subject, and that details of the subject are lost or blurred due to excessive motion blur. It is a shutter speed at which deterioration is hardly perceived. That is, if a subject is imaged at a shutter speed faster than the optimum shutter speed, it can be determined that jerkiness degradation has occurred in the captured image. On the other hand, if a subject is imaged at a shutter speed slower than the optimum shutter speed, it can be determined that the captured image has been blurred.
Therefore, the optimum shutter speed calculation / discrimination unit 114 calculates the optimum shutter speed corresponding to the motion vector of each pixel by associating the motion vector of each pixel block with the subject speed in FIG.

Note that the optimum shutter speed curve SS0 shown by the solid line in FIG. 10 is an example showing the correspondence between an arbitrary subject speed and the optimum shutter speed, and is specifically obtained based on a psychological experiment. It is a curve connecting the experimental result values.
Here, the motion blur area A1 shown in FIG. 10 is an area where it is determined that the motion blur due to the motion of the subject is excessively included based on the optimum shutter speed curve SS0. Similarly, the jerkiness area A2 is an area where it is determined that jerkiness deterioration has occurred in visual characteristics based on the optimum shutter speed curve SS0 without motion blur due to the movement of the subject.
When the optimum shutter speed curve SS0 indicated by the solid line is directly used to obtain the optimum shutter speed corresponding to the motion vector, the optimum shutter speed information corresponding to the motion vector at an arbitrary step size is stored as a table. In advance, the recording medium may be referred to.

なお、この（数５）における各パラメータＡ、Ｂ、γは、図１０中に示す最適シャッタ速度曲線の曲線形状に応じて適当な値を選択して用いるようにすればよい。シャッタ速度曲線の具体例として、図１０では、（数５）中の各パラメータのうち、Ａ、Ｂの値を固定にして、γを３段階に変化させたときの曲線形状ＳＳ１〜ＳＳ３を示している。 In the present embodiment, the optimum shutter speed corresponding to the motion vector may be calculated by using a function that approximates the optimum shutter speed curve indicated by the solid line.
In this case, the optimum shutter speed calculation / discrimination unit 114 calculates the optimum shutter speed SSD ′ by using an approximate function of the optimum shutter speed curve shown in the following (Equation 5), where v is a motion vector in a certain pixel block.

The parameters A, B, and γ in (Equation 5) may be selected and used in accordance with the curve shape of the optimum shutter speed curve shown in FIG. As a specific example of the shutter speed curve, FIG. 10 shows curve shapes SS1 to SS3 when the values of A and B among the parameters in (Equation 5) are fixed and γ is changed in three stages. ing.

When the optimum shutter speed SSD ′ corresponding to the motion vector is calculated, in step S32 of FIG. 9, the optimum shutter speed calculation / determination unit 114 determines the optimum shutter speed SSD ′ and the actually captured shutter speed SSD. To determine whether or not the pixel block unit corresponds to the jerkiness area A2 shown in FIG.
It should be noted that the actually captured shutter speed SSD only needs to be added as metadata in the moving image data ID, for example. Further, when the image processing apparatus 1 (100) of this example is mounted in the imaging apparatus, it can be acquired as shutter speed information at the time of imaging by the own apparatus.

From the determination result of step S32, when the shutter speed SSD is faster than the optimal shutter speed SSD ′ and corresponds to the jerkiness area A2 in the current pixel block to be processed, the motion vector correction unit 115 proceeds to step S33. move on.
When the shutter speed SSD is slower than the optimum shutter speed SSD ′ and does not correspond to the jerkiness area A2 in the current pixel block to be processed, the motion vector correction unit 115 proceeds to step S34.

In step S33, since the jerkiness deterioration has occurred in the pixel block to be processed, the motion vector correcting unit 115 increases, for example, as the shutter speed SSD increases, and the function fs (SSD that converges to 1). ) Is multiplied by the value of the motion vector.
The motion vector correction unit 153 replaces the function fs (SSD) with fs (VD) using the motion vector VD as a variable, or fs (SSD, VD) using the shutter speed SSD and the motion vector VD as two variables. You may make it perform a multiplication process using.
In step S34, since the jerkiness deterioration does not occur in the pixel block to be processed, the motion vector correction unit 153 performs a mask process for invalidating the value of the motion vector by, for example, multiplying by 0.

  In this way, the optimum shutter speed calculation / determination unit 114 determines whether or not jerkiness degradation has occurred in consideration of the shutter speed SSD when the moving image to be processed is actually captured. . Then, the motion vector correction unit 115 performs a correction process for adding an appropriate motion blur to the motion vector of the pixel block that is determined to have jerkiness degradation. This contributes to the motion blur addition process performed by the motion blur addition filter 112 being a more natural moving image in terms of visual characteristics.

The filter parameter calculation unit 111 calculates the following filter parameters for each pixel in order to add motion blur to each pixel constituting the processing target frame.
First, the filter parameter calculation unit 111 specifies a pixel (hereinafter referred to as a parameter calculation target pixel) located on the motion vector of each target pixel, with a pixel having effective motion vector information as the target pixel. Then, the filter parameter calculation unit 111 calculates a filter parameter corresponding to the relative position of the specified parameter calculation target pixel with respect to the target pixel as follows.

ここで、式中の強度σを二乗した値が後段の動きぼけ付加フィルタ１１２におけるガウス型関数の分散となるように、（数６）は設定されている。 That is, as shown in FIG. 11, the filter parameter calculation unit 111 sets all pixels located on a vector having the midpoint between the start point S and the end point E of the motion vector as the position of the target pixel P0 as parameter calculation target pixels. Identify. As shown in FIG. 11, the absolute value v is the absolute value of the motion vector of the target pixel.
Subsequently, the filter parameter calculation unit 111 according to the absolute value v of the motion vector and the distance d between the pixel position of the target pixel P0 and the pixel position of the parameter calculation target pixel P1 specified by the above-described processing, The motion blur added strength σ is calculated by the following (Equation 6).

Here, (Equation 6) is set so that the value obtained by squaring the intensity σ in the equation becomes the variance of the Gaussian function in the subsequent motion blur addition filter 112.

In addition, the filter parameter calculation unit 111 sets the coordinate point on the orthogonal coordinate plane xy of each parameter calculation target pixel P1 when the target pixel P0 is the origin as an angle for adding motion blur (x ₁ , y ₁ ). The direction θ is calculated from the following (Equation 7).

  In this way, the filter parameter calculation unit 111 specifies the parameter calculation target pixel from the motion vector of the target pixel, sets the parameter information (σ, θ) for each specified parameter calculation target pixel, and sets the processing target This is supplied to the motion blur addition filter 112 in units of frames.

  Note that, in the processing related to the filter parameter calculation unit 111, there are cases where a parameter calculation target pixel is specified redundantly for a certain pixel. In this case, in order to simplify the processing, for example, information having a large value of σ among the parameter information specified redundantly may be set as the parameter information of the pixel. In addition, the filter parameter calculation unit 111 applies subsequent smoothing processing such as Gaussian function type filter processing and median filter processing to the parameter information (σ, θ) of each parameter calculation target pixel, thereby adding motion blur in the subsequent stage. The image quality of the moving image output from the filter 112 can be improved.

The motion blur addition filter 112 performs the following processing target on the pixel value of each pixel of the processing target frame of the input moving image data ID according to the parameter information supplied from the filter parameter calculation unit 111. Apply spatial filtering within the frame.
In the present embodiment, the motion blur addition filter 112 outputs an image with motion blur added by executing one or both of the following first filter processing and second filter processing.

なお、入力Ｉ（ｘ＋ｉ，ｙ＋ｊ）となる周辺画素は、動きベクトルを付加する角度方向に応じて設定される。また、ｒは、動きぼけ付加対象画素と周辺画素との間の距離を示す。 First, the first filter process will be described. In the first filter processing, the motion blur addition filter 112 inputs the pixel value of the motion blur addition target pixel before the motion blur addition filter processing and the pixel values of peripheral pixels located around this pixel as input I (x + i, y + j). ), The pixel value J (x, y) of the target image after the filter processing is output by a Gaussian function as shown in the following (Equation 8).

It should be noted that the peripheral pixels that become the input I (x + i, y + j) are set according to the angular direction to which the motion vector is added. R represents the distance between the motion blur addition target pixel and the surrounding pixels.

  The motion blur addition filter 112 performs the above-described filter processing and updates the pixel value for each pixel for which parameter information (σ, θ) is set among all the pixels constituting the processing target frame. In this way, the motion blur addition filter 112 can supply the moving image data BD with reduced jerkiness degradation to the encoding processing unit 2 at the subsequent stage.

  By the way, some peripheral pixels located around the pixel of interest originally have no movement, that is, a background region. Peripheral pixels located in such a background area need not be considered in order to add motion blur to the pixel of interest. A processing method paying attention to such a point is the second filter processing described below.

That is, in the second filter processing, when the motion vector value of the target pixel is 0 or invalid, the motion blur addition filter 112 has a motion vector value of 0 among the peripheral pixels located around the target pixel. Alternatively, the pixel value I (x + i ₀ , y + j ₀ ) of the invalid pixel is replaced with the pixel value I (x, y) of the target pixel, and the pixel value J (x, y) of the target pixel according to the above (Equation 8). Is calculated. In this manner, the motion blur addition filter 112 outputs an image in which jerkiness degradation is naturally reduced in terms of visual characteristics, compared to the first filter processing.

The motion blur addition processing unit 11 can output moving image data BD subjected to motion blur addition processing as in the above example.
Particularly in the case of this example, the optimum shutter speed calculation / discrimination unit 114 corrects the motion vector according to the shutter speed information at the time of capturing a moving image, so that the motion blur added intensity calculated by the subsequent filter parameter calculation unit 111 is obtained. Since the value of σ is controlled, an appropriate motion blur can be added according to the shutter speed information at the time of imaging by the motion blur addition filter 112, and a moving image in which jerkiness degradation is reduced more naturally in terms of human visual characteristics. Data BD can be output.

  Next, a configuration example of FIG. 12 will be described as a configuration example of the motion blur addition processing unit 11. As illustrated in FIG. 12, the motion blur addition processing unit 11 may include a filter parameter calculation unit 111 and a motion blur addition filter 112. In other words, the processing block related to motion vector correction is deleted from the configuration of FIG.

As described above, in the present embodiment, the motion blur addition processing unit is performed on the moving image data ID in the preceding stage of the encoding processing unit 2, and one of the purposes is encoding efficiency. It is an improvement.
Then, by using the configuration example of FIG. 7 above, it is possible to optimize the degree of blur addition according to the perceptual characteristics, but this is not essential if the viewpoint of coding efficiency is emphasized. In that case, the configuration of FIG. 12 may be used.

In the configuration example of FIG. 12, the motion vector VD supplied from the motion vector generation processing unit 12 is directly input to the filter parameter calculation unit 111.
The filter parameter calculation unit 111 performs filter parameter calculation by performing the same process as described above using the uncorrected motion vector VD. The motion blur addition filter 112 performs the above-described motion blur addition processing based on the filter parameter.

According to this configuration example, motion blur is added faithfully to the magnitude of the motion vector detected by the motion vector generation processing unit 12.
This relates to the second image processing apparatus 4 (400) to be described later. By doing so, it is possible to more easily perform faithful motion blur reduction processing.

The configuration and operation of the motion blur addition processing unit 11 have been described above with reference to FIGS. 7 and 12, but various configuration examples and operation examples other than these can be considered.
The motion blur addition processing unit 11 adds motion blur to image data using other motion information instead of the spatial filter processing that adds motion blur to each unit image using motion vectors. May be.
For example, a process of adding motion blur to the moving image data ID may be performed by performing a temporal filter process of superimposing a plurality of frames on one frame. In this case, the motion blur addition processing unit 11 detects a moving body image region based on a difference between frames as motion information instead of a motion vector. Then, the information indicating the detected moving body image area is corrected based on the imaging information, and then subjected to temporal filter processing using the motion information, so that the appropriate information can be obtained according to the shutter speed information at the time of imaging. Motion blur can be added.

[2. Second image processing apparatus]
[2-1: Configuration Example of Image Processing Device]

Subsequently, a second image processing apparatus as an embodiment of the present invention will be described. The second image processing device is the moving image data transmitted from the first image processing device described above, that is, the moving image data ED (hereinafter referred to as “encoding state”) subjected to motion blur addition processing and encoding processing. The video processing device performs decoding and motion blur reduction processing on the video data ED ”.

A configuration example of the second image processing apparatus will be described with reference to FIGS. 13, 14, and 15.
Note that the processing blocks in each drawing are shown as logical components, and each processing block may be in the same casing or in another casing. In each figure, the image processing device 4 surrounded by a broken line or the image processing device 400 also surrounded by a broken line can be considered as the configuration of the second embodiment of the present invention.

  FIG. 13 shows a configuration including a motion blur reduction processing unit 41 as the image processing apparatus 4. Or the structure provided with the decoding process part 3 and the motion blur reduction process part 41 as the image processing apparatus 400 is shown.

The decoding processing unit 3 receives moving image data ED in an encoded state. The decoding processing unit 3 performs decoding processing on the encoded moving image data ED and outputs decoded moving image data DD. For example, the decoding process for the encoding executed by the encoding processing unit 2 in the first image processing apparatus 100 described above is executed.
Accordingly, the output moving image data DD can be considered as moving image data in a state in which motion blur is added by the motion blur addition processing unit 11 of the first image processing apparatus 100.

The motion blur reduction processing unit 41 performs motion blur reduction processing on the image data DD decoded by the preceding decoding processing unit 3, and outputs the processed video data OD.
The motion blur reduction processing unit 41 is similarly input to each frame and each divided region of the decoded moving image data DD, and motion information MI corresponding to each frame and each divided region of the moving image data DD. Filtering processing according to the above is adaptively performed on the moving image data DD to reduce motion blur.
The moving image data OD subjected to the motion blur reduction process is output to a display device (not shown).
As shown in the figure, it is also conceivable to output the moving image data DD decoded by the decoding processing unit 3 to a display device or the like.

  Here, the following three functions (i), (ii), and (iii) are conceivable as functions and effects of the second image processing apparatus 4 (400) of this example.

(I) Higher image quality by reducing blur degradation of transmitted video data.
The moving image data OD that has been subjected to motion blur reduction processing by the motion blur reduction processing unit 41 and output to a display device or the like has a blur deterioration that impairs image details compared to the case where the decoded image data DD is directly displayed on a display device. It will be resolved and displayed in high quality.
That is, it is possible to obtain an effect of reducing blur deterioration by the motion blur reduction process under the assumption that the transmission source of moving image data is not specified.

(Ii) Improvement in image quality of moving image data transmitted from the first image processing apparatus 100 by reducing jerkiness deterioration and blur deterioration.
Considering that the moving image data ED in the encoded state in which the jerkiness degradation is eliminated by the motion blur addition processing unit 11 on the first image processing apparatus 100 side is input to the second image processing apparatus 400. In this case, the moving image data OD subjected to the motion blur reduction process becomes high-quality moving image data in which both the jerkiness deterioration and the blur deterioration are reduced.
That is, in this sense, the moving image data DD is moving image data in which jerkiness deterioration is reduced, and the moving image data OD is moving image data in which both jerkiness deterioration and blur deterioration are reduced.

(Iii) Restoration of moving image data before processing of the first image processing apparatus 100.
Furthermore, in this example, the moving image data OD can be an image equivalent to the original moving image data ID input to the first image processing apparatus 100. In other words, the motion blur added by the motion blur addition processing unit 11 is faithfully reduced by the motion blur reduction processing unit 41. This is because motion blur is added to emphasize the viewpoint of improving the encoding efficiency in the encoding processing unit 2 in the first image processing apparatus 100, and on the second image processing apparatus 400 side as the transmission destination, This is suitable when it is desired to restore the original moving image data.
For example, as described above, assuming input of image data obtained by encoding / decoding data with motion blur added using the motion blur addition processing unit 11 shown in the configuration example of FIG. The motion blur is added without correction, faithfully to the magnitude of the motion vector VD.
Therefore, when the motion blur reduction process is performed by the motion blur reduction processing unit 41 using only the motion vector VD as a parameter, in principle, before the motion blur is added by the first image processing apparatus 1 (100). Should be obtained.

Next, FIG. 14 illustrates a configuration including a motion blur reduction processing unit 41 and a motion vector generation processing unit 42 as the image processing device 4. Or the structure provided with the decoding process part 3, the motion blur reduction process part 41, and the motion vector production | generation process part 42 as the image processing apparatus 400 is shown.
FIG. 14 shows a configuration example using a motion vector VD as an example of the motion information MI in FIG.

The motion vector generation processing unit 42 generates a motion vector VD for each region (pixel unit or pixel block unit of a plurality of pixels) of each frame of the decoded moving image data DD.
The motion blur reduction processing unit 41 adaptively performs filtering on the moving image data DD (each frame and each divided region) with respect to the decoded moving image data DD using the motion vector VD. Perform reductions. The moving image data OD subjected to the motion blur reduction process is output to a display device (not shown).
Also in this case, the function / effect of any of the above (i), (ii), and (iii) is assumed.

Next, FIG. 15 is a configuration example in which a motion vector used in the decoding processing unit 3 is used for motion blur reduction processing.
The motion blur estimation process is performed in the encoding process in predictive encoding based on the motion information of an image such as MPEG, and the motion blur estimation result is sent to the decoding processing unit 3 together with the encoded data. Is being transmitted.
The decoding processing unit 3 executes decoding processing using the transmitted motion vector. In the configuration example illustrated in FIG. 15, motion blur reduction processing is executed using this motion vector.

FIG. 15 shows a simple MPEG decoding processing block.
The encoded moving image data ED is input to the entropy decoding unit 301 in the decoding processing unit 3. The entropy decoding unit 301 performs a decoding process on the input encoded moving image data ED, obtains quantized data, and extracts a motion vector VD. That is, this is, for example, the motion vector VD generated by the motion vector generation processing unit 210 in FIG. 4 and superimposed on the encoded moving image data in the encoding by the entropy encoding unit 204.
The entropy decoding unit 301 outputs the quantized data to the inverse quantization unit 302, and outputs the motion vector VD to the motion compensation unit 306 and the motion blur reduction processing unit 41.

The inverse quantization unit 302 inversely quantizes the input quantized data to generate a DCT coefficient, and outputs the DCT coefficient to the inverse DCT unit 303.
The inverse DCT unit 303 performs inverse DCT processing on the DCT coefficient input from the inverse quantization unit 302 to generate image data, and outputs the image data to the addition unit 304.
The adder 304 adds the image data that has been motion compensated by the motion compensator 306 and the image data input from the inverse DCT unit 303 and outputs the result as decoded moving image data DD. Output to. The image data temporarily stored in the frame memory 305 is output to the motion compensation unit 306.
Based on the motion vector input from the entropy decoding unit 301, the motion compensation unit 306 performs motion compensation processing on the image data input from the addition unit 304 via the frame memory 305, and outputs the motion data to the addition unit 304. .
The moving image data DD decoded by the decoding processing unit 3 is supplied to the motion blur reduction processing unit 41. The motion blur reduction processing unit 41 is also supplied with a motion vector VD. The motion blur reduction processing unit 41 performs motion blur reduction processing on the moving image data DD using the motion vector VD. The moving image data OD subjected to the motion blur reduction process is output to a display device (not shown).
As in this example, it is also possible to perform motion blur reduction processing using a motion vector extracted in the MPEG decoding process.
Also in this case, the function / effect of any of the above (i), (ii), and (iii) is assumed.

Further, the image processing apparatus 4 (400) of each example described above refers to motion information and performs appropriate filtering for each region. Therefore, compared to the case of executing a spatially uniform low-pass filter at the time of encoding (that is, a spatially uniform high-pass filter at the time of decoding), it is adapted for each region according to the magnitude of motion. If necessary, the motion blur can be reduced including adjustment of the motion blur amount as post-processing, and a natural image quality can be presented to the observer of the moving image.
Further, even when the parameter of motion blur addition processing in the processing before encoding is not transmitted (or cannot be performed), motion blur reduction processing can be performed using only motion information.
In other words, it can be executed only by using a motion vector transmitted for use in encoding / decoding or motion information obtained after decoding. This is because if a low pass filter that is spatially uniform at the time of encoding (that is, a high pass filter that is spatially uniform at the time of decoding) is executed, some filter information at the time of the low pass filter must be transmitted. This is an advantage over what shouldn't be handled.

[2-2: Motion blur reduction processing unit]

The motion blur reduction processing unit 41 in the configuration examples of FIGS. 13, 14, and 15 will be described. Here, the example of FIG. 16 and the example of FIG. 18 will be described as the configuration of the motion blur reduction processing unit 41, respectively.
First, a configuration example of the motion blur reduction processing unit 41 in FIG. 16 will be described.
The motion blur reduction processing unit 41 in FIG. 16 includes a moving average filter characteristic conversion unit 411, a moving average filter unit 412, a subtraction unit 413, and an addition unit 414. The motion blur reduction processing unit 41 performs motion on the input moving image data DD. Realize reduction of blur.

  The moving average filter used as the moving average filter unit 412 is a kind of simplest low-pass filter, and is a filter that calculates and outputs an average value with surrounding pixels every time a pixel to be processed moves one pixel. . For example, at a certain point in time, an average is obtained with n (four in the figure) samples including the current sample value as shown in FIG. Even at the next time point, an average is obtained with n (four in the figure) samples including the current sample value as shown in FIG. The sample value here becomes a pixel value, and an average of n samples of peripheral pixels including the processing target pixel is output every time the processing target pixel moves by one pixel.

A motion vector VD is input to the moving average filter characteristic conversion unit 411. The moving average filter characteristic conversion unit 411 extracts a motion vector VD corresponding to the divided area in the moving image data DD as a filter parameter from the input motion vector VD, and based on this, the moving average filter unit The filter characteristic of the process performed at 412 is determined.
For example, one moving average filter may be prepared in advance for each of the plurality of motion vectors VD, and a filter to be used in the target pixel may be determined. Specifically, this is performed as follows.

Considering the characteristics of the moving average filter by how many pixels around the target pixel are subjected to the averaging process, a motion vector VD is used as an example of a filter parameter. At this time, a table for determining one number of pixels used for the moving average filter is prepared for the motion vector VD, and the number of pixels used for the moving average filter is output each time the motion vector VD of each region is input. To do.
The determined number of pixels used for the moving average filter is output to the moving average filter unit 412.

The moving average filter unit 412 (low-pass filter) applies a moving average filter whose characteristics are determined by the moving average filter characteristic conversion unit 411 to a predetermined block including the target pixel in the processing target frame. The pixel value of the target pixel is converted. The pixel value of the target pixel converted by the moving average filter unit 412 is output to the subtraction unit 413.
That is, the pixel value of the pixel of interest converted by the moving average filter unit 412 is input to the subtraction unit 413 with the polarity reversed. The subtraction unit 413 also receives a target pixel in the processing target frame in the input moving image data DD.
The subtraction unit 413 obtains a difference between the pixel value of the pixel in the moving image data DD and the pixel value of the target pixel converted by the moving average filter unit 412, and outputs the difference value to the addition unit 414.
In this way, the difference before and after the moving average filter is input to the adding unit 414. Also, the pixel of interest in the processing target frame in the moving image data DD is input to the adding unit 414. The adding unit 414 adds the difference value before and after the moving average filter to the pixel value before correction of the target pixel, and outputs the addition result as an output image (part thereof).

The above is the processing content of the motion blur reduction processing unit 41. In order to understand the meaning of this processing, it is easy to understand when considered in the frequency domain.
When the difference before and after the moving average filter, which is the output signal of the subtraction unit 413, is considered in the frequency domain, the gain of the input image signal and the gain of the image signal after the moving average filter is applied at the frequency of interest. Is the gain of the output signal of the subtracting unit 413. Further, the gain of the output image signal of the adding unit 414 is a value obtained by adding the difference gain before and after the moving average filter to the gain of the input image signal. That is, at each frequency, the gain of the output image signal is raised by the difference gain before and after the moving average filter compared to the gain of the input image.
Considering that the moving average filter is a low-pass filter, in other words, the motion blur reduction processing unit 41 shown in the configuration example of FIG. 16 as a whole performs processing that is basically equivalent to processing that applies a high-pass filter. Will be.

Next, a configuration example of FIG. 18 will be described as the motion blur reduction processing unit 41.
The configuration example of FIG. 18 is different from FIG. 16 in that an element of motion vector correction processing necessary to adjust the amount of motion blur to be added to the amount of motion blur where the jerkiness deterioration is most reduced is added. is there.

  Specifically, an optimum shutter speed calculation / discrimination unit 415 and a motion vector correction unit 416 are added, and the optimum shutter speed calculation / discrimination unit 114 and motion in the motion blur addition processing unit 11 shown in FIG. The same motion vector scaling processing as that of the vector correction unit 115 is performed. The difference is that there is no input of the imaging shutter speed SSD.

However, assuming that the moving image data that has been subjected to the motion blur addition processing by the motion blur addition processing unit 11 of FIG. 12 described above is input, the motion data VD is faithfully corrected without being corrected. You can think that blur is added.
Then, it may be considered that the decoded moving image data DD is uniformly added with motion blur equivalent to an open shutter, and is imaged at the longest shutter speed in FIG. 10, for example. Processing may be performed assuming that moving image data is always input.
Specifically, the motion vector correction unit 416 corrects (reduces) the input motion vector in accordance with, for example, the optimum shutter speed curve SS0 in FIG. 10 and outputs the corrected motion vector to the subsequent moving average filter characteristic conversion unit 411. That's fine. The processes of the moving average filter characteristic conversion unit 411, the moving average filter unit 412, the subtractor 413, and the adder 414 are the same as described above.

As described above, the motion blur reduction processing can be realized by using the configuration examples shown in FIGS. Of course, the configuration and operation of the motion blur reduction processing unit 41 are not limited to the above.

[3. Information transmission between the first and second image processing apparatuses]

Next, sharing of motion blur additional information between the first image processing device 1 serving as a preprocessing device for encoding and the second image processing device 4 serving as a post-processing device for decoding will be described.
Up to this point, the image processing apparatus 4 that performs post-processing of decoding processing has been described. Here, the parameters used when the image processing apparatus 1 adds motion blur as pre-coding processing are transmitted to the decoding processing side, and the image processing device 4 receives the decoding post-processing. A mode of determining whether or not to execute motion blur reduction processing and the degree of motion blur to be reduced using motion blur additional information is shown.

FIG. 19 shows the image processing apparatus 1 (100) of FIG. 3 and the image processing apparatus 4 (400) of FIG.
In particular, the motion blur addition processing unit 11 transmits to the encoding processing unit 2 the motion blur addition parameter PM used when the motion blur is added in addition to the processed moving image data BD. Yes.
The encoding processing unit 2 adds the motion blur addition parameter PM as metadata of the moving image data ED in the encoded state, and transmits and transmits it.

On the image processing device 4 (400) side, the decoding processing unit 3 performs decoding processing and outputs the moving image data DD to the motion blur reduction processing unit 41, and extracts the motion blur addition parameter PM from the metadata. This is supplied to the motion blur reduction processing unit 41.
The motion blur reduction processing unit 41 determines the content of motion blur reduction processing based on the motion blur addition parameter PM.

  In addition, although the method used as metadata of an encoded signal is shown as a transmission method of the motion blur addition parameter PM, the transmission method is not limited to this. You may transmit as another stream.

In the motion blur reduction processing unit 41 in FIG. 13 and the like, assuming input of a signal with motion blur added faithfully to the motion vector (for example, without performing vector correction processing as in FIG. 12) The process for reducing the motion blur is executed according to the configuration examples of FIGS. 16 and 18 without requiring the motion blur additional parameter PM.
However, as shown in FIG. 7, processing such as selecting only a region where jerkiness degradation is likely to occur and adding motion blur, or adjusting and adding motion blur considering human perception of jerkiness degradation, etc. In the case of a signal for which motion blur is executed, it is effective to transmit the motion blur addition parameter PM used when motion blur is added to the subsequent stage.

As a specific example of the motion blur addition parameter PM, first, as the simplest, a flag signal of “0” or “1” indicating whether motion blur is added to the pixel (or the region) in the image. Can be considered.
The motion blur reduction processing unit 41 refers to this and determines whether or not to perform motion blur reduction processing.

As a specific example of the motion blur addition parameter PM, it is conceivable to transmit the vector itself corrected in the configuration of FIG. That is, the output of the motion vector correction unit 115 in FIG. 7 is transmitted.
The motion blur reduction processing unit 41 executes motion blur reduction processing using this correction vector.

Also, since the transmission of the vector itself requires a large amount of data to be added, only the correction value obtained by correcting the vector is sent as the motion blur addition parameter PM, or only the vector conversion rules applied at the time of correction are transmitted. It is also conceivable to perform vector inverse transform on the motion blur reduction processing unit 41 side.
For example, in each pixel or each block, the vector correction value performed when the motion blur is added is added to each pixel or each block and transmitted. The simplest is to transmit a real number α between 0 and 1. In the motion blur reduction processing unit 41, the vector V is reduced by the equation V ′ = α × V and then motion blur is performed. Will be reduced. If no motion blur is added, “0” is transmitted.
In the case of a real number, the amount of data increases. For example, it is realistic to transmit a correction value quantized with a value from 0 to N.

  Further, as the motion blur addition parameter PM, for example, it is also conceivable to transmit the motion blur addition filter parameter output from the filter parameter calculation unit 111 of FIG.

Furthermore, the following advantages can be considered by transmitting the motion blur additional parameter PM.
Even if motion blur is added excessively by the motion blur addition processing unit 11 as pre-coding processing (so that blur degradation occurs when displayed as it is), the motion blur reduction processing unit 41 reduces motion blur. This reduces blurring.
That is, by adding an effective amount of motion blur for improving compression efficiency, even if the image quality is deteriorated, the parameter when the motion blur is added is known and the subsequent motion blur reduction processing is performed. Therefore, it is possible to restore the image quality.

[4. program]

In the above embodiment, the image processing apparatuses 1 (100) and 4 (400) have been described. However, the present invention can be applied to various devices that perform image processing. For example, an image reproducing device, an imaging device, a communication device, an image recording device, a game machine, a video editing machine, and the like are assumed.
Furthermore, it is naturally assumed that the image processing apparatuses 1 (100) and 4 (400) are realized in a general-purpose personal computer or other information processing apparatus.
For example, by providing a program for causing an arithmetic processing unit to execute the operation of each processing block of FIGS. 1, 3, 4, 13, 14, and 15 as image processing application software, Image processing can be realized.

That is, the program that realizes image processing of the image processing apparatus 100 corresponds to motion information MI (for example, motion vector VD) indicating the motion of an image between unit images constituting the motion image data ID with respect to the motion image data ID. A motion blur addition step for performing the motion blur addition process, and a coding step for encoding and outputting the moving image data BD subjected to the motion blur addition process in the motion blur addition step. Is a program that causes the information processing apparatus to execute the program.
If there is only the motion blur adding step, it is a program for realizing the image processing of the image processing apparatus 1.

The program for realizing the image processing of the image processing device 400 includes a decoding step for decoding the moving image data ED in an encoded state after the motion blur addition processing is performed, and a decoding obtained by the decoding step. By applying a filtering process to the converted moving image data DD in accordance with motion information MI (for example, a motion vector VD) indicating the motion of the image between unit images constituting the moving image data, motion blur is reduced. A program that causes an information processing apparatus to execute a motion blur reduction step for executing a reduction process.
If there is only the motion blur reduction step, it is a program for realizing the image processing of the image processing device 4.

  With such a program, the present invention can be executed in a personal computer, a cellular phone, a PDA (Personal Digital Assistant), and other various information processing apparatuses using image data.

Such a program can be recorded in advance in an HDD as a recording medium built in a device such as a personal computer, a ROM in a microcomputer having a CPU, a flash memory, or the like.
Alternatively, temporarily or permanently on a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magnet optical) disk, DVD, Blu-ray disc, magnetic disk, semiconductor memory, memory card, etc. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.
In addition to installing the program from a removable recording medium to a personal computer or the like, the program can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

  1, 100, 4,400 Image processing device, 2 encoding processing unit, 3 decoding processing unit, 11 motion blur addition processing unit, 12, 42 motion vector generation processing unit, 41 motion blur reduction processing unit

Claims (6)

Motion vector generating means for generating a motion vector as motion information indicating the motion of an image between unit images constituting the moving image data from the moving image data which has been subjected to motion blur addition processing and encoded ,
Coded video data in terms of the motion blur addition process has been performed is input is decoded, with respect to the entered the video data, the motion generated by the motion vector generating means It is equipped with motion blur reduction means for performing motion blur reduction processing by applying filter processing using vectors ,
The above motion blur reduction means is
An image for selecting whether or not to perform the motion blur reduction process based on the motion blur addition parameter used when the motion blur addition process is performed, or for adjusting the degree of the motion blur reduction process Processing equipment. The motion blur addition parameter, flag information indicating whether added with the motion blur for each area of the unit image of the moving image data, or the image processing apparatus according to claim 1 is the correction information of the motion vector. The image processing apparatus according to claim 1, wherein the motion blur reducing unit performs a filter process for reducing motion blur using a motion vector used for decoding moving image data in a preceding decoding process. The image processing apparatus according to claim 1, further comprising a decoding unit configured to decode the moving image data encoded after the motion blur addition process. A motion vector generation step for generating a motion vector as motion information indicating the motion of an image between unit images constituting the moving image data from the moving image data which has been subjected to the motion blur addition process; ,
A decoding step for decoding moving picture data encoded in terms of the motion blur addition process has been applied,
A motion blur that performs a motion blur reduction process by performing a filtering process using the motion vector generated in the motion vector generation step on the decoded moving image data obtained in the decoding step. Reduction steps and
And having a,
In the above motion blur reduction step,
An image for selecting whether or not to perform the motion blur reduction process based on the motion blur addition parameter used when the motion blur addition process is performed, or for adjusting the degree of the motion blur reduction process Processing method. A motion vector generation step for generating a motion vector as motion information indicating the motion of an image between unit images constituting the moving image data from the moving image data which has been subjected to the motion blur addition process; ,
A decoding step for decoding moving picture data encoded in terms of the motion blur addition process has been applied,
A motion blur reduction process is performed on the decoded moving image data obtained by the decoding step by performing a filtering process using the motion vector generated by the motion vector generation step . A motion that selects whether or not to execute the motion blur reduction process based on the motion blur addition parameter used when the motion blur addition process is performed, or adjusts the degree of the motion blur reduction process program for executing a blur reduction step to the information processing apparatus.

Images