JP4441865B2 - Image processing method, image processing apparatus, program, and program recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce data by taking into account the visual effect of human eyes. <P>SOLUTION: The image processing method includes steps of: filtering moving picture data at a frame rate of 1/t<SB>o</SB>and applying 1/4-down-sampling to the moving picture data in a time direction by using a region R<SB>1101</SB>, extended in a principal component direction of the moving picture data at the frame rate of 1/t<SB>o</SB>and including a particular width 2&pi;/(4t<SB>0</SB>) in a direction of a frequency axis T in the time direction in a frequency domain defined by the frequency axis T in the time direction and frequency axes X, Y in a space direction, for a pass band to obtain moving picture data at a frame rate of 1/(4t<SB>0</SB>); determining the region R<SB>1101</SB>extended in the principal component direction of the moving picture data at the frame rate of 1/(4t<SB>0</SB>) to be the pass band; and applying up-sampling in the time direction to the moving picture data at the frame rate of 1/(4t<SB>0</SB>) while applying filtering with the pass band of R<SB>1101</SB>to the moving picture data to obtain moving picture data at the frame rate of 1/t<SB>0</SB>. The present invention can be applied to processing of moving pictures at a high frame rate. <P>COPYRIGHT: (C)2006,JPO&amp;NCIPI

Description

The present invention relates to an image processing method and image processing apparatus, and a program and a program recording medium thereof, in particular, for example, an image processing method capable of performing reduction of the moving image data in consideration of visual effect of the human eye and an image processing apparatus, and a program and a program recording medium thereof.

  Conventionally, moving image (moving image) data is recorded (recorded) and reproduced at a frame rate of about 30 to 60 frames per second (30 to 60 fps (Frame per second)). However, at such a frame rate, a moving subject (moving subject) is perceived as a blurred image in human vision, and thus it cannot be said that good image quality is obtained for humans.

  By the way, it is known that when moving image data is recorded and reproduced at a frame rate of about 240 fps, image quality good for human beings can be obtained.

  Video data with a frame rate of 240 fps can be displayed on a display device that supports a high frame rate of 240 fps, and further, for example, by thinning out frames and reducing the frame rate, a low frame rate such as 30 fps or 60 fps It can also be displayed on the display device.

  However, when the frames of moving image data having a frame rate of 240 fps are simply thinned out, the smoothness of the motion is lost in the image.

  Therefore, Patent Document 1 proposes a method in which an average value of a plurality of frames of high frame rate moving image data is used as low frame rate moving image data, instead of simply thinning out high frame rate moving image data. . Further, Patent Document 1 proposes a method of hierarchically encoding high frame rate moving image data into a plurality of layers corresponding to a plurality of frame rates.

Here, for example, when two-layer encoding of a 120 fps layer and a 60 fps layer is performed, for example, data D ₁ of a certain frame and data D of the next frame of 120 fps moving image data are used. average of _{2 (D 1 + D 2)} / 2 is set to the hierarchy of the moving image data of 60 fps, of one of the data D ₁ and D _2, and the example, only the data D ₁ is 120 fps hierarchy Video data. In this case, the data D ₂ is the moving image data of 60fps hierarchy _{(D 1 + D 2) /} 2, it can be obtained from the moving picture data D ₁ Metropolitan of 120fps hierarchy.

  As described above, high-frame-rate video data such as 240 fps can provide good image quality, but 240 fps video data is simpler than conventional 30-fps or 60-fps video data. The amount of data is 8 times or 4 times larger. This data amount does not change even when the above-described hierarchical encoding is performed. An enormous amount of moving image data increases the data processing load of the moving image data, and is not preferable for transmission or recording of moving image data.

JP 2004-088244 A.

  The present invention has been made in view of such a situation, and makes it possible to reduce moving image data in consideration of the visual effect of the human eye.

The first image processing method of the present invention detects a principal component direction that is a direction of a first principal component of moving image data on a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction. The principal component direction detection step and the region extending in the principal component direction in the frequency domain and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction are determined as the passband of the filter. This is an image processing method including an area determining step to output and an output step of outputting the pass band of the filter as filter information .

The first image processing apparatus of the present invention detects a principal component direction that is a direction of a first principal component of moving image data on a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction. The principal component direction detection means and the region extending in the principal component direction in the frequency domain and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction are determined as the passband of the filter. And an area determination means for outputting as filter information .

A first program of the present invention is a principal component for detecting a principal component direction which is a direction of a first principal component of moving image data on a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction. A direction detection step and a region that extends in the main component direction in the frequency domain and has a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction as a pass band of the filter A program for causing a computer to execute image processing including a determination step and an output step of outputting the pass band of the filter as filter information .

The program recorded on the first program recording medium of the present invention is the direction of the first main component of the moving image data on the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. A principal component direction detection step for detecting a principal component direction, and a region extending in the principal component direction in the frequency domain and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction, A program for causing a computer to execute image processing including an area determining step for determining a pass band of a filter and an output step for outputting the pass band of the filter as filter information .

The second image processing method of the present invention is a principal component that is the direction of the first principal component of the moving image data of the first frame rate in the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. The first frame rate moving image data is filtered using the region extending in the direction and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction, and in the time direction. An image processing method including a filtering step of obtaining moving image data having a second frame rate by down-sampling, and an output step of outputting moving image data having a second frame rate obtained in the filtering step .

The second image processing apparatus of the present invention is a principal component that is the direction of the first principal component of the moving image data of the first frame rate in the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. The first frame rate moving image data is filtered using the region extending in the direction and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction, and in the time direction. The image processing apparatus includes a filter unit that obtains moving image data having a second frame rate by down-sampling .

The second program of the present invention is arranged in a principal component direction that is a direction of the first principal component of the moving image data of the first frame rate in the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. The first frame rate video data is filtered and the down-sampling is performed in the time direction using a region extending as a pass band in the direction of the frequency axis in the time direction and having a specific width corresponding to the visual integration function. By doing so, the program causes the computer to execute image processing including a filter step of obtaining moving image data of the second frame rate and an output step of outputting moving image data of the second frame rate obtained in the filtering step. .

The program recorded on the second program recording medium of the present invention is a first principal component of moving image data having a first frame rate in a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction. The moving image data of the first frame rate is filtered using a region extending in the principal component direction, which is a direction, and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction. In addition, the image processing includes a filter step of obtaining moving image data of the second frame rate by down-sampling in the time direction, and an output step of outputting moving image data of the second frame rate obtained in the filtering step Is a program that causes a computer to execute.

The third image processing method of the present invention is the direction of the first principal component of the moving image data of the second frame rate on the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. A principal component direction acquisition step for acquiring a principal component direction, and a region extending in the principal component direction in the frequency domain and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction, By performing upsampling in the time direction while applying a passband determining step for determining the passband of the filter and applying a filter that passes the frequency component of the passband to the moving image data of the second frame rate, And a filter step for obtaining moving image data of one frame rate .

The third image processing apparatus of the present invention is the direction of the first main component of the moving image data of the second frame rate on the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. A principal component direction acquisition means for acquiring a principal component direction; and a region extending in the principal component direction in the frequency domain, and having a specific width according to a visual integration function in the direction of the frequency axis in the time direction, By performing upsampling in the time direction while applying a passband determining means that determines the passband of the filter and a filter that passes the frequency component of the passband to the moving image data of the second frame rate, An image processing apparatus comprising: filter means for obtaining moving image data having a frame rate of 1 .

The third program of the present invention is a principal component that is the direction of the first principal component of the moving image data of the second frame rate on the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. A main component direction acquisition step for acquiring a direction, and a region extending in the main component direction in the frequency domain and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction. By performing upsampling in the time direction while applying a passband determination step for determining the passband and a filter that passes the frequency component of the passband to the moving image data of the second frame rate, A program for causing a computer to execute image processing including a filter step for obtaining frame rate moving image data .

The program recorded on the third program recording medium of the present invention is the first of the moving image data of the second frame rate on the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction . A principal component direction acquisition step for acquiring a principal component direction, which is a principal component direction, and a region extending in the principal component direction in the frequency domain, and a specific direction corresponding to a visual integration function in the direction of the frequency axis in the time direction Up-sampling in the time direction while applying a pass band determining step for determining a region having a width as a pass band of the filter and a filter that passes the frequency component of the pass band to moving image data of the second frame rate by performing, professional to execute image processing including a filter to obtain a moving picture data of the first frame rate to the computer It is a lamb.

In the first image processing method, image processing apparatus, program, and program recording medium of the present invention, the first moving image data on the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction . The principal component direction that is the direction of the principal component is detected, and in the frequency domain, a region that extends in the principal component direction and has a specific width according to the visual integration function in the direction of the frequency axis in the time direction, It is determined as the pass band of the filter.

In the second image processing method, image processing apparatus, program, and program recording medium of the present invention, the moving image having the first frame rate in the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction. An area extending in the principal component direction, which is the direction of the first principal component of data, and having a specific width in accordance with the visual integration function in the direction of the frequency axis in the time direction is used as the first frame. The moving image data of the second frame rate is obtained by filtering the moving image data of the rate and down-sampling in the time direction.

In the third image processing method, image processing apparatus, program, and program recording medium of the present invention, the second frame rate on the frequency domain defined by the time axis in the time direction and the frequency axis in the spatial direction. The principal component direction that is the direction of the first principal component of the moving image data is acquired and is a region extending in the principal component direction in the frequency domain, and is specified in accordance with the visual integration function in the direction of the frequency axis in the time direction. A region having a width is determined as the passband of the filter. Then, the moving image data having the first frame rate is obtained by performing upsampling in the time direction while applying the filter that passes the frequency component of the pass band to the moving image data having the second frame rate.

  According to the present invention, data can be reduced.

  Next, before describing the embodiment of the present invention, a theoretical matter will be described for moving images.

  FIG. 1 shows moving image data in a three-dimensional space in which a time direction (t) is added to a two-dimensional plane (x, y).

In FIG. 1, an image P ₁₀₁ is an image at a certain time (frame), an image P ₁₀₂ is an image at the next time, and an image P ₁₀₃ is an image at the next time. The image P ₁₀₄ is an image at the next time. Although illustration is omitted, there are also images at a time before the image P ₁₀₁ and a time after the image P ₁₀₄ .

In FIG. 1, the images P _{101 to} P ₁₀₄ show an object (subject) that moves in the y direction over time.

  When a movie such as the above (moving image data) is viewed by a human, when the frame rate of the moving image is higher than a certain frame rate, the difference between two adjacent frames (images) of the moving image can be recognized. Absent. As a result of actually conducting a visual experiment on the subject, it has been found that the frame rate at which a human cannot recognize the difference between two adjacent frames (images) is about 240 fps or more.

Here, if the time (frame period) between two adjacent frames is expressed as t ₀ , the frame rate can be expressed as 1 / t ₀ .

In the following description, the frame period t ₀ is about 1/240 seconds. However, the frame period t ₀ may not be about 1/240 seconds, for example, about 1/120 seconds. good. However, when the frame period t ₀ is longer than the frame period (about 1/240 seconds) in which a human cannot recognize the difference between two adjacent frames (images thereof), for example, 1/120 second. The image quality of the moving image displayed on the display device 3 (FIG. 24) described later is somewhat perceived.

Next, r ₀ represents a human's recognition ability in the spatial direction, that is, a limit distance at which two adjacent points can be recognized (perceived) as two points instead of one point.

  In this case, the range of moving images that can be recognized (perceived) by human vision can be expressed as shown in FIG.

  That is, FIG. 2 shows a frequency domain defined by the frequency axis T in the time t direction and the frequency axes X and Y in the spatial directions x and y.

In order to avoid complication of the figure, in FIG. 2, the frequency axis X in the spatial direction x and the frequency axis Y in the spatial direction y are represented by one axis. That is, in FIG. 2, the frequency axes X and Y are taken from left to right. In FIG. 2, the frequency axis T is taken from the top to the bottom. Further, in FIG. 2, the frequency axis T (vertical direction) is divided in units of 2π / (4t ₀ ), and the frequency axes X and Y directions (lateral direction) are divided in units of 2π / r ₀ .

  Here, other diagrams of the frequency domain, which will be described later, are the same as those in FIG.

Humans, shown in FIG. 2, in the horizontal is _{2 × 2π / (2r 0)} , the vertical of _{2 × 2π / (2r 0)} , recognizes the high frequency components are outside the range of the region R ₂₀₁ centered at the origin Can not do it. Therefore, the following description will be made assuming that all moving image data exists in the region _R201 .

In FIG. 2, the region R ₂₀₁ is illustrated as a rectangular region, but the region R ₂₀₁ is actually a rectangular parallelepiped region. That is, the region R ₂₀₁ has an X direction of − (π / r ₀ ) to + (π / r ₀ ), a Y direction of − (π / r ₀ ) to + (π / r ₀ ), and T The direction is a region in the range of − (π / t ₀ ) to + (π / t ₀ ).

  Next, the position (distribution) on the frequency domain of the data of each part of the moving image will be described.

  In a moving image, a subject projected on a certain portion is stationary, and a subject projected on the moving portion is moving in another portion.

  First, the stationary moving image portion (the moving image portion on which the stationary subject is projected) does not change with respect to the time direction.

FIG. 3 shows a region R ₃₀₁ on the frequency domain in which data of such a still moving image portion is distributed.

The region R ₃₀₁ is a region in which the X and Y directions are −2π / (2r ₀ ) to + 2π / (2r ₀ ) and the T direction is −2π / (4t ₀ ) to + 2π / (4t ₀ ). .

Note that if the data is a completely stationary part of the video, T = 0, that is, the region R ₃₀₁ has a width in the T direction of 0, but here the subject is moving slightly. In consideration of the above, the region R ₃₀₁ has a width in the T direction that is not 0 but 2π / (2t ₀ ). The X and Y directions are limited to the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ) as described with reference to FIG. This is because it does not exist in a range exceeding the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ) (outside the range of the region R _{201 in} FIG. 2).

  Here, in the frequency domain, the data of the stationary part is distributed in the direction of the straight line (plane) of T = 0, so this direction is the main component (for example, the first data of the stationary part). Main component) direction.

Next, FIG. 4 shows a region R401 on the frequency domain in which data of a portion where a subject projected on a moving image moves at a speed (r ₀ / t ₀ ) / 2 is distributed.

  Here, in order to simplify the explanation as appropriate, the points on the frequency domain are expressed by using only two coordinates A and B in the X and Y directions, and (A, B). It expresses.

  The region R401 extends from the origin (0,0) to the point (π / r0, −π / (2t0)) and has a width of 2π / (2t0) in the T direction. .

It should be noted that if the subject does not deform and moves accurately at a constant linear velocity at a speed (r ₀ / t ₀ ) / 2, the region R401 has a width in the T direction of 0. The width in the T direction is not 0 but 2π / (2t0) in consideration of some deformation and the movement speed slightly deviating from (r ₀ / t ₀ ) / 2.

In the region R401, the X and Y directions are limited to the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ), but this is limited to the range of the region R _{301 in} the X and Y directions. For the same reason that has been done.

Here, the data of the portion where the subject projected on the moving image moves at a speed (r ₀ / t ₀ ) / 2 is obtained from the origin (0,0) to the point (π / r0, −π / (2t0) ) Is distributed for the following reason.

That is, the waveform R _{501 in} FIG. 5 shows the distribution of the subject in the spatial direction x at a certain time t ₁ .

For example, when the subject is moving at a speed (r ₀ / t ₀ ) / 2, for example, in the spatial direction x, the moving subjects are waveform R ₅₀₁ , R ₅₀₂ , R ₅₀₃ , R ₅₀₄ , R ₅₀₅ , R ₅₀₆ ,...

Here, the waveform R ₅₀₁ represents the subject at a certain time (frame) t ₁ . Waveform R ₅₀₂ represents a subject at the next time (frame) t ₁ + t ₀ of time t ₁ of the waveform R _501, from the position of the waveform R _501, only r _0/2, moves in the x-direction Yes. Hereinafter, similarly, waveforms R ₅₀₃ , R ₅₀₄ , R ₅₀₅ , and R ₅₀₆ represent subjects at times t ₁ + 2t ₀ , t ₁ + 3t ₀ , t ₁ + 4t ₀ , and t ₁ + 5t _{0, respectively} . Yes.

The waveform in FIG. 5 is considered as a function of the spatial position x and time t, and this waveform is defined as Func (x, t). As is clear from the waveforms R ₅₀₁ , R ₅₀₂ , R ₅₀₃ , R ₅₀₄ , R ₅₀₅ , R ₅₀₆ ,..., Func (x, t) = Func (x − ((r ₀ / t ₀ ) / 2) × t, 0).

Assuming that the data on the frequency domain obtained by two-dimensional Fourier transform of the function Func with two variables x and t is (X, T), the component of (X, T) is apparent from the definition of Fourier transform. In addition, a component in which the period in the spatial direction x is X and the period in the time direction t is T is represented.

From the relationship Func (x, t) = Func (x − ((r ₀ / t ₀ ) / 2) × t, 0), (X, T) is a two-dimensional vector (r _0/2 , t ₀ ) Can be said to be orthogonal.

  Although described with reference to FIG. 5, the spatial direction is actually two-dimensional, and when the time axis is added, it becomes three-dimensional, and a three-dimensional Fourier transform must be considered. Then, it demonstrates again using FIG. FIG. 6 is a diagram for explaining three-dimensionally as in FIG. 1.

A waveform R ₆₀₁ on the two-dimensional space (x, y) in FIG. 6 shows the distribution in the spatial direction (x, y) of the subject at a certain time t ₁ .

For example, when the subject is moving at a speed (r ₀ / t ₀ ) / 2, the moving subject is represented by waveforms R ₆₀₁ , R ₆₀₂ , R ₆₀₃ , R _604,. Can do. Note that r ₀ is a two-dimensional vector on a two-dimensional space (x, y) unlike the description of FIG.

Here, the waveform R ₆₀₁ represents the subject at a certain time (frame) t ₁ . Waveform R ₆₀₂ represents a subject at the next time (frame) t ₁ + t ₀ of time t ₁ of the waveform R _601, from the position of the waveform R _601, is moved by r _0/2. Hereinafter, similarly, waveforms R ₆₀₃ and R ₆₀₄ represent subjects at times t ₁ + 2t ₀ and t ₁ + 3t _0, respectively.

The waveform in FIG. 6 is considered as a function of the spatial position (x, y) and time t, and this waveform is defined as Func ((x, y), t). As apparent from the waveforms R ₆₀₁ , R ₆₀₂ , R ₆₀₃ , R ₆₀₄ ,... In FIG. 6, Func ((x, y), t) = Func ((x, y) − ((r ₀ / t ₀ ) / 2) × t, 0).

  Now, if the data on the frequency domain obtained by three-dimensional Fourier transform of the function Func with three variables x, y and t is (X, Y, T), the component of (X, Y, T) is Fourier transform. As is clear from the definition of FIG. 4, a component in which the period in the spatial direction x is X, the period in the spatial direction y is Y, and the period in the time direction t is T is represented.

From the relationship Func ((x, y), t) = Func ((x, y)-((r ₀ / t ₀ ) / 2) × t, 0), (X, Y, T) is three-dimensional It can be said that it is orthogonal to the vector (r _0/2 , t ₀ ).

As shown in FIGS. 5 and 6, since (X, T) or (X, Y, T) is orthogonal to the vector (r _0/2 , t ₀ ), for example, as shown in FIG. There is data at a point (2π / (2r ₀ ), −2π / (4t ₀ )) (position indicated by a circle in FIG. 7) on the frequency domain, or a point (2π / (4r ₀ ), −2π / There is data on (8t ₀ )) (the position indicated by Δ in FIG. 7).

That is, each waveform moving at a speed (r ₀ / t ₀ ) / 2 is a straight line (plane) R passing through the origin (0,0) and the point (π / r0, -π / (2t0)). Distributed on ₇₀₁ .

Similarly, the data (waveform) of the other frequency component X of the portion moving at the speed (r ₀ / t ₀ ) / 2 is also distributed on the straight line R ₇₀₁ .

The moving parts at a rate _{(r 0 / t 0) /} 2 , when the time advances by t _0, since r _0/2 only spatial directions position shifts, as described above, in a frequency domain, the origin ( 0,0) and the point (π / r0, -π / ( 2t0) distributed on a straight line R ₇₀₁ through a).

In the frequency domain, the data of the portion moving at the speed (r ₀ / t ₀ ) / 2 is distributed in the direction of the straight line (plane) R ₇₀₁ (π / r0, −π / (2t0)). This direction (π / r0, −π / (2t0)) is the direction of the principal component of the data of the portion moving at the speed (r ₀ / t ₀ ) / 2.

Next, FIG. 8 shows a region R ₈ 01 on the frequency domain in which data of a portion in which a subject projected on a moving image moves at a speed r ₀ / t ₀ is distributed.

The region R ₈ 01 is a region extending in the direction from the origin (0,0) to the point (π / r0, −π / t0) and having a width of 2π / (2t0) in the T direction. .

That is, the portion moving at the speed r ₀ / t ₀ is shifted in position in the spatial direction by r ₀ when the time advances by t _0, so that the origin (0,0) and the point (π / r0, -π / t0).

Note that if the subject does not deform and moves accurately linearly at a constant velocity r ₀ / t ₀ , the region R ₈ 01 has a width of 0 in the T direction. In consideration of deformation and the movement speed slightly deviating from r ₀ / t ₀ , the width in the T direction is not 0 but 2π / (2t 0).

In the region R ₈ 01, the X and Y directions are limited to the range of −2π / (2r ₀ ) to + 2π / (2r ₀ ), which is the same as the region R _{301 in} FIG. For the same reason that the range of directions is limited.

Further, in the region R ₈ 01, the T direction is limited to a range of −2π / (2t ₀ ) to + 2π / (2t ₀ ), which is the frequency of the moving image data as described with reference to FIG. This is because the component does not exist in the T direction beyond the range of −2π / (2t ₀ ) to + 2π / (2t ₀ ) (outside the range of the region R _{201 in} FIG. 2).

Here, in the frequency domain, the data of the portion moving at the velocity r ₀ / t ₀ is the direction of the straight line (π / t) passing through the origin (0,0) and the point (π / r0, −π / t0). r0, -π / t0), this direction (π / r0, -π / t0) is the direction of the main component of the data of the portion moving at the speed r ₀ / t ₀ .

Next, FIG. 9 shows a region R ₉ 01 on the frequency domain data of the part object is projected onto a video is moving at about the speed 2r ₀ / t ₀ is distributed.

Region R ₉ 01 is the point from the origin (0,0) (π / r0, -2π / t0) an area extending in the direction of, and is a region having a width of 2π / (2t0) in the T direction .

That is, the portion moving at the speed 2r ₀ / t ₀ is shifted in the spatial direction by 2r ₀ when the time advances by t _0, so that the origin (0,0) and the point (π / r0, -2π / t0).

Incidentally, without the subject is deformed, if exactly uniform linear motion at a speed 2r ₀ / t _0, the region R ₉ 01, although the width of the T direction is 0, here, some of the subject In consideration of deformation and the movement speed slightly deviating from r ₀ / t ₀ , the width in the T direction is not 0 but 2π / (2t 0).

In the region R ₉ 01, the T direction is limited to the range of −2π / (2t ₀ ) to + 2π / (2t ₀ ). This is because the frequency of the moving image data is as described with reference to FIG. This is because the component does not exist in the T direction beyond the range of −2π / (2t ₀ ) to + 2π / (2t ₀ ) (outside the range of the region R _{201 in} FIG. 2).

Here, in the frequency domain, the data of the portion moving at the speed 2r ₀ / t ₀ is the direction of the straight line (π / 0) passing through the origin (0,0) and the point (π / r0, -2π / t0). r0, -2π / t0), this direction (π / r0, -2π / t0) is the direction of the main component of the data of the portion moving at the speed 2r ₀ / t ₀ .

As described above, when the subject is stationary, and when the subject is moving at a speed (r ₀ / t ₀ ) / 2, r ₀ / t ₀ , 2r ₀ / t ₀ , such a subject is The distribution on the frequency domain of the data of the portion of the moving image being projected has been described, but the data of the portion moving at other speeds is similarly distributed in the frequency domain.

  Next, human visual effects will be described.

  The human eye recognizes the light emitted from the object by integrating it in the time direction. This integration function (visual integration function) in the human eye is called the Block's Law. The integration time of light in the human eye is approximately 1/60 seconds, although it varies depending on the environment.

Here, if t ₀ is about 1/240 seconds as described above, the integration time of light in the human eye can be expressed as 4 × t ₀ seconds.

An image in which a stationary subject is projected in a moving image is recognized by a human gaze without moving the line of sight. This is equivalent to an image obtained by filtering with a low-pass filter of about “4 × t ₀ ” seconds.

The pass band of this low-pass filter can be represented as a region R ₁₀₀₁ shown in FIG. 10 on the frequency domain.

In the region R _{1001 in} FIG. 10, the range in the X and Y directions is −2π / (2r ₀ ) to + 2π / (2r ₀ ), which is the same as the region R _{201 in} FIG. 2, and the range in the T direction is − ( The region is 2π / (4t ₀ )) / 2 to (2π / (4t ₀ )) / 2. The width in the T direction of this region R ₁₀₀₁ is 2π / (4t ₀ ), which represents the integration of the time of 4t ₀ .

As shown in FIG. 3, the data of the still part of the moving image exists in the region R ₃₀₁ , but the human recognizes only the information (data) in the region R ₁₀₀₁ in the region R _301. Can not do it. Therefore, information outside the region R ₁₀₀₁ in the region R ₃₀₁ is useless information for humans.

Next, while moving the line of sight in the direction in which the subject moves in the part of the moving image where the subject being projected is moving at a speed of “(r ₀ / t ₀ ) / 2” An image that is recognized by gazing, that is, an image that a human recognizes by following vision, uses data of a portion moving at a speed of “(r ₀ / t ₀ ) / 2” in the time direction by a visual integration function. This is equivalent to an image obtained by filtering with a low-pass filter of about “4 × t ₀ ” seconds.

However, in the following tracking in this case, the gazing point moves at a speed of about “(r ₀ / t ₀ ) / 2” with the passage of time. Therefore, the pass band of the low-pass filter is as shown in FIG. Can be represented as a region R ₁₁₀₁ shown in FIG.

A region R _{1101 in} FIG. 11 is a region extending in the direction from the origin (0,0) to the point (π / r0, −π / (2t0)), similarly to the region R401 in FIG. Further, a region R ₁₁₀₁ is a region having a width of 2π / (4t0) in the T direction, and this represents an integration of time of 4t ₀ . Note that the area R ₁₁₀₁ is limited to the area R ₂₀₁ where the moving image data exists, as shown in FIG.

Speed _{"(r 0 / t 0) /} 2 " data of the portion moving at a degree, as shown in FIG. 4, are present in the region R _401, human, of its region R ₄₀₁ Only the information in the region R ₁₁₀₁ can be recognized. Therefore, information outside the region R ₁₁₀₁ in the region R ₄₀₁ is useless information for humans.

Next, the image in which the subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ” by human follow-up is still the visual integration. This function is equivalent to an image obtained by filtering data of a portion moving at a speed of about “r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

However, in the following tracking in this case, the gazing point moves at a speed of about “r ₀ / t ₀ ” with the passage of time, so the pass band of the low-pass filter is the region R shown in FIG. 12 on the frequency domain. Can be represented as ₁₂₀₁ .

Region R _{1201 in} FIG. 12 is a region extending in the direction from the origin (0, 0) to the point (π / r 0, −π / t 0), similarly to the region R ₈ 01 in FIG. Further, the region R ₁₂₀₁ is a region having a width of 2π / (4t0) in the T direction, and this represents the integration of time of 4t ₀ . Note that the region R ₁₂₀₁ is limited to the region R ₂₀₁ where the moving image data exists as shown in FIG.

As shown in FIG. 8, the data of the portion moving at the speed “r ₀ / t ₀ ” is present in the region R ₈₀₁ , but the human is the region R _{1201 in the} region R _801. Only the information inside can be recognized. Therefore, information outside the region R ₁₂₀₁ in the region R ₈₀₁ is useless information for humans.

Next, an image that is recognized by humans following the moving part of the moving subject at a speed of about “2r ₀ / t ₀ ” in the moving image is still a visual integral. This function is equivalent to an image obtained by filtering data of a portion moving at a speed of about “2r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

However, in the follow-up view in this case, since the gazing point moves at a speed of about “2r ₀ / t ₀ ” with time, the pass band of the low-pass filter is the region R shown in FIG. 13 on the frequency domain. Can be represented as ₁₃₀₁ .

A region R _{1301 in} FIG. 13 is a region extending in the direction from the origin (0,0) to the point (π / r0, −2π / t0), similarly to the region R ₉₀₁ in FIG. Further, the region R ₁₃₀₁ is a region having a width of 2π / (4t0) in the T direction, and this represents the integration of time of 4t ₀ . Note that the region R ₁₃₀₁ is limited to the region R ₂₀₁ where the moving image data exists as shown in FIG.

As shown in FIG. 9, the data of the portion moving at the speed “2r ₀ / t ₀ ” exists in the region R ₉₀₁ , but the human is the region R _{1301 in the} region R _901. Only the information inside can be recognized. Therefore, information outside the region R ₁₃₀₁ in the region R ₉₀₁ is useless information for humans.

  Next, a case where a human performs fixed vision will be described. Here, fixed vision is different from follow-up vision in the case of moving the line of sight different from the movement of the subject. For example, humans do not move the line of sight (keep the viewing direction constant) Applicable when watching a video.

An image that is recognized by a human fixed view of a moving part of a moving image at a speed of about “(r ₀ / t ₀ ) / 2” in a moving image is obtained by a visual integration function. This is equivalent to an image obtained by filtering data in a portion moving at a speed of “(r ₀ / t ₀ ) / 2” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

The pass band of this low-pass filter can be represented by the same region on the frequency domain as the case where a stationary portion is watched, that is, a region R ₁₀₀₁ in FIG.

Here, FIG. 14 shows a region R ₄₀₁ (FIG. 4) in which data of a portion moving at a speed of “(r ₀ / t ₀ ) / 2” in the frequency domain is distributed, and a case where a human performs fixed vision. FIG. 10 shows a region R ₁₀₀₁ (FIG. 10) as a pass band of the low-pass filter.

Humans, of area R _401, information in the region R _1001, that is, can not be a region R ₄₀₁ and R ₁₀₀₁ are only recognized information in the region R ₁₄₀₁ of the portions overlapping. Therefore, information outside the region R ₁₀₀₁ in the region R ₄₀₁ is useless information for humans.

In FIG. 14, a region R ₁₀₀₁ as a pass band of the low-pass filter when a human performs fixed vision is a region R ₁₄₀₂ and R ₁₄₀₃ (parts R ₁₄₀₂ and R _{1403 in the} figure) that do not overlap with the region R ₄₀₁ where the moving image data exists. In this case, there is no moving image data in the regions R ₁₄₀₂ and R ₁₄₀₃ .

Further, as described above, when a human is fixedly viewing a portion that is moving at a speed of about “(r ₀ / t ₀ ) / 2”, the human gaze direction is the moving subject (moving subject). Therefore, even a slightly blurred image is recognized as a good moving image.

Therefore, the data in the region R ₁₄₀₁ in FIG. 14 has a low-pass filter when the gazing point moves at a speed of “(r ₀ / t ₀ ) / 2”, that is, the region R _{1101 in} FIG. Even if it is filtered with a low-pass filter, it is recognized as a moving image with good image quality.

From the above, as shown in FIG. 15, if there is data in a region R ₁₅₀₁ of the region R ₁₄₀₁ that overlaps the region R ₁₁₀₁ (shown in black in the drawing), the speed “(r ₀ / A portion moving at about t ₀ ) / 2 ”is recognized as a moving image with good image quality.

Therefore, in the case where a portion of a moving image moving at a speed of “(r ₀ / t ₀ ) / 2” is fixedly viewed by a human, the data of that portion, that is, the data in the region R ₄₀₁ is fixed. Region R ₁₀₀₁ as the pass band of the low-pass filter when viewing is performed, and region R as the pass band of the low-pass filter when the gazing point moves at a speed of “(r ₀ / t ₀ ) / 2” The data in the area excluding the area R ₁₅₀₁ that overlaps the area ₁₁₀₁ is useless (even if it does not, it hardly affects the image quality recognized by humans).

Next, an image that is recognized by a human fixed view of a moving part of a moving image at a speed of about “r ₀ / t ₀ ” in a moving image is obtained by a visual integration function, This is equivalent to an image obtained by filtering data in a portion moving at a speed of about “r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

The pass band of this low-pass filter can be represented by the same region on the frequency domain as the case where a stationary portion is watched, that is, a region R ₁₀₀₁ in FIG.

Here, FIG. 16 shows a region R ₈₀₁ (FIG. 8) in which data of a portion moving at a speed of “r ₀ / t ₀ ” in the frequency domain is distributed, and a low-pass filter when a human performs fixed vision. A region R ₁₀₀₁ (FIG. 10) as a pass band is shown.

Humans, of area R _801, information in the region R _1001, that is, can not be a region R ₈₀₁ and R ₁₀₀₁ are only recognized information in the region R ₁₆₀₁ of the portions overlapping. Therefore, information outside the region R ₁₀₀₁ in the region R ₈₀₁ is useless information for humans.

In FIG. 16, a region R ₁₀₀₁ as a pass band of the low-pass filter when a human performs fixed vision is a portion of regions R ₁₆₀₂ and R ₁₆₀₃ that do not overlap with the region R ₈₀₁ where moving image data exists (in the drawing) In this case, there is no moving image data in these areas R ₁₆₀₂ and R ₁₆₀₃ .

In addition, as described above, when a human is fixedly viewing a portion that is moving at a speed of about “r ₀ / t ₀ ”, the direction of the human gaze is the movement of the moving subject (moving subject). Since it moves differently, even a slightly blurred image is recognized as a good moving image.

Therefore, the data in the region R ₁₆₀₁ in FIG. 16 is filtered by a low-pass filter when the gazing point moves at a speed of “r ₀ / t ₀ ”, that is, a low-pass filter having the region R _{1201 in} FIG. Even so, it is recognized as a moving image with good image quality.

From the above, as shown in FIG. 17, if there is data in a region R ₁₇₀₁ (shown in black in the drawing) that overlaps the region R ₁₂₀₁ in the region R ₁₆₀₁ , the speed “r ₀ / t A portion moving at about “ ₀ ” is recognized as a moving image with good image quality.

Therefore, when a human views a moving part of a moving image at a speed of about “r ₀ / t ₀ ”, the fixed view is performed from the data of that part, that is, the data in the region R ₈₀₁ . Region R ₁₀₀₁ as the pass band of the low-pass filter in the case where the gazing point moves and the region R ₁₂₀₁ as the pass band of the low-pass filter when the gazing point moves at a speed of about “r ₀ / t ₀ ” The data in the area excluding ₁₇₀₁ is useless.

Next, an image that is recognized by a human fixed view of a moving part of a moving image at a speed of about “2r ₀ / t ₀ ” in a moving image is obtained by a visual integration function, This is equivalent to an image obtained by filtering data in a portion moving at a speed of about “r ₀ / t ₀ ” with a low-pass filter of about “4 × t ₀ ” seconds in the time direction.

The pass band of this low-pass filter can be represented by the same region on the frequency domain as the case where a stationary portion is watched, that is, a region R ₁₀₀₁ in FIG.

Here, FIG. 18 shows a region R ₉₀₁ (FIG. 9) in which data of a portion moving at a speed of about “2r ₀ / t ₀ ” is distributed in the frequency domain, and a low-pass filter when a human performs fixed vision. A region R ₁₀₀₁ (FIG. 10) as a pass band is shown.

Humans, of area R _901, information in the region R _1001, that is, can not be a region R ₉₀₁ and R ₁₀₀₁ are only recognized information in the region R ₁₈₀₁ of the portions overlapping. Therefore, information outside the region R ₁₀₀₁ in the region R ₉₀₁ is useless information for humans.

In FIG. 18, a region R ₁₀₀₁ as a pass band of the low-pass filter when a human performs fixed vision is a region R ₁₈₀₂ and R _{1803 of} portions that do not overlap with the region R ₉₀₁ where moving image data exists (in the drawing) In this case, there is no moving image data in the regions R ₁₈₀₂ and R ₁₈₀₃ .

In addition, as described above, when a human is fixedly viewing a portion that is moving at a speed of about “2r ₀ / t ₀ ”, the direction of the human line of sight is the movement of the moving subject (moving subject). Since it moves differently, even a slightly blurred image is recognized as a good moving image.

Accordingly, the data in the region R ₁₈₀₁ in FIG. 18 is filtered by a low-pass filter when the point of interest moves at a speed of about “2r ₀ / t ₀ ”, that is, a low-pass filter having the region R _{1301 in} FIG. Even so, it is recognized as a moving image with good image quality.

From the above, as shown in FIG. 19, if there is data in a region R ₁₉₀₁ (shown in black in the drawing) that overlaps the region R ₁₃₀₁ in the region R ₁₈₀₁ , the speed “2r ₀ / t A portion moving at about “ ₀ ” is recognized as a moving image with good image quality.

Therefore, when a human views a moving part of a moving image at a speed of about “2r ₀ / t ₀ ”, the fixed view is performed from the data of that part, that is, the data in the region R ₉₀₁ . Region R ₁₀₀₁ as the pass band of the low-pass filter when the gazing point moves and the region R ₁₃₀₁ as the pass band of the low-pass filter when the gazing point moves at a speed of about “2r ₀ / t ₀ ” The data in the area excluding ₁₉₀₁ is useless.

  The ranges (regions) on the frequency domain that can be recognized by humans in the case of fixed vision and follow-up vision are summarized as shown in FIGS.

  That is, FIG. 20 shows an area where there is data that can be recognized by humans in a portion where a stationary object (subject) is projected in a moving image.

As shown in FIG. 20, the region on the frequency domain of the data that can be recognized by humans for a portion where a stationary object is projected in the moving image is a region R ₁₀₀₁ shown in FIG. The data outside the area R ₁₀₀₁ is useless.

Next, FIG. 21 shows data in which a person can recognize a portion of a moving image where a subject projected on the moving image moves at a speed of “(r ₀ / t ₀ ) / 2”. The area to be shown is shown.

A region on the frequency domain of data that can be recognized by humans by following vision for a portion moving at a speed of “(r ₀ / t ₀ ) / 2” in the moving image is a region R shown in FIG. ₁₁₀₁ . Furthermore, the region on the frequency domain of data that can be recognized by fixed vision is a region R ₁₅₀₁ shown in FIG. 15, and this region R ₁₅₀₁ is included in the region R ₁₁₀₁ shown in FIG.

Therefore, the region on the frequency domain of the data that can be recognized by humans for the portion moving at the speed “(r ₀ / t ₀ ) / 2” is shown in FIG. 11 as shown in FIG. a region R _1101, data in the region other than the region R ₁₁₀₁ is waste.

Next, FIG. 22 shows an area where data that can be recognized by humans exists in a portion of the moving image where the subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ”. ing.

A region on the frequency domain of data that can be recognized by a human by tracking vision for a portion moving at a speed of “r ₀ / t ₀ ” in the moving image is a region R ₁₂₀₁ shown in FIG. Furthermore, the region on the frequency domain of data that can be recognized by fixed vision is a region R ₁₇₀₁ shown in FIG. 17, and this region R ₁₇₀₁ is included in the region R ₁₂₀₁ shown in FIG.

Therefore, as shown in FIG. 22, the region on the frequency domain of the data that can be recognized by humans for the portion moving at the speed “r ₀ / t ₀ ” is a region R ₁₂₀₁ shown in FIG. _Yes , the data outside the area R ₁₂₀₁ is useless.

Next, FIG. 23 shows an area where data that can be recognized by humans exists in a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”. ing.

A region on the frequency domain of the data that can be recognized by humans by following vision for a portion moving at a speed of “2r ₀ / t ₀ ” in the moving image is a region R ₁₃₀₁ shown in FIG. Furthermore, the region on the frequency domain of data that can be recognized with fixed vision is a region R ₁₉₀₁ shown in FIG. 19, and this region R ₁₉₀₁ is included in the region R ₁₃₀₁ shown in FIG.

Therefore, as shown in FIG. 23, the region on the frequency domain of the data that can be recognized by humans for the portion moving at the speed “2r ₀ / t ₀ ” is a region R ₁₃₀₁ shown in FIG. _Yes , data outside the area R ₁₃₀₁ is useless.

In the above case, the width in the T direction of the region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 is 2π / (4t ₀ ). However, this width may be a width corresponding to the integration time by the visual integration function, and is not limited to 2π / (4t ₀ ).

  The same can be said for the data of the moving part of the video at other speeds.

  As described above, by utilizing the human visual effect (visual characteristics), that is, the visual integration function, on the frequency domain, moving image data can be recognized as data in a region that can be recognized by humans and other waste. Data.

  Then, if only data in a region that can be recognized by humans is extracted from video data on the frequency domain, that is, if unnecessary data other than that is deleted, image quality is not degraded ( The image quality recognized when viewing the video), and the video data can be compressed.

  FIG. 24 shows an example of the configuration of an image processing system that compresses moving image data using a visual integration function and decodes (decompresses) the compressed moving image data.

In FIG. 24, the transmission apparatus 1 is supplied with moving image data having a high frame rate such as 1 / t ₀ = 240 fps. The transmission apparatus 1 filters the moving image data supplied thereto with a pass band in a region R _{1001 in} FIG. 20, a region R ₁₁₀₁ in FIG. 21, a region R ₁₂₀₁ in FIG. 22, a region R _{1301 in} FIG. Band-pass filter or band-limiting filter), and outputs the data in the frequency domain that can be recognized by humans. Data output from the transmission device 1 is recorded on a recording medium 11 such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. Alternatively, it is transmitted via a transmission medium 12 such as a telephone line, a terrestrial wave, a satellite line, the Internet, a wired line, or a wireless LAN (Local Area Network).

  The receiver 2 is supplied with data reproduced from the recording medium 11 or data transmitted via the transmission medium 12. The receiving device 2 receives the data supplied thereto, performs predetermined processing, and the moving image data obtained as a result is composed of, for example, a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), or the like. It is supplied to the display device 3 and displayed.

  In FIG. 24, the transmission device 1, the reception device 2, and the display device 3 can be configured as physically independent devices. Further, the transmitter 1 and the receiver 2 can be physically configured as one device as a whole as shown by being surrounded by a dotted line in FIG. Furthermore, the whole of the transmission device 1, the reception device 2, and the display device 3, or the reception device 2 and the display device 3 can be physically configured as one device.

  Next, FIG. 25 illustrates a first configuration example of the transmission device 1 of FIG.

  The buffer unit 21 is supplied with the high frame rate moving image data supplied to the transmission apparatus 1. The buffer unit 21 sequentially stores the moving image data supplied thereto.

  The filter unit 22 appropriately reads out the moving image data stored in the buffer unit 21, filters the moving image data according to the filter information supplied from the filter generation unit 23, which will be described later, and obtains the moving image data obtained by the filtering, This is supplied to the encoding unit 24.

  The filter generation unit 23 appropriately reads the moving image data stored in the buffer unit 21, generates filter information that is filter information for filtering each part of the moving image data, and supplies the filter information to the filter unit 22.

  That is, the filter generation unit 23 includes a principal component direction acquisition unit 31 and a filter information supply unit 32.

  The principal component direction acquisition unit 31 acquires the principal component direction which is the direction of the principal component on the frequency domain for each part of the moving image data read from the buffer unit 21, and filters the information on the principal component direction. The information is supplied to the information supply unit 32.

The filter information supply unit 32 is a region extending in the principal component direction in the frequency domain according to the information on the principal component direction from the principal component direction acquisition unit 31, and has a specific width in the direction of the frequency axis T in the time direction. For example, a region having 2π / (4t ₀ ), that is, a region R _{1001 in} FIG. 20, a region R ₁₁₀₁ in FIG. 21, a region R ₁₂₀₁ in FIG. 22, and a region R _{1301 in} FIG. The pass band is determined, and the pass band is output to the filter unit 22 as filter information.

  The encoding unit 24 encodes the moving image data supplied from the filter unit 22 by a predetermined encoding (encoding) method such as MPEG (Moving Picture Experts Group) 1 or 2, and outputs the encoded data obtained as a result. To do. This encoded data is recorded on the recording medium 11 of FIG. 24 or transmitted via the transmission medium 12.

  Next, processing of the transmission apparatus 1 in FIG. 25 will be described with reference to the flowchart in FIG.

The buffer unit 21 is supplied with moving image data having a high frame rate (first frame rate) of 240 fps, for example, at a frame rate 1 / t ₀ and sequentially stored.

  In step S1, the filter generation unit 23 reads the moving image data stored in the buffer unit 21, inputs the moving image data to itself (filter generation unit 23), and proceeds to step S2.

In step S2, a region in which the filter generation unit 23 (its principal component direction acquisition unit 31 and filter information supply unit 32) can recognize each part of the moving image data input in step S1 on the frequency domain. That is, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 are obtained as filter passbands, and the passbands are used as filter information. The “processing for obtaining a pass band of necessary information” described later with reference to FIG.

  In step S3, the filter unit 22 reads data used for filtering by the filter of the filter information supplied from the filter generation unit 23 in step S2 from the buffer unit 21, and the filter generation unit 23 in step S2 for the data. Downsampling is performed by thinning out the number of samples in the time direction to 1/4 while applying the filter of the passband represented by the filter information supplied from.

In other words, the filter unit 22 filters the data read from the buffer unit 21 with a filter in the passband represented by the filter information supplied from the filter generation unit 23 in step S2, while sampling in the time direction, that is, the frame period. However, moving image data having a low frame rate of 4t ₀ , which is four times the original moving image data, is output to the encoding unit 24, and the process proceeds to step S4.

  In step S4, the encoding unit 24 encodes the moving image data from the filter unit 22, and outputs the resulting encoding.

  Note that the processing in steps S1 to S4 is performed for all the data of each part of the moving image stored in the buffer unit 21.

  Further, the downsampling in step S3 is a process of outputting a filtering result of one frame for every four frames of moving image data stored in the buffer unit 21. Accordingly, for every four frames of the moving image data stored in the buffer unit 21, one frame out of the four frames may be filtered by a passband filter represented by the filter information supplied from the filter generation unit 23. .

As described above, regions that can be recognized by humans on the frequency domain, that is, for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. 22, region R _{1301 in} FIG. Since the moving image data is filtered by a filter having a pass band and further down-sampling is performed, the moving image data can be reduced in consideration of the visual effect of the human eye.

That is, as shown in FIG. 2, the regions on the frequency domain that humans can visually recognize are 2 × 2π / (2r ₀ ) in the X and Y directions and 2 × 2π / (2t in the T direction. ₀ ) of the region R ₂₀₁ centered on the origin, the highest quality video that can be recognized by humans is the video with a frame rate of 1 / t ₀ and a pixel pitch r ₀ in the spatial direction. It is.

If the visual effects described with reference to FIGS. 10 to 23 are not considered, for example, if the frame rate is reduced below 1 / t ₀ by down-sampling the moving image in the time direction, the frame rate is reduced. When a reduced moving image is displayed, a person who views the moving image recognizes the deterioration of the image quality.

On the other hand, in the transmission apparatus 1, the processing in consideration of the visual effect described with reference to FIGS. 10 to 23, that is, an area that can be recognized by a human on the frequency domain, for example, an area R _{1001 in} FIG. In addition, since the moving image data is filtered by a filter having the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22 and the region R _{1301 in} FIG. 23 as a pass band, the frame rate of the moving image is reduced to 1 / (4t ₀ ). Even if it is reduced, that is, even if the data amount of the moving image data is reduced to ¼, it is possible to display a moving image in which humans do not feel deterioration in image quality.

That is, FIG. 27 to FIG. 30 show a high frame rate by a filter in which the region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. videos while filtering data (frame rate video 1 / t _0), low frame rate video obtained by performing 1/4 down-sampling in the time direction (frame rate 1 / (4t ₀₎ Video) distribution on the frequency domain.

First, the data of the stationary part in the moving image is filtered by the filter unit 22 with a filter having a region R _{1001 in} FIG. 20 as a pass band. Therefore, the moving image data after filtering exists only in the region R ₁₀₀₁ .

When the moving image data in the region R ₁₀₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 in} the spatial direction x, y and 4 × in the time direction t. t Data sampled at ₀ intervals.

For this reason, as shown in FIG. 27, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, since the region R ₁₀₀₁ is a region in which the X and Y directions are 2π / r ₀ and the T direction is 2π / (4t ₀ ) as described with reference to FIG. 10, the aliasing components do not overlap.

  In FIG. 27, the shaded part is the part where the down-sampled video data exists.

Next, the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2” is stored in the region R _{1101 of} FIG. Filtered by a filter as a pass band. Therefore, the moving image data after filtering exists only in the region R ₁₁₀₁ .

When the moving image data in the region R ₁₁₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 in} the spatial direction x, y and 4 × in the time direction t. t Data sampled at ₀ intervals.

Therefore, as shown in FIG. 28, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, the region R ₁₁₀₁ has a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, −π / (2t0)), as described in FIG. And the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ) and the T direction is-(π / t ₀ ) to + (π / t ₀ ). Since it is a range area, the aliasing components do not overlap.

  In FIG. 28, the shaded part is the part where the down-sampled moving image data exists.

Next, the data of the portion of the moving image where the subject projected on the moving image is moving at a speed of about “r ₀ / t ₀ ” has a region R _{1201 in} FIG. Filtered by the filter. Therefore, the moving image data after filtering exists only in the region R ₁₂₀₁ .

When the moving image data in the region R ₁₂₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 with} respect to the spatial directions x and y, and 4 × with respect to the time direction t. t Data sampled at ₀ intervals.

Therefore, as shown in FIG. 29, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, the region R ₁₂₀₁ has a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, -2π / (2t0)), as described in FIG. And the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ) and the T direction is-(π / t ₀ ) to + (π / t ₀ ). Since it is a range area, the aliasing components do not overlap.

  In FIG. 29, the shaded portion is the portion where the down-sampled moving image data exists.

Next, the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ” has a region R _{1301 in} FIG. Filtered by the filter. Therefore, the moving image data after filtering exists only in the region R ₁₃₀₁ .

When the moving image data in the region R ₁₃₀₁ is downsampled to 1/4 in the time direction, the moving image data is sampled at intervals of r _{0 in} the spatial direction x, y and 4 × in the time direction t. t Data sampled at ₀ intervals.

Therefore, as shown in FIG. 30, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, the region R ₁₃₀₁ has a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, −2π / t0), as described in FIG. area, and, X, Y direction, - at (π / r ₀₎ to _{+ (π / r 0),} T direction, - the range of (π / t ₀₎ to + (π / t ₀₎ Since it is a region, the aliasing components do not overlap.

  In FIG. 30, the shaded part is the part where the down-sampled moving image data exists.

As shown in FIGS. 27 to 30 above, the fact that the aliasing components of the downsampled video data do not overlap in the frequency domain means that the original data, that is, 20 area R ₁₀₀₁ , area R ₁₁₀₁ in FIG. 21, area R ₁₂₀₁ in FIG. 22, and area R ₁₃₀₁ in FIG. 23 can be extracted. This means that necessary information (information that can be recognized by humans) taking into consideration human visual characteristics is accurately held.

  Note that in order to decode (restore) the moving image data after downsampling into moving image data before downsampling, information on what kind of filter is applied to the data of each part of the moving image is required.

Therefore, in the transmission device 1 in FIG. 25, the filter information supply unit 32 transmits to the encoding unit 24 the passband as the filter information of the filter applied to the data of each part of the moving image, for example, the region R in FIG. ₁₀₀₁ , information on the area R ₁₁₀₁ in FIG. 21, area R ₁₂₀₁ in FIG. 22, area R _{1301 in} FIG. 23, and information on the principal component directions of the areas R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ , In the encoding unit 24, the information can be multiplexed with the encoded data and output.

  Next, with reference to the flowchart of FIG. 31, the “process for obtaining the passband of necessary information” performed by the filter generation unit 23 in step S2 of FIG. 26 will be described.

  First, in step S11, the principal component direction acquisition unit 31 of the filter generation unit 23 (FIG. 25) performs the frequency axis X in the spatial directions x and y for each part of the moving image data input in step S1 of FIG. , Y, and the principal component direction in the frequency domain defined by the frequency axis T in the time direction t are acquired, supplied to the filter information supply unit 32, and the process proceeds to step S12. The principal component direction can be obtained, for example, by performing Fourier transform (three-dimensional Fourier transform) on the three-dimensional direction of the spatial direction x, y and the time direction t.

In step S12, the filter information supply unit 32 is a region extending in the principal component direction from the principal component direction acquisition unit 31 from the origin (0,0) in the frequency domain, and 2π / (4 × t in the T direction. ₀ ), the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ), and the T direction is-(π / t ₀ ) to + (π / t ₀ ). a range of area, i.e., and a region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R ₁₃₀₁ in FIG. 23, determined as the pass band of the filter, the flow proceeds to step S13.

  In step S13, the filter information supply unit 32 supplies the filter pass band (information representing) obtained in step S12 to the filter unit 22 as filter information, and performs a “process for obtaining the pass band of necessary information”. finish.

According to the “process for obtaining the pass band of necessary information” as described above, for example, the region R _{1001 in} FIG. 20 is used as filter information for a portion where a stationary subject is projected in a moving image. Can be sought. Further, for a portion where the subject projected in the moving image moves at a speed of “(r ₀ / t ₀ ) / 2”, the region R _{1101 in} FIG. 21 can be obtained as filter information. Furthermore, the region R _{1201 in} FIG. 22 can be obtained as filter information for a portion where the subject projected in the moving image moves at a speed of about “r ₀ / t ₀ ”. Further, for a portion where the subject projected in the moving image moves at a speed of about “2r ₀ / t ₀ ”, the region R _{1301 in} FIG. 23 can be obtained as filter information. Even in a portion of the moving image where the subject is moving at other speeds, a region in the frequency domain that can be recognized by humans can be obtained as filter information.

  In the present embodiment, in the filter information supply unit 32 of the filter generation unit 23 (FIG. 25), the region on the frequency domain extending in the main component direction supplied from the main component direction acquisition unit 31 is defined as the passband of the filter. Although the pass band is supplied to the filter unit 22 as filter information, the filter information supply unit 32 obtains filter information as the pass band as described above for each of a plurality of principal component directions in advance. It may be stored and the filter information corresponding to the principal component direction supplied from the principal component direction acquisition unit 31 may be selected from the stored filter information and supplied to the filter unit 22.

  Further, the filter generation unit 23 can be configured as an independent device that outputs filter information different from the transmission device 1.

  By the way, in step S11 of FIG. 31, the principal component direction of each part of the moving image data is obtained by the three-dimensional Fourier transform. However, since the calculation amount of the three-dimensional Fourier transform is enormous, the principal component in step S11 The amount of calculation required for the process of acquiring the direction is enormous.

  A method for acquiring the principal component direction with a smaller amount of calculation than in the case of performing the three-dimensional Fourier transform in step S11 of FIG. 31 will be described with reference to the flowchart of FIG.

  First, in step S21, the principal component direction acquisition unit 31 of the filter generation unit 23 (FIG. 25) stores the data of each frame of the moving image data stored in the buffer unit 21, for example, horizontal × vertical is 16 × 16. Dividing into blocks such as pixels, the process proceeds to step S22. Here, the image data of each block is the data of each part of the above-described moving image data.

  Here, the block does not need to be composed of a plurality of pixels, and may be one pixel.

  In step S22, the principal component direction acquisition unit 31 sequentially sets each block obtained in step S21 as a target block, and for the target block, a frame subsequent to the frame of the target block (hereinafter, referred to as a target frame as appropriate). Correlation information representing the correlation with is obtained. Further, in step S22, the principal component direction acquisition unit 31 obtains the position in the spatial direction on the frame next to the frame of interest where the correlation represented by the correlation information of the block of interest is the highest. That is, in step S22, the principal component direction acquisition unit 31 detects a two-dimensional motion vector in the spatial direction x, y of the block of interest by so-called block matching or the like.

  Here, as the correlation information, a so-called correlation coefficient can be adopted. Here, considering the calculation cost, for example, the target block is placed in the spatial direction x, y within the search range with the motion vector. , The square error of the pixel value of each pixel of the target block at the position shifted by u, v and the pixel of the next frame of the target frame at the same position as the pixel, the sum of absolute differences, etc. adopt. In this case, the position u, v in the spatial direction where the “value” of the correlation information is the minimum is the position in the spatial direction on the next frame of the frame of interest where the correlation represented by the correlation information is the highest.

Note that the spatial direction positions u and v at which the correlation represented by the correlation information is maximum (correlation information value is minimum) are represented by u ₀ and v ₀ , respectively, in the spatial directions x and y detected in step S22. The motion vector is represented as (u ₀ , v ₀ ).
The

Thereafter, the process proceeds from step S22 to S23, and the principal component direction acquisition unit 31 adds the frame period t ₀ of the original moving image data to the motion vector (u ₀ , v ₀ ) detected in step S22 and the component in the time direction t. The direction orthogonal to the direction of the three-dimensional motion vector (u ₀ , v ₀ , t ₀ ) added as is detected as the principal component direction, supplied to the filter information supply unit 32, and the process is terminated.

According to the above processing, the principal component direction of the block is detected as a plane (expansion direction) perpendicular to the motion vector (u ₀ , v ₀ , t ₀ ).

That is, when a motion vector (u ₀ , v ₀ , t ₀ ) is detected for the block of interest, ideally, in a three-dimensional space defined by the spatial direction x, y and the temporal direction t, the motion vector ( u ₀ , v ₀ , t ₀ ) in the direction of the pixel value of the target block, that is, the value of the video data at the point (x, y, t) where the target block is This is the same as the value of the moving image data at the point (x + mu ₀ , y + mv ₀ , t + mt ₀ ) on the three-dimensional space represented by x, y, t.

Therefore, the frequency component of the data of the spatio-temporal block of interest, that is, the data obtained by converting the spatio-temporal block of interest data into the data on the frequency domain is the motion vector (u ₀ , v ₀ , t ₀ ). It exists only on a vertical plane.

In the method according to the flowchart of FIG. 32, this is used to detect a plane perpendicular to the motion vector (u ₀ , v ₀ , t ₀ ) as the principal component direction. Therefore, in this case, in order to acquire the principal component direction, it is only necessary to detect a motion vector, and it is not necessary to perform a three-dimensional Fourier transform. Therefore, the principal component direction can be detected with a small amount of calculation.

  As the correlation information, for example, a correlation coefficient can be adopted. In this case, the correlation information value (correlation coefficient) is the maximum when the correlation represented by the correlation information is the maximum.

  Further, the encoding unit 24 of the transmission device 1 (FIG. 25) can output the moving image data from the filter unit 22 as it is without encoding.

  Next, FIG. 33 illustrates a configuration example of the receiving device 2 of FIG.

The encoded data output from the transmission apparatus 1 in FIG. 25 is encoded low-frame-rate video data with a frame rate of 1 / (4t ₀ ). When the image is supplied and displayed on a display device having the same frame rate as the rate, in addition to the frequency component that should be displayed, a part of the aliasing component shown in FIGS. 28 to 30 is also displayed. As a result, the image quality of the moving image displayed on the display device deteriorates.

Therefore, the receiving apparatus 2 upsamples the low frame rate moving image data with the frame rate of 1 / (4t ₀ ) to the original frame rate 1 / t ₀ , and the resulting frame rate is 1 / t _0. The high-frame-rate moving image data is supplied to the display device 3 (FIG. 24), so that the display device 3 displays the high-quality moving image data (the deterioration of the image quality is not recognized). Here, it is assumed that the display device 3 is configured to display a moving image at a frame rate of 1 / t ₀ .

In the receiving device 2, as described above, in order to display a high-quality moving image on the display device 3, a low frame rate (second frame) supplied from the transmitting device 1 and having a frame rate of 1 / (4t ₀ ). (Rate) up-sampling in the time direction is performed while applying a filter that passes a frequency component of a predetermined pass band.

  That is, the decoding unit 50 is supplied with encoded data from the transmission device 1 (FIG. 24) via the recording medium 11 or the transmission medium 12. The decoding unit 50 decodes the encoded data, and supplies the low frame rate moving image data obtained as a result to the buffer unit 51.

  The buffer unit 51 sequentially stores the low frame rate moving image data supplied from the decoding unit 50.

  The filter unit 52 appropriately reads out the moving image data stored in the buffer unit 51, up-samples the moving image data while filtering according to the filter information supplied from the filter generation unit 53 described later, and performs the filtering and up-sampling. The resulting high frame rate video data is output. The high frame rate moving image data output from the filter unit 52 is supplied to the display device 3 (FIG. 24) and displayed.

  The filter generation unit 53 appropriately reads out the moving image data stored in the buffer unit 51, generates filter information that is filter information to be filtered using each portion of the moving image data, and supplies the filter information to the filter unit 52.

  That is, the filter generation unit 53 includes a main component direction acquisition unit 61 and a filter information supply unit 62.

  The principal component direction acquisition unit 61 acquires, for each part of the moving image data read from the buffer unit 51, a principal component direction that is the direction of the principal component on the frequency domain, and filters the information on the principal component direction. The information is supplied to the information supply unit 62.

The filter information supply unit 62 is an area extending in the principal component direction in the frequency domain according to the information of the principal component direction from the principal component direction acquisition unit 61, and has a specific width in the direction of the frequency axis T in the time direction. For example, a region having 2π / (4t ₀ ), that is, a region R _{1001 in} FIG. 20, a region R ₁₁₀₁ in FIG. 21, a region R ₁₂₀₁ in FIG. 22, and a region R _{1301 in} FIG. The pass band is determined, and the pass band is output to the filter unit 52 as filter information.

  Next, processing of the receiving device 2 in FIG. 33 will be described with reference to the flowchart in FIG.

The decoding unit 50 is supplied with encoded data. The decoding unit 50 decodes the encoded data into moving image data with a frame rate of 1 / (4t ₀ ) and a low frame rate, and supplies it to the buffer unit 51. The buffer unit 51 sequentially stores moving image data having a low frame rate of 1 / (4t ₀ ) supplied from the decoding unit 50.

  In step S31, the filter generation unit 53 generates a certain part of the moving image data stored in the buffer unit 51, that is, for example, a block of 16 × 16 pixels as described in step S21 of FIG. Data is read out and input to itself (filter generation unit 53), and the process proceeds to steps S32 to S34 in sequence.

  In steps S32 to S34, the filter generation unit 53 (the principal component direction acquisition unit 61 and the filter information supply unit 62) can recognize a region that can be recognized by humans on the frequency domain for the data input in step S31. A process for obtaining the pass band of the filter and supplying the pass band as filter information to the filter unit 52, that is, a process similar to the above-described "process for obtaining the pass band of necessary information" in FIG. 31 is performed.

  That is, in step S32, the principal component direction acquisition unit 61 of the filter generation unit 53 uses the frequency axes X and Y in the spatial directions x and y and the frequency axis T in the time direction t for the moving image data input in step S31. The principal component direction in the defined frequency domain is acquired and supplied to the filter information supply unit 62, and the process proceeds to step S33.

  In step S32, the principal component direction can be obtained by performing a three-dimensional Fourier transform as described in FIG. 31, or can be obtained by detecting a motion vector as described in FIG. it can. Further, as described above, when the principal component direction (information thereof) is multiplexed in the encoded data from the transmission device 1, the decoding unit 50 separates the principal component direction from the encoded data, In the direction acquisition unit 61, the principal component direction can be acquired by receiving the supply of the principal component direction (information thereof) from the decoding unit 50.

Furthermore, encoded data in the spatial direction x, in the case that contains the y motion vector (u _0, v ₀₎ is the motion vector (u _0, v _0), the frame synchronization 4t of the moving image data included in the encoded data ₀ is added as a component in the time direction t to obtain a three-dimensional motion vector (u ₀ , v ₀ , 4t ₀ ) in the spatial direction x, y and the time direction t, and as shown in FIG. The plane orthogonal to the motion vector (u ₀ , v ₀ , 4t ₀ ) can be the principal component direction.

  That is, in the encoding unit 24 of the transmission apparatus 1 of FIG. 25, when moving image data is encoded by an encoding method such as MPEG that uses at least motion compensation, the encoded data is used for motion compensation. Contains motion vectors. Therefore, the decoding unit 50 extracts the motion vector (information thereof) from the encoded data, and supplies the motion vector included in the encoded data to the principal component direction acquisition unit 61 via the buffer unit 51, thereby the principal component direction. The acquisition unit 61 can obtain (detect) a plane orthogonal to the motion vector as the principal component direction.

  In MPEG, a motion vector does not exist in an I picture block (macroblock), but an I picture motion vector may be estimated from a motion vector of a P or B picture block, for example.

In step S33, the filter information supply unit 62 is a region extending in the principal component direction from the principal component direction acquisition unit 61 from the origin (0, 0) in the frequency domain, and is 2π / (4 × t in the T direction. ₀ ), the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ), and the T direction is-(π / t ₀ ) to + (π / t ₀ ). a range of area, i.e., and a region R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R ₁₃₀₁ in FIG. 23, determined as the pass band of the filter, the flow proceeds to step S34.

  In step S34, the filter information supply unit 62 supplies the filter passband (information representing) obtained in step S33 to the filter unit 52 as filter information, and the process proceeds to step S35.

  In step S35, the filter unit 52 reads the moving image data from the buffer unit 51, and applies the filter of the passband represented by the filter information supplied from the filter generation unit 53 in step S34 to the moving image data. Upsampling is performed to quadruple the number of samples in the direction.

  That is, for example, assuming that the moving image data input in step S31 is a block (data) of a certain frame of moving image data at a low frame rate, that block is the attention block, and the frame of the attention block is noted. Frame.

The filter unit 52 converts the moving image data read from the buffer unit 51 into a passband filter represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region R _{1101 in} FIG. The data (pixel value) of each pixel of the block of interest is obtained by filtering with a filter that uses the region R ₁₂₀₁ in FIG. 22 and the region R _{1301 in} FIG. 23 as a pass band.

Further, the filter unit 52 filters the moving image data read from the buffer unit 51 in the passband represented by the filter information supplied from the filter generation unit 53, that is, for example, the region _{R1001 in} FIG. 20 or the region in FIG. By filtering with R ₁₁₀₁ , a region R ₁₂₀₁ in FIG. 22 and a filter having a region R _{1301 in} FIG. 23 as a pass band, the time 4 t ₀ from the frame of interest to the next frame is divided into three equal to the time t ₀ . Data of pixels on the straight line extending in the direction of the three-dimensional motion vector of the block of interest through each pixel of the block of interest is obtained at each time. Note that the three-dimensional motion vector of the block of interest can be obtained by, for example, the principal component direction acquisition unit 61 and supplied to the filter unit 52 together with the filter information via the filter information supply unit 62.

The filter unit 52 performs the above filtering and upsampling to obtain high frame rate moving image data with a frame rate of 1 / t ₀ , and proceeds from step S35 to step S36.

  In step S36, the filter unit 52 outputs the high frame rate moving image data obtained in step S35 to the display device 3 (FIG. 24), and ends the process.

  Note that the processing in steps S31 to S34 is performed for all portions of the moving image data stored in the buffer unit 51.

As described above, regions that can be recognized by humans on the frequency domain, that is, for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. 22, region R _{1301 in} FIG. Since the video data is filtered by a filter that uses a passband, and further upsampling is performed, the amount of data is reduced in consideration of the visual effects of the human eye. , Obtain high-quality video data, that is, video data with the same image quality as the original high-frame-rate video data (restore video data that can recognize the same image quality as the original high-frame-rate video data Can be displayed).

That is, FIGS. 35 to 38, respectively, the regions R ₁₀₀₁ in FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R ₁₃₀₁ in FIG. 23, the filter having a pass band to 27 A high frame rate video (frame rate is obtained by performing upsampling four times in the time direction while filtering the data of the low frame rate video (video with a frame rate of 1 / (4t ₀ )) in FIG. 1 / t ₀ (moving data) on the frequency domain.

First, the data of the still portion in the moving image is up-sampled in the filter unit 52 while performing filtering by a filter having the region R _{1001 in} FIG. 20 as a pass band. Accordingly, the moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₀₀₁ in FIG. 20, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 35, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 35, the shaded portion is a portion where moving image data exists.

Next, with respect to data of a portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2”, the filter unit 52 uses the area shown in FIG. Upsampling is performed while filtering with a filter having R ₁₁₀₁ as a pass band. Accordingly, the moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₁₀₁ in FIG. 21, and is sampled at intervals of r _{0 with} respect to the spatial direction x, y, and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 36, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 36, the shaded portion is a portion where moving image data exists.

Next, for the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ”, the filter unit 52 passes the region R _{1201 in} FIG. Upsampling is performed while filtering with a band filter. Therefore, the moving image data obtained by the filtering and up-sampling has frequency components in the region R ₁₂₀₁ in FIG. 22, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y, and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 37, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 37, the shaded portion is a portion where moving image data exists.

Next, for the data of the portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, the filter unit 52 passes the region R _{1301 in} FIG. Upsampling is performed while filtering with a band filter. Accordingly, the moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₃₀₁ in FIG. 23, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y, and in the time direction t. On the other hand, the data is sampled at t ₀ intervals.

Therefore, as shown in FIG. 38, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction. . That is, the period in the T direction in which the aliasing component occurs is four times that shown in FIG.

  In FIG. 38, the shaded portion is a portion where moving image data exists.

As described in FIG. 2, humans can recognize that the X and Y directions are − (π / r ₀ ) to + (π / r ₀ ) and the T direction is − (π / t ₀ ) to + (π / t ₀ ) range R ₂₀₁ , and in FIG. 35 to FIG. 38, there is no aliasing component in this region R ₂₀₁ .

Therefore, when moving image data with a high frame rate of 1 / t ₀ obtained by filtering and upsampling in the filter unit 52 is displayed on the display device 3, The same image quality as that of the moving image (the moving image supplied to the transmission device 1) can be recognized.

  As described above, the encoding unit 24 of the transmission device 1 in FIG. 25 can multiplex the filter information of the filter used in the filter unit 22 into the encoded data, but the filter information is multiplexed into the encoded data. If it is configured, the receiving device 2 (FIG. 33) can be configured without the filter generation unit 53. In this case, the receiving device 2 may be configured such that the decoding unit 50 separates the filter information from the encoded data and supplies it to the filter unit 52.

In the above case, 1 / (4t ₀ ) low frame rate moving image data obtained by processing the high frame rate moving image data of 1 / t ₀ in the transmitting device 1 of FIG. And the receiving device 2 converts the low-frame-rate moving image data into high-frame-rate moving image data of 1 / t ₀ , so that the display device 3 has high image quality (input to the transmitting device 1). Although a moving image having the same image quality as the moving image is displayed, even if moving image data with a low frame rate of 1 / (4t ₀ ) is originally input to the receiving device 2, the display device 3 has a certain amount of high quality. It is possible to display a moving image of image quality.

That is, in the moving image, in a portion where a (substantially) stationary subject is projected, there is almost no frequency component in the time direction, and thus the region obtained by removing the region R ₁₀₀₁ from the region R ₃₀₁ shown in FIG. There are almost no frequency components in.

In the moving image, in the part where the subject projected there is moving at a speed of “(r ₀ / t ₀ ) / 2”, the direction of the movement (spatial direction x, y and time direction t three-dimensional Since there is almost no frequency component other than the frequency component in the direction perpendicular to the direction of the movement of (1), there is almost no frequency component in the region excluding the region R ₁₁₀₁ from the region R ₄₀₁ shown in FIG.

Further, in the moving image, there is almost no frequency component other than the frequency component in the direction perpendicular to the direction of the movement in the portion where the object projected there is moving at a speed of about “r ₀ / t ₀ ”. There are almost no frequency components in the region excluding the region R ₁₂₀₁ from the region R ₈₀₁ shown in FIG.

Further, in the moving image, in the portion where the subject projected there is moving at a speed of about “2r ₀ / t ₀ ”, there is almost no frequency component in the direction perpendicular to the direction of the movement. There are almost no frequency components in the region excluding the region R ₁₃₀₁ from the region R ₉₀₁ shown in FIG.

However, in actuality, in the moving image, the projected image of the subject changes in a complicated manner with time, so even a portion that is stationary in the moving image is a region from the region R ₃₀₁ shown in FIG. There is some data (frequency components) in the area excluding _R1001 .

Similarly, in the moving part of the moving image at a speed of “(r ₀ / t ₀ ) / 2”, there is some data in the area excluding the area R ₁₁₀₁ from the area R ₄₀₁ shown in FIG. Further, in the moving part of the moving image at the speed “r ₀ / t ₀ ”, there is some data in the area excluding the area R ₁₂₀₁ from the area R ₈₀₁ shown in FIG. Furthermore, in the moving part of the moving image at a speed of about “2r ₀ / t ₀ ”, there is some data in the area excluding the area R ₁₃₀₁ from the area R ₉₀₁ shown in FIG.

In order to delete data existing outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ , the filter unit 22 of the transmission device 1 (FIG. 25) uses the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , R Filtering is performed by a filter having a pass band of ₁₃₀₁ .

Therefore, when the low frame rate moving image data input to the receiving device 2 is not obtained as a result of the processing of the transmitting device 1, some data outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , R ₁₃₀₁ There can be data. Then, when data exists outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ , moving image data obtained by filtering and upsampling in the filter unit 52 of the receiving device 2 (FIG. 33) is shown in FIG. The aliasing component is mixed in the region R ₂₀₁ that can be recognized by the human being, and the image quality of the moving image deteriorates due to the aliasing component.

However, the data existing outside the regions R ₁₀₀₁ , R ₁₁₀₁ , R ₁₂₀₁ , and R ₁₃₀₁ is very small, and image quality degradation caused by the aliasing component due to such a small amount of data is also small. Therefore, originally when moving image data with a low frame rate of 1 / (4t ₀ ) is originally input to the receiving device 2, it is compared with when inputting moving image data with a low frame rate obtained by the transmitting device 1. Although there is some degradation in image quality, it is still possible to display movies with improved image quality compared to the case of displaying 1 / (4t ₀ ) low frame rate movie data as it is. it can.

Note that when the 1 / (4t ₀ ) low-frame-rate moving image data input to the receiving device 2 is captured by an imaging device that employs a so-called electronic shutter, the receiving device 2 (FIG. 33). In the filter information supply unit 62, a region in which the width in the T direction is limited corresponding to the exposure time at the time of capturing the moving image data at the low frame rate can be determined as the pass band of the filter.

That is, 1 / the (4t ₀₎ exposure time for the imaging of each frame of the moving image data of a low frame rate, with greater than t _0, when it is t ₀ 'is smaller than 4t ₀ is on each frame The projected image of the subject is equal to a value (light reception amount) obtained by integrating light from the subject for a time t ₀ ′. Therefore, when the subject is moving, blur (motion blur) corresponding to time t ₀ ′ is generated in the image of each frame. When t ₀ is about 1/240 seconds, for example, t ₀ ′ is about 1/120 seconds, for example.

The integration of only time t ₀ ′ is equivalent to performing filtering by a low-pass filter having a region R ₃₉₀₁ having a width of 2π / t ₀ ′ in the frequency domain T direction as shown in FIG. Therefore, it can be said that the data of each frame of the moving image captured with the exposure time t ₀ ′ by the electronic shutter is the data that has passed through the low-pass filter having the region R ₃₉₀₁ as the pass band.

Note that the region R ₃₉₀₁ has a width of 2π / t ₀ ′ in the range of −π / t ₀ ′ to π / t ₀ ′ in the T direction, and a human recognizes the X and Y directions. This is a region having a width of 2π / r _{0 in} the range of −π / r _{0 to} π / r ₀ . As described above, for example, when t ₀ is 1/240 seconds and t ₀ ′ is 1/120 seconds, and therefore 2t ₀ = t ₀ ′, the region R ₃₉₀₁ has the T direction It has a width of π / t _{0 in} the range of −π / (2t ₀ ) to π / (2t ₀ ).

As described above, the data of each frame of the imaged moving image exposure time as t ₀ ', since the data that has passed through the low-pass filter having a pass band region R _3901, in the frequency domain, in addition to regions R ₃₉₀₁ There is no data.

Therefore, for example, even the data in the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R ₁₃₀₁ in FIG. Since the data outside the area R ₃₉₀₁ is not the original data of the moving image such as noise, it is desirable to delete it as unnecessary data.

Therefore, in the filter information supply unit 62 of the receiving device 2 (FIG. 33), the T direction of the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R ₁₃₀₁ in FIG. A region limited by 39 regions R ₃₉₀₁ can be determined as the passband of the filter.

For example, as described above, assuming that 2t ₀ = t ₀ ′, the region R _{1001 in} FIG. 20 is included in the region R ₃₉₀₁ in FIG. For the stationary part, the region R _{1001 in} FIG. 20 can be determined as it is as the pass band of the filter.

In addition, as shown in FIG. 40, the region R _{1101 in} FIG. 21 has X, Y in the T direction in the region R ₁₁₀₁ in the range of −π / (2t ₀ ) to π / (2t ₀ ). direction, in the region R ₄₀₀₁ in the range of - [pi] / r ₀ to [pi / r _0, since overlapping the region R ₃₉₀₁ in FIG. 39, the filter information supply section 62, the subject being projected video speed "( For a portion moving at about “r ₀ / t ₀ ) / 2”, the region R _{4001 in} FIG. 40 can be determined as the passband of the filter.

Furthermore, as shown in FIG. 41, the region R _{1201 in} FIG. 22 has X, Y in the T direction in the region R ₁₂₀₁ in the range of −π / (2t ₀ ) to π / (2t ₀ ). Since the direction R overlaps with the region R _{3901 in} FIG. 39 in the region R ₄₁₀₁ in the range of −π / r _{0 to} π / r ₀ , the filter information supply unit 62 uses the velocity “r For a portion moving at about “ ₀ / t ₀ ”, the region R _{4101 in} FIG. 41 can be determined as the passband of the filter.

In addition, as shown in FIG. 42, the region R _{1301 in} FIG. 23 has X, Y in the T direction in the region R ₁₃₀₁ in the range of −π / (2t ₀ ) to π / (2t ₀ ). direction, in the region R ₄₂₀₁ in the range of - [pi] / r ₀ to [pi / r _0, since overlapping the region R ₃₉₀₁ in FIG. 39, the filter information supply unit 62, the object being projected video speed "2r For the portion moving at about “ ₀ / t ₀ ”, the region R _{4201 in} FIG. 42 can be determined as the passband of the filter.

In the filter information supply unit 62 of the receiving apparatus 2 (FIG. 33), the pass band of the filter is determined as described above, and filtering is performed by the filter unit 52, so that the original moving image such as noise outside the region R ₃₉₀₁ is obtained. Since unnecessary data other than the above data is removed, moving image data with improved S / N (Signal to Noise ratio) and the like can be obtained.

Here, in the receiving apparatus 2, when the pass band of the filter is limited by the region R ₃₉₀₁ in FIG. 39, the exposure time t ₀ ′ at the time of capturing the low frame rate moving image data input to the receiving apparatus 2 is Although necessary, the exposure time t ₀ ′ can be input to the receiving device 2 by multiplexing the moving image data input to the receiving device 2, for example.

  When the data input to the receiving device 2 is not encoded, the decoding unit 50 supplies the data to the buffer unit 51 without decoding the data.

By the way, for example, the principal component direction acquisition unit 31 of the transmission device 1 (FIG. 25) detects the motion vector (u ₀ , v ₀ ) of the block of interest to the next frame of the frame of interest, as described above. From the motion vector (u ₀ , v ₀ ), the principal component direction in the frequency domain of the block of interest can be obtained. That is, in the principal component direction acquisition unit 31, a three-dimensional motion vector (u ₀ , v ₀ , t ₀ ) obtained by adding the component t ₀ in the time direction t to the motion vector (u ₀ , v ₀ ) of the block of interest An orthogonal plane can be obtained as the principal component direction in the frequency domain of the block of interest.

In this case, for example, a vector obtained by multiplying the motion vector (u ₀ , v ₀ ) of the block of interest to the next frame of the frame of interest by, for example, -2, -1, 2 or 3 times, If it matches the motion vector to the previous frame, previous frame, next next frame, next next next frame of the frame of interest, it can be obtained from the motion vector (u ₀ , v ₀ ) The principal component direction of the block of interest is accurate.

However, a vector obtained by multiplying the motion vector (u ₀ , v ₀ ) by, for example, −2 times, −1 times, 2 times, or 3 times is the frame before the attention frame or the previous frame of the attention block. If the motion vector is shifted from the motion vector to the next next frame, the next next frame, the principal component direction of the target block obtained from the motion vector (u ₀ , v ₀ ) is It contains errors and is inaccurate.

  That is, FIG. 43 shows high frame rate moving image data supplied to the transmission apparatus 1.

  In FIG. 43, the moving image data is shown with the horizontal axis as the spatial direction x. In addition, a moving subject is projected on the moving image, and the moving subject moves only in the spatial direction x (does not move in the spatial direction y). In FIG. 43, the moving subject moves from left to right as time passes.

In FIG. 43, in order from the top, time (frames) t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + Image data in each of 4t ₀ is shown.

If the image data at time t is used as a reference, the image data at time t-4t ₀ is only K ₄ , the image data at time t-3t ₀ is only K ₃ , and the time t Image data at -2t ₀ is K ₂ only, image data at time tt ₀ is K ₁ only, image data at time t + t ₀ is H ₁ only, image data at time t + 2t ₀ is H ₂ only, time t The image data at + 3t ₀ is shifted by H _{3 and} the image data at time t + 4t ₀ is shifted by H ₄ in the spatial direction x.

Therefore, when a certain block R ₄₃₀₀ in the image data at time t and the subject block, the same image data and the block of interest R ₄₃₀₀ (image data corresponding to the target block R _4300), the image data at time t-4t ₀ is , from the target block R ₄₃₀₀ present in a region R ₄₃₁₄ of the position shifted by K ₄ in the spatial direction x, the image data at time t-3t _0, from the target block R ₄₃₀₀ of position shifted in the spatial direction x by K ₃ Exists in region R ₄₃₁₃ . Similarly, the same image data as the _target block R ₄₃₀₀ is the image data at time tt _{0 in} the region R ₄₃₁₂ at a position shifted by K _{2 in the} spatial direction x from the _target block R _{4300 in} the image data at time t−2t _0. In the data, the region R ₄₃₁₁ at a position shifted by K _{1 in the} spatial direction x from the _target block R ₄₃₀₀ , and at the time t + t ₀ image data at the position shifted by H _{1 in the} spatial direction x from the _target block R ₄₃₀₀ . In the region R ₄₃₀₁ , in the image data at time t + 2t ₀ , the image data at time t + 3t ₀ is shifted from the _target block R ₄₃₀₀ to R ₄₃₀₂ at a position shifted by H _{2 in the} spatial direction x from the _target block R _4300. In the region R ₄₃₀₃ at a position shifted by H _{3 in the} spatial direction x, the image data at time t + 4t ₀ exists in the region R ₄₃₀₄ at a position shifted by H _{4 in the} spatial direction x from the target block R ₄₃₀₀ , respectively. .

In FIG. 43, the magnitude of the movement of the moving subject is not constant. Therefore, the rate of increase from K ₄ to K ₃ , the rate of increase from K ₃ to K ₂ , the rate of increase from K ₂ to K ₁ , The rate of increase from K ₁ to 0, the rate of increase from 0 to H ₁ , the rate of increase from H ₁ to H ₂ , the rate of increase from H ₂ to H ₃ and the rate of increase from H ₃ to H ₄ are also constant. It is not.

As shown in FIG. 43, when the subject is moving, the motion vector (u ₀ , v ₀ ) of the block of interest R ₄₃₀₀ is obtained as shown in FIG. 44 as a vector (H ₁ , 0). Therefore, the principal component direction acquisition unit 31 of the transmission device 1 (FIG. 25) obtains the principal component direction of the _target block R _{4300 from} the motion vector (H ₁ , 0) of the target block R ₄₃₀₀ .

Then, in the filter information supply unit 32 of the transmission apparatus 1, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R _{1301 in} FIG. Is determined as the pass band of the filter that performs filtering in the filter unit 22.

As the pass band of the filter, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R _{1301 in} FIG. / (4t ₀ ), therefore, filtering by such a passband filter, image data of about 4 frames before and after the frame of the block of interest R ₄₃₀₀ (frame of interest), that is, for example, time t-4t ₀ , T-3t ₀ , t-2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t ₀ range of image data or time t-3t ₀ , T−2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t _{0 and} other operations using image data in a range of 7 frames (performed by the filter unit 22 of the transmission apparatus 1) By calculation), the image data of the _target block R ₄₃₀₀ as the filtering result is obtained.

That is, the block of interest position with R ₄₃₀₀ of pixels (x, y), the pixel value as a filtering result, for example, the position in the image data at time _{t + t 0 (x, y} ) a motion vector (H ₁ from Pixel value near the position shifted by 0), position shifted by the vector (2H ₁ , 0) twice the motion vector (H ₁ , 0) from the position (x, y) in the image data at time t + 2t ₀ Near pixel value, pixel value near the position shifted by the vector (3H ₁ , 0) three times the motion vector (H ₁ , 0) from the position (x, y) in the image data at time t + 3t ₀ A pixel value in the vicinity of a position shifted by a vector (4H ₁ , 0) four times the motion vector (H ₁ , 0) from the position (x, y) in the image data at t + 4t _{0, in} the image data at time tt ₀ position (x, y) -1 times the vector (-H _1, 0) shifted by the pixel value of the vicinity of the position of the motion vector from the (H _1, 0), the position in the image data at time t-2t ₀ (x, -2 times the motion vector (H _1, 0) from y) Vector (-2H _1, 0) only the pixel values of the shifted around position, the position in the image data at time _{t-3t 0 (x, y} ) a motion vector (H _1, 0) from -3 fold vector (-3H _1, 0) -4 times the vector of pixel values around a position shifted by a position in the image data at time _{t-4t 0 (x, y} ) from the motion vector (H _1, 0) (-4H _1, 0) It is obtained by using pixel values in the vicinity of the position shifted by a distance.

Here, the motion vector (H _1, 0), as shown in FIG. 44, from the target block R _4300, the image data at time t + t _0, corresponding to the target block R ₄₃₀₀ and (attention block R ₄₃₀₀ Since this is a vector to the position of the region R ₄₃₀₁ (where the same image data exists), the position on the image data at time t + t ₀ shifted by the motion vector (H ₁ , 0) from the block of interest R ₄₃₀₀ It matches the position of the region R ₄₃₀₁ corresponding to the block R ₄₃₀₀ .

However, since the vector (2H ₁ , 0) is a vector obtained by doubling the motion vector (H ₁ , 0) as shown in FIG. 45, the vector (2H ₁ , 0) is shifted from the _target block R ₄₃₀₀ by the vector (2H ₁ , 0). The position on the image data at time t + 2t ₀ does not necessarily match the position of the region R ₄₃₀₂ corresponding to the block of interest R ₄₃₀₀ .

Further, as shown in FIG. 46, the vector (3H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by three, so that it is shifted from the _target block R ₄₃₀₀ by the vector (3H ₁ , 0). The position on the image data at time t + 3t ₀ does not always match the position of the region R ₄₃₀₃ corresponding to the block of interest R ₄₃₀₀ .

Furthermore, since the vector (4H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by 4 as shown in FIG. 47, the vector (4H ₁ , 0) is shifted from the _target block R ₄₃₀₀ by the vector (4H ₁ , 0). The position on the image data at time t + 4t ₀ does not necessarily match the position of the region R ₄₃₀₄ corresponding to the target block R ₄₃₀₀ .

Further, as shown in FIG. 48, the vector (−H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by −1. Therefore, the vector (−H ₁ , 0) from the _target block R ₄₃₀₀ The position on the image data at time tt _{0 that} is shifted by a certain amount does not necessarily match the position of the region R ₄₃₁₁ corresponding to the block of interest R ₄₃₀₀ .

Furthermore, since the vector (-2H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by -2 as shown in FIG. 49, the vector (-2H ₁ , 0) from the target block R ₄₃₀₀ The position on the image data at time t−2t _{0 that} is shifted by a certain amount does not always match the position of the region R ₄₃₁₂ corresponding to the block of interest R ₄₃₀₀ .

Further, since the vector (−3H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by −3 as shown in FIG. 50, the vector (−3H ₁ , 0) from the target block R ₄₃₀₀ The position on the image data at time t-3t _{0 that} is shifted by a certain amount does not necessarily match the position of the region R ₄₃₁₃ corresponding to the block of interest R ₄₃₀₀ .

Furthermore, since the vector (−4H ₁ , 0) is a vector obtained by multiplying the motion vector (H ₁ , 0) by −4 as shown in FIG. 51, the vector (−4H ₁ , 0) from the target block R ₄₃₀₀ The position on the image data at time t-4t _{0 that} is shifted by a certain amount does not necessarily match the position of the region R ₄₃₁₄ corresponding to the block of interest R ₄₃₀₀ .

For this reason, in filtering focused on the _target block R ₄₃₀₀ , the image data at times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t ₀ , The pixel values of the regions shifted from the regions R ₄₃₁₄ , R ₄₃₁₃ , R ₄₃₁₂ , R ₄₃₁₁ , R ₄₃₀₂ , R ₄₃₀₃ , R ₄₃₀₄ corresponding to the target block R ₄₃₀₀ are used, and as a result, the true main Data of only frequency components in a region on the frequency domain extending in the component direction (for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. 22, region R _{1301 in} FIG. 23), There may be cases where it is difficult to obtain accurately.

  This is because the motion vector from the target block to the next frame can be determined by using the motion vector from the target block to the next frame. Therefore, when the principal component direction is calculated using the motion vector from the block of interest to the next frame even though the subject is not moving at a constant speed, an incorrect principal component direction is obtained. This is due to the fact that

  Therefore, in the principal component direction acquisition unit 31 of the transmission device 1 (FIG. 25), as described above, not only the correlation information indicating the correlation between the block of interest and the next frame, but also the block of interest and a plurality of frames before and after the block. By using a plurality of pieces of correlation information representing the correlation between the target block and each of the plurality of frames, a so-called average motion vector (hereinafter referred to as an average motion vector as appropriate) is obtained. The principal component direction with high accuracy can be obtained for the target block.

  FIG. 52 shows a configuration example of the principal component direction acquisition unit 31 in the case where the principal component direction acquisition unit 31 in FIG. 25 obtains the principal component direction from the average motion vector.

The buffer unit 101 is supplied with the 1 / t ₀ high frame rate moving image data read from the buffer unit 21 (FIG. 25), and the buffer unit 101 temporarily stores the moving image data.

  The block extraction unit 102 divides the moving image data stored in the buffer unit 101 into, for example, the above-described 16 × 16 pixel blocks, sequentially selects the target block, and supplies the selected block to the correlation calculation unit 103.

  The correlation calculation unit 103 reads, from the buffer unit 101, a plurality of frames (data) before and after the frame of the block of interest (the frame of interest) supplied from the block extraction unit 102. The correlation information representing the correlation with each of the frames is calculated and supplied to the scaling synthesis unit 104.

  Here, as the correlation information, for example, as in the case described in step S22 of FIG. 32, each pixel of the target block at a position where the target block is shifted by u and v in the spatial directions x and y, respectively. And the sum of absolute difference values of pixel values from pixels in other frames at the same position as the pixel. In this case, as described above, the position u, v in the spatial direction where the “value” of the correlation information is the minimum is the position in the spatial direction on the other frame where the correlation represented by the correlation information is the highest.

  Note that in the correlation calculation unit 103, the range of the frame for which the correlation information with the target block is calculated can be a range corresponding to the frame range used for filtering in the filter unit 22 (FIG. 25).

That is, as described with reference to FIG. 43, when the image data at time t is used as a reference (when time t is the frame of interest), the filtering in the filter unit 22 performs time t-4t ₀ , t-3t ₀ , t− 9 frames of image data of 2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t ₀ , time t-3t ₀ , t-2t ₀ , tt ₀ , t , T + t ₀ , t + 2t ₀ , t + 3t ₀ , and the like are used. In the correlation calculation unit 103, the frame for calculating the correlation information is image data in a range almost similar to the image data of 9 frames or 7 frames, that is, for example, time t-4t ₀ , t-3t ₀ , t-2t. _0, tt _0, t, can be _{t + t 0, t + 2t} 0, t + 3t 0, t + 4t 0 each image data.

Note that since the image data at time t is a frame of the block of interest (frame of interest), there is no need to calculate correlation information for that time t. That is, the correlation calculation unit 103 calculates the correlation information with the block of interest at times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t The image data is + 3t ₀ and t + 4t ₀ respectively.

  Here, the frame for which the correlation information with the block of interest is calculated is hereinafter referred to as a correlation calculation target frame as appropriate.

  The scaling synthesis unit 104 scales the correlation information obtained from each of the plurality of correlation calculation target frames supplied from the correlation calculation unit 103 in the spatial directions x and y as described later, and further, after the scaling. The correlation information is synthesized to obtain synthesized correlation information. Then, the scaling combining unit 104 supplies the combined correlation information to the minimum value detecting unit 105.

The minimum value detecting unit 105 detects a maximum correlation position that is a position in the spatial direction where the correlation represented by the combined correlation information from the scaling combining unit 104 is maximum, and calculates a vector to the maximum correlation position as an average motion of the target block. Ask as a vector. Then, the minimum value detection unit 105 adds the frame period t ₀ of the moving image data read from the buffer unit 21 (FIG. 25) to the average motion vector as a component in the time direction t. A direction orthogonal to the direction is detected and output as the principal component direction.

Next, FIG. 53 shows the block of interest R ₄₃₀₀ shown in FIG. 43 and the times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t, t + t ₀ , t + 2t ₀ , t. Correlation information with each image data (frame) of + 3t ₀ and t + 4t ₀ is shown.

Note that the correlation information shown in FIG. 53 is obtained by the correlation calculation unit 103 in FIG. However, as described above, the correlation information between the block of interest R ₄₃₀₀ and the image data at time t, which is the frame of interest, does not need to be calculated.

  In FIG. 53, as described above, the sum of the absolute differences of the pixel values is used as the correlation information. Therefore, the correlation information is a downward convex function, and the “value” of the correlation information is The position in the spatial direction that becomes the minimum is the position in the spatial direction where the correlation represented by the correlation information is the highest. A vector from the target block to that position represents a motion vector.

As described in FIG. 43, the same image data and the block of interest R ₄₃₀₀ is at time t-4t _0, present in the block of interest R ₄₃₀₀ position of the region R ₄₃₁₄ shifted by K ₄ in the spatial direction x from the time At t−3t ₀ , it exists in a region R ₄₃₁₃ at a position shifted from the target block R ₄₃₀₀ by K _{3 in the} spatial direction x. Similarly, the same image data as the _target block R ₄₃₀₀ is _displayed at the time t-2t ₀ in the region R ₄₃₁₂ at a position shifted by K ₂ from the _target block R _{4300 in the} spatial direction x, and at the time tt ₀ , the _target block R _At time t + t ₀ at a region R ₄₃₁₁ at a position shifted by K _{1 in the} spatial direction x from ₄₃₀₀ , at time t + 2t at a region R ₄₃₀₁ at a position shifted by H _{1 in the} spatial direction x from the _target block R _{4300 At 0} , it moves to R ₄₃₀₂ at a position shifted by H _{2 in the} spatial direction x from the _target block R ₄₃₀₀ , and at time t + 3t ₀ , it _enters a region R ₄₃₀₃ at a position shifted by H ₃ from the _target block R _{4300 in the} spatial direction x. At time t + 4t ₀ , each exists in a region R ₄₃₀₄ at a position displaced from the target block R ₄₃₀₀ by H _{4 in the} spatial direction x.

Accordingly, the correlation information at time t-4t ₀ is minimum at position x = K ₄ , and the correlation information at time t-3t ₀ is minimum at position x = K ₃ . Similarly, the correlation information at time t-2t ₀ is at position x = K ₂ , the correlation information at time tt ₀ is at x = K ₁ , and the correlation information at time t + t ₀ is at position x = H ₁ . The correlation information at time t + 2t ₀ is at position x = H ₂ , the correlation information at time t + 3t ₀ is at position x = H ₃ , and the correlation information at time t + 4t ₀ is at position x = H ₄ In each, it becomes the minimum.

In the principal component direction acquisition unit 31 in FIG. 52, the motion of the _target block R ₄₃₀₀ is approximated to be constant at the time from time t-4t _{0 to} t + 4t ₀ , and the vector representing the constant motion is the average motion. Required as a vector.

Therefore, the scaling synthesis unit 104 of the principal component direction acquisition unit 31 performs the times t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ , t Correlation information at each of + 4t ₀ is scaled in the spatial directions x and y, and the correlation information after the scaling is combined to obtain combined correlation information.

That is, when it is approximated that the motion of the _target block R ₄₃₀₀ is constant at the time from time t-4t _{0 to} time t + 4t ₀ , for example, when the correlation information at time t + t ₀ is used as a reference, time t-4t The correlation information at ₀ has a difference of −4 times with respect to the position (spatial direction x, y) with respect to the correlation information at time t + t ₀ . The correlation information at time t-3t ₀ is shifted by -3 times with respect to the correlation information at time t + t ₀ . Similarly, the correlation information at time t-2t ₀ is shifted by -2 times with respect to the position, and the correlation information at time tt ₀ is shifted by -1 times with respect to the correlation information at time t + t ₀ . However, the correlation information at time t + 2t ₀ is twice as much as the position, the correlation information at time t + 3t ₀ is three times as much as the position, and the correlation information at time t + 4t ₀ is about the position. There is a 4x shift.

Therefore, the scaling synthesis unit 104 performs scaling by multiplying the scale of the position of the correlation information at time t-4t ₀ by −1/4 (reversing and reducing). Similarly, the scaling composition unit 104 performs scaling to increase the scale of the position of the correlation information at time t-3t ₀ by −1/3, and scales the position of the position of the correlation information at time t−2t ₀ by −1/2. The scaling of the correlation information position at time tt ₀ is multiplied by -1 (reversed), and the scaling of the correlation information position at time t + 2t ₀ is halved (reduced). ) Scaling is performed so that the scale of the position of the correlation information at time t + 3t ₀ is ３ times, and the scaling is performed to increase the scale of the position of the correlation information at time t + 4t ₀ to ¼. .

For the correlation information at time t + t ₀ , the scaling composition unit 104 does not perform any scaling, but it may be seen that the fact that the scaling is not performed means that the position is scaled to 1 time. it can.

Here, FIG. 54 shows time t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + 2t ₀ , t + 3t ₀ , t + based on the correlation information at time t + t ₀ . The correlation information after scaling when the correlation information at 4t ₀ is scaled is shown.

Scaling, correlation information at time t-4t ₀ is becomes minimum at the position x = -K _4/4, the correlation information at time t-3t ₀ is at the minimum position x = -K _3/3. Similarly, the correlation information at time t-2t _0, at the position x = -K _2/2, correlation information at time tt _0, at x = -K _1, the correlation information at time t + 2t _0, the position x in = H _2/2, correlation information at time t + 3t _0, at the position x = H _3/3, correlation information at time t + 4t _0, at position x = H _4/4, respectively become minimum.

Since scaling is performed with reference to correlation information at time t + t _0, the correlation information at time t + t ₀ is minimum at position x = H ₁ before and after scaling.

In the principal component direction acquisition unit 31 in FIG. 52, the scaling synthesis unit 104 performs the time t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t _0. , T + 4t ₀ , the correlation information after scaling is synthesized to obtain synthesized correlation information.

  FIG. 55 shows combined correlation information obtained by combining the correlation information of FIG.

The combined correlation information of FIG. 55 is the time t-4t ₀ , t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ , t + 4t shown in FIG. It can be obtained by performing a synthesis of adding the correlation information after scaling in each of ₀ .

In the minimum value detection unit 105 of the principal component direction acquisition unit 31 in FIG. 52, the position x = L ₁ ((x, y) = (L ₁ , 0)) at which the composite correlation information in FIG. It is obtained as a vector (L ₁ , 0).

  Note that the combined correlation information can be obtained by adding the weighted addition in addition to simply adding the correlation information after scaling. In other words, the combined correlation information can be obtained by adding to the correlation information the correlation information closer to the time t of the block of interest with a greater weight. In this case, an average motion vector in which an image at a time close to the time t of the block of interest is regarded as important is obtained.

The minimum value detection unit 105 obtains the principal component direction of the block of interest from the average motion vector (L ₁ , 0) obtained as described above. Therefore, it is possible to obtain the principal component direction with relatively high accuracy for the block of interest.

That is, according to the principal component direction (that is, a plane perpendicular to (L ₁ , 0, t ₀ )) obtained from the average motion vector (L ₁ , 0) as described above, the filter unit 22 in FIG. For example, the pixel value as a filtering result of a pixel at a certain position (x, y) of the _target block R ₄₃₀₀ is a motion vector (L ₁ , 0) from the position (x, y) in the image data at time t + t ₀ The pixel value near the position shifted by a certain amount, the position near the position shifted by a vector (2L ₁ , 0) twice the motion vector (L ₁ , 0) from the position (x, y) in the image data at time t + 2t ₀ Pixel value, pixel value near time t + 3t ₀ position in image data at position (x, y), pixel value near position shifted by vector (3L ₁ , 0) three times motion vector (L ₁ , 0), time t + A pixel value in the vicinity of a position shifted by a vector (4L ₁ , 0) that is four times the motion vector (L ₁ , 0) from the position (x, y) in the image data at 4t ₀ , and a position in the image data at time tt ₀ ( x, y) Torr (L _1, 0) -1 times the vector (-L _1, 0) pixel values around a position shifted by a position in the image data at time _{t-2t 0 (x, y} ) from the motion vector (L ₁ , 0), the pixel value in the vicinity of the position shifted by -2 times the vector (−2L ₁ , 0), the position of the motion vector (L ₁ , 0) from the position (x, y) in the image data at time t-3t ₀ -3 times the pixel value in the vicinity of the position shifted by the vector (-3L ₁ , 0), from the position (x, y) in the image data at time t-4t ₀ to -4 times the motion vector (L ₁ , 0) Pixel values in the vicinity of the position shifted by the vector (−4L ₁ , 0) are respectively obtained.

Here, FIG. 56 shows the position in the image data at time t + t ₀ that is shifted from the position (x, y) of the _target block R _{4300 by} the motion vector (L ₁ , 0). Similarly, FIG. 57 to FIG. 63 show that at time t + 2t ₀ shifted from the position (x, y) of the _target block R ₄₃₀₀ by a vector (2L ₁ , 0) that is twice the motion vector (L ₁ , 0). Position in the image data at time t + 3t ₀ where the position in the image data is shifted from the position (x, y) of the _target block R ₄₃₀₀ by a vector (3L ₁ , 0) that is three times the motion vector (L ₁ , 0) Is shifted from the position (x, y) of the _target block R ₄₃₀₀ by a vector (4L ₁ , 0) four times the motion vector (L ₁ , 0) to the position in the image data at time t + 4t ₀ The position in the image data at time tt _{0 that} is shifted from the position (x, y) of R ₄₃₀₀ by a vector (−L ₁ , 0) that is −1 times the motion vector (L ₁ , 0) is the position of the target block R ₄₃₀₀ . (x, y) position in the -2 times the vector (-2L _1, 0) image data at time t-2t ₀ shifted by the motion vector (L _1, 0) from the position of the target block R ₄₃₀₀ (x , -3 times the vector of the motion vector from y) (L _1, 0) Le (-3L _1, 0) position in the image data at time t-3t ₀ shifted by -4 times the vector of position (x, y) from the motion vector of the target block _{R 4300 (L 1, 0)} ( The positions in the image data at time t-4t ₀ shifted by −4L ₁ , 0) are shown respectively.

56 to 63 and FIG. 44 to 51 described above are compared with each other, and in the case of FIGS. 56 to 63, the attention block R as a whole is more than the case of FIGS. ₄₃₀₀ using the pixel values of the position close to the corresponding region _{R 4301, R 4302, R 4303} , R 4304, R 4311, R 4312, R 4313, R 4314 , a is understood that the filtering of the block of interest R ₄₃₀₀ is performed .

  Next, processing performed by the principal component direction acquisition unit 31 in FIG. 52 will be described with reference to the flowcharts in FIGS.

  Note that the processing according to the flowcharts of FIGS. 64 to 67 corresponds to the processing of step S11 of FIG.

  52, the moving image data read from the buffer unit 21 (FIG. 25) is supplied to the buffer unit 101, and the buffer unit 101 temporarily stores the moving image data.

  64, the block extraction unit 102 divides the moving image data stored in the buffer unit 101 into blocks of 16 × 16 pixels, for example, as in step S21 of FIG. To the unit 103. In the subsequent processing, the blocks obtained by the block extraction unit 102 are sequentially performed as the target block.

Here, the position of the block of interest (for example, the position of the upper left pixel of the block of interest) is represented as (x ₀ , y ₀ ). The frame of the block of interest (frame of interest) is assumed to be a frame at time t.

After the processing in step S101, the process proceeds to step S102 in FIG. 65, in which the correlation calculation unit 103 reads out the image data at time t + t ₀ that is the image data of the frame next to the frame of interest from the buffer unit 101, and If, with the image data at time t + t _0, the function E ₁ representing the sum of the absolute differences of each other corresponding pixel (u _1, v _1), the image data at time t + t ₀ to correspond to the block of interest (X ₀ + u ₁ , y ₀ + v ₁ ) while changing the position (determined while changing (u ₁ , v ₁ )), and the function E ₁ (u ₁ , v ₁ ) is obtained at time t + t The correlation information at ₀ is supplied to the scaling synthesis unit 104, and the process proceeds to step S103.

In step S103, the correlation calculation unit 103 reads out the image data at time t + 2t ₀ that is image data of the next frame after the frame of interest from the buffer unit 101, and reads the block of interest and the image at time t + 2t ₀ . The function E ₂ (u ₂ , v ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t + 2t ₀ (x ₀ + u ₂ , y ₀ + v ₂ ) is changed and the function E ₂ (u ₂ , v ₂ ) is supplied as correlation information at time t + 2t _{0 to} the scaling synthesis unit 104, and the process proceeds to step S104.

In step S104, the correlation calculation unit 103 reads out the image data at time t + 3t ₀ that is temporally previous image data by 3 frames of the frame of interest from the buffer unit 101, and the block of interest and time t + 3t _0. The function E ₃ (u ₃ , v ₃ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data position of the image data at time t + 3t ₀ (x ₀ + u ₃ , y ₀ + v ₃ ), and the function E ₃ (u ₃ , v ₃ ) is supplied as correlation information at time t + 3t _{0 to} the scaling synthesis unit 104, and the process proceeds to step S105.

In step S105, image data at time t + 4t ₀ that is temporally preceding image data by four frames of the target frame is read from the buffer unit 101, and the target block and the image data at time t + 4t ₀ are The function E ₄ (u ₄ , v ₄ ) representing the sum of absolute differences between corresponding pixels is set to the position (x ₀ + u ₄ , y ₀ + v) of the image data at time t + 4t _{0 to which} the target block is associated. ₄ ), and the function E ₄ (u ₄ , v ₄ ) is supplied to the scaling synthesis unit 104 as correlation information at time t + 4t ₀ , and the process proceeds to step S106 in FIG.

In step S106, correlation calculation section 103, the image data at time tt ₀ is image data of a previous frame of the frame of interest, from the buffer unit 101, and the block of interest, the image data at time tt _0, the corresponding The function F ₁ (r ₁ , s ₁ ) that represents the sum of absolute differences between pixels is changed by changing the position (x ₀ + r ₁ , y ₀ + s ₁ ) of the image data at time tt ₀ to which the block of interest is associated. The function F ₁ (r ₁ , s ₁ ) is obtained as correlation information at time tt ₀ and supplied to the scaling synthesizer 104, and the process proceeds to step S107.

In step S107, the correlation calculation unit 103 reads out the image data at time t-2t ₀ that is image data of the previous frame before the frame of interest from the buffer unit 101, and reads the block of interest and the image at time t-2t ₀ . The function F ₂ (r ₂ , s ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t-2t ₀ (x ₀ + r ₂ , y ₀ + s ₂ ) is changed and the function F ₂ (r ₂ , s ₂ ) is supplied as correlation information at time t−2t _{0 to} the scaling synthesis unit 104, and the process proceeds to step S108.

In step S108, the correlation calculating unit 103, the image data at time t-3t ₀ is image data before in time by three frames of the frame of interest, from the buffer unit 101, and the block of interest, the time t-3t ₀ The function F ₃ (r ₃ , s ₃ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data position of the image data at time t-3t ₀ (x ₀ + r ₃ , y ₀ + s ₃ ), and the function F ₃ (r ₃ , s ₃ ) is supplied to the scaling combiner 104 as correlation information at time t-3t ₀ , and the process proceeds to step S109.

In step S109, the correlation calculation unit 103 reads out the image data at time t-4t ₀ that is temporally previous image data by four frames of the target frame from the buffer unit 101, reads the target block, and the time t-4t _0. The function F ₄ (r ₄ , s ₄ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data at the time t-4t ₀ corresponding to the block of interest (x ₀ + r ₄ , y ₀ + s ₄ ), and the function F ₄ (r ₄ , s ₄ ) is supplied to the scaling synthesis unit 104 as correlation information at time t-4t ₀ , and step S110 in FIG. Proceed to

In step S <b> 110, the scaling composition unit 104 receives the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), supplied from the correlation calculation unit 103. E ₄ (u ₄ , v ₄ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), F ₄ (r ₄ , s ₄ ) Based on the scale of the position (u ₁ , v ₁ ) of the correlation information E ₁ (u ₁ , v ₁ ), the above-described scaling is performed, and the correlation information after the scaling is further synthesized to produce a combined correlation Find information E (p, q). That is, the scaling combining unit 104, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (u 4 _{/ 4, v 4/4)} = (- r 1, -s 1) = (- r 2/2, -s 2/2) = (- r 3/3, -s 3/3) = (- r _4/4 , -s _4/4 ), and further scaling correlation information E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q), Add E ₄ (4p, 4q), F ₁ (-p, -q), F ₂ (-2p, -2q), F ₃ (-3p, -3q), F ₄ (-4p, -4q) Therefore, the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + E ₄ (4p, 4q) + F ₁ (- p, -q) + F ₂ (-2p, -2q) + F ₃ (-3p, -3q) + F ₄ (-4p, -4q)).

Note that the combined correlation information E (p, q) can be obtained by simply adding the correlation information after scaling, or by weighted addition as described above. That is, the combined correlation information E (p, q) is, for example, an expression 8 × E ₁ (p, q) + 4 × E _{2 in} which correlation information at a time closer to the time t of the block of interest is added with a greater weight. (2p, 2q) + 2 × E ₃ (3p, 3q) + 1 × E ₄ (4p, 4q) + 8 × F ₁ (-p, -q) + 4 × F ₂ (-2p, -2q) + It can be obtained by calculating 2 × F ₃ (−3p, −3q) + 1 × F ₄ (−4p, −4q).

  The scaling combining unit 104 obtains the combined correlation information E (p, q) in step S110, supplies the combined correlation information E (p, q) to the minimum value detecting unit 105, and proceeds to step S111. .

In step S111, the minimum value detection unit 105 has a position (relative position from the target block) in the spatial direction where the correlation represented by the combined correlation information E (p, q) from the scaling combining unit 104 is maximum (p, q) is detected as the maximum correlation position (p ₀ , q ₀ ), that is, the position (p, q) that minimizes the `` value '' of the combined correlation information E (p, q) is the maximum correlation position (p ₀ , q ₀ ), and the vector to the maximum correlation position (p ₀ , q ₀ ) is set as the average motion vector (p ₀ , q ₀ ), and the process proceeds to step S112.

In step S112, the minimum value detecting section 105, the average motion vector detected at step _{S111 (p 0, q 0)} , 3 -dimensional frame period t ₀ of the original video data was added as a component in the time direction t The direction orthogonal to the direction of the motion vector (p ₀ , q ₀ , t ₀ ) is the principal component direction, and the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) (FIG. 25) is supplied and the process is terminated.

In this case, for example, the filter information supply unit 32 directly uses the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) supplied from the minimum value detection unit 105 of the principal component direction acquisition unit 31 as filter information. It supplies to the filter part 22 (FIG. 25).

  In the filter unit 22, the process according to the flowchart shown in FIG. 68 is performed as the process of step S3 in FIG.

That is, in step S131, the filter unit 22 receives the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) supplied as filter information from the filter information supply unit 32, and proceeds to step S132. .

In step S132, the filter unit 22 calculates the pixel value C (t, x, y) of the pixel at the position (x, y) in the target block of the frame at time t, for example, the equation (1/16) × D ( t-3t ₀ , x-3p ₀ , y-3q ₀ ) + (2/16) × D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x, y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2 / 16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ) Perform filtering. Note that D (t, x, y) represents a pixel value at a position (x, y) at time t of moving image data used for filtering.

Here, equation (1/16) × D (t- 3t 0, x-3p 0, y-3q 0) + (2/16) × D (t-2t 0, x-2p 0, y-2q 0 ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x, y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2/16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), the so-called tap coefficients are 1/16, 2/16, 3/16, 4/16, 3/16, 2/16, 1/16, and 7 taps in the time direction It is equivalent to performing filtering by a FIR (Finite Impulse Response) filter having (filtering using image data of 7 frames). Such a tap coefficient FIR filter is a low-pass filter.

In step S132, the filtering as the calculation of the pixel value C (t, x, y) is performed while down-sampling is performed in which the number of samples in the time direction is reduced to 1/4, that is, the ratio of one frame every four frames. Done in That is, t is a multiple of 4 × t ₀ , for example.

After the process of step S132, the process proceeds to step S133, and the filter unit 22 performs the filtering result, that is, the expression (1/16) × D (t−3t ₀ , x−3p ₀ , y−3q ₀ ) + (2 / 16) × D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x , y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2/16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), the pixel value C (t, x, y) obtained by calculating the position (x , y) is output to the encoding unit 24 as a pixel value.

  In FIG. 68, the filtering is performed by the FIR filter having 7 taps in the time direction. However, for example, the filtering may be performed by the FIR filter having 9 taps in the time direction.

Also, the expression (1/16) × D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) + (2/16) × D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) + (3/16) × D (tt ₀ , xp ₀ , yq ₀ ) + (4/16) × D (t, x, y) + (3/16) × D (t + t ₀ , x + p ₀ , y + q ₀ ) + (2/16) × D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) + (1/16) × D (t + 3t ₀ , x + 3p ₀ , The calculation of y + 3q ₀ ) is equivalent to filtering by a low-pass filter, but it can also be regarded as weighted addition with a larger weight as it is closer to the frame of the target block.

In this way, when the pixel value C (t, x, y) as a filtering result is obtained by performing weighted addition with a larger weight as it is closer to the frame of the target block, the combined correlation information E (p, In the synthesis for obtaining q), as described above, the correlation information at the time closer to the time t of the block of interest is added with a larger weight and is added. 8 × E ₁ (p, q) + 4 × E ₂ ( 2p, 2q) + 2 × E ₃ (3p, 3q) + 1 × E ₄ (4p, 4q) + 8 × F ₁ (-p, -q) + 4 × F ₂ (-2p, -2q) +2 It is desirable to perform the calculation by × F ₃ (−3p, −3q) + 1 × F ₄ (−4p, −4q).

  As described above, not only the correlation information indicating the correlation between the block of interest and the next frame, but also the plurality of pieces of correlation information indicating the correlation between the block of interest and each of a plurality of frames before and after the block, Since an average motion vector representing an average motion for each of the frames is obtained, a highly accurate principal component direction can be obtained for the target block from the average motion vector.

  Further, the range of the plurality of frames used when obtaining the average motion vector is a range corresponding to the range of frames used for filtering in the filter unit 22 (FIG. 25), that is, a frame used for filtering (in FIG. 68, Since the range is almost the same as (7 frames centered on the frame of the target block) (9 frames centering on the frame of the target block in FIGS. 64 to 67), the filter unit 22 (FIG. 25) It is possible to obtain a filtering result (a filtering result of only frequency components that should be passed through).

  In FIG. 68, filtering is performed using 7 frames centered on the frame of the target block, and in FIGS. 64 to 67, the average motion vector is obtained using 9 frames centered on the frame of the target block. However, for example, filtering and calculation of the average motion vector can be performed using 7 frames or 9 frames centering on the frame of the target block. Filtering can also be performed using 9 frames centered on the frame of the target block, and the average motion vector can be calculated using 7 frames centered on the frame of the target block. That is, the frame range used for filtering and the frame range used for calculating the average motion vector may be completely coincident with each other or may be almost coincident with each other.

  As described above, by obtaining an average motion vector using a plurality of frames in a range substantially similar to the range of frames used for filtering in the filter unit 22 (FIG. 25), a plurality of average motion vectors are obtained from the average motion vector. A relatively accurate principal component direction can be obtained in the frame range.

  Next, as described above, according to the average motion vector, a relatively accurate principal component direction can be basically obtained.

  However, if the average motion vector is always obtained using all of a plurality of frames, an average motion vector having a large error may be obtained. In this case, the accuracy in the principal component direction is also deteriorated. This filtering result is also an inappropriate moving image (an image that cannot be restored to a moving image in which a human does not recognize the deterioration in image quality in the receiving device 2).

  That is, for example, in moving image data, when a stationary object is used as a background and a moving object exists as a foreground on the front side of the background, when the block of interest includes only the background, another frame (frame of interest In other frames, the background portion corresponding to the block of interest may be hidden behind the foreground and invisible. In this case, the value of the correlation information with the block of interest in the frame in which the background portion corresponding to the block of interest is hidden behind the foreground is not always the minimum at the position of the background portion corresponding to the block of interest. First, when the composite correlation information is calculated using such correlation information and the average motion vector is obtained, an average motion vector with a large error may be obtained.

  Specifically, it is assumed that the moving image data input to the transmission device 1 is, for example, moving image data on which a subject shown in FIG. 69 is projected.

Here, the horizontal axis in FIG. 69 represents the position x in the spatial direction. 69 shows the waveform of the subject P _{6901 as} the background, and the lower side of FIG. 69 shows the waveform of the subject P _{6902 as} the foreground.

The subject P ₆₉₀₁ shown on the upper side of FIG. 69 is stationary, and the subject P ₆₉₀₂ shown on the lower side of FIG. 69 is moving in the spatial direction x at a speed of g / t ₀ [m / s].

Incidentally, in the above case, in the calculation of the average motion vector, but using the image data at time t-4t ₀ to 9 times t + 4t ₀ (9 frames), in the following, the time t-3t ₀ to t Image data at 7 times (7 frames) of + 3t ₀ is used. Since the time from time t-3t _{0 to} t + 3t ₀ is as short as 6t ₀ , the speed of the subject P _{6902 at} that time can be regarded as being constant (approximate), and the above-described g / t ₀ Represents the constant speed.

70 and 71 show image data at seven times (7 frames) from time t-3t _{0 to} t + 3t ₀ on which the subjects P ₆₉₀₁ and P ₆₉₀₂ shown in FIG. 69 are projected.

In FIG. 70 and FIG. 71, image data at seven times from time t-3t _{0 to} t + 3t ₀ are illustrated in order from the top. FIG. 70 and FIG. 71 are the same figures except that the parts to which the reference numerals are attached are different.

According to the processing according to the flowcharts of FIGS. 64 to 67 described above (hereinafter, referred to as first motion detection processing as appropriate), for example, the image data at time t is divided into blocks (step S101 in FIG. 64). ). 70 and 71, the image data at time t is divided into 16 blocks B _{1 to} B ₁₆ .

For example, if the first motion detection process is performed on the assumption that the block B ₃ including only the subject P ₆₉₀₁ that is the background is the block of interest, as shown in FIG. 70, the image data at time t + t ₀ , a region corresponding to the target block B ₃ is because it is the block of interest B ₃ and the region B ₁₀₁ in the same position is determined at step S102 of FIG. 65, the correlation information at time _{t + t 0 E 1 (u} 1, v The value of ₁ ) is minimized when the position (u ₁ , v ₁ ) = (0,0).

Similarly, in the image data at times t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ , the region corresponding to the target block B ₃ is the same position as the target block B ₃ , respectively. region _{B 201, B 301, B -101} , B -201, a B _-301. Therefore, the correlation at time t + 2t ₀ Info _{E 2 (u 2, v 2} ), the correlation information E ₃ at time _{t + 3t 0 (u 3,} v 3), the correlation at time tt ₀ information F ₁ (r _1, s ₁ ), correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , and correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ are all minimized. , (U ₂ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ ) are (0,0) It is.

Therefore, according to the first motion detection process, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u) after scaling obtained in step S111 of FIG. ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), and F ₃ (r ₃ , s ₃ ), the combined correlation information E (p, q) The minimum (p, q), that is, the average motion vector (p ₀ , q ₀ ) is (0,0). As a result, for the block of interest B ₃ , (0, 0, t ₀ ) is obtained.

In this case, in FIG. 68, for each pixel of the block of interest B _3, area _{B -301, B -201, B -101} , to _{B 3, B 101, B 201} , B 301 pixel values of respective corresponding pixels Filtering for performing weighted addition with weights 1/16, 2/16, 3/16, 4/16, 3/16, 2/16, 1/16 is performed. Region _{B -301, B -201, B -101} , B 3, B 101, B 201, B 301 are both an area corresponding to the target block B _3, therefore, the region B _-301, B _-201 , B _-101 , B ₃ , B ₁₀₁ , B ₂₀₁ , B ₃₀₁ , data of only frequency components that can be recognized by human vision, that is, (0,0, t _{0 in this case)} ) To obtain appropriate data of only the frequency component in the region R ₁₀₀₁ (FIG. 20) extending in the direction orthogonal to (the principal component direction of the data of the target block B ₃ ) and having a width in the T direction of 2π / (4t ₀ ). be able to.

Next, for example, if the first motion detection process is performed assuming that the block B ₁₀ including only the subject P ₆₉₀₂ which is the foreground is the _target block, as shown in FIG. 70, the image data at time t + t ₀ is displayed. in the region corresponding to the block of interest B ₁₀ it is because it is the region B ₁₀₂ of position shifted by g in the x-direction from the target block B _10, obtained in the step S102 of FIG. 65, the correlation information at time t + t ₀ The value of E ₁ (u ₁ , v ₁ ) is minimized when the position (u ₁ , v ₁ ) = (g, 0).

Similarly, in the image data at times t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ , the regions corresponding to the target block B ₁₀ are respectively in the x direction from the target block B _10. the position of the region B ₂₀₂ shifted by 2g, the block of interest B ₁₀ position of the region B ₃₀₂ shifted by 3g in the x direction from the block of interest B ₁₀ position of the region B _-102 shifted in the x direction by -g from attention area at a position shifted by -2g from the block B ₁₀ in the x direction B _-202, a region B _-302 position shifted by -3g in the x direction from the target block B _10.

Therefore, the _value of the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is minimized when the position (u ₂ , v ₂ ) is (2g, 0) and the time t + The _value of the correlation information E ₃ (u ₃ , v ₃ ) at 3t ₀ is minimized when the position (u ₃ , v ₃ ) is (3g, 0), and the correlation information F _{1 at} time tt ₀ (r _1, s ₁₎ the value of the is minimized, positions (r _1, s ₁₎ is (-g, 0) is when the correlation at time t-2t ₀ information F ₂ (r _2, The value of s ₂ ) is minimized when the position (r ₂ , s ₂ ) is (-2g, 0), and the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ The value is minimized when the position (r ₃ , s ₃ ) is ( ₋₃ g, 0).

From the above, according to the first motion detection process, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (after scaling) obtained in step S111 of FIG. composite correlation information E (p, q) that is a composite result of u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) (P, q), that is, the average motion vector (p ₀ , q ₀ ) is (g, 0). As a result, for the target block B ₁₀ , (g , 0, t ₀ ).

In this case, in FIG. 68, for each pixel of the block of interest B _10, region _{B -302, B -202, B -102} , relative to _{B 10, B 102, B 202} , B 302 pixel values of respective corresponding pixels Filtering for performing weighted addition with weights 1/16, 2/16, 3/16, 4/16, 3/16, 2/16, 1/16 is performed. Region _{B -302, B -202, B -102} , B 10, B 102, B 202, B 302 are both an area corresponding to the target block B _10, therefore, the region B _-302, B _-202 , B _-102 , B ₁₀ , B ₁₀₂ , B ₂₀₂ , B ₃₀₂ , data of only frequency components that can be recognized by human vision, that is, (g, 0, t _{0 in this case)} ) Proper data of only frequency components in a region extending in a direction orthogonal to (a principal component direction of data of the target block B ₁₀ ) and having a width in the T direction of 2π / (4t ₀ ) can be obtained.

As described above, when all of the regions corresponding to the target block and the target block of other frames include only the subject P ₆₉₀₁ that is the background or only the subject P ₆₉₀₂ that is the foreground, the _target block By accurately obtaining the principal component direction of the data and filtering based on the principal component direction, it is possible to obtain appropriate data of only frequency components that can be recognized by human vision.

  On the other hand, if the area corresponding to the target block or the target block in another frame includes a plurality of subjects with different speeds, that is, includes a boundary portion of a plurality of subjects with different speeds, A large average motion vector may be required. In this case, from such an average motion vector, the principal component direction of the data of the block of interest cannot be accurately determined, and as a result, only frequency components that can be recognized by human vision by filtering based on the principal component direction. It may not be possible to obtain appropriate data.

That is, for example, in FIG. 71, if the first motion detection process is performed assuming that the block B ₇ including only the subject P _{6901 as the} background is the _target block, the _target block B in the image data at time t + t ₀ is performed. _Since the area corresponding to ₇ is the area B ₁₁₁ at the same position as the target block B ₇ , the _value of the correlation information E ₁ (u ₁ , v ₁ ) at time t + t ₀ obtained in step S102 of FIG. Is minimized when the position (u ₁ , v ₁ ) = (0,0).

Similarly, in the image data at time _{t + 2t 0, t + 3t} 0, tt 0, t-2t 0, a region corresponding to the block of interest B _7, which may each focus block B ₇ in the same position of the region B _211, B _311, B _-111, is a B _-211. Therefore, the correlation at time t + 2t ₀ Info _{E 2 (u 2, v 2} ), the correlation information E ₃ at time _{t + 3t 0 (u 3,} v 3), the correlation at time tt ₀ information F ₁ (r _1, s ₁ ) and the _value of the correlation information F ₂ (r ₂ , s ₂ ) at the time t-2t ₀ are minimized because the positions (u ₂ , v ₂ ), (u ₃ , v ₃ ), This is when (r ₁ , s ₁ ) and (r ₂ , s ₂ ) are (0,0).

However, in the image data at time t-3t _0, a region corresponding to the block of interest B _7, which may Originally, it should be region B _-311 at the same position as the block of interest B _7, FIG. 71, moving Hidden behind the subject P ₆₉₀₂ , and does not exist. Therefore, the _value of the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ is minimized when the position (r ₃ , s ₃ ) is (0,0). The value of the correlation information F ₃ (r ₃ , s ₃ ) is not always the minimum when the position (r ₃ , s ₃ ) is (0,0).

Then, according to the first motion detection process, in step S111 in FIG. 67, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) (P, q), that is, an average motion vector (p ₀ , q ₀ ) is obtained, and (p ₀ , q ₀ , t ₀ ) is obtained as a three-dimensional motion vector.

The three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) is not always the minimum value when the position (r ₃ , s ₃ ) is (0,0). The correlation information F ₃ (r ₃ , s ₃ ) is obtained from the average motion vector (p ₀ , q ₀ ) that is (p, q) that minimizes the composite correlation information E (p, q) obtained using The direction orthogonal to the vector (p ₀ , q ₀ , t ₀ ) does not always accurately represent the principal component direction of the data of the block of interest B ₇ .

In such filtering based on a direction orthogonal to the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), it is difficult to obtain appropriate data of only frequency components that can be recognized by human vision. It becomes.

Next, for example, in FIG. 71, if the first motion detection process is performed assuming that the block B ₁₄ including only the subject P _{6901 as the} background is the _target block, the _target block is included in the image data at time t + t ₀ . a region corresponding to the B ₁₄ is because it is the region B ₁₁₂ of the block of interest B ₁₄ at the same position is determined at step S102 of FIG. 65, the correlation information at time t + t ₀ E ₁ of (u _1, v ₁₎ The value is minimized when the position (u ₁ , v ₁ ) = (0,0).

Similarly, in the image data at time _{tt 0, t-2t 0,} t-3t 0, a region corresponding to the block of interest B _14, respectively, the block of interest B ₁₄ at the same position in the region B _-112, B _-212, B- ₃₁₂ . Therefore, correlation information F ₁ (r ₁ , s ₁ ) at time tt _0, correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , and correlation information F ₃ (r ₃ , r at time t-3t ₀ The value of s ₃ ) is minimized when the positions (r ₁ , s ₁ ), (r ₂ , s ₂ ), and (r ₃ , s ₃ ) are (0, 0). .

However, in the image data at time t + 2t _0, a region corresponding to the block of interest B ₁₄ is originally, but it should be region B ₂₁₂ of the block of interest B ₁₄ at the same position, moving the subject P ₆₉₀₂ are It is hidden behind and does not exist. Therefore, the _value of the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is minimized when the position (u ₂ , v ₂ ) is (0,0). The value of the correlation information E ₂ (u ₂ , v ₂ ) is not necessarily the minimum when the position (u ₂ , v ₂ ) is (0,0).

Similarly, in the image data at time t + 3t _0, a region corresponding to the block of interest B ₁₄ are originally of interest but block B ₁₄ and it should be region B ₃₁₂ in the same position, a moving subject P ₆₉₀₂ It is hidden behind and does not exist. Therefore, the _value of the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is minimized when the position (u ₃ , v ₃ ) is (0,0). The value of the correlation information E ₃ (u ₃ , v ₃ ) is not necessarily the minimum when the position (u ₃ , v ₃ ) is (0,0).

Then, according to the first motion detection process, in step S111 in FIG. 67, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) (P, q), that is, an average motion vector (p ₀ , q ₀ ) is obtained, and (p ₀ , q ₀ , t ₀ ) is obtained as a three-dimensional motion vector.

The three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) is not always the minimum when the position (u ₂ , v ₂ ) is (0,0). The correlation information F ₂ (u ₂ , v ₂ ) and the correlation information F ₃ (u ₃ , v ₃ ), the value of which is not necessarily minimized when the position (u ₃ , v ₃ ) is (0,0) Since it is obtained from the average motion vector (p ₀ , q ₀ ) that is (p, q) that minimizes the information E (p, q), the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) the direction perpendicular, not necessarily the main ingredient direction of the data block of interest B ₁₄ accurately represents.

In such filtering based on a direction orthogonal to the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), it is difficult to obtain appropriate data of only frequency components that can be recognized by human vision. It becomes.

Next, for example, in FIG. 71, if the block B ₉ including the subject P _{6901 that} is the background and the subject P ₆₉₀₂ that is the foreground is the _target block, and the first motion detection process is performed, the target block B ₉ is Since both the stationary subject P ₆₉₀₁ and the moving subject P ₆₉₀₂ are included, that is, at a position of the stationary subject P ₆₉₀₁ , the moving subject P ₆₉₀₂ is a stationary subject. Since P ₆₉₀₁ is hidden, the areas corresponding to such a target block B ₉ are time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t−2t ₀ , t It does not exist in any of the image data at -3t ₀ .

For this reason, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , t at time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) Positions (u ₁ , v ₁ ), (u ₂ ) most similar to the block of interest B ₉ in the image data at t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ )

Then, according to the first motion detection process, in step S111 in FIG. 67, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) (P, q), that is, an average motion vector (p ₀ , q ₀ ) is obtained, and (p ₀ , q ₀ , t ₀ ) is obtained as a three-dimensional motion vector.

This three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) has the minimum correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v) at the position most similar to the target block B _{9. 2} ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) Since it is obtained from the average motion vector (p ₀ , q ₀ ) that is (p, q) that minimizes the information E (p, q), the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) the direction perpendicular, not necessarily the main ingredient direction of the data block of interest B ₉ accurately represents.

In such filtering based on a direction orthogonal to the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), it is difficult to obtain appropriate data of only frequency components that can be recognized by human vision. It becomes.

  As described above, when the region corresponding to the block of interest or the block of interest in another frame includes a plurality of subjects having different speeds, that is, if the region includes boundary portions of a plurality of subjects having different speeds, the first In this motion detection processing, the average motion vector, and thus the principal component direction of the data of the block of interest, cannot be obtained accurately, and the filtering based on such principal component direction is performed, so that it is recognized by human vision. It may be difficult to obtain appropriate data of only frequency components that can be obtained.

  Therefore, FIG. 72 shows the average motion vector, and thus the principal component direction of the data of the target block, even when the region corresponding to the target block or the target block of another frame includes a plurality of subjects having different velocities. 25 shows an example of the configuration of the principal component direction acquisition unit 31 in FIG. 25 that performs a process for accurately obtaining (hereinafter referred to as a second motion detection process as appropriate).

  In the figure, portions corresponding to those in FIG. 52 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the principal component direction acquisition unit 31 in FIG. 72 is different from the scaling synthesis unit 104 and the minimum value detection unit 105 in that the scaling synthesis unit 124 and the minimum value detection unit 125 are provided in the case of FIG. It is configured in the same way.

  Similar to the scaling synthesis unit 104 in FIG. 52, the scaling synthesis unit 124 scales the correlation information obtained from each of the correlation calculation target frames supplied from the correlation calculation unit 103 in the spatial directions x and y, Further, the correlation information after scaling is synthesized to obtain synthesized correlation information. However, the scaling combining unit 124 obtains a plurality of pieces of combined correlation information by changing the number of pieces of correlation information after scaling (the number of pieces of correlation information to be combined). Then, the scaling synthesis unit 124 supplies the plurality of synthesis correlation information to the minimum value detection unit 125.

  The minimum value detecting unit 125 obtains maximum combined correlation information that is the combined correlation information having the maximum correlation among the plurality of combined correlation information from the scaling combining unit 124. That is, the minimum value detection unit 125 obtains the minimum value of “value” of each of the plurality of combined correlation information from the scaling combining unit 124. Further, the minimum value detection unit 125 obtains the composite correlation information having the minimum minimum value as the maximum composite correlation information from the plurality of composite correlation information.

Further, the minimum value detection unit 125 sets the position in the spatial direction where the correlation represented by the maximum combined correlation information is maximum, that is, the position in the spatial direction that minimizes the “value” of the maximum combined correlation information as the maximum correlation position. The vector to the maximum correlation position is detected as an average motion vector. Then, the minimum value detection unit 125 obtains a three-dimensional motion vector obtained by adding the frame period t ₀ of the moving image data stored in the buffer unit 21 (FIG. 25) to the average motion vector as a component in the time direction t. The direction orthogonal to the direction of the three-dimensional motion vector is detected and output as the principal component direction of the block of interest.

  Further, the minimum value detection unit 125 represents a range of frames used for calculating the correlation information synthesized to obtain the maximum synthesized correlation information among a plurality of frames used to obtain the synthesized correlation information. The kernel size to be calculated is further obtained as the effective range of the motion vector (average motion vector) of the target block, that is, the effective range of the main component direction of the target block, and is output together with the main component direction of the target block.

  Next, the processing of the filter generation unit 23 in FIG. 25 when the principal component direction acquisition unit 31 is configured as shown in FIG. 72 will be described with reference to the flowcharts in FIGS. 73 to 77.

  Of the processes of steps S151 to S176 according to the flowcharts of FIGS. 73 to 77, the processes of steps S151 to S172 are second motion detection processes and correspond to the process of step S11 of FIG. Further, the processing in steps S173 to S175 corresponds to the processing in step S12 in FIG. 31, and the processing in step S176 corresponds to the processing in step S13 in FIG.

  In the second motion detection process, an area corresponding to the target block among a plurality of frames for calculating correlation information with the target block using the so-called property of a moving image on which a plurality of subjects with different speeds are projected. Only the correlation information obtained for a frame having a frame is combined to obtain combined correlation information, and an average motion vector capable of obtaining an accurate principal component direction of the block of interest is obtained using the combined correlation information.

That is, for example, as shown in FIG. 69 to FIG. 71, seven frames from time t-3t _{0 to} t + 3t ₀ where the moving subject P ₆₉₀₂ exists on the front side of the stationary subject P ₆₉₀₁ are shown. Since the time between them is as short as 6t ₀ , it can be approximated that the speed of the moving subject P ₆₉₀₂ is constant.

Then, when a block having image data at time t is set as a target block, in the case where the moving subject P ₆₉₀₂ moves away from the _target block, a time later than time t ( In the image data of (future time), there is an area corresponding to the block of interest. That is, for example, in FIG. 71, when the block B ₇ is the _target block, the subject P ₆₉₀₂ moves away from the _target block B ₇ , but in this case, the time later than the time t. At times t + t ₀ , t + 2t ₀ , and t + 3t ₀ , areas B ₁₁₁ , B ₂₁₁ , and B ₃₁₁ corresponding to the block of interest B ₇ exist.

In addition, in the case where the moving subject P ₆₉₀₂ approaches the block of interest, there is an area corresponding to the block of interest in the image data at a time prior to the time t (past time). To do. That is, for example, in FIG. 71, when the block B ₁₄ is the _target block, the subject P ₆₉₀₂ approaches the _target block B ₁₄ , but in this case, the time before the time t At times tt ₀ , t-2t ₀ , and t-3t ₀ , there are regions B ₋₁₁₂ , B ₋₂₁₂ , and B ₋₃₁₂ corresponding to the block of interest B ₁₄ .

In the case where the stationary block includes the stationary subject P ₆₉₀₁ and the moving subject P ₆₉₀₂ , there is no region corresponding to the focused block in the image data at times before and after the time t. That is, for example, in FIG. 71, when the block B ₉ is the target block, the target block B ₉ includes a stationary subject P ₆₉₀₁ and a moving subject P _{6902. In} this case, , The block of interest B _{9 at} any of the times tt ₀ , t-2t ₀ , t-3t ₀ before the time t and the times t + t ₀ , t + 2t ₀ , t + 3t ₀ after the time t There is no corresponding area.

  In the second motion detection process, a process using the nature of a moving image in which a plurality of subjects with different speeds is projected as described above is performed.

  That is, in the principal component direction acquisition unit 31 in FIG. 72, the moving image data read from the buffer unit 21 (FIG. 25) is supplied to the buffer unit 101, and the buffer unit 101 temporarily stores the moving image data.

  In step S151 in FIG. 73, the block extraction unit 102 divides the moving image data stored in the buffer unit 101 into, for example, blocks of 16 × 16 pixels, as in step S101 in FIG. To the unit 103. In the subsequent processing, the blocks obtained by the block extraction unit 102 are sequentially performed as the target block.

Here, as in FIGS. 64 to 67, the position of the block of interest (for example, the position of the upper left pixel of the block of interest) is represented as (x ₀ , y ₀ ). The frame of the block of interest (frame of interest) is assumed to be a frame at time t.

After the process of step S151, the process proceeds to step S152, and the correlation calculation unit 103 reads out the image data at time t + t ₀ that is the image data of the frame next to the frame of interest from the buffer unit 101, The function E ₁ (u ₁ , v ₁ ) representing the sum of absolute differences between corresponding pixels with the image data at t + t ₀ is expressed as the position of the image data at time t + t ₀ ( x ₀ + u ₁ , y ₀ + v ₁ ), and the function E ₁ (u ₁ , v ₁ ) is supplied to the scaling synthesis unit 124 as correlation information at time t + t ₀ . The process proceeds to S153.

In step S153, the correlation calculation unit 103 reads the image data at time t + 2t ₀ that is the image data of the next frame after the frame of interest from the buffer unit 101, and reads the block of interest and the image at time t + 2t ₀ . The function E ₂ (u ₂ , v ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t + 2t ₀ (x ₀ + u ₂ , y ₀ + v ₂ ) is changed and the function E ₂ (u ₂ , v ₂ ) is supplied as correlation information at time t + 2t _{0 to} the scaling synthesis unit 124, and the process proceeds to step S154.

In step S154, the correlation calculation unit 103 reads out the image data at time t + 3t ₀ that is temporally preceding image data by three frames of the target frame from the buffer unit 101, and sets the target block and time t + 3t _0. The function E ₃ (u ₃ , v ₃ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data position of the image data at time t + 3t ₀ (x ₀ + u ₃ , y ₀ + v ₃ ), and the function E ₃ (u ₃ , v ₃ ) is supplied to the scaling combiner 124 as correlation information at time t + 3t ₀ , and step S155 in FIG. Proceed to

In step S155, the correlation calculation unit 103 reads the image data at time tt ₀ that is the image data of the frame before the frame of interest from the buffer unit 101, and corresponds the block of interest to the image data at time tt _0. Change the position (x ₀ + r ₁ , y ₀ + s ₁ ) of the image data at the time tt ₀ to associate the block of interest with the function F ₁ (r ₁ , s ₁ ) that represents the sum of absolute differences between pixels. The function F ₁ (r ₁ , s ₁ ) is obtained as correlation information at time tt ₀ and supplied to the scaling synthesizer 124, and the process proceeds to step S156.

In step S156, the correlation calculation unit 103 reads the image data at the time t-2t ₀ that is the image data of the previous frame before the target frame from the buffer unit 101, and reads the target block and the image at the time t-2t ₀ . The function F ₂ (r ₂ , s ₂ ) representing the sum of absolute differences between corresponding pixels with the data is expressed as the position of the image data at the time t-2t ₀ (x ₀ + r ₂ , y ₀ + s ₂ ) is changed, and the function F ₂ (r ₂ , s ₂ ) is supplied as the correlation information at time t-2t _{0 to} the scaling combiner 124, and the process proceeds to step S157.

In step S157, the correlation calculation unit 103 reads out the image data at time t-3t ₀ that is temporally previous image data by three frames of the frame of interest from the buffer unit 101, reads the block of interest, and time t-3t _0. The function F ₃ (r ₃ , s ₃ ) representing the sum of absolute differences between corresponding pixels with the image data of the image data position of the image data at time t-3t ₀ (x ₀ + r ₃ , y ₀ + s ₃ ), and the function F ₃ (r ₃ , s ₃ ) is supplied to the scaling combiner 124 as correlation information at time t-3t ₀ , and the process proceeds to step S158.

In step S <b> 158, the scaling synthesis unit 124 includes six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), the position of the correlation information E ₁ (u ₁ , v ₁ ) Scaling is performed with the scale of (u ₁ , v ₁ ) as a reference, and further, the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (- r _{1, -s 1) = (-} r 2/2, -s 2/2) = (- r 3/3, performs scaling with -s _3/3), further, the correlation information after the scaling E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q), F ₁ (-p, -q), F ₂ (-2p, -2q), F ₃ (-3p, -3q), the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (-p, -q) + F ₂ (-2p, -2q) + F ₃ (-3p, -3q)).

  Note that the combined correlation information E (p, q) can be obtained by simply adding the correlation information after scaling, or by weighted addition as described above.

As described above, the scaling synthesizer 124, in step S158, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t _0, correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , correlation information F ₂ (r ₂ , s at time t-2t _{0 2} ), combined correlation information E (p, q) that combines all of the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ , that is, later in time with respect to time t of the frame of interest After obtaining the combined correlation information E (p, q), which is a total of 6 correlation information for each of the 3 frames and the previous 3 frames in time, the combined correlation information E (p, q) is obtained. The obtained correlation information is normalized by dividing the combined correlation information E (p, q) by 6, which is the number of correlation information combined, and further, a predetermined offset is added to the normalized combined correlation information E (p, q). It is added to the value L ₃₃ By the final synthesized correlation information E (p, q) (= {E 1 (p, q) + E 2 (2p, 2q) + E 3 (3p, 3q) + F 1 (-p, -q _{) + F 2 (-2p, -2q} ) + F 3 (-3p, -3q)} / 6 + L 33) obtained. Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S158 to S159.

In step S159, the minimum value detecting unit 125 supplies each of the three frames that are temporally subsequent to the time t of the frame of interest and the three frames that are temporally previous supplied from the scaling composition unit 124 in step S158. (P ₃₃ , q ₃₃ ), which is the position (p, q) in the spatial direction at which the correlation represented by the combined correlation information E (p, q) that is obtained by combining the correlation information of Detect (p ₃₃ , q ₃₃ ), which is the position (p, q) that minimizes the `` value '' of (p, q), and position (p, q) is the maximum correlation position (p ₃₃ , q ₃₃ ) Stored together with the “value” of the combined correlation information E (p ₃₃ , q ₃₃ ) in some cases, that is, the minimum value E ₃₃ , proceeds to step S 160 in FIG.

In step S <b> 160, the scaling synthesis unit 124 includes six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), five correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), correlation information E ₁ (u ₁ , v ₁ ) Is scaled with reference to the scale of the position (u ₁ , v ₁ ), and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (- r _{1, -s 1) = (-} r 2/2, performs scaling with -s _2/2), further, the correlation information E ₁ after the scaling _{(p, q), E 2} (2p, 2q ), E ₃ (3p, 3q), F ₁ (-p, -q), and F ₂ (-2p, -2q) are added to obtain the combined correlation information E (p, q) (= E ₁ (p , q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (−p, −q) + F ₂ (−2p, −2q)).

As described above, the scaling synthesizer 124, in step S160, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t _0, correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , correlation information F ₂ (r ₂ , s at time t-2t _{0 2} ), the combined correlation information E (p, q), that is, the sum of the three frames after the time t and the two frames before the time t of the frame of interest. The composite correlation information E (p, q) obtained by synthesizing the five correlation information in the above is obtained, and then the composite correlation information E is obtained by five, which is the number of correlation information synthesized in obtaining the composite correlation information E (p, q). Normalization is performed by dividing (p, q) .Further, by adding a predetermined offset value L ₂₃ to the normalized combined correlation information E (p, q), the final combined correlation information E ( p, q) (= {E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (-p, -q) + F ₂ (-2p, -2q)} / 5 + L ₂₃ ) Then, the scaling synthesis unit 124 supplies the final synthesis correlation information E (p, q) to the minimum value detection unit 125, and proceeds from step S160 to S161.

In step S161, the minimum value detector 125 supplies each of the three frames that are temporally subsequent to the time t of the frame of interest and the two frames that are temporally previous, supplied from the scaling composition unit 124 in step S160. (P ₂₃ , q ₂₃ ), which is the position (p, q) in the spatial direction where the correlation represented by the combined correlation information E (p, q), which is obtained by synthesizing the correlation information, is detected, that is, the combined correlation information E (p ₂₃ , q ₂₃ ), which is the position (p, q) that minimizes the (value) of (p, q), is detected, and the position (p, q) is the maximum correlation position (p ₂₃ , q ₂₃ ) Stored together with the “value” of the composite correlation information E (p ₂₃ , q ₂₃ ) in some cases, that is, the minimum value E ₂₃ , proceeds to step S162.

In step S <b> 162, the scaling synthesis unit 124 includes six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), four correlation information E ₁ (u ₁ , v ₁ ), E _{2 (u 2, v 2)} , E 3 (u 3, v 3), the _{F 1 (r 1, s 1} ), the position of the correlation information _{E 1 (u 1, v 1} ) (u 1, v 1) Is scaled with reference to the scale, and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (u 3/3, v 3/3) = (- r ₁ , -s ₁ ), and the correlation information E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q), F ₁ (- By adding (p, -q), the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q) + F ₁ (- p, -q)).

As described above, the scaling synthesizer 124, in step S162, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Combined correlation information E (p, q) obtained by combining four pieces of correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t _{0 and} correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , That is, the combined correlation information E (p, q) obtained by synthesizing a total of four pieces of correlation information for each of the three frames later in time with respect to the time t of the frame of interest and the one frame earlier in time. After obtaining, the composite correlation information E (p, q) is normalized by dividing the composite correlation information E (p, q) by 4 which is the number of correlation information synthesized in obtaining the composite correlation information E (p, q). By adding a predetermined offset value L ₁₃ to the composite correlation information E (p, q) of the final composite correlation information E (p, q) (= {E ₁ (p, q) + E ₂ ( 2p, 2q) + E ₃ (3p, 3q) + F ₁ (-p, -q)} / 4 + L ₁₃ ) Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S162 to S163.

In step S163, the minimum value detector 125 supplies each of the three frames later in time with respect to the time t of the frame of interest and the first frame temporally supplied from the scaling composition unit 124 in step S162. (P ₁₃ , q ₁₃ ), which is the position (p, q) in the spatial direction where the correlation represented by the combined correlation information E (p, q), which is obtained by synthesizing the correlation information, is maximized, that is, the combined correlation information E (p ₁₃ , q ₁₃ ), which is the position (p, q) that minimizes the (value) of (p, q), is detected, and the position (p, q) is the maximum correlation position (p ₁₃ , q ₁₃ ) The value is stored together with the “value” of the composite correlation information E (p ₁₃ , q ₁₃ ) in some cases, that is, the minimum value E ₁₃ , and the process proceeds to step S164.

In step S164, the scaling synthesis unit 124 receives the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), three correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ) and E ₃ (u ₃ , v ₃ ) are scaled based on the scale of the position (u ₁ , v ₁ ) of the correlation information E ₁ (u ₁ , v ₁ ), and Then, the scaled correlation information is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = With _{(u 3/3, v 3} /3) By performing scaling and further adding the correlation information E ₁ (p, q), E ₂ (2p, 2q), E ₃ (3p, 3q) after the scaling, the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q)) is obtained.

As described above, the scaling synthesizer 124, in step S164, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Combined correlation information E (p, q) obtained by combining three pieces of correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ , that is, 3 later in time with respect to time t of the target frame This is the number of pieces of correlation information synthesized for obtaining the combined correlation information E (p, q) after obtaining the combined correlation information E (p, q) obtained by combining a total of three pieces of correlation information for each frame. 3 is normalized by dividing the composite correlation information E (p, q) by 3 and, further, by adding a predetermined offset value L ₀₃ to the composite correlation information E (p, q) after normalization, Synthetic correlation information E (p, q) (= {E ₁ (p, q) + E ₂ (2p, 2q) + E ₃ (3p, 3q)} / 3 + L ₀₃ ) is obtained. Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S164 to S165.

In step S165, the minimum value detection unit 125 synthesizes the correlation information supplied from the scaling synthesis unit 124 in step S164 and synthesizes the correlation information for each of the three frames after the time t of the frame of interest. (P ₀₃ , q ₀₃ ), which is the position (p, q) in the spatial direction where the correlation represented by E (p, q) is maximum, is detected, that is, the “value” of the combined correlation information E (p, q) (P ₀₃ , q ₀₃ ), which is a position (p, q) that minimizes the position of the signal, is detected, and the combined correlation information E (p (p, q) is the maximum correlation position (p ₀₃ , q ₀₃ ) ₀₃ , q ₀₃ ), ie, the minimum value E ₀₃ , and the process proceeds to step S166 in FIG.

In step S <b> 166, the scaling synthesis unit 124 includes the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), five correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), correlation information E ₁ (u ₁ , v ₁ ) Is scaled with reference to the scale of the position (u ₁ , v ₁ ), and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (u 2/2, v 2/2) = (- r 1, -s 1) = (- r 2 / _{2, -s 2/2) =} (- r 3/3, performs scaling with -s _3/3), further, the correlation information E ₁ after the scaling _{(p, q), E 2} (2p , 2q), F ₁ (−p, −q), F ₂ (−2p, −2q), and F ₃ (−3p, −3q), the combined correlation information E (p, q) (= E ₁ (p, q) + E ₂ (2p, 2q) + F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)) is obtained.

As described above, the scaling synthesizer 124, in step S166, the correlation information E ₁ at time _{t + t 0 (u 1,} v 1), the correlation information E ₂ at time _{t + 2t 0 (u 2,} v 2) , Correlation information F ₁ (r ₁ , s ₁ ) at time tt _0, correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , correlation information F ₃ (r ₃ , s at time t-3t _{0 3} ) combined correlation information E (p, q), that is, 5 frames in total for each of the two frames later in time and the previous three frames in time with respect to the time t of the frame of interest. After obtaining the combined correlation information E (p, q) obtained by combining the correlation information, the combined correlation information E (p, q) is obtained by 5 which is the number of the correlation information combined in obtaining the combined correlation information E (p, q). q) is divided by normalization, and the final combined correlation information E (p, q is added by adding a predetermined offset value L ₃₂ to the normalized combined correlation information E (p, q). ) (= {E ₁ (p, q ) + E ₂ (2p, 2q) + F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)} / 5 + L ₃₂ ). Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S166 to S167.

In step S167, the minimum value detection unit 125 supplies each of the two frames that are temporally subsequent to the time t of the frame of interest and the three frames that are temporally previous supplied from the scaling composition unit 124 in step S166. (P ₃₂ , q ₃₂ ), which is the position (p, q) in the spatial direction where the correlation represented by the combined correlation information E (p, q), which is obtained by combining the correlation information of (p ₃₂ , q ₃₂ ), which is the position (p, q) that minimizes the (value) of (p, q), is detected, and the position (p, q) is the maximum correlation position (p ₃₂ , q ₃₂ ) Stored together with the “value” of the composite correlation information E (p ₃₂ , q ₃₂ ) in some cases, that is, the minimum value E ₃₂ , proceeds to step S168.

In step S168, the scaling synthesis unit 124 receives the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), four correlation information E ₁ (u ₁ , v ₁ ), F _{1 (r 1, s 1)} , F 2 (r 2, s 2), F 3 (r 3, s 3) for the correlation information E ₁ (u _1, v ₁₎ the position of the (u _1, v ₁₎ Is scaled with reference to the scale, and the correlation information after the scaling is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (u 1, v 1) = (- r 1, -s 1) = (- r 2/2, -s 2/2) = (- r _3/3 performs scaling with -s _3/3), further, its the scaled correlation information _{E 1 (p, q),} F 1 (-p, -q), F 2 (-2p, -2q) and F ₃ (-3p, -3q) are added to obtain the combined correlation information E (p, q) (= E ₁ (p, q) + F ₁ (-p, -q) + F ₂ Find (-2p, -2q) + F ₃ (-3p, -3q)).

As described above, the scaling synthesizer 124, in step S168, the correlation at time t + t ₀ information _{E 1 (u 1, v 1} ), the correlation information at time _{tt 0 F 1 (r 1,} s 1), the time Synthetic correlation information E (p, q) obtained by synthesizing correlation information F ₂ (r ₂ , s ₂ ) at t-2t _{0 and} correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ , that is, attention After obtaining the combined correlation information E (p, q), which is a total of four pieces of correlation information for one frame later in time with respect to time t of the frame and three frames earlier in time. Then, the composite correlation information E (p, q) is normalized by dividing the composite correlation information E (p, q) by 4 which is the number of correlation information synthesized in obtaining the composite correlation information E (p, q). By adding a predetermined offset value L ₃₁ to the information E (p, q), the final combined correlation information E (p, q) (= {E ₁ (p, q) + F ₁ (−p, -q) + F ₂ (-2p, -2q) + F ₃ (-3p, -3q)} / 4 + L ₃₁ ) obtain. Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S168 to S169.

In step S169, the minimum value detecting unit 125 supplies each of one frame later in time with respect to time t of the target frame and three frames earlier in time supplied from the scaling composition unit 124 in step S168. (P ₃₁ , q ₃₁ ), which is the position (p, q) in the spatial direction where the correlation represented by the combined correlation information E (p, q), which is obtained by combining the correlation information, is maximized, that is, the combined correlation information E (p ₃₁ , q ₃₁ ), which is the position (p, q) that minimizes the (value) of (p, q), is detected, and the position (p, q) is the maximum correlation position (p ₃₁ , q ₃₁ ) Stored together with the “value” of the composite correlation information E (p ₃₁ , q ₃₁ ) in some cases, that is, the minimum value E ₃₁ , proceeds to step S170.

In step S <b> 170, the scaling synthesis unit 124 includes the six pieces of correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , v ₂ ), E ₃ (u ₃ , v ₃ ) supplied from the correlation calculation unit 103. ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ), three correlation information F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ) and F ₃ (r ₃ , s ₃ ) are scaled with reference to the scale of the position (u ₁ , v ₁ ) of the correlation information E ₁ (u ₁ , v ₁ ), and Then, the scaled correlation information is synthesized to obtain synthesized correlation information E (p, q). That is, the scaling synthesizer 124, the position (p, q) = (- r 1, -s 1) = (- r 2/2, -s 2/2) = (- r 3/3, -s 3 / Scaling is performed by setting 3), and the correlation information F ₁ (-p, -q), F ₂ (-2p, -2q), and F ₃ (-3p, -3q) after the scaling are added. Thus, the composite correlation information E (p, q) (= F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)) is obtained.

As described above, in step S170, the scaling composition unit 124 calculates the correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ , the correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ , and the time The combined correlation information E (p, q) obtained by synthesizing the correlation information F ₃ (r ₃ , s ₃ ) at t-3t ₀ , that is, for each of the three frames temporally prior to the time t of the frame of interest After obtaining the combined correlation information E (p, q) obtained by combining the three pieces of correlation information in total, the number of the correlation information combined in determining the combined correlation information E (p, q) is 3 to determine the combined correlation. normalized by dividing the information E (p, q), further synthesized correlation information E after the normalization (p, q) in, by adding a predetermined offset value L _30, final synthesized correlation information E (p, q) (= {F ₁ (−p, −q) + F ₂ (−2p, −2q) + F ₃ (−3p, −3q)} / 3 + L ₃₀ ) is obtained. Then, the scaling combining unit 124 supplies the final combined correlation information E (p, q) to the minimum value detecting unit 125, and proceeds from step S170 to S171.

In step S171, the minimum value detection unit 125 synthesizes the correlation information that is supplied from the scaling synthesis unit 124 in step S170 and synthesizes the correlation information for each of the previous three frames with respect to the time t of the frame of interest. (P ₃₀ , q ₃₀ ) is detected as a position (p, q) in the spatial direction where the correlation represented by E (p, q) is maximum, that is, the `` value '' of the combined correlation information E (p, q) (P ₃₀ , q ₃₀ ), which is the position (p, q) that minimizes, is detected, and the combined correlation information E (p (p, q) is the maximum correlation position (p ₃₀ , q ₃₀ ) ₃₀ , q ₃₀ ), ie, the minimum value E ₃₀ , and the process proceeds to step S 172 in FIG. 77.

In step S <b> 172, the minimum value detection unit 125 obtains the maximum combined correlation information that is the combined correlation information with the maximum correlation among the plurality of combined correlation information from the scaling combining unit 124. In other words, the minimum value detection unit 125 synthesizes the combined correlation information E (p, q) obtained by synthesizing the six correlation information of the three frames later in time and the previous three frames with respect to the time t of the frame of interest. The minimum value E ₃₃ (step S159), the combined correlation information E (p, q) obtained by synthesizing the five correlation information of the temporally subsequent three frames and the temporally previous two frames with respect to the time t of the frame of interest. The minimum value E ₂₃ (step S161), the combined correlation information E (p, q) obtained by synthesizing the four correlation information of the temporally subsequent three frames and the temporally previous one frame with respect to the time t of the frame of interest. Minimum value E ₁₃ (step S163), minimum value E _{03 of} combined correlation information E (p, q) obtained by synthesizing three correlation information of three frames that are temporally subsequent to time t of the frame of interest (step S165) , 2 frames later in time and 3 frames earlier in time with respect to time t of the frame of interest The minimum value E ₃₂ (step S167) of the combined correlation information E (p, q) obtained by synthesizing the five pieces of correlation information of the first frame, one frame later in time with respect to the time t of the frame of interest, and the previous three in time The minimum value E _{31 of} the combined correlation information E (p, q) obtained by combining the four pieces of correlation information of the frame (step S169), and the three correlation information of the previous three frames with respect to the time t of the target frame are combined. The combined correlation information that gives the minimum value of the minimum values E ₃₀ (step S171) of the combined correlation information E (p, q) is obtained as the maximum combined correlation information.

Here, in order to obtain the combined correlation information that has become the maximum combined correlation information, k + If the correlation information of h is synthesized (here, k, h = 0,1,2,3), among the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ Can be expressed as E _hk .

In step S172, the minimum value detecting unit 125 further minimizes the “value” of the maximum combined correlation information, that is, the position in the spatial direction that gives the minimum value E _hk of the maximum combined correlation information (p _hk , q _hk ), together with the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ , the stored positions (p ₃₃ , q ₃₃ ), (p ₂₃ , q ₂₃ ), ( p ₁₃ , q ₁₃ ), (p ₀₃ , q ₀₃ ), (p ₃₂ , q ₃₂ ), (p ₃₁ , q ₃₁ ), (p ₃₀ , q ₃₀ ) Let it be the position (p ₀ , q ₀ ).

Then, the minimum value detection unit 125 sets the maximum correlation position (p ₀ , q ₀ ) as the average motion vector (p ₀ , q ₀ ), and uses the original motion picture data as the average motion vector (p ₀ , q ₀ ). A three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) obtained by adding the frame period t ₀ of the above as a component in the time direction t is obtained, and the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) Assuming that the direction orthogonal to the direction is the principal component direction, a three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) is supplied to the filter information supply unit 32 (FIG. 25), and the maximum combined correlation information The minimum value E _hk is supplied to the filter information supply unit 32. Further, the minimum value detection unit 125 uses (h, k), which is the suffix of the minimum value E _hk of the maximum combined correlation information, and the range of frames used for the calculation of the correlation information combined to obtain the maximum combined correlation information. Is supplied to the filter information supply unit 32, and the process proceeds from step S172 to S173.

Here, the offset values L ₃₃ (step S158), L ₂₃ (step S160), L ₁₃ (step S162), L ₀₃ (step S164), L ₃₂ (step S166), L ₃₁ (step S168), L described above. ₃₀ (step S170), the formula _{L 33 ≦ L 32 ≦ L 31} ≦ L 30, and a small constant satisfying the formula _{L 33 ≦ L 23 ≦ L 13} ≦ L 03.

In step S173, the filter information supply unit 32 (FIG. 25) determines that the minimum value E _hk of the maximum combined correlation information from the principal component direction acquisition unit 31 (the minimum value detection unit 125 thereof) in FIG. It is determined whether (less than). Here, the threshold value ε used in step S173 is larger than the offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L _{30 and} can be determined by, for example, simulation. . In addition, the threshold value ε can be set according to a user operation.

If it is determined in step S173 that the minimum value E _hk of the maximum combined correlation information is equal to or smaller than the predetermined threshold ε, that is, the minimum value E _hk of the maximum combined correlation information is small to some extent, and therefore corresponds to the block of interest. When the region exists in the frame before or after time t, the process proceeds to step S174, and the filter information supply unit 32 performs the principal component from the principal component direction acquisition unit 31 from the origin (0, 0) in the frequency domain. A region extending in the direction, having a width of 2π / (4 × t ₀ ) in the T direction, and the X and Y directions being − (π / r ₀ ) to + (π / r ₀ ), and the T direction Are in the range of − (π / t ₀ ) to + (π / t ₀ ), that is, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region in FIG. R ₁₃₀₁ is determined as the pass band of the filter, and the minimum value detection of the principal component direction acquisition unit 31 in FIG. (H, k) from the unit 125) is directly determined as the kernel size, and the process proceeds to step S176.

On the other hand, if it is determined in step S173 that the minimum value E _hk of the maximum combined correlation information is not less than or equal to the predetermined threshold ε, that is, the minimum value E _hk of the maximum combined correlation information is large, and therefore corresponds to the block of interest. When the region does not exist in any of the frames before and after time t, the process proceeds to step S175, and the filter information supply unit 32 starts from the origin (0, 0) in the frequency domain as in step S174. , A region extending in the principal component direction from the principal component direction acquisition unit 31 and having a width of 2π / (4 × t ₀ ) in the T direction, and the X and Y directions being − (π / r ₀ ) to A region where + (π / r ₀ ) and the T direction is in the range of − (π / t ₀ ) to + (π / t ₀ ) is determined as the passband of the filter. Further, in step S175, the filter information supply unit 32 replaces (h, k) from the principal component direction acquisition unit 31 (the minimum value detection unit 125 thereof) in FIG. 72 with (h, k) = (0, 0) is determined as the kernel size, and the process proceeds to step S176.

In step S176, the filter information supply unit 32 uses, for example, a three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) as filter information indicating the pass band of the filter determined in step S174 or S175, and the filter unit 22. (FIG. 25) and the kernel size (h, k) are supplied to the filter unit 22 and the process is terminated.

According to the process in accordance with the flowchart shown in FIG. 73 to FIG. 77, the block of interest, and three-dimensional motion vectors as filter information such as: _{(p 0, q 0, t} 0), the kernel size (h , k).

For example, in the moving image data shown in FIG. 70, when the block B ₃ is the target block, as described above, the times t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t−2t at _0, respectively image data in t-3t _0, block of interest B ₃ with the corresponding region _{B 101, B 201, B 301} , B -101, B -201, B -301 is present, the time t + t ₀ Correlation information E ₁ (u ₁ , v ₁ ), correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ , correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ , time tt correlation information in ₀ F ₁ of (r _1, s _1), the correlation information F ₂ at time _{t-2t 0 (r 2,} s 2), the correlation information F ₃ at time _{t-3t 0 (r 3,} s 3) The values for all are (u ₂ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ ) are (0, 0) is the smallest and almost zero.

In this case, the position (p ₃₃ , q ₃₃ ) that gives the minimum value E ₃₃ of the combined correlation function E (p, q) obtained in step S159 of FIG. 74 is (0,0), and the minimum value E ₃₃ is approximately equal to the offset value L _33. Similarly, the position (p ₂₃ , q ₂₃ ) that gives the minimum value E ₂₃ of the combined correlation function E (p, q) obtained in step S161 in FIG. 75 is (0,0), and the minimum value E ₂₃ is approximately equal to the offset value L _23. In addition, the position (p ₁₃ , q ₁₃ ) that gives the minimum value E ₁₃ of the combined correlation function E (p, q) obtained in step S163 in FIG. 75 is (0,0), and the minimum value E ₁₃ is almost equal to equal to the offset value L _13. Further, the position (p ₀₃ , q ₀₃ ) that gives the minimum value E ₀₃ of the combined correlation function E (p, q), which is obtained in step S165 of FIG. 75, is (0, 0), and the minimum value E ₀₃ is almost equal to equal to the offset value L _03. In addition, the position (p ₃₂ , q ₃₂ ) that gives the minimum value E ₃₂ of the combined correlation function E (p, q) obtained in step S167 of FIG. 76 is (0,0), and the minimum value E ₃₂ is almost equal to equal to the offset value L _32. Further, the position (p ₃₁ , q ₃₁ ) that gives the minimum value E ₃₁ of the combined correlation function E (p, q) obtained in step S169 in FIG. 76 is (0,0), and the minimum value E ₃₁ is almost equal to equal to the offset value L _31. In addition, the position (p ₃₀ , q ₃₀ ) that gives the minimum value E ₃₀ of the combined correlation function E (p, q) obtained in step S171 in FIG. 76 is (0, 0), and the minimum value E ₃₀ is almost equal to equal to the offset value L _30.

As described above, the offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L ₃₀ are the expressions L ₃₃ ≦ L ₃₂ ≦ L ₃₁ ≦ L ₃₀ and the expressions L ₃₃ ≦ L _23. Since ≦ L ₁₃ ≦ L ₀₃ is satisfied, for the target block B ₃ , the minimum value E _{33 that} is substantially the offset value L ₃₃ is basically the minimum value E ₃₃ , E ₂₃ , E in step S172 of FIG. ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ are detected as the smallest one, and ( ₀ , 0) is obtained as the average motion vector (p ₀ , q ₀ ). Further, in step S172, (0,0, t ₀ ) is obtained as the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (3,3) is obtained as the kernel size (h, k). .

Further, the offset value L _33, as described above, since the smaller the threshold ε in step S173 (or less is), the minimum value E ₃₃ of substantially the offset value L ₃₃ is the minimum value E _33, E _23, E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ regarding the target block B ₃ detected as the smallest one in step S 174 of FIG. 77, the three-dimensional motion vector (p ₀ , q _0, t ₀₎ as (0, 0, with t ₀₎ is obtained, the kernel size (h, k) as the (3,3) is obtained.

This time t-ht ₀ from t + kt _0, that is, between time t-3t ₀ of t + 3t _0, the block of interest B ₃ is (0,0, t ₀₎ of the speed (time t ₀ It is shown that it is moving at a speed of (0,0) moving in the spatial direction.

Accordingly, the kernel size (h, k) is determined from the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and thus the target block obtained from the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ). It can be said that it represents the effective range of the principal component direction.

Then, for example, in the moving image data shown in FIG. 70, when the block B ₁₀ is the target block, as described above, the time _{t + t 0, t + 2t} 0, t + 3t 0, tt 0, in each image data in the _{t-2t 0, t-3t} 0, region B ₁₀₂ corresponding to a target block _{B 3, B 202, B 302} , B -102, B -202, is B _-302 exist, time t + The _value of the correlation information E ₁ (u ₁ , v ₁ ) at t ₀ is the correlation information E ₂ (u ₂ , u at time t + 2t ₀ when the position (u ₁ , v ₁ ) is (g, 0). v is the value of _2), the value of the position (u _2, v ₂₎ is (2 g, 0 when), correlation information at time _{t + 3t 0 E 3 (u} 3, v 3) , the position (u ₃ , v ₃ ) is (3g, 0), the _value of correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ is when position (r ₁ , s ₁ ) is (-g, 0) Further, the _value of the correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ is the correlation information at time t-3t ₀ when the position (r ₂ , s ₂ ) is (-2g, 0). the value of _{F 3 (r 3, s 3} ) , the position _{(r 3, s 3),} - when the (3 g, 0) , Each becomes a minimum, it is substantially zero.

In this case, the position (p ₃₃ , q ₃₃ ) that gives the minimum value E ₃₃ of the combined correlation function E (p, q) obtained in step S159 of FIG. 74 is (g, 0), and the minimum value E ₃₃ is approximately equal to the offset value L _33. Similarly, the position (p ₂₃ , q ₂₃ ) that gives the minimum value E ₂₃ of the combined correlation function E (p, q) obtained in step S161 in FIG. 75 is (g, 0), and the minimum value E ₂₃ is approximately equal to the offset value L _23. In addition, the position (p ₁₃ , q ₁₃ ) that gives the minimum value E ₁₃ of the combined correlation function E (p, q) obtained in step S163 in FIG. 75 is (g, 0), and the minimum value E ₁₃ is almost equal to equal to the offset value L _13. Further, the position (p ₀₃ , q ₀₃ ) that gives the minimum value E ₀₃ of the combined correlation function E (p, q), which is obtained in step S165 of FIG. 75, is (g, 0), and the minimum value E ₀₃ is almost equal to equal to the offset value L _03. In addition, the position (p ₃₂ , q ₃₂ ) that gives the minimum value E ₃₂ of the combined correlation function E (p, q) obtained in step S167 in FIG. 76 is (g, 0), and the minimum value E ₃₂ is almost equal to equal to the offset value L _32. Further, the position (p ₃₁ , q ₃₁ ) that gives the minimum value E ₃₁ of the combined correlation function E (p, q) obtained in step S169 in FIG. 76 is (g, 0), and the minimum value E ₃₁ is almost equal to equal to the offset value L _31. In addition, the position (p ₃₀ , q ₃₀ ) that gives the minimum value E ₃₀ of the combined correlation function E (p, q) obtained in step S171 of FIG. 76 is (g, 0), and the minimum value E ₃₀ is almost equal to the offset value L _30.

As described above, the offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L ₃₀ are the expressions L ₃₃ ≦ L ₃₂ ≦ L ₃₁ ≦ L ₃₀ and the expressions L ₃₃ ≦ L _23. Since ≦ L ₁₃ ≦ L ₀₃ is satisfied, for the target block B ₁₀ , the minimum value E _{33 that} is substantially the offset value L ₃₃ is basically the minimum value E ₃₃ , E ₂₃ , E in step S172 of FIG. ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ are detected as the smallest one, and (g, 0) is obtained as the average motion vector (p ₀ , q ₀ ). In step S172, (g, 0, t ₀ ) is obtained as a three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (3, 3) is obtained as a kernel size (h, k). .

Further, as described above, since the offset value L ₃₃ is smaller than the threshold value ε in step S173, the minimum value E _{33 that} is substantially the offset value L ₃₃ becomes the minimum value E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E _With respect to the target block B ₁₀ detected as the smallest of ₃₂ , E ₃₁ , and E ₃₀ , in step S174 in FIG. 77, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) serving as filter information is obtained. (G, 0, t ₀ ) is obtained, and (3, 3) is obtained as the kernel size (h, k).

This time t-ht ₀ from t + kt _0, that is, between time t-3t ₀ of t + 3t _0, the block of interest B ₁₀ is (g, 0, t ₀₎ of the speed (time t ₀ It is shown that it is moving at a speed (moving by (g, 0)) in the spatial direction.

Next, for example, in the moving image data shown in FIG. 71, when the block B ₇ is the target block, as described above, the times t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , in each image data in the t-2t _0, region B _111, B ₂₁₁ corresponding to a target block _{B 7, B 311, B -111} , and B _-211 is present, the correlation information at time t + t ₀ E ₁ (u ₁ , v ₁ ) is the position (u ₁ , v ₁ ) is (0,0) and the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is the position ( When u ₂ , v ₂ ) is (0,0), the _value of the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is the position (u ₃ , v ₃ ) at (0,0 ), The _value of the correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ is the correlation information at time t-2t ₀ when the position (r ₁ , s ₁ ) is (0,0). The value of F ₂ (r ₂ , s ₂ ) is minimized when the position (r ₂ , s ₂ ) is (0, 0), and is almost zero.

However, in the image data at time t-3t _0, a region corresponding to the block of interest B _7, which may originally, but it should be region B _-311 at the same position as the block of interest B _7, a moving subject P ₆₉₀₂ It is hidden behind and does not exist. Therefore, the _value of the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ is minimized when the position (r ₃ , s ₃ ) is (0,0). The value of the correlation information F ₃ (r ₃ , s ₃ ) is not always the minimum when the position (r ₃ , s ₃ ) is (0,0). That is, the value of the correlation information F ₃ (r ₃ , s ₃ ) is minimum at a certain position (r ₃ , s ₃ ) that is not necessarily (0, 0). Become.

In this case, the minimum value E _{33 of the} combined correlation function E (p, q) obtained using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ in step S158 in FIG. The position (p ₃₃ , q ₃₃ ) that gives is a position that is not necessarily (0, 0), and the minimum value E ₃₃ is considerably larger than the offset value L ₃₃ . Similarly, the minimum value E _{32 of the} combined correlation function E (p, q) obtained using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ in step S166 of FIG. The position (p ₃₂ , q ₃₂ ) that gives is a position that is not necessarily (0, 0), and the minimum value E ₃₂ is considerably larger than the offset value L ₃₂ . Further, in step S168 of FIG. 76, the minimum value E ₃₁ is also set for the combined correlation function E (p, q) obtained using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ as well. The given position (p ₃₁ , q ₃₁ ) is a position that is not necessarily (0,0), and the minimum value E ₃₁ is considerably larger than the offset value L ₃₁ . Further, in step S170 of FIG. 76, the time t-3t ₀ correlation information F ₃ in (r _3, s ₃₎ is also a synthetic correlation function E obtained (p, q) for even the minimum value E ₃₀ The given position (p ₃₀ , q ₃₀ ) is a certain position that is not necessarily (0, 0), and the minimum value E ₃₀ is considerably larger than the offset value L ₃₀ .

On the other hand, in step S160 of FIG. 75, for a time t-3t correlation in ₀ information _{F 3 (r 3, s 3} ) determined without using the synthesized correlation function E (p, q), the minimum value E ₂₃ The given position (p ₂₃ , q ₂₃ ) is (0, 0), and the minimum value E ₂₃ is substantially equal to the offset value L ₂₃ . Similarly, the minimum value E _{13 of the} combined correlation function E (p, q) obtained without using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ in step S162 in FIG. The position (p ₁₃ , q ₁₃ ) that gives is (0,0), and the minimum value E ₁₃ is substantially equal to the offset value L ₁₃ . Further, in step S164 of FIG. 75, the minimum value _E03 is also set for the combined correlation function E (p, q) obtained without using the correlation information F ₃ (r ₃ , s ₃ ) at time t-3t ₀ . The given position (p ₀₃ , q ₀₃ ) is (0, 0), and the minimum value E ₀₃ is substantially equal to the offset value L ₀₃ .

Since the offset values L ₂₃ , L ₁₃ , and L ₀₃ satisfy the formula L ₂₃ ≦ L ₁₃ ≦ L ₀₃ from the above description, the target block B ₇ is basically substantially offset in step S172 of FIG. The minimum value E ₂₃ having the value L ₂₃ is detected as the minimum of the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ and the average motion vector (p ₀ , (0,0) is obtained as q ₀ ). Further, in step S172, (0,0, t ₀ ) is obtained as the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (2,3) is obtained as the kernel size (h, k). .

Further, the offset value L _23, as described above, is smaller than the threshold value ε in step S173, the minimum value E _23, which is substantially offset value L ₂₃ is the minimum value _{E 33, E 23, E 13} , E 03, E _With respect to the target block B ₇ detected as the smallest of ₃₂ , E ₃₁ , and E _30, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) that becomes the filter information in step S174 of FIG. (0,0, t ₀ ) is obtained, and (2,3) is obtained as the kernel size (h, k).

This is because the target block B ₇ moves at a speed of (0,0, t ₀ ) from time t-ht ₀ to t + kt ₀ , that is, from time t-2t ₀ to t + 3t _0. It shows that there is.

Then, for example, in the moving image data shown in FIG. 71, when the block B ₁₄ is the target block, as described above, at time _{t + t 0, tt 0,} t-2t 0, t-3t 0 In each image data, there are regions B ₁₁₂ , B ₋₁₁₂ , B ₋₂₁₂ , and B ₋₃₁₂ corresponding to the block of interest B _14, and the _value of the correlation information E ₁ (u ₁ , v ₁ ) at time t + t ₀ When the position (u ₁ , v ₁ ) is (0,0), the _value of the correlation information F ₁ (r ₁ , s ₁ ) at time tt ₀ is the position (r ₁ , s ₁ ) is (0 , 0), the _value of the correlation information F ₂ (r ₂ , s ₂ ) at time t-2t ₀ is the time t-3t when the position (r ₂ , s ₂ ) is (0,0). _{The value} of the correlation information F ₃ (r ₃ , s ₃ ) at 0 is minimum and almost 0 when the position (r ₃ , s ₃ ) is (0,0).

However, in the image data at time t + 2t _0, a region corresponding to the block of interest B ₁₄ is originally, but it should be region B ₂₁₂ of the block of interest B ₁₄ at the same position, moving the subject P ₆₉₀₂ are It is hidden behind and does not exist. For this reason, the _value of the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is minimized when the position (u ₂ , v ₂ ) is (0,0). The value of the correlation information E ₂ (u ₂ , v ₂ ) is not always the minimum when the position (u ₂ , v ₂ ) is (0,0). That is, the value of the correlation information E ₂ (u ₂ , v ₂ ) is minimum at a certain position (u ₂ , v ₂ ) that is not necessarily (0,0). Become.

Further, in the image data at time t + 3t _0, a region corresponding to the block of interest B ₁₄ is originally but should a region B ₃₁₂ of the block of interest B ₁₄ at the same position, moving the subject P ₆₉₀₂ are It is hidden behind and does not exist. Therefore, the _value of the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is minimized when the position (u ₃ , v ₃ ) is (0,0). The value of the correlation information E ₃ (u ₃ , v ₃ ) is not always the minimum when the position (u ₃ , v ₃ ) is (0,0). That is, the value of the correlation information E ₃ (u ₃ , v ₃ ) is minimum at a certain position (u ₃ , v ₃ ) that is not necessarily (0,0). Become.

In this case, in step S158 in FIG. 74, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also obtained. For the resultant correlation function E (p, q), the position (p ₃₃ , q ₃₃ ) giving the minimum value E ₃₃ is not necessarily (0,0), and the minimum value E ₃₃ is offset. The value is much larger than the value L ₃₃ Similarly, in step S160 in FIG. 75, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also used. The position (p ₂₃ , q ₂₃ ) that gives the minimum value E ₂₃ is also a position that is not necessarily (0,0), and the minimum value E ₂₃ is an offset value. It becomes a much larger value than L _23. 75, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also obtained. Also for the composite correlation function E (p, q), the position (p ₁₃ , q ₁₃ ) giving the minimum value E ₁₃ is not always (0,0), and the minimum value E ₁₃ is an offset value. It becomes a much larger value than L _13. Further, in step S164 of FIG. 75, the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are also obtained. Also for the composite correlation function E (p, q), the position (p ₀₃ , q ₀₃ ) giving the minimum value E ₀₃ is not always (0,0), and the minimum value E ₀₃ is an offset value. It is considerably larger than L ₀₃ .

In step S166 of FIG. 76, the correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is not used, but the correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ is used. The position (p ₃₂ , q ₃₂ ) that gives the minimum value E ₃₂ of the composite correlation function E (p, q) obtained by using the position is not always (0,0), and the minimum value E ₃₂ becomes considerably larger than the offset value L _32.

On the other hand, in step S168 in FIG. 76, neither correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ nor correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ is used. The position (p ₃₁ , q ₃₁ ) giving the minimum value E ₃₁ is (0,0), and the minimum value E ₃₁ is almost equal to the offset value L ₃₁ . Will be equal. Similarly, in step S170 of FIG. 76, both correlation information E ₂ (u ₂ , v ₂ ) at time t + 2t ₀ and correlation information E ₃ (u ₃ , v ₃ ) at time t + 3t ₀ are used. The position (p ₃₀ , q ₃₀ ) that gives the minimum value E ₃₀ of the composite correlation function E (p, q) obtained without any change is (0, 0), and the minimum value E ₃₀ is almost equal to the offset value L _30. Is equal to

Since the offset values L ₃₁ and L ₃₀ satisfy the expression L ₃₁ ≦ L ₃₀ from the above description, the block B ₁₄ of interest is basically the minimum that substantially becomes the offset value L ₃₁ in step S172 of FIG. value E ₃₁ is detected as the smallest ones within the minimum _{E 33, E 23, E 13} , E 03, E 32, E 31, E 30, average motion vector (p _0, q ₀₎ as (0 , 0) is required. Further, in step S172, (0,0, t ₀ ) is obtained as the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ), and (3,1) is obtained as the kernel size (h, k). .

Further, the offset value L _31, as described above, is smaller than the threshold value ε in step S173, the minimum value E _31, which is substantially offset value L ₃₁ is the minimum value _{E 33, E 23, E 13} , E 03, E For the block of interest B ₁₄ detected as the smallest of ₃₂ , E ₃₁ , and E ₃₀ , in step S174 of FIG. 77, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) serving as filter information is obtained. (0,0, t ₀ ) is obtained, and (3,1) is obtained as the kernel size (h, k).

This is because the target block B ₁₄ moves at a speed of (0,0, t ₀ ) from time t-ht ₀ to t + kt ₀ , that is, from time t-3t ₀ to t + t _0. It shows that there is.

Next, for example, in the moving image data shown in FIG. 71, when the block B ₉ is the target block, as described above, the target block B ₉ moves at a position where the subject P ₆₉₀₁ is stationary. Since the stationary subject P ₆₉₀₂ hides the stationary subject P ₆₉₀₁ , the time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t−2t ₀ , t− There is no region corresponding to the target block B _{9 in} any of the image data at 3t ₀ .

For this reason, the correlation information E ₁ (u ₁ , v ₁ ), E ₂ (u ₂ , t at time t + t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t ₀ v ₂ ), E ₃ (u ₃ , v ₃ ), F ₁ (r ₁ , s ₁ ), F ₂ (r ₂ , s ₂ ), F ₃ (r ₃ , s ₃ ) The position (u ₁ , v ₁ ), (u ₂ ) of the image data at t ₀ , t + 2t ₀ , t + 3t ₀ , tt ₀ , t-2t ₀ , t-3t _{0 that} is most similar to the target block B ₉ , v ₂ ), (u ₃ , v ₃ ), (r ₁ , s ₁ ), (r ₂ , s ₂ ), (r ₃ , s ₃ ) are the smallest, but the value is somewhat large Value.

In this case, for the combined correlation function E (p, q) obtained in step S158 of FIG. 74, the position (p ₃₃ , q ₃₃ ) giving the minimum value E ₃₃ is a certain position, and the minimum value E ₃₃ is It becomes considerably larger than the offset value L _33. Similarly, for the combined correlation function E (p, q) obtained in step S160 of FIG. 75, the position (p ₂₃ , q ₂₃ ) giving the minimum value E ₂₃ is a certain position, and the minimum value E ₂₃ is an offset. It becomes a much larger value than the value L _23. Also, with respect to the composite correlation function E (p, q) obtained in step S162 in FIG. 75, the position (p ₁₃ , q ₁₃ ) giving the minimum value E ₁₃ is a certain position, and the minimum value E ₁₃ is the offset value. It becomes a much larger value than L _13. Further, with respect to the composite correlation function E (p, q) obtained in step S164 of FIG. 75, the position (p ₀₃ , q ₀₃ ) giving the minimum value E ₀₃ is a certain position, and the minimum value E ₀₃ is an offset value. It is considerably larger than L ₀₃ . Also, for the combined correlation function E (p, q) obtained in step S166 of FIG. 76, the position (p ₃₂ , q ₃₂ ) giving the minimum value E ₃₂ is a certain position, and the minimum value E ₃₂ is the offset value. It becomes considerably to a value greater than L _32. Further, with respect to the composite correlation function E (p, q) obtained in step S168 of FIG. 76, the position (p ₃₁ , q ₃₁ ) giving the minimum value E ₃₁ is a certain position, and the minimum value E ₃₁ is an offset value. Much larger than L ₃₁ . Also, for the combined correlation function E (p, q) obtained in step S170 of FIG. 76, the position (p ₃₀ , q ₃₀ ) giving the minimum value E ₃₀ is a certain position, and the minimum value E ₃₀ is the offset value. It becomes considerably to a value greater than L _30.

Accordingly, the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , and E ₃₀ are all large values, and any of them is selected as the minimum in step S172 of FIG. even if they are, larger than the threshold ε in step S173 (becomes higher), for the block of interest B _9, in step S175 of FIG. 77, three-dimensional motion vectors, which are filter information _{(p 0, q 0, t} 0 ) Is obtained as (p ₀ ′, q ₀ ′, t ₀ ), and (0,0) is obtained as the kernel size (h, k).

This means that the target block B ₉ is moving at a speed of (p ₀ ′, q ₀ ′, t ₀ ) only from time t-ht ₀ to t + kt ₀ , that is, at the timing of time t. Is shown. In other words, the kernel size (h, k) being (0, 0) means that the area corresponding to the target block B ₉ does not exist in the image data at any other time.

Note that (p ₀ ', q ₀ ') is a position (p, q) that gives the smallest one of the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E _30. To express. Further, (p, q) when the kernel size (h, k) is (0,0) is dummy data and is not used.

As described above, according to the processing according to the flowcharts shown in FIGS. 73 to 77, the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) as the filter information and the kernel size ( h, k) is obtained, and as a result, for the block of interest, the block of interest is moving at a speed of (p ₀ , q ₀ , t ₀ ) from time t-ht ₀ to t + kt ₀ Can be obtained.

The offset values L ₃₃ , L ₂₃ , L ₁₃ , L ₀₃ , L ₃₂ , L ₃₁ , and L ₃₀ are the expressions L ₃₃ ≦ L ₃₂ ≦ L ₃₁ ≦ L ₃₀ and the expressions L ₃₃ ≦ L ₂₃ ≦ L _13. By satisfying ≦ L ₀₃ , basically, in step S 172 of FIG. 77, h or k of the minimum values E ₃₃ , E ₂₃ , E ₁₃ , E ₀₃ , E ₃₂ , E ₃₁ , E ₃₀ A large minimum value E _hk is easily selected. As a result, even if there is a slight difference in pixel values between the block of interest and the region corresponding to the block of interest in another frame, the block of interest will be (p ₀ , q ₀ , t ₀ ) can be prevented from underestimating the range (time) of moving at a speed of (t ₀ ).

Next, the filter generation unit 23 in which the principal component direction acquisition unit 31 of FIG. 25 is configured as shown in FIG. 72 performs the processing according to FIGS. When the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) and the kernel size (h, k) as filter information are supplied from the information supply unit 32) to the filter unit 22, the filter unit 22 Then, the process according to the flowchart shown in FIG. 78 is performed as the process of step S3 of FIG.

That is, in step S191, the filter unit 22 receives the three-dimensional motion vector (p ₀ , q ₀ , t ₀ ) and the kernel size (h, k) supplied as filter information from the filter information supply unit 32. The process proceeds to step S192.

In step S192, the filter unit 22 calculates the pixel value C (t, x, y) of the pixel at the position (x, y) in the target block of the frame at time t, for example, from time t-3t _{0 to} t + 3t. pixel values of the pixels in the _{0 D (t-3t 0,} x-3p 0, y-3q 0), D (t-2t 0, x-2p 0, y-2q 0), D (tt 0, xp 0, yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D Of (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), filtering is performed by calculation using pixel values in a time (frame) range corresponding to the kernel size (h, k). Note that D (t, x, y) represents a pixel value at a position (x, y) at time t of moving image data used for filtering.

That is, in step S192, the filter unit 22 performs the filtering, that is, the calculation of the pixel value C (t, x, y) of the pixel at the position (x, y) in the target block according to the kernel size (h, k). This is performed using only the pixel values of the frame within the effective range of the motion vector (p ₀ , q ₀ , t ₀ ) represented.

Specifically, in step S192, when the kernel size (h, k) is (3, 3), the filter unit 22 determines the pixel value D (t−t) of the pixels at times t−3t _{0 to} t + 3t ₀ . 3t ₀ , x-3p ₀ , y-3q ₀ ), D (t-2t ₀ , x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (1). In this case, in the same manner as in step S132 of FIG. 68, filtering is performed using a filter that extends in the principal component direction of the block of interest and has a region in the T direction with a width of 2π / (4t ₀ ).

・・・（１）

... (1)

In addition, when the kernel size (h, k) is (2,3), the filter unit 22 has a pixel value D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) at time t-3t ₀ . ) Without using the pixel values D (t-2t ₀ , x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ) at times t-2t _{0 to} t + 3t ₀ , D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (2).

・・・（２）

... (2)

Further, when the kernel size (h, k) is (1,3), the filter unit 22 outputs the pixel value D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) of the pixel at time t-3t ₀ . ), and time t-2t pixel value of the pixel at _{0 D (t-2t 0,} x-2p 0, y-2q 0) is not used, the time tt ₀ to t + 3t pixel value of the pixel at ₀ D ( tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, according to equation (3), the pixel value C (t, x, y) of the block of interest is calculated Do.

・・・（３）

... (3)

In addition, when the kernel size (h, k) is (0,3), the filter unit 22 has a pixel value D (t-3t ₀ , x-3p ₀ , y-3q ₀ ) at time t-3t ₀ . ), Pixel value D (t-2t ₀ , x-2p ₀ , y-2q ₀ ) of the pixel at time t-2t ₀ , and pixel value D (tt ₀ , xp ₀ , yq ₀ ) of the pixel at time tt ₀ Is not used, and the pixel values D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ ) at times t to t + 3t ₀ are used. , x + 2p ₀ , y + 2q ₀ ), D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ), for example, according to equation (4), the pixel value C (t, Calculate x, y).

・・・（４）

... (4)

Similarly, when the kernel size (h, k) is (3, 2), the filter unit 22 uses the pixel value D (t + 3t ₀ , x + 3p ₀ , y + 3q) at the time t + 3t ₀ . ₀ ) is not used, and pixel values D (t-3t ₀ , x-3p ₀ , y-3q ₀ ), D (t-2t ₀ , x-2p ₀ ) at times t-3t _{0 to} t + 2t _{0 0} , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ), for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (5).

・・・（５）

... (5)

Further, when the kernel size (h, k) is (3, 1), the filter unit 22 uses the pixel value D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ) at time t + 3t ₀ . ), And the pixel value D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) of the pixel at time t + 2t ₀ , and the pixel value of the pixel at time t-3t _{0 to} t + t ₀ D (t-3t ₀ , x-3p ₀ , y-3q ₀ ), D (t-2t ₀ , x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D ( t, x, y), D (t + t ₀ , x + p ₀ , y + q ₀ ), for example, according to equation (6), the pixel value C (t, x, y) of the block of interest Perform the operation.

・・・（６）

... (6)

In addition, when the kernel size (h, k) is (3,0), the filter unit 22 has a pixel value D (t + 3t ₀ , x + 3p ₀ , y + 3q ₀ ) at time t + 3t ₀ . ), The pixel value D (t + 2t ₀ , x + 2p ₀ , y + 2q ₀ ) of the pixel at time t + 2t ₀ , and the pixel value D (t + t ₀ , x + of the pixel at time t + t ₀ p _0, y + q ₀₎ without the pixel value D of the pixel at time t-3t ₀ to _{t (t-3t 0, x} -3p 0, y-3q 0), D (t-2t 0, x-2p ₀ , y-2q ₀ ), D (tt ₀ , xp ₀ , yq ₀ ), D (t, x, y), for example, according to the equation (7), the pixel value C ( t, x, y) is calculated.

・・・（７）

... (7)

On the other hand, when the kernel size (h, k) is (0,0), the same area (image area) as the target block (area corresponding to the target block) is set to other times t-3t ₀ , t- 2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , and t + 3t ₀ are not present in this case. In this case, temporal filtering results in degradation of image quality. (Frame of interest), that is, using only the pixel value D (t, x, y) of the pixel at time t, for example, the pixel value C (t, x, y) of the block of interest is calculated according to equation (8). Do. Equation (8) is equivalent to not performing filtering.

・・・（８）

... (8)

Here, as a case where the region corresponding to the block of interest does not exist in the images at other times t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ , For example, when the moving speed of the subject projected in the block of interest is so high that humans cannot recognize it, or the subject projected in the block of interest is in the frame at time t (frame of interest). There are cases where it only appears suddenly.

Finding the minimum value of correlation information or composite correlation information is equivalent to detecting a motion vector by so-called block matching, but the subject projected in the target block exceeds the search range in block matching. Even if it has moved, the area corresponding to the block of interest is displayed in the images at other times t-3t ₀ , t-2t ₀ , tt ₀ , t + t ₀ , t + 2t ₀ , t + 3t ₀ . Applicable when it does not exist.

  In step S192, the filtering as the calculation of the pixel value C (t, x, y) is performed while down-sampling that thins out the number of samples in the time direction to 1/4, that is, the ratio of one frame every four frames. Done in

  After the process of step S192, the process proceeds to step S193, and the filter unit 22 outputs the pixel value C that is the filtering result, that is, the filtering result in step S192 (the calculation result of any one of the expressions (1) to (8)). (t, x, y) is output to the encoding unit 24 as a pixel value at the position (x, y) at time t.

As described above, in the second motion detection process (the processes in steps S151 to S172 of the processes in steps S151 to S176 in FIGS. 73 to 77), the filter passes as in the first motion detection process. By using multiple pieces of correlation information representing the correlation with each of multiple frames corresponding to the width in the T direction of the band, 2π / (4t ₀ ), the average movement from the target block to multiple frames can be said. The average motion vector that represents the average motion of the target block is calculated because the correlation information is not used for the frame in which the region corresponding to the target block does not exist. Furthermore, an accurate principal component direction can be obtained for the block of interest.

  In the second motion detection process, the kernel size (h, k) is output, and only the pixels of the frame in the range corresponding to the kernel size (h, k) are used for filtering in the filter unit 22. In other words, since the pixel of the frame in which the region corresponding to the block of interest does not exist is not used, appropriate data of only frequency components that can be recognized by human vision can be obtained.

Then, in the image processing system of FIG. 24, when the display device 3 has a frame rate of 1 / t ₀ is capable of displaying a video frame rate of 1 / t ₀ where the receiving apparatus 2 outputs , the display device 3, 1 / if t has a _non-zero frame rate, it is difficult to view the video frame rate of 1 / t ₀ where the receiving apparatus 2 outputs.

  Therefore, FIG. 79 shows a configuration example of the transmission apparatus 1 of FIG. 24 that outputs encoded data that can display a moving image corresponding to the frame rate of the display apparatus 3.

  79, parts corresponding to those in FIG. 25 are denoted by the same reference numerals, and description thereof will be omitted as appropriate. That is, the transmission apparatus 1 in FIG. 79 is configured in the same manner as in FIG. 25 except that the filter unit 202 is provided instead of the filter unit 22.

The filter unit 202 is an area extending in the principal component direction of each part (each block) of moving image data having a frame rate of 1 / t _{0 in} the frequency domain and having a width of 2π / (4t ₀ ) in the T direction. As the passband, the video with frame rate 1 / t ₀ is filtered, and the video with frame rate 1 / (4t ₀ ) is divided into multiple frequency components by downsampling 1/4 in the time direction. Data is obtained and output to the encoding unit 24. Therefore, the filtering in the filter unit 202 is that the moving image data divided into a plurality of frequency components is obtained, and the moving image data not divided into the plurality of frequency components is obtained in the filter unit 22 of FIG. Different from filtering.

  Next, the details of filtering in the filter unit 202 will be described. Before that, the display capability of the display device 3 when the display device 3 has a certain frame rate will be described.

FIG. 80 shows a region R as a display capability when the display device 3 has a frame rate of 1 / t ₀ and can display one pixel per pixel at intervals of r _{0 in} the spatial direction. _{Represents 8001} .

Region R ₈₀₀₁ is a rectangular parallelepiped region in which the X and Y directions are in the range of −π / r ₀ to + π / r ₀ and the T direction is in the range of −π / t ₀ to + π / t ₀ on the frequency domain. It is. When the display device 3 has a frame rate of 1 / t ₀ and can display one pixel per pixel at intervals of r ₀ , the frequency component data in the region R ₈₀₀₁ is stored. Can be displayed.

Figure 81 is a display device 3, for example, a frame rate of _{1 / (4t 0/3)} , and display capability if one pixel 1 pixel can be displayed at intervals of r ₀ in the spatial direction As a region R ₈₁₀₁ .

Region R _8101, in a frequency domain, X, in the range in the Y direction, respectively - [pi] / r ₀ from the + π / r _0, T direction -π / (4t _0/3) from + π / (4t _0/3 ) Is a rectangular parallelepiped region. The display device 3 has a frame rate of _{1 / (4t 0/3)} , and, in the case where one pixel 1 pixel can be displayed at intervals of r ₀ in the spatial direction, the frequency of the region R ₈₁₀₁ The component data can be displayed.

FIG. 82 shows the display capability when the display device 3 has a frame rate of 1 / (2t ₀ ), for example, and can display one pixel per pixel at intervals of r ₀ . Region R ₈₂₀₁ is represented.

The region R ₈₂₀₁ has, on the frequency domain, the X and Y directions are in the range of −π / r ₀ to + π / r ₀ and the T direction is in the range of −π / (2t ₀ ) to + π / (2t ₀ ). It is a rectangular parallelepiped area. When the display device 3 has a frame rate of 1 / (2t ₀ ) and can display one pixel per pixel at intervals of r ₀ , the frequency components in the region R ₈₂₀₁ Data can be displayed.

FIG. 83 shows the display capability when the display device 3 has a frame rate of 1 / (4t ₀ ) and can display one pixel per pixel at intervals of r _{0 in} the spatial direction. Region R ₈₃₀₁ is represented.

The region R ₈₃₀₁ has, on the frequency domain, the X and Y directions are in the range of −π / r ₀ to + π / r ₀ , and the T direction is in the range of −π / (4t ₀ ) to + π / (4t ₀ ). It is a rectangular parallelepiped area. When the display device 3 has a frame rate of 1 / (4t ₀ ) and can display one pixel per pixel at intervals of r ₀ , the frequency component in the region R ₈₃₀₁ Data can be displayed.

79 filters the moving image data of the frame rate 1 / t ₀ so that the display device 3 (FIG. 24) can display any of the displays described in FIG. 80 to FIG. 83, for example. Even if you have the capability, the video with the frame rate 1 / (4t ₀ ) divided appropriately (optimally) into multiple frequency components so that you can perform optimal display within the range of the display capability Get the data.

By the way, as described above, the human recognizes only the data in the region R ₁₀₀₁ shown in FIG. 20 for the portion where the stationary subject is projected in the moving image. Further, regarding a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ ) / 2”, the human being has the data in the region R ₁₁₀₁ shown in FIG. Only recognize. Furthermore, for a portion of the moving image where the subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ”, a human recognizes only the data in the region R ₁₂₀₁ shown in FIG. Further, for a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, a human recognizes only data in the region R ₁₃₀₁ shown in FIG. The same applies to moving subjects moving at other speeds in moving images.

Accordingly, when displaying a moving image, depending on the moving speed of the subject in the moving image, for example, data in the region R ₁₀₀₁ in FIG. 20, data in the region R ₁₁₀₁ in FIG. 21, and data in the region R ₁₂₀₁ in FIG. If there is data, data in the region R ₁₃₀₁ shown in FIG. 23, humans do not recognize the degradation of the image quality of the moving image, so it is sufficient that there is data in such a region.

On the other hand, when the display device 3 has a display capability capable of displaying the frequency component data in the region R _{8001 of} FIG. 80, the (frequency component) data outside the region R ₈₀₀₁ is unnecessary. It is sufficient if there is data in region R ₈₀₀₁ .

Similarly, the display device 3, when having a display capability may display data of the frequency components in the region R ₈₁₀₁ in FIG. 81, data outside the region R ₈₁₀₁ is unnecessary, the area R ₈₁₀₁ Is sufficient.

Further, the display device 3, when having a display capability may display data of the frequency components in the region R ₈₂₀₁ in FIG. 82, data outside the region R ₈₂₀₁ is unnecessary, the area R ₈₂₀₁ of the Data is enough.

Further, in the case where the display device 3 has a display capability capable of displaying the frequency component data in the region R ₈₃₀₁ in FIG. 83, the data outside the region ₈₃₀₁ is unnecessary, and the data in the region ₈₃₀₁ is If there is enough.

Therefore, the filter unit 202 in FIG. 79 performs the same filtering as the filter unit 22 in FIG. 25 in order to deal with the display device 3 having the display capabilities described in FIGS. Frequency component in region R ₈₃₀₁ , frequency component outside region R _{8301 in} FIG. 83 and within region R ₈₂₀₁ in FIG. 82, frequency component outside region R _{8201 in} FIG. 82 and within R ₈₁₀₁ in FIG. Further, filtering is performed to divide the data into four frequency component data of the frequency components outside the R _{8101 in} FIG. 81 and in the region R ₈₀₀₁ in FIG.

Here, the frequency component data in the region R ₈₃₀₁ in FIG. 83 among the above-described four frequency component data will be referred to as “low-frequency component moving image data”. Also, the frequency component data outside the region R _{8301 in} FIG. 83 and within the region R ₈₂₀₁ in FIG. 82 is referred to as “middle low frequency component moving image data”, and outside the region R ₈₂₀₁ in FIG. The frequency component data in the region R ₈₁₀₁ is referred to as “middle and high frequency component moving image data”. Further, the frequency component data outside the region R _{8101 in} FIG. 81 and within the region R ₈₀₀₁ in FIG. 80 is referred to as “high frequency component moving image data”.

  Next, “low frequency component video data”, “middle low frequency component video data”, “middle and high frequency component video data”, and “high frequency component video data” in each part of the video data will be described.

For the portion of the moving image where a stationary subject is projected, it is sufficient if there is data in the region R ₁₀₀₁ in FIG. The area R ₁₀₀₁ in FIG. 20 is equal to the area R ₈₃₀₁ in FIG. 83, and therefore the data of the area R _{1001 in} FIG. 20 does not exist outside the area R _{8301 in} FIG. Therefore, for the portion of the moving image where the stationary subject is projected, all of the data in the region R _{1001 in} FIG. 20 is “low-frequency component moving image data”, and “middle-low-frequency component moving image data”. , “Middle and high frequency component moving image data” and “high frequency component moving image data” do not exist (is 0).

Next, with respect to a portion of the moving image where the subject projected on the moving image is moving at a speed of “(r ₀ / t ₀ ) / 2”, it is sufficient that there is data in the region R ₁₁₀₁ in FIG. It is. In addition, the region R _{1101 in} FIG. 21 has a T direction that protrudes from the region R _{8201 in} FIG. 82, but is included in the region R ₈₁₀₁ in FIG.

Accordingly, as shown in FIG. 84, the portion moving at a speed of “(r ₀ / t ₀ ) / 2” is within the region R ₈₃₀₁ in FIG. 83 of the data in the region R ₁₁₀₁ in FIG. The data of the region R ₈₄₀₁ that is the data is “low-frequency component moving image data”.

Also, as shown in FIG. 85, of the data in the region R ₁₁₀₁ in FIG. 21, the data in the region R ₈₅₀₁ that is outside the region R _{8301 in} FIG. 83 and is in the region R ₈₂₀₁ in FIG. “Middle and low range moving image data”.

Further, as shown in FIG. 86, the data in the region R ₈₆₀₁ which is the data outside the region R _{8201 in} FIG. 82 (and in the region R ₈₁₀₁ in FIG. 81) among the data in the region R ₁₁₀₁ in FIG. Is “middle and high frequency component moving image data”.

Note that the region R _{1101 in} FIG. 21 includes the direction T in the region R ₈₁₀₁ in FIG. 81, and therefore there is no data outside the region R _{8101 in} FIG. 81, so the speed “(r ₀ / t ₀ ) / There is no “high-frequency component moving image data” for a portion that moves at about “2”.

Next, with respect to a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ )”, it is sufficient if there is data in the region R ₁₂₀₁ in FIG. . In the region R _{1201 in} FIG. 22, the T direction exists up to the boundary of the region R ₈₀₀₁ in FIG.

Accordingly, as shown in FIG. 87, the portion moving at the speed “(r ₀ / t ₀ )” is the data in the region R ₈₃₀₁ in FIG. 83 among the data in the region R ₁₂₀₁ in FIG. The data of the region R ₈₇₀₁ is “low-frequency component moving image data”.

As shown in FIG. 88, the data in the region R ₈₈₀₁ outside the region R _{8301 in} FIG. 83 and the data in the region R ₈₂₀₁ in FIG. 82 out of the data in the region R ₁₂₀₁ in FIG. “Middle and low range moving image data”.

Further, as shown in FIG. 89, the data in the region R ₈₉₀₁ outside the region R _{8201 in} FIG. 82 and within the region R ₈₁₀₁ in FIG. 81 out of the data in the region R ₁₂₀₁ in FIG. “Middle and high frequency component moving image data”.

Also, as shown in FIG. 90, among the data in the region R ₁₂₀₁ in FIG. 22, the data in the region R ₉₀₀₁ that is the data outside the region R _{8101 in} FIG. 81 (and in the region R ₈₀₀₁ in FIG. 80). Is “high-frequency component moving image data”.

Next, with respect to the portion of the moving image where the subject projected on the moving image moves at a speed of “(2r ₀ / t ₀ )”, it is sufficient if there is data in the region R ₁₃₀₁ in FIG. . In the region R _{1301 in} FIG. 23, the T direction exists up to the boundary of the region R ₈₀₀₁ in FIG.

Accordingly, as shown in FIG. 91, the data in the region R ₈₃₀₁ in FIG. 83 among the data in the region R ₁₃₀₁ in FIG. 83 is shown for the portion moving at the speed “(2r ₀ / t ₀ )”. The data of the region R ₉₁₀₁ is “low-frequency component moving image data”.

Also, as shown in FIG. 92, the data in the area R ₉₂₀₁ outside the area R _{8301 in} FIG. 83 and the data in the area R ₈₂₀₁ in FIG. 82 out of the data in the area R ₁₃₀₁ in FIG. “Middle and low range moving image data”.

Further, as shown in FIG. 93, of the data in the region R ₁₃₀₁ in FIG. 23, the data in the region R ₉₃₀₁ outside the region R _{8201 in} FIG. 82 and in the region R ₈₁₀₁ in FIG. “Middle and high frequency component moving image data”.

Further, as shown in FIG. 94, the data in the region R ₉₄₀₁ which is the data outside the region R _{8101 in} FIG. 81 (and in the region R ₈₀₀₁ in FIG. 80) among the data in the region R ₁₃₀₁ in FIG. Is “high-frequency component moving image data”.

  Next, processing of the transmission apparatus 1 in FIG. 79 will be described with reference to the flowcharts in FIGS. 95 to 97.

The buffer unit 21 is supplied with moving image data having a high frame rate of 240 fps, for example, at a frame rate of 1 / t ₀ and sequentially stored.

  In steps S201 and S202, the same processing as in steps S1 and S2 of FIG. 26 is performed.

  That is, in step S201, the filter generation unit 23 reads out the moving image data stored in the buffer unit 21, inputs it to itself (filter generation unit 23), and proceeds to step S202.

In step S202, the filter generation unit 23 (the principal component direction acquisition unit 31 and the filter information supply unit 32) performs a human operation on the frequency domain for each part (for example, each block) of the moving image data input in step S201. The regions that can be recognized, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. The “processing for obtaining a pass band of necessary information” described with reference to FIG. 31 and the like described above with reference to FIG.

  Here, the pass band of the filter obtained in step S202 is hereinafter referred to as a basic band as appropriate.

In step S203, the filter unit 202 reads from the buffer unit 21 data used for filtering by the filter of the filter information supplied from the filter generation unit 23 in step S202, and the filter generation unit 23 performs step S202 on the data. _Downsampling that thins out the number of samples in the time direction to 1/4 while applying a filter that passes the frequency component of the basic band represented by the filter information supplied from and passes the frequency component of the region R _{8301 in} FIG. I do.

Then, the process proceeds from step S203 to S204, and the filter unit 202 supplies the data obtained by the filtering and downsampling in step S203, that is, the “low frequency component moving image data” of the frame rate 1 / (4t ₀ ) to the encoding unit 24. The process proceeds to step S205 in FIG.

In step S205, the filter unit 202 allows the data read from the buffer unit 21 in step S203 to pass the frequency component of the base band represented by the filter information supplied from the filter generation unit 23 in step S202, and FIG. Down-sampling is performed to reduce the number of samples in the time direction to 1/4 while applying a filter (band limiting filter) that passes the frequency component of region R ₈₂₀₁ of FIG. 83 but does not pass the frequency component of region R ₈₃₀₁ of FIG. .

Then, the process proceeds from step S205 to S206, and the filter unit 202 encodes the data obtained by the filtering and downsampling in step S205, that is, the “middle and low frequency range moving image data” having the frame rate 1 / (4t ₀ ). And the process proceeds to step S207.

In step S207, the filter unit 202 passes the frequency component of the base band represented by the filter information supplied from the filter generation unit 23 in step S202 to the data read from the buffer unit 21 in step S203. _Downsampling is performed to thin out the number of samples in the time direction to 1/4 while applying a filter that passes the frequency component of the region R ₈₁₀₁ of FIG. 82 but does not pass the frequency component of the region R ₈₂₀₁ of FIG.

Then, the process proceeds from step S207 to S208, and the filter unit 202 supplies the data obtained by the filtering and downsampling in step S207, that is, the “middle and high frequency component moving image data” of the frame rate 1 / (4t ₀ ) to the encoding unit 24. The process proceeds to step S209 in FIG.

In step S209, the filter unit 202 passes the frequency component of the base band represented by the filter information supplied from the filter generation unit 23 in step S202 to the data read from the buffer unit 21 in step S203. Down-sampling is performed to thin out the number of samples in the time direction to 1/4 while applying a filter that passes the frequency component of the region R ₈₀₀₁ of FIG. 81 but does not pass the frequency component of the region R ₈₁₀₁ of FIG.

Then, the process proceeds from step S209 to step S210. The filter unit 202 supplies the data obtained by the filtering and downsampling in step S209, that is, the “high frequency component moving image data” having the frame rate 1 / (4t ₀ ) to the encoding unit 24. Output.

In the encoding unit 24, the moving image data of the frame rate 1 / (4t ₀ ) divided into four frequency components from the filter unit 202, that is, “low frequency component moving image data”, “middle low frequency component moving image data”, Each of “middle and high frequency component moving image data” and “high frequency component moving image data” is encoded as necessary, further multiplexed, and output as encoded data. The encoded data is recorded on the recording medium 11 or transmitted via the transmission medium 12 as described with reference to FIG.

  Note that the processing in steps S202 to S210 is performed for all data of each part (each block) of the moving image stored in the buffer unit 21.

  Further, the downsampling in steps S203, S205, S207, and S209 is a process of outputting a filtering result of one frame for every four frames of moving image data stored in the buffer unit 21. Accordingly, the filter unit 202 may perform filtering for one frame out of the four frames of the moving image data stored in the buffer unit 21.

Further, in the encoding unit 24 of the transmission apparatus 1 in FIG. 79, as in the case in FIG. 25, for example, the region R in FIG. 20 represents the basic band as the filter information of the filter applied to the data of each part of the moving image. ₁₀₀₁ and, region R ₁₃₀₁ in FIG. 21, the region R ₁₃₀₁ in FIG. 22, and information of the region R ₁₃₀₁ in FIG. 23, the main component direction information of the region _{R 1001, R 1301, R 1301} , R 1301, encoded data Can be multiplexed and output.

As described above, regions that can be recognized by humans on the frequency domain, that is, for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. 22, region R _{1301 in} FIG. 80, a band limited by a region R _{8001 in} FIG. 80, a region R ₈₁₀₁ in FIG. 81, a region R ₈₂₀₁ in FIG. 82, and a region R ₈₃₀₁ in FIG. Since the moving image data is filtered and downsampling is performed, the moving image data can be reduced in consideration of the visual effect of the human eye. Further, the region R _{8001 in} FIG. 80 and the region in FIG. Moving image data appropriately divided for the display device 3 having the display capability represented by R ₈₁₀₁ , region R ₈₂₀₁ in FIG. 82, and region R _{8301 in} FIG. 83 can be obtained.

  Next, FIGS. 98 to 109 show “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, “ The distribution on the frequency domain of each “high frequency component moving image data” is shown.

First, for a portion of a moving image where a stationary subject is projected, a region R _{1001 in} FIG. 20 is a basic band. As described above, all of the data in the region R _{1001 in} FIG. 20 is “low frequency component video data”, “middle low frequency component video data”, “middle high frequency component video data”, and None of the “high frequency component moving image data” exists. That is, the moving image data after filtering is only “low-frequency component moving image data” in the region R ₁₀₀₁ .

When the “low-range component video data” that is the data of region R ₁₀₀₁ is downsampled to 1/4 in the time direction, the data after downsampling is sampled at r ₀ intervals in the spatial direction x, y In the time direction t, the data is sampled at 4 × t ₀ intervals.

For this reason, as shown in FIG. 98, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / (4t ₀ ) in the T direction. Occurs.

However, since the region R ₁₀₀₁ has been described with reference to FIG. 10, the region R ₁₀₀₁ is a rectangular (cuboid) region centered at the origin, in which the X and Y directions are 2π / r ₀ and the T direction is 2π / (4t ₀ ). Therefore, the folding components do not overlap.

  In FIG. 98, the shaded portion is the portion where the “low-frequency component moving image data” after down-sampling exists.

Next, for a portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2”, a region R _{1101 in} FIG. 21 is a basic band. As for the data in the region R _{1101 in} FIG. 21, as described above, the data in the region R _{8401 in} FIG. 84 is “low-frequency component moving image data”, and the data in the region R _{8501 in} FIG. , “Middle and low range component moving image data”, and the data in region R _{8601 in} FIG. 86 is “middle and high range component moving image data”. Further, “high-frequency component moving image data” does not exist for the data in the region R _{1101 in} FIG.

84. “Low-frequency component moving image data” that is data in region R _{8401 in} FIG. 84, “Middle and low-frequency component moving image data” that is data in region R _{8501 in} FIG. 85, and “Middle and high-frequency data that is data in region R _{8601 in} FIG. When `` component video data '' is downsampled to 1/4 in the time direction, the data after the downsampling is sampled at intervals of r _{0 in the} spatial direction x, y, and in the time direction t Becomes data sampled at intervals of 4 × t ₀ .

Therefore, with regard to “low frequency component moving image data” that is data of region R _{8401 in} FIG. 84, as shown in FIG. 99, about “middle low frequency component moving image data” that is data of region R _{8501 in} FIG. As shown in FIG. 100, the “middle and high frequency component moving image data” that is the data of the region R _{8601 in} FIG. 86 is 2π in the X and Y directions on the frequency domain, respectively, as shown in FIG. Folding components occur at intervals of / r ₀ , and folding components occur at intervals of 2π / (4t ₀ ) in the T direction.

However, the region R ₈₄₀₁ in FIG. 84, the region R ₈₅₀₁ in FIG. 85, and the region R _{8601 in} FIG. 86 are regions obtained by dividing the region R _{1101 in} FIG. 21, and the region R ₁₁₀₁ is described with reference to FIG. , A region having a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, −π / (2t0)), and the X and Y directions are − Since (π / r ₀ ) to + (π / r ₀ ) and the T direction is in the range of − (π / t ₀ ) to + (π / t ₀ ), as shown in FIG. Even if sampling is performed at 4 × t ₀ intervals in the time direction, the aliasing components do not overlap. Accordingly, the “low-frequency component moving image data” (FIG. 99) in the region R ₈₄₀₁ in FIG. 84 obtained by dividing the region R _{1101 in} FIG. 21 and the “middle low-frequency component moving image data” in the region R ₈₅₀₁ in FIG. 100), and the “middle / high range component moving image data” (FIG. 101) in the region R _{8601 in} FIG. 86 does not overlap the aliasing components.

  In FIG. 99 to FIG. 101, the shaded portion is a portion where the down-sampled moving image data exists.

Next, for a portion of the moving image where the subject projected thereon moves at a speed of about “r ₀ / t ₀ ”, a region R _{1201 in} FIG. 22 is the basic band. As for the data in the region R _{1201 in} FIG. 22, as described above, the data in the region R _{8701 in} FIG. 87 is “low-frequency component moving image data”, and the data in the region R _{8801 in} FIG. 89, the data in region R _{8901 in} FIG. 89 is “middle and high frequency component video data”, and the data in region R _{9001 in} FIG. 90 is “high frequency component video data”. is there.

A data area R ₈₇₀₁ in FIG. 87, "low-frequency component video data" is data area R ₈₈₀₁ of FIG. 88 "medium low-frequency component video data" is data area R ₈₉₀₁ in FIG. 89, "high range When the “high-frequency component moving image data”, which is the data of the component moving image data ”and the region R _{9001 in} FIG. 90, is down-sampled to ¼ in the time direction, Data is sampled at intervals of r ₀ for y and sampled at intervals of 4 × t ₀ for time direction t.

Therefore, as for “low-frequency component moving image data” that is data of region R _{8701 in} FIG. 87, as shown in FIG. 102, “middle and low-frequency component moving image data” that is data of region R _{8801 in} FIG. 103, “middle and high frequency component moving image data” that is data of region R _{8901 in} FIG. 89 is “high frequency component moving image data that is data of region R _{9001 in} FIG. 90”, as shown in FIG. As for “data”, as shown in FIG. 105, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and 2π / (4t ₀ ) intervals in the T direction. Thus, a folding component is generated.

However, the region R ₈₇₀₁ in FIG. 87, the region R ₈₈₀₁ in FIG. 88, the region R ₈₉₀₁ in FIG. 89, the region R ₉₀₀₁ in FIG. 90 is an area obtained by dividing the area R ₁₂₀₁ in FIG. 22, the region R ₁₂₀₁ is As described in FIG. 12, the region having a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, −π / t0), and X, Since the Y direction is a region in the range of − (π / r ₀ ) to + (π / r ₀ ) and the T direction is in the range of − (π / t ₀ ) to + (π / t ₀ ), FIG. As shown in FIG. 4, even if sampling is performed at 4 × t ₀ intervals in the time direction, the aliasing components do not overlap. Therefore, “low-frequency component moving image data” (FIG. 102) in the region R ₈₇₀₁ in FIG. 87 obtained by dividing the region R _{1201 in} FIG. 22 and “middle low-frequency component moving image data” in the region R ₈₈₀₁ in FIG. 103), the “mid-high frequency component moving image data” (FIG. 104) in the region R _{8901 in} FIG. 89 and the “high frequency component moving image data” (FIG. 105) in the region R _{9001 in} FIG. 90 do not overlap.

  In FIGS. 102 to 105, the shaded portion is the portion where the down-sampled moving image data exists.

Next, for a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, a region R _{1301 in} FIG. 23 is a basic band. 23, the data in the region R _{9101 in} FIG. 91 is “low-frequency component moving image data”, and the data in the region R _{9201 in} FIG. 92 is the data in the region R ₁₃₀₁ in FIG. , The data in the region R _{9301 in} FIG. 93 is “middle and high frequency component video data”, and the data in the region R _{9401 in} FIG. 94 is “high frequency component video data”. is there.

“Low-frequency component moving image data” that is data of region R _{9101 in} FIG. 91, “Middle and low-frequency component moving image data” that is data of region R _{9201 in} FIG. 92, and “Middle and high-frequency region” that is data of region R _{9301 in} FIG. When the “high-frequency component moving image data”, which is the data of the region R _{9401 in} FIG. 94, is down-sampled to 1/4 in the time direction, the data after the down-sampling are all in the spatial direction x, Data is sampled at intervals of r ₀ for y and sampled at intervals of 4 × t ₀ for time direction t.

For this reason, with regard to “low-frequency component moving image data” that is data of region R _{9101 in} FIG. 91, as shown in FIG. 106, “middle and low-frequency component moving image data” that is data of region R _{9201 in} FIG. As shown in FIG. 107, “middle and high frequency component moving image data” that is the data of region R _{9301 in} FIG. 93 is “high frequency component moving image” that is the data of region R _{9401 in} FIG. With respect to “data”, as shown in FIG. 109, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions on the frequency domain, and 2π / (4t ₀ ) intervals in the T direction. Thus, a folding component is generated.

However, the region R ₉₁₀₁ in FIG. 91, the region R ₉₂₀₁ in FIG. 92, the region in FIG. 93 R _9301, region R ₉₄₀₁ in FIG. 94 is an area obtained by dividing the area R ₁₃₀₁ in FIG. 23, the region R ₁₃₀₁ is As described in FIG. 13, the region having a width of 2π / (4t0) in the T direction from the straight line connecting the origin (0,0) and the point (π / r0, −2π / t0), and X, Since the Y direction is a region in the range of − (π / r ₀ ) to + (π / r ₀ ) and the T direction is in the range of − (π / t ₀ ) to + (π / t ₀ ), FIG. As shown in FIG. 4, even if sampling is performed at 4 × t ₀ intervals in the time direction, the aliasing components do not overlap. Accordingly, the “low-frequency component moving image data” (FIG. 106) in the region R ₉₁₀₁ in FIG. 91 obtained by dividing the region R _{1301 in} FIG. 23 and the “middle low-frequency component moving image data” in the region R ₉₂₀₁ in FIG. 107), “middle / high-frequency component moving image data” (FIG. 108) in region R _{9301 in} FIG. 93 and “high-frequency component moving image data” (FIG. 109) in region R _{9401 in} FIG. 94 do not overlap.

  In FIGS. 106 to 109, the shaded portion is the portion where the down-sampled moving image data exists.

As shown in FIGS. 98 to 109, in the frequency domain, the aliasing components of the down-sampled video data do not overlap with each other from the down-sampled video data. “Low-frequency component moving image data” in 20 regions R ₁₀₀₁ , “Low-frequency component moving image data” in region R _{8401 in} FIG. 84, “Middle and low-frequency component moving image data” in region R _{8501 in} FIG. 85, and Region R in FIG. "intermediate and high frequency components moving data" _8601, "the low-frequency component video data" of the region R ₈₇₀₁ in FIG. 87, "medium low-frequency component video data" of the region R ₈₈₀₁ in FIG. 88, "crowned area R ₈₉₀₁ in FIG. 89 90 ”,“ High-frequency component video data ”in region R _{9001 in} FIG. 90,“ Low-frequency component video data ”in region R _{9101 in} FIG. 91,“ Middle-low-frequency component video data ”in region R _{9201 in} FIG. "," middle-to-high-frequency component video region R ₉₃₀₁ in FIG. 93 Over data "means that it is possible to extract the" high-frequency component video data "of the region R ₉₄₀₁ in FIG. 94, in other words, necessary information video data after downsampling, in consideration of human visual characteristics (Information that can be recognized by humans) is accurately held.

Next, “Low-frequency component video data”, “Middle-low frequency component video data”, “Middle-high frequency component video data” and “High-frequency component video data” with the frame rate 1 / (4t ₀ ) as described above are multiplexed. The configuration and processing of the receiving apparatus 2 when the encoded data is supplied to the receiving apparatus 2 via the recording medium 11 or the transmission medium 12 (FIG. 24) will be described.

FIG. 110 shows a case where the display device 3 has the display capability represented by the region R ₈₃₀₁ in FIG. 83, that is, a frame rate of 1 / (4t ₀ ), and one pixel per pixel in the spatial direction. It shows an exemplary configuration of a receiving apparatus 2 in FIG. 24 in the case where it is possible to display at intervals of r _0.

  In FIG. 110, portions corresponding to those in FIG. 33 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the receiving apparatus 2 in FIG. 110 is configured in the same manner as in FIG. 33 except that the filter generation unit 53 is not provided and an extraction unit 211 is provided instead of the filter unit 52. .

The extraction unit 211 extracts “low-frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51 and supplies the extracted data to the display device 3 (FIG. 24).

  Next, processing of the receiving device 2 in FIG. 110 will be described with reference to the flowchart in FIG.

The decoding unit 50 is supplied with encoded data. The decoding unit 50 decodes the encoded data into moving image data with a frame rate of 1 / (4t ₀ ) and a low frame rate, and supplies it to the buffer unit 51. The buffer unit 51 sequentially stores moving image data having a low frame rate of 1 / (4t ₀ ) supplied from the decoding unit 50.

In step S231, the extracting unit 211 extracts “low-frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and the process proceeds to step S232.

  In step S232, the extraction unit 211 outputs the “low frequency component moving image data” extracted in step S231 to the display device 3 (FIG. 24).

In this case, “low-frequency component moving image data” having a frame rate of 1 / (4t ₀ ) is supplied and displayed on the display device 3 having the display capability represented by the region R ₈₃₀₁ in FIG.

That is, for example, for a portion of a moving image where a stationary subject is projected, the “low frequency component moving image data” of the frame rate 1 / (4t ₀ ) is the frequency component data shown in FIG. Yes, as shown in FIG. 112 with a shadow, the display device 3 displays only the data in the region R ₈₃₀₁ in FIG. 83 out of the frequency component data shown in FIG. Then, the aliasing component shown in FIG. 98 is not displayed.

Next, for example, for a portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2”, “low-frequency component moving image data” is shown in FIG. In the display device 3, as shown in FIG. 113 with a shadow, the data in the region R ₈₃₀₁ in FIG. Only data in an area R ₈₄₀₁ in FIG. 84 is displayed. The display device 3 does not display the aliasing component shown in FIG.

Next, for example, in a portion of the moving image where the subject projected on the moving object moves at a speed of about “r ₀ / t ₀ ”, the “low frequency component moving image data” has the frequency shown in FIG. In the display device 3, as shown in FIG. 114 with a shadow, in the frequency component data shown in FIG. 102, the data in the region R ₈₃₀₁ in FIG. Only the data in area R ₈₇₀₁ is displayed. The aliasing component shown in FIG. 102 is not displayed.

Next, for example, for a portion of the moving image where the subject projected thereon moves at a speed of about “2r ₀ / t ₀ ”, the “low frequency component moving image data” has the frequency shown in FIG. In the display device 3, as shown in FIG. 115 with a shadow, in the frequency component data shown in FIG. 106, the data in the region R ₈₃₀₁ in FIG. Only the data in area R ₉₁₀₁ is displayed. The aliasing component shown in FIG. 106 is not displayed.

As described above, when the display device 3 has a frame rate of 1 / (4t ₀ ), only the “low frequency component moving image data” is supplied to the display device 3. The frequency component data that can be displayed by the capability can be appropriately (optimally) displayed without being affected by the aliasing component.

  As can be seen from the above description, the moving image data actually used after the buffer unit 51 is “low frequency component moving image data”, “middle low frequency component moving image data”, “middle and high frequency component moving image data”, and “ Of the four data of “high frequency component moving image data”, only one data of “low frequency component moving image data” is provided. Therefore, the decoding unit 50 in FIG. 110 decodes “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” among the four data transmitted from the transmission device 1. You don't have to.

Next, FIG. 116 illustrates a case where the display device 3 has the display capability represented by the region R ₈₂₀₁ in FIG. 82, that is, has a frame rate of 1 / (2t ₀ ), and 1 pixel per pixel. 25 shows a configuration example of the receiving device 2 in FIG. 24 when it can be displayed at intervals of r _{0 in} the spatial direction.

  116, portions corresponding to those in FIG. 33 are denoted by the same reference numerals, and description thereof will be omitted as appropriate. That is, the receiving apparatus 2 in FIG. 116 is the same as in FIG. 33 except that an extracting unit 221 and a combining unit 222 are newly provided, and a filter unit 223 is provided instead of the filter unit 52. It is configured.

The extraction unit 221 extracts “low frequency component video data” and “middle low frequency component video data” from the video data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies it to the synthesis unit 222. .

Combining unit 222, the synthesis of the extraction unit 221, a frame rate 1 / (4t ₀₎ of synthesizing the "low-frequency component video data" and "medium low-frequency component video data", the frame rate 1 / (4t ₀₎ The moving image data is supplied to the filter unit 223.

The filter unit 223 uses the combined moving image data of the frame rate 1 / (4t ₀ ) supplied from the combining unit 222 to filter information from the filter information supply unit 62 as in the filter unit 52 of FIG. Perform corresponding filtering and upsampling. However, the filter unit 52 in FIG. 33 performs upsampling four times, but the filter unit 223 performs upsampling twice.

  Next, processing of the receiving device 2 in FIG. 116 will be described with reference to the flowchart in FIG. 117.

The decoding unit 50 is supplied with encoded data. The decoding unit 50 decodes the encoded data into moving image data with a frame rate of 1 / (4t ₀ ) and a low frame rate, and supplies it to the buffer unit 51. The buffer unit 51 sequentially stores moving image data having a low frame rate of 1 / (4t ₀ ) supplied from the decoding unit 50.

In step S241, the extracting unit 221 extracts “low-frequency component moving image data” from the moving image data at the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the extracted low-frequency component moving image data to the combining unit 222. Proceed to step S242. In step S242, the extraction unit 221 extracts “middle and low frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies it to the synthesizing unit 222. The process proceeds to S243.

In step S243, the synthesizing unit 222 synthesizes the “low frequency component moving image data” and the “middle low frequency component moving image data” supplied from the extracting unit 211, that is, for example, “low frequency component moving image data” and “ “Middle-low frequency component moving image data” is added, and the resultant combined moving image data of the frame rate 1 / (4t ₀ ) is supplied to the filter unit 223.

  After that, in steps S244 to S246, the filter generation unit 53 (the principal component direction acquisition unit 61 and the filter information supply unit 62) applies to each portion of the synthesized moving image data (for example, a block of 16 × 16 pixels as described above). , A region that can be recognized by humans on the frequency domain is obtained as a pass band of the filter, and the process of supplying the pass band as filter information to the filter unit 223, that is, the “required information of FIG. The same processing as “processing for obtaining passband” is performed.

  That is, in step S244, the principal component direction acquisition unit 61 of the filter generation unit 53 uses the frequency axes X and Y in the spatial directions x and y and the frequency axis T in the time direction t for the data of each part of the synthesized moving image data. The principal component direction in the defined frequency domain is acquired and supplied to the filter information supply unit 62, and the process proceeds to step S245.

  Note that in step S244, the principal component direction can be obtained by performing a three-dimensional Fourier transform on synthetic moving image data (or “low-frequency component moving image data”) as described with reference to FIG. As described in FIG. 32, it can also be obtained by detecting a motion vector. Further, as described above, when the principal component direction (information thereof) is multiplexed in the encoded data from the transmission device 1, the decoding unit 50 separates the principal component direction from the encoded data, The direction acquisition unit 61 can also acquire the principal component direction by receiving the supply of the principal component direction from the decoding unit 50.

Further, as described in step S32 in FIG. 34, encoded data in the spatial direction x, in the case that contains the y motion vector (u _0, v _0), the motion vector (u _0, v _0), The time direction component t (here, t = 4t ₀ ) is added to obtain a three-dimensional motion vector (u ₀ , v ₀ , t) in the spatial direction x, y and time direction t, as described in FIG. In addition, a plane orthogonal to the three-dimensional motion vector (u ₀ , v ₀ , t) can be set as the principal component direction.

  That is, in the encoding unit 24 of the transmission device 1, when “low-frequency component moving image data” is encoded by an encoding method such as MPEG that uses at least motion compensation, the encoded data includes motion compensation. Contains the motion vector to be used. Therefore, in the decoding unit 50, the motion vector included in the encoded data is supplied to the principal component direction acquisition unit 61 via the buffer unit 51. In the principal component direction acquisition unit 61, a plane orthogonal to the motion vector is supplied. Can be obtained (detected) as the principal component direction.

In step S245, the filter information supply unit 62 is an area extending in the principal component direction from the principal component direction acquisition unit 61 from the origin (0, 0) in the frequency domain, and 2π / (4 × t in the T direction. ₀ ), the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ), and the T direction is-(π / t ₀ ) to + (π / t ₀ ). , Ie, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 are determined as the passband of the filter, and the process proceeds to step S246.

  In step S246, the filter information supply unit 62 supplies the filter passband obtained in step S245 (information indicating) to the filter unit 223 as filter information, and the process proceeds to step S247.

  In step S247, the filter unit 223 applies the passband filter represented by the filter information supplied from the filter generation unit 53 (the filter information supply unit 62) in step S246 to the synthesized moving image data from the synthesis unit 222. However, upsampling is performed to double the number of samples in the time direction.

  That is, for example, a block as a portion of the combined moving image data whose principal component direction has been acquired in step S244 is set as a target block, and a frame of the target block is set as a target frame.

The filter unit 223 uses the passband filter represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region R _{1101 in} FIG. The data (pixel value) of each pixel of the block of interest is obtained by filtering with a filter that uses the region R ₁₂₀₁ in FIG. 22 and the region R _{1301 in} FIG. 23 as a pass band.

Further, the filter unit 223 filters the synthesized moving image data from the synthesis unit 222 in the passband represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region in FIG. R ₁₁₀₁ , a region R ₁₂₀₁ in FIG. 22 and a filter having a region R _{1301 in} FIG. 23 as a passband are used to filter the time 4t ₀ from the frame of interest to the next frame into time 2t ₀ . Data of pixels on a straight line extending at the time and passing through each pixel of the block of interest and extending in the direction of the three-dimensional motion vector of the block of interest is obtained. Note that the three-dimensional motion vector of the block of interest can be obtained by, for example, the principal component direction acquisition unit 61 and supplied to the filter unit 223 together with the filter information via the filter information supply unit 62.

The filter unit 223 obtains composite video data having a frame rate of 1 / (2t ₀ ) by performing the above filtering and upsampling on the composite video data having a frame rate of 1 / (4t ₀ ). The process proceeds from step S247 to S248.

In step S248, the filter unit 223 outputs the combined moving image data of the frame rate 1 / (2t ₀ ) obtained in step S247 to the display device 3 (FIG. 24).

In this case, the composite moving image data having the frame rate 1 / (2t ₀ ) is supplied and displayed on the display device 3 having the display capability represented by the region R ₈₂₀₁ in FIG.

  That is, in this case, the combined moving image data is data obtained by combining “low-frequency component moving image data” and “middle-low frequency component moving image data”.

For example, as for the portion of the moving image where the stationary subject is projected, as described above, the “low frequency component moving image data” of the frame rate 1 / (4t ₀ ) is shown in FIG. This is frequency component data, and there is no “middle and low frequency component moving image data” with a frame rate of 1 / (4t ₀ ).

Therefore, the combined moving image data of the frame rate 1 / (4t ₀ ) obtained by combining the “low frequency component moving image data” of the frequency component shown in FIG. 98 and the “middle low frequency component moving image data” that does not exist is The frequency component data shown in FIG. 98 is obtained.

Then, with respect to the composite moving image data (frame portion data in the moving image) with the frame rate 1 / (4t ₀ ) in FIG. 98, the filter unit 223 uses the filter having the region R _{1001 in} FIG. Filtering by 2 and upsampling of 2 times are performed (step S247 in FIG. 117). Therefore, the synthesized moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₀₀₁ in FIG. 20, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y, and the time direction t Is data sampled at 2t ₀ intervals.

Therefore, as shown in FIG. 118, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / (2t ₀ ) in the T direction. Occurs.

In FIG. 118, the shaded portion is a portion where the stationary portion of the combined moving image data having the frame rate 1 / (2t ₀ ) exists.

In the display device 3, only the data in the area R ₈₂₀₁ in FIG. 82 of the combined moving image data at the frame rate 1 / (2t ₀ ) shown in FIG. 118 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₂₀₁ in FIG. 82, only the data in the region indicated by the shadow in FIG. 119 is displayed, and the aliasing component shown in FIG. 118 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving image is moving at a speed of “(r ₀ / t ₀ ) / 2”, as described above, the frame rate 1 / (4t "low-frequency component video data" ₀₎ is the data of the frequency components shown in FIG. 99, the frame rate 1 / (4t ₀₎ "medium low-frequency component video data", the frequency components shown in FIG. 100 It is data of.

Therefore, the frame rate 1 / (4t ₀ ) obtained by synthesizing the “low frequency component moving image data” of the frequency component shown in FIG. 99 and the “middle low frequency component moving image data” of the frequency component shown in FIG. The combined moving image data is frequency component data shown in FIG.

Then, with respect to the composite moving image data (the data moving at the speed “(r ₀ / t ₀ ) / 2”) at the frame rate 1 / (4t ₀ ) in FIG. Filtering by a filter having the region R ₁₁₀₁ as a pass band and up-sampling twice are performed (step S247 in FIG. 117). Therefore, as shown in FIG. 121, the synthesized moving image data obtained by the filtering and upsampling is sampled at intervals of r ₀ for the spatial direction x, y and sampled at intervals of 2t ₀ for the time direction t. Data.

That is, as shown in FIG. 121, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / (2t ₀ ) in the T direction. Arise.

In FIG. 121, the shaded portion is the data of the portion moving at a speed of about “(r ₀ / t ₀ ) / 2” in the combined moving image data of the frame rate 1 / (2t ₀ ). Is the part that exists.

In the display device 3, only the data in the region R ₈₂₀₁ in FIG. 82 of the combined moving image data at the frame rate 1 / (2t ₀ ) shown in FIG. 121 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₂₀₁ in FIG. 82, only the data in the region indicated by the shadow in FIG. 122 is displayed, and the aliasing component shown in FIG. 121 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving object moves at a speed of about “r ₀ / t ₀ ”, as described above, “1” of the frame rate 1 / (4t ₀ ) The “low frequency component moving image data” is the frequency component data shown in FIG. 102, and the “middle low frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the frequency component data shown in FIG. .

Therefore, the frame rate 1 / (4t ₀ ) obtained by combining the “low frequency component moving image data” of the frequency component shown in FIG. 102 and the “middle low frequency component moving image data” of the frequency component shown in FIG. The synthesized moving image data is the data of the frequency component shown in FIG.

Then, with respect to the synthetic moving image data (the data moving at the speed “r ₀ / t ₀ ”) at the frame rate 1 / (4t ₀ ) in FIG. 123, the filter unit 223 uses the region R _{1201 in} FIG. Filtering using a filter as a pass band and upsampling of 2 times are performed (step S247 in FIG. 117). Therefore, as shown in FIG. 124, the synthesized moving image data obtained by the filtering and upsampling is sampled at intervals of r ₀ for the spatial directions x and y, and sampled at intervals of 2t ₀ for the time direction t. Data.

That is, as shown in FIG. 124, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / (2t ₀ ) in the T direction. Arise.

In FIG. 124, the shaded portion is the portion where the data of the portion moving at the speed “r ₀ / t ₀ ” of the combined moving image data of the frame rate 1 / (2t ₀ ) exists. It is.

In the display device 3, only the data in the region R ₈₂₀₁ in FIG. 82 of the combined moving image data at the frame rate 1 / (2t ₀ ) shown in FIG. 124 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₂₀₁ in FIG. 82, only the data in the region indicated by the shadow in FIG. 125 is displayed, and the aliasing component shown in FIG. 124 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving image is moving at a speed of about “2r ₀ / t ₀ ”, as described above, the frame rate 1 / (4t ₀ ) “ The “low frequency component moving image data” is the frequency component data shown in FIG. 106, and the “middle low frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the frequency component data shown in FIG. .

Therefore, the frame rate 1 / (4t ₀ ) obtained by synthesizing the “low frequency component moving image data” of the frequency component shown in FIG. 106 and the “middle low frequency component moving image data” of the frequency component shown in FIG. The combined moving image data is frequency component data shown in FIG.

Then, with respect to the synthetic moving image data (the data moving at the speed “2r ₀ / t ₀ ”) at the frame rate 1 / (4t ₀ ) in FIG. 126, the filter unit 223 uses the region R _{1301 in} FIG. Filtering using a filter as a pass band and upsampling of 2 times are performed (step S247 in FIG. 117). Therefore, the synthesized moving image data obtained by the filtering and upsampling is sampled at intervals of r ₀ in the spatial direction x, y and sampled at intervals of 2t ₀ in the time direction t, as shown in FIG. Data.

That is, as shown in FIG. 127, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / (2t ₀ ) in the T direction. Arise.

In FIG. 127, the shaded portion is the portion where the data of the portion moving at a speed of about “2r ₀ / t ₀ ” in the combined moving image data of the frame rate 1 / (2t ₀ ) exists. It is.

In the display device 3, only the data in the region R ₈₂₀₁ in FIG. 82 of the combined moving image data at the frame rate 1 / (2t ₀ ) shown in FIG. 127 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₂₀₁ in FIG. 82, only the data in the region indicated by the shadow in FIG. 128 is displayed, and the aliasing component shown in FIG. 127 is not displayed.

As described above, when the display device 3 has a frame rate of 1 / (2t ₀ ), the “low frequency component moving image data” and the “middle low frequency component moving image data” are combined to generate 1 / ( 4t ₀ ) frame rate composite video data is obtained, up-sampling is performed twice while filtering based on the principal component direction, and the resulting 1 / (2t ₀ ) frame rate composite video data is displayed. By supplying to the device 3, the display device 3 can appropriately display the data of the frequency component that can be displayed with the display capability without being influenced by the aliasing component.

  As can be seen from the above description, the moving image data actually used after the buffer unit 51 is “low frequency component moving image data”, “middle low frequency component moving image data”, “middle and high frequency component moving image data”, and “ Of the four data of “high frequency component moving image data”, there are only two data of “low frequency component moving image data” and “middle low frequency component moving image data”. Therefore, the decoding unit 50 in FIG. 116 does not need to decode “middle / high-frequency component moving image data” and “high-frequency component moving image data” among the four data transmitted from the transmission device 1.

Next, FIG. 129, the display unit 3, if having a display capability represented by region R ₈₁₀₁ in FIG. 81, i.e., having a frame rate of _{1 / (4t 0/3)} , and 1 pixel 1 25 shows a configuration example of the receiving device 2 in FIG. 24 in a case where pixels can be displayed at intervals of r _{0 in} the spatial direction.

  In FIG. 129, portions corresponding to those in FIG. 33 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the receiving apparatus 2 in FIG. 129 is the same as in FIG. 33 except that an extracting unit 231 and a combining unit 232 are newly provided, and a filter unit 233 is provided instead of the filter unit 52. It is configured.

The extraction unit 231 obtains “low frequency component video data”, “middle low frequency component video data”, and “middle high frequency component video data” from the video data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51. Extracted and supplied to the synthesis unit 232.

The synthesizing unit 232 synthesizes the “low frequency component moving image data”, “middle and low frequency component moving image data”, and “middle and high frequency component moving image data” of the frame rate 1 / (4t ₀ ) from the extracting unit 231, The combined moving image data of the rate 1 / (4t ₀ ) is supplied to the filter unit 233.

The filter unit 233 uses the combined moving image data of the frame rate 1 / (4t ₀ ) supplied from the combining unit 232 to filter information from the filter information supply unit 62 as in the filter unit 52 of FIG. Perform corresponding filtering and upsampling. However, although the filter unit 52 in FIG. 33 performs upsampling four times, the filter unit 233 performs upsampling three times.

  Next, processing of the reception device 2 in FIG. 129 will be described with reference to the flowcharts in FIGS. 130 and 131.

The decoding unit 50 is supplied with encoded data. The decoding unit 50 decodes the encoded data into moving image data with a frame rate of 1 / (4t ₀ ) and a low frame rate, and supplies it to the buffer unit 51. The buffer unit 51 sequentially stores moving image data having a low frame rate of 1 / (4t ₀ ) supplied from the decoding unit 50.

In step S251, the extracting unit 231 extracts “low-frequency component moving image data” from the moving image data having the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the extracted low-frequency component moving image data to the combining unit 232. Proceed to step S252. In step S252, the extracting unit 231 extracts “middle and low frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the extracted “middle and low frequency component moving image data” to the combining unit 232. The process proceeds to S253. In step S253, the extraction unit 231 extracts “middle and high frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the extracted “middle and high frequency component moving image data” to the combining unit 232 as shown in FIG. The process proceeds to step S254.

In step S254, the synthesizing unit 232 synthesizes the “low frequency component moving image data”, “middle and low frequency component moving image data”, and “middle and high frequency component moving image data” supplied from the extracting unit 211, that is, for example, “ The low-frequency component moving image data ”, the“ middle-low-frequency component moving image data ”, and the“ middle-high frequency component moving image data ”are added, and the resultant moving image data having the frame rate 1 / (4t ₀ ) is added to the filter unit 233. Supply.

  Thereafter, in steps S255 to S257, the filter generation unit 53 (the principal component direction acquisition unit 61 and the filter information supply unit 62) performs the same processing as in steps S244 to S246 of FIG. For each part of the video data (for example, a block of 16 × 16 pixels as described above), an area that can be recognized by humans on the frequency domain is obtained as a filter passband, and the passband is used as filter information. This is supplied to the filter unit 233.

That is, in step S255, the principal component direction acquisition unit 61 of the filter generation unit 53 uses the frequency axes X and Y in the spatial directions x and y and the frequency axis T in the time direction t for the data of each part of the synthesized moving image data. The principal component direction in the defined frequency domain is acquired, supplied to the filter information supply unit 62, and the process proceeds to step S256. In step S256, the filter information supply unit 62 is a region extending in the principal component direction from the principal component direction acquisition unit 61 from the origin (0, 0) in the frequency domain, and 2π / (4 × t in the T direction. ₀ ), the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ), and the T direction is-(π / t ₀ ) to + (π / t ₀ ). , Ie, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 are determined as the passband of the filter, and the process proceeds to step S257. In step S257, the filter information supply unit 62 supplies the filter passband obtained in step S256 (information indicating) to the filter unit 233 as filter information, and the process proceeds to step S258.

  In step S258, the filter unit 233 applies the passband filter represented by the filter information supplied from the filter generation unit 53 (the filter information supply unit 62) in step S257 to the synthesized moving image data from the synthesis unit 232. However, upsampling is performed to triple the number of samples in the time direction.

  That is, for example, a block as a portion of the composite moving image data whose principal component direction has been acquired in step S255 is set as a target block, and a frame of the target block is set as a target frame.

The filter unit 233 uses the passband filter represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region R _{1101 in} FIG. The data (pixel value) of each pixel of the block of interest is obtained by filtering with a filter that uses the region R ₁₂₀₁ in FIG. 22 and the region R _{1301 in} FIG. 23 as a pass band.

Further, the filter unit 233 filters the synthesized moving image data from the synthesis unit 232 in the passband represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region in FIG. R _1101, a region R ₁₂₀₁ in FIG. 22, by filtering by a filter to pass band region R ₁₃₀₁ in FIG. 23, a time 4t ₀ from the target frame to the next, aliquoted into time 4t _0/3 Data of pixels on two straight lines that extend in the direction of the three-dimensional motion vector of the block of interest through the pixels of the block of interest at each of the two times is obtained. Note that the three-dimensional motion vector of the block of interest can be obtained by the principal component direction acquisition unit 61 and supplied to the filter unit 233 along with the filter information via the filter information supply unit 62, for example.

Filter unit 233, the subject composite video data of the frame rate is 1 / (4t _0), by performing filtering and upsampling as described above, synthetic moving image data frame rate 1 / (4t _0/3) And proceeds from step S258 to S259.

In step S259, the filter unit 233, the synthetic moving image data obtained frame rate 1 / (4t _0/3) in step S258, and outputs to the display device 3 (FIG. 24).

In this case, the display device 3 having a display capability represented by region R ₈₁₀₁ in FIG. 81, synthetic moving image data of a frame rate of 1 / (4t _0/3) is displayed is supplied.

  In other words, in this case, the combined moving image data is data obtained by combining “low frequency component moving image data”, “middle low frequency component moving image data”, and “middle and high frequency component moving image data”.

For example, as for the portion of the moving image where the stationary subject is projected, as described above, the “low frequency component moving image data” of the frame rate 1 / (4t ₀ ) is shown in FIG. This is frequency component data, and there is no “middle low frequency component moving image data” and “middle high frequency component moving image data” with a frame rate of 1 / (4t ₀ ).

Accordingly, the frame rate 1 / (4t obtained by synthesizing the “low frequency component moving image data” of the frequency component shown in FIG. 98 with the “middle low frequency component moving image data” and “middle high frequency component moving image data” that do not exist. ₀ ) is the frequency component data shown in FIG.

Then, with respect to the composite moving image data (data of the stationary portion in the moving image) having the frame rate 1 / (4t ₀ ) in FIG. 98, the filter unit 233 uses the filter with the region R _{1001 in} FIG. And the upsampling of 3 times is performed (step S258 in FIG. 131). Therefore, the synthesized moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₀₀₁ in FIG. 20, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y, and the time direction t the sampled data 4t _0/3 interval for.

Therefore, as shown in FIG. 132, on the frequency domain, X, the Y-direction, at 2 [pi / r ₀ interval, cause aliasing components, in the T direction at 2π / (4t _0/3) interval, A folding component is generated.

Note that in FIG. 132, the portion that is shaded is a portion in which the data exists in the still to have part of the composite moving data frame rate _{1 / (4t 0/3)} .

In the display device 3, of the synthetic moving image data of the frame rate 1 / shown in FIG. 132 (4t _0/3), only the data in the region R ₈₁₀₁ in FIG. 81 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₁₀₁ in FIG. 81, only the data in the region indicated by the shadow in FIG. 133 is displayed, and the aliasing component shown in FIG. 132 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving image is moving at a speed of “(r ₀ / t ₀ ) / 2”, as described above, the frame rate 1 / (4t "low-frequency component video data" ₀₎ is the data of the frequency components shown in FIG. 99, the frame rate 1 / (4t ₀₎ "medium low-frequency component video data", the frequency components shown in FIG. 100 It is data of. Further, the “middle and high frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the data of the frequency component shown in FIG.

Therefore, the “low frequency component moving image data” of the frequency component shown in FIG. 99, the “middle low frequency component moving image data” of the frequency component shown in FIG. 100, and the “middle high frequency component moving image data of the frequency component shown in FIG. The synthesized moving image data of the frame rate 1 / (4t ₀ ) obtained by synthesizing “” is the frequency component data shown in FIG.

Then, with respect to the composite moving image data (the data moving at the speed “(r ₀ / t ₀ ) / 2”) at the frame rate 1 / (4t ₀ ) in FIG. Filtering by a filter having the region R ₁₁₀₁ as a pass band and upsampling three times are performed (step S258 in FIG. 131). Accordingly, the synthetic moving image data obtained by the filtering and upsampling as shown in Figure 135, the spatial direction x, with respect to the y is sampled at r ₀ intervals, 4t _0/3 intervals for the time direction t The data sampled at.

That is, as shown in FIG. 135, on the frequency domain, X, the Y-direction, at 2 [pi / r ₀ interval, cause aliasing components, in the T direction at 2π / (4t _0/3) interval, folded Ingredients are produced.

The portion in FIG. 135, the portion that is shaded is in motion at a speed _{"(r 0 / t 0) /} 2 " degree of synthesized moving data frame rate 1 / (4t _0/3) This is the part where data exists.

In the display device 3, of the synthetic moving image data of the frame rate 1 / shown (4t _0/3) in FIG. 135, only the data in the region R ₈₁₀₁ in FIG. 81 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₁₀₁ in FIG. 81, only the data in the region indicated by the shadow in FIG. 136 is displayed, and the aliasing component shown in FIG. 135 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving object moves at a speed of about “r ₀ / t ₀ ”, as described above, “1” of the frame rate 1 / (4t ₀ ) The “low frequency component moving image data” is the frequency component data shown in FIG. 102, and the “middle low frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the frequency component data shown in FIG. . Further, the “middle and high frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the data of the frequency component shown in FIG.

Therefore, the “low frequency component moving image data” of the frequency component shown in FIG. 102, the “middle low frequency component moving image data” of the frequency component shown in FIG. 103, and the “middle high frequency component moving image data of the frequency component shown in FIG. synthesis moving image data "frame rate 1 / obtained by combining (4t ₀₎ becomes the data of the frequency components shown in Figure 137.

Then, with respect to the synthetic moving image data (the data moving at the speed “r ₀ / t ₀ ”) at the frame rate 1 / (4t ₀ ) in FIG. 137, the filter unit 233 uses the region R _{1201 in} FIG. Filtering by a filter as a pass band and upsampling three times are performed (step S258 in FIG. 131). Accordingly, the synthetic moving image data obtained by the filtering and upsampling as shown in Figure 138, the spatial direction x, with respect to the y is sampled at r ₀ intervals, 4t _0/3 intervals for the time direction t The data sampled at.

That is, as shown in FIG. 138, on the frequency domain, X, the Y-direction, at 2 [pi / r ₀ interval, cause aliasing components, in the T direction at 2π / (4t _0/3) interval, folded Ingredients are produced.

Note that in FIG. 138, the portion that is shaded is the presence data of the moving parts at a degree rate "r ₀ / t _0" of the synthetic moving image data of a frame rate of 1 / (4t _0/3) is It is a part to do.

In the display device 3, of the synthetic moving image data of the frame rate 1 / shown in FIG. 138 (4t _0/3), only the data in the region R ₈₁₀₁ in FIG. 81 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₁₀₁ in FIG. 81, only the data in the region indicated by the shadow in FIG. 139 is displayed, and the aliasing component shown in FIG. 138 is not displayed.

Next, as described above, the portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ” is set to “low range” of the frame rate 1 / (4t ₀ ). The “component moving image data” is the frequency component data shown in FIG. 106, and the “middle low frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the frequency component data shown in FIG. Further, the “middle and high frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the data of the frequency component shown in FIG.

Therefore, the “low frequency component moving image data” of the frequency component shown in FIG. 106, the “middle low frequency component moving image data” of the frequency component shown in FIG. 107, and the “middle high frequency component moving image data of the frequency component shown in FIG. The combined moving image data of the frame rate 1 / (4t ₀ ) obtained by combining “” is the frequency component data shown in FIG.

Then, with respect to the synthetic moving image data (the data moving at the speed “2r ₀ / t ₀ ”) at the frame rate 1 / (4t ₀ ) in FIG. 140, the filter unit 233 uses the region R _{1301 in} FIG. Filtering by a filter as a pass band and upsampling three times are performed (step S258 in FIG. 131). Accordingly, the synthetic moving image data obtained by the filtering and upsampling as shown in Figure 141, the spatial direction x, with respect to the y is sampled at r ₀ intervals, 4t _0/3 intervals for the time direction t The data sampled at.

Therefore, as shown in FIG. 141, on the frequency domain, X, the Y-direction, at 2 [pi / r ₀ interval, cause aliasing components, in the T direction at 2π / (4t _0/3) interval, A folding component is generated.

That is, in FIG. 141, the portion that is shaded is the presence data of the moving parts at a degree rate "r ₀ / t _0" of the synthetic moving image data of a frame rate of 1 / (4t _0/3) is It is a part to do.

In the display device 3, of the synthetic moving image data of the frame rate 1 / shown in FIG. 141 (4t _0/3), only the data in the region R ₈₁₀₁ in FIG. 81 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₁₀₁ in FIG. 81, only the data in the region indicated by the shadow in FIG. 142 is displayed, and the aliasing component shown in FIG. 141 is not displayed.

As described above, the display device 3, 1 / if having a frame rate of (4t _0/3), the "low-frequency component video data", "medium low-frequency component video data" and "high range component video Data ”to obtain composite video data with a frame rate of 1 / (4t ₀ ), upsampling three times while performing filtering based on the principal component direction, and the resulting 1 / (4t ₀ / 3) frame rate composite video data is supplied to the display device 3 so that the display device 3 can appropriately display the frequency component data that can be displayed with the display capability without being affected by the aliasing component. Can be displayed.

  As can be seen from the above description, the moving image data actually used after the buffer unit 51 is “low frequency component moving image data”, “middle low frequency component moving image data”, “middle and high frequency component moving image data”, and “ Of the four data of “high frequency component moving image data”, there are only three data of “low frequency component moving image data”, “middle and low frequency component moving image data”, and “middle and high frequency component moving image data”. Therefore, the decoding unit 50 in FIG. 129 does not need to decode “high frequency component moving image data” among the four data transmitted from the transmission device 1.

Next, FIG. 143 shows a case where the display device 3 has the display capability represented by the region R ₈₀₀₁ in FIG. 80, that is, has a frame rate of 1 / t ₀ and one pixel per pixel in the spatial direction. FIG. 24 shows a configuration example of the receiving apparatus 2 in FIG. 24 when it can be displayed at intervals of r ₀ .

  In FIG. 143, portions corresponding to those in FIG. 33 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the receiving apparatus 2 in FIG. 143 is configured in the same manner as in FIG. 33 except that an extracting unit 241 and a combining unit 242 are newly provided.

The extraction unit 241 extracts data of all frequency components from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, that is, “low frequency component moving image data” and “middle low frequency component moving image data”. , “Middle and high frequency component moving image data” and “high frequency component moving image data” are extracted and supplied to the synthesis unit 242.

The synthesizing unit 242 outputs the “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component” of the frame rate 1 / (4t ₀ ) from the extracting unit 241. The moving image data ”is combined, and the combined moving image data having the frame rate 1 / (4t ₀ ) is supplied to the filter unit 52.

  Next, processing of the receiving apparatus 2 in FIG. 143 will be described with reference to the flowcharts in FIGS. 144 and 145.

The decoding unit 50 is supplied with encoded data. The decoding unit 50 decodes the encoded data into moving image data with a frame rate of 1 / (4t ₀ ) and a low frame rate, and supplies it to the buffer unit 51. The buffer unit 51 sequentially stores moving image data having a low frame rate of 1 / (4t ₀ ) supplied from the decoding unit 50.

In step S261, the extracting unit 241 extracts “low-frequency component moving image data” from the moving image data with the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the extracted low-frequency component moving image data to the combining unit 242. Proceed to step S262. In step S262, the extracting unit 241 extracts “middle and low frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the extracted “middle low frequency component moving image data” to the combining unit 242. The process proceeds to S263. In step S263, the extraction unit 241 extracts “middle and high frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the extracted “middle and high frequency component moving image data” to the combining unit 242. Proceed to In step S264, the extraction unit 241 extracts “high frequency component moving image data” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the high frequency component moving image data to the combining unit 242. The process proceeds to step S265.

In step S265, the synthesizing unit 242 receives the “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” supplied from the extracting unit 211. Combining, that is, for example, adding “low frequency component video data”, “middle low frequency component video data”, “middle and high frequency component video data”, and “high frequency component video data”, and the resulting frame rate The 1 / (4t ₀ ) composite video data is supplied to the filter unit 52.

  Thereafter, in steps S266 to S268, the filter generation unit 53 (the principal component direction acquisition unit 61 and the filter information supply unit 62) performs the same processing as in steps S244 to S246 in FIG. For each part of the video data (for example, a block of 16 × 16 pixels as described above), an area that can be recognized by humans on the frequency domain is obtained as a filter passband, and the passband is used as filter information. This is supplied to the filter unit 52.

That is, in step S266, the principal component direction acquisition unit 61 of the filter generation unit 53 uses the frequency axes X and Y in the spatial directions x and y and the frequency axis T in the time direction t for the data of each part of the synthesized moving image data. The principal component direction in the defined frequency domain is acquired and supplied to the filter information supply unit 62, and the process proceeds to step S267. In step S267, the filter information supply unit 62 is a region extending in the principal component direction from the principal component direction acquisition unit 61 from the origin (0, 0) in the frequency domain, and 2π / (4 × t in the T direction. ₀ ), the X and Y directions are-(π / r ₀ ) to + (π / r ₀ ), and the T direction is-(π / t ₀ ) to + (π / t ₀ ). , That is, for example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. move on. In step S268, the filter information supply unit 62 supplies the filter passband (information indicating) obtained in step S267 to the filter unit 52 as filter information, and the process proceeds to step S269.

  In step S269, the filter unit 52 performs the same processing as in step S35 of FIG. 34 on the synthesized moving image data from the synthesis unit 242, that is, the filter generation unit 53 (the filter information supply unit 62) in step S268. Upsampling is performed to increase the number of samples in the time direction by four times while applying a filter in the passband represented by the filter information supplied from.

  That is, for example, a block as a portion of the composite moving image data whose principal component direction has been acquired in step S266 is a target block, and a frame of the target block is a target frame.

The filter unit 52 uses the passband filter represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region R _{1101 in} FIG. The data (pixel value) of each pixel of the block of interest is obtained by filtering with a filter that uses the region R ₁₂₀₁ in FIG. 22 and the region R _{1301 in} FIG. 23 as a pass band.

Further, the filter unit 52 filters the synthesized moving image data from the synthesis unit 242 with a passband filter represented by the filter information supplied from the filter generation unit 53, for example, the region R _{1001 in} FIG. 20 or the region in FIG. By filtering with R ₁₁₀₁ , a region R ₁₂₀₁ in FIG. 22 and a filter having a region R _{1301 in} FIG. 23 as a pass band, the time 4 t ₀ from the frame of interest to the next frame is divided into three equal to the time t ₀ . Data of pixels on the straight line extending in the direction of the three-dimensional motion vector of the block of interest through each pixel of the block of interest is obtained at each time. Note that the three-dimensional motion vector of the block of interest can be obtained by, for example, the principal component direction acquisition unit 61 and supplied to the filter unit 52 together with the filter information via the filter information supply unit 62.

The filter unit 52 performs the above filtering and upsampling on the composite video data with the frame rate of 1 / (4t ₀ ), thereby obtaining the composite video data with the frame rate of 1 / t ₀ , The process proceeds from step S269 to S270.

In step S270, the filter unit 52 outputs the combined moving image data of the frame rate 1 / t ₀ obtained in step S269 to the display device 3 (FIG. 24).

In this case, the composite moving image data having the frame rate 1 / t ₀ is supplied and displayed on the display device 3 having the display capability represented by the region R ₈₀₀₁ in FIG.

  In other words, in this case, the composite video data is data obtained by combining all of the “low frequency component video data”, “middle low frequency component video data”, “mid / high frequency component video data”, and “high frequency component video data”. is there.

For example, as for the portion of the moving image where the stationary subject is projected, as described above, the “low frequency component moving image data” of the frame rate 1 / (4t ₀ ) is shown in FIG. This is frequency component data, and there is no “middle low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” with a frame rate of 1 / (4t ₀ ).

Therefore, the “low frequency component moving image data” of the frequency component shown in FIG. 98 is synthesized with the “middle low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” that do not exist. The synthesized moving image data of the frame rate 1 / (4t ₀ ) obtained as described above is the frequency component data shown in FIG.

Then, for the composite moving image data (data of a stationary portion in the moving image) with the frame rate 1 / (4t ₀ ) in FIG. 98, the filter unit 52 uses the region R _{1001 in} FIG. And the upsampling of 4 times is performed (step S269 in FIG. 145). Therefore, the synthesized moving image data obtained by the filtering and upsampling has frequency components in the region R ₁₀₀₁ in FIG. 20, and is further sampled at intervals of r _{0 with} respect to the spatial directions x and y, and the time direction t the sampled data at t ₀ interval for.

Therefore, as shown in FIG. 146, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction. .

In FIG. 146, the shaded portion is the portion where the stationary portion of the combined moving image data with the frame rate 1 / t ₀ exists.

In the display device 3, only the data in the area R ₈₀₀₁ in FIG. 80 of the combined moving image data with the frame rate 1 / t ₀ shown in FIG. 146 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₀₀₁ in FIG. 80, only the data in the region indicated by the shadow in FIG. 147 is displayed, and the aliasing component shown in FIG. 146 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving image is moving at a speed of “(r ₀ / t ₀ ) / 2”, as described above, the frame rate 1 / (4t "low-frequency component video data" ₀₎ is the data of the frequency components shown in FIG. 99, the frame rate 1 / (4t ₀₎ "medium low-frequency component video data", the frequency components shown in FIG. 100 It is data of. Further, the frame rate 1 / a (4t ₀₎ "intermediate and high frequency components moving image data" is data of the frequency components shown in FIG. 101, "high-frequency component video data" frame rate 1 / (4t ₀₎ is present do not do.

Therefore, the “low frequency component moving image data” of the frequency component shown in FIG. 99, the “middle low frequency component moving image data” of the frequency component shown in FIG. 100, and the “middle high frequency component moving image data of the frequency component shown in FIG. ”And the non-existing“ high frequency component moving image data ”, the combined moving image data of the frame rate 1 / (4t ₀ ) is the data of the frequency component shown in FIG.

Then, the combined moving image data (data of the portion moving at the speed “(r ₀ / t ₀ ) / 2)” at the frame rate 1 / (4t ₀ ) in FIG. Filtering using a filter having the region R ₁₁₀₁ as a pass band and upsampling four times are performed (step S269 in FIG. 145). Therefore, as shown in FIG. 149, the synthesized moving image data obtained by the filtering and upsampling is sampled at intervals of r ₀ in the spatial direction x, y and sampled at intervals of t ₀ in the time direction t. Data.

That is, as shown in FIG. 149, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction.

In FIG. 149, the shaded part is the data of the part moving at a speed of about “(r ₀ / t ₀ ) / 2” in the composite video data with the frame rate 1 / t _0. It is a part to do.

In the display device 3, only the data in the region R ₈₀₀₁ in FIG. 80 of the combined moving image data with the frame rate 1 / t ₀ shown in FIG. 149 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₀₀₁ in FIG. 80, only the data in the region indicated by the shadow in FIG. 150 is displayed, and the aliasing component shown in FIG. 149 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving object moves at a speed of about “r ₀ / t ₀ ”, as described above, “1” of the frame rate 1 / (4t ₀ ) The “low frequency component moving image data” is the frequency component data shown in FIG. 102, and the “middle low frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the frequency component data shown in FIG. . Further, the frame rate 1 / a (4t ₀₎ "intermediate and high frequency components moving image data" is data of the frequency components shown in FIG. 104, the frame rate 1 / a (4t ₀₎ "high-frequency component video data" It is the data of the frequency component shown in FIG.

Accordingly, the “low frequency component moving image data” of the frequency component shown in FIG. 102, the “middle low frequency component moving image data” of the frequency component shown in FIG. 103, and the “middle high frequency component moving image data” of the frequency component shown in FIG. The synthesized moving image data of the frame rate 1 / (4t ₀ ) obtained by synthesizing the “high frequency component moving image data” of the frequency component shown in FIG. 105 becomes the data of the frequency component shown in FIG.

Then, with respect to the composite moving image data (the data moving at the speed “r ₀ / t ₀ ”) at the frame rate 1 / (4t ₀ ) in FIG. 151, the filter unit 52 uses the region R _{1201 in} FIG. Filtering by a filter to be a pass band and upsampling four times are performed (step S269 in FIG. 145). Therefore, as shown in FIG. 152, the synthesized moving image data obtained by the filtering and upsampling is sampled at intervals of r ₀ for the spatial directions x and y, and sampled at intervals of t ₀ for the time direction t. Data.

That is, as shown in FIG. 152, on the frequency domain, aliasing components occur at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components occur at intervals of 2π / t _{0 in the} T direction.

In FIG. 152, the shaded portion is a portion where there is data of a portion moving at a speed of about “r ₀ / t ₀ ” in the combined moving image data of the frame rate 1 / t _0. .

In the display device 3, only the data in the area R ₈₀₀₁ in FIG. 80 of the combined moving image data with the frame rate 1 / t ₀ shown in FIG. 152 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₀₀₁ in FIG. 80, only the data in the region indicated by the shadow in FIG. 153 is displayed, and the aliasing component shown in FIG. 152 is not displayed.

Next, for example, in the portion of the moving image where the subject projected on the moving image is moving at a speed of about “2r ₀ / t ₀ ”, as described above, the frame rate 1 / (4t ₀ ) “ The “low frequency component moving image data” is the frequency component data shown in FIG. 106, and the “middle low frequency component moving image data” at the frame rate 1 / (4t ₀ ) is the frequency component data shown in FIG. . Further, the frame rate 1 / a (4t ₀₎ "intermediate and high frequency components moving image data" is data of the frequency components shown in FIG. 108, the frame rate 1 / a (4t ₀₎ "high-frequency component video data" It is the data of the frequency component shown in FIG.

Therefore, the “low frequency component moving image data” of the frequency component shown in FIG. 106, the “middle low frequency component moving image data” of the frequency component shown in FIG. 107, and the “middle high frequency component moving image data” of the frequency component shown in FIG. , And the combined moving image data of the frame rate 1 / (4t ₀ ) obtained by combining the “high frequency component moving image data” of the frequency component shown in FIG. 109 is the data of the frequency component shown in FIG.

Then, with respect to the synthesized moving image data (the data moving at the speed “2r ₀ / t ₀ ”) at the frame rate 1 / (4t ₀ ) in FIG. 154, the filter unit 52 uses the region R _{1301 in} FIG. Filtering by a filter to be a pass band and upsampling four times are performed (step S269 in FIG. 145). Therefore, as shown in FIG. 155, the synthesized moving image data obtained by the filtering and upsampling is sampled at intervals of r ₀ for the spatial directions x and y, and sampled at intervals of t ₀ for the time direction t. Data.

That is, as shown in FIG. 155, on the frequency domain, aliasing components are generated at intervals of 2π / r _{0 in the} X and Y directions, and aliasing components are generated at intervals of 2π / t _{0 in the} T direction.

In FIG. 155, the shaded portion is a portion where there is data of a portion moving at a speed of about “r ₀ / t ₀ ” in the combined moving image data of the frame rate 1 / t _0. .

In the display device 3, only the data in the region R ₈₀₀₁ in FIG. 80 of the combined moving image data with the frame rate 1 / t ₀ shown in FIG. 155 is displayed. That is, in the display device 3 having the display capability represented by the region R ₈₀₀₁ in FIG. 80, only the data in the region indicated by the shadow in FIG. 156 is displayed, and the aliasing component shown in FIG. 155 is not displayed.

As described above, when the display device 3 has a frame rate of 1 / t ₀ , “low frequency component moving image data”, “middle low frequency component moving image data”, “middle and high frequency component moving image data”, and “ High-frequency component video data "is synthesized to obtain 1 / (4t ₀ ) frame rate composite video data, and upsampling is performed 4 times while filtering based on the principal component direction. By supplying the composite moving image data with the frame rate of / t ₀ to the display device 3, the display device 3 can appropriately display the frequency component data that can be displayed with the display capability without being affected by the aliasing component. Can be displayed.

146, 149, 152, and 155 are the same as FIGS. 35 to 38 described above, and therefore, the reception of FIG. 143 is the same as the case described in FIGS. When moving image data with a frame rate of 1 / t ₀ output by the device 2 is displayed on the display device 3, a human is supplied to the original moving image (supplied to the transmitting device 1) with respect to the displayed moving image. Image quality similar to that of a video).

Next, in the receiving device 2 in FIG. 24, the display device 3 is represented by any of the region R ₈₀₀₁ in FIG. 80, the region R ₈₁₀₁ in FIG. 81, the region R ₈₂₀₁ in FIG. 82, and the region R _{8301 in} FIG. Even if it has a display capability, it can be configured so that it can be dealt with.

  FIG. 157 shows a configuration example of such a receiving apparatus 2. In FIG. 157, portions corresponding to those in FIG. 33 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the receiving apparatus 2 in FIG. 157 is the same as in FIG. 33 except that an extracting unit 251 and a combining unit 252 are newly provided and a filter unit 253 is provided instead of the filter unit 52. It is configured.

  In the receiving device 2 of FIG. 157, the display capability (information indicating) of the display device 3 is supplied to the extracting unit 251, the combining unit 252, and the filter unit 253. Here, when the display device 3 has a function of supplying display capability, the display capability of the display device 3 is supplied from the display device 3 to the extraction unit 251, the synthesis unit 252, and the filter unit 253, for example. You can make it. In addition, the display capability of the display device 3 is, for example, from the operation unit to the extraction unit 251, the synthesis unit 252, and the filter unit 253 when the user operates the operation unit (not shown) of the reception device 2. It is also possible to supply them.

The extraction unit 251 selects and extracts a frequency component corresponding to the display capability of the display device 3 from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51, and supplies the frequency component to the synthesis unit 252.

That is, when the display device 3 has the display capability represented by the region R ₈₃₀₁ in FIG. 83, the extraction unit 251 extracts the “low frequency range” from the moving image data with the frame rate 1 / (4t ₀ ) stored in the buffer unit 51. “Component moving image data” is extracted and supplied to the synthesis unit 252.

When the display device 3 has the display capability represented by the region R ₈₂₀₁ in FIG. 82, the extraction unit 251 extracts the “low frequency” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51. “Component moving image data” and “middle and low range component moving image data” are extracted and supplied to the combining unit 252.

Furthermore, when the display device 3 has the display capability represented by the region R ₈₁₀₁ in FIG. 81, the extraction unit 251 extracts the “low frequency range” from the moving image data having the frame rate 1 / (4t ₀ ) stored in the buffer unit 51. “Component moving image data”, “middle and low range component moving image data”, and “middle and high range component moving image data” are extracted and supplied to the synthesis unit 252.

When the display device 3 has the display capability represented by the region R ₈₀₀₁ in FIG. 80, the extraction unit 251 extracts the “low frequency range” from the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51. All of “component moving image data”, “middle and low range component moving image data”, “middle and high range component moving image data”, and “high range component moving image data” are extracted and supplied to the synthesis unit 252.

  The combining unit 252 combines the frequency components supplied from the extraction unit 251 according to the display capability of the display device 3, and supplies the combined moving image data to the filter unit 253.

That is, when the display device 3 has the display capability represented by the region R ₈₃₀₁ in FIG. 83, the “low frequency component moving image data” having the frame rate 1 / (4t ₀ ) is transmitted from the extraction unit 251 to the synthesis unit 252. Since the image data is supplied, the synthesis unit 252 supplies the “low-frequency component moving image data” as it is to the filter unit 253 as synthesized image data.

If the display device 3 has the display capability represented by the region R ₈₂₀₁ in FIG. 82, the extraction unit 251 to the synthesis unit 252 send “low-frequency component moving image data” having a frame rate of 1 / (4t ₀ ) and Since “middle low-frequency component video data” is supplied, the synthesis unit 252 synthesizes the “low-frequency component video data” and “middle-low frequency component video data”, and filters the resultant synthesized image data as a filter. Supplied to the unit 253.

Further, when the display device 3 has the display capability represented by the region R ₈₁₀₁ in FIG. 81, the extraction unit 251 to the synthesis unit 252 send “low-frequency component moving image data” having a frame rate of 1 / (4t ₀ ), Since “middle low frequency component moving image data” and “middle high frequency component moving image data” are supplied, the synthesis unit 252 performs “low frequency component moving image data”, “middle low frequency component moving image data”, and “middle high frequency component video data”. The moving image data ”is synthesized, and the resultant synthesized image data is supplied to the filter unit 253.

Further, when the display device 3 has the display capability represented by the region R ₈₀₀₁ in FIG. 80, the “low frequency component moving image data” having the frame rate 1 / (4t ₀ ) is sent from the extraction unit 251 to the synthesis unit 252. Since “middle low frequency component video data”, “middle high frequency component video data”, and “high frequency component video data” are supplied, the synthesizer 252 performs the “low frequency component video data”, “middle low frequency component data”, The “moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” are combined, and the resultant combined image data is supplied to the filter unit 253.

The filter unit 253 uses the combined moving image data with the frame rate 1 / (4t ₀ ) supplied from the combining unit 232 to filter information from the filter information supply unit 62 in the same manner as in the filter unit 52 of FIG. Perform corresponding filtering and upsampling. However, the filter unit 253 performs multiple upsampling according to the display capability of the display device 3.

That is, when the display device 3 has the display capability represented by the region R ₈₃₀₁ in FIG. 83, the filter unit 253 performs upsampling by one time, that is, the frame rate 1 / (4t ₀ from the synthesis unit 252. ) Is output to the display device 3 without upsampling.

When the display device 3 has the display capability represented by the region R ₈₂₀₁ in FIG. 82, the filter unit 253 performs upsampling twice, and synthesizes the frame rate 1 / (2t ₀ ) obtained as a result. The moving image data is output to the display device 3.

Further, the display device 3, if having a display capability represented by region R ₈₁₀₁ in FIG. 81, the filter unit 253 performs the triple upsampling, the frame rate 1 / (4t _0/3) the resulting Are output to the display device 3.

Further, when the display device 3 has the display capability represented by the region R ₈₀₀₁ in FIG. 80, the filter unit 253 performs upsampling four times, and the resultant moving image data having the frame rate 1 / t ₀ obtained as a result. Is output to the display device 3.

As described above, in the receiving device 2 in FIG. 157, the display device 3 is any one of the region R ₈₀₀₁ in FIG. 80, the region R ₈₁₀₁ in FIG. 81, the region R ₈₂₀₁ in FIG. 82, and the region R _{8301 in} FIG. Even if it has the display capability represented by this, the suitable synthetic | combination moving image data according to the display capability can be output to the display apparatus 3. FIG.

Next, in the transmission device 1 in FIG. 79, for example, the basic band such as the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, the region R _{1301 in} FIG. By filtering the moving image data using the band limited by the area R _{8001 in} FIG. 80, the area R ₈₁₀₁ in FIG. 81, the area R ₈₂₀₁ in FIG. 82, and the area R ₈₃₀₁ in FIG. Although divided into “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” as described in FIGS. “Low-frequency component video data”, “Middle-low frequency component video data”, “Middle-high frequency component video data”, “High-frequency component video data” may also be data of the following frequency components, for example: it can.

That is, for example, moving image data of the portion stationary subjects in the moving image is being projected, that is, the data region R ₁₀₀₁ in FIG. 20, as in the case described above, the region R ₁₀₀₁ in FIG. 20 The data is “low frequency component video data”, and “middle low frequency component video data”, “middle and high frequency component video data”, and “high frequency component video data” can all be zero.

Further, for example, moving image data of a portion in which a subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2”, that is, data of region R _{1101 in} FIG. 84, “Low-frequency component moving image data” in region R _{8401 in} FIG. 84, “Middle and low-frequency component moving image data” in region R _{8501 in} FIG. 85, “Middle and high-frequency component moving image data” in region R _{8601 in} FIG. In place of “high frequency component moving image data” (0), for example, the following frequency component data are respectively converted into “low frequency component moving image data”, “middle low frequency component moving image data”, and “middle high frequency component data”. "Video data" and "high frequency component video data".

That is, as shown in FIG. 158, in the frequency domain, a straight line (plane) passing through the origin and extending in the principal component direction of the data in the region R _{1101 in} FIG. 21 is defined as L ₁₁₀₁ , and the straight line L ₁₁₀₁ and 1 / (4t the two intersections between the region R ₈₃₀₁ of Figure 83 that represents the display capability of a frame rate of _0), and P ₁₅₈₀₁ and P _{15 802.}

Further, as shown in FIG. 159, two intersection points of a straight line L ₁₁₀₁ and a region R _{8201 of} FIG. 82 representing the display capability of the frame rate of 1 / (2t ₀ ) are represented by P ₁₅₉₀₁ and P ₁₅₉₀₂ in the frequency domain. And

As shown in FIG. 160, straight lines (planes) passing through points P ₁₅₈₀₁ and P ₁₅₈₀₂ in FIG. 158 and points P ₁₅₉₀₁ and P ₁₅₉₀₂ in FIG. 159 and parallel to the T direction are respectively shown in L ₁₅₈₀₁ and L _15802. , L ₁₅₉₀₁ and L ₁₅₉₀₂ .

In this case, as shown in FIG. 161, the data in the region R _{1101 in} FIG. 21 _includes the data in the region R ₁₆₁₀₁ sandwiched between the straight lines L ₁₅₈₀₁ and L ₁₅₈₀₂ in the region R _1101. Data ".

Further, as for the data of the region R ₁₁₀₁ , as shown in FIG. 162, the region sandwiched between the straight lines L ₁₅₈₀₁ and L ₁₅₈₀₂ is _excluded from the region sandwiched between the straight lines L ₁₅₉₀₁ and L ₁₅₉₀₂ in the region R _1101. The data of the region R ₁₆₂₀₁ is “middle low-frequency component video data”, and “middle high-frequency component video data” and “high-frequency component video data” can both be zero.

In this case, the “low frequency component moving image data” in the region R _{8401 in} FIG. 84 is the data in the region R _{16101 in} FIG. 161, and the “middle low frequency component moving image data” in the region R _{8501 in} FIG. 85 is the region in FIG. In the data of R ₁₆₂₀₁ , “middle and high range component moving image data” in the region R ₈₆₀₁ in FIG.

Next, for example, moving image data of a portion of the moving image in which a subject projected on the moving image moves at a speed of about “r ₀ / t ₀ ”, that is, data of the region R ₁₂₀₁ in FIG. “Low-frequency component moving image data” in 87 region R ₈₇₀₁ , “Middle and low-frequency component moving image data” in region R _{8801 in} FIG. 88, “Middle and high-frequency component moving image data” in region R _{8901 in} FIG. 89, and region R in FIG. Instead of ₉₀₀₁ "High-frequency component video data", for example, the following frequency component data are respectively "Low-frequency component video data", "Middle-low frequency component video data", "Middle-high frequency component video data""," High-frequency component moving image data ".

That is, as shown in FIG. 163, in the frequency domain, a straight line (plane) passing through the origin and extending in the principal component direction of the data in the region R _{1201 in} FIG. 22 is L ₁₂₀₁ , and the straight line L ₁₂₀₁ and 1 / (4t Two intersection points with the region R _{8301 in} FIG. 83 representing the display capability of the frame rate of ₀ ) are P ₁₆₃₀₁ and P ₁₆₃₀₂ .

Further, as shown in FIG. 164, in the frequency domain, two intersection points of the straight line L ₁₂₀₁ and the region R _{8201 of} FIG. 82 representing the display capability of the frame rate of 1 / (2t ₀ ) are represented by P ₁₆₄₀₁ and P _16402. And

Furthermore, as shown in FIG. 165, in the frequency domain, the straight line L _1201, the two intersections between the region R ₈₁₀₁ of Figure 81 that represents the display capability of a frame rate of _{1 / (4t 0/3)} , and P ₁₆₅₀₁ P ₁₆₅₀₂ .

Also, as shown in FIG. 166, in the frequency domain, two intersections of the straight line L ₁₂₀₁ and the region R _{8001 in} FIG. 80 representing the display capability of the 1 / t ₀ frame rate are P ₁₆₆₀₁ and P ₁₆₆₀₂ . .

Then, as shown in FIG. 167, point P _16301, P ₁₆₃₀₂ in FIG 163, the point P _16401, P ₁₆₄₀₂ in FIG 164, the point P _16501, P ₁₆₅₀₂ in FIG 165, the point P _16601, P ₁₆₆₀₂ in FIG 166 respectively The straight lines (planes) parallel to the T direction are L ₁₆₃₀₁ , L ₁₆₃₀₂ , L ₁₆₄₀₁ , L ₁₆₄₀₂ , L ₁₆₅₀₁ , L ₁₆₅₀₂ , L ₁₆₆₀₁ , and L ₁₆₆₀₂ , respectively.

In this case, as shown in FIG. 168, for the data in the region R _{1201 in} FIG. 22, the data in the region R ₁₆₈₀₁ between the straight lines L ₁₆₃₀₁ and L ₁₆₃₀₂ in the region R ₁₂₀₁ Data ".

Further, for the data of the area R _1201, except as shown in Figure 169, of the region R _1201, a region sandwiched by the straight line L ₁₆₄₀₁ and L _16402, the region sandwiched by the straight line L ₁₆₃₀₁ and L ₁₆₃₀₂ The data in the region R ₁₆₉₀₁ can be “middle and low range component moving image data”.

Also, the data of the area R _1201, except as shown in Figure 170, of the region R _1201, a region sandwiched by the straight line L ₁₆₅₀₁ and L _16502, the region sandwiched by the straight line L ₁₆₄₀₁ and L ₁₆₄₀₂ The data in the region R ₁₇₀₀₁ can be “middle and high range component moving image data”.

Furthermore, as for the data of the region R ₁₂₀₁ , as shown in FIG. 171, the region sandwiched between the straight lines L ₁₆₅₀₁ and L ₁₆₅₀₂ is _excluded from the region sandwiched between the straight lines L ₁₆₆₀₁ and L ₁₆₆₀₂ in the region R _1201. The data in the region R ₁₇₁₀₁ can be “high frequency component moving image data”.

In this case, the “low frequency component moving image data” in the region R _{8701 in} FIG. 87 is the data in the region R _{16801 in} FIG. 168, and the “middle low frequency component moving image data” in the region R _{8801 in} FIG. 88 is the region in FIG. In the data of R ₁₆₉₀₁ , the “middle and high frequency component moving image data” in the region R _{8901 in} FIG. 89 is the data in the region R _{17001 in} FIG. 170, and the “high frequency component moving image data” in the region R ₉₀₀₁ in FIG. _Are approximated with the data of the region R ₁₇₁₀₁ .

Next, for example, moving image data of a portion of the moving image in which a subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, that is, data in the region R ₁₃₀₁ in FIG. 91, “Low-frequency component moving image data” in region R ₉₁₀₁ , “Middle and low-frequency component moving image data” in region R _{9201 in} FIG. 92, “Middle and high-frequency component moving image data” in region R _{9301 in} FIG. 93, and region R in FIG. _Instead of ₉₄₀₁ “high-frequency component video data”, for example, the following frequency component data are respectively “low-frequency component video data”, “middle-low-frequency component video data”, and “middle-high-frequency component video data”. "," High-frequency component moving image data ".

That is, as shown in FIG. 172, in the frequency domain, a straight line (plane) passing through the origin and extending in the principal component direction of the data in the region R _{1301 in} FIG. 23 is L ₁₃₀₁ , and the straight line L ₁₃₀₁ and 1 / (4t Two intersection points with the region R _{8301 in} FIG. 83 representing the display capability of the frame rate of ₀ ) are P ₁₇₂₀₁ and P ₁₇₂₀₂ .

Further, as shown in FIG. 173, in the frequency domain, two intersection points of a straight line L ₁₃₀₁ and a region R _{8201 of} FIG. 82 representing the display capability of the frame rate of 1 / (2t ₀ ) are represented by P ₁₇₃₀₁ and P _17302. And

Furthermore, as shown in FIG. 174, in the frequency domain, the straight line L _1301, the two intersections between the region R ₈₁₀₁ of Figure 81 that represents the display capability of a frame rate of _{1 / (4t 0/3)} , and P ₁₇₄₀₁ P ₁₇₄₀₂ .

Also, as shown in FIG. 175, in the frequency domain, two intersections of the straight line L ₁₃₀₁ and the region R _{8001 in} FIG. 80 representing the display capability of the frame rate of 1 / t ₀ are _designated as P ₁₇₅₀₁ and P ₁₇₅₀₂ . .

Then, as shown in FIG. 176, point P _17201, P ₁₇₂₀₂ in FIG 172, the point P _17301, P ₁₇₃₀₂ in FIG 173, the point P _17401, P ₁₇₄₀₂ in FIG 174, the point P _17501, P ₁₇₅₀₂ in FIG 175 respectively The straight lines (planes) parallel to the T direction are _denoted by L ₁₇₂₀₁ , L ₁₇₂₀₂ , L ₁₇₃₀₁ , L ₁₇₃₀₂ , L ₁₇₄₀₁ , L ₁₇₄₀₂ , L ₁₇₅₀₁ , and L ₁₇₅₀₂ , respectively.

In this case, as shown in FIG. 177, for the data in the region R _{1301 in} FIG. 23, the data in the region R ₁₇₇₀₁ between the straight lines L ₁₇₂₀₁ and L ₁₇₂₀₂ in the region R ₁₃₀₁ Data ".

Furthermore, as for the data of region R ₁₃₀₁ , as shown in FIG. 178, the region sandwiched between straight lines L ₁₇₂₀₁ and L ₁₇₂₀₂ is _excluded from the region sandwiched between straight lines L ₁₇₃₀₁ and L ₁₇₃₀₂ in region R _1301. The data of the region R ₁₇₈₀₁ can be “middle and low range component moving image data”.

As for the data of the region R ₁₃₀₁ , as shown in FIG. 179, the region sandwiched between the straight lines L ₁₇₃₀₁ and L ₁₇₃₀₂ is _excluded from the region sandwiched between the straight lines L ₁₇₄₀₁ and L ₁₇₄₀₂ in the region R _1301. The data in the region R ₁₇₉₀₁ can be “middle and high range component moving image data”.

Furthermore, for the data of region R ₁₃₀₁ , as shown in FIG. 180, the region sandwiched between straight lines L ₁₇₅₀₁ and L ₁₇₅₀₂ in the region R ₁₃₀₁ is _excluded from the region sandwiched between straight lines L ₁₇₄₀₁ and L _{17402. Focus on} the data in the region RR ₁₈₀₀₁ . In this case, since data also exists in the region RRR ₁₈₀₀₁ excluding the region sandwiched between the straight lines L ₁₇₅₀₁ and L ₁₇₅₀₂ in the region R ₁₃₀₁ , the “high frequency component moving image data” is the region RR. _The region R ₁₈₀₀₁ is a combination of the two regions ₁₈₀₀₁ and RRR ₁₈₀₀₁ . In other words, in any case, the “high-frequency component video data” includes three of the basic bands, “low-frequency component video data”, “middle-low frequency component video data”, and “middle-high frequency component video data”. It may be said that it is the data of the area excluding one data.

In this case, the “low frequency component moving image data” in the region R _{9101 in} FIG. 91 is the data in the region R _{17701 in} FIG. 177, and the “middle low frequency component moving image data” in the region R _{9201 in} FIG. 92 is the region in FIG. In the data of R ₁₇₈₀₁ , the “middle and high frequency component moving image data” in the region R _{9301 in} FIG. 93 is the data in the region R _{17901 in} FIG. 179, and the “high frequency component moving image data” in the region R ₉₄₀₁ in FIG. _Are approximated by the data of the region R ₁₈₀₀₁ .

  When the “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” described with reference to FIGS. The moving image data (synthetic moving image data) supplied to the apparatus 3 may include a folding component, but even if the folding component is included, the image quality is not so much deteriorated.

  Further, when the “low frequency component video data”, “middle low frequency component video data”, “middle high frequency component video data”, and “high frequency component video data” described with reference to FIGS. Compared with the case where “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” described in FIGS. Can be reduced or the circuit scale can be reduced.

That is, when “low frequency component moving image data”, “middle low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” described in FIGS. The region R ₁₁₀₁ shown in FIG. 21, the region R ₁₂₀₁ shown in FIG. 22, and the region R ₁₃₀₁ shown in FIG. 23 are filtered in the pass band of the region divided in the direction orthogonal to the T direction as shown in FIGS. It is necessary to perform such filtering on a frame other than the frame of interest in addition to the frame of the block of interest (frame of interest). Therefore, in this case, it is necessary to store frames other than the target frame (for example, one or more frames before and after the target block), and a frame memory for that purpose is required. That is, filtering in the time direction is necessary, which means that filtering processing using a frame other than the frame of interest is necessary.

On the other hand, when the “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” described in FIGS. 21 corresponds to the region R _{1101 in} FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 as shown in FIG. 161, FIGS. 162, 168 to 171 and 177 to 180, respectively. It is only necessary to filter the passband of the region divided in the direction parallel to the direction, and the filtering need only be performed on the frame of interest. Therefore, in this case, it is not necessary to store frames other than the frame of interest, and a frame memory for that purpose is also unnecessary. As a result, the memory capacity can be reduced or the circuit scale can be reduced. That is, it is only necessary to perform filtering in the spatial direction, that is, filtering processing using only the pixels in the frame of interest, so that processing that spans between frames is unnecessary.

  The video data is divided into “low frequency component video data”, “middle and low frequency component video data”, “middle and high frequency component video data”, and “high frequency component video data” described with reference to FIGS. Alternatively, when dividing into “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” described in FIGS. The region on the frequency domain where each of the “component video data”, “middle and low frequency component video data”, “middle and high frequency component video data”, and “high frequency component video data” is distributed is the magnitude of movement of the subject in the video data ( Speed).

  That is, in the video data, for example, as a part to be divided into “low frequency component video data”, “middle low frequency component video data”, “middle high frequency component video data”, “high frequency component video data” When the motion vector of a block is small, most of the region on the frequency domain where the data of the block of interest is distributed is the region of “low-frequency component video data”, and almost the region of “high-frequency component video data” 0. Then, the larger the motion vector of the block of interest is, the more “high-frequency component moving image data” area is.

Specifically, for example, when the motion vector of the block of interest with respect to the next frame is approximately 0, that is, when a stationary object in the moving image is projected onto the block of interest, A region on the frequency domain in which the data of the block of interest is distributed, that is, for example, all of the region R _{1001 in} FIG. 20 is a region of “low-frequency component moving image data”.

Further, for example, the block of interest, if the magnitude of the motion vector for the next frame is about r _0/2, i.e., in the block of interest, subject velocity _{"(r 0 / t 0) /} 2 " moves to the extent If it is, the region on the frequency domain data of the block of interest is distributed, i.e., for example, of the region R ₁₁₀₁ in FIG. 21, most of such regions R ₁₆₁₀₁ in the region R ₈₄₀₁ and Figure 161 in Figure 84 Is the “low-frequency component moving image data” region, and the region R _{8501 in} FIG. 85, the region R _{16201 in} FIG. 162, etc. is the “middle low-frequency component moving image data” region. Then, a slight region such as the region R _{8601 in} FIG. 86 becomes the region of “middle and high range component moving image data”.

Further, for example, when the size of the motion vector of the block of interest for the next frame is about r _0, that is, when the subject is moving at a speed of about “r ₀ / t ₀ ” in the block of interest, A region on the frequency domain in which the data of the block of interest is distributed, that is, for example, region R _{8701 in} FIG. 87 and region R _{16801 in} FIG. 168 in region R ₁₂₀₁ in FIG. The region R _{8801 in} FIG. 88, the region R _{16901 in} FIG. 169, and the like become “middle and low range component moving image data”. The region R _{8901 in} FIG. 89, the region R _{17001 in} FIG. 170, and the like become the “middle and high frequency component moving image data” region, and the region R _{9001 in} FIG. 90 and the region R _{17101 in} FIG. Data "area.

Further, for example, when the size of the motion vector of the block of interest for the next frame is larger, for example, about 2r _0, that is, the subject moves at a speed of about “2r ₀ / t ₀ ” in the block of interest. If you are at a region on the frequency domain data of the block of interest is distributed, i.e., for example, of the region R ₁₃₀₁ in FIG. 23, and a region R ₁₇₇₀₁ in the region R ₉₁₀₁ and Figure 177 in FIG. 91, " The region R _{9201 in} FIG. 92, the region R _{17801 in} FIG. 178, and the like become “middle and low range component moving image data”. Then, the region R _{9301 in} FIG. 93 and the region R _{17901 in} FIG. 179 become the “middle and high frequency component moving image data” region, and the region R _{9401 in} FIG. 94 and the region R _{18001 in} FIG. Data "area.

  In the above case, the moving image data is divided according to the four display capabilities described with reference to FIGS. 80 to 83. However, the moving image data may be divided according to other display capabilities. Is possible.

  Furthermore, the number of divisions of moving image data is limited to the above-mentioned “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data”. It may be two, three, or five or more.

  In addition, the number of divisions of the moving image data and the display capability for dividing the moving image data can be set in advance according to the user's operation, for example. You can also.

Further, in FIGS. 95 to 97, filtering that passes frequency components in the basic band of the moving image data, and “low frequency component moving image data”, “middle low frequency component moving image data”, “middle and high frequencies” from the moving image data. The component moving image data "or the filtering for dividing the" high frequency component moving image data "is performed at the same time. That is, for example, in step S203 of FIG. “Low frequency component moving image data” is obtained by applying a filter that allows the frequency component to pass and the frequency component of region R _{8301 in} FIG. 83 to pass. However, within the basic band of the moving image data, Filtering to pass the frequency component of the video, and from the video data "low frequency component video data", "middle low frequency component video data", "middle high frequency component video data Or filtering to split the "high-frequency component video data" may be performed separately. That is, first, filtering is performed to pass the frequency component in the basic band of the moving image data, and the filtering result is set as “low frequency component moving image data”, “middle low frequency component moving image data”, “middle high frequency range”. It is possible to perform filtering that divides “component moving image data” or “high frequency component moving image data”.

  Next, in the transmission device 1 of FIG. 79, the moving image data is converted into “low frequency component moving image data”, “middle low frequency component moving image data”, “medium / high frequency” according to the four display capabilities described in FIGS. Since the display device 3 (FIG. 24) is divided into four display capabilities described with reference to FIGS. In the receiving device 2 (FIG. 24), “low frequency component moving image data”, “middle low frequency component moving image data”, “middle and high frequency component moving image data”, according to the display capability of the display device 3, By synthesizing the “high frequency component moving image data”, the display device 3 can appropriately display the moving image data as described above.

  However, when the display device 3 (FIG. 24) does not have any of the four display capabilities described with reference to FIGS. 80 to 83, the display device 3 appropriately displays the moving image data. It becomes difficult to display.

Further, unlike the transmission device 1 of FIG. 79, the moving image data is not necessarily divided into a plurality of frequency components. In such a case, it becomes difficult for the display device 3 that has only a display capability of a frame rate lower than the frame rate of 1 / t ₀ to perform appropriate display. In other words, unless the transmission device 1 divides and transmits the moving image data, the receiving device 2 cannot generate appropriate moving image data according to the display capability of the display device 3.

  Therefore, the receiving device 2 (FIG. 24) needs to be improved so that moving image data corresponding to the display capability of the display device 3 can be generated and supplied to the display device 3.

Hereinafter, such a receiving device 2 will be described, but before that, moving image data corresponding to the display capability of the display device 3 will be described. Here, for example, in the receiving device 2, moving image data (encoded data) having a frame rate of 1 / (4t ₀ ) output from the transmitting device 1 in FIG. 25 is recorded on the recording medium 11 (FIG. 24) or It is assumed that it is supplied via the transmission medium 12. The receiving device 2 includes a plurality of frequency components such as “low frequency component moving image data”, “middle and low frequency component moving image data”, “middle and high frequency component moving image data”, and “high frequency component moving image data” as described above. However, in this case, the receiving device 2 combines the plurality of frequency components and performs processing on the moving image data obtained as a result of the combination.

For example, when the display device 3 has the display capability described in FIG. 83, that is, when the display device 3 has a frame rate of 1 / (4t ₀ ), the display device 3 stores the data in the region R ₈₃₀₁ in FIG. Can be displayed.

Then, for example, in the moving image data from the transmission apparatus 1, for a portion where a stationary subject is projected, the data exists in the region R ₁₀₀₁ in FIG. 20, as shown in FIG. Data in a region R _{18101 in which} the region R ₁₀₀₁ and the region R _{8301 in} FIG. 83 overlap may be supplied to the display device 3.

Further, for example, in the moving image data from the transmission apparatus 1, for a portion where the subject projected thereon is moving at a speed of about “(r ₀ / t ₀ ) / 2”, an area R _{1101 in} FIG. Therefore, as shown in FIG. 182, the data of the region R ₁₈₂₀₁ where the region R ₁₁₀₁ and the region R _{8301 of} FIG. 83 overlap may be supplied to the display device 3.

Further, in the moving image data from the transmission apparatus 1, for example, for a portion where the subject projected thereon moves at a speed of about “r ₀ / t ₀ ”, the data exists in the region R ₁₂₀₁ in FIG. Therefore, as shown in FIG. 183, the data of the region R ₁₈₃₀₁ in which the region R ₁₂₀₁ and the region R _{8301 in} FIG. 83 overlap may be supplied to the display device 3.

Further, for example, in the moving image data from the transmission apparatus 1, data exists in the region R ₁₃₀₁ in FIG. 23 for a portion where the subject projected thereon moves at a speed of about “2r ₀ / t ₀ ”. Therefore, as shown in FIG. 184, data in a region R ₁₈₄₀₁ in which the region R ₁₃₀₁ and the region R _{8301 in} FIG. 83 overlap may be supplied to the display device 3.

  Hereinafter, the same applies to the portion of the subject moving at other speeds in the moving image data from the transmission device 1.

Next, for example, when the display device 3 has the display capability described with reference to FIG. 82, that is, when the display device 3 has a frame rate of 1 / (2t ₀ ), the display device 3 includes the region R ₈₂₀₁ in FIG. Can be displayed.

Then, for example, in the moving image data from the transmission apparatus 1, for a portion where a stationary subject is projected, data exists in the region R ₁₀₀₁ in FIG. 20, as shown in FIG. 185, Data in a region R _{18501 in which} the region R ₁₀₀₁ and the region R _{8201 in} FIG. 82 overlap may be supplied to the display device 3.

Further, for example, in the moving image data from the transmission apparatus 1, for a portion where the subject projected thereon is moving at a speed of about “(r ₀ / t ₀ ) / 2”, an area R _{1101 in} FIG. Therefore, as shown in FIG. 186, the data of the region R ₁₈₆₀₁ where the region R ₁₁₀₁ overlaps the region R _{8201 of} FIG. 82 may be supplied to the display device 3.

Further, in the moving image data from the transmission apparatus 1, for example, for a portion where the subject projected thereon moves at a speed of about “r ₀ / t ₀ ”, the data exists in the region R ₁₂₀₁ in FIG. Therefore, as shown in FIG. 187, data in a region R ₁₈₇₀₁ in which the region R ₁₂₀₁ and the region R _{8201 in} FIG. 82 overlap may be supplied to the display device 3.

Further, for example, in the moving image data from the transmission apparatus 1, data exists in the region R ₁₃₀₁ in FIG. 23 for a portion where the subject projected thereon moves at a speed of about “2r ₀ / t ₀ ”. Therefore, as shown in FIG. 188, data in a region R ₁₈₈₀₁ in which the region R ₁₃₀₁ and the region R _{8201 in} FIG. 82 overlap may be supplied to the display device 3.

  Hereinafter, the same applies to the portion of the subject moving at other speeds in the moving image data from the transmission device 1.

Then, for example, the display device 3, if having a display capability described in FIG. 81, i.e., in a case having a frame rate of 1 / (4t _0/3) is, in the display device 3, a region in FIG. 81 R Data in ₈₁₀₁ can be displayed.

Then, for example, in the moving image data from the transmission apparatus 1, for a portion where a stationary subject is projected, data exists in the region R ₁₀₀₁ in FIG. 20, as shown in FIG. Data in a region R _{18901 in which} the region R ₁₀₀₁ and the region R _{8101 in} FIG. 81 overlap may be supplied to the display device 3.

Further, for example, in the moving image data from the transmission apparatus 1, for a portion where the subject projected thereon is moving at a speed of about “(r ₀ / t ₀ ) / 2”, an area R _{1101 in} FIG. Therefore, as shown in FIG. 190, data in a region R _{19001 in which} the region R ₁₁₀₁ and the region R _{8101 in} FIG. 81 overlap may be supplied to the display device 3.

Further, in the moving image data from the transmission apparatus 1, for example, for a portion where the subject projected thereon moves at a speed of about “r ₀ / t ₀ ”, the data exists in the region R ₁₂₀₁ in FIG. Therefore, as shown in FIG. 191, the data of a region R ₁₉₁₀₁ in which the region R ₁₂₀₁ and the region R _{8101 in} FIG.

Further, for example, in the moving image data from the transmission apparatus 1, data exists in the region R ₁₃₀₁ in FIG. 23 for a portion where the subject projected thereon moves at a speed of about “2r ₀ / t ₀ ”. Therefore, as shown in FIG. 192, data of a region R ₁₉₂₀₁ in which the region R ₁₃₀₁ and the region R _{8101 in} FIG. 81 overlap may be supplied to the display device 3.

  Hereinafter, the same applies to the portion of the subject moving at other speeds in the moving image data from the transmission device 1.

Next, for example, when the display device 3 has the display capability described in FIG. 80, that is, when the display device 3 has a frame rate of 1 / t ₀ , the display device 3 uses the data in the region R ₈₀₀₁ in FIG. Can be displayed.

Then, for example, in the moving image data from the transmission apparatus 1, for a portion where a stationary subject is projected, data exists in the region R ₁₀₀₁ in FIG. 20, as shown in FIG. 193, Data in a region R _{19301 in which} the region R ₁₀₀₁ and the region R _{8001 in} FIG. 80 overlap may be supplied to the display device 3.

Further, for example, in the moving image data from the transmission apparatus 1, for a portion where the subject projected thereon is moving at a speed of about “(r ₀ / t ₀ ) / 2”, an area R _{1101 in} FIG. Therefore, as shown in FIG. 194, the data of the region R ₁₉₄₀₁ where the region R ₁₁₀₁ overlaps the region R _{8001 of} FIG. 80 may be supplied to the display device 3.

Further, in the moving image data from the transmission apparatus 1, for example, for a portion where the subject projected thereon moves at a speed of about “r ₀ / t ₀ ”, the data exists in the region R ₁₂₀₁ in FIG. Therefore, as shown in FIG. 195, the data of a region R ₁₉₅₀₁ in which the region R ₁₂₀₁ and the region R _{8001 of} FIG.

Further, for example, in the moving image data from the transmission apparatus 1, data exists in the region R ₁₃₀₁ in FIG. 23 for a portion where the subject projected thereon moves at a speed of about “2r ₀ / t ₀ ”. Therefore, as shown in FIG. 196, the data of a region R ₁₉₆₀₁ in which the region R ₁₃₀₁ and the region R _{8001 of} FIG.

  Hereinafter, the same applies to the portion of the subject moving at other speeds in the moving image data from the transmission device 1.

  From the above, the following moving image data may be supplied to the display device 3 according to the display capability.

That is, when the frame rate as the display capability of the display device 3 is expressed as 1 / (d × t ₀ ) using the variable d, the display capability is expressed in the following region R _B on the frequency domain. Is done.

Region R _B = {(X, Y, T) | −π / r ₀ ≦ X ≦ π / r ₀ , and −π / r ₀ ≦ Y ≦ π / r ₀ ,
And −π / (d × t ₀ ) ≦ T ≦ π / (d × t ₀ )}

Further, regions on the frequency domain where data of each part of the moving image data from the transmission apparatus 1 exists, that is, for example, region R _{1001 in} FIG. 20, region R ₁₁₀₁ in FIG. 21, region R ₁₂₀₁ in FIG. As described above, the region R ₁₃₀₁ of 23 is a region having a width in the T direction of 2π / (4t ₀ ) extending in the main component direction of the data of each part of the moving image data. As described above, the principal component direction of the data is a three-dimensional data obtained by adding the frame period as a component in the time direction t to the motion vector (two-dimensional motion vector) in the spatial direction x, y of the data. Orthogonal to the motion vector.

Now, assuming that the motion vector in the spatial direction x, y of the data is (u ₀ , v ₀ ), the frame period of the moving image data from the transmission apparatus 1 is 4t ₀ , so the three-dimensional motion vector is (u ₀ , v ₀ , 4t ₀ ). The principal component direction is orthogonal to the motion vector (u ₀ , v ₀ , 4t ₀ ).

Therefore, the region on the frequency domain where the data of each part of the moving image data from the transmission device 1 exists (exactly, ± π / (4t in the T direction from the straight line (plane) passing through the origin and extending in the principal component direction). The region in the range of ₀ ) is represented by the following region R _C.

Through the region R _C = {point (0,0, -π / (4t ₀ ))
A plane perpendicular to the motion vector (u ₀ , v ₀ , 4t ₀ ),
Through the point (0,0, π / (4t ₀ ))
Region surrounded by a plane perpendicular to the motion vector (u ₀ , v ₀ , 4t ₀ )}

The display device 3 may be supplied with moving image data in a region R _A (= R _B ∩R _C ) where the regions R _B and R _C overlap. Here, the region on the frequency domain of the moving image data to be supplied to the display device 3 is hereinafter referred to as a supply region as appropriate. Region _RA is a supply region.

FIG. 197 shows a configuration example of the receiving device 2 of FIG. 24 that obtains moving image data in the supply region _RA and supplies the moving image data to the display device 3.

  In FIG. 197, portions corresponding to those in FIG. 33 are denoted by the same reference numerals, and description thereof will be omitted as appropriate. That is, the receiving apparatus 2 in FIG. 197 is configured in the same manner as in FIG. 33 except that a filter unit 301 and a filter information supply unit 302 are provided instead of the filter unit 52 and the filter information supply unit 62, respectively. ing.

  In FIG. 197, the display capability (information indicating) of the display device 3 is supplied to the filter unit 301 and the filter information supply unit 302. Here, when the display device 3 has a function of supplying display capability, the display capability of the display device 3 is supplied from the display device 3 to the filter unit 301 and the filter information supply unit 302, for example. can do. The display capability of the display device 3 is supplied from the operation unit to the filter unit 301 and the filter information supply unit 302 by, for example, having the user operate the operation unit (not shown) of the reception device 2. It is also possible to do so.

The filter information supply unit 302 obtains the supply region _RA from the principal component direction from the principal component direction acquisition unit 61 and the display capability of the display device 3.

That is, the filter information supply unit 302 is an area extending in the principal component direction in the frequency domain based on the information on the principal component direction from the principal component direction acquisition unit 61, similarly to the filter information supply unit 62 in FIG. The above-mentioned region R _C (for example, the region R _{1001 in} FIG. 20 or the region R _{1101 in} FIG. 21) having a specific width in the direction of the frequency axis T in the time direction, for example, 2π / (4t ₀ ). A region in which the principal component direction is not limited, such as a region R ₁₂₀₁ in FIG. 22 and a region R _{1301 in} FIG.

Here, in order to determine the region R _C, it is necessary motion vector (u _0, v _0), the motion vector (u _0, v _0), recognizing the major component direction from the main component direction acquisition unit 61 can do. As described above, the principal component direction acquisition unit 61 can obtain the principal component direction from the motion vector. In this case, the motion vector (two-dimensional or three-dimensional motion is obtained without obtaining the principal component direction. Vector) can be supplied to the filter information supply unit 302 as it is.

Filter information supply section 302 further determines the region R _B representing the display capability of the display device 3, and the region R _B, a region where the region R _C obtained previously overlapped, obtains a supply region R _A. The region R _B is, as a display capability of the display device 3, for example, it can be obtained from the display device 3 of the frame rate _{1 / (d × t 0)} .

Filter information supply unit 302, when obtaining the feed region R _A As described above, the supply region R _A, was determined as the passing band of the filter, the passband as the filter information, and outputs to the filter section 301 .

The filter unit 301 uses the moving image data of the frame rate 1 / (4t ₀ ) stored in the buffer unit 51 and corresponds to the filter information from the filter information supply unit 302 as in the filter unit 52 of FIG. Filtering and upsampling. However, the filter unit 301 performs multiple upsampling according to the display capability of the display device 3.

That is, the filter unit 301 performs upsampling of 4 / d times in accordance with, for example, the frame rate 1 / (d × t ₀ ) of the display device 3 as the display capability of the display device 3. Accordingly, the filter unit 301 obtains moving image data having a frame rate of 1 / (d × t ₀ ) from moving image data having a frame rate of 1 / (4t ₀ ) and supplies the moving image data to the display device 3.

  Next, processing of the receiving device 2 in FIG. 197 will be described with reference to the flowchart in FIG. 198.

The decoding unit 50 is supplied with encoded data. The decoding unit 50 decodes the encoded data into moving image data with a frame rate of 1 / (4t ₀ ) and a low frame rate, and supplies it to the buffer unit 51. The buffer unit 51 sequentially stores moving image data having a low frame rate of 1 / (4t ₀ ) supplied from the decoding unit 50.

  In step S301, the filter generation unit 53 selects a certain portion of the moving image data stored in the buffer unit 51, that is, a block of 16 × 16 pixels as described in step S21 in FIG. Data is read out and input to itself (filter generation unit 53), and the process proceeds to steps S302 to S304 in sequence.

In steps S302 to S304, the filter generation unit 53 (the principal component direction acquisition unit 61 and the filter information supply unit 302) can recognize the data input in step S301 on the frequency domain, and _A supply area _RA that is an area of data that can be displayed on the display device 3 is obtained as a filter pass band, and the pass band is supplied to the filter unit 301 as filter information.

  That is, in step S302, the principal component direction acquisition unit 61 of the filter generation unit 53 performs the frequency axes X and Y in the spatial directions x and y and the frequency axis T in the time direction t for the data of the part input in step S301. The principal component direction in the frequency domain defined by is acquired, supplied to the filter information supply unit 302, and the process proceeds to step S303.

  In step S302, the principal component direction can be obtained by performing a three-dimensional Fourier transform as described in FIG. 31, or can be obtained by detecting a motion vector as described in FIG. it can. Further, as described above, when the principal component direction (information thereof) is multiplexed in the encoded data from the transmission device 1, the decoding unit 50 separates the principal component direction from the encoded data, The direction acquisition unit 61 can also acquire the principal component direction by receiving the supply of the principal component direction from the decoding unit 50.

Further, as described in step S32 in FIG. 34, encoded data in the spatial direction x, in the case that contains the y motion vector (u _0, v _0), the motion vector (u _0, v _0), A component t in the time direction (here, t = 4t ₀ ) is added to obtain a three-dimensional motion vector (u ₀ , v ₀ , t) in the spatial direction x, y and the time direction t, and the three-dimensional motion vector ( The direction orthogonal to u ₀ , v ₀ , t) can be the principal component direction.

  Here, in step S302, the principal component direction acquisition unit 61 may supply a motion vector instead of the principal component direction to the filter information supply unit 302.

In step S303, the filter information supply unit 302 is based on the motion vector supplied from the principal component direction acquisition unit 61 or the motion vector obtained from the principal component direction, that is, a data region R _C that can be recognized by humans, that is, For example, the region R _{1001 in} FIG. 20, the region R ₁₁₀₁ in FIG. 21, the region R ₁₂₀₁ in FIG. 22, and the region R _{1301 in} FIG. 23 (regions in which the principal component direction is not limited) are obtained. Furthermore, filter information supply section 302, based on the frame rate 1 / (d × t ₀₎ as the display capability of the display device 3, obtains the region R _B representing the display capability. Then, the filter information supply unit 302 obtains a region where the regions R _B and R _C overlap as the supply region R _A , determines the supply region R _A as a pass band of the filter, and proceeds from step S303 to S304. .

  In step S304, the filter information supply unit 302 supplies the filter passband (information indicating) obtained in step S303 to the filter unit 301 as filter information, and the process proceeds to step S305.

  In step S305, the filter unit 301 reads the moving image data from the buffer unit 51, and applies the filter of the passband represented by the filter information supplied from the filter generation unit 53 in step S304 to the data while moving in the time direction. Perform upsampling to increase the number of samples to 4 / d times.

The filter unit 301 obtains moving image data with a frame rate of 1 / (d × t ₀ ) by performing filtering and upsampling, and proceeds from step S305 to S306.

In step S306, the filter unit 301 outputs the video data of the frame rate 1 / afford (d × t ₀₎ at step S305, the frame rate to 1 / (d × t ₀₎ of the display device 3 (FIG. 24) To do.

As described above, the moving image data is obtained by the filter using the supply region R _{A in} which the region R _C that can be recognized by humans on the frequency domain and the region R _B representing the display capability of the display device 3 overlap each other. Since filtering and further upsampling according to the display capability of the display device 3 are performed, the display device is obtained from the low frame rate moving image data in which the data amount is reduced in consideration of the visual effect of the human eye. Thus, moving image data suitable for the display capability 3 can be obtained and displayed.

  In this way, by adopting the configuration on the receiving side as described with reference to FIGS. 181 to 198, moving image data divided into a plurality of frequency components as described with reference to FIGS. 79 to 109 (for example, “ Even when the low-frequency component moving image data, “middle and low-frequency component moving image data”, “middle and high-frequency component moving image data” and “high-frequency component moving image data” are not transmitted from the transmission device 1, By restoring only the optimum frequency component with the receiving device 2 (the device shown in FIG. 197), it is possible to display the optimum moving image data on the display device 3.

Next, FIG. 199 shows another configuration example of the receiving device 2 of FIG. 24 that obtains moving image data in the supply region _RA and supplies the moving image data to the display device 3.

  In FIG. 199, portions corresponding to those in FIG. 197 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, in the receiving apparatus 2 of FIG. 199, the decoding unit 50 is configured by the MPEG decoder 311 and the filter information supply unit 312, and the filter generation unit 53 is not provided, as in FIG. 197. It is configured.

In the decoding unit 50 of the receiving device 2 in FIG. 199, encoded data obtained by MPEG encoding the moving image data of the frame rate 1 / (4t ₀ ) in the encoding unit 24 of the transmitting device 1 in FIG. 25, for example. The recording medium 11 (FIG. 24) or the transmission medium 12 is supplied.

In the decoding unit 50, the MPEG decoder 311 performs MPEG decoding on the encoded data supplied to the decoding unit 50 based on the filter information supplied from the filter information supply unit 312, and the frame rate 1 / (4t ₀₎ obtained as a result. ) Is supplied to the buffer unit 51.

The filter information supply unit 312 is supplied with the frame rate 1 / (d × t ₀ ) of the display device 3 as the display capability of the display device 3. Here, when the display device 3 has a function of supplying display capability, the display capability of the display device 3 can be supplied from the display device 3 to the filter information supply unit 312, for example. . Further, the display capability of the display device 3 is supplied from the operation unit to the filter information supply unit 312 when the user operates the operation unit (not shown) of the reception device 2, for example. It is also possible. The display capability of the display device 3 is supplied not only to the filter information supply unit 312 but also to the filter unit 301.

Filter information supply section 312, based on the frame rate 1 / display device 3 as the display capability of the display device 3 (d × t _0), obtains the region R _B. Further, the filter information supply unit 312 acquires a motion vector used by the MPEG decoder 311 to MPEG-decode encoded data, and obtains a region _RC based on the motion vector. The filter information supply section 312 obtains a supply region R _A is a region where the region R _B and R _C are overlapped, the supply region R _A, was determined as the passing band of the filter, the pass band filter Information is output to the filter unit 301.

  Further, the filter information supply unit 312 also supplies the filter information to the MPEG decoder 311, thereby controlling the MPEG decoder 311.

  Next, processing of the reception device 2 in FIG. 199 will be described with reference to the flowchart in FIG.

  The decoding unit 50 is supplied with encoded data. In step S311, the decoding unit 50 inputs the encoded data to the MPEG decoder 311 and proceeds to step S312.

  In step S312, the MPEG decoder 311 extracts (extracts) a motion vector from the encoded data, supplies it to the filter information supply unit 312, and proceeds to step S313.

In step S313, the filter information supply unit 312 uses the motion vector supplied from the MPEG decoder 311 to infer a motion vector of a macroblock that does not include the motion vector in the encoded data, and thereby encodes the encoded data into the MPEG encoded data. Motion vectors are obtained for all the macroblocks of the moving image data having the frame rate 1 / (4t ₀ ).

  That is, when the moving image data is MPEG-encoded, each picture (frame or field) of the moving image data is classified into, for example, one of I (Intra), P (Predictive), and B (Bidirectionally Predictive) pictures. . Since the I picture is encoded without using motion compensation, no motion vector exists in the macroblock of the I picture. Therefore, in step S313, the motion vector of the macroblock having no motion vector is estimated by, for example, interpolation or extrapolation using the motion vector of the macroblock of the preceding and succeeding frames.

Thereafter, the process proceeds from step S313 to S314, and the filter information supply unit 312 is an area of data that can be recognized by a human on the frequency domain and displayed on the display device 3, for example, for a macroblock. The supply area _RA is obtained as a filter pass band, and the pass band is supplied to the filter unit 301 and the MPEG decoder 311 as filter information.

That is, in step S314, as in step S303 in FIG. 198, the filter information supply unit 312 obtains a data area _RC that can be recognized by humans based on the motion vector of the macroblock. Furthermore, filter information supply section 312, based on the frame rate 1 / (d × t ₀₎ as the display capability of the display device 3, obtains the region R _B representing the display capability. The filter information supply section 312, a region overlapping and a region R _B and R _C, determined as the supply region R _A, the supply region R _A, determined as the passband of the filter, represents the passband ( Information) is supplied to the filter unit 301 and the MPEG decoder 311 as filter information.

Thereafter, the process proceeds from step S314 to step S315, and the MPEG decoder 311 is used when obtaining filter information while applying a band limiting filter that passes only data in the passband represented by the filter information from the filter information supply unit 312. The macroblock having the motion vector thus obtained is MPEG-decoded (decompressed), and the resulting frame rate 1 / (4t ₀ ) moving image data is supplied to the buffer unit 51 for storage.

  That is, here, since the encoded data is obtained by MPEG encoding moving image data, the moving image data is converted into DCT (Discrete Cosine Transform) coefficients that are data on the frequency axis. In step S315, the MPEG decoder 311 forcibly sets the DCT coefficient corresponding to the frequency component outside the passband represented by the filter information to 0 and then performs MPEG decoding, so that the passband represented by the filter information is equivalent. The encoded data can be MPEG-decoded while applying a band-limiting filter that passes only the data inside.

  Then, the process proceeds from step S315 to S316, and the filter unit 301 reads the moving picture data after MPEG decoding from the buffer unit 51, and the passband represented by the filter information supplied from the filter generation unit 53 in step S314 for the data. While applying this filter, up-sampling is performed to increase the number of samples in the time direction by 4 / d times.

The filter unit 301 obtains moving image data having a frame rate of 1 / (d × t ₀ ) by performing filtering and upsampling, and proceeds from step S316 to S317.

In step S317, the filter unit 301 outputs the video data of the frame rate 1 / afford (d × t ₀₎ at step S316, the frame rate to 1 / (d × t ₀₎ of the display device 3 (FIG. 24) To do.

As described above, the moving image data is obtained by the filter using the supply region R _{A in} which the region R _C that can be recognized by humans on the frequency domain and the region R _B representing the display capability of the display device 3 overlap each other. Since filtering and further upsampling according to the display capability of the display device 3 are performed, the display device is obtained from the low frame rate moving image data in which the data amount is reduced in consideration of the visual effect of the human eye. Thus, moving image data suitable for the display capability 3 can be obtained and displayed.

Incidentally, when obtaining the feed region R _A, instead of the region R _C, it is possible to adopt an area R _C 'as follows.

Region R _C '= {(X, Y, T) | X and Y satisfy −4π / d ≦ (u ₀ × X + v ₀ × Y) ≦ 4π / d,
Further, −π / r ₀ ≦ X ≦ π / r ₀ and −π / r ₀ ≦ Y ≦ π / r ₀ are satisfied. } ∩R _C

Here, (u ₀ , v ₀ ) is a macroblock motion vector.

According to the definition of the region R _C ′, X and Y on the frequency domain are limited to small values as the motion vector, that is, u ₀ or v ₀ is larger. Therefore, the region R _C ′ expands so that the smaller the motion vector, the higher the spatial direction frequency X, Y, and the larger the motion vector, the lower the spatial direction frequency X, Y. It narrows.

Instead of the region R _C, when 'the supply area in the case of adopting the R _A' region R _C and is represented as the supply region R _A 'is equal to or less than the above region.

That is, for example, when the display device 3 has a frame rate of 1 / (4t ₀ ), the supply region R _A ′ is the following region.

For example, for a portion of a moving image where a stationary subject is projected, the region R _A ′ is the same region R _{20101 as} the region R _{1001 in} FIG. 20, as shown in FIG.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ ) / 2”, the region R _A ′ is as shown in FIG. , of the region R ₁₁₀₁ in FIG. 21, a region R ₂₀₂₀₁ sandwiched by the straight line L ₁₅₈₀₁ and L ₁₅₈₀₂ shown in FIG 160.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ )”, the region R _A ′ is shown in FIG. Of the 22 regions R _1201, the region R ₂₀₃₀₁ is sandwiched between the straight lines L ₁₆₃₀₁ and L ₁₆₃₀₂ shown in FIG. 167.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, the region R _A ′ is shown in FIG. Of the region R _1301, the region R ₂₀₄₀₁ is sandwiched between the straight lines L ₁₇₂₀₁ and L ₁₇₂₀₂ shown in FIG. 176.

Next, for example, when the display device 3 has a frame rate of 1 / (2t ₀ ), the supply region R _A ′ is the following region.

For example, for a portion of a moving image where a stationary subject is projected, the region R _A ′ is the same region R _{20501 as} the region R _{1001 in} FIG. 20, as shown in FIG.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2”, the region R _A ′ is as shown in FIG. , of the region R ₁₁₀₁ in FIG. 21, the region R ₂₀₆₀₁ sandwiched by the straight line L ₁₅₉₀₁ and L ₁₅₉₀₂ shown in FIG 160, i.e., the same region R ₂₀₆₀₁ and the region R _1101.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ )”, the region R _A ′ is shown in FIG. Of the 22 regions R _1201, the region R ₂₀₇₀₁ is sandwiched between the straight lines L ₁₆₄₀₁ and L ₁₆₄₀₂ shown in FIG. 167.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, the region R _A ′ is shown in FIG. Of the region R _1301, the region R ₂₀₈₀₁ is sandwiched between the straight lines L ₁₇₃₀₁ and L ₁₇₃₀₂ shown in FIG. 176.

Then, for example, the display device 3, in the case where a frame rate of 1 / (4t _0/3), the supply region R _A 'is equal to or less than the above region.

For example, for a portion of a moving image where a stationary subject is projected, the region R _A ′ is the same region R _{20901 as} the region R _{1001 in} FIG. 20, as shown in FIG.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ ) / 2”, the region R _A ′ is as shown in FIG. The region R ₂₁₀₀₁ is the same as the region R _{1101 in} FIG.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ )”, the region R _A ′ is shown in FIG. Of the 22 regions R _1201, the region R ₂₁₁₀₁ is sandwiched between the straight lines L ₁₆₅₀₁ and L ₁₆₅₀₂ shown in FIG. 167.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, the region R _A ′ is shown in FIG. Of the region R _1301, the region R ₂₁₂₀₁ is sandwiched between the straight lines L ₁₇₄₀₁ and L ₁₇₄₀₂ shown in FIG.

Next, for example, when the display device 3 has a frame rate of 1 / t ₀ , the supply region R _A ′ is the following region.

For example, in a portion of the moving image where a stationary subject is projected, the region R _A ′ is the same region R _{21301 as} the region R _{1001 in} FIG. 20, as shown in FIG.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of about “(r ₀ / t ₀ ) / 2”, the region R _A ′ is as shown in FIG. The region R ₂₁₄₀₁ is the same as the region R _{1101 in} FIG.

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of “(r ₀ / t ₀ )”, the region R _A ′ is shown in FIG. The region R ₂₁₅₀₁ is the same as the 22 region R ₁₂₀₁ .

Further, for example, in a portion of the moving image where the subject projected on the moving image moves at a speed of about “2r ₀ / t ₀ ”, the region R _A ′ is shown in FIG. The region R ₂₁₆₀₁ is the same as the region R ₁₃₀₁ .

The region R _A is, for example, the region illustrated in FIGS. 181 to 196, and the region R _A ′ is, for example, the region illustrated in FIGS. 201 to 216. Each of the regions R _A shown in FIGS. 181 to 196 and each of the regions R _A ′ shown in FIGS. 201 to 216 are substantially the same regions.

Accordingly, the receiving apparatus 2, instead of filtering the pass band region R _A, it is possible to perform filtering to pass band region R _A '.

Further, the display device 3, if having a frame rate of _{1 / (d × t 0)} , the display device 3, in the time direction, filtering the pass band region R _B is performed (equivalently). Therefore, in the receiving apparatus 2, if the moving image data having the frame rate 1 / (4t ₀ ) from the transmitting apparatus 1 is filtered using the area R _C ′ instead of the area R _A ′, the display apparatus 3, substantially the same display can be performed as when filtering is performed with the region R _A ′ and thus the region R _A as the pass band.

  In this case, the memory capacity can be reduced or the circuit scale can be reduced.

That is, the region R _A, as shown in FIGS. 181 to Fig. 196, in a direction perpendicular to the T direction, since the region with a limited region R _C, the filtering of such regions R _A and passband In addition to the frame of the block of interest (frame of interest), the frame other than the frame of interest is the target. Therefore, in this case, it is necessary to store frames other than the target frame (for example, one or more frames before and after the target block), and a frame memory for that purpose is required. That is, filtering in the time direction is necessary, which means that filtering processing using a frame other than the frame of interest is necessary.

On the other hand, since the region R _C ′ is not limited to a direction orthogonal to the T direction, that is, limited to a direction parallel to the T direction, the filtering is performed only on the target frame. I'll do it. Therefore, in this case, it is not necessary to store frames other than the frame of interest, and a frame memory for that purpose is also unnecessary. As a result, the memory capacity can be reduced or the circuit scale can be reduced. That is, it is only necessary to perform filtering in the spatial direction, that is, filtering processing using only the pixels in the frame of interest, so that processing that spans between frames is unnecessary.

In the above-described case, 1 / (4t ₀ ) of 1 / t 4 obtained by processing the high-frame-rate moving image data of 1 / t ₀ in the transmission device 1 of FIG. The moving image data with a low frame rate is input, and the receiving device 2 converts the moving image data into moving image data with a frame rate corresponding to the frame rate of the display device 3, but the receiving devices in FIG. 197 and FIG. 2 may be originally input video data having a frame rate of 1 / (4t ₀ ). In addition, moving image data having a frame rate other than 1 / (4t ₀ ) may be input to the receiving device 2 in FIGS. 197 and 199.

  Next, the series of processes described above can be performed by dedicated hardware or by software. When a series of processing is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

  Therefore, FIG. 217 shows a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

  A CPU (Central Processing Unit) 401, which is a main controller of the computer, is connected to each unit via a bus 408, and executes the application program under the control of the operating system (OS) to thereby execute the above-described operation. Perform a series of processing. The application program executed by the CPU 401 includes a program for performing the above-described series of processing. For example, the CPU 401 starts from a VTR (Video Tape Recorder) 410, a VTR interface 409, a bus 408, and an external device interface. The moving image data downloaded to the HDD (Hard disk Drive) 414 via 407 is processed.

  The memory 402 is a storage device used for storing program codes executed by the CPU 401 and temporarily storing work data being executed. The memory 402 includes both a nonvolatile memory such as a ROM (Read Only Memory) and a volatile memory such as a DRAM (Dynamic Random Access Memory).

  The display controller 403 is a dedicated controller for actually processing a drawing command issued by the CPU 401. The drawing data processed in the display controller 403 is temporarily written in a frame buffer (not shown), for example, and then output on the screen by the display 411. For example, the moving image reproduced from the HDD 414 is displayed on the screen of the display 411 as described above.

  The input device interface 404 is a device for connecting user input devices such as a keyboard 412 and a mouse 413 to a computer. A user can input a command or the like for playing back a moving image via the keyboard 412 or the mouse 413.

  The network interface 405 can connect the computer to a local network such as a LAN (Local Area Network) and further to a wide area network such as the Internet according to a predetermined communication protocol such as Ethernet (registered trademark). On the network, a plurality of host terminals and servers (not shown) are connected in a transparent state, and a distributed computing environment is constructed. On the network, it is possible to provide a distribution service such as a software program including an application program for executing the above-described series of processing and data / content including encoded data. Further, the computer can receive moving image data and encoded data from a server storing moving images taken by others by the network interface 405 via the network and download them to the HDD 414.

  The external device interface 407 is a device for connecting external devices such as the HDD 414 and the media drive 415 to the computer.

  The HDD 414 is a random-accessible external storage device in which a magnetic disk as a storage medium is fixedly mounted, and is superior to other external storage devices in terms of storage capacity and data transfer speed. The HDD 414 is installed with an application program that executes the above-described series of processing. Here, “install” means placing the software program on the HDD 414 in an executable state. The HDD 414 stores the operating system program code to be executed by the CPU 401, application programs, device drivers, and the like in a nonvolatile manner. Note that the application program can be downloaded from the portable medium 416 or via the network interface 405 and installed in the HDD 414. The application program can be installed in the HDD 414 in advance.

  The media drive 415 is a device for loading portable media 416 such as CD (Compact Disc), MO (Magneto-Optical disc), DVD (Digital Versatile Disc), etc., and accessing the data recording surface.

  The portable media 416 is used mainly for the purpose of backing up software programs, data files, and the like as data in a computer-readable format, and for moving them between systems (that is, including sales, distribution, and distribution). . The application program for executing the above-described series of processing and the encoded data obtained by executing the application program can be physically distributed and distributed among a plurality of devices using the portable medium 416. The encoded data can also be distributed via the network interface 405.

  The VTR interface 409 is a device for taking a moving image reproduced from the VTR 410 into the computer.

  As the computer shown in FIG. 217, a compatible computer or a successor of IBM's personal computer “PC / AT (Personal Computer / Advanced Technology)” can be employed. Of course, you may employ | adopt the computer provided with the other architecture.

In the embodiment described above, the frame rate of 1 / t ₀ is obtained by performing downsampling while performing filtering using the region on the frequency domain that can be recognized by humans as the passband in the transmission device 1. the video data, 1 / has been set as the moving picture data of the frame rate (4t _0), downsampling is not necessarily performed. When downsampling is not performed, the number of frames of video data after filtering is the same as that before filtering, but the frequency component is limited to a region on the frequency domain that can be recognized by humans. That is, since the frequency component outside the region is removed, the data amount is reduced by the amount of the removed frequency component (by applying an encoding that performs conversion on the frequency axis to the moving image data after filtering). It can be performed.

It is a figure which shows the moving image data in the three-dimensional space which added the time direction (t) to the two-dimensional plane (x, y). It is a figure which shows the range on the frequency domain of the moving image which can be recognized by human vision. It is a figure which shows the area | region on the frequency domain where the data of the part of the moving image which are still are distributed. Subject is a diagram showing a region on the frequency domain data is distributed in the moving parts at a rate _{(r 0 / t 0) /} 2 approximately. It is a waveform diagram showing a moving subject at a rate _{(r 0 / t 0) /} 2. It is a diagram showing a moving subject at a rate _{(r 0 / t 0) /} 2. It is a diagram for explaining a portion of a frequency domain data of the moving parts at a rate _{(r 0 / t 0) /} 2 approximately. Subject is a diagram showing a region on the frequency domain data is distributed in the moving parts in the order rate r ₀ / t _0. Subject is a diagram showing a region on the frequency domain data is distributed in the moving parts in the order rate 2r ₀ / t _0. It is a figure explaining the area | region on the frequency domain which a human can recognize about the still part of a moving image. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. It is a figure explaining the area | region on the frequency domain which a human can recognize about the still part of a moving image. It is a diagram for explaining a region of the frequency domain that can recognize human the portion where the subject is moving at a speed _{(r 0 / t 0) /} 2 approximately. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate r ₀ / t _0. Subject is a diagram for explaining a region of the frequency domain that a human can recognize the portion in motion at about the rate 2r ₀ / t _0. It is a block diagram which shows the structural example of one Embodiment of the image processing system to which this invention is applied. 3 is a block diagram illustrating a first configuration example of a transmission apparatus 1. FIG. 4 is a flowchart for explaining processing of the transmission device 1. It is a figure explaining the moving image data which the transmitter 1 outputs. It is a figure explaining the moving image data which the transmitter 1 outputs. It is a figure explaining the moving image data which the transmitter 1 outputs. It is a figure explaining the moving image data which the transmitter 1 outputs. 5 is a flowchart for explaining processing of a filter generation unit 23. It is a flowchart explaining the method of acquiring a principal component direction. 3 is a block diagram illustrating a first configuration example of a receiving device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. 6 is a diagram for explaining moving image data after up-sampling in the receiving device 2; 6 is a diagram for explaining moving image data after up-sampling in the receiving device 2; 6 is a diagram for explaining moving image data after up-sampling in the receiving device 2; 6 is a diagram for explaining moving image data after up-sampling in the receiving device 2; It is a figure explaining the integration by an electronic shutter. It is a figure explaining the pass band of a filter. It is a figure explaining the pass band of a filter. It is a figure explaining the pass band of a filter. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. FIG. 4 is a waveform diagram showing high frame rate moving image data supplied to the transmission apparatus 1. 3 is a block diagram illustrating a configuration example of a principal component direction acquisition unit 31. FIG. It is a figure which shows correlation information. It is a figure which shows the correlation information after scaling. It is a figure which shows synthetic | combination correlation information. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. It is a figure which shows a motion vector. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 5 is a flowchart illustrating processing of a principal component direction acquisition unit 31. 4 is a flowchart for explaining processing of a filter unit 22; It is a figure which shows the to-be-photographed subject and the to-be-moved subject. FIG. 6 is a waveform diagram showing moving image data in which a stationary subject and a moving subject are projected. FIG. 6 is a waveform diagram showing moving image data in which a stationary subject and a moving subject are projected. 10 is a block diagram illustrating another configuration example of the principal component direction acquisition unit 31. FIG. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 5 is a flowchart for explaining processing of a filter generation unit 23. 4 is a flowchart for explaining processing of a filter unit 22; 6 is a block diagram illustrating a second configuration example of the transmission device 1. FIG. FIG. 3 is a diagram showing an area representing display capability of the display device 3. FIG. 3 is a diagram showing an area representing display capability of the display device 3. FIG. 3 is a diagram showing an area representing display capability of the display device 3. FIG. 3 is a diagram showing an area representing display capability of the display device 3. It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "high region component moving image data." It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "high region component moving image data." 4 is a flowchart for explaining processing of the transmission device 1. 4 is a flowchart for explaining processing of the transmission device 1. 4 is a flowchart for explaining processing of the transmission device 1. It is a figure which shows "low-frequency component moving image data". It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "high region component moving image data." It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "high region component moving image data." 6 is a block diagram illustrating a second configuration example of the reception device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. 4 is a diagram showing moving image data displayed on the display device 3. FIG. 4 is a diagram showing moving image data displayed on the display device 3. FIG. 4 is a diagram showing moving image data displayed on the display device 3. FIG. 4 is a diagram showing moving image data displayed on the display device 3. FIG. 11 is a block diagram illustrating a third configuration example of the reception device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low frequency component moving image data" and "medium low frequency component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low frequency component moving image data" and "medium low frequency component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low frequency component moving image data" and "medium low frequency component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. 10 is a block diagram illustrating a fourth configuration example of the reception device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. 5 is a flowchart for explaining processing of the reception device 2. It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low region component moving image data", "middle low region component moving image data", and "middle and high region component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low region component moving image data", "middle low region component moving image data", and "middle and high region component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low region component moving image data", "middle low region component moving image data", and "middle and high region component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. 10 is a block diagram illustrating a fifth configuration example of the receiving device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. 5 is a flowchart for explaining processing of the reception device 2. It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. It is a figure which shows the moving image data obtained by synthesize | combining "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure which shows the moving image data after upsampling. 4 is a diagram showing moving image data displayed on the display device 3. FIG. 11 is a block diagram illustrating a sixth configuration example of the receiving device 2. FIG. It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "high region component moving image data." It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure explaining the division | segmentation method into "low frequency component moving image data", "middle low frequency component moving image data", "middle and high frequency component moving image data", and "high frequency component moving image data". It is a figure which shows "low-frequency component moving image data". It is a figure which shows "medium-low-range component moving image data." It is a figure which shows "mid-high range component moving image data." It is a figure which shows "high region component moving image data." It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. 11 is a block diagram illustrating a seventh configuration example of the receiving device 2. FIG. 5 is a flowchart for explaining processing of the reception device 2. FIG. 20 is a block diagram illustrating an eighth configuration example of the receiving device 2. 5 is a flowchart for explaining processing of the reception device 2. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a figure which shows the moving image data which should be supplied to the display apparatus. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Transmission apparatus, 2 Reception apparatus, 3 Display apparatus, 11 Recording medium, 12 Transmission medium, 21 Buffer part, 22 Filter part, 23 Filter production | generation part, 24 Encoding part, 31 Main component direction acquisition part, 32 Filter information supply part, 50 Decoding section, 51 Buffer section, 52 Filter section, 53 Filter generation section, 61 Principal component direction acquisition section, 62 Filter information supply section, 101 Buffer section, 102 Block extraction section, 103 Correlation calculation section, 104 Scaling composition section, 105 Minimum value detection unit, 124 synthesis unit, 125 minimum value detection unit, 202 filter unit, 211 extraction unit, 221 extraction unit, 222 synthesis unit, 223 filter unit, 231 extraction unit, 232 synthesis unit, 233 filter unit, 241 extraction unit , 242 Generation unit, 251 extraction unit, 252 synthesis unit, 253 filter unit, 301 filter unit, 302 filter information supply unit, 311 MPEG decoder, 312 filter information supply unit, 401 CPU, 402 memory, 403 display controller, 404 input device interface, 405 network interface, 407 external device interface, 409 VTR interface, 410 VTR, 411 display, 412 keyboard, 413 mouse, 414 HDD, 415 media drive, 416 portable media

Claims (30)

In an image processing method for obtaining filter information, which is filter information for filtering video data,
A principal component direction detection step of detecting a principal component direction which is a direction of the first principal component of the moving image data on a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction;
In the frequency domain, a region that extends in the principal component direction and has a specific width corresponding to a visual integration function in the direction of the frequency axis in the time direction is determined as a passband of the filter Steps,
Images processing method and an output step of outputting the passband of the filter, as the filter information. In the principal component direction detection step, a direction orthogonal to the direction of the vector on the frequency domain, which is the same component as the vector representing the motion of the block composed of one or more pixels in the moving image data, is detected as the principal component direction.
The image processing method according to 請 Motomeko 1. The principal component direction detection step includes:
A correlation information calculation step of calculating correlation information indicating a correlation between the block of the target frame of interest in the moving image data and each of a plurality of frames temporally preceding and following the target frame;
A scaling and combining step for obtaining combined correlation information obtained by scaling and combining correlation information between the block and each of the plurality of frames in a spatial direction;
A position detecting step of detecting a maximum correlation position that is a position in a spatial direction where the correlation represented by the combined correlation information is maximum, and obtaining a vector to the maximum correlation position as a motion vector of the block, and
A direction orthogonal to the direction of the vector on the frequency domain of the same component as the motion vector of the block is detected as the principal component direction.
The image processing method according to 請 Motomeko 2. The range of the plurality of frames is a range corresponding to a range of frames used for filtering by the filter.
The image processing method according to 請 Motomeko 3. The principal component direction detection step includes:
A correlation information calculation step of calculating correlation information indicating a correlation between the block of the target frame of interest in the moving image data and each of a plurality of frames temporally preceding and following the target frame;
A scaling combining step of obtaining a plurality of combined correlation information obtained by scaling and combining the correlation information between the block and each of the plurality of frames in a spatial direction, by changing the number of the correlation information to be combined;
Among the plurality of composite correlation information, the maximum composite correlation information which is the maximum correlation information is obtained, and the maximum correlation position which is the position in the spatial direction where the correlation represented by the maximum composite correlation information is maximum is detected. And a position detecting step for obtaining a vector to the maximum correlation position as a motion vector of the block,
A direction orthogonal to the direction of the vector on the frequency domain of the same component as the motion vector of the block is detected as the principal component direction.
The image processing method according to 請 Motomeko 2. In the position detection step, a frame range used for calculating the correlation information synthesized to obtain the maximum synthesized correlation information among the plurality of frames is further obtained as an effective range of the motion vector.
The image processing method according to 請 Motomeko 5. Filtering by the filter is performed using pixels in the frame in the effective range.
The image processing method according to 請 Motomeko 6. In an image processing apparatus that obtains filter information that is filter information for filtering video data,
Principal component direction detection means for detecting a principal component direction which is a direction of the first principal component of the moving image data on a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction;
In the frequency domain, a region extending in the principal component direction and having a specific width corresponding to a visual integration function in the direction of the frequency axis in the time direction is determined as a passband of the filter, images processing device and a region determination unit that outputs a filter information. In a program for causing a computer to execute image processing for obtaining filter information, which is filter information for filtering moving image data,
A principal component direction detection step of detecting a principal component direction which is a direction of the first principal component of the moving image data on a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction;
In the frequency domain, a region that extends in the principal component direction and has a specific width corresponding to a visual integration function in the direction of the frequency axis in the time direction is determined as a passband of the filter Steps,
A program for causing a computer to execute image processing including an output step of outputting the pass band of the filter as the filter information. In a program recording medium in which a program for causing a computer to execute image processing for obtaining filter information that is filter information for filtering moving image data is recorded.
A principal component direction detection step of detecting a principal component direction which is a direction of the first principal component of the moving image data on a frequency domain defined by a frequency axis in a time direction and a frequency axis in a spatial direction;
In the frequency domain, a region that extends in the principal component direction and has a specific width corresponding to a visual integration function in the direction of the frequency axis in the time direction is determined as a passband of the filter Steps,
A program recording medium in which a program for causing a computer to execute image processing including an output step of outputting the pass band of the filter as the filter information is recorded. In the image processing method of filtering moving image data having a first frame rate, down-sampling in the time direction, and outputting moving image data having a second frame rate lower than the first frame rate,
In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, a region extending in the principal component direction that is the direction of the first principal component of the moving image data at the first frame rate, the time By filtering the moving image data of the first frame rate with a region having a specific width corresponding to the visual integration function in the direction of the frequency axis of the direction as a pass band, and down-sampling in the time direction, the first A filter step for obtaining video data having a frame rate of 2;
Images processing method and an output step of outputting the video data of the second frame rate obtained by the filter step. A principal component direction that is a direction of a first principal component of the moving image data at the first frame rate on the frequency domain is detected, and an area extending in the principal component direction in the frequency domain, the time domain an area with a specified width in the direction of the frequency axis direction, further comprising a pass band calculating step of determining as said pass band
The image processing method according to 請 Motomeko 11. In the output step, the moving image data of the second frame rate is output together with the principal component direction.
The image processing method according to 請 Motomeko 11. In the passband calculation step, a direction orthogonal to the direction of the vector on the frequency domain of the same component as the vector representing the motion of the block composed of one or more pixels in the moving image data of the first frame rate is set to the main domain. Detect as component direction
The image processing method according to 請 Motomeko 12. The passband calculation step includes
Correlation information calculation step for calculating correlation information indicating the correlation between the block of the target frame of interest in the moving image data of the first frame rate and each of a plurality of frames temporally preceding and following the target frame. When,
A scaling and combining step for obtaining combined correlation information obtained by scaling and combining correlation information between the block and each of the plurality of frames in a spatial direction;
A position detecting step of detecting a maximum correlation position that is a position in a spatial direction where the correlation represented by the combined correlation information is maximum, and obtaining a vector to the maximum correlation position as a motion vector of the block, and
A direction orthogonal to the direction of the vector on the frequency domain of the same component as the motion vector of the block is detected as the principal component direction.
The image processing method according to 請 Motomeko 14. The range of the plurality of frames is a range corresponding to a range of frames used when filtering moving image data having the first frame rate.
The image processing method according to 請 Motomeko 15. The passband calculation step includes
Correlation information calculation step for calculating correlation information indicating the correlation between the block of the target frame of interest in the moving image data of the first frame rate and each of a plurality of frames temporally preceding and following the target frame. When,
A scaling combining step of obtaining a plurality of combined correlation information obtained by scaling and combining the correlation information between the block and each of the plurality of frames in a spatial direction, by changing the number of the correlation information to be combined;
Among the plurality of composite correlation information, the maximum composite correlation information which is the maximum correlation information is obtained, and the maximum correlation position which is the position in the spatial direction where the correlation represented by the maximum composite correlation information is maximum is detected. And a position detecting step for obtaining a vector to the maximum correlation position as a motion vector of the block,
A direction orthogonal to the direction of the vector on the frequency domain of the same component as the motion vector of the block is detected as the principal component direction.
The image processing method according to 請 Motomeko 14. In the position detection step, a frame range used for calculating the correlation information synthesized to obtain the maximum synthesized correlation information among the plurality of frames is further obtained as an effective range of the motion vector.
The image processing method according to 請 Motomeko 17. The filtering of the moving image data at the first frame rate is performed using pixels of the frames in the effective range.
The image processing method according to 請 Motomeko 18. In the image processing apparatus that filters moving image data having a first frame rate and down-samples the moving image data in a time direction to output moving image data having a second frame rate lower than the first frame rate.
In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, a region extending in the principal component direction that is the direction of the first principal component of the moving image data at the first frame rate, the time By filtering the moving image data of the first frame rate with a region having a specific width corresponding to the visual integration function in the direction of the frequency axis of the direction as a pass band, and down-sampling in the time direction, the first Filter means for obtaining moving image data of frame rate 2
Images processing device. A program for causing a computer to execute image processing for filtering moving image data having a first frame rate and down-sampling the moving image data in a time direction to output moving image data having a second frame rate lower than the first frame rate In
In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, a region extending in the principal component direction that is the direction of the first principal component of the moving image data at the first frame rate, the time By filtering the moving image data of the first frame rate with a region having a specific width corresponding to the visual integration function in the direction of the frequency axis of the direction as a pass band, and down-sampling in the time direction, the first A filter step for obtaining video data having a frame rate of 2;
A program that causes a computer to execute image processing including an output step of outputting moving image data of the second frame rate obtained in the filtering step. A program for causing a computer to execute image processing for filtering moving image data having a first frame rate and down-sampling the moving image data in a time direction to output moving image data having a second frame rate lower than the first frame rate In a program recording medium on which is recorded,
In the frequency domain defined by the frequency axis in the time direction and the frequency axis in the spatial direction, a region extending in the principal component direction that is the direction of the first principal component of the moving image data at the first frame rate, the time By filtering the moving image data of the first frame rate with a region having a specific width corresponding to the visual integration function in the direction of the frequency axis of the direction as a pass band, and down-sampling in the time direction, the first A filter step for obtaining video data having a frame rate of 2;
A program recording medium on which a program for causing a computer to execute image processing including an output step of outputting moving image data of the second frame rate obtained in the filter step is recorded. While moving image data having a second frame rate lower than the first frame rate of moving image data having a first frame rate, up-sampling is performed in the time direction, and moving image data having the first frame rate is output. In the image processing method to
Acquisition of principal component direction for acquiring a principal component direction which is a direction of the first principal component of the moving image data of the second frame rate on a frequency domain defined by a frequency axis in the time direction and a frequency axis in the spatial direction. Steps,
In the frequency domain, a region that extends in the principal component direction and has a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction is determined as a passband for the filter. Steps,
A filter that obtains moving image data of the first frame rate by performing upsampling in the time direction while applying a filter that passes the frequency component of the pass band to the moving image data of the second frame rate. images processing method comprising the steps. In the principal component direction obtaining step, the principal component direction is obtained by detecting the principal component direction from the moving image data of the second frame rate.
The image processing method according to claim 23 . In the principal component direction acquisition step, a direction orthogonal to the direction of the vector on the frequency domain of the same component as the vector representing the motion of the block composed of one or more pixels in the moving image data of the second frame rate is Detect as principal component direction
The image processing method according to claim 24 . In the case where the moving image data of the second frame rate is encoded by an encoding method using at least motion compensation,
In the principal component direction acquisition step, a direction orthogonal to the direction of the vector on the frequency domain of the same component as the motion vector used for the motion compensation is acquired as the principal component direction.
The image processing method according to claim 23 . In the passband determining step, the time direction of the region extending in the principal component direction in the frequency domain and having a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction A region in which the width in the direction of the frequency axis is limited corresponding to the exposure time at the time of capturing the moving image data at the second frame rate is determined as the pass band of the filter.
The image processing method according to claim 23 . While moving image data having a second frame rate lower than the first frame rate of moving image data having a first frame rate, up-sampling is performed in the time direction, and moving image data having the first frame rate is output. In the image processing apparatus to
Acquisition of principal component direction for acquiring a principal component direction which is a direction of the first principal component of the moving image data of the second frame rate on a frequency domain defined by a frequency axis in the time direction and a frequency axis in the spatial direction. Means,
In the frequency domain, a region that extends in the principal component direction and has a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction is determined as a passband for the filter. Means,
A filter that obtains moving image data of the first frame rate by performing upsampling in the time direction while applying a filter that passes the frequency component of the pass band to the moving image data of the second frame rate. images processing and means. While moving image data having a second frame rate lower than the first frame rate of moving image data having a first frame rate, up-sampling is performed in the time direction, and moving image data having the first frame rate is output. In a program for causing a computer to execute image processing to be performed,
Acquisition of principal component direction for acquiring a principal component direction which is a direction of the first principal component of the moving image data of the second frame rate on a frequency domain defined by a frequency axis in the time direction and a frequency axis in the spatial direction. Steps,
In the frequency domain, a region that extends in the principal component direction and has a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction is determined as a passband for the filter. Steps,
A filter that obtains moving image data of the first frame rate by performing upsampling in the time direction while applying a filter that passes the frequency component of the pass band to the moving image data of the second frame rate. A program for causing a computer to execute image processing including steps. While moving image data having a second frame rate lower than the first frame rate of moving image data having a first frame rate, up-sampling is performed in the time direction, and moving image data having the first frame rate is output. In a program recording medium on which a program for causing a computer to execute image processing is recorded,
Acquisition of principal component direction for acquiring a principal component direction which is a direction of the first principal component of the moving image data of the second frame rate on a frequency domain defined by a frequency axis in the time direction and a frequency axis in the spatial direction. Steps,
In the frequency domain, a region that extends in the principal component direction and has a specific width corresponding to the visual integration function in the direction of the frequency axis in the time direction is determined as a passband for the filter. Steps,
A filter that obtains moving image data of the first frame rate by performing upsampling in the time direction while applying a filter that passes the frequency component of the pass band to the moving image data of the second frame rate. A program recording medium on which a program for causing a computer to execute image processing including steps is recorded.

Images