JP2003331292A - Image processor and method for processing image, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably display by compensating a camera shake. <P>SOLUTION: A block extracting part 12 extracts two portions of a still object as reference blocks from a frame image input at first, to be stored in a reference block storage part 13, and the center position in either of the reference blocks is output to a locus storage part 18 to be stored. A vector extracting part 14 extracts a reference vector from an inter-center position of the two reference blocks, a comparing part 15 makes the reference vector stored in a reference vector storage part 16. A target vector is extracted from a frame image thereafter by the same manner, the reference vector is compared with the target vector, and the comparing part 15 issues a command to the block extracting part 12 to substitute the block extracting part 12 reference block with the target block at that time, when a difference in the comparison hereinbefore comes to a prescribed threshold value or more. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to an image processing apparatus and method for more stably realizing correction of camera shake that occurs during imaging. The present invention relates to a recording medium and a program.

[0002]

2. Description of the Related Art It is known that so-called camera shake occurs when capturing an image with a video camera or the like, and a technique for correcting this camera shake is becoming popular.

This is to correct an image at each time point with respect to a picked-up moving image according to the degree of "camera shake" so as to correct the moving image without "camera shake". This "camera shake" correction generally involves detecting the degree of "camera shake" by determining the change in the projected image of an object (especially a stationary object) in the moving image, and shifting the captured image according to that degree. Target. That is, a part of the projected image of the predetermined object in the frame image captured in the past is used as a reference block, and a part of the projected image of the predetermined object in the frame image captured now is used as a target block, and this reference block is used. For example, if there is a change of 10 pixels to the right as a whole (if it is moving), it is understood that the image pickup position of the video camera has moved to the right by 10 pixels due to camera shake. By shifting the frame to the left by 10 pixels, "camera shake" can be corrected.

[0004]

By the way, an image (reference block) of a predetermined object serving as a reference for detecting a change in camera shake needs to be updated with the progress of imaging.
As the method of updating the reference block, a method of extracting the captured frame images for each predetermined number of frame images, extracting a part of the predetermined object as a reference block and updating it, and a method of always updating the frame immediately preceding in time. There is a method of extracting a part of a predetermined object included in an image as a reference block and updating it.

However, the reference block is extracted from the frame image extracted for each predetermined frame image,
When detecting changes due to camera shake by comparison with target blocks extracted from subsequent frame images,
If there is a large change in the frame images during the number of frame images shorter than the predetermined number of frame images (when there is a large movement in the predetermined object itself from which the reference block is extracted),
The change between the reference block and the currently captured target block may be too large, and in this case, the change due to camera shake cannot be obtained, and as a result, camera shake may not be corrected.

Further, when the reference block is always extracted from the frame image immediately preceding in time to detect a change due to camera shake, compared with the case where the reference block is updated from the frame images picked up every predetermined number of frame images. , The reference block is always extracted from the previous frame image.
It will be easier to follow changes in the frame image,
There is a problem that the processing amount becomes enormous.

Further, it is known that when the reference block is updated, a slight fluctuation occurs in the image. This is corrected when the reference block is updated at a predetermined number of frame images. The effect on the image is negligible. However, when the reference block is updated for each frame image, fluctuations that occur in one update are accumulated along with the change of the reference block, and as a result, the effect of the image stabilization is visually reduced. There was a problem.

The present invention has been made in view of such circumstances, and makes it possible to optimally process an input frame image by changing the reference block according to the movement of the image.

[0009]

A first image processing apparatus of the present invention comprises a reference block extracting means for extracting a plurality of reference blocks from a frame at a predetermined time, a storage means for storing the plurality of reference blocks, A reference positional relationship information generating unit that generates reference positional relationship information representing the positional relationship between the plurality of reference blocks stored by the storage unit, and a plurality of reference blocks stored in the storage unit from a frame after a predetermined time. Similar block extraction means for extracting a plurality of similar blocks, similar positional relationship information generation means for generating similar positional relationship information indicating the positional relationship between the plurality of similar blocks, similar positional relationship information and reference positional relationship information Difference generating means for generating a difference between the plurality of groups, a comparing means for comparing the difference with a predetermined threshold, and a plurality of groups stored in the storing means corresponding to the comparison result. Based on the replacement means for replacing the block with a plurality of similar blocks, the position information of a predetermined reference block of the plurality of reference blocks, and the position information of a predetermined similar block of the plurality of similar blocks, a predetermined time It is characterized by comprising processing means for performing a predetermined processing on the subsequent frames.

The reference positional relationship information may be a reference vector as a vector between the centers of two reference blocks, and the similar positional relationship information may be similar as a vector between the centers of two similar blocks. It can be a vector.

The difference may be a difference between the reference vector and the similar vector.

The processing means determines a frame after a predetermined time based on position information of a predetermined reference block of the plurality of reference blocks and position information of a predetermined similar block of the plurality of similar blocks. As the predetermined processing, it is possible to perform camera shake correction processing or image combination processing.

According to the first image processing method of the present invention, a reference block extraction step of extracting a plurality of reference blocks from a frame at a predetermined time, a storage step of storing a plurality of reference blocks, and a storage step are stored. The reference positional relationship information generating step of generating reference positional relationship information representing the positional relationship between the plurality of reference blocks described above, and the frame after the predetermined time are similar to the plurality of reference blocks stored in the processing of the storing step. A similar block extracting step of extracting a plurality of similar blocks, a similar positional relationship information generating step of generating similar positional relationship information indicating a positional relationship between a plurality of similar blocks, and a difference between the similar positional relationship information and the reference positional relationship information. A difference generation step for generating
A comparison step of comparing the difference with a predetermined threshold value, a replacement step of replacing the plurality of reference blocks stored in the processing of the storage step with a plurality of similar blocks corresponding to the comparison result, and a plurality of reference blocks among the plurality of reference blocks. Of the predetermined reference block and position information of a predetermined similar block of the plurality of similar blocks, the processing step of performing a predetermined process on a frame after a predetermined time. .

The program of the first recording medium of the present invention is
A reference block extraction control step that controls extraction of a plurality of reference blocks from a frame at a predetermined time, a storage control step that controls storage of the plurality of reference blocks, and a plurality of references whose storage is controlled by the processing of the storage control step. A reference positional relationship information generation control step that controls the generation of reference positional relationship information that represents the positional relationship between blocks, and a frame after a predetermined time is similar to a plurality of reference blocks whose storage is controlled by the processing of the storage control step. A similar block extraction control step for controlling the extraction of the plurality of similar blocks, a similar positional relationship information generation control step for controlling the generation of similar positional relationship information representing the positional relationship between the plurality of similar blocks, and a similar positional relationship information, The difference generation control step for controlling the generation of the difference from the reference positional relationship information and the comparison of the difference with a predetermined threshold are controlled. And a replacement control step for controlling replacement of a plurality of reference blocks whose storage is controlled by the processing of the storage control step with a plurality of similar blocks corresponding to the comparison result, and a plurality of reference blocks. Of the plurality of similar blocks, and a processing control step of controlling a predetermined processing performed on a frame after a predetermined time based on the position information of the predetermined similar block of the plurality of similar blocks. It is characterized by

A first program of the present invention is a reference block extraction control step for controlling extraction of a plurality of reference blocks from a frame at a predetermined time, a storage control step for controlling storage of a plurality of reference blocks, and a storage control. The reference position relation information generation control step for controlling the generation of the reference position relation information indicating the position relation between the plurality of reference blocks whose storage is controlled by the step process, and the process of the storage control step from the frame after the predetermined time. Similar block extraction control step that controls the extraction of a plurality of similar blocks similar to the plurality of reference blocks whose storage is controlled by the similar block, and the similar position that controls the generation of similar positional relationship information indicating the positional relationship between the similar blocks. Relation information generation control step and difference generation control for controlling generation of difference between similar positional relation information and reference positional relation information Step and a comparison control step for controlling the comparison of the difference with a predetermined threshold value, and a plurality of reference blocks whose storage is controlled by the processing of the storage control step corresponding to the comparison result to a plurality of similar blocks. Based on the replacement control step of controlling the replacement, the position information of a predetermined reference block of the plurality of reference blocks, and the position information of a predetermined similar block of the plurality of similar blocks, the frame after the predetermined time And a process control step of controlling a predetermined process to be performed.

The second image processing apparatus of the present invention comprises a reference block extracting means for extracting a reference block from a frame at a predetermined time, a storage means for storing the reference block, and a reference block from a frame after the predetermined time. Similar block extracting means for extracting similar similar blocks, difference generating means for generating a difference between the reference block and similar block, comparing means for comparing the difference with a predetermined threshold value, and storage means corresponding to the comparison result. A replacement unit that replaces the reference block stored in the same block with a similar block, position information of the reference block, and processing unit that performs a predetermined process on a frame after a predetermined time based on the position information of the similar block. Is characterized by.

The difference may be based on a conversion parameter when affine-transforming the reference block and the similar block.

The conversion parameter for the affine transformation may include an enlargement rate or a reduction rate and a rotation angle when the reference block is affine transformed into the similar block.

The processing means, on the basis of the position information of the reference block and the position information of the similar block, a camera shake correction process as a predetermined process for a frame after a predetermined time.
Alternatively, it is possible to perform an image combining process.

According to the second image processing method of the present invention, a reference block extraction step of extracting a reference block from a frame at a predetermined time, a storage step of storing the reference block, and a reference block from the frame after the predetermined time are made. A similar block extraction step of extracting similar similar blocks, a difference generation step of generating a difference between the reference block and the similar block, a comparison step of comparing the difference with a predetermined threshold value, and a storage step corresponding to the comparison result. The substituting step of replacing the reference block stored by the process of with the similar block, the position information of the reference block, and the processing step of performing a predetermined process on a frame after a predetermined time based on the position information of the similar block. It is characterized by including.

The program of the second recording medium of the present invention is
A reference block extraction control step that controls the extraction of the reference block from the frame at a predetermined time, a storage control step that controls the storage of the reference block, and a extraction of a similar block similar to the reference block from the frame after the predetermined time A similar block extraction control step, a difference generation control step for controlling the generation of the difference between the reference block and the similar block, a comparison control step for controlling the comparison of the difference with a predetermined threshold, and a memory corresponding to the comparison result. The replacement control step for controlling the replacement of the reference block whose storage is controlled by the step processing with the similar block, the position information of the reference block, and the position information of the similar block are applied to the frames after a predetermined time. A process control step of controlling a predetermined process.

A second program of the present invention is a reference block extraction control step for controlling extraction of a reference block from a frame at a predetermined time, a storage control step for controlling storage of the reference block, and a frame after the predetermined time. A similar block extraction control step for controlling extraction of a similar block more similar to the reference block, a difference generation control step for controlling generation of a difference between the reference block and the similar block, and a comparison of a difference with a predetermined threshold value. A comparison control step, a replacement control step that controls replacement of a reference block whose storage is controlled by the processing of the storage step with a similar block corresponding to the comparison result, position information of the reference block, and position information of the similar block The process control step for controlling the predetermined process performed on the frame after the predetermined time based on Characterized in that to execute the door to the computer.

A third image processing apparatus of the present invention comprises a reference block extracting means for extracting a reference block from a frame at a predetermined time, a storage means for storing the reference block, and a reference block from a frame after the predetermined time. Similar block extracting means for extracting similar similar blocks, difference generating means for generating a conversion parameter for affine-transforming a reference block into similar blocks as a difference between the reference block and the similar block, and affine the reference block by the difference. A conversion unit for converting, a comparison unit for comparing the difference with a predetermined threshold value, and a replacement for replacing the reference block stored in the storage unit with the reference block affine-transformed by the conversion unit and storing the reference block according to the comparison result. Based on the means, the position information of the reference block, and the position information of the similar block, Wherein the in subsequent frames and processing means for performing a predetermined processing.

The conversion parameter for the affine conversion as the difference may include the enlargement ratio or reduction ratio of the reference block with respect to the similar block, and the rotation angle.

The processing means, on the basis of the position information of the reference block and the position information of the similar block, a camera shake correction process as a predetermined process for a frame after a predetermined time.
Alternatively, it is possible to perform an image combining process.

According to a third image processing method of the present invention, a reference block extracting step of extracting a reference block from a frame at a predetermined time, a storing step of storing the reference block, and a reference block from a frame after the predetermined time are made. A similar block extraction step of extracting similar similar blocks, a difference generation step of generating a conversion parameter when affine-transforming a reference block into a similar block as a difference between the reference block and the similar block, and an affine reference block based on the difference. A conversion step of converting, a comparison step of comparing the difference with a predetermined threshold value, and the reference block stored by the processing of the storage step is replaced with the reference block affine-transformed by the processing of the conversion step according to the comparison result. Replacement step to be stored as If, based on the position information of the similar block, characterized in that it comprises a processing step of performing predetermined processing at a predetermined time after the frame.

The program of the third recording medium of the present invention is
A reference block extraction control step that controls the extraction of the reference block from the frame at a predetermined time, a storage control step that controls the storage of the reference block, and a extraction of a similar block similar to the reference block from the frame after the predetermined time A similar block extraction control step, a difference generation control step for controlling generation of a conversion parameter when the reference block is affine transformed into a similar block as a difference between the reference block and the similar block, and an affine transformation of the reference block by the difference. Of the reference block whose storage is controlled by the processing of the storage control step in accordance with the comparison result, and the comparison control step of controlling the comparison with the predetermined threshold value of the difference. Control the replacement and storage with the reference block affine transformed by A replacement control step, the positional information of the reference block, based on the position information of the similar block, characterized in that it comprises a processing control step of controlling the predetermined processing to a predetermined subsequent time frame.

A third program of the present invention is a reference block extraction control step for controlling extraction of a reference block from a frame at a predetermined time, a storage control step for controlling storage of the reference block, and a frame after a predetermined time. A similar block extraction control step for controlling the extraction of a similar block more similar to the reference block, and a difference for controlling the generation of the conversion parameter when the reference block is affine-transformed into the similar block as the difference between the reference block and the similar block. The generation control step, the conversion control step for controlling the affine transformation of the reference block by the difference, the comparison control step for controlling the comparison of the difference with a predetermined threshold, and the storage control step according to the comparison result It is affine transformed in the transformation step of the controlled reference block. A replacement control step of controlling replacement and storage with the reference block, position information of the reference block, and a processing control step of controlling a predetermined process for a frame after a predetermined time based on the position information of the similar block. It is characterized by causing a computer to execute.

A first image processing apparatus and method of the present invention,
In addition, in the program, a plurality of reference blocks are extracted from the frame at a predetermined time, the plurality of reference blocks are stored, reference positional relationship information indicating the positional relationship between the stored plurality of reference blocks is generated, and the predetermined positional relationship is generated. A plurality of similar blocks similar to the plurality of stored reference blocks are extracted from the frame after the time, and similar positional relationship information indicating the positional relationship between the plurality of similar blocks is generated. Similar positional relationship information and reference positional relationship A difference from the information is generated, the difference is compared with a predetermined threshold value, and the plurality of reference blocks stored are replaced with a plurality of similar blocks according to the comparison result, and a predetermined one of the plurality of reference blocks is replaced. Based on the position information of the reference block and the position information of a predetermined similar block among the plurality of similar blocks, Predetermined processing is performed on.

A second image processing apparatus and method of the present invention,
In the second program, the reference block is extracted from the frame at the predetermined time, the reference block is stored, the similar block similar to the reference block is extracted from the frame after the predetermined time, and the reference block and the similar block are extracted. Difference is generated, the difference is compared with a predetermined threshold value, the stored reference block is replaced with a similar block according to the comparison result, and the reference block position information and the similar block position information are used. , A predetermined process is performed on frames after a predetermined time.

A third image processing apparatus and method of the present invention,
In the third program, the reference block is extracted from the frame at the predetermined time, the reference block is stored, the similar block similar to the reference block is extracted from the frame after the predetermined time, and the reference block is changed to the similar block. A conversion parameter for affine transformation is generated as a difference between a reference block and a similar block, the reference block is affine-transformed by the difference, the difference is compared with a predetermined threshold value, and the reference value stored in accordance with the comparison result. The block is replaced with the affine-transformed reference block, and a predetermined process is performed on a frame after a predetermined time based on the position information of the reference block and the position information of the similar block.

[0032]

1 is a block diagram showing the configuration of an embodiment of an image processing apparatus according to the present invention.

The image input unit 11 is, for example, an image storage device that stores images, and supplies the images stored in advance to the block extraction unit 12 and the image memory 17. Of course, an image captured by an imaging device such as a CCD (Charge Coupled Device) and supplied may be directly supplied to the block extraction unit 12 and the image memory 17.

When no reference block is stored in the reference block storage unit 13, the block extraction unit 12 selects two AC portions (for example, by an optical system (not shown) at the time of image pickup in the first input frame image. In the state where the focus position exists, a clear portion such as a static object captured by the image input unit 11 with little blurring) is extracted as a reference block and stored in the reference block storage unit 13. For example, when a frame image as shown in FIG. 2 is input at the same time as the start of image capturing, the door knob 32 and the window frame 33 of the door 30 among them are input.
When the corner portion of the block is the AC portion, the block extraction unit 12
Is a block corresponding to the door knob 32 in the figure as a reference block 31'-1, and the vicinity of the corner of the window frame 33 is extracted as a reference block 31'-2 and stored in the reference block storage unit 13. Reference block 31'-1,
The coordinates of the respective central positions of 31'-2 are output to the vector extraction unit 14, and the reference blocks 31'-1, 3 '
The coordinates of the central position of 1'-2 are output to the locus storage unit 18 and stored therein.

In the following description, the reference block 3
When it is not necessary to distinguish 1'-1 and 31'-2, they are simply referred to as reference blocks 31 '. The same applies to other cases.

Further, when the reference block is stored in the reference block storage unit 13 (when the reference block is stored once), the block extraction unit 12 stores the sequentially input frame images in the reference block storage unit 13. A block similar to the stored reference block is searched by pattern matching (a process of moving the reference block in the horizontal direction and the vertical direction on the frame image to search for a similar portion), and this is extracted as a target block, Vector extraction unit 1 calculates the coordinates of the center position of the target block.
4 to the reference blocks 31'-1, 31'-
The coordinates of the center position of 2 are output to the locus storage unit 18 and stored therein.

That is, for example, as shown in FIG. 2, when the reference blocks 31'-1, 31'-2 are stored in the reference block storage unit 13, for example, a frame image as shown in FIG. When input, the block extraction unit 12
Is the door knob 32 stored in the reference block storage unit 13.
By performing pattern matching with the reference block 31′-1 in the vicinity, the similar target block 41
Search for -1. As a result, the target block 41
As -1, as shown in FIG. 3, the blocks around the door knob 42 are searched.

Similarly, the block extraction unit 12 performs pattern matching with the reference blocks 31'-2 around the corners of the window frame 33 stored in the reference block storage unit 13 to obtain similar target blocks 41- Search for 2. As a result, as the target block 41-2, as shown in FIG. 3, the blocks around the window frame 43 are searched.

Then, the block extracting unit 12 outputs the center coordinates of the extracted target blocks 41-1 and 41-2 to the vector extracting unit 14.

Further, the block extraction unit 12 temporarily stores the extracted target block until the processing of the next frame starts, receives the notification of the comparison result from the comparison unit 15, and extracts at that timing ( The target block of the frame image (stored at that time) is overwritten on the data in the reference block storage unit 13 to update the reference block.

The vector extraction unit 14 is a block extraction unit 1.
Vectors are respectively obtained from the coordinates of the central portions of the two reference blocks or the coordinates of the central portions of the two target blocks, which are input from 2 and are output to the comparison unit 15. Therefore, in the case of FIG. 2, for example, the reference block 31 '
-1, 31'-2 as the starting point of the central part of each, and
The vector that is the end point will be obtained. Further, in the case of FIG. 3, for example, the target blocks 41-1 and 41
A vector with the center point of each −2 as the start point and the end point is obtained.

The comparison unit 15 stores the first vector input from the vector extraction unit 14 in the reference vector storage unit 16 as the reference vector, and the vector input thereafter is used as the target vector, and the difference vector from the reference vector. Is calculated and the absolute value of the difference vector is compared with a predetermined threshold value. More specifically, the comparison unit 15
Calculates the difference vector between the reference vector and the target vector, divides the absolute value of the difference vector by the absolute value of the reference vector, and compares it with the threshold value. Then, the comparison unit 15 replaces the reference vector stored in the reference vector storage unit 16 with the input target vector and overwrites and stores the reference vector when the comparison result is larger than the threshold value. 12 is instructed to replace the reference block with the target block extracted from the frame at that time and store it in the reference block storage unit 13.

The image memory 17 sequentially stores the images supplied from the image input unit 11, and outputs them to the camera shake correction unit 19 when the reading is completed. The locus storage unit 18 is
The central coordinate positions of the reference block and the target block supplied from the block extraction unit 12 are stored, and the target object of the block composed of the reference block and the target block (for example, a door knob and a window in a frame image described later) The locus of movement of the (corner of the frame) is stored and output as locus information to the camera shake correction unit 19 at the timing when the supply of the image from the image input unit 11 is completed.

The camera shake correction unit 19 corrects the camera shake by shifting the image supplied from the image memory 17 according to the trajectory information from the trajectory storage unit 18, and thus the input frame image, and the image output unit 20. Output to. The image output unit 20 outputs the image in which the camera shake is corrected by the camera shake correction unit 19 to a storage device or a display device (not shown).

Next, referring to the flow chart of FIG.
The image stabilization processing by the image processing apparatus in FIG. 1 when the frame images from the first frame to the sixth frame are input will be described. In this example, the processing from the first frame to the sixth frame will be described.
Actually, the processing is performed in frames of hundreds or thousands, and here, for convenience of description, the processing up to the sixth frame will be described.

In step S1, the block extraction unit 1
Reference numeral 2 is a first frame input from the image input unit 11 as shown in FIG. 2, that is, an alternating current portion (a portion that becomes a focused still object in the frame image) of the first frame as a reference block. It is extracted and stored in the reference block storage unit 13. That is, if the first frame image is input at the same time as the start of image capturing, and if the corners of the door knob 32 and the window frame 33 of the door 30 are AC images, the block extraction unit 12 operates as described above. , A reference block 31'-1 and a window frame 33 corresponding to the door knob 32 in the figure
Extract the vicinity of the corner of the block as the reference block 31'-2,
It is stored in the reference block storage unit 13, and the vector extraction unit 1
Output to 4.

In step S2, the image memory 17
Stores the frame image input from the image input unit 11, that is, the image of the first frame in this case.

In step S3, the block extraction unit 1
2 is the reference blocks 31′-1, 31′-2 stored in the reference block storage unit 13 from the input frame image.
To extract the target block. In the case of FIG.
The block extraction unit 12 extracts the target blocks 31-1 and 31-2 by pattern matching and outputs the coordinate position of the center thereof to the vector extraction unit 14. In the first process, since the reference blocks 31′-1, 31′-2 and the target blocks 31-1, 31-2 are the same object, the coordinates of the center of the reference blocks 31′-1, 31′-2 are It will be output to the vector extraction unit 14.

In step S4, the block extraction unit 1
2 outputs the center coordinate position of the target blocks 31-1 and 31-2 to the locus storage unit 18 as locus information and stores the locus information.

In step S5, the vector extraction unit 1
4 extracts a target vector from the coordinates of the center of the target blocks 31-1, 31-2 = reference blocks 31′-1, 31′-2 and outputs it to the comparison unit 15. In this case, the vector extraction unit 14 outputs the target vector 34 shown in FIG. 2 to the comparison unit 15.

In step S6, the comparison section 15 determines whether or not the reference vector is stored in the reference vector storage section 16. In this case, since it is the first frame, the reference vector is not stored, so it is determined that the reference vector is not stored in the reference vector storage unit 16, and the process proceeds to step S7.

In step S7, the comparison section 15 commands the block extraction section 12 to store the target block of the current frame as a reference block (commands to update the reference block). However, in this case, the reference block 31 of the first frame has already been used.
Since -1, 31-2 are stored in the reference block storage unit 13, the reference block is not substantially updated.

In step S8, the comparison section 15 replaces the target vector 34 as the reference vector 34 'with the reference vector storage section 16 and stores it. However,
In this case, in the first process, since nothing is stored in the reference vector storage unit 16, the target vector 34 is newly stored as the reference vector 34 '.

In step S9, it is determined whether or not the next frame image exists, and in this case, the second frame shown in FIG. 3 is input.
Return to 2.

In step S2, the image memory 17
Stores the image input from the image input unit 11, that is, the image of the second frame in this case.

In step S3, the block extraction unit 1
2 uses the two reference blocks stored in the reference block storage unit 13 from the input frame image to extract two target blocks by pattern matching, and the coordinates of the central portion are extracted by the vector extraction unit 14. Output. That is, the door 30 extracted from the first frame
Reference block 31'-1 corresponding to the door knob 32 of
Since the reference block 31′-2 corresponding to the corner portion of the window frame 33 is stored in the reference block storage unit 13, in the second frame, the reference blocks 31′-1, 31 ′.
-2 is used to perform pattern matching, and a similar portion is extracted as a target block. Second of FIG.
In the case of a frame, the target block 41-1 near the doorknob 42 of the door 40 is similar to the reference block 31'-1 near the doorknob 32 (FIG. 2), and the target block 41-1 near the corner of the window frame 43. The block 41-2 is a reference block 31'-2 near the corner of the window frame 33 (FIG. 2),
Are similar, the target blocks 41-1, 41-
2 is extracted, and the coordinates of the center are output to the vector extraction unit 14.

In step S4, the block extraction unit 1
2 outputs the center coordinate position of the target blocks 41-1 and 41-2 to the locus storage unit 18 as locus information to be stored therein.

In step S5, the vector extraction unit 1
4 is the target block 41-, as shown in FIG.
The target vector 44 is obtained from the coordinates of the center of 1, 41-2, and is output to the comparison unit 15.

In step S6, the comparison section 15 determines whether or not the reference vector is stored in the reference vector storage section 16. In this case, the reference vector 34 'is stored. Go to step S10.

In step S10, the comparison section 15
Difference between the reference vector and the target vector dn (n is
(Indicates the frame number currently input).
More specifically, the comparison unit 15 obtains a difference vector Vdn between the reference vector and the target vector (n indicates a frame number currently input), and further, an absolute value of the difference vector Vdn is an absolute value of the reference vector. The difference dn is obtained by dividing by the value. That is, the second shown in FIG.
In the case of a frame, as shown in FIG. 5, the comparison unit 15 obtains the difference vector Vd2 between the reference vector 34 ′ and the target vector 44, and the absolute value S2 ′ of the difference vector Vd2.
Is divided by the absolute value S1 of the reference vector 34 'to obtain the difference d.
2 (= S2 ′ / S1). The reference vector 34 'is displayed in black in FIG. Also,
Here, the difference vector of the nth frame is Vdn,
The absolute value of the frame difference vector Vdn is represented by Sn ′, and the absolute value of the reference vector 34 ′ is represented by S1, respectively, and the same applies below.

In step S11, the comparison section 15
The difference d2 (n = 2) is compared with the threshold D to determine whether the difference d2> threshold D. In the following description, for convenience of description, when the threshold D is set to 1/2 (the difference d2 is 1/2, the difference vector Vdn is 1/2 of the absolute value S1 of the reference vector). About)
Proceed with the story, but it's not limited to this.

In step S11, as shown in FIG. 5, since the difference d2> threshold value D is not satisfied, the process proceeds to step S9, and the next frame is input. Therefore, the process is step S2. Return to.

Here, in step S2, the image memory 17 stores the image of the third frame input from the image input unit 11 as shown in FIG.

In step S3, the block extraction unit 1
2 performs pattern matching using the reference blocks 31′-1 and 31′-2 stored in the reference block storage unit 13, and targets near the door knob 52 of the door 50 similar to the reference block 31′-1. Block 51-1
Then, the target block 51-2 near the corner of the window frame 53 similar to the reference block 31′-2 is extracted, and the coordinates of the central portion thereof are output to the vector extraction unit 14.

In step S4, the block extraction unit 1
2 outputs the center coordinate position of the target blocks 51-1 and 51-2 to the locus storage unit 18 as locus information to be stored therein.

In step S5, the vector extraction unit 1
4 is the input target blocks 51-1, 51-
As shown in FIG. 6, the target vector 54 is obtained from the coordinates of the center of 2 and is output to the comparison unit 15.

In step S6, since the reference vector 34 'is already stored in the reference vector storage unit 16, it is determined that the reference vector is stored. The process proceeds to step S10 and the comparison unit 15 As shown in FIG. 7, the difference vector Vd3 is calculated by the same method as the method shown in FIG. 5 to obtain the difference d3. That is,
As shown in FIG. 7, the comparison unit 15 obtains the difference vector Vd3 between the target vector 54 and the reference vector 34 ′, and further divides the absolute value S3 ′ of the difference vector Vd3 by the absolute value S1 of the reference vector 34 ′, Difference d3 (= S
3 '/ S1) is calculated.

In step S11, the comparison section 17
The difference d3 and the threshold D are compared. In this case, the difference d3> threshold value D is not satisfied as shown in FIG. 7, so the process proceeds to step S9, and the fourth frame is input, so the process returns to step S2.

Here, in step S2, the image memory 17 stores the image of the fourth frame as shown in FIG. 8 input from the image input unit 11.

In step S3, when the fourth frame is input as shown in FIG. 8, the block extraction unit 12
Using the reference blocks 31′-1, 31′-2 stored in the reference block storage unit 13, as described above,
Pattern matching is performed, and the target block 61-1 near the door knob 62 of the door 60 similar to the reference block 31′-1 and the target block 61-2 near the corner of the window frame 63 similar to the reference block 31′-2. Is extracted and the coordinates of the central portion thereof are output to the vector extraction unit 14.

In step S4, the block extraction unit 1
2 outputs the center coordinate position of the target blocks 61-1 and 61-2 to the locus storage unit 18 as locus information for storage.

In step S5, the vector extraction unit 1
4 is the input target blocks 61-1 and 61-
As shown in FIG. 8, the target vector 64 is obtained from the coordinates of the center of 2 and is output to the comparison unit 15.

In step S6, since the reference vector 34 'is already stored in the reference vector storage unit 16, it is determined that the reference vector is stored. As shown, the difference vector Vd4 is calculated to obtain the difference d4.
That is, as shown in FIG. 9, the comparison unit 15 determines the difference vector V between the target vector 64 and the reference vector 34 ′.
d4 is obtained, and the absolute value S of the difference vector Vd4 is calculated.
4'is divided by the absolute value S1 of the reference vector 34 'to obtain the difference d4 (= S4' / S1).

In step S11, the comparison section 15
It is determined whether or not the difference d4> threshold value D. In the case of the fourth frame, as shown in FIG. 9, the difference vector Vd4 is
Since the difference d4> threshold value D is larger than 1/2 of the absolute value S1 of the reference vector 34 ′, the process proceeds to step S7.

In step S7, the comparison section 15 commands the block extraction section 12 to replace the reference block 31 'with the current target block 61. At this time, the block extracting unit 12 determines that the reference block 3
1'-1, 31'-2 are replaced with target blocks 61-1, 61-2 obtained from the current frame 4 and stored in the reference block storage unit 13 as reference blocks 61'-1, 61'-2. Let

In step S8, the comparing section 15 compares the target vector 64 currently obtained with the reference vector 6
4 ′ is stored in the reference vector storage unit 16 (updated from the reference vector 34 ′ stored until then),
The process proceeds to step S9.

That is, since the difference d4 exceeds the threshold value at the time when the fourth frame is input, the pattern matching using the current reference block is sufficient for the images of the future (that is, the fifth frame and thereafter). It is determined that the search is impossible, and the newly obtained target block is stored as the reference block stored until then. As a result, in the reference block storage unit 13, the reference blocks 31′-1, 31′-2 correspond to the door knob 62, and the reference block 61′-1 (= target block 61-1).
And a reference block 61′-2 corresponding to the corner of the window frame 63
(= Target block 61-2) is updated. At the same time, the reference vector storage unit 16 updates the reference vector 34 'to the reference vector 64'.

At step S9, since the subsequent fifth frame is input, the processing returns to step S2.

Here, in step S2, the image memory 17 stores the image of the fifth frame as shown in FIG.

In step S3, when the fifth frame image as shown in FIG. 10 is input, the block extraction unit 1
2 performs pattern matching using the reference blocks 61′-1 and 61′-2 stored in the reference block storage unit 13 and extracts similar portions as target blocks. In the case of the fifth frame in FIG. 10, the target block 71-1 near the door knob 72 of the door 70 and the door knob 6
2 (FIG. 8) near the reference block 61′-1 and the target block 7 near the corner of the window frame 73.
1-2 is similar to the reference block 61-2 near the corner of the window frame 63 (FIG. 8), the target block 71-
1, 71-2 are respectively extracted, and the coordinates of the center thereof are output to the vector extraction unit 14.

If the reference block in the fourth frame is not updated, the old reference block 31'-1,
31'-2 is used to perform the pattern matching. Since the amount of expansion and rotation of the fifth frame is too large compared to the first frame, the reference block 31′-
It is expected that it will be impossible to search for a part similar to 1,31′-2. However, in practice, the reference block is updated to 61′-1, 61′-2 of the fourth frame, and the amount of enlargement and rotation in the fifth frame is reduced, so that similar portions are searched for. Is possible.

In step S4, the block extraction unit 1
2 outputs the center coordinate position of the target blocks 71-1 and 71-2 to the locus storage unit 18 as locus information and stores the locus information.

In step S5, the vector extraction unit 1
4 is the target block 71-, as shown in FIG.
The target vector 74 is obtained from the coordinates of the center of 1, 71-2, and is output to the comparison unit 15.

In step S6, the comparison section 15 determines whether or not the reference vector is stored in the reference vector storage section 16. In this case, the reference vector 64 'is stored. Go to step S10.

In step S10, the comparison section 15
Difference d between the reference vector 64 'and the target vector 74
Ask for 5. More specifically, as shown in FIG. 11, the comparison unit 15 sets the reference vector 64 ′ and the target vector 7
4 is obtained, and the absolute value S5 ′ of the difference vector Vd5 is converted to the absolute value S of the reference vector 64 ′.
The difference d5 (S5 ′ / S2) is obtained by dividing by 2.

In step S11, the comparison section 15
It is determined whether or not the difference d5> threshold value D. In the present case,
As shown in FIG. 11, the absolute value S of the difference vector Vd5
Since 5 ′ is not ½ of the reference vector 64 ′, the difference d5> threshold D is not satisfied, and the process proceeds to step S9.

In step S9, since the sixth frame is input, the process returns to step S2. here,
In step S2, the image memory 17 stores the image of the sixth frame input from the image input unit 11 as shown in FIG.

In step S3, when the sixth frame as shown in FIG. 12 is input, the block extraction unit 12
Performs pattern matching using the reference blocks 61′-1 and 61′-2 stored in the reference block storage unit 13 to target the target block 81-1 near the door knob 82 similar to the reference block 61′-1. And the target blocks 81-2 near the corners of the window frame 83 similar to the reference block 61′-2 are extracted, and the coordinates of the central portion are output to the vector extraction unit 14, respectively.

In step S4, the block extraction unit 1
2 outputs the center coordinate position of the target blocks 81-1 and 81-2 to the locus storage unit 18 as locus information to be stored therein.

In step S5, the vector extraction unit 1
4 is the input target blocks 81-1 and 81-
As shown in FIG. 12, a target vector 84 is obtained from the coordinates of the center of 2 and is output to the comparison unit 15.

In step S6, since the reference vector is already stored in the reference vector storage unit 16,
When it is determined that the reference vector is stored, step S
10, the comparing unit 15 uses the method similar to the method illustrated in FIG. 11 to calculate the difference vector Vd6 as illustrated in FIG.
Is calculated to obtain the difference d6. More specifically, the comparison unit 1
13, the difference vector Vd6 between the reference vector 64 'and the target vector 84 is obtained, and the absolute value S6' of the difference vector Vd6 is divided by the absolute value S2 of the reference vector to obtain the difference d5 (S6 '/ S
2) is asked.

In step S11, the comparison section 17
It is determined whether or not the difference d6> threshold value D. In the present case,
As shown in FIG. 13, the absolute value S of the difference vector Vd6
Since 6 ′ is not ½ of the reference vector 64 ′, the difference d6> threshold D is not satisfied, and the process proceeds to step S9. Since the next frame is not input, the process is step S12. Proceed to.

In step S12, the image stabilization unit 1
Reference numeral 9 reads the image stored in the image memory 17, corrects the camera shake based on the trajectory information stored in the trajectory storage unit 18, and outputs the image with the camera shake corrected from the image output unit 20. That is, the locus storage unit 18 stores the coordinates of the central portions of the target blocks 31 to 81. The locus of this coordinate is the door knobs 32 to 82 on the image.
And the loci of the corners of the window frames 33 to 83, the image blur correction unit 19 shifts the frame image according to the locus to correct the image blur, and the image output unit 20 outputs the image with the image blur corrected. To do.

As described above, the difference dn obtained from the absolute value of the reference vector and the absolute value of the difference vector Vdn is
When it becomes larger than the threshold value D (when the fourth frame is input in the above description), the reference block and the reference vector as information indicating the positional relationship between the reference blocks are the target blocks obtained at that time. By replacing the target block with the target vector as the information indicating the positional relationship between the target blocks, the reference block and the reference vector can be updated in accordance with the motion, and thus proper image stabilization can be performed. As a result, for example, in the case of severe camera shake or in the case of an image with a lot of motion, the reference block and the reference vector are updated for each frame image, and there is almost no camera shake or an image with little motion. In addition, it is possible to prevent the reference block from being updated, and it is possible to perform the camera shake correction process according to the degree of camera shake and the movement of the image.

Furthermore, it is also possible to synthesize a moving image by using this movement locus, and when synthesizing a projected image of a virtual object created by CG (computer graphics) with the moving image, Since the trajectory of the projected image of the object shown in the moving image can be obtained, for example, even in the process of putting a virtual helmet made of CG on an actual person shown in the moving image, a moving image can be obtained. It is possible to find the trajectory of the person's head in the image and synthesize the projected image of the helmet at that position.

In the above example, the difference vector is obtained from the reference vector and the target vector obtained from the reference block and the target block, respectively, and the reference vector is updated by comparing the difference based on the difference vector with the threshold value. By the way, an example in which the image stabilization is appropriately processed has been described, but the reference block in the image stabilization can be updated also from the enlargement ratio (or reduction ratio) and the rotation angle of the reference block with respect to the target block. .

Therefore, the configuration of an image processing apparatus for performing camera shake correction processing from the enlargement ratio (or reduction ratio) of the reference block with respect to the target block and the rotation angle will be described next with reference to the block diagram of FIG. In the figure,
Portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate.

When the reference block storage unit 94 does not store a reference block among the input frame images, the block extraction unit 91 uses the AC portion of the input frame image as a reference. It is extracted as a block and stored in the reference block storage unit 94. For example, when a frame image in which the door 110 and the window frame 113 are captured as shown in FIG. 15 is input at the same time as the image capturing is started, when the door knob 112 of the door 110 is the image of the AC portion, the block is displayed. The extraction unit 91 extracts a portion corresponding to the door knob 112 in the drawing as a reference block 111 ′ and stores it in the reference block storage unit 94.

Further, when the reference block is stored in the reference block storage unit 94 (when the reference block is stored once), the block extraction unit 91 stores the sequentially input frame images in the reference block storage unit 94. A block similar to the stored reference block is searched by pattern matching, and this block is extracted as a target block and output to the transformation parameter extraction unit 92. That is, for example, as shown in FIG. 15, in a state where the reference block 111 ′ is stored in the reference block storage unit 94, for example, a frame image in which the door 120 and the window frame 123 are imaged as shown in FIG. When input, the block extraction unit 91
The target block 121 near the door knob 122 of the door 120 is searched by performing pattern matching using the reference block 111 ′ near the door knob 112 stored in the reference block storage unit 94. Output to the extraction unit 92.

Further, the block extraction unit 91 temporarily stores the extracted target block until the processing of the next frame is started, receives the notification of the comparison result from the comparison unit 93, and extracts at that timing. Then, the target block of the frame image (stored at that time) is overwritten on the data of the reference block storage unit 94, and the reference block is updated.

The block extraction unit 91 also outputs the coordinates of the center of the extracted target block to the locus storage unit 18 as locus information.

The transformation parameter extraction unit 92, with respect to the target block input from the block extraction unit 91,
The expansion rate L and the rotation angle θ of the reference block stored in the reference block 94 are obtained as the deformation parameter (L, θ) and output to the comparison unit 93. The enlargement ratio here is a ratio (enlargement ratio) that represents the size of the texture of the target block when the size of the texture of the reference block is used as a reference, and is larger than that of the reference block 111 ′ in FIG. 16, the target block 121 of FIG.
As shown by, the enlargement ratio L is L2 / L1 and the rotation angle is θ2. For example, as shown in FIG. 17, L1, L
2 is the size of the door knobs 112 and 113, and the rotation angle of the door knob is θ2. The same applies below.

Further, in practice, the reference block and the target block have slightly different shapes because they were photographed at different times, and since noise is also included, scaling (L times) and rotation ( Even if they are deformed by θ), they do not match. Therefore, here, the value that minimizes the difference from the target block by scaling and rotating the reference block is the deformation parameter (L, θ). Is required as. That is, the deformation parameter (L, θ) when the reference block and the target block are most similar to each other is obtained by variously changing the values of the deformation parameter (L, θ). It should be noted that the transformation parameter (L, θ) here is also a transformation parameter for transforming the reference block into an target block by affine transformation (conversion using a determinant of rotation and scaling parameters). I can say.

The comparison unit 93 includes a transformation parameter extraction unit 92.
Based on the deformation parameters (L, θ) input by
The difference g is calculated and compared with the threshold value G, and according to the comparison result, the reference block storage unit 9 sets the current target block as the reference block to the block extraction unit 91.
4 to update. Here, the difference g is
For example, g in the case of FIG. 18 is shown and the difference g (= √ (L ² + 1-2 × L × c) with respect to the deformation parameter (L, θ).
osθ)) is defined. This difference g indicates the amount of change caused by rotating or enlarging the reference block with respect to the target block.

Next, with reference to the flow chart of FIG. 19, the camera shake correction processing by the image processing apparatus of FIG. 14 when the frame images from the first frame to the sixth frame are input will be described.

In step S31, the block extraction unit 91 extracts the AC portion of the first frame, ie, the first frame, input from the image input unit 11 as shown in FIG. It is stored in the storage unit 94. That is, the door 110 and the window frame 1
When the first frame image of 13 is input at the same time as the start of image capturing, and the door knob 112 is an AC image, the block extracting unit 91 extracts the reference block 111 ′ corresponding to the door knob 112 in the figure. Then, it is stored in the reference block storage unit 94 and output to the transformation parameter extraction unit 92.

In step S32, the image memory 17
Stores the frame image input from the image input unit 11, that is, the image of the first frame in this case.

In step S33, the block extraction unit 91 extracts the target block from the input first frame by using the reference block 111 'stored in the reference block storage unit 94. In the case of FIG. 15, the block extraction unit 91 extracts the target block 111 by pattern matching. In the first process, since the reference block 111 ′ and the target block 111 are the same object, the reference block 111 ′ is the deformation parameter extraction unit 9
2 will be output.

In step S34, the block extraction unit 91 outputs the center coordinate position of the target block 111 to the locus storage unit 18 as locus information and stores it therein.

In step S35, the transformation parameter extraction unit 92 inputs the target block 111 that has been input.
And the reference block 111 ′ stored in the reference block storage unit 94 are compared with each other to obtain the deformation parameter (L, θ) and output to the comparison unit 93. In this case, since it is the first frame, the reference block 111 ′ and the target block 11
Since 1 is the same as shown in FIG. 15, the transformation parameter (1, 0) is extracted and output to the comparison unit 93.

In step S36, the comparison section 93
The difference g1 between the target block and the reference block is calculated. In this case, the difference g1 has a deformation parameter of (1,
Since it is 0), it is calculated as 0. In the following,
The difference of the nth frame is called a difference gn.

In step S37, the comparison section 93
The difference g1 is compared with a predetermined threshold G to determine whether the difference gn> the threshold G. In this case, the difference g1 is 0 and the difference g1> threshold value G is not satisfied, and therefore the process proceeds to step S38.

In step S38, it is determined whether or not the next frame image exists. In this case, the second frame shown in FIG. 16 is input, so the process returns to step S32.

In step S32, the image memory 17
Stores the frame image input from the image input unit 11, that is, the image of the second frame in this case.

In step S33, the block extracting section 91 extracts a target block from the input frame image by pattern matching using the reference block 111 'stored in the reference block storage section 94. That is, since the reference block 111 ′ corresponding to the door knob 112 extracted from the first frame is stored in the reference block storage unit 94, the door 120 and the window frame 1
In the second frame in which 23 is imaged, pattern matching is performed using this reference block 111 ′,
A similar part is extracted as a target block. Figure 1
In the case of the second frame of No. 6, the door knob 122 of the door 120
Since the target block 121 in the vicinity and the reference block 111 ′ in the vicinity of the door knob 112 (FIG. 15) are similar, the target block 121 is extracted and output to the deformation parameter extraction unit 92.

In step S34, the block extracting unit 91 outputs the center coordinate position of the target block 121 to the locus storage unit 18 as locus information and stores it therein.

In step S35, the transformation parameter extraction unit 92 inputs the input target block 121.
And the reference block 111 ′ stored in the reference block storage unit 94 are compared with each other to obtain the deformation parameter (L, θ) and output to the comparison unit 93. In this case, the reference block 11
1'and the target block 121 can be shown as shown in FIG. 17, for example. Therefore, when the second frame shown in FIG. 16 is input, the transformation parameter extraction unit 92
Compares the deformation parameter (L2 / L1, θ2) with the comparison unit 93.
Output to.

In step S36, the comparison section 93
A difference g2 between the target block 121 and the reference block 111 ′ is calculated. In this case, since the deformation parameter of the difference g2 is (L2 / L1, θ2), the difference g2 (= √)
((L2 / L1) ² + 1-2 × (L2 / L1) × cos θ
2)) is calculated.

In step S37, the comparison section 93
The difference g2 is compared with a predetermined threshold G to determine whether the difference g2> threshold G. For example, if the difference g2> threshold value G is not satisfied, that is, if there is no significant change in the second frame compared to the first frame, the process proceeds to step S38.

In step S38, it is determined whether or not the next frame image is present. In this case, the third frame shown in FIG. 20 is input, so the process returns to step S32.

In step S32, the image memory 17
Stores the frame image input from the image input unit 11, that is, the image of the third frame in this case.

In step S33, the block extracting unit 91 extracts a target block from the input frame image by pattern matching using the reference block 111 'stored in the reference block storage unit 94. That is, in the third frame in which the door 130 and the window frame 133 are imaged, pattern matching is performed using this reference block 111 ′, and a similar portion is extracted as a target block. In the case of the third frame of FIG. 20, since the target block 131 near the door knob 132 of the door 130 and the reference block 111 ′ near the door knob 112 (FIG. 15) are similar, the target block 131 is extracted and the deformation parameter extraction is performed. It is output to the section 92.

In step S34, the block extracting unit 91 outputs the center coordinate position of the target block 131 to the locus storage unit 18 as locus information and stores it therein.

In step S35, the transformation parameter extraction unit 92 receives the input target block 131.
And the reference block 111 ′ stored in the reference block storage unit 94 are compared with each other to obtain the deformation parameter (L, θ) and output to the comparison unit 93. In this case, the reference block 11
1'and the target block 131 can be shown as shown in FIG. 21, for example. Therefore, when the third frame shown in FIG. 20 is input, the transformation parameter extraction unit 92
Compares the deformation parameter (L3 / L1, θ3) with the comparison unit 93.
Output to.

In step S36, the comparison section 93
A difference g3 between the target block 131 and the reference block 111 ′ is calculated. In this case, since the deformation parameter of the difference g3 is (L3 / L1, θ3), the difference g3 (= √)
((L3 / L1) ² + 1-2 × (L3 / L1) × cos θ
3)) is calculated.

In step S37, the comparison section 93
The difference g3 is compared with a predetermined threshold G to determine whether the difference g3> threshold G. For example, when the difference g3> threshold value G is not satisfied, that is, when it is determined that there is no significant change in the third frame after the second frame as compared with the first frame, the process proceeds to step S38.

In step S38, it is determined whether or not the next frame image is present. In this case, the fourth frame shown in FIG. 22 is input, so the process returns to step S32.

In step S32, the image memory 17
Stores the frame image input from the image input unit 11, that is, the image of the fourth frame in this case.

In step S33, the block extraction unit 91 performs pattern matching on the input fourth frame in which the door 140 and the window frame 143 are imaged by using the reference block 111 ', and a similar portion is obtained. Extract as a target block. In the case of the fourth frame of FIG. 22, since the target block 141 near the door knob 142 of the door 140 and the reference block 111 ′ near the door knob 112 (FIG. 15) are similar, the target block 141 is extracted and the deformation parameter is extracted. It is output to the section 92.

In step S34, the block extracting unit 91 outputs the center coordinate position of the target block 141 to the locus storage unit 18 as locus information and stores it therein.

In step S35, the transformation parameter extraction unit 92 receives the input target block 141.
And the reference block 111 ′ stored in the reference block storage unit 94 are compared with each other to obtain the deformation parameter (L, θ) and output to the comparison unit 93. In this case, the reference block 11
1'and the target block 141 can be shown as shown in FIG. 23, for example. Therefore, when the fourth frame shown in FIG. 22 is input, the transformation parameter extraction unit 92
Compares the deformation parameter (L4 / L1, θ4) with the comparison unit 93.
Output to.

In step S36, the comparison section 93
The difference g4 between the target block 141 and the reference block 111 ′ is calculated. In this case, since the deformation parameter of the difference g4 is (L4 / L1, θ4), the difference g4 (= √)
((L4 / L1) ² + 1-2 × (L4 / L1) × cos θ
4)) is calculated.

In step S37, the comparison section 93
The difference g4 is compared with a predetermined threshold G to determine whether the difference g4> threshold G. For example, when it is determined that the difference g4> threshold value G, that is, when it is determined that there is a large change in the fourth frame as compared with the first frame, the process proceeds to step S39.

In step S39, the comparing section 93
The block extraction unit 91 is caused to replace the current target block as the reference block. At this time,
The block extracting unit 91 determines the reference block 111 ′ as the target block 141 obtained from the current fourth frame.
Is stored in the reference block storage unit 94 as the reference block 141 ′, and the process is performed in step S38.
Proceed to.

In step S38, the fifth frame in which the door 150 and the window frame 153 shown in FIG. 24 are picked up is input, and the process returns to step S32.

In step S32, the image memory 17
Stores the frame image input from the image input unit 11, that is, the image of the fifth frame in this case.

In the step S33, in the case of the fifth frame in FIG. 24, the block extracting unit 12 makes the target block 151 near the door knob 152 of the door 150 and the reference block 141 'near the door knob 142 (FIG. 22) similar to each other. Therefore, the target block 151 is extracted,
It is output to the transformation parameter extraction unit 92.

If the reference block in the fourth frame is not updated, pattern matching will be performed using the old reference block 111 '. First
It is expected that it will be impossible to find a part similar to the reference block 111 ′ in the fifth frame because the amount of expansion and rotation in the fifth frame is too large. However, in reality, since the reference block is updated to the reference block 141 ′ of the fourth frame, the amount of enlargement and rotation in the fifth frame becomes small, so that it is possible to search for a similar portion.

In step S34, the block extraction unit 91 outputs the center coordinate position of the target block 151 to the locus storage unit 18 as locus information and stores it therein.

In step S 35, the transformation parameter extraction unit 92 inputs the target block 151 that has been input.
And the reference block 141 ′ stored in the reference block storage unit 94 are compared with each other to obtain the deformation parameter (L, θ) and output to the comparison unit 93. In this case, the reference block 14
1 ′ and the target block 151 can be shown as shown in FIG. 25, for example. Therefore, when the fifth frame shown in FIG. 24 is input, the transformation parameter extraction unit 92
Compares the deformation parameter (L5 / L4, θ5) with the comparison unit 93.
Output to.

In step S36, the comparison section 93
The difference g5 between the target block 151 and the reference block 141 ′ is calculated. In this case, since the deformation parameter of the difference g5 is (L5 / L4, θ5), the difference g5 (= √)
((L5 / L4) ² + 1-2 × (L5 / L4) × cos θ
5)) is calculated.

In step S37, the comparison section 93
The difference g5 is compared with a predetermined threshold G to determine whether the difference g5> threshold G. For example, when it is determined that the difference g5> threshold G is not satisfied, that is, when it is determined that there is no large change in the fifth frame compared to the updated fourth frame, the process is It proceeds to step S38.

In step S38, it is determined whether or not there is a next frame image. In this case, the sixth frame shown in FIG. 26 is input, so the process returns to step S32.

In step S32, the image memory 17
Stores the frame image input from the image input unit 11, that is, the image of the sixth frame in this case.

In step S33, the block extraction unit 91 performs pattern matching on the input sixth frame in which the door 160 and the window frame 163 are picked up by using the reference block 141 ', and a similar portion is obtained. Extract as a target block. In the case of the sixth frame in FIG. 26, since the target block 161 near the door knob 162 of the door 160 and the reference block 141 ′ near the door knob 142 (FIG. 22) are similar, the target block 161 is extracted and the deformation parameter extraction is performed. It is output to the section 92.

In step S34, the block extracting unit 91 outputs the center coordinate position of the target block 161 to the locus storage unit 18 as locus information and stores it therein.

In step S35, the transformation parameter extraction unit 92 inputs the input target block 161.
And the reference block 141 ′ stored in the reference block storage unit 94 are compared with each other to obtain the deformation parameter (L, θ) and output to the comparison unit 93. In this case, the reference block 14
1'and the target block 161 can be shown as shown in FIG. 27, for example. Therefore, when the sixth frame shown in FIG. 26 is input, the transformation parameter extraction unit 92
Compares the deformation parameter (L6 / L4, θ6) with the comparison unit 93.
Output to.

In step S36, the comparison section 93
The difference g6 between the target block 161 and the reference block 141 ′ is calculated. In this case, since the deformation parameter of the difference g6 is (L6 / L4, θ6), the difference g6 (= √)
((L6 / L4) ² + 1-2 × (L6 / L4) × cos θ
6)) is calculated.

In step S37, the comparison section 93
The difference g6 is compared with a predetermined threshold G to determine whether the difference g6> threshold G. For example, when it is determined that the difference g6> threshold value G is not satisfied, that is, when it is determined that there is no significant change in the sixth frame as compared with the fourth frame, the process proceeds to step S38.

In step S38, it is determined whether or not the next frame image is present. In this case, the processing is up to the sixth frame, so the processing is step S40.
Proceed to.

In step S40, the camera shake correction unit 1
Reference numeral 9 reads the frame image stored in the image memory 17, corrects the camera shake based on the trajectory information stored in the trajectory storage unit 18, and causes the image output unit 20 to output the image with the camera shake corrected. That is, in the locus storage unit 18,
The coordinates of the center of the target blocks 111 to 161 are stored. Since the locus of this coordinate is the locus of the door knobs 112 to 162 on the image, the image stabilization unit 19
Corrects the camera shake in accordance with this locus, and the image output unit 2
An image in which camera shake is corrected is output from 0.

If the trajectory of another object is also required when performing the camera shake correction in the processing of step S40, the processing of steps S31 to S39 is performed for that object in the same manner as described above. After that, the process of step S40 is executed.

As described above, the deformation parameter of the target block based on the reference block for each frame image is calculated based on the difference between the reference block and the target block.
By replacing the reference block with the target block obtained at that time, it is possible to update the reference block in accordance with the movement, so that proper image stabilization can be performed. As a result, for example, when the camera shake is severe, or when the image has a lot of movement, the reference block is updated for each frame image, and when there is almost no camera shake or when the image has little movement, It is possible to prevent the reference block from being updated, and it is possible to perform image stabilization processing according to the degree of image stabilization and image movement.

Further, as shown in FIG. 27, the displacement of the target block can be regarded as a locus of movement of the image of interest. Therefore, while changing the reference block by the same method, the displacement of the image of interest is noted on the frame image. Image (here
It is also possible to find the trajectory of the doorknob).

Therefore, as in the case of the image processing apparatus of FIG. 1, CG (computer graphics) is added to the moving image.
When synthesizing the projection image of the virtual object created by, the trajectory of the projection image of the object shown in the moving image can be obtained. Therefore, for example, by using CG for the actual person shown in the moving image, Even in the process of covering the virtual helmet created, the trajectory of the head of the person shown in the moving image can be obtained and the projected image of the helmet can be synthesized at that position.

In addition, in the image processing apparatus of FIG.
Since the deformation parameter (L, θ) is obtained, the reference block up to that point is enlarged (or reduced) L times by the conversion parameter, rotated by the rotation angle θ, and the reference block is updated. May be.

FIG. 28 shows an example of the structure of an image processing apparatus for updating a reference block using such deformation parameters (L, θ). In the figure, parts corresponding to those in FIG. 14 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate.

The basic function of the block extraction unit 201 is as follows.
Similar to the block extraction unit 91, but the reference block update method is different. The block extraction unit 201 includes a comparison unit 20.
The deformation parameter (L,
θ) is received and the reference block stored in the reference block storage unit 94 at that time is enlarged by the transformation unit 201a at the enlargement ratio L based on the transformation parameter (L, θ), and the rotation angle θ is obtained. After rotating (after affine transformation), the data in the reference block storage unit 94 is overwritten,
Update the reference block.

Next, with reference to the flow chart of FIG. 29, the camera shake correction processing by the image processing apparatus of FIG. 28 will be described. The processes of steps S61 to S68 and S71 in the flowchart of FIG. 29 are the same as steps S31 to S3 described with reference to the flowchart of FIG.
Since it is the same as the process of S8 and S40, the description thereof will be omitted.

In step S69, the comparison unit 202
Is the transformation parameter (L, θ) for the block extraction unit 201.
And the deformation parameter (L, θ) is used to instruct the reference block currently stored in the reference block storage unit 94 to be enlarged at the enlargement ratio L and rotated by the rotation angle θ. In step S70, the block extraction unit 201 controls the transformation unit 201a to enlarge the reference block stored in the reference block storage unit 94 at the enlargement ratio L based on the transformation parameter (L, θ). After rotating at the rotation angle θ (after affine transformation), the data in the reference block storage unit 94 is overwritten,
Update the reference block.

According to the above, since the reference block to be referred to is updated according to the image in the imaged frame during camera shake correction, it is possible to stably correct the camera shake that occurs during image pickup. become.

The series of processes described above can be executed by hardware or software. When a series of processes is executed by software, various functions can be executed by installing a computer in which a program configuring the software is incorporated in dedicated hardware or various programs. It is installed from a recording medium into a possible general-purpose personal computer or the like.

FIG. 30 shows the configuration of an embodiment of a personal computer when the image processing apparatus is realized by software. CP of personal computer
The U1001 controls the entire operation of the personal computer. Further, the CPU 1001 is responsive to a command input from a user via the bus 1004 and the input / output interface 1005 from an input unit 1006 including a keyboard, a mouse, etc.
The program stored in 1002 is executed. Alternatively, the CPU 1001 loads a program read from the magnetic disk 1021, optical disk 1022, magneto-optical disk 1023, or semiconductor memory 1024 connected to the drive 1010 and installed in the storage unit 1008 to a RAM (Random Access Memory) 1023. And executes the program, and the output unit 1007 outputs the execution result.
Further, the CPU 1001 controls the communication unit 1009,
Communicates with the outside and exchanges data.

The recording medium on which the program is recorded is
As shown in FIG. 30, a magnetic disk 1021 (including a flexible disk) on which a program is recorded and an optical disk 1022 (CD-ROM (Compact
Disc-Read Only Memory), DVD (Digital VersatileDis
c)), magneto-optical disk 1023 (MD (Mini-Dis
(including c)), or a package medium including a semiconductor memory 1024, etc.
It is configured by a ROM 1002 in which a program is recorded, which is provided to the user in a state where the computer is installed in advance, a hard disk included in the storage unit 1008, and the like.

In the present specification, the steps for writing the program recorded on the recording medium are not limited to the processing performed in time series according to the order described and not necessarily the processing performed in time series. , Which include processes executed in parallel or individually.

[0166]

According to the present invention, it is possible to more stably correct camera shake that occurs during image pickup.

[Brief description of drawings]

FIG. 1 is a block diagram of an image processing apparatus to which the present invention has been applied.

FIG. 2 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 3 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 4 is a flowchart illustrating a camera shake correction process.

FIG. 5 is a diagram illustrating a difference vector.

6 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 7 is a diagram illustrating a difference vector.

FIG. 8 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 9 is a diagram illustrating a difference vector.

FIG. 10 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 11 is a diagram illustrating a difference vector.

12 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 13 is a diagram illustrating a difference vector.

FIG. 14 is a block diagram of an image processing apparatus to which the present invention has been applied.

FIG. 15 is a diagram showing a frame image input to the image processing apparatus of FIG.

16 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 17 is a diagram illustrating a difference.

FIG. 18 is a diagram illustrating a difference.

FIG. 19 is a flowchart illustrating a camera shake correction process.

20 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 21 is a diagram illustrating a difference.

22 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 23 is a diagram illustrating a difference.

24 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 25 is a diagram illustrating a difference.

FIG. 26 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 27 is a diagram illustrating a difference.

28 is a diagram showing a frame image input to the image processing apparatus of FIG.

FIG. 29 is a flowchart illustrating a process of camera shake correction.

FIG. 30 is a diagram illustrating a medium.

[Explanation of symbols]

11 image input unit, 12 block extraction unit, 13
Reference block storage unit, 14 vector extraction unit, 15
Comparison unit, 16 reference vector storage unit, 17 image storage unit, 18 trajectory storage unit, 19 camera shake correction unit,
20 image output unit, 31, 31-1, 31-2 target block, 31 ', 31'-1, 31'-2
Reference block, 32 door knob, 33 window frame, 3
4 target vector, 34 'reference vector,
41, 41-1 and 41-2 target block, 4
2 doorknob, 43 window frame, 44 target vector, 51,51-1, 51-2 target block, 52 doorknob, 53 window frame, 54 target vector, 61,61-1, 61-2 target block, 61 ', 61 '-1,61'-2 reference block, 62 door knob, 63 window frame, 64 target vector, 64' reference vector, 71,7
1-1, 71-2 target block, 72 door knob, 73 window frame, 74 target vector, 8
1, 81-1, 81-2 Target block, 82
Door knob, 83 window frame, 84 target vector, 91 block extraction unit, 92 deformation parameter extraction unit, 93 comparison unit, 94 reference block storage unit, 95 image stabilization unit, 111 target block, 111 'reference block, 112 door knob, 121 target Block, 122 Door Knob, 131 Target Block, 132 Door Knob, 141 Target Block, 141 'Reference Block, 142 Door Knob, 151 Target Block, 152 Door Knob, 161 Target Block, 162 Door Knob, 201 Block Extraction Section, 201a Deformation Section, 202 Comparison Section

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] June 5, 2002 (2002.6.5)

[Procedure Amendment 1]

[Document name to be corrected] Drawing

[Name of item to be corrected] Fig. 19

[Correction method] Change

[Correction content]

FIG. 19

Claims (20)

[Claims] 1. A reference block extraction means for extracting a plurality of reference blocks from a frame at a predetermined time, a storage means for storing the plurality of reference blocks, and a plurality of reference blocks stored by the storage means. Reference positional relationship information generating means for generating reference positional relationship information indicating the positional relationship, and a plurality of similar blocks similar to the plurality of reference blocks stored in the storage means, from the frame after the predetermined time. Similar block extracting means, similar positional relationship information generating means for generating similar positional relationship information indicating the positional relationship between the plurality of similar blocks, and difference for generating a difference between the similar positional relationship information and the reference positional relationship information. Generating means, comparing means for comparing the difference with a predetermined threshold value, and stored in the storing means in correspondence with the comparison result. A replacement unit that replaces the plurality of reference blocks with the plurality of similar blocks, position information of a predetermined reference block of the plurality of reference blocks, and position information of a predetermined similar block of the plurality of similar blocks. And a processing unit that performs a predetermined process on the frame after the predetermined time based on the above. 2. The reference positional relationship information is a reference vector as a vector between the centers of two reference blocks, and the similar positional relationship information is a similarity vector as a vector between the centers of two similar blocks. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. 3. The image processing apparatus according to claim 2, wherein the difference is a difference between the reference vector and the similar vector. 4. The processing means, based on position information of a predetermined reference block of the plurality of reference blocks and position information of a predetermined similar block of the plurality of similar blocks, the predetermined time. The image processing apparatus according to claim 1, wherein the subsequent frame is subjected to a camera shake correction process or an image combining process as a predetermined process. 5. A reference block extracting step of extracting a plurality of reference blocks from a frame at a predetermined time, a storing step of storing the plurality of reference blocks, and the plurality of reference blocks stored in the processing of the storing step. A reference positional relationship information generating step of generating reference positional relationship information indicating a positional relationship between the plurality of similarities similar to the plurality of reference blocks stored in the processing of the storing step from the frame after the predetermined time. A similar block extracting step of extracting a block, a similar positional relationship information generating step of generating similar positional relationship information indicating a positional relationship between the plurality of similar blocks, and a difference between the similar positional relationship information and the reference positional relationship information A difference generation step of generating a difference, a comparison step of comparing the difference with a predetermined threshold, and the comparison Corresponding to the result, the replacement step of replacing the plurality of reference blocks stored in the processing of the storage step with the plurality of similar blocks, and position information of a predetermined reference block among the plurality of reference blocks, And a processing step of performing a predetermined process on a frame after the predetermined time based on position information of a predetermined similar block of the plurality of similar blocks. 6. A reference block extraction control step for controlling extraction of a plurality of reference blocks from a frame at a predetermined time, a storage control step for controlling storage of the plurality of reference blocks, and storage by processing of the storage control step. A reference positional relationship information generation control step for controlling the generation of the reference positional relationship information representing the positional relationship between the plurality of controlled reference blocks, and a frame after the predetermined time is stored in the processing of the storage control step. A similar block extraction control step of controlling extraction of a plurality of similar blocks similar to the plurality of reference blocks, and a similar position controlling generation of similar positional relationship information indicating a positional relationship between the plurality of similar blocks. A relationship information generation control step, controlling generation of a difference between the similar positional relationship information and the reference positional relationship information A minute generation control step, a comparison control step for controlling comparison with a predetermined threshold value of the difference, and a plurality of reference blocks whose storage is controlled in the processing of the storage control step, corresponding to the comparison result. A replacement control step of controlling replacement with the plurality of similar blocks, position information of a predetermined reference block of the plurality of reference blocks, and position information of a predetermined similar block of the plurality of similar blocks. A recording medium having a computer-readable program recorded thereon, the processing medium including a processing control step of controlling a predetermined processing performed on the frame after the predetermined time. 7. A reference block extraction control step for controlling extraction of a plurality of reference blocks from a frame at a predetermined time, a storage control step for controlling storage of the plurality of reference blocks, and storage by processing of the storage control step. A reference positional relationship information generation control step for controlling the generation of the reference positional relationship information representing the positional relationship between the plurality of controlled reference blocks, and a frame after the predetermined time is stored in the processing of the storage control step. A similar block extraction control step of controlling extraction of a plurality of similar blocks similar to the plurality of reference blocks, and a similar position controlling generation of similar positional relationship information indicating a positional relationship between the plurality of similar blocks. A relationship information generation control step, controlling generation of a difference between the similar positional relationship information and the reference positional relationship information A minute generation control step, a comparison control step for controlling comparison with a predetermined threshold value of the difference, and a plurality of reference blocks whose storage is controlled in the processing of the storage control step, corresponding to the comparison result. A replacement control step of controlling replacement with the plurality of similar blocks, position information of a predetermined reference block of the plurality of reference blocks, and position information of a predetermined similar block of the plurality of similar blocks. A program that causes a computer to execute a process control step for controlling a predetermined process performed on a frame after the predetermined time based on the program. 8. A reference block extracting means for extracting a reference block from a frame at a predetermined time, a storage means for storing the reference block, and a similar block similar to the reference block from a frame after the predetermined time. A similar block extracting unit, a difference generating unit that generates a difference between the reference block and the similar block, a comparing unit that compares the difference with a predetermined threshold, and a storage unit that corresponds to the comparison result. A replacement unit that replaces the stored reference block with the similar block, a processing unit that performs a predetermined process on a frame after the predetermined time based on the position information of the reference block and the position information of the similar block. An image processing apparatus comprising: 9. The image processing apparatus according to claim 8, wherein the difference is based on a conversion parameter when affine-transforming the reference block and the similar block. 10. The conversion parameter for the affine transformation includes an enlargement ratio or a reduction ratio and a rotation angle when the reference block is affine-transformed into the similar block. The image processing device described. 11. The processing means, based on position information of the reference block and position information of the similar block,
9. The image processing apparatus according to claim 8, wherein a camera shake correction process or an image synthesizing process is performed as a predetermined process on a frame after the predetermined time. 12. A reference block extracting step of extracting a reference block from a frame at a predetermined time, a storing step of storing the reference block, and a similar block similar to the reference block from frames after the predetermined time. A similar block extracting step, a difference generating step of generating a difference between the reference block and the similar block, a comparing step of comparing the difference with a predetermined threshold, and a storing step of the storing step corresponding to the comparison result. A replacing step of replacing the reference block stored by the process with the similar block, a predetermined process is performed on the frame after the predetermined time based on the position information of the reference block and the position information of the similar block. An image processing method comprising: a processing step. 13. A reference block extraction control step of controlling extraction of a reference block from a frame at a predetermined time, a storage control step of controlling storage of the reference block, and a reference block from the frame after the predetermined time to the reference block. A similar block extraction control step for controlling extraction of similar similar blocks, a difference generation control step for controlling generation of a difference between the reference block and the similar block, and a comparison for controlling comparison with a predetermined threshold of the difference A control step, a replacement control step corresponding to the comparison result, which controls replacement of the reference block whose storage is controlled by the processing of the storage step with the similar block, position information of the reference block, and Based on the position information of similar blocks, a predetermined value to be applied to frames after the predetermined time And a processing control step for controlling the processing of step 1. The recording medium having a computer-readable program recorded thereon. 14. A reference block extraction control step of controlling extraction of a reference block from a frame at a predetermined time, a storage control step of controlling storage of the reference block, and a reference block from the frame after the predetermined time to the reference block. A similar block extraction control step for controlling extraction of similar similar blocks, a difference generation control step for controlling generation of a difference between the reference block and the similar block, and a comparison for controlling comparison with a predetermined threshold of the difference A control step, a replacement control step corresponding to the comparison result, which controls replacement of the reference block whose storage is controlled by the processing of the storage step with the similar block, position information of the reference block, and Based on the position information of similar blocks, a predetermined value to be applied to frames after the predetermined time And a program for causing a computer to execute a process control step for controlling the process of. 15. A reference block extraction means for extracting a reference block from a frame at a predetermined time, a storage means for storing the reference block, and a similar block similar to the reference block from a frame after the predetermined time. Similar block extraction means, a difference generation means for generating a conversion parameter when affine-transforming the reference block into a similar block as a difference between the reference block and the similar block, and an affine transformation of the reference block by the difference. Conversion means for comparing, the comparison means for comparing the difference with a predetermined threshold value, the reference block stored in the storage means according to the comparison result, to the reference block affine-transformed by the conversion means. Replacement means for replacing and storing the position information of the reference block; Similar based on the position information of the block, the image processing apparatus characterized by comprising a processing unit for performing predetermined processing on the predetermined time after the frame. 16. The conversion parameter at the time of performing the affine transformation as the difference includes an enlargement ratio or a reduction ratio of the reference block with respect to the similar block, and a rotation angle. Image processing device. 17. The processing means, based on position information of the reference block and position information of the similar block,
The image processing apparatus according to claim 15, wherein a camera shake correction process or an image synthesizing process is performed as a predetermined process on a frame after the predetermined time. 18. A reference block extraction step of extracting a reference block from a frame at a predetermined time, a storage step of storing the reference block, and a similar block similar to the reference block from frames after the predetermined time. A similar block extraction step, a difference generation step of generating a conversion parameter when affine-transforming the reference block into a similar block as a difference between the reference block and the similar block, and an affine transformation of the reference block by the difference. And a comparison step of comparing the difference with a predetermined threshold value, and the reference block stored by the processing of the storage step is affine-transformed by the processing of the conversion step according to the comparison result. Replacement for replacing and storing with the reference block An image processing method comprising: a step, position information of the reference block, and a processing step of performing a predetermined process on a frame after the predetermined time based on the position information of the similar block. 19. A reference block extraction control step of controlling extraction of a reference block from a frame at a predetermined time, a storage control step of controlling storage of the reference block, and a reference block from the frame after the predetermined time to the reference block. A similar block extraction control step for controlling the extraction of similar similar blocks; and a difference generation for controlling generation of a conversion parameter when the reference block is affine-transformed into a similar block as a difference between the reference block and the similar block. A control step, a conversion control step of controlling the affine transformation of the reference block by the difference, a comparison control step of controlling comparison with a predetermined threshold value of the difference, and a storage control step of the storage control step according to the comparison result. The conversion block of the reference block whose storage is controlled by processing. A replacement control step for controlling replacement and storage with the reference block affine-transformed by the processing of step, position information of the reference block, and position information of the similar block, based on the frame after the predetermined time And a process control step of controlling a predetermined process of 1. A recording medium having a computer-readable program recorded thereon. 20. A reference block extraction control step of controlling extraction of a reference block from a frame at a predetermined time, a storage control step of controlling storage of the reference block, and a reference block from the frame after the predetermined time to the reference block. A similar block extraction control step for controlling the extraction of similar similar blocks; and a difference generation for controlling generation of a conversion parameter when the reference block is affine-transformed into a similar block as a difference between the reference block and the similar block. A control step, a conversion control step of controlling the affine transformation of the reference block by the difference, a comparison control step of controlling comparison with a predetermined threshold value of the difference, and a storage control step of the storage control step according to the comparison result. The conversion block of the reference block whose storage is controlled by processing. A replacement control step of controlling replacement and storage of the reference block affine-transformed by the processing of the step, position information of the reference block, and position information of the similar block, to the frame after the predetermined time. And a program for causing a computer to execute a process control step of controlling a predetermined process of.