JP4936222B2 - Motion vector collation device, image composition device, and program - Google Patents

Description

The present invention relates to a motion vector collation device, an image composition device, and a program. For example, in a photographing device such as a digital camera, a motion that can be applied to continuous image composition used as one of techniques for improving photographing sensitivity such as night scenes. The present invention relates to a vector matching device, an image composition device, and a program.

  An imaging apparatus such as a digital camera using a semiconductor imaging device such as a CCD or CMOS can change the imaging sensitivity (the imaging sensitivity of the semiconductor imaging device; also referred to as ISO sensitivity) with the camera itself. However, if the shooting sensitivity is too high, there is an adverse effect that the noise of the semiconductor imaging device increases and the image quality deteriorates.

  Therefore, in the following Patent Document 1, two images are continuously shot at a low shutter speed (for example, 1/60 seconds) and a high shutter speed (for example, 1/1000 seconds), and these images are combined, A technique is described that can substantially expand the dynamic range of a semiconductor imaging device.

  The point of this technique is that an image shot at a low shutter speed (hereinafter referred to as a low-speed image) is an image having excellent black level reproducibility, while an image shot at a high shutter speed (hereinafter referred to as a high-speed image). .), On the other hand, is an image with excellent white level reproducibility. In other words, in a low-speed image, a bright part is likely to be a white-out image, whereas in a high-speed image, a dark part is Although it has the disadvantage that it tends to be a blackened “blackout” image, by combining these two images, it compensates for the shortcomings of both images (out-of-brightness and underexposure). In other words, an image having no dynamic range, that is, an image having a wide dynamic range can be obtained.

  In the technique of this Patent Document 1, the image composition is merely detected by detecting a white-out part and a black-out part and applying the corresponding part of the image on the other side to those parts. For example, there is a drawback that it is difficult to perform accurate image composition when there is a moving subject or when there is a camera shake at the time of shooting.

  On the other hand, in Patent Document 2 below, in order to prevent camera shake that occurs in still image shooting, in the image stabilization mode, exposure is performed a plurality of times with a short exposure time, and from the images obtained as a result of these exposures. A technique is disclosed in which feature points composed of pixels with high luminance are extracted, a motion vector is obtained, coordinate transformation is performed based on these data, and an image is synthesized.

Japanese Patent No. 3110797 JP 2004-357202 A

  However, in the technique of Patent Document 2, since the feature point detection target is a high-luminance region, for example, even when the composition of continuously captured images is different, the feature point is one. There has been a problem that improper image composition is performed.

The present invention has been made in view of such a problem, and is a movement for reliably determining coincidence / non-coincidence between continuously photographed images and performing more accurate image composition. The object is to provide a vector verification device, an image composition device, and a program.

According to the first aspect of the present invention, image acquisition means for continuously acquiring a plurality of image data, and first image data among the plurality of image data continuously acquired by the image acquisition means as reference image data As a reference, the position detection means for detecting the position of the first pixel block having a large amount of feature from the reference image data, and the position detected by the position detection means as a reference, the second of the plurality of image data. The first search means for searching the position of the second pixel block corresponding to the position of the first pixel block from the tracked image data, and the first search first movement between the positions of the first pixel blocks obtained search result by means well the position of the first pixel block and the position of the second pixel block vector Holding means for holding, a changing means for changing the position of the first pixel block or the block size of the first pixel block detected by the detecting means after holding by the holding means, and the changing means A second position corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block or the position of the block size of the first pixel block changed by Second search means for searching for the position of the pixel block; the position of the changed first pixel block obtained as a result of the search by the second search means; and the position of the changed first pixel block; a second motion vector between the position of the second pixel blocks, the position of the first pixel block held by the holding means A motion vector matching means for each collating the serial first motion vector, motion based on the collation result by the motion vector checking means, for selecting a first motion vector in the second motion vector within a certain distance It comprises a vector selection means and a reliability determination means for determining the reliability of the first motion vector selected by the motion vector selection means .
The invention described in claim 2 further comprises setting means for setting a range to be collated from the first image data acquired by the image acquisition means in the invention described in claim 1, wherein the changing means includes: It includes means for changing the collation target range set by the setting means.
According to a third aspect of the present invention, in the first aspect of the present invention, the changing unit shifts the position of the first pixel block detected by the detecting unit by a predetermined pixel, thereby changing the first pixel block. The position is changed.
According to a fourth aspect of the present invention, in the first aspect of the invention, the changing unit changes the block size of the first pixel block at the position detected by the detecting unit to be small. To do.
According to a fifth aspect of the invention, there is provided image acquisition means for continuously acquiring a plurality of image data, and first image data among the plurality of image data continuously acquired by the image acquisition means as reference image data. As a reference, the position detection means for detecting the position of the first pixel block having a large amount of feature from the reference image data, and the position detected by the position detection means as a reference, the second of the plurality of image data. The first search means for searching the position of the second pixel block corresponding to the position of the first pixel block from the tracked image data, and the first search first movement between the positions of the first pixel blocks obtained search result by means well the position of the first pixel block and the position of the second pixel block vector Holding means for holding, a changing means for changing the position of the first pixel block or the block size of the first pixel block detected by the detecting means after holding by the holding means, and the changing means A second position corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block or the position of the block size of the first pixel block changed by Second search means for searching for the position of the pixel block; the position of the changed first pixel block obtained as a result of the search by the second search means; and the position of the changed first pixel block; a second motion vector between the position of the second pixel blocks, the position of the first pixel block held by the holding means A motion vector matching means for each collating the serial first motion vector, based on the collation result by the motion vector checking means, the position and the first motion vector of the first pixel block held in said holding means projection motion vector selecting means, based on the position and the motion vector of said set of first pixel blocks of a predetermined number selected by the motion vector selection unit, the second image data for selecting a predetermined number of sets The image processing apparatus is characterized by comprising combining means for converting and combining with the first image data.
The invention according to claim 6 further comprises setting means for setting a range to be collated from the first image data acquired by the image acquisition means in the invention according to claim 5, wherein the changing means comprises: It includes means for changing the collation target range set by the setting means.
According to a seventh aspect of the present invention, in the invention according to the fifth aspect, the changing unit shifts the position of the first pixel block detected by the detecting unit by a predetermined pixel, thereby shifting the first pixel block. The position is changed.
The invention according to claim 8 is the invention according to claim 5, wherein the changing means changes so as to reduce the block size of the first pixel block at the position detected by the detecting means. To do.
According to the ninth aspect of the present invention, the computer uses the image acquisition means for continuously acquiring a plurality of image data, and the first image data of the plurality of image data continuously acquired by the image acquisition means as a reference. As image data, position detection means for detecting the position of the first pixel block having a large amount of feature from the reference image data, and the position detected by the position detection means as a reference, the first of the plurality of image data. The first search means for searching for the position of the second pixel block corresponding to the position of the first pixel block from the tracked image data, using the second image data as the tracked image data, and the first search means during a first position of said first pixel blocks obtained search result by the search means and the position of the first pixel block and the position of the second pixel block Holding means for holding a motion vector, after holding by the holding means, changing means for changing the block size of the position or the first pixel blocks of said first pixel blocks detected by said detecting means, the changing means A second position corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block or the position of the block size of the first pixel block changed by Second search means for searching for the position of the pixel block, the position of the changed first pixel block obtained as a result of the search by the second search means, and the position of the changed first pixel block, a second motion vector between the position of the second pixel block, the first pixel block held by the holding means Motion vector matching means for each matching said the location motion vector, based on the collation result by the motion vector checking means, the motion vector selection for selecting a first motion vector in the second motion vector within a certain distance And a function of reliability determination means for determining the reliability of the first motion vector selected by the motion vector selection means .
According to a tenth aspect of the present invention, there is provided a computer based on image acquisition means for continuously acquiring a plurality of image data, and based on the first image data of the plurality of image data continuously acquired by the image acquisition means. As image data, position detection means for detecting the position of the first pixel block having a large amount of feature from the reference image data, and the position detected by the position detection means as a reference, the first of the plurality of image data. The first search means for searching for the position of the second pixel block corresponding to the position of the first pixel block from the tracked image data, using the second image data as the tracked image data, and the first search means the between the positions of the first pixel blocks obtained search result by the search means and the position of the first pixel block and the position of the second pixel block Holding means for holding a motion vector of after holding by the holding means, changing means for changing the block size of the position or the first pixel blocks of said first pixel blocks detected by said detecting means, this change Second position corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block changed by the means or the position of the block size of the first pixel block . Second search means for searching for the position of the pixel block, a position of the changed first pixel block obtained as a result of the search by the second search means, and a position of the changed first pixel block, wherein a second motion vector between the position of the second pixel blocks, wherein the first pixel block held by the holding means Position and the first motion vector matching means for each collating the motion vector, on the basis of the collation result by the motion vector checking means, the position of the first motion vector of the first pixel block held in said holding means motion vector selection unit for the set to select a predetermined number, based on the position and the motion vector of the first pixel block of the set of a predetermined number selected by the motion vector selection unit, projecting the second image data It is characterized by functioning as a combining means for converting and combining with the first image data.

  According to the present invention, it is possible to reliably determine coincidence / non-coincidence between continuously shot images, and to perform image synthesis with higher accuracy.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a configuration diagram of a photographing apparatus according to the embodiment. In this figure, a photographing apparatus 1 forms an electric image by forming an optical system 2 including a photographing lens and a diaphragm mechanism, a shutter mechanism 3 that can mechanically block an optical path, and light passing through the optical system 2 and the shutter mechanism 3. CCD 4 for converting to a typical image signal, A / D converter 5 for converting the output of CCD 4 from analog to digital, DRAM 6 for temporarily holding the output of A / D converter 5, and an image signal held in DRAM 6 Is converted into a predetermined display format and output to the liquid crystal display screen 8, the liquid crystal display screen 8 displaying the image output from the liquid crystal display controller 7, and the image displayed on the liquid crystal display screen 8 is desired. The shutter button 9 that is operated by the user when the composition becomes, and the mode of the photographing apparatus 1 is set to the normal photographing mode or the continuous photographing mode, or these modes are set. Mode button 10 for or in or or predetermined system setting mode or the like in the opposite or in the playback mode of recorded images, the integrated circuit 11, and an external storage memory M for storing the captured image.

  The integrated circuit unit 11 includes a shutter control unit 12 that controls the shutter mechanism 3, a light reception control unit 13 that controls the CCD 4 and the A / D converter 5, and image signals stored in the DRAM 6 as YUV signals (color signals) and luminance. A demosaic unit 14 for converting to a signal, a feature amount calculation unit 15 for calculating feature amounts of two images shot in the continuous shooting mode, and an SRAM 16 used as a work memory for feature amount calculation, block matching, and image synthesis The block matching unit 17 that performs block matching of two images shot in the continuous shooting mode, and the image deformation / synthesis adding unit 18 that performs image deformation and synthesis / addition processing of the two images based on the result of the block matching. A CPU core 19 that controls the overall operation of the integrated circuit unit 11 is included.

  FIG. 2 is a diagram showing a schematic flow of image composition processing in the present embodiment. In this flow, first, the user operates the mode button 10 to enter the continuous shooting mode, adjusts to a desired composition while viewing the display on the liquid crystal display screen 8, and then operates the shutter button 9 to display two images. When continuous shooting (step S1) is performed, these two images are captured as raw image (RAW image) data from the CCD 4 to the DRAM via the A / D converter 5. At this time, it is assumed that there is a slight positional shift caused by camera shake in the two captured images.

  The two captured RAW images are converted into two types of images, a YUV image and a luminance image, by the demosaic unit 14 of the integrated circuit unit 11 and stored in the DRAM 6. The block matching unit 17 designates one of the two luminance images as a “reference image” and the other image as a “tracked image” (the side to be combined and added to the reference image). Block matching is performed between the reference image and the tracked image. Thus, the two luminance images are used for block matching, and the YUV image is used as combined addition data described later.

Next, window setting on the reference image side (first image) is performed (step S2).
FIG. 3 is a conceptual diagram of window setting. As shown in this figure, the entire reference image 20 is divided into a plurality of areas of the same size (here, 4 × 3 = 12 areas E0 to E11 as an example), and these 12 areas E0 to E11 One window W is formed. Each area is further divided into a plurality of blocks of the same size (here, 3 × 3 = 9 blocks B0 to B8 as an example), and each block B0 to B8 has the same number of pixels (16 × here). 16 pixels). A fixed offset OFST is set between the origin P0 of the reference image 20 and the origin P1 of the window W.

  Next, for each block B <b> 0 to B <b> 8 set and classified in step S <b> 2, a feature amount is obtained using a Harris corner detection evaluation function. For each area E0 to E11, the block Bi having the largest (highest) feature value is selected as a block matching block (step S3).

  FIG. 4 is a diagram showing a selection block. In this figure, the hatched blocks indicate an example of the block Bi having the largest feature amount in each of the areas E0 to E11. For example, in the area E0, the block B7 (indicated as E0_B7 in the figure), the area E1 Block B6 (denoted as E1_B6 in the figure), block B1 (denoted as E2_B1 in the figure) in area E2, block B7 (denoted as E3_B7 in the figure) in area E3, block B5 (denoted as E4_B5 in the figure) in area E4 In area E5, block B2 (indicated as E5_B2 in the figure), in area E6, block B4 (indicated in the figure as E6_B4), in area E7, in block B3 (indicated as E7_B3 in the figure), in area E8 Block B4 (E8 in the figure) B4 (indicated as E9_B6 in the figure), block B1 (indicated as E10_B1 in the figure) in area E10, and block B4 (indicated as E11_B4 in the figure) in area E11. The block Bi has the largest feature amount in the area.

  Next, with respect to each selected block Bi, the block matching unit 17 performs block matching to obtain a motion vector (step S4). Specifically, the sum of squares of differences from the selected block of the reference image is obtained for each coordinate within a designated search range around the same coordinate block on the tracked image side (second image). Then, the motion vector of the block is obtained from the position having the smallest difference square sum. At this time, since a large number of multiplications of 16 × 16 = 256 times are performed for one block matching, the block on the reference image side and the data in the search area on the tracked image side are once integrated circuit unit 11. The block matching unit 17 performs high-speed parallel processing using the acquired data.

  Next, using the motion vector data of each block, for example, by the RANSAC method, outliers (motion vectors of blocks moved irregularly due to subject movement, etc.) are excluded, and a projective transformation matrix representing the positional relationship between the two images is obtained. Obtained (step S5). RANSAC (RANdom Sample Consensus) is a method of parameter estimation. For a parameter estimation candidate obtained from a small number of points, the number of points that match the estimation from a large number of points and the accuracy of the matching This is a method of calculating the degree of the evaluation, that is, the number of supports and adopting an estimation candidate having a large number of supports as a final estimation result.

  Next, the reliability of the motion vector data of each block is determined (step S6). This determination can be made based on, for example, whether or not the support rate of the projective transformation matrix obtained by the RANSAC method is a certain value or more. The support rate of the projective transformation matrix obtained by the RANSAC method indicates the ratio of inliers (samples for which the transformation matrix is valid) to the total number of samples. If this support rate is a certain value or more, the projective transformation by the RANSAC method This is because the generation of the matrix can be regarded as successful.

  If it is determined in step S6 that the motion vector data of each block is reliable (if the determination result in step S6 is “YES”), then, based on the transformation matrix, a composite addition of the two images is performed. (Step S7), the combined and added image data is stored in the external storage memory M (step S8), and the flow is terminated.

  On the other hand, when it is determined in step S6 that the motion vector data of each block is not reliable (when the determination result in step S6 is “NO”), that is, the support rate of the projection transformation matrix obtained by the RANSAC method is constant. If it is not equal to or greater than the value, next, the motion vector data of each block, which is a feature of the present embodiment, is verified according to the following processing.

First, a window Wa is set by shifting the origin P1 of the window W set in the previous step S2 by a predetermined amount (step S9).
FIG. 5 is a conceptual diagram of setting the window Wa. As shown in (a) of this figure, the origin P1 ′ of the window Wa is obtained by shifting the origin P1 of the window W set in the previous step S2 by a predetermined amount in the lower right direction. As shown in (b), the amount corresponds to 1/2 of the vertical and horizontal size of one block.

  The area and block configuration of the window Wa are the same as those of the window W set in the previous step S2. That is, this window Wa is also divided into a plurality of areas E0 to E11 of the same size as a whole of the reference image 20, and these 12 areas E0 to E11 constitute one window W. It is divided into a plurality of blocks B0 to B8 of the same size, and each block B0 to B8 is common in that it is composed of the same number of pixels (16 × 16 pixels). The difference is that the offset from the origin P0 of the reference image 20 to the origin P1 ′ of the window Wa is larger than the offset OFST of the window W set in the previous step S2 by the predetermined amount.

  Next, as in step S3, a block having a high feature value is selected for each area E0 to E11 of the window Wa (step S10). At this time, since the origin P1 ′ of the window Wa is shifted by 1/2 block (see FIG. 5B), the block at the same position as the block selected in the above step S3 is not selected.

  Next, as in step S4, block matching unit 17 performs block matching on each selected block Bi to obtain a motion vector (step S11).

  The motion vector obtained in the previous step S4 (motion vector obtained by applying the window W) and the motion vector obtained in step S11 (motion vector obtained by applying the window Wa) are Collation is performed for each of the areas E0 to E11, and only the motion vector group in which both motion vectors are within a certain distance is selected (step S12).

  Next, outlier removal and projective transformation matrix generation by the RANSAC method is attempted again using the selected motion vector group (step S13).

  Then, the reliability of the motion vector data of each block is determined (step S14). This determination can also be made by checking whether or not the support rate of the projective transformation matrix generated in step S13 is equal to or greater than a certain value, as in the previous step S6.

  If the support rate is equal to or greater than a certain value, it is determined that the motion vector data of each block is reliable (the determination result of step S14 is “YES”), and then two images are obtained based on the transformation matrix obtained in step S13. (Step S7), the combined and added image data is stored in the external storage memory M (step S8), and the flow is terminated. If the support rate is not equal to or higher than a certain value, When it is determined that the motion vector data of the block is not reliable (determination result “NO” in step S14), a required warning message (for example, “Image composition failed. Please try again”). The information is displayed on the liquid crystal display screen 8 (step S15), and the flow ends.

  As described above, in this embodiment, when the reliability of the motion vector obtained by block matching in each area is not sufficient (“NO” in step S6 in FIG. 2), the reliability is simply insufficient and synthesis is impossible. Since not only the determination but also a new process for verifying the validity of the data by collating with another motion vector in the same area (steps S9 to S14 in FIG. 2) is newly executed, block matching In addition to improving the reliability of the data, even if it is temporarily determined that the synthesis is impossible (“NO” in step S6 in FIG. 2), the validity verification of the newly executed data Depending on the result (determination result in step S14 in FIG. 2), still image alignment synthesis may be successful, and the success rate of image synthesis can be increased. Effect can be obtained.

  6 and 7 are examples of images showing the effects of the present embodiment. Specifically, FIG. 6 is an image when the present embodiment is not applied (when steps S9 to S14 in FIG. 2 are not performed). FIG. 7 shows an example of an image when this embodiment is applied (when Steps S9 to S14 of FIG. 2 are executed). Incidentally, the image in FIG. 6 is obtained when the RANSAC support rate is 59.6% when steps S9 to S14 in FIG. 2 are not executed, and the image in FIG. 7 is obtained in steps S9 to S14 in FIG. When RANSAC is supported, the support rate is 89.1%. When comparing the two images, blurring remains in the entire image of FIG. 6, but blurring is hardly noticeable in the image of FIG. 7, and this embodiment is applied (image of FIG. 7). The advantage of has been revealed.

  In the above-described embodiment, the data of the same area obtained by shifting the origin of the search window (P1 → P1 ′: see FIG. 5) is collated. However, the present invention is not limited to this. You may make it like the modification of.

  FIG. 8 is a diagram illustrating a first modification. As shown in this figure, the matching data of the block with the largest feature amount in each area of the reference image 20 is matched with the matching data of the block shifted by several pixels from the block with the largest feature amount, and each vector data If the values are close to each other, data having the largest feature amount may be adopted.

  FIG. 9 is a diagram illustrating a second modification. As shown in this figure, the matching data of the block having the largest feature amount in each area of the reference image 20 is collated with the matching data of the block having the smallest feature amount and the size of the block having the largest feature amount. If the values are close to each other, data having the largest feature amount may be adopted.

It is a block diagram of the imaging device which concerns on embodiment. It is a figure which shows the schematic flow of the image composition process in this embodiment. It is a conceptual diagram of a window setting. It is a figure which shows the selection block. It is a setting conceptual diagram of window Wa. It is a figure which shows the example of an image when not applying this embodiment (when not performing step S9-step S14 of FIG. 2). It is a figure which shows the example of an image when this embodiment is applied (when step S9-step S14 of FIG. 2 are performed). It is a figure which shows a 1st modification. It is a figure which shows the 2nd modification.

Explanation of symbols

1 Shooting device 4 CCD
15 Feature Quantity Calculation Unit 16 SRAM
17 Block matching unit 18 Image deformation synthesis addition unit 19 CPU core

Claims (10)

Image acquisition means for continuously acquiring a plurality of image data;
Position for detecting the position of the first pixel block having a large amount of feature from the reference image data using the first image data of the plurality of image data continuously acquired by the image acquisition means as the reference image data. Detection means;
Using the position detected by the position detection means as a reference, second image data of the plurality of image data as tracked image data, and corresponding to the position of the first pixel block from the tracked image data a first search means for searching the position of the second pixel block,
The position of the first pixel block obtained as a result of the search by the first search means and the first motion vector between the position of the first pixel block and the position of the second pixel block are held. Holding means;
Change means for changing the position of the first pixel block detected by the detection means or the block size of the first pixel block after the holding by the holding means;
Corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block or the position of the block size of the first pixel block changed by the changing means. Second search means for searching for the position of the second pixel block;
The position of the changed first pixel block obtained as a result of the search by the second search means and the second position between the changed position of the first pixel block and the position of the second pixel block . A motion vector collating unit that collates each of the first motion vector with a position of the first pixel block held by the holding unit, and the first motion vector ,
A motion vector selecting means for selecting a first motion vector within a certain distance from the second motion vector based on the result of matching by the motion vector matching means;
A motion vector collating apparatus comprising: a reliability determining unit that determines the reliability of the first motion vector selected by the motion vector selecting unit . Further comprising setting means for setting a range to be collated from the first image data acquired by the image acquisition means;
2. The motion vector matching apparatus according to claim 1, wherein the changing means includes means for changing a matching target range set by the setting means. The motion vector according to claim 1, wherein the changing unit changes the position of the first pixel block by shifting the position of the first pixel block detected by the detecting unit by a predetermined pixel. Verification device. The motion vector matching apparatus according to claim 1, wherein the changing unit changes the block size of the first pixel block at the position detected by the detecting unit to be small. Image acquisition means for continuously acquiring a plurality of image data;
Position for detecting the position of the first pixel block having a large amount of feature from the reference image data using the first image data of the plurality of image data continuously acquired by the image acquisition means as the reference image data. Detection means;
Using the position detected by the position detection means as a reference, second image data of the plurality of image data as tracked image data, and corresponding to the position of the first pixel block from the tracked image data a first search means for searching the position of the second pixel block,
The position of the first pixel block obtained as a result of the search by the first search means and the first motion vector between the position of the first pixel block and the position of the second pixel block are held. Holding means;
Change means for changing the position of the first pixel block detected by the detection means or the block size of the first pixel block after the holding by the holding means;
Corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block or the position of the block size of the first pixel block changed by the changing means. Second search means for searching for the position of the second pixel block;
The position of the changed first pixel block obtained as a result of the search by the second search means and the second position between the changed position of the first pixel block and the position of the second pixel block . A motion vector collating unit that collates each of the first motion vector with a position of the first pixel block held by the holding unit, and the first motion vector ,
A motion vector selecting means for selecting a predetermined number of sets of the positions of the first pixel blocks and the first motion vectors held in the holding means based on the result of matching by the motion vector matching means ;
Based on the positions and motion vectors of the first pixel blocks of a predetermined number of sets selected by the motion vector selection means , the second image data is projectively transformed and synthesized with the first image data. And an image synthesizing device. Further comprising setting means for setting a range to be collated from the first image data acquired by the image acquisition means;
6. The image synthesizing apparatus according to claim 5, wherein the changing means includes means for changing a collation target range set by the setting means.   6. The image composition according to claim 5, wherein the changing unit changes the position of the first pixel block by shifting the position of the first pixel block detected by the detecting unit by a predetermined pixel. apparatus.   The image synthesizing apparatus according to claim 5, wherein the changing unit changes the block size of the first pixel block at the position detected by the detecting unit to be small. Computer
Image acquisition means for continuously acquiring a plurality of image data;
Position for detecting the position of the first pixel block having a large amount of feature from the reference image data using the first image data of the plurality of image data continuously acquired by the image acquisition means as the reference image data. Detection means,
Using the position detected by the position detection means as a reference, second image data of the plurality of image data as tracked image data, and corresponding to the position of the first pixel block from the tracked image data a first search means for searching the position of the second pixel block,
The position of the first pixel block obtained as a result of the search by the first search means and the first motion vector between the position of the first pixel block and the position of the second pixel block are held. Holding means,
Change means for changing the position of the first pixel block or the block size of the first pixel block detected by the detection means after being held by the holding means;
Corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block or the position of the block size of the first pixel block changed by the changing means. Second search means for searching for the position of the second pixel block;
The position of the changed first pixel block obtained as a result of the search by the second search means and the second position between the changed position of the first pixel block and the position of the second pixel block . the motion vector and the motion vector matching means, each matching the position of the first pixel block held between the motion vector by said holding means,
A motion vector selecting means for selecting a first motion vector within a certain distance from the second motion vector based on the result of matching by the motion vector matching means;
A program that functions as a reliability determination unit that determines the reliability of the first motion vector selected by the motion vector selection unit . Computer
Image acquisition means for continuously acquiring a plurality of image data;
Position for detecting the position of the first pixel block having a large amount of feature from the reference image data using the first image data of the plurality of image data continuously acquired by the image acquisition means as the reference image data. Detection means,
Using the position detected by the position detection means as a reference, second image data of the plurality of image data as tracked image data, and corresponding to the position of the first pixel block from the tracked image data a first search means for searching the position of the second pixel block,
The position of the first pixel block obtained as a result of the search by the first search means and the first motion vector between the position of the first pixel block and the position of the second pixel block are held. Holding means,
Change means for changing the position of the first pixel block or the block size of the first pixel block detected by the detection means after being held by the holding means;
Corresponding to the position of the first pixel block from the second image data with reference to the position of the first pixel block or the position of the block size of the first pixel block changed by the changing means. Second search means for searching for the position of the second pixel block;
The position of the changed first pixel block obtained as a result of the search by the second search means and the second position between the changed position of the first pixel block and the position of the second pixel block . of a motion vector, a motion vector matching means for each collating the position and the first motion vector of the first pixel block held by said holding means,
Motion vector selection means for selecting a predetermined number of pairs of the positions of the first pixel blocks and the first motion vectors held in the holding means based on the result of matching by the motion vector matching means;
Based on the positions and motion vectors of the first pixel blocks of a predetermined number of sets selected by the motion vector selection means , the second image data is projectively transformed and synthesized with the first image data. A program characterized by functioning as a synthesis means.

Images