JP4321251B2 - Apparatus and method for generating and displaying composite image - Google Patents

Description

  The present invention relates to an apparatus for automatically generating and displaying a composite image by calculating a common portion from a plurality of images and connecting the images, and more specifically, by comparing a plurality of edge features and hue feature groups in a plurality of images. The present invention relates to an apparatus and a method for detecting a common part of images, generating one or a plurality of images synthesized by reducing miscorrespondence between feature points based on the relative positional relationship of feature points, and displaying them.

  When one image is generated from a plurality of images, a method is generally used in which a human determines a common feature in each image and generates a single image by manual selection. In addition, when automating, it may be difficult to extract many common features in each image, and in the matching of feature points based on correlation calculation, miscorrespondence is often detected. It is known that it is difficult to synthesize the image automatically and accurately (see, for example, Patent Document 1 and Non-Patent Documents 1 to 3).

JP 2000-90232 A

Naoki Chiba, Haruo Hatanaka, Takashi Iida, “High-speed and high-accuracy panoramic image synthesis software based on image features”, Sanyo Electric Technical Report, Vol.35, No.1, pp.75-82, 2003. John Krumm, “Object Detection with Vector Quantized Binary Features”, Proc. Of the 1997 IEEE Int. Conf. On Computer Vision and Pattern Recognition, pp.179-185, 1997. Tatsuya Yoshida, Masataka Kagezawa, Tetsuya Hatanaka, Katsushi Ikeuchi, “Recognition of Vehicles by Local Feature Recognition Algorithm”, IEICE Technical Report, Vol.101, No.302, pp.9-14, 2001.

  By detecting a common part between images from multiple images obtained from an image input device or a storage device that already stores images, the multiple images are connected and automatically generated as a single image. The construction of a display device is an issue. Note that one or a plurality of images generated from the plurality of images is defined as a composite image, and this designation is used below.

  In order to solve the above-described problems, image input is performed by controlling an image input device or an image already stored, and an edge image and a hue image are generated from the input image. A point with low edge or hue value similarity to a neighborhood at a certain point is extracted as a feature point, and a set of images having a common feature point is calculated by collating the feature points between multiple images. Calculate only the feature points for which the relative positional relationship is maintained between images having feature points, and calculate the average value of the relative positions of the feature point groups for which the relative positional relationship is maintained between images having the same feature points. The calculated relative position is used as a relative position between the images, and the combined images are generated and displayed by superimposing the images.

  According to the present invention, common features in a plurality of images can be extracted and collated, and automatically generated and displayed as a single composite image. In particular, a large number of edge and hue feature points are extracted from each of a plurality of images and collated between the images to detect a common part between the images, and between the feature points by comparing the relative positions of the feature points. This makes it possible to automatically generate and display a synthesized image with higher accuracy.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows an outline of the processing flow of the present invention. When the processing is started (101), image input is performed by controlling an image input device such as a camera (102), and an edge image and a hue image are generated from the input images. A point having a low edge similarity to a neighborhood or a point having a low hue value similarity at a certain point is extracted as a feature point (103), and a common feature point is obtained by collating the feature points between a plurality of images. A set of images possessed is calculated, and only the feature points for which the relative positional relationship is maintained in the calculated set of images are calculated (104), and the relative positional relationship is maintained between images having common feature points. The average value of the relative positions of the feature point groups being calculated is calculated, a composite image is generated using the relative position as the relative position between the images (105), the generated composite image is displayed (106), and the program is To completion (107).

  Based on the outline of the above processing flow, FIG. 2 shows a more detailed processing flow of FIG. 2 in FIG. 1, 102 in FIG. 1, 203 to 206 in FIG. 2, 103 in FIG. 1 in 207 to 209 and 212 in FIG. 2, 104 in FIG. 1 to 210 in FIG. 105 in FIG. 1 corresponds to 211 in FIG.

  FIG. 3 shows details of the processing 208 in FIG.

When the program is started (201), the image input device such as a camera is controlled as an image input (102) process, and a plurality of images are captured (202).
Next, image feature calculation (103) is performed. FIG. 4 shows examples 204 and 206 in the processing (203 to 206) of the image feature calculation (103). For each of the plurality of images input by the image input (102), an edge image binarized by brightness is first generated (203). A window (402) is provided in the generated edge image (401), and the window is scanned in the image (403). At this time, for each pixel of the window (404) at a certain position, a binary value indicating either white or black is read as a pixel value, and each pixel value of the window (405) provided around this window 1 is calculated when the values match, and 0 is set when the values do not match, and the pixel values in the window are evaluated to calculate the sum and obtain the Hamming distance. When the Hamming distance is equal to or smaller than the threshold, the window is used as a feature point, and the coordinates in the image and each pixel value in the window centered on the coordinates are recorded (204). Here, in setting the threshold, a ratio that matches when each pixel value between the windows is the same is set as the threshold. In FIG. 4, a window provided around the window (402) at a certain position is simply shown as one window (405), but in reality, it is provided around the entire window (402).

  Further, the image feature calculation (103) generates a hue image from a plurality of captured images (205). Similar to the processing in the edge image, a window (402) is provided in the generated hue image (401), and the window is scanned in the image (403). At this time, by taking a range from 0 degree to 360 degrees for each pixel of the window (404) at a certain position, a value indicating color is read and set as a pixel value. Similarly, each pixel value of the window (405) provided around the window (404) is read. Next, the difference between the pixel values of the pixels having the same coordinates between the window (404) and the window (405) when the upper left corner of each window is the origin is obtained, and the sum of the differences between the pixel values of the pixels is obtained. Specifically, the sum of the differences between the pixel values of each pixel is calculated by the following equations, Equations 1 and 2.

Here, n is the number of pixels in the vertical or horizontal direction in the window, and H1 and H2 are the pixel values of a certain pixel in each window. Further, it is assumed that the equations 1 and 2 are evaluated only when i = j, and when the absolute value dij when H2 is subtracted from H1 is 180 or more, d = 360−dij is obtained, and dij is changed to dij Shall be substituted.
Next, a point where the sum of the differences is equal to or greater than a threshold value is used as a feature point, and the coordinates in the image and each pixel value in the window centered on the coordinates are recorded (206). This threshold value is defined as the ratio of the sum of the differences of the pixel values for each pixel between the windows being compared to the sum of the differences when the difference of the pixel values between the windows is the largest. If the threshold value R is defined here, when comparing the similarity of windows in actuality, the current difference is compared with the sum of the differences when the difference between the pixel values between the windows is the largest by Equation 3. The ratio of the sum of the differences in pixel values for each pixel between the current windows is calculated.

When r calculated | required by this is smaller than the threshold value R, it determines with not being similar.

  Next, image feature matching (104) is performed. Image feature matching (104) is a process of obtaining a combination of images having edge feature points or hue feature points corresponding to a plurality of images (207) and among the image combinations obtained by the processing of 207. The processing consists of the processing (208) for selecting only the feature point group having the same relative position among the feature point groups that are common to each other. First, FIG. 5 shows an example 207 which is a process of image feature matching (104). The feature points extracted in the other images (506 to 508) captured by the image input (102) using the feature point groups (502 to 505) of the edges obtained for the certain image (501) as reference images. Is checked. For example, for the image 506, the similarity is compared with the feature point group (509 to 512) already extracted by the image feature extraction (103).

  At this time, when the pixel values in the window match each other, the evaluation is performed with respect to each pixel value in the window by setting 1 when the pixel values do not match and 0 when the pixel values do not match, and the sum is calculated to obtain the Hamming distance. Find a point above the threshold and record its coordinates. As a result, the position of the feature point of the edge common between the images is calculated and recorded. Similarly to the matching of the feature points in the edge image, scanning is performed in the other images (506 to 508) using the feature points (502 to 505) of the hue obtained for the certain image (501) as a reference image. While collating. At this time, the difference between each pixel value of the image in the window and each pixel value of the image in the window provided at the reference destination is calculated, and the sum of the differences between the pixel values is obtained. This is equivalent to the processing of Formula 1 to Formula 2 in the description of the image feature extraction (103). Find a point where Equation 3 is greater than or equal to the threshold and record its coordinates. As a result, the position of the feature point of the hue common to the images is calculated and recorded. As described above, all combinations of images having edge feature points or hue feature points are obtained. FIG. 6 shows an example 208 of image feature matching (104) processing. In 208, an image (601, 607) having a plurality of feature points in common with a plurality of feature points in a certain image is selected, and a combination of feature points that best matches the relative positions of the feature points is obtained. First, when superimposing images 601 and 607 having common feature points, an appropriate feature point (604) is selected as a reference (301).

  Next, the images 601 and 607 are arranged so as to overlap with the feature point (604) as a reference (302). At this time, the relative distance between the feature points that coincide between the images is calculated (303). Here, the relative distance is a distance from the reference point to the feature point. Among feature points that match between images, the number of feature points whose relative distances corresponding to each other are equal to or less than a threshold is counted (304). An example of the relationship between the arbitrarily selected reference point, relative distance, and threshold is shown in FIG. Here, the reference point may be an arbitrary feature point, or an arbitrary place other than the feature point may be selected as the reference point. Similarly, when the images of 601 and 607 are overlapped using the feature point (602) as a reference, when the relative positions between the feature points that match between images are equal to or less than a threshold value, and all feature points are used as a reference, The number of features whose distance between feature points common to the images is equal to or less than a threshold value is counted. When counting of feature points whose relative distance is equal to or less than the threshold when all feature points are used as a reference (305), the feature point used as a reference when the count value is the largest is selected (306).

  When the images are arranged in such a way that the feature points have the maximum count value, the number of feature points having the relative positional relationship is the largest among the feature point groups common to the images. Become. Based on this, when images are arranged based on the feature points selected in 306, the feature points that match between the images are excluded when the distance between the feature points is greater than or equal to a threshold, and only feature points that are less than or equal to the threshold are selected. The image in the feature point window and its coordinates are recorded as a file (307). For example, in FIG. 6, when the images (601, 607) are arranged with the feature point (602) as a reference, among the sets of matching features between the images, a set of 602 and 608, a set of 603 and 609, and a set of 605 and 610 And the number of features whose matching feature points are equal to or less than the threshold is larger than when the images are overlapped with reference to 604 as a reference. Therefore, the feature point 602 is selected and recorded as a reference (307). At this time, the feature point pairs (613 and 616) have the same feature, but are excluded from the recording because the distance between them is equal to or greater than the threshold (308).

  Next, it is evaluated whether the number of feature points recorded in the process (208) of the image feature matching (104) and the distance between the feature points on the image are more than a threshold value (209). For example, it is determined that a feature point necessary for generating the composite image cannot be obtained, and an error is displayed to notify the operator that the composite image cannot be generated (212). If the number of feature points recorded in the process (208) of the image feature matching (104) and the distance between the feature points on the image are equal to or greater than the threshold value, each of the images that match between the images as the composite image generation (105) The relative positions of the feature points are calculated, the average value of the relative positions of all the feature points that match each other is obtained, and this value is set as the relative position of the images having the common feature points. The relative positions of the images having the common feature points are sequentially obtained, and a single image is generated so that the images having the common feature points overlap each other by the relative position, thereby obtaining a composite image (210 ). Finally, the generated composite image is displayed as the image display (105) (211), and the program ends (213).

  Next, the configuration of the apparatus according to the embodiment of the present invention will be described with reference to FIG. This apparatus includes a main processor (701) for processing data based on a program, a main memory (707) such as a memory for holding the program and data, an auxiliary memory (706) such as a hard disk, a display (702) and a graphic board. (703), a camera (704) for performing image input (102) and a video capture board (705). This configuration can be realized in terms of hardware by a processor, memory, or other LSI, and in terms of software, it is realized by a program loaded in the memory. It is described. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The program and data that operate in the main memory (707) will be described. An operating system (709) is basic software for managing execution of programs and the like. As a program for generating a composite image from a plurality of images, a main program (710) for controlling the overall processing flow, an image input program (711) corresponding to the processing of the camera image input (102) in FIG. Image feature calculation program (712) corresponding to the processing of calculation (103), image feature verification program (713) corresponding to the processing of image feature verification (104), and composite image generation corresponding to the processing of composite image generation (105) There is an image display program (715) corresponding to the processing of the program (714) and image display (106). In addition, there is an image buffer (716) that holds processing results in image processing. The auxiliary memory (706) stores various programs (709 to 715) to be read into the main memory as programs (708). These programs are read into a memory and processed by being executed by a processor.

  The processing of each program will be described. The main program (710) controls the processing flow of the entire program from image input to composite image generation / display shown in FIGS. The image input program (711) controls the video capture board (705) and the camera (704) as the camera image input (102), whereby the image captured by the camera is stored in the image buffer (716) of the main memory (707). Capture (202 in FIG. 2). The image buffer (716) is also used as a work area for various programs described below.

  Next, the image feature calculation program (712) generates an edge image from the image captured by the image input (102) (203 in FIG. 2), and extracts a point having a low degree of edge similarity with the vicinity as a feature point. (204 in FIG. 2). Also, a hue image is generated (205 in FIG. 2), and a point having a low degree of hue similarity with the vicinity is extracted as a feature point (206 in FIG. 2). The image feature matching program (713) calculates a combination of feature points common to each edge and hue between images (207 in FIG. 2).

  More specifically, the difference between the relative position of each feature point in the reference image is equal to or less than the threshold value based on the relative position of each feature point in a certain image among the edge and hue feature points common to the images. Are selected and recorded (208 in FIG. 2). The number of matched feature points and whether or not the distance between the feature points on the image is equal to or greater than the threshold value are evaluated. When the number of matched feature points and the distance between the feature points on the image are equal to or larger than the threshold value, an average value of relative positions between the images is obtained from the matched feature point group by the composite image generation program (714), and the relative position A plurality of images are synthesized as one synthesized image according to (211 in FIG. 2). Next, the image display program (715) displays the composite image generated by the composite image generation program (714) (211 in FIG. 2).

  It is also possible to supply a recording medium in which a software program for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and for the processor of the system or apparatus to read and execute the program stored in the recording medium. Needless to say, the object of the present application is achieved.

  In the above description, an example of an apparatus using a camera as an image input device has been shown as an embodiment of the invention. As an image input device, an electronic notebook such as a camera-equipped mobile phone or a camera-equipped PDA (Personal Digital Assistant) is used. In addition to the video camera, the scanner, etc., the same result as that of the device example shown can be obtained even when the image is input from a storage device on the network storing the still camera image.

  In addition, by using the present invention, as an example of industrial application, one composite image is automatically generated and displayed from a plurality of images taken by a camera mounted on a moving object such as an automobile or an aircraft or a satellite. It is possible to automatically generate and display one composite image from a plurality of images by a camera array in which a plurality of cameras are arranged, and it can be expected to save work and shorten the work time.

It is a block diagram of the whole function of this invention. It is a figure which shows the flow of the whole process of this invention. It is a figure which shows the flow of the process 208 in connection with an image feature collation among FIG. 2 which shows the flow of the whole process. FIG. 3 is a diagram showing processing 204 and 206 related to image feature extraction in FIG. 2 showing the overall processing flow. It is a figure which shows the process 207 regarding image feature collation among FIG. 2 which shows the flow of the whole process. It is a figure which shows the process 208 in connection with an image feature collation among FIG. 2 which shows the flow of the whole process. It is a block diagram of the whole apparatus in one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Program start process, 102 ... Image input process, 103 ... Image feature extraction process, 104 ... Image feature collation process, 105 ... Image display process, 106 ... Program end process

Claims (2)

An image generation device that generates a composite image from a plurality of images,
An image input means for inputting an image;
Image feature calculating means for calculating features in each image of the plurality of images input from the image input means;
Image feature collating means for comparing features of each image obtained by the image feature calculating means and collating images having similar features;
Image generating means for generating a composite image by superimposing at least two of the plurality of images based on a matching result in the image feature matching means;
Have a means for displaying the composite image,
The image feature calculation means generates an edge image and a hue image for each of the plurality of images, extracts feature points in each image based on the edge image and the hue image,
The image feature matching unit is configured to determine a pixel value of a feature point in one image and a pixel value of another image based on the pixel value of the edge image and the hue image of the feature point in each of the plurality of images. When the difference between the pixel values of the feature points is equal to or less than a predetermined threshold, it is determined that the one feature point is similar to the other feature points, and the feature points are among at least two images having the similar feature points. Determining whether one feature point of a plurality of feature points in each image is a reference point, and whether another plurality of feature points in the image are within a predetermined distance from the reference point, An image generating apparatus , wherein when any of a plurality of other feature points is outside the predetermined distance, the feature points outside the predetermined distance are deleted from the extracted feature points . The image generation apparatus according to claim 1,
The image generating device is characterized in that the image generating means combines two or more images having the similar feature points by superimposing positions of the feature points .

Images