JP2007304733A - Image detector and image detection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image detection method and an image detector, capable of satisfactorily detecting a marker from images taken in various conditions. <P>SOLUTION: A detecting image generation part 11 uses a pattern of a maker as a registered image and generates detecting images based on the registered image. A detecting image inspection part 12 verifies the detecting images generated by the generation part 11, and selects a verified detecting image from the detecting images based on the verification result. A marker detection part 15 detects the marker from the photographic image by comparing the verified detecting image selected by the inspection part 12 with the photographic image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to Augmented Reality technology that makes it appear as if there is a virtual object in real space, and more particularly to an image detection apparatus and an image detection method for realizing Augmented Reality using images taken under various conditions.

  Augmented Reality is a technology that reinforces real-world information with virtual world data (CG) built inside the computer, and is expected to open up a completely different application field from Virtual Reality that handles only the virtual world by CG. Yes. In recent years, a lot of research on Augmented Reality has been conducted.

  For example, Non-Patent Document 1 discloses a technique for realizing Augmented Reality. In Non-Patent Document 1, a black frame marker with a picture written inside is used. When the camera captures a space in front of the user's eyes and detects a marker in the captured image (captured image), an image is generated by combining CG at the marker position of the captured image. At that time, the three-dimensional relative position between the camera and the black frame is calculated based on the size and shape of the black frame in the photographed image, and the picture written inside the black frame is compared with the photographed image based on the relative position. CG is determined. According to this, it is possible to synthesize a plurality of different CGs by changing the pattern of the marker.

  In this method, a marker is detected by extracting the inside of a black frame in a captured image and comparing the image with a previously registered image. Since this method compares images, it is effective when the illumination conditions are relatively stable such as indoors and the camera parameters are constant.

Patent Document 1 proposes a technique for realizing Augmented Reality by capturing a marker using a camera function of a mobile phone or the like and detecting the marker from the captured image.
Billinghurst & Kato, “Collaborative Mixed Reality.”, ISMAL '99 JP 2004-341642 A

  However, there are various environments for photographing with a mobile phone, and generally lighting conditions are not stable. For this reason, there is a problem that the color of the pattern of the marker photographed depending on the illumination condition changes, and the marker cannot be detected. In addition, there are various types of mobile phones, and camera parameters are different from each other, which makes marker detection more difficult.

  As described above, the conventional application using Augmented Reality has a problem that accurate marker detection cannot be performed because the color of the marker in the camera image changes depending on the illumination condition. In particular, when a mobile phone or the like is used, there is a problem that the marker detection rate is further deteriorated due to various shooting environments.

  An object of the present invention is to provide an image detection apparatus and an image detection method that can detect a marker satisfactorily from images taken under various conditions.

In order to achieve the above object, the image detection apparatus of the present invention includes:
An image detection device that detects the marker from a captured image in which the marker is captured,
A detection image generation unit that generates a detection image based on the registered image using the pattern of the marker as a registered image;
A detection image inspection unit that verifies the detection image generated by the detection image generation unit and selects a post-verification detection image from the detection image based on the verification result;
A marker detection unit that detects the marker from the photographed image by comparing the post-verification detection image selected by the detection image inspection unit and the photographed image.

  According to the present invention, since the marker is detected from the photographed image by comparing the detection image generated based on the registered image of the marker with the photographed image, the marker is detected from the photographed image that is different from the marker depending on the photographing condition. It can be detected well.

  In addition, the detection image generation unit may use the same image as the registered image and an image obtained by performing image processing on the registered image as the detection image.

  Further, the detection image generation unit may generate an image in which at least one of color, brightness, contrast, resolution, and focus of the registered image is changed by the image processing.

In addition, the detected image inspection unit,
A plurality of rotation images obtained by rotating the detection image are generated, the detection image and the rotation image are compared, and a similar rotation image exists from the detection image generated by the detection image generation unit. The post-verification detection image may be selected by excluding the detection image.

  According to this, the detection image is compared with the rotation image, and only the detection image (post-verification detection image) remaining by excluding the detection image having a similar rotation image is used for marker detection. It is possible to prevent erroneous recognition of a marker by using an image obtained by changing an image for detection.

In addition, the detected image inspection unit,
For each of the detection images, the number of rotated images having a degree of similarity with the detection image greater than a predetermined threshold is obtained as the number of similar images, and the detection image having the largest number of similar images and the detection image are rotated. The process of excluding the rotated image may be repeated until the number of similar images of all the detection images becomes zero, and the remaining detection images may be used as the post-verification detection images.

In addition, the detected image inspection unit,
Among the detection images, the same image as the registered image may not be excluded even if there is a similar rotated image.

  Since it can be assumed that the marker of the captured image matches the registered image with a higher probability than the other detected image, according to this, the same detection image as the registered image is given priority. By leaving it, the marker detection rate can be increased.

  In addition, when the marker cannot be detected from the captured image, the marker detection unit may cut out a pattern portion of the marker from the captured image and add it to the post-verification detection image.

  According to this, since the post-verification detection image that was not generated by the image processing can be added, the marker can be detected from the captured image captured under the same conditions after the next time. Thus, the marker detection rate is improved.

There are several markers with different patterns,
The detection image generation unit sets each of the plurality of markers as a registered image, generates a plurality of detection images for each registered image,
The detection image inspection unit generates a plurality of rotation images obtained by rotating each of the detection images, includes a rotation image of a marker with a different pattern, compares the detection image with the rotation image, and Selecting the post-verification detection image by excluding a detection image having a similar rotated image from the plurality of detection images generated by the detection image generation unit;
The marker detection unit may detect the marker from the photographed image by comparing the post-verification detection image for the plurality of markers with the photographed image.

  According to the present invention, since the marker is detected from the photographed image by comparing the detection image generated based on the registered image of the marker with the photographed image, the marker is detected from the photographed image that is different from the marker depending on the photographing condition. It can be detected well.

  An embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the image detection apparatus according to this embodiment. Referring to FIG. 1, the image detection apparatus 10 includes a detection image generation unit 11, a detection image inspection unit 12, and a marker detection unit 15. The detection image inspection unit 12 includes a rotated image generation unit 13 and a similar inspection unit 14.

  The detection image generation unit 11 receives an image of a marker prepared in advance, performs image processing on the image, and generates a plurality of detection images P1 to Pn that can correspond to various illumination conditions and camera parameters.

  The detection image inspection unit 12 excludes detection images that cause misrecognition from the detection images P1 to Pn generated by the detection image generation unit 11, and detects post-verification detection images PK1 to PKm (m ≦ n) is selected.

  The marker detection unit 15 detects a marker from the captured image using the post-verification detection images PK1 to PKm selected by the detection image inspection unit 12. Here, the marker with the black frame described above is used as an example. A pattern to be synthesized at the marker position is drawn in the black frame. The marker is a black frame as an example, but the present invention is not limited to the black frame, and may be a frame of another color.

  FIG. 2 is a diagram for explaining the operation of the image detection apparatus of the present embodiment. Referring to FIG. 2, first, the detection image generation unit 11 receives a registered image T, which is a pattern of a marker to be detected, and changes color, brightness, contrast, resolution, and focus (blur condition) by image processing. A plurality of detection images P1 to Pn are generated. The detection images P1 to Pn include the same image as the registered image T. The detection images P1 to Pn are given to the rotation image generation unit 13 and the similarity inspection unit 14 of the detection image inspection unit 12. The detection images P1 to Pn include the same image as the registered image T that is not subjected to image processing.

  The rotation image generation unit 13 of the detection image inspection unit 12 receives the detection images P1 to Pn, rotates the detection images P1 to Pn by 90 degrees to the right, and rotates 180 degrees to the right. The rotated images PB1 to PBn that have been rotated and the rotated images PC1 to PCn that have been rotated 270 degrees to the right are generated.

  The similarity inspection unit 14 compares the detection images P1 to Pn from the detection image generation unit 11 with the rotation images PA1 to PAn, PB1 to PBn, and PC1 to PCn from the rotation image generation unit 13, and Exclude detection images with close rotation images. For example, if the pattern of a certain low-contrast detection image is close to the pattern of a certain low-contrast rotated image, the low-contrast detection image is excluded.

  FIG. 3 is a flowchart showing an operation algorithm of the similar inspection unit. Referring to FIG. 3, the similarity inspection unit 14 sets i = 1 (step 101), receives the detection images P1 to Pn and the rotated images PA1 to PAn, PB1 to PBn, and PC1 to PCn, and inputs the detection image. Pi and each rotated image are compared to determine the number of similar patterns (number of similar images) Ki (step 102). Whether or not the images are similar may be determined based on whether or not the degree of similarity of the pattern is greater than a predetermined threshold. As a method for obtaining the similarity, for example, there is a method using an image comparison algorithm such as DP (Dynamic Programming) matching. What is necessary is just to judge that the pattern whose similarity exceeded a predetermined threshold is similar. As an image comparison algorithm, in addition to DP matching, pattern matching, a feature point extraction collation method, or the like can be used.

  DP matching is an algorithm that calculates the degree of similarity between data based on element-by-element matching while associating elements constituting two data to be compared. Thereby, the similarity between data with different pattern lengths (number of elements) can be obtained. Specifically, when the correspondence between elements becomes optimal, the square root of the sum of squares of the difference between corresponding elements becomes the distance (dissimilarity) between the two data. It can be said that the closer the distance between the two data is, the higher the similarity is, and therefore, the reciprocal of this distance may be used as the similarity. Note that the Euclidean square distance may be used instead of the square root of the square sum described above.

  If DP matching is used to calculate the similarity between the captured image and the post-verification detection image, an appropriate similarity calculation and marker can be used when the number of pixels differs between the marker portion of the captured image and the post-verification detection image. Can be detected. Depending on the maximum number of pixels of the camera and the distance between the camera and the marker, the number of pixels in the marker portion in the captured image data changes. However, if DP matching is used, the similarity can be calculated even if the number of pixels is not constant. .

  Subsequently, the similarity inspection unit 14 adds 1 to i (step 103), and determines whether i is larger than n (step 104). If i is not greater than n, the similarity checking unit 14 returns to step 102. On the other hand, if i is larger than n, the similarity checking unit 14 determines whether or not all the similar image counts Ki (1 ≦ i ≦ n) obtained so far are 0 (step 105).

  If all the similar image numbers Ki are 0, the similar inspection unit 14 ends the process. On the other hand, if any one of the number of similar images Ki is not 0, the similarity inspection unit 14 obtains i that maximizes the number of similar images Ki (step 106). Subsequently, the similarity inspection unit 14 excludes the i detection image Pi obtained in step 106 and the rotated images PAi, PBi, and PCi rotated (step 107), returns to step 101, and repeats the processing.

  In these processes, the process of excluding similar images in steps 101 to 107 is repeated until the determination in step 105 is “YES”, that is, until there is no image similar in design. Then, the detection image without the similar rotated image remaining after the processing is provided to the marker detection unit 15 as post-verification detection images PK1 to PKm (m ≦ n).

  Returning to FIG. 2, the marker detection unit 15 receives the captured image I as an input, and detects markers from the captured image I using the post-verification detection images PK <b> 1 to PKm given from the similarity inspection unit 14 as templates. At this time, an existing method may be used for marker detection. For example, a marker can be detected using ARTToolKit disclosed in Non-Patent Document 1.

  As described above, according to the present embodiment, in addition to the registered image T of the marker, an image in which the color, brightness, or the like is changed is used as a detection image, and by comparing the captured image with a plurality of detection images. Since the marker is detected from the photographed image, the marker can be favorably detected from the photographed image that is different from the marker depending on the photographing condition.

  Further, according to the present embodiment, the detection image and the rotation image are compared, and only the detection image (post-verification detection image) remaining by excluding the detection image having a similar rotation image is used for marker detection. Therefore, it is possible to prevent erroneous recognition of the marker due to the use of an image whose color and brightness are changed for detection.

  FIG. 3 shows an example in which the detection image having the maximum number of similar images is excluded regardless of whether the detection image Pi is the same as the registered image or obtained by image processing of the registered image. However, the present invention is not limited to this. As another example, the same detection image as the registered image may be excluded even if there is a similar rotated image. In general, it can be assumed that the marker of a captured image is more likely to match the registered image than the possibility to match the other detection images, so the same detection image as the registered image is given priority. By leaving it, the marker detection rate can be increased.

  FIG. 4 is a flowchart showing the operation algorithm of the similar inspection unit when the same detection image as the registered image is not excluded. Referring to the example of FIG. 4, step 201 is performed instead of step 105 of FIG. 3, and step 202 is performed instead of step 106. In step 201, the similar inspection unit 14 determines whether or not the number of similar images Ki of the detection image Pi that is not the same as the registered image is all zero. As a result, the processing is repeated until there is no similar image to any of the rotated images among the detection images Pi other than the same image as the registered image T. In step 202, the similarity inspection unit 14 obtains i that maximizes the number of similar images Ki among i of the detection images Pi that are not the same as the registered image T. As a result, in step 107, the detection image having the maximum number of similar images is excluded from the detection images excluding the same registered image.

  Moreover, in embodiment mentioned above, it does not specifically limit about the case where a marker cannot be detected in the marker detection part 15. FIG. However, the marker detection unit 15 additionally registers the pattern portion of the marker of the captured image I in which the marker cannot be detected in the post-verification detection image, and the detection image is also used for marker detection from the next time. May be.

  FIG. 5 is a flowchart showing the operation of the marker detection unit when a post-verification detection image is added if a marker cannot be detected. Referring to FIG. 5, the marker detection unit 15 executes a process of detecting a marker from a captured image by comparing the post-verification detection image and the captured image as described above (step 301). Subsequently, the marker detection unit 15 determines whether or not the marker is correctly detected (step 302). If the marker is correctly detected, the marker detection unit 15 outputs the detection result (step 303) and ends the process. On the other hand, if the marker is not correctly detected, the marker detection unit 15 cuts out the pattern portion of the marker from the captured image I, and additionally registers it in the post-verification detection image (step 304). According to this, since the post-verification detection image that was not generated by the image processing can be added, the marker can be detected from the captured image captured under the same conditions after the next time. Thus, the marker detection rate is improved.

  In the above-described embodiment, an example in which a marker is detected from a captured image based on one registered image has been described. However, the present invention is not limited to this. As another example, the image detection apparatus 10 may detect a plurality of markers having different patterns.

  As another embodiment, an image detection apparatus that detects a plurality of markers having different patterns will be described. The image detection apparatus of this embodiment has the same configuration as that shown in FIG. However, in the present embodiment, since there are a plurality of markers to be detected, the detection image generation unit 11 sets the patterns of the plurality of markers to be detected as registered images T1 to Tk, and colors for each of the registered images T1 to Tk. A plurality of detection images with different brightness are generated. The detection image inspection unit 12 determines a post-verification detection image by excluding detection images similar to any of the rotation images, including rotation images of markers with different patterns, from the detection images of the respective markers. To do. However, at this time, the same images as the registered images T1 to Tk may not be excluded even if there are similar rotated images. The marker detection unit 15 detects a marker from the captured image I using the post-verification detection image. According to this, even when detecting a plurality of markers, the marker is detected from the photographed image by comparing the detection image generated based on the registered image of the marker with the photographed image, so that the image differs from the marker depending on the photographing conditions. The marker can be detected well from the captured image.

  In the above-described embodiment, the detection image generation unit 11 receives the registered image T as an input, and generates a plurality of detection images P1 to Pn in which color, brightness, contrast, resolution, and focus are changed by image processing. However, the present invention is not limited to this. As another example of image processing, the detection image generation unit 11 may use an image obtained by enlarging or reducing the registered image T as a detection image.

  In the above-described embodiment, the detection image inspection unit 13 generates a rotation image from the detection image, and compares the detection image with the rotation image, thereby excluding the detection image that causes misrecognition. Although an example is shown, the present invention is not limited to this. As another example, a deformed image may be generated instead of or in addition to the rotated image, and the detection image and the deformed image may be compared. For example, a deformed image (such as a trapezoidal image) corresponding to the case where the marker is photographed from an oblique direction may be generated and compared with the detection image.

  Further, although the detection image inspection unit 12 of the above-described embodiment has shown an example in which the processing is repeated while excluding the detection images having the maximum number of similar images one by one, the present invention is limited to this. is not. As another example, at the beginning of the verification process, all detection images having a number of similar images larger than a predetermined threshold are excluded, and thereafter, a process of excluding one detection image having the largest number of similar images one by one is started. You may decide. For example, in the first determination, verification images having three or more similar images may be excluded at a time. According to this, the time required for the verification process for selecting the post-verification detection image can be shortened.

It is a block diagram which shows the structure of the image detection apparatus by this embodiment. It is a figure for demonstrating operation | movement of the image detection apparatus of this embodiment. It is a flowchart which shows the operation | movement algorithm of a similar test | inspection part. It is a flowchart which shows the operation | movement algorithm of the similar test | inspection part when not excluding the image for a detection same as a registration image. It is a flowchart which shows operation | movement of the marker detection part in the case of adding the image for a detection after verification, if a marker cannot be detected.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image detection apparatus 11 Image generator for detection 12 Image inspection part for detection 13 Rotation image generation part 14 Similar inspection part 15 Marker detection part 101-107, 201-202, 301-304 step

Claims (14)

An image detection device that detects the marker from a captured image in which the marker is captured,
A detection image generation unit that generates a detection image based on the registered image using the pattern of the marker as a registered image;
A detection image inspection unit that verifies the detection image generated by the detection image generation unit and selects a post-verification detection image from the detection image based on the verification result;
An image detection apparatus comprising: a marker detection unit that detects the marker from the captured image by comparing the post-verification detection image selected by the detection image inspection unit with the captured image.   The image detection apparatus according to claim 1, wherein the detection image generation unit uses the same image as the registered image and an image obtained by performing image processing on the registered image as the detection image.   The image detection apparatus according to claim 2, wherein the detection image generation unit generates an image in which at least one of color, brightness, contrast, resolution, and focus of the registered image is changed by the image processing. The detected image inspection unit includes:
A plurality of rotation images obtained by rotating the detection image are generated, the detection image and the rotation image are compared, and a similar rotation image exists from the detection image generated by the detection image generation unit. The image detection apparatus according to claim 1, wherein the post-verification detection image is selected by excluding the detection image. The detected image inspection unit includes:
For each of the detection images, the number of rotated images having a degree of similarity with the detection image greater than a predetermined threshold is obtained as the number of similar images, and the detection image having the largest number of similar images and the detection image are rotated. The image detection apparatus according to claim 4, wherein the process of excluding the rotated image is repeated until the number of similar images of all detection images becomes zero, and the remaining detection image is used as the post-verification detection image. The detected image inspection unit includes:
The image detection apparatus according to claim 4, wherein, among the detection images, the same image as the registered image is not excluded even if there is a similar rotated image.   The said marker detection part cuts out the pattern part of a marker from this picked-up image, and adds it to the said post-verification detection image, when the said marker cannot be detected from the said picked-up image. The image detection apparatus described in 1. There are multiple markers with different patterns,
The detection image generation unit sets each of the plurality of markers as a registered image, generates a plurality of detection images for each registered image,
The detection image inspection unit generates a plurality of rotation images obtained by rotating each of the detection images, includes a rotation image of a marker with a different pattern, compares the detection image with the rotation image, and Selecting the post-verification detection image by excluding a detection image having a similar rotated image from the plurality of detection images generated by the detection image generation unit;
8. The marker detection unit according to claim 1, wherein the marker detection unit detects the marker from the captured image by comparing the post-verification detection image for the plurality of markers with the captured image. 9. Image detection device. An image detection method for detecting the marker from a captured image in which the marker is captured,
Using the pattern of the marker as a registered image, generating a detection image based on the registered image,
Validate the generated detection image, and select a post-verification detection image from the detection image based on the verification result;
An image detection method for detecting the marker from the captured image by comparing the selected post-verification detection image and the captured image.   A plurality of rotation images obtained by rotating the detection image are generated, the detection image and the rotation image are compared, and a similar rotation image exists from the detection image generated by the detection image generation unit. The image detection method according to claim 9, wherein the post-verification detection image is selected by excluding the detection image.   For each of the detection images, the number of rotated images having a degree of similarity with the detection image greater than a predetermined threshold is obtained as the number of similar images, and the detection image having the largest number of similar images and the detection image are rotated. The image detection method according to claim 10, wherein the process of excluding the rotated image is repeated until the number of similar images of all detection images becomes zero, and the remaining detection images are used as the post-verification detection images.   The image detection method according to claim 10 or 11, wherein an image that is the same as the registered image among the detection images is not excluded even if there is a similar rotated image.   The image detection method according to claim 9, wherein when the marker cannot be detected from the captured image, a pattern portion of the marker is cut out from the captured image and added to the post-verification detection image. . There are multiple markers with different patterns,
Each of the plurality of markers as a registered image to generate a plurality of detection images for each registered image;
A plurality of rotation images obtained by rotating each of the detection images are generated, the rotation images of the markers having different patterns are included, the detection image and the rotation image are compared, and the plurality of detection images are similar. By selecting the post-verification detection image by excluding the detection image with the rotated image to be
The image detection method according to claim 9, wherein the marker is detected from the captured image by comparing the post-verification detection image for the plurality of markers with the captured image.

Images