KR20110104243A - Recognition apparatus and method of occluded marker in augmented reality - Google Patents

Abstract

According to the present invention, in augmented reality (AR) based on a real environment, when a marker to be recognized is obscured by another object or the shape is distorted, the marker is extracted by distinguishing the feature points of the marker, and the extracted feature points are already stored. If it matches a feature point or more than a certain threshold value, it is recognized as a valid marker and applied to measuring the position and posture of the marker, thereby enabling more accurate marker recognition and tracking, thereby providing augmented reality.

Description

RECOGNITION APPARATUS AND METHOD OF OCCLUDED MARKER IN AUGMENTED REALITY

The present invention relates to a marker recognition technology in augmented reality, and more particularly, when a marker to be recognized is obscured or distorted by another object, the marker is extracted by distinguishing the feature points of the marker, and the extracted feature points are already stored. The present invention relates to an apparatus and method for recognizing a shielded marker in augmented reality, which is recognized as a valid marker if it matches a feature point equal to or greater than a predetermined threshold.

Virtual Reality (VR) is a field that mainly interacts with 3D virtual spaces created by computer systems, and virtual spaces are constructed based on the real world. The user has an immersive feeling by feeling the virtual space through the five senses (visual, auditory, olfactory, taste, and tactile) of the human body.

Augmented Reality (AR) is a technology derived from one field of virtual reality (VR), which means combining the real world and the virtual experience. In other words, it is a computer graphics technique that synthesizes virtual objects in the real environment and makes them look like objects existing in the original environment.

Virtual reality creates a virtual environment through a computer and interacts with the user, but augmented reality has a feature that the user can improve the reality by interacting with a virtual object based on the real world. Therefore, in augmented reality, the user recognizes the real environment in which the user is located and also recognizes the virtual information expressed on the real image.

As described above, since augmented reality displays a combination of real images and virtual graphics, virtual objects must be accurately positioned on a screen in order to obtain a more realistic and accurate image. To implement this, three-dimensional coordinates for the virtual object are required, which must be coordinates with respect to the camera.

However, the problem is to obtain the three-dimensional coordinates of the camera for a certain point or object in the real world in the image of the camera, which requires two or more cameras, it is not easy to determine the three-dimensional position.

Accordingly, as a countermeasure, in augmented reality, a marker-based recognition for estimating a position and posture at which a virtual object is placed by arranging a marker to locate the virtual object in a real world space and extracting relative coordinates based on the marker I'm using technology.

For example, the user randomly creates a marker for a cat, and matches a cat marker with a 3D virtual image in which a real cat's shape is formed into a 3D graphic image. When the marker is recognized by an image recognition means such as a camera connected to a computer, a three-dimensional virtual image of a cat appears on the computer monitor.

However, in the existing marker-based recognition technology, the marker to be recognized is covered by a moving hand or other object for interacting with the user (hereinafter, a shielded marker), or the shape is distorted (hereinafter, distorted marker). There is a problem that is not recognized as a useful marker.

As a solution to this problem, if an occluded marker exists, it is removed to recognize the marker. In this case, an operation error for marker recognition can be solved, but since some components of the virtual object matching the removed markers are deleted or crushed into other components, there is a problem of degrading reality. In particular, when providing a virtual image by recognizing a plurality of markers, the above problem can be made larger. For example, when recognizing a plurality of markers and providing a multifunction device, which is a virtual image, some markers may be mapped to the control button of the multifunction device and some may be mapped to the output tray of the multifunction device. The control button of the multifunction device may be displayed in a deleted state or may be displayed as an awkward multifunction device without an output tray.

The present invention was devised to solve the above problems, and when the marker to be recognized is shielded by another object or the shape is distorted, the feature point of the marker is extracted, and the extracted feature point is consistent with the feature point of the marker already stored. It is an object of the present invention to provide an apparatus and method for recognizing a shielded marker, which is recognized as a valid marker if it matches a threshold value or more and applied to marker tracking.

Another object of the present invention is to enable a more accurate marker recognition and tracking by applying usefully to a technique for recognizing a marker in an augmented reality environment in which a plurality of markers are arranged.

To this end, according to a first aspect of the present invention, in the apparatus for recognizing a marker from a real image obtained by photographing a real object in a real environment, a feature point of the marker is extracted by dividing a marker to be recognized from the real image. Feature point extraction unit; When the feature point extracted from the feature point extracting unit is applied, the original feature point information and the threshold value information of the corresponding marker stored in advance are detected, and whether the feature point extracted from the feature point extracting unit matches the original feature point information of the corresponding marker more than a threshold value. A validity determination unit that determines validity according to whether or not; A marker detector for detecting a valid marker determined by the validity determiner; And a storage unit that stores threshold information for calculating a degree of correspondence between the original feature point information of each marker to be recognized from the actual image and the original feature point information for each marker.

According to a second aspect of the present invention, a method of recognizing a marker from a real image obtained by photographing a real object in a real environment, the method comprising: extracting a feature point of the marker by distinguishing a marker to be recognized from the real image; Detecting original feature point information and threshold information previously stored for the marker, and determining whether the feature point extracted in the step is equal to or greater than a threshold value of the original feature point information of the marker; And as a result of the determination, recognizing a valid marker when the extracted feature point matches the original feature point information of the marker more than a threshold value and recognizing it as an invalid marker when the extracted feature point is less than the threshold value. And detecting the effective marker as a final marker and applying the marker to the position and posture measurement of the marker.

According to the present invention, even in augmented reality environment in which a plurality of markers are disposed, even a marker that is shielded or distorted by another object and distorted by a certain object satisfies a predetermined condition and is recognized and applied as a valid marker, thereby enabling accurate marker recognition and tracking. Do. Thus, there is an effect that can provide augmented reality compared to the existing.

1 is a view showing a state where a marker is shielded by a finger in augmented reality according to an embodiment of the present invention.
2 is a block diagram showing an apparatus for recognizing a shielded marker in augmented reality according to an embodiment of the present invention.
3 is a view for explaining a method of determining the validity according to the robustness of the marker in the marker recognition according to an embodiment of the present invention.
4 is a flowchart illustrating a method of recognizing a shielded marker in augmented reality according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The configuration of the present invention and the operation and effect thereof will be clearly understood through the following detailed description. Prior to the detailed description of the present invention, the same components will be denoted by the same reference numerals even if they are displayed on different drawings, and the detailed description will be omitted when it is determined that the well-known configuration may obscure the gist of the present invention. do.

1 is a view showing a state in which a marker is shielded by a finger in augmented reality according to an embodiment of the present invention.

The reference numeral 10 is an actual image of the real environment recognized through an image recognition means such as a web camera.

In the process of detecting and recognizing each of the markers 12 and 14 from the real image 10, the first marker 12 which is not influenced by the outside can accurately recognize a pattern in which a geometric pattern is used in the search definition frame. have.

On the other hand, in the second marker 14, a user's hand is disposed in front of the marker to cover a portion of the area. When the second marker 14 is recognized, the pattern of the region where the hand is not placed can be accurately recognized, but the region where the hand is placed 16 is covered by the hand image and cannot be accurately recognized.

However, if the area where the hand is placed is not the area corresponding to the pattern 14a of the marker but the area corresponding to the black border 14b, it is not a big problem to recognize the marker. In addition, although the area 16 on which the hand is placed includes a portion of the pattern 14a of the marker, it is not a big problem for recognizing the marker unless it obscures the characteristics of the pattern.

That is, even if a part of the marker is shielded, it can be applied as an effective marker as long as the minimum characteristic for identifying the marker can be known.

Accordingly, the present invention focuses on the above-described points, and extracts the feature points of each marker from the actual image to determine whether or not the markers are used depending on whether the extracted feature points coincide with the information of the original feature points of the corresponding markers by a predetermined threshold value or more. Determine.

The configuration for implementing this will be described in detail.

FIG. 2 is a block diagram illustrating an apparatus for recognizing a shielded marker in augmented reality according to an embodiment of the present invention, and FIG. 3 illustrates a method of determining effectiveness according to the robustness of the marker in marker recognition according to an embodiment of the present invention. It is a figure for following.

First, referring to FIG. 2, the marker recognition apparatus 100 according to an exemplary embodiment of the present invention may include an image recognition unit 110 and a marker recognition unit 120 as a feature point extractor 122, an effective determination unit 124, and a marker. The detector 126 includes a storage unit 130, a virtual object synthesis unit 140, and a display unit 150.

The image recognition unit 110 acquires an image (hereinafter, referred to as a real image) of the real environment by photographing a real object existing in the real environment. Then, it is transmitted to the marker recognition unit 120. The image recognition unit 110 may be a camera or a webcam connected to an external control device such as a computer, or a camera embedded in a personal portable terminal.

The feature point extractor 122 classifies the marker from the actual image acquired by the image recognizer 110 and extracts the feature point using color information, line segment information, edge information, or the like of the marker.

When the feature point extracted from the feature point extractor 122 is applied, the validity determination unit 124 detects the original feature point information and the threshold value information of the marker previously stored in the storage unit 130, and the extracted feature point is the source of the marker. It compares and judges whether it matches more than the threshold value from the characteristic point information. As a result of the determination, if it matches more than the threshold value, the validity determination unit 124 recognizes the marker as a valid marker, and if it is less than the threshold value, the validity determination unit 124 recognizes the invalid marker.

The marker detector 126 extracts the valid marker according to the determination result of the validity determiner 124 to detect the final marker. In this case, the marker detector 126 may restore the effective marker to information of the corresponding marker stored in the storage 130 in some cases.

Therefore, the marker recognition unit 120 according to the above configuration extracts the feature point of the target marker to be recognized from the actual image and determines the validity. In the case of the effective marker, the marker detection unit 126 stores the marker stored in the storage unit 130. Recognize markers based on information.

The storage unit 130 may include a marker DB 132 storing information on various markers to be recognized, and a virtual object DB 134 storing information on a virtual object mapped to a marker stored in the marker DB 132. It may include.

The marker DB 132 stores feature point information using color information, line segment, and edge information of the marker, and previously stores threshold value information for calculating a degree of agreement with the feature point information.

As described above, the threshold value becomes a criterion for determining validity according to the degree of correspondence between the feature point extracted from the actual image by the validity determination unit 124 and the feature point information of the corresponding marker previously stored in the storage unit 130. .

Threshold information according to an embodiment of the present invention may be determined according to the robustness of the marker.

A marker's robustness is a measure of how accurate and easy it is to recognize a marker's pattern.A higher robustness means a higher recognition rate, and a lower toughness means a lower recognition rate. It is likely to fail.

Therefore, if the robustness of the marker is high, the minimum level of the threshold value is low since the minimum marker recognition is possible even when the marker shielding area is relatively large. On the contrary, when the toughness of the marker is low, since the marker recognition is possible only when the shielding area of the marker is small, the threshold level may be determined to be high.

3 illustrates a method of determining the effectiveness according to the robustness of the marker as an example.

That is, it is assumed that the robustness of the marker is 80 and 60, respectively, for the markers A and B that are detected from the actual image, and the threshold values are 60% and 80%, respectively, considering that the thresholds are inversely related to the robustness of the markers.

When the same occluded region occurs in the marker A and the marker B as shown in FIG. 3B, recognition of each marker is based on the region excluding the occluded region.

When the range of the shielding area is about 30%, the feature point of the marker A extracted from the area excluding the shielding area satisfies the condition of 60% or more of the threshold value matching the original feature point information of the marker A. Thus, shielded marker A can be seen as a valid marker.

However, the feature point of the marker B extracted from the region excluding the shielding region may be regarded as an invalid marker because the feature point of the marker B does not satisfy the condition of 80% or more of the threshold value of the original feature point information of the marker B.

As such, when the marker recognizer 120 determines whether to use the marker by determining the validity, the virtual object synthesizer 140 stores the virtual object mapped to the final marker determined by the marker recognizer 120. Is detected, and synthesized with the actual image acquired through the image recognizer 110. The synthesized image is output to the display unit 150.

Then, the display unit 150 is an image input from the virtual object synthesizing unit 140, that is, the actual image obtained through the image recognition unit 110 and a visual virtual object such as two-dimensional or three-dimensional image, video is synthesized one Display the video.

Accordingly, the marker recognition apparatus 100 of the present invention according to the above configuration does not exclude even if a part or one or more of the plurality of markers are shielded or distorted, and according to the degree to match the original feature of the marker It is recognized as an effective marker and applied to the measurement of the marker's orientation (specifically, position and posture).

On the other hand, in a situation where a plurality of effective markers are required for azimuth recognition in augmented reality, the markers are shielded or distorted due to an error or an external environment of the marker recognition, and thus, a plurality of effective markers required for azimuth recognition cannot be recognized. May occur. In this case, in the present embodiment, it is determined whether the number of markers required for orientation recognition is basically determined from the feature points extracted from the feature point extractor 122 of FIG. The validity judgment is performed through the unit (124 of FIG. 2). As a result, even a shielded or distorted marker may be recognized as an effective marker to increase the recognition rate of the marker.

Next, the process of recognizing the shielded marker using the configuration of the marker recognition apparatus 100 described above will be described in detail.

4 is a flowchart illustrating a method of recognizing a shielded marker in augmented reality according to an embodiment of the present invention.

First, before recognizing a marker, a feature point is defined for at least one marker to be recognized, and the defined feature point information (hereinafter, referred to as original feature point information) is stored in advance in the storage unit 130 for each marker. The virtual object matching the at least one marker to be recognized is stored in the storage 130 in advance.

Thereafter, the marker recognizer 120 captures a real object existing in the real environment through the image recognizer 110 to obtain a real image of the real environment (S100).

Thereafter, the marker recognizer 120 classifies and detects markers from the actual image acquired by the image recognizer 110, and extracts feature points using color information, line segment, and edge information of the detected markers (S110 and S120). ).

Then, it is determined whether the extracted feature point is shielded or distorted (S130).

Whether it is shielded or distorted may be easily determined depending on whether the extracted feature point is different from the original feature point information of the corresponding marker previously stored in the storage unit 130.

As a result of the determination, if the extracted feature point is not shielded or distorted, it is recognized as a valid marker (S160). However, if it is determined that the extracted feature point is shielded or distorted, the validity determination unit 124 determines that the marker is stored from the storage unit 130. The original feature point information and the threshold value information are detected, and a comparison is made to determine whether the extracted feature point matches the original feature point information of the corresponding marker above the threshold value (S140).

Through the determination, if the extracted feature points match the threshold or more, the validity determination unit 124 recognizes the corresponding marker as a valid marker (S160), and if it is less than the threshold value, recognizes the invalid marker (S150). After recognition, the valid marker is detected as the final marker and applied to the orientation measurement of the marker.

In this case, if necessary, the valid marker may be restored to the information of the corresponding marker stored in the storage 130 and detected as the final marker.

Thereafter, the virtual object mapped with the marker is detected from the storage 130 (S170).

Thereafter, the detected virtual object is positioned on the marker and synthesized with the actual image, and then provided as one image through the display unit 150 (S180).

Thus, even if the marker is shielded or distorted through the above process, if a certain condition is satisfied, the marker can be used as an effective marker for more accurate marker recognition and tracking, thereby providing augmented reality.

On the other hand, the present invention can be applied to a variety of playback apparatus by implementing the augmented reality method as a software program to record on a computer-readable predetermined recording medium.

Various playback devices may be PCs, laptops, portable terminals, and the like.

The recording medium may be an internal hard disk of each playback device, or an external optical disk such as a CD-R or CD-RW, a compact flash card, a smart media, a memory stick, or a multimedia card.

In this case, the program recorded on the computer-readable recording medium, as described above, the process of extracting the feature points of the marker by distinguishing the marker to be recognized from the actual image of the real environment, and the previously stored circle for the extracted marker Detecting the feature point information and the threshold value information, and determining whether or not the feature points of the marker extracted in the above process is equal to or greater than the threshold stored in the original feature point information, the determination result, the original feature point information If it is equal to or greater than a threshold value, the recognition may be performed as a valid marker, and if it is less than the threshold value, the invalid marker may be detected, and the effective marker may be detected as a final marker and applied to azimuth measurement of the marker.

The above description is merely illustrative of the present invention, and various modifications may be made by those skilled in the art without departing from the technical spirit of the present invention. Therefore, the embodiments disclosed in the specification of the present invention are not intended to limit the present invention. The scope of the present invention should be construed according to the following claims, and all the techniques within the scope of equivalents should be construed as being included in the scope of the present invention.

In augmented reality (AR) based on a conventional reality environment, when a marker is shielded or distorted, marker recognition is performed by excluding the marker, but the present invention classifies and extracts the feature points of the marker, and the extracted feature points are already stored. If it matches more than a certain threshold with the feature point of the marker, it is recognized as a valid marker and applied to measuring the position and posture of the marker to enable more accurate marker recognition and tracking, thereby providing augmented reality compared to the existing one. have.

10: real image 12, 14: first and second markers
14a: pattern of the marker 14b: border of the marker
16: area where the hand is placed 100: marker recognition device
110: image recognition unit 120: marker recognition unit
122: feature point extracting unit 124: validity determining unit
126: marker detection unit 130: storage unit
132: marker DB 134: virtual object DB
140: virtual object synthesizing unit 150: display unit

Claims (11)

An apparatus for recognizing one or more markers from a real image obtained by photographing a real object in a real environment,
A feature point extracting unit for extracting feature points of the plurality of markers by separating the plurality of markers to be recognized from the real image;
When the feature point extracted from the feature point extracting unit is applied, the original feature point information and the threshold value information of the corresponding marker stored in advance are detected, and whether the feature point extracted from the feature point extracting unit matches the original feature point information of the corresponding marker more than a threshold value. A validity determination unit that determines validity according to whether or not;
A marker detector for detecting a valid marker determined by the validity determiner; And
A storage unit storing threshold information for calculating a degree of correspondence between the original feature point information of each marker to be recognized from the real image and the original feature point information for each marker
Apparatus for recognizing a shielded marker in augmented reality comprising a.
The method of claim 1,
The threshold information is
Apparatus for recognizing a shielded marker in augmented reality, characterized in that calculated according to the robustness (robustness) of the plurality of markers.
The method of claim 1,
When the feature point extracted from the feature point extracting unit is applied, the validity determining unit determines whether the number of markers basically required for orientation recognition is insufficient from the feature point, and when the number of markers is insufficient, the validity determination is performed. A device for recognizing a shielded marker in augmented reality.
The method of claim 1,
The feature point is
The apparatus for recognizing a shielded marker in augmented reality, characterized in that the color information, line segment information, edge information that is characteristic of the marker.
The method of claim 1,
The marker detection unit
The apparatus for recognizing a shielded marker in augmented reality, wherein the effective marker can be restored by detecting information of the corresponding marker stored in the storage unit.
In a method of recognizing a marker from a real image obtained by photographing a real object in a real environment,
Extracting feature points of the marker by dividing a marker to be recognized from the real image;
Detecting original feature point information and threshold information previously stored for the marker, and determining whether the feature point extracted in the step is equal to or greater than a threshold value of the original feature point information of the marker;
If the extracted feature point matches the original feature point information of the marker above a threshold value, recognizing it as a valid marker; And
Detecting the effective marker as a final marker and applying the marker to orientation measurement of the marker
Recognition method of a shielded marker in augmented reality, characterized in that it comprises a.
The method of claim 6,
Extracting the feature point of the marker,
The method of recognizing a shielded marker in augmented reality, characterized in that the extraction using the color information, line segment information, edge information of the marker.
The method of claim 6,
Before the step of determining whether or not to match the original feature point information of the marker more than a threshold,
In the augmented reality, it is determined whether the number of markers basically required for orientation recognition is insufficient from the feature points of the extracted markers, and if the number of markers is insufficient, it is determined whether or not to match the threshold. How to recognize markers.
The method of claim 6,
The threshold information is
The method of recognizing a shielded marker in augmented reality, characterized in that determined according to the robustness (robustness) of the marker.
The method of claim 6,
Detecting the effective marker as the final marker,
A method of recognizing a shielded marker in augmented reality, wherein it is possible to recover the effective marker with information of a corresponding marker stored in advance, if necessary.
A computer-readable recording medium having recorded thereon a program for executing the process of any one of claims 6 to 10.

Images