KR101314756B1 - Augmented Reality Providing Apparatus And Method Using Control Marker - Google Patents

Abstract

The disclosed technology relates to an apparatus and method for providing augmented reality using a control marker. The present invention relates to a photographing unit for photographing an image in the real world, a detector for detecting an object marker and a control marker included in the image, and augmenting and controlling a virtual object in the object marker. An implementation unit for augmenting a menu in the marker and a controller for controlling the augmented virtual object with a command included in the menu. Therefore, even if a part of the object marker or the control marker is covered, the original shape of the marker may be detected to augment the virtual object or the menu. The augmented virtual object is expressed in various ways with the commands included in the menu. This accurately detects markers and provides realistic augmented reality.

Description

Augmented Reality Providing Apparatus And Method Using Control Marker}

The disclosed technology relates to an apparatus and method for providing augmented reality, which is more detailed but without limitation, easily extracts a marker even if a portion of the marker is covered, and augments a virtual object augmented in the extracted object marker to a control marker. An apparatus and method for providing augmented reality for controlling using a menu are provided.

Augmented Reality (Augmented Reality) is a technology that reinforces additional information that is difficult to obtain in the real world by overlapping virtual objects or information on the basis of the real world, unlike a virtual reality that assumes a perfect virtual world. The terms used in the sense of augmented reality are extended reality and mixed reality. Unlike virtual reality, which presupposes a complete virtual world, it combines virtual objects over the real world environment, making it the next generation display technology in terms of realism and scalability.

In an apparatus and method using augmented reality, the main technique is to find a coordinate of an object to be augmented by detecting a marker. In the conventional augmented reality implementation technology, when a part of the marker is covered, there is a problem of failing to construct the system by not detecting the marker, or incorrectly detecting the marker to construct a wrong system. As a technique related to a method of detecting a marker, there is a Japanese Laid-Open Patent Publication No. 2005-293141 (marker detection method, marker detection device, position pose estimation method, and composite reality space presentation method).

The disclosed technology solves the problems of the aforementioned prior art in detecting the hidden markers, and provides an apparatus and method for providing augmented reality that realistically expresses a virtual object using a control marker.

In order to achieve the above technical problem, a first aspect of the disclosed technology includes a photographing unit for photographing an image representing a real world, a detection unit for detecting a control marker and at least one object marker from the photographed image, and a virtual area of the object marker. An augmented reality providing apparatus comprising an implementation unit for augmenting an object and augmenting at least one menu item in an area of the control marker and a controller for controlling the virtual object by using a command corresponding to the menu item. .

According to a second aspect of the present invention, there is provided a method of photographing an image representing a real world, detecting a control marker and at least one object marker from the photographed image, and augmenting a virtual object in an area of the object marker. And augmenting at least one menu item in an area of the control marker, and controlling the virtual object by using a command corresponding to the menu item.

Embodiments of the disclosed technique may have effects that include the following advantages. It should be understood, however, that the scope of the disclosed technology is not to be construed as limited thereby, since the embodiments of the disclosed technology are not meant to include all such embodiments.

According to an embodiment of the disclosed technology, the apparatus and method for detecting a marker has an effect of augmenting a virtual object to be augmented by the original marker to a user by detecting the hidden marker by the components and methods included in the disclosed technology.

In addition, according to another embodiment of the disclosed technology to express the virtual object augmented by the control device and method using the control marker has the effect of providing realism and interest.

1 is a diagram of a singular or plural object markers and a control marker.
2 is a block diagram of an apparatus for providing augmented reality according to an embodiment of the disclosed technology.
FIG. 3 illustrates an embodiment of a component included in a detector among the components of FIG. 2.
4 shows steps for marker detection included in one embodiment of the disclosed technique.
5 is a flowchart of an embodiment of an apparatus for providing augmented reality included in an embodiment of the disclosed technology.

The description of the disclosed technique is merely an example for structural or functional explanation and the scope of the disclosed technology should not be construed as being limited by the embodiments described in the text. That is, the embodiments are to be construed as being variously embodied and having various forms, so that the scope of the disclosed technology should be understood to include equivalents capable of realizing technical ideas.

Meanwhile, the meaning of the terms described in the present application should be understood as follows.

The terms " first ", " second ", and the like are used to distinguish one element from another and should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" to another element, it may be directly connected to the other element, but there may be other elements in between. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions describing the relationship between the components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring to", should be interpreted as well.

Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "include" or "have" refer to features, numbers, steps, operations, components, parts, or parts thereof described. It is to be understood that the combination is intended to be present, but not to exclude in advance the possibility of the presence or addition of one or more other features or numbers, steps, operations, components, parts or combinations thereof.

Each step may occur differently from the stated order unless the context clearly dictates the specific order. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosed technology belongs. Terms such as those defined in the commonly used dictionaries should be construed to be consistent with the meanings in the context of the related art and should not be construed as having ideal or overly formal meanings unless expressly defined in this application. .

1 is a diagram for an object marker and a control marker. FIG. 1 (1) is a diagram of one object marker 110 and a control marker 120, respectively, and FIG. 1 (2) shows a plurality of object markers 110a, 110b, 110c and control marker 120. This is the drawing. The marker is divided into the object marker 110 and the control marker 120 according to the type. For example, the method of distinguishing the object marker 110 from the control marker 120 may be a method of changing the color of the marker. As another example, as shown in FIG. 1, a symbol or a text may be inserted into the marker to distinguish the object marker 110 from the control marker 120. The object marker 110 is a marker indicating an area where a general virtual object is to be augmented. The object marker 110 may have information about the virtual object that is augmented in the image viewed by the user. The control marker 120 displays an area where a menu for controlling the virtual object is to be augmented. Menus contain commands to control virtual objects. For example, if the augmented virtual object is a dog, the menu may include commands such as object color conversion, object reduction, object enlargement, and the like. In addition, the menu may include a command having an association with the virtual object. For example, if the virtual object augmented at the object marker 120 is a dog, the commands included in the menu augmented at the control marker 120 may be provided with sitting, barking, and tailing in association with the dog. According to an embodiment, the object marker 110 may exist as a plurality of object markers 110a, 110b, and 110c in one image, as shown in FIG. 1 (2). In addition, the control marker 120 may control the plurality of object markers 110a, 110b, and 110c at the same time or a desired portion. For example, the control marker 120 may calculate distances from the object markers 110a, 110b, and 110c, and control the object marker 110c located closest thereto. As another example, the control marker 120 may control all of the plurality of object markers 110a, 110b, and 110c included in the image at once.

2 is a block diagram of an apparatus 200 for providing augmented reality according to an embodiment of the disclosed technology. Referring to FIG. 2, the apparatus for providing augmented reality 200 includes a photographing unit 210, a detector 220, an implementation unit 230, and a controller 240. The photographing unit 210 captures an image of the real world. The captured image may include at least one object marker 110 and a control marker 120. For convenience of explanation, it is assumed that the captured image includes at least one object marker 110 and a control marker 120 as shown in FIG. 1. The detector 220 detects the object marker 110 and the control marker 120 from the captured image. For example, if a part of the object marker 110 and the control marker 120 to be detected is damaged or obscured by an obstacle, the hidden part may be detected based on a canny edge algorithm and a hop transform algorithm. In addition, the detector 220 detects an operation means in the captured image. The operation means accesses the control marker 120 area to select the augmented menu item. For example, the manipulation means may be a tool (eg, a hand-held light emitting device, a baton) to recognize the body end in the captured image. In another example, the manipulation means may be the body end (eg, finger, etc.) itself. The implementation unit 230 augments the virtual object in the area of the object marker 110 and augments at least one menu item in the area of the control marker 120. For example, only one menu item (object collapse) may be augmented to the control marker 120, and as another example, a plurality of menu items (sitting, barking, and tailing) may be augmented. The controller 240 controls the virtual object using a command corresponding to the menu. For example, if the detected object markers 110a, 110b, 110c are plural, the control unit 240 may augment the virtual marker in one of the selected object marker regions based on the coordinate difference between the object markers 110a, 110b, 110c. You can control the object. According to an exemplary embodiment, the controller 240 may control the virtual object according to the manipulation of the manipulation means detected by the detector 220. For example, if it is determined that the manipulation means is located in a menu item augmented by the control marker 120, the controller 240 may control the virtual object according to a corresponding command. In addition, according to an exemplary embodiment, when there are a plurality of menu items enhanced in the control marker 120, the controller 240 may control the virtual object according to a command corresponding to the menu item selected by the manipulation means.

3 is a block diagram illustrating the detector of FIG. 2 in accordance with an embodiment of the disclosed technology. Referring to FIG. 3, the detector 220 includes a straight line detector 310, a connection line generator 320, and a vertex determiner 330. The straight line detector 310 detects straight lines in the captured image. The straight line detector 310 according to an embodiment may include a boundary line extractor 310a and a straight line extractor 310b. The boundary extractor 310a extracts a boundary existing in the image. For example, the boundary extractor 310a may remove noise of an image and extract a boundary based on the Kenny edge algorithm. The straight line extracting unit 310b extracts a straight line from the boundary lines extracted by the boundary extracting unit 310a. For example, the straight line extractor 310b may extract, as a straight line, a boundary line having a high intensity of straight lines among the boundary lines based on the hop transform algorithm. The connection line generator 320 connects the marker reference points to generate connection lines 420a and 420b passing through the center of the marker. Marker reference points are symmetry points that exist in a straight line with each other between the marker center points. For example, the connection line generator 320 may generate the connection lines 420a and 420b that connect two predetermined reference points to pass through the center point. As another example, the center point may be displayed on the marker itself to generate connection lines 420a and 420b connected from the center point to the reference point. The vertex determination unit 330 restores the marker by determining a vertex estimated to meet the corner of the masked marker. For example, the vertex determiner 330 first calculates the intersection points of the straight lines and the extension lines detected by the straight line detector 310 as vertex candidates. Thereafter, the vertex determiner 330 may determine, as a vertex, an intersection present at the shortest distance from the connection line generated by the connection line generator 320 among the intersection points calculated as the vertex candidates.

4 illustrates a process of detecting an object marker or a control marker that is partially hidden. FIG. 4 (1) is a diagram showing a marker with a portion hidden. 4 shows a process of calculating intersection points of straight lines or extension lines detected in an image. Since the straight lines 410a, 410b, 410c, and 410d shown in (2) of FIG. 4 may be inner and outer marker lines, an intersection point between them may be regarded as a vertex of the marker. 4 (3) shows the state of detecting the vertex of the hidden portion of the marker. Since the vertices of the hidden portions become reference points of the inner and outer sides of the marker, the original shape of the marker can be restored. In order to find the vertex, connecting lines 420a and 420b are generated to find an intersection point closest to the connecting line among the intersection points calculated in FIG. For example, as illustrated in (3) of FIG. 4, the connecting line 420a may be generated by connecting the marker center point with the reference point of the inner edge of the marker. As another example, the connection line 420a may be generated by connecting a marker center point with a reference point outside the marker. The connection line 420b may be generated by passing through the same center point as the connection line 420a but connecting another reference point not used to generate the connection line 420a. The connection line 420b may find vertices 430a and 430b that exist at the shortest distance among the vertices intersecting the straight lines 410a, 410b, 410c, and 410d inside and outside the marker. Fig. 4 (4) is the original state of the marker detected through the procedure of the preceding (1), (2) and (3).

5 is a flowchart illustrating a method of providing augmented reality according to an embodiment of the disclosed technology. In operation 510, the photographing unit captures and inputs an image representing the real world. For example, the captured image may be a camera, a camcorder, or a smartphone. The photographed image may include at least one object marker 110 and a control marker 120 because the photographed image is photographed for the purpose of providing augmented reality. In operation 520, the object marker 110 and the control marker 120 included in the image are detected. For example, the detection of the marker may be made in a method based on the Kenny edge algorithm and the hop transform algorithm. In operation 530, a menu for the virtual object and the virtual object is augmented in the detected object marker 110 area and the control marker 120 area, respectively. The menu includes at least one command for the augmented virtual object. In operation 540, a motion is detected by detecting an operation means for controlling a virtual object by accessing an enhanced menu. In one example, the manipulating means may be a body end that is a hand, foot or head. In another example, the manipulation means may be an object that is an indicator rod or a rod. In addition, a method of accessing a menu by recognizing a manipulation means on a screen or using a touchpad as a manipulation means may be used. In step 550, a command provided from a menu is selected. The menu contains commands to control the virtual object. The menu may have a different form depending on the singular or plural augmented virtual objects. In one embodiment, if the augmented virtual object is a dog, it may be a menu that provides commands indicating the behavior or appearance of the dog. In another embodiment, if more than one augmented virtual object is provided, the commands of the menu may be bundled or separated according to the association between the virtual objects. For example, if augmented virtual objects are dogs, cats, and monkeys, a menu containing common commands by the association of animals may be provided together to provide sitting, standing, and barking together. It may be a menu that provides a separate command. In operation 560, the operation corresponding to the command of the menu is represented through the virtual object. For example, if the virtual object is a person and the command of the selected menu is sitting, the virtual person may be shown to sit as the corresponding command.

Apparatus and method for providing augmented reality through embodiments of the disclosed technology have been described with reference to the embodiments illustrated in the drawings for clarity, but these are merely exemplary and various modifications therefrom can be made by those skilled in the art. And other equivalent embodiments are possible. Accordingly, the true scope of protection of the disclosed technology should be determined by the appended claims.

110: object marker 120: control marker
110a, 110b, 110c: multiple object markers
200: augmented reality providing device 210: the photographing unit
220: detector 230: implementer
240 control unit 310 linear detection unit
310a: boundary line extraction unit 310b: straight line extraction unit
320: connection line generation unit 330: vertex determination unit
410a, 410b, 410c, 410d: straight lines 420a, 420b: connecting line
430a, 430b: vertices
510: Image input step 520: Marker detection step
530: object augmentation step 540: pattern recognition step
550: menu selection step 560: object operation step

Claims (15)

A photographing unit which photographs an image representing a real world;
A detector detecting a control marker and at least one object marker in the captured image;
An implementation unit for augmenting a virtual object in an area of the object marker and augmenting at least one menu item in an area of the control marker; And
And a controller for controlling the virtual object by using a command corresponding to the menu item.
Wherein:
A straight line detector for detecting straight lines in the image;
A connection line generation unit configured to generate a connection line by connecting reference points symmetrical to one of the center markers of the control marker and the at least one object marker; And
And a vertex determining unit configured to determine, as vertices of the markers to be detected by the detector, the intersection between the straight lines or the intersection points of the straight lines having the shortest distance from the connection line.
The reference points, the augmented reality providing apparatus comprising a vertex existing in or out of the control marker and the at least one object marker. The apparatus of claim 1,
The apparatus for providing augmented reality that controls the virtual object based on a distance between an area of the object marker and an area of the control marker. The method of claim 1, wherein when the detector detects a plurality of object markers,
The implementation unit augments each virtual object corresponding to each object marker in an area of each object marker,
The control unit, augmented reality providing apparatus for controlling an augmented virtual object in the region of any one object marker selected based on the coordinate difference between the control marker and the plurality of object markers of the plurality of object markers. The method of claim 1, wherein the implementation unit augments a plurality of menu items in an area of the control marker.
The detection unit detects an operation means in the captured image,
The controller selects any one of the plurality of menu items according to the movement of the manipulation means and controls the virtual object by using a command corresponding to the selected one of the menu items. . delete delete The method of claim 1, wherein the center point,
The apparatus for providing augmented reality displayed on the center of the control marker or the at least one object marker or on the center of the connecting line. The method of claim 1, wherein the straight line detector,
A boundary line extracting unit for extracting boundary lines existing in the image based on a canny edge algorithm; And
And a straight line extracting unit configured to extract straight lines among the boundary lines based on a straight line strength calculated according to a hop transform algorithm. Photographing an image representing a real world;
Detecting a control marker and at least one object marker from the captured image;
Augmenting a virtual object in an area of the object marker and augmenting at least one menu item in an area of the control marker; And
Controlling the virtual object using a command corresponding to the menu item;
Wherein the detecting comprises:
Detecting straight lines in the captured image;
Generating a connection line by connecting reference points symmetrical to a center point of any one of the control marker and the at least one object marker; And
Determining the vertex of the marker to detect the intersection point of the shortest distance from the connection line among the intersection points between the straight lines or the extension lines of the straight lines;
The reference points, augmented reality providing method comprising a vertex existing in or inside the control marker and the at least one object marker. The method of claim 9, wherein the controlling step,
And controlling the virtual object based on a distance between the area of the object marker and the area of the control marker. The method of claim 10, wherein when a plurality of object markers are detected in the captured image,
Wherein the controlling comprises:
Selecting an object marker having a shortest distance from the control marker among the plurality of object markers; And
And controlling an augmented virtual object in an area of the selected object marker. The method of claim 9, wherein when a plurality of menu items are augmented in an area of the control marker,
Detecting an operation means in the captured image, and tracking a movement of the detected operation means;
The controlling may include selecting one menu item among the plurality of menu items according to the movement of the manipulation means, and controlling the virtual object using a command corresponding to the selected one menu item. Augmented reality providing method comprising a. delete delete The method of claim 9, wherein the detecting of the straight lines comprises:
Extracting boundary lines existing in the image based on a canny edge algorithm; And
And extracting straight lines from the boundary lines based on a straight line strength calculated according to a hop transform algorithm.

Images