KR101537651B1 - System for supplying virtual menu conneted and method thereof - Google Patents

Abstract

The present invention relates to a system and a method for providing a virtual menu, and more particularly, to a technique capable of operating a function-executing command by operating only an electronic device having a camera phone in a situation where a visual marker of an external appearance has a plurality of identical electronic devices .
To this end, the virtual menu providing system according to the present invention includes a database, an identification information extracting unit, an identification information receiving unit, an object recognizing unit, and a virtual menu enhancing unit. The database stores identification information about at least one object and information about a virtual menu corresponding to the identification information. The identification information extracting unit extracts the first identification information of the first object from the image received through the camera. The identification information receiving unit receives the second identification information from one or more objects having the same first identification information as the extracted first identification information. The object recognition unit recognizes the first object based on the received second identification information. The virtual menu enhancement unit provides a virtual menu for the recognized first object.

Description

TECHNICAL FIELD [0001] The present invention relates to a virtual menu providing system,

The present invention relates to a system and a method for providing a virtual menu, and more particularly, to a technique capable of operating a function-executing command by operating only an electronic device having a camera phone in a situation where a visual marker of an external appearance has a plurality of identical electronic devices .

The present invention has been derived from research carried out by the Ministry of Knowledge Economy and the Information and Communication Industry Promotion Agency as part of the 3D and Smart TV Competitiveness Enhancement Project [Project Number: I2904-12-1003, Title: 3D content based on AR technology Emotional Communication App and Smart Object Development].

Augmented Reality (AR) is a technique for superimposing virtual images on the real world that a user sees. It is also referred to as a mixed reality (MR) because a virtual world including additional information is combined into a single image . Such augmented reality contrasts with virtual reality. In other words, the virtual reality technology generally allows the user to immerse in the virtual environment, so that the user can not see the actual environment, whereas the augmented reality technology allows the user to see the real environment, .

Such augmented reality technology has been generally used in a high performance specification system because of its large amount of computation processing. However, recent mobile communication terminals are equipped with a camera as a basic specification, It is equipped with high specification and provides various services such as 3D rendering or camera image processing to users and furthermore provides virtual reality technology or augmented reality technology which was possible on high performance desktop.

Currently, there are "hot code" and "visual code recognition systems" where augmented reality technology is applied to handsets. First, "hot code" is a technique using a conventional bar code. When a user simply illuminates a bar code with a terminal equipped with a camera (aka a camera phone), the camera of the camera phone captures the bar code, Augmented reality technology that moves to a space on the screen. The "visual code recognition system" is an augmented reality technology that recognizes bar codes and pre-stores information (i.e., virtual objects such as text, voice, or images) stored in the storage unit of the terminal on the display of the terminal.

In Korean Patent Application No. 10-2012-0032336 entitled " Mobile Terminal and Display Method of Augmented Reality Information ", when an AR information icon of each object is superimposed on the object in a preview image photographed by a camera of a terminal, Discloses a technique for providing an AR information screen in which an AR information icon of an object in a preview image is displayed in front of the preview image.

On the other hand, a user may control an electronic device through an enhanced menu by illuminating an electronic device capable of wireless communication as a camera phone. At this time, if the visual markers such as the appearance of the electronic device, specifically the image on the external appearance are the same in the neighborhood of the electronic device which is represented by the camera phone (within a distance enabling wireless communication with the camera phone) It is difficult to determine which electronic device should be controlled. That is, there is a problem that it is difficult to determine to which electronic device the control message should be transmitted only with the real image information containing the appearance of the electronic device. Therefore, there is a need for a technique capable of operating a function-executing command by operating only an electronic device having a camera phone in a situation where there are a plurality of the same electronic devices.

Korean Patent Publication No. 2010-0138689 "Management Server and Method Providing Cloud Computing Service"

In the present invention, when the visual markers such as the appearance of the electronic device, specifically the image on the external appearance are the same in the vicinity of the electronic device which is represented by the camera phone (within a distance within which wireless communication with the camera phone is possible) Which is difficult to judge which electronic device should perform the control according to the present invention.

The purpose is to operate only the electronic device that has been displayed as a camera phone and to issue a function execution command in a situation where there are a plurality of visual devices having the same visual markers.

To achieve these and other advantages and in accordance with the purpose of the present invention, as embodied and broadly described herein, a virtual menu providing system includes a database, an identification information extracting unit, an identification information receiving unit, an object recognizing unit, and a virtual menu enhancing unit. The database stores identification information about at least one object and information about a virtual menu corresponding to the identification information. The identification information extracting unit extracts the first identification information of the first object from the image received through the camera. The identification information receiving unit receives the second identification information from one or more objects having the same first identification information as the extracted first identification information. The object recognition unit recognizes the first object based on the received second identification information. The virtual menu enhancement unit provides a virtual menu for the recognized first object.

As described above, according to the present invention, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera, (Additional markers), the visual marker can distinguish the same objects and issue a control command only to the objects rendered by the camera.

Further, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera, different second identification information is assigned to each of the objects, The first object can be recognized by receiving the second identification information from the first object and comparing it with the second identification information designated. Particularly, the method of comparing the second identification information recognized through the screen with the designated second identification information can be applied to a case where only one-way communication with the object is possible in the mobile communication terminal.

Also, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera, the mobile communication terminal requests and receives the second identification information from each object, The first object and the second identification information of the first object recognized through the first object and the second identification information of the first object recognized through the second object, A function execution command according to the operation of the virtual menu can be issued.

Also, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera, the mobile communication terminal may perform a 'tap object' or ' And if the user takes an action in accordance with the guidance message, performs wireless communication for a function operation with the first object that transmitted the shake information, The user can issue a function execution command corresponding to the operation of the virtual menu only to the object desired by the user. Particularly, since this method uses the shaking information as the second identification information, it is possible that the mobile communication terminal does not illuminate the object.

FIG. 1 is a diagram showing a schematic configuration of a virtual menu providing system according to an embodiment of the present invention.
2 is a diagram showing an example in which a virtual menu is enhanced on an object A on the screen according to the present invention.
FIG. 3 is a view showing an additional visual marker displayed on objects having the same appearance according to the present invention.
4 is a diagram illustrating a schematic configuration of a virtual menu providing system using a method of assigning second identification information to an object according to another embodiment of the present invention.
FIG. 5 is a flowchart illustrating an operation procedure of the virtual menu providing system implemented in FIG.
FIG. 6 is a diagram illustrating a schematic configuration of a virtual menu providing system using a method of requesting second identification information to an object according to another embodiment of the present invention.
FIG. 7 is a flowchart illustrating an operation procedure of the virtual menu providing system implemented in FIG.
FIG. 8 is a diagram illustrating a schematic configuration of a virtual menu providing system using shaking information of an object according to another embodiment of the present invention.
FIG. 9 is a flowchart illustrating an operation procedure of the virtual menu providing system implemented in FIG.
FIG. 10 is a flowchart illustrating a method of providing a virtual menu according to an exemplary embodiment of the present invention. Referring to FIG.
11 is a flowchart illustrating a method of providing a virtual menu using a method of assigning second identification information to an object according to another embodiment of the present invention.
12 is a flowchart illustrating a method of providing a virtual menu using a method of requesting second identification information to an object according to another embodiment of the present invention.
13 is a flowchart illustrating a method of providing a virtual menu using shaking information of an object according to another embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. Further, in the description of the embodiments of the present invention, specific values are merely examples, and exaggerated values may be presented for convenience and understanding of the present invention.

<Description of System>

FIG. 1 is a diagram illustrating a schematic configuration of a mobile communication terminal to which a virtual menu providing system according to an embodiment of the present invention is applied. The mobile communication terminal 20 can be implemented in various forms such as a mobile phone, a smart phone, a notebook computer, a digital broadcasting terminal, a PDA, and a PMP.

1, a virtual menu providing system 100 according to the present invention includes a database 110, an image receiving unit 120, an identification information extracting unit 130, an identification information receiving unit 140, an object recognizing unit 150, A virtual menu enhancement unit 160, and a control unit 170. [

The database 110 receives and stores identification information about at least one object and information about a virtual menu corresponding to the identification information from the server 10. The information on the virtual menu means information defined such that a virtual menu is associated with each identification information. The object is an actual object identified by an image, and the identification information of the object is information (e.g., marker information or non-marker information such as minutia information) for identifying the object. 2, the virtual menu is a menu (button) to be augmented on an object, and is a UI component (User Interface Component) capable of manipulating the function of the object object or a function associated with the object.

The image receiving unit 120 receives the real image photographed by the camera 21 when the real image of the real object to be viewed by the user is photographed by the camera 21. [

The identification information extracting unit 130 extracts the first identification information of the first object from the real image received through the photographing of the camera 21. [ The first identification information is visual marker information in the appearance of the object. The visual markers may be black rectangular frames used in the Augmented Reality Toolkit or overlapping rectangles used in the QRCode, and the present invention is not limited by such a configuration.

The identification information receiving unit 140 receives the second identification information from one or more objects having the same first identification information as the extracted first identification information. The at least one object is paired with the mobile communication terminal 20 by Bluetooth or the like, and performs wireless communication. The second identification information is information on additional markers that can recognize objects having the same first identification information, and refers to information related to protocol methods mutually determined between the mobile communication terminal 20 and objects. For example, when there is a display available for an electronic device, the electronic device may display an identifiable image or the like to indicate the second identification information. As another example, if there is no display in the electronic device and an LED is present as shown in FIG. 3, the LED indicates flickering (coded signal generation that can be identified by changing the flickering period such as Morse code) to indicate the second identification information It is possible.

The object recognition unit 150 recognizes the first object based on the received second identification information.

The virtual menu enhancement unit 160 provides a virtual menu for the recognized first object on the electronic device in the form of a UI component as shown in FIG. For example, an operation button for listening to music such as volume control and play control is virtually augmented and displayed on the surface of the speaker. Basically, the UI components that can be applied to the virtual menu may be all UI components used in a general PC or a smart phone, and the present invention is not limited thereto.

The controller 170 processes a function corresponding to the virtual menu on the first object when the virtual menu is selected by the user. For example, while the mobile communication terminal 20 keeps illuminating the object A, the virtual menu defined in advance is augmented and displayed at the object A position in the real image, and the virtual menu operation command of the user is transmitted to the object A. If object A disappears from the camera image, additional visual marker display such as LED blinking on the electronic device need not be continued.

In this way, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera 21, (Additional markers), the visual markers can distinguish the same objects and can issue control commands only to the objects displayed on the camera 21.

The present invention is also applicable to a state in which the communication of the object displayed by the camera 21 is turned off. Further, in the present invention, when a plurality of objects having the same first identification information are recognized, or when the communication between the object and the mobile communication terminal 20 is bidirectional or unidirectional, the second identification information is information The present invention will be described in detail.

(APPLICATION EXAMPLE 1 - Case where the mobile communication terminal designates the second identification information for each object)

FIG. 4 is a diagram illustrating a schematic configuration of a virtual menu providing system according to another embodiment of the present invention, and FIG. 5 is a flowchart illustrating an operation procedure of the virtual menu providing system implemented in FIG.

4, a virtual menu providing system 200 according to the present invention includes a database 210, an image receiving unit 220, an identification information extracting unit 230, an identification information transmitting unit 240, an identification information receiving unit 250 An object recognition unit 260, a virtual menu enhancement unit 270, and a control unit 280. [ A configuration similar to the above-described configuration will be briefly described.

The database 210 stores identification information about at least one object and information about a virtual menu corresponding to the identification information.

The image receiving unit 220 receives the real image photographed by the camera 21 when the real image of the real object to be viewed by the user is photographed by the camera 21. [

The identification information extracting unit 230 extracts the first identification information of the first object from the image received through the photographing of the camera 21. [

The identification information transmitting unit 240 specifies different second identification information to one or more objects having the same first identification information as the extracted first identification information and transmits the same.

The method of checking the second identification information differs depending on whether one or more objects are paired. In case of one, object recognition is performed only when the second identification information is correctly identified in mutually agreed manner. If there are two or more objects, it is necessary to know what object to control. In this case, the second identification number must be specified differently for each object so that identification is possible.

Accordingly, in the present invention, the mobile communication terminal 20 may designate a different method for each object in advance by checking the second identification information, and check what type of second identification information actually received. For example, when the object A is illuminated by the camera 21 as shown in FIG. 5A and the object B having the same first identification information of the object A exists, as shown in FIG. 5B, Designates the flicker period of the LEDs on the objects A and B as 1 second and 2 seconds, respectively. In this example, the flicker cycle represents the second identification information. Then, according to the above designation, as shown in (c), the object A flickers every one second, and the object B flickers every two seconds.

The identification information receiving unit 250 receives the second identification information from one or more objects having the same first identification information as the extracted first identification information, and the reception is based on a screen generated by photographing the communication network or the camera 21 Lt; / RTI &gt;

The object recognition unit 260 recognizes the first object based on the received second identification information and the second identification information recognized through the shooting of the camera 21. [ That is, as shown in FIG. 5, when the mobile communication terminal 20 designates flicker periods of the objects A and B as 1 second and 2 seconds, respectively, when the objects blink according to the designation, It is equal to the flicker period specified for object A. Alternatively, if the information indicating that the objects are blinking is transmitted to the mobile communication terminal 20 as described above, the mobile communication terminal 20 recognizes the object A that transmitted the information of one second cycle as an object to be operated.

The virtual menu enhancement unit 270 provides a virtual menu for the recognized first object (see (d) of FIG. 5).

The control unit 280 processes a function corresponding to the virtual menu selected on the first object by the user so that the corresponding function is performed on the first object.

As described above, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera 21, The first object can be recognized by specifying the information, receiving the second identification information from the first object through the communication or the screen, and comparing the received second identification information with the designated second identification information. In particular, the method of comparing the second identification information recognized through the screen with the designated second identification information can be applied to a case where only one-way communication with the object is possible in the mobile communication terminal.

(Application Example 2 - When the mobile communication terminal requests the second identification information to the object)

FIG. 6 is a diagram illustrating a schematic configuration of a virtual menu providing system according to another embodiment of the present invention, and FIG. 7 is a flowchart illustrating an operation procedure of the virtual menu providing system implemented in FIG.

6, a virtual menu providing system 300 according to the present invention includes a database 310, an image receiving unit 320, an identification information extracting unit 330, an identification information requesting unit 340, an identification information receiving unit 350 An object recognition unit 360, a virtual menu enhancement unit 370, and a control unit 380. A configuration similar to the above-described configuration will be briefly described.

The database 310 stores identification information about at least one object and information about a virtual menu corresponding to the identification information.

The image receiving unit 320 receives the real image photographed by the camera 21 when the real image of the real object to be viewed by the user is photographed by the camera 21. [

The identification information extracting unit 330 extracts the first identification information of the first object from the image received through the photographing of the camera 21.

The identification information request unit 340 requests the second identification information through the communication network to one or more objects having the same first identification information as the extracted first identification information. For example, when the mobile communication terminal 20 is paired with the objects A and B and illuminates the object A as shown in FIG. 7A, the identification information requesting unit 340 requests the objects A and B The second identification information, in this embodiment, the flicker information of the LED. Here, it is possible that the flicker period of the objects is set in advance.

The identification information receiving unit 350 receives second identification information from one or more objects according to the request. (c), each object transmits information about the blink period to the mobile communication terminal 20, and blinks the LED according to the set period.

The object recognition unit 360 recognizes the first object based on the received second identification information. That is, the mobile communication terminal 20 compares the flicker information received from the objects A and B with the flicker period of the object A recognized through the screen of the camera 21.

The virtual menu enhancement unit 370 provides a virtual menu for the recognized first object (see (d) of FIG. 7).

The controller 380 processes a function corresponding to the virtual menu selected on the first object by the user so that the corresponding function is performed on the first object.

As described above, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera 21, the mobile communication terminal requests the second identification information Comparing the first identification information with the second identification information of the first object recognized through the screen, and performing wireless communication for the first object and the subsequent operation if the comparison result is identical, The function execution command according to the operation of the virtual menu can be issued only to the object photographed by the camera 21. [

Since the second identification information of the objects received by the mobile communication terminal 20 may be the same, the second identification information is composed of a plurality of different second identification information. If the received second identification information is identical, the identification information requesting unit 340 requests the one or more objects to re-request the second identification information different from the same second identification information. For example, in a process in which two objects transmit each second identification information (for example, a message such as a visual marker + flashing period and a flashing period of one second) to the mobile communication terminal 20, When the mobile communication terminal 20 transmits the second identification information, the mobile communication terminal 20 requests the second identification information again for each object, and when the object that has been requested again transmits the second identification information, the mobile communication terminal 20 receives and processes the second identification information. Here, the second identification information may be defined in advance, and a plurality of pre-defined sets may be known to the mobile communication terminal 20 or the object.

As described above, if the second identification information is composed of a plurality of pieces of information and the second identification information of the objects received by the mobile communication terminal 20 is the same, the second identification information is requested again so that the present invention can be implemented without any problem.

In addition, in the present invention, when each object receives a message requesting re-request of the second identification information from the mobile communication terminal 20, the identification information receiving unit 350 extracts a second identification Information is received.

Thus, there is an effect that the probability that the second identification information received from each object is the same is reduced. In this case, when the second identification information is randomly selected from a plurality of objects and transmitted to the mobile communication terminal 20, it is preferable that the pre-defined set is large so as to minimize the probability of selecting the same one.

(Application Example 3 - When using a sensor embedded in an object)

Hereinafter, a system capable of transmitting a control command only to an object illuminated by the camera 21 without using an additional visual marker will be described.

FIG. 8 is a diagram illustrating a schematic configuration of a virtual menu providing system according to another embodiment of the present invention, and shows an operation procedure of the virtual menu providing system implemented in FIG.

8, a virtual menu providing system 400 according to the present invention includes a database 410, an image receiving unit 420, an identification information extracting unit 430, an operation command outputting unit 440, an identification information receiving unit 450 An object recognition unit 460, a virtual menu enhancement unit 470, and a control unit 480. [ A configuration similar to the above-described configuration will be briefly described.

The database 410 stores identification information about at least one object and information about a virtual menu corresponding to the identification information.

The image receiving unit 420 receives the real image photographed by the camera 21 when the real image of the real object to be viewed by the user is photographed by the camera 21. [

The identification information extracting unit 430 extracts the first identification information of the first object from the image received through the photographing of the camera.

The operation command output unit 440 outputs an operation command for the recognized object through the shooting of the camera 21. [ For example, when the object A is illuminated by the camera 21 as shown in Fig. 9A and the object B having the same first identification information of the object A exists, the operation command output unit 440 outputs (b) Such as 'tap the object' or 'please tilt the object slightly' or 'let the object be heard and released'. As a result, the user can take an action corresponding to the guidance phrase and lightly touch the object A as shown in (c). When the sensor signal of the object A changes according to the action of the user, the object A transmits information indicating that the mobile communication terminal 20 has shaken. In the present invention, it is preferable that the object includes a sensor capable of detecting vibration such as an acceleration sensor or a gyro sensor so as to easily grasp the shaking of the object.

The identification information receiving unit 450 receives the second identification information about whether or not the user has operated according to the output operation command. For example, if the user touches the object A lightly within a predetermined time after the mobile communication terminal 20 displays the guidance message as shown in (b), the information indicating that the object A has shaken is displayed on the mobile communication terminal 20 (20).

The object recognition unit 460 recognizes the first object based on the received second identification information. That is, as shown in (c), the object A outputting the shake information is recognized, and wireless communication is performed for the object A and the function operation.

The virtual menu enhancement unit 470 provides a virtual menu for the recognized first object (see (d) of FIG. 9).

The controller 480 processes a function corresponding to the virtual menu on the first object when the virtual menu is selected by the user.

As described above, even if there is one or more objects having first identification information identical to the first identification information (visual marker) of the first object exposed by the camera 21, Quot ;, &quot; Please listen to the object &quot;, and &quot; Please release the object &quot;, and if the user takes an action according to the guidance message, performs wireless communication for the function operation with the first object that transmitted the shake information , It is possible to issue a function execution command according to the operation of the virtual menu only to the object desired by the user even in the case of a plurality of the same electronic apparatuses. Particularly, this method uses shaking information as the second identification information, so that it is possible that the mobile communication terminal 20 does not illuminate the object.

On the other hand, in the present invention, when information indicating that the mobile communication terminal has been shaken at a plurality of objects simultaneously is received under the assumption that the same objects exist, a guidance message such as 'tap object' or 'tilt the object slightly' . Even if the object disappears from the image and the object is recognized again after a while even under the above-mentioned premise, the guiding phrase may be displayed.

<Description of Method>

A method of providing a virtual menu according to an embodiment of the present invention will be described with reference to a flowchart shown in FIG. 10, for convenience.

1. Storing identification information on an object and information on a corresponding virtual menu < S101 >

And stores identification information on at least one object in the database and information on a virtual menu corresponding to the identification information. The information on the virtual menu means information defined such that a virtual menu is associated with each identification information. The object is a real-life object identified by an image, and the identification information of the object is information (e.g., non-marker information such as marker information or minutia information) for identifying the object. The virtual menu is a menu that is augmented on the object, and can manipulate the function of the object object or the function associated with it.

2. Receiving a real image shot from a camera < S102 >

When a real image of a physical object that the user wants to view is photographed by a camera, the real image photographed from the camera is received.

3. Extracting the first identification information of the first object < S103 >

The first identification information of the first object is extracted from the image received through the photographing of the camera in step S102. The first identification information is visual marker information in the appearance of the object.

4. Receiving the second identification information from one or more objects having the same first identification information as the first identification information <S104>

And receives the second identification information from the object having the first identification information identical to the first identification information extracted in step S103. The at least one object is paired with the mobile communication terminal by Bluetooth or the like, and performs wireless communication. The second identification information is additional marker information that can recognize objects having the same first identification information, and refers to information related to protocol methods mutually determined between the mobile communication terminal and the objects. For example, when there is a display available for an electronic device, the electronic device may display an identifiable image or the like to indicate the second identification information. As another example, if there is no display in the electronic device and an LED is present as shown in FIG. 3, the LED indicates flickering (coded signal generation that can be identified by changing the flickering period such as Morse code) to indicate the second identification information It is possible.

5. Recognizing the first object based on the second identification information <S105>

And recognizes the first object based on the received second identification information.

6. Augmenting the virtual menu on the first object < S106 >

The virtual menu for the first object recognized in step S105 is provided on the electronic device in the form of a UI component as shown in FIG. For example, the operation buttons for listening to music such as volume control and play control are virtually displayed on the speaker surface.

7. When the virtual menu is selected by the user, the step of transmitting the command controlling the first object < S107 >

When a virtual menu enhanced on the first object is selected by the user, the corresponding function is processed in the first object. For example, while the mobile communication terminal 20 keeps illuminating the object A, the virtual menu defined in advance is augmented and displayed at the object A position in the real image, and the virtual menu operation command of the user is transmitted to the object A. If object A disappears from the camera image, additional visual marker display such as LED blinking on the electronic device need not be continued.

In this way, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera, the mobile communication terminal can extract the second identification information ), The visual marker can distinguish the same objects and output control commands only to the objects displayed on the camera.

The present invention can also be applied to a state in which the communication of an object displayed by a camera is turned off. In addition, in the present invention, when a plurality of objects having the same first identification information are recognized, or when the communication between the object and the mobile communication terminal is bidirectional or unidirectional, the second identification information is not information recognized on the screen Hereinafter, this will be described in detail.

(APPLICATION EXAMPLE 1 - Case where the mobile communication terminal designates the second identification information for each object)

11 is a flowchart illustrating a method of providing a virtual menu according to another embodiment of the present invention. Steps similar to those described above will be briefly described.

1. Step S201 of storing identification information on an object and information on a virtual menu corresponding thereto,

And stores identification information on at least one object in the database and information on a virtual menu corresponding to the identification information.

2. Step S202 of receiving the photographed image photographed from the camera < RTI ID = 0.0 >

When a real image of a physical object that the user wants to view is photographed by a camera, the real image photographed from the camera is received.

3. Extracting the first identification information of the first object < S203 >

In step S202, the first identification information of the first object is extracted from the image received through the camera.

4. Step S204 of assigning different second identification information to an object having the same first identification information as the first identification information and transmitting the same,

Different second identification information is assigned to one or more objects having the same first identification information as the first identification information extracted in step S203 and transmitted.

The method of checking the second identification information differs depending on whether one or more objects are paired. In case of one, object recognition is performed only when the second identification information is correctly identified in mutually agreed manner. If there are two or more objects, it is necessary to know what object to control. In this case, the second identification number must be specified differently for each object so that identification is possible. Accordingly, in the present invention, the mobile communication terminal designates a different method for each object in advance by checking the second identification information, and it is checked how the actually received second identification information is. For example, when the object A is illuminated by a camera and the object B having the same first identification information of the object A is present as shown in FIG. 5A, the mobile communication terminal displays the objects A and B Specify the blink period of each LED as 1 second and 2 seconds. In this example, the flicker period indicates the second identification information. Then, according to the above designation, as shown in (c), the object A flickers every one second, and the object B flickers every two seconds.

5. Receiving the second identification information from one or more objects having the same first identification information as the first identification information <S205>

And receives second identification information from at least one object having first identification information identical to the extracted first identification information, and the reception may be performed through a screen generated by shooting a communication network or a camera.

6. Recognizing the first object based on the second identification information < S206 >

The first object is recognized based on the second identification information received in step S205 and the second identification information recognized through photographing of the camera. That is, as shown in FIG. 5, when the mobile communication terminal designates the flicker periods of the objects A and B as 1 and 2 seconds, respectively, and when the objects are blinking according to the designation, Cycle. Alternatively, if information indicating that the objects are blinking is transmitted to the mobile communication terminal as described above, the mobile communication terminal recognizes the object A that transmitted the information of one second cycle as a target for the function operation.

7. Augmenting the virtual menu on the first object < S207 >

The virtual menu for the first object recognized in step S206 is provided on the electronic device in the form of a UI component (see FIG. 5 (d)).

8. When the virtual menu is selected by the user, the step of transmitting the command controlling the first object < S208 >

When a virtual menu enhanced on the first object is selected by the user, the corresponding function is processed in the first object.

As described above, even if one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera exist, the mobile communication terminal assigns different second identification information to each of the objects The first object can be recognized by receiving second identification information from the first object through communication or a screen and comparing the second identification information with the second identification information. In particular, the method of comparing the second identification information recognized through the screen with the designated second identification information can be applied to a case where only one-way communication with the object is possible in the mobile communication terminal.

(Application Example 2 - When the mobile communication terminal requests the second identification information to the object)

12 is a flowchart illustrating a method of providing a virtual menu according to another embodiment of the present invention. Steps similar to those described above will be briefly described.

1. Step S301 of storing identification information on an object and information on a virtual menu corresponding thereto,

And stores identification information on at least one object in the database and information on a virtual menu corresponding to the identification information.

2. Receiving a real image shot from a camera < S302 >

When a real image of a physical object that the user wants to view is photographed by a camera, the real image photographed from the camera is received.

3. Extracting the first identification information of the first object < S303 >

In step S302, the first identification information of the first object is extracted from the image received through the camera.

4. Requesting the second identification information to the object via the communication network <S304>

The second identification information is requested through the communication network to one or more objects having the same first identification information as the first identification information extracted in step S303. For example, when the mobile communication terminal is paired with the objects A and B and illuminates the object A as shown in FIG. 7A, the second identification information is provided to the objects A and B as shown in (b) Of the flicker information. Here, it is possible that the flicker period of the objects is set in advance.

5. Receiving the second identification information from one or more objects having the same first identification information as the first identification information <S305>

And receives second identification information from one or more objects according to the request of step S304. (c), each object transmits information about a flicker period to the mobile communication terminal, and blinks the LED according to the set period.

6. Recognizing the first object based on the second identification information < S306 >

And recognizes the first object based on the second identification information received in step S305. That is, the mobile communication terminal compares the flicker information received from the objects A and B with the flicker period of the object A recognized through the camera screen.

7. Augmenting the virtual menu on the first object < S307 >

The virtual menu for the first object recognized in step S306 is provided on the electronic device in the form of a UI component (see FIG. 7 (d)).

8. When the virtual menu is selected by the user, the step of transmitting the command controlling the first object < S208 >

When a virtual menu enhanced on the first object is selected by the user, the corresponding function is processed in the first object.

As described above, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera, the mobile communication terminal requests and receives the second identification information from each object , And compares the second identification information with the second identification information of the first object recognized through the screen. If the comparison result is identical, wireless communication is performed for the first object and subsequent operations, A function execution command according to the manipulation of the virtual menu can be issued only to the object that is being manipulated.

Meanwhile, since the second identification information of the objects received by the mobile communication terminal may be the same, the second identification information is composed of a plurality of different second identification information. In step S304, if the received second identification information is the same, the second identification information different from the same second identification information is requested again to one or more objects. For example, in a process in which two objects transmit each second identification information (e.g., a message such as a visual marker + flashing period and a flashing period of one second) to the mobile communication terminal, The mobile communication terminal re-requests each object with the second identification information, and if the object that has been re-requested transmits the second identification information again, the mobile communication terminal receives and processes the second identification information. Here, the second identification information may be defined in advance, and a plurality of pre-defined sets may be known to the mobile communication terminal or the object.

As described above, if the second identification information is composed of a plurality of pieces of information and the second identification information of the objects received by the mobile communication terminal is the same, the second identification information is requested again so that the present invention can be implemented without any problem.

In addition, in the present invention, when each object receives a message requesting re-request of the second identification information from the mobile communication terminal, the step S304 receives the second identification information randomly selected from one or more objects in response to the re-request.

Thus, there is an effect that the probability that the second identification information received from each object is the same is reduced. In this case, when the second identification information is randomly selected from a plurality of objects and transmitted to the mobile communication terminal 20, it is preferable that the pre-defined set is large so as to minimize the probability of selecting the same one.

(Application Example 3 - When using a sensor embedded in an object)

Hereinafter, a method of transmitting a control command only to an object illuminated by a camera without using an additional visual marker will be described.

13 is a flowchart illustrating a method of providing a virtual menu according to another embodiment of the present invention. Steps similar to those described above will be briefly described.

1. Step S401 of storing identification information on an object and information on a corresponding virtual menu,

And stores identification information on at least one object in the database and information on a virtual menu corresponding to the identification information.

2. Step (S402) of receiving the photographed image photographed from the camera

When a real image of a physical object that the user wants to view is photographed by a camera, the real image photographed from the camera is received.

3. Extracting the first identification information of the first object < S403 >

In step S402, the first identification information of the first object is extracted from the image received through the camera. When the user operates the camera of the mobile communication terminal to illuminate the object A, the mobile communication terminal recognizes the first identification information of the object A, and searches for the same object as the object A in the current proximity.

4. Step S404 of outputting an operation command for the recognized object through camera shooting,

And outputs an operation command for the recognized object through shooting of the camera. For example, when an object A is illuminated by a camera as shown in FIG. 9A and an object B having the same first identification information of the object A exists, 'attach object' or 'object' , 'Please listen to the object and release it'. As a result, the user can take an action corresponding to the guidance phrase and lightly touch the object A as shown in (c). When the sensor signal of the object A changes according to the action of the user, the object A transmits information indicating that the mobile communication terminal has shaken. In the present invention, it is preferable that the object includes a sensor capable of detecting vibration such as an acceleration sensor or a gyro sensor so as to easily grasp the shaking of the object.

5. Receiving the second identification information from one or more objects having the same first identification information as the first identification information <S405>

And receives second identification information regarding whether or not the user has operated according to the operation command output in step S404. For example, if the user touches the object A within a predetermined period of time after the guidance message (b) is displayed, the mobile communication terminal 20 transmits information indicating that the object A has been shaken to the mobile communication terminal 20, Lt; / RTI &gt;

6. Recognizing the first object based on the second identification information < S406 >

And recognizes the first object based on the second identification information received in step S405. That is, as shown in (c), the object A outputting the shake information is recognized, and wireless communication is performed for the object A and the function operation.

7. Augmenting the virtual menu on the first object < S407 >

The virtual menu for the first object recognized in step S406 is provided on the electronic device in the form of a UI component (see (d) of FIG. 9).

8. When the virtual menu is selected by the user, step S408 of transmitting a command for controlling the first object

When a virtual menu enhanced on the first object is selected by the user, the corresponding function is processed in the first object.

As described above, even if there is one or more objects having the same first identification information as the first identification information (visual marker) of the first object exposed by the camera, the mobile communication terminal may 'tap the object' or ' Quot ;, &quot; Please listen to the object &quot;, and if the user takes an action according to the announcement phrase, perform wireless communication for function operation with the first object that transmitted the shake information, It is possible to issue a function execution command according to the operation of the virtual menu only to the object desired by the user even in the case of the same electronic apparatus. Particularly, since this method uses the shaking information as the second identification information, it is possible that the mobile communication terminal does not illuminate the object.

On the other hand, in the present invention, when information indicating that the mobile communication terminal has been shaken at a plurality of objects simultaneously is received under the assumption that the same objects exist, a guidance message such as 'tap object' or 'tilt the object slightly' . Even if the object disappears from the image and the object is recognized again after a while even under the above-mentioned premise, the guiding phrase may be displayed.

Meanwhile, the virtual menu providing method according to an exemplary embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The foregoing description is merely illustrative of the technical idea of the present invention and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

100, 200, 300, 400: virtual menu providing system

Claims (15)

delete A database for storing identification information about at least one object and information about a virtual menu corresponding to the identification information;
An identification information extracting unit for extracting first identification information of a first object from an image received through camera shooting;
An identification information transmitting unit for assigning different second identification information to one or more objects having the same first identification information as the extracted first identification information and transmitting the second identification information;
An identification information receiver for receiving the designated second identification information from the one or more objects;
An object recognition unit for recognizing the first object based on the received second identification information and second identification information recognized through shooting of the camera; And
A virtual menu enhancement unit for providing a virtual menu for the recognized first object,
The virtual menu providing system comprising: A database for storing identification information about at least one object and information about a virtual menu corresponding to the identification information;
An identification information extracting unit for extracting first identification information of a first object from an image received through camera shooting;
An identification information request unit for requesting at least one object having first identification information identical to the extracted first identification information to second identification information through a communication network;
An identification information receiver for receiving the second identification information from the one or more objects according to the request;
An object recognition unit for recognizing the first object based on the received second identification information and second identification information recognized through shooting of the camera; And
A virtual menu enhancement unit for providing a virtual menu for the recognized first object,
The virtual menu providing system comprising: The method of claim 3,
Wherein the second identification information comprises a plurality of different second identification information,
Wherein the identification information requesting unit requests the one or more objects to transmit the second identification information different from the second identification information to the one or more objects when the second identification information received from the first object is identical to the second identification information received from the one or more objects Re-requesting
Virtual menu providing system. 5. The method of claim 4,
And the identification information receiving unit receives the second identification information randomly selected from the one or more objects in response to the re-request
Virtual menu providing system. 3. The method of claim 2,
Further comprising an operation command output unit for outputting an operation command for an object recognized through shooting of the camera,
Wherein the identification information receiving unit receives the second identification information as to whether or not the user has operated according to the output operation command
Virtual menu providing system. The method according to claim 6,
Wherein the at least one object includes an acceleration sensor or a gyro sensor
Virtual menu providing system. delete (a) storing identification information about at least one object and information about a virtual menu corresponding to the identification information in a database;
(b) extracting first identification information of the first object from the image received through the photographing of the camera;
(c) assigning different second identification information to one or more objects having the same first identification information as the extracted first identification information, and transmitting
(d) receiving the designated second identification information from the one or more objects;
(e) recognizing the first object based on the received second identification information and second identification information recognized through shooting of the camera; And
(f) providing a virtual menu for the recognized first object
/ RTI &gt; (a) storing identification information about at least one object and information about a virtual menu corresponding to the identification information in a database;
(b) extracting first identification information of the first object from the image received through the photographing of the camera;
(c) requesting second identification information through one or more objects having a first identification information identical to the extracted first identification information through a communication network
(d) receiving the second identification information from the one or more objects according to the request;
(e) recognizing the first object based on the received second identification information and second identification information recognized through shooting of the camera; And
(f) providing a virtual menu for the recognized first object
/ RTI &gt; 11. The method of claim 10,
Wherein the second identification information comprises a plurality of different second identification information,
If the second identification information received from the first object is identical to the second identification information received from the at least one object in the step (d), the second identification information different from the second identification information To re-request
How to provide a virtual menu. 12. The method of claim 11,
In the step (d), in response to the re-request, the second identification information selected randomly from the one or more objects is received
How to provide a virtual menu. 10. The method of claim 9,
Further comprising the step of outputting an operation command for the recognized object through the shooting of the camera,
In the step (d), in response to the output operation command, second identification information on whether or not the user operates is received
How to provide a virtual menu. 14. The method of claim 13,
Wherein the at least one object includes an acceleration sensor or a gyro sensor
How to provide a virtual menu. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 9 to 14.

Images