KR102636996B1 - Method for providing a virtual menu based on hand recognition for ar glasses apparatus and ar glasses system for providing a virtual menu based on hand recognition - Google Patents

Abstract

The present invention relates to a method of providing a virtual menu based on hand recognition for an augmented reality glasses device, and more specifically, to a method of providing a virtual menu in which each step is performed in an augmented reality glasses system, including: (1) a virtual menu located within the wearer's field of view; recognizing the wearer's palm; (2) identifying the direction of the palm recognized in step (1); and (3) displaying and providing a virtual menu determined based on the direction of the identified palm on the recognized palm.
In addition, the present invention relates to an augmented reality glasses system that provides a virtual menu based on hand recognition, and more specifically, to an augmented reality glasses device, an optical display unit disposed in front of both eyes of the wearer and providing augmented reality to the wearer. ; A recognition unit that recognizes the wearer's palm located within the wearer's field of view; An identification unit that identifies the direction of the palm recognized by the recognition unit; and a menu providing unit that displays and provides a virtual menu determined based on the direction of the identified palm on the recognized palm.
According to the method of providing a virtual menu based on hand recognition for an augmented reality glasses device proposed by the present invention and the augmented reality glasses system that provides a virtual menu based on hand recognition, the wearer's palm is recognized and the information determined based on the palm direction is provided. By displaying a virtual menu on the palm, quick menus and full menus are selectively displayed depending on the direction of the palm, making it possible to easily provide the wearer with the desired menu using only the direction of the palm. By entering the input signal for the virtual menu with the opposite finger, The wearer can control the virtual menu intuitively and conveniently.

Description

Method for providing a virtual menu based on hand recognition for an augmented reality glasses device and an augmented reality glasses system providing a virtual menu based on hand recognition VIRTUAL MENU BASED ON HAND RECOGNITION}

The present invention relates to a method of providing a virtual menu for an augmented reality glasses device and an augmented reality glasses system that provides a virtual menu. More specifically, a method of providing a virtual menu based on hand recognition for an augmented reality glasses device and a method of providing a virtual menu based on hand recognition for an augmented reality glasses device. This is about an augmented reality glass system that provides a virtual menu.

In accordance with the trend toward lighter and smaller digital devices, various wearable devices are being developed. A head mounted display, a type of wearable device, refers to a variety of devices that a user can wear on their head to receive multimedia content. Here, the head mounted display (HMD) is worn on the user's body and provides images to the user in various environments as the user moves. These head-mounted displays (HMDs) are divided into see-through and see-closed types. The see-through type is mainly used for Augmented Reality (AR), and the closed type is mainly used for virtual reality (Virtual Reality). It is used for Reality (VR).

Because it is difficult to use input methods like existing touch screens for head-mounted displays, an optimized input method for user interaction is needed. Input methods that can be used in the augmented reality of a head-mounted display may include buttons provided on the HMD, separate input devices connected to the HMD, gesture recognition, and voice recognition.

Head-mounted displays for augmented reality (hereinafter referred to as ‘AR glasses’ or ‘augmented reality glasses’) display menus similar to smartphones and smart watches, and select icons on the menu to access various options. You can run several functions. However, unlike VR, AR glasses do not have control aids and are mainly controlled based on gesture recognition, so controlling and manipulating menus using hand gestures is one of the most important tasks in the use of AR glasses.

Therefore, in AR glasses that use a gesture-oriented input method, there is a need to develop a virtual menu and operation method that can be used intuitively and minimize concerns about operation malfunction.

Meanwhile, as prior art related to the present invention, Registered Patent No. 10-1700569 (title of the invention: HMD device capable of gesture-based user authentication and gesture-based user authentication method of the HMD device, registration date: January 2017 23), etc. have been launched.

The present invention was proposed to solve the above-mentioned problems of previously proposed methods. It recognizes the wearer's palm and displays a virtual menu determined based on the palm direction on the palm, thereby providing a quick menu and menu according to the palm direction. Since the entire menu is displayed selectively, the desired menu can be easily provided to the wearer just by pointing the palm, and by inputting input signals for the virtual menu with the opposite finger, the wearer can intuitively and conveniently control the virtual menu. The purpose is to provide a method of providing a hand recognition-based virtual menu for a glass device and an augmented reality glasses system that provides a hand recognition-based virtual menu.

A method of providing a virtual menu based on hand recognition for an augmented reality glass device according to the characteristics of the present invention to achieve the above object is,

A method of providing a virtual menu in which each step is performed in an augmented reality glass system,

(1) recognizing the wearer's palm located within the wearer's field of view;

(2) identifying the direction of the palm recognized in step (1); and

(3) It is characterized in that it includes the step of displaying and providing a virtual menu determined based on the direction of the identified palm on the recognized palm.

Preferably, in step (2),

It is possible to identify whether the direction of the recognized palm is horizontal or vertical.

More preferably, in step (3),

If the direction of the palm identified in step (2) is horizontal, the quick virtual menu is displayed, and the position of the palm is tracked and the quick virtual menu is moved according to the movement of the palm, so that the quick virtual menu is displayed on the palm. can do.

Even more preferably, after step (3),

If the palm is out of the wearer's field of view, the quick virtual menu may not be displayed.

More preferably, in step (3),

If the direction of the palm identified in step (2) is vertical, the entire virtual menu is displayed based on the palm, and after displaying the entire virtual menu, the entire virtual menu is displayed within the wearer's field of view regardless of the change in the position of the palm. The position of the virtual menu can be fixed and displayed.

Preferably, after step (3),

(4) further comprising controlling the virtual menu using an input signal for the displayed virtual menu,

The input signal may be a signal using the fingers of the hand opposite to the palm recognized in step (1).

The augmented reality glasses system that provides a virtual menu based on hand recognition to achieve the above purpose is,

As an augmented reality glass system,

An optical display unit disposed in front of both eyes of the wearer to provide augmented reality to the wearer;

A recognition unit that recognizes the wearer's palm located within the wearer's field of view;

An identification unit that identifies the direction of the palm recognized by the recognition unit; and

Its structural feature includes a menu providing unit that displays and provides a virtual menu determined based on the direction of the identified palm on the recognized palm.

Preferably, the identification unit,

Identify whether the recognized palm orientation is horizontal or vertical,

The menu provision section,

If the direction of the palm identified by the identification unit is horizontal, a quick virtual menu is displayed, and the position of the palm is tracked and the quick virtual menu is moved according to the movement of the palm, so that the quick virtual menu is displayed on the palm, If the direction of the palm identified by the identification unit is vertical, the entire virtual menu is displayed based on the palm, and after displaying the entire virtual menu, the entire virtual menu is displayed within the wearer's field of view regardless of the change in the position of the palm. The position of can be fixed and displayed.

According to the method of providing a hand recognition-based virtual menu for an augmented reality glasses device proposed by the present invention and the augmented reality glasses system that provides a hand recognition-based virtual menu, the wearer's palm is recognized and determined based on the palm direction. By displaying a virtual menu on the palm, quick menus and full menus are selectively displayed depending on the direction of the palm, making it possible to easily provide the wearer with the desired menu using only the direction of the palm. By entering the input signal for the virtual menu with the opposite finger, The wearer can control the virtual menu intuitively and conveniently.

Figure 1 is a diagram illustrating the configuration of an augmented reality glasses system that provides a virtual menu based on hand recognition according to an embodiment of the present invention.
Figure 2 is a diagram showing an example of an augmented reality glass device that provides a virtual menu based on hand recognition according to an embodiment of the present invention.
Figure 3 is a diagram illustrating the flow of a method for providing a virtual menu based on hand recognition for an augmented reality glasses device according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a method of providing a virtual menu based on hand recognition for an augmented reality glass device according to an embodiment of the present invention, where a quick virtual menu is provided when the palm is in a vertical direction.
Figure 5 is a diagram illustrating how a virtual menu is quickly moved and displayed according to the movement of the palm in a method of providing a virtual menu based on hand recognition for an augmented reality glasses device according to an embodiment of the present invention.
Figure 6 is a diagram illustrating an entire virtual menu being provided when the palm is in a horizontal direction in a method of providing a virtual menu based on hand recognition for an augmented reality glasses device according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating an entire virtual menu being provided regardless of palm movement in a method of providing a virtual menu based on hand recognition for an augmented reality glass device according to an embodiment of the present invention.
Figure 8 is a diagram illustrating inputting an input signal for a virtual menu in a method of providing a virtual menu based on hand recognition for an augmented reality glasses device according to an embodiment of the present invention.

Hereinafter, with reference to the attached drawings, preferred embodiments will be described in detail so that those skilled in the art can easily practice the present invention. However, when describing preferred embodiments of the present invention in detail, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the same symbols are used throughout the drawings for parts that perform similar functions and actions.

In addition, throughout the specification, when a part is said to be 'connected' to another part, this is not only the case when it is 'directly connected', but also when it is 'indirectly connected' with another element in between. Includes. Additionally, ‘including’ a certain component does not mean excluding other components, but rather including other components, unless specifically stated to the contrary.

Figure 1 is a diagram showing the configuration of an augmented reality glasses system that provides a virtual menu based on hand recognition according to an embodiment of the present invention. As shown in FIG. 1, the augmented reality glass system that provides a virtual menu based on hand recognition according to an embodiment of the present invention includes an optical display unit (optical display unit) disposed in front of both eyes of the wearer and providing augmented reality to the wearer. 100); A recognition unit 200 that recognizes the wearer's palm located within the wearer's field of view; An identification unit 300 that identifies the direction of the palm recognized by the recognition unit 200; and a menu providing unit 400 that displays and provides a virtual menu determined based on the direction of the identified palm on the recognized palm, and includes a control unit 500, a camera unit 600, and a sensor unit 700. ) and a communication unit 800.

More specifically, the identification unit 300 identifies whether the direction of the recognized palm is horizontal or vertical, and the menu providing unit 400 provides a quick virtual menu if the direction of the palm identified by the identification unit 300 is horizontal. , If the direction of the palm is vertical, the entire virtual menu can be displayed based on the palm. In other words, the quick menu and the full menu are selectively displayed as images on the palm of the wearer's palm according to the direction of the wearer's palm, making it possible to easily provide the wearer with the desired menu only by the direction of the palm.

Figure 2 is a diagram showing an example of an augmented reality glass device 10 that provides a virtual menu based on hand recognition according to an embodiment of the present invention. As shown in FIG. 2, the augmented reality glasses device 10 that provides a virtual menu based on hand recognition according to an embodiment of the present invention includes goggles with closed areas around the eyes or glasses with open sides. It can be implemented as follows.

The augmented reality glasses system that provides a virtual menu based on hand recognition according to an embodiment of the present invention may be composed of the augmented reality glasses device 10 and a server, but may also be implemented solely with the augmented reality glasses device 10. . For example, the optical display unit 100 and the camera unit 600 are components of the augmented reality glass device 10, and the recognition unit 200 and identification unit 300 are implemented as components of the server. You can. That is, the image captured by the camera unit 600 of the augmented reality glasses device 10 is transmitted to the server, and the recognition unit 200 and identification unit 300 of the server recognize the palm based on the artificial intelligence model, calculate and After analyzing and generating a control event, a virtual menu according to the control event can be provided to the wearer in augmented reality through the optical display unit 100 of the augmented reality glass device 10. As such, in the augmented reality glasses system that provides a virtual menu based on hand recognition according to an embodiment of the present invention, each function handled by the augmented reality glasses device 10 and the server can be implemented in various ways.

Below, each configuration of the augmented reality glasses system that provides a virtual menu based on hand recognition according to an embodiment of the present invention will be described in detail.

Here, the optical display unit 100 is disposed in front of both eyes of the wearer and can provide augmented reality to the wearer. That is, the optical display unit 100 is a component corresponding to the AR glasses of the augmented reality glass device 10, and may be configured to include a display 110 and an optical system 120 as shown in FIG. 1.

The display 110 may output image light so that image information can be provided to the wearer. More specifically, the display 110 is coupled to the optical system 120, which will be described in detail below, so that image information can be provided to the wearer, and outputs image light transmitted by the optical system 120 toward the wearer's eyes. and may be composed of a pair of displays 110 for binocular display. The display 110 may be configured in various ways, such as OLED.

The optical system 120 is disposed in front of both eyes of the wearer wearing the augmented reality glass device 10 and can provide augmented reality by combining real world light and image light. More specifically, the optical system 120 transmits at least a portion of the light of the real world through the wearer's field of view and transmits the image light output from the display 110 toward the wearer's eyes to provide augmented reality. can do. That is, the optical system 120 may be configured so that a wearer wearing the augmented reality glasses device 10 can experience augmented reality.

In addition, the optical system 120 is composed of a plurality of lenses and mirrors and can be implemented in various ways, for example, an optical diffraction method, a beam splitter method, a pin mirror method, etc.

The recognition unit 200, identification unit 300, menu provision unit 400, and control unit 500 will be described below as a hand recognition-based virtual device for the augmented reality glass device 10 according to an embodiment of the present invention. We will explain in detail how to provide the menu.

FIG. 3 is a diagram illustrating the flow of a method for providing a virtual menu based on hand recognition for the augmented reality glass device 10 according to an embodiment of the present invention. As shown in FIG. 3, the hand recognition-based virtual menu providing method for the augmented reality glasses device 10 according to an embodiment of the present invention is a virtual menu providing method in which each step is performed in the augmented reality glasses system. , a step of recognizing the wearer's palm (S110), a step of identifying the direction of the palm (S120), and a step of displaying and providing a virtual menu determined based on the direction of the palm on the recognized palm (S130). It can be implemented by further including controlling the virtual menu using an input signal for the virtual menu (S140).

In step S110, the recognition unit 200 may recognize the wearer's palm located within the wearer's field of view. More specifically, the recognition unit 200 may recognize the palm from the wearer's gaze direction image obtained from the camera unit 600. For example, when the wearer looks at the hand, the recognition unit 200 can identify the palm by recognizing the shape of the hand in the camera image. At this time, the object recognized in step S110 may be the wearer's left palm, and the shape of the left hand open and the palm visible can be recognized.

Here, the recognition unit 200 can estimate the hand shape or feature points using vision recognition technology based on image processing, artificial intelligence, etc., and can recognize palms or gestures using the estimated hand shape or feature points. You can. Depending on the embodiment, the recognition unit 200 may use a hand image acquired from an RGB camera, an infrared image camera, a depth sensor, etc. of the augmented reality glass device 10.

More specifically, the recognition unit 200 may use an infrared hand image obtained from an infrared image camera of the augmented reality glasses device 10. Infrared images are single-channel images, and gesture recognition using infrared hand images can reduce the computational cost used in image processing, so gesture recognition can be effectively performed even in the embedded environment of the augmented reality glass device 10. In particular, if the augmented reality glass device 10 is heavy, it puts a burden on the neck and shoulders and head movement is restricted, making it difficult to wear for a long time, so it is important to make it lightweight and compact. Therefore, since many computational resources cannot be mounted on the augmented reality glass device 10, if the computational cost is reduced so that it can be used in an embedded environment by using an infrared image, which is a single-channel image, the calculation in step S110 is performed not on the server, but on the augmented reality glass device ( 10) can also be processed.

In step S120, the identification unit 300 may identify the direction of the palm recognized in step S110. More specifically, in step S120, it is possible to identify whether the direction of the recognized palm is horizontal or vertical. Here, the identification unit 300 calculates the angle of the palm based on the hand shape or feature points estimated by the recognition unit 200, and determines whether the angle of the palm falls within a predetermined horizontal angle range or vertical angle range. The orientation of the palm can be identified as horizontal or vertical.

In step S130, the menu providing unit 400 may display and provide a virtual menu determined based on the direction of the palm identified by the identification unit 300 on the recognized palm. In other words, different virtual menus can be provided when the palm orientation identified in step S120 is horizontal and vertical.

Figure 4 shows that in the method of providing a virtual menu 20 based on hand recognition for the augmented reality glass device 10 according to an embodiment of the present invention, a quick virtual menu 20 is provided when the palm is in a vertical direction. This is a drawing showing. As shown in FIG. 4, in step S130 of the method for providing a virtual menu 20 based on hand recognition for the augmented reality glasses device 10 according to an embodiment of the present invention, the direction of the palm identified in step S120 is If it is horizontal, a quick virtual menu (20) can be displayed. That is, as shown on the left side of FIG. 4, when the wearer wears the augmented reality glass device 10 and looks at the palm with the left hand spread out vertically, the palm is recognized, the direction of the palm is identified, and the palm direction ( The quick virtual menu 20 determined according to the length may be displayed as a pop-up on the left palm, as shown on the right side of FIG. 4.

Here, the quick virtual menu 20 may be a basic function menu or a favorite menu. For example, it may be a menu composed of frequently used functions and/or applications such as phone, web browser, album, camera, etc. The type, order, and arrangement of functions and/or applications that make up the quick virtual menu 20 can be directly edited by the wearer. Therefore, each wearer can quickly customize and use the virtual menu 20.

FIG. 5 illustrates a method of providing a virtual menu 20 based on hand recognition for an augmented reality glass device 10 according to an embodiment of the present invention, where a quick virtual menu 20 is moved and displayed according to the movement of the palm. It is a drawing. As shown in FIG. 5, in step S130 of the method for providing a virtual menu 20 based on hand recognition for the augmented reality glass device 10 according to an embodiment of the present invention, the position of the palm is tracked and the palm is moved. Accordingly, the menu providing unit 400 moves the quick virtual menu 20 so that the quick virtual menu 20 is displayed on the palm. That is, the augmented reality glass device 10 tracks the position of the palm on which the quick virtual menu 20 is displayed as a pop-up, and when the palm moves, the quick virtual menu 20 can move together as if it were fixed on the palm.

Additionally, after displaying the quick virtual menu 20 in step S130, if the palm leaves the wearer's field of view, the quick virtual menu 20 may not be displayed. In other words, when the left palm leaves the field of view, the quick virtual menu 20 also disappears.

Figure 6 shows that in the method of providing a virtual menu 20 based on hand recognition for the augmented reality glass device 10 according to an embodiment of the present invention, the entire virtual menu 20 is provided when the palm is in a horizontal direction. This is a drawing showing. As shown in FIG. 6, in step S130 of the method for providing a virtual menu 20 based on hand recognition for the augmented reality glasses device 10 according to an embodiment of the present invention, the direction of the palm identified in step S120 is If it is vertical, the entire virtual menu (20) can be displayed based on the palm. That is, as shown on the left side of FIG. 6, when the wearer wears the augmented reality glass device 10 and looks at the palm with the left hand spread horizontally, the palm is recognized, the direction of the palm is identified, and the palm direction ( The entire virtual menu 20 determined according to the width (horizontal) may be displayed as a pop-up on the left palm, as shown on the right side of FIG. 6.

Here, the entire virtual menu 20 may be a list menu of all functions and applications provided by the augmented reality glasses device 10 or installed on the augmented reality glasses device 10. The type, order, and arrangement of functions and/or applications that make up the entire virtual menu 20 can be directly edited by the wearer. Therefore, each wearer can customize and use the entire virtual menu 20 in the desired order and arrangement.

Since this full virtual menu 20 contains more icons than the quick virtual menu 20, it can be output on the entire screen of the view field, as shown on the right side of FIG. 6. At this time, the entire virtual menu 20 can be displayed based on the position of the identified left palm. For example, the left palm is positioned at the center of the entire virtual menu 20, or the left palm is positioned at the center of the entire virtual menu 20, or the left palm is positioned at the center of the entire virtual menu 20. As shown, the left edge of the entire virtual menu 20 can be displayed aligned with the beginning of the left palm.

FIG. 7 shows how the entire virtual menu 20 is provided regardless of palm movement in a method of providing a virtual menu 20 based on hand recognition for the augmented reality glass device 10 according to an embodiment of the present invention. It is a drawing. As shown in FIG. 7, in step S130 of the method for providing a virtual menu 20 based on hand recognition for the augmented reality glass device 10 according to an embodiment of the present invention, after displaying the entire virtual menu 20 Can display the entire virtual menu 20 by fixing its position within the wearer's field of view, regardless of changes in the position of the palm. That is, unlike the quick virtual menu 20, the entire virtual menu 20 can be provided fixedly within the field of view regardless of whether the wearer moves his or her left hand or head.

Additionally, in step S130, even if the palm leaves the wearer's field of view, the entire virtual menu 20 can be maintained within the wearer's field of view. That is, unlike the quick virtual menu 20, the entire virtual menu 20 can be displayed as is even if the left palm is out of sight. In order to remove the entire virtual menu 20 from view, a separate input signal for termination may be required.

In step S140, the menu providing unit 400 and the control unit 500 may control the virtual menu 20 using an input signal for the virtual menu 20. The control unit 500 can execute a selected function or application in response to an input signal, and can edit the virtual menu 20 or terminate the output of the virtual menu 20 in conjunction with the menu providing unit 400.

FIG. 8 is a diagram illustrating inputting an input signal for the virtual menu 20 in a method of providing a virtual menu 20 based on hand recognition for the augmented reality glass device 10 according to an embodiment of the present invention. am. As shown in FIG. 8, in step S140 of the method for providing a virtual menu 20 based on hand recognition for the augmented reality glasses device 10 according to an embodiment of the present invention, the hand opposite to the palm recognized in step S110 The virtual menu 20 can be controlled by using a signal using the finger as an input signal. In other words, the wearer can execute the corresponding function by selecting the icon with the index finger of the right hand in the virtual menu 20 (quick virtual menu 20 or full virtual menu 20) displayed on the upper left hand, change the icon placement, etc. You can also perform menu editing.

Meanwhile, the recognition unit 200 can control the virtual menu 20 by recognizing various gestures used by the wearer to control the menu, extracting input signals, and transmitting them to the control unit 500. Here, each gesture may consist of at least one of the appearance (posture) and movement of the hand.

For example, the recognition unit 200 detects the shape of the right hand (the shape of only the index finger extended, as shown in FIG. 8) from the image captured by the camera, estimates the key point of the hand, and extracts the coordinate value of the end point of the index finger. Thus, among the icons constituting the virtual menu 20, the icon pointed to by the wearer can be specified. Additionally, it is possible to determine whether or not the corresponding icon is touched by estimating the depth of the tip of the index finger.

Meanwhile, the augmented reality glass device 10 that provides a virtual menu 20 based on hand recognition according to an embodiment of the present invention further includes a camera unit 600, a sensor unit 700, and a communication unit 800. It can be configured as follows.

That is, the control unit 500 generates image information to be provided to the wearer based on the information collected by the camera unit 600 and the sensor unit 700 and controls it to be transmitted to the display 110, allowing the wearer to experience augmented reality. Through this, the virtual menu 20 and additional information about the external environment can be delivered on an optimized screen.

The camera unit 600 can capture an image in the wearer's viewing direction and transmit it to the recognition unit 200 to recognize palms, gestures, etc. based on vision recognition. More specifically, the camera unit 600 can provide hand images necessary for palm and gesture recognition, including an RGB camera, an infrared image camera, and a depth sensor, according to the palm and gesture recognition algorithm of the recognition unit 200. there is.

The sensor unit 700 may include at least one sensor, and more specifically, may include a pupil tracking sensor, an iris recognition sensor, etc. According to the embodiment, the wearer's gaze is tracked using a pupil tracking sensor, and the recognition unit 200 is configured to recognize the wearer's palm when the wearer looks at the palm, so that the virtual menu (20) is displayed in situations where the wearer does not intend. ) can be prevented from being displayed. At this time, malfunctions can be minimized by allowing step S110 to be performed when the palm is looked at for more than a predetermined time.

Meanwhile, when the wearer wears the augmented reality glass device 10 and performs user authentication, the control unit 500 displays the virtual menu 20 customized by each wearer, the individual wearer's motion gesture usage pattern, preferred illuminance, and IPD (Inter- Various user-specific settings, such as Pupillary Distance (Pupillary Distance) and interface, can be automatically applied. At this time, user authentication may use an iris recognition sensor included in the sensor unit 700.

The communication unit 800 is installed on one side of the augmented reality glasses device 10 and can transmit and receive various signals and data to other augmented reality glasses devices 10 or servers. Here, the network used by the communication unit 800 is a wired network or a mobile network such as a Local Area Network (LAN), Wide Area Network (WAN), or Value Added Network (VAN). radio communication network), satellite communication network, Bluetooth, Wibro (Wireless Broadband Internet), HSDPA (High Speed Downlink Packet Access), LTE (Long Term Evolution), 3/4/5/6G (3/4/5/ It can be implemented with all types of wireless networks such as 6th Generation Mobile Telecommunication).

As described above, according to the method of providing a hand recognition-based virtual menu 20 for the augmented reality glasses device 10 and the augmented reality glasses system providing the hand recognition-based virtual menu 20 proposed in the present invention. , By recognizing the wearer's palm and displaying a virtual menu (20) determined based on the palm direction on the palm, quick menus and full menus are selectively displayed depending on the palm direction, so the desired menu is easily provided to the wearer only by the palm direction. The wearer can intuitively and conveniently control the virtual menu 20 by inputting an input signal for the virtual menu 20 with the opposite finger.

Meanwhile, the present invention may include a computer-readable medium containing program instructions for performing operations implemented in various communication terminals. For example, computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD_ROM and DVD, and floptical disks. It may include magneto-optical media and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc.

Such computer-readable media may include program instructions, data files, data structures, etc., singly or in combination. At this time, program instructions recorded on a computer-readable medium may be specially designed and configured to implement the present invention, or may be known and available to those skilled in the computer software art. For example, it may include not only machine language code such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

The present invention described above can be modified or applied in various ways by those skilled in the art, and the scope of the technical idea according to the present invention should be determined by the claims below.

10: Augmented reality glass device
20: Virtual menu
100: Optical display unit
110: display
120: Optical system
200: Recognition unit
300: identification unit
400: Menu provision unit
500: Control unit
600: Camera unit
700: Sensor unit
800: Department of Communications
S110: Step of recognizing the wearer's palm
S120: Identifying the direction of the palm
S130: Step of displaying and providing a virtual menu determined based on the direction of the palm on the recognized palm.
S140: Step of controlling the virtual menu using an input signal to the virtual menu

Claims (8)

A method of providing a virtual menu (20) in which each step is performed in an augmented reality glass system,
(1) recognizing the wearer's palm located within the wearer's field of view from the wearer's gaze direction image obtained by the camera unit 600;
(2) identifying whether the orientation of the palm recognized in step (1) is horizontal or vertical; and
(3) A virtual menu 20 determined based on the direction of the identified palm is displayed and provided on the recognized palm. If the palm is horizontal, a quick virtual menu 20 is provided, and if the palm is vertical, the virtual menu 20 is provided. It includes the step of providing the entire virtual menu (20) by displaying it as an image on the palm of each hand,
In step (3),
If the direction of the palm identified in step (2) is horizontal, the quick virtual menu 20 is displayed, and the position of the palm is tracked and the quick virtual menu 20 is moved according to the movement of the palm. Make the menu (20) appear in the palm of your hand,
After step (3) above,
If the palm leaves the wearer's field of view, the quick virtual menu 20 is not displayed,
In step (3),
If the direction of the palm identified in step (2) is vertical, the entire virtual menu 20 is displayed based on the palm, and after displaying the entire virtual menu 20, the wearer is An augmented reality glass device, characterized in that the position of the entire virtual menu 20 is fixed and displayed within the field of view of the wearer, and the entire virtual menu 20 is displayed as is even if the palm leaves the wearer's field of view. Method for providing a virtual menu (20) based on hand recognition for 10).
delete delete delete delete The method of claim 1, wherein after step (3),
(4) further comprising controlling the virtual menu (20) using an input signal for the displayed virtual menu (20),
A method of providing a virtual menu (20) based on hand recognition for an augmented reality glass device (10), wherein the input signal is a signal using a finger of the hand opposite to the palm recognized in step (1).
As an augmented reality glass system,
An optical display unit 100 disposed in front of both eyes of the wearer to provide augmented reality to the wearer;
A recognition unit 200 that recognizes the wearer's palm located within the wearer's field of view from the wearer's gaze direction image obtained by the camera unit 600;
An identification unit 300 that identifies whether the direction of the palm recognized by the recognition unit 200 is horizontal or vertical; and
A virtual menu 20 determined based on the direction of the identified palm is displayed and provided on the recognized palm. If the palm is horizontal, a quick virtual menu 20 is displayed, and if the palm is vertical, the virtual menu 20 is displayed based on the palm. It includes a menu providing unit 400 that displays the entire virtual menu 20 as an image on each palm,
The menu providing unit 400,
If the direction of the palm identified by the identification unit 300 is horizontal, the quick virtual menu 20 is displayed, and the position of the palm is tracked and the quick virtual menu 20 is moved according to the movement of the palm. The virtual menu 20 is displayed on the palm, and when the palm leaves the wearer's field of view, the quick virtual menu 20 is not displayed,
If the direction of the palm identified by the identification unit 300 is vertical, the entire virtual menu 20 is displayed based on the palm, and after the entire virtual menu 20 is displayed, the entire virtual menu 20 is displayed regardless of the change in the position of the palm. A hand recognition-based device, characterized in that the position of the entire virtual menu (20) is fixed and displayed within the wearer's field of view, and the entire virtual menu (20) is displayed as is even if the palm deviates from the wearer's field of view. Augmented reality glass system that provides a virtual menu (20). delete

Images