WO2017164584A1

WO2017164584A1 - Hmd device capable of gesture-based user authentication and gesture-based user authentication method for hmd device

Info

Publication number: WO2017164584A1
Application number: PCT/KR2017/002930
Authority: WO
Inventors: 박지만; 예정민
Original assignee: 주식회사 다날
Priority date: 2016-03-23
Filing date: 2017-03-20
Publication date: 2017-09-28
Also published as: KR101700569B1

Abstract

The present invention relates to a technology wherein: predetermined pieces of first figure information are stored in order to perform user authentication; a camera capable of photographing an external environment is mounted on the front side of an HMD device, so as to photograph the external environment and a movement of a user's hand through the camera; object tracking for a movement of the user's hand is performed from an image photographed through the camera, so as to generate figure information corresponding to a movement trajectory of the user's hand; the figure information corresponding to the movement trajectory of the user's hand is compared with the predetermined pieces of first figure information; and then the user authentication can be completed when it is determined that the figure information matches the predetermined pieces of first figure information as a result of the comparison, whereby a user can easily perform an authentication process even in an HMD device which does not have a separate input unit.

Description

HMD device capable of gesture based user authentication and gesture based user authentication method of the HMD device

The present invention relates to a technology that enables a user to easily perform user authentication in a head mounted display (HMD) device capable of implementing virtual reality (VR).

Recently, with the widespread use of the Internet, the use of services such as purchase of goods online and banking through online is increasing rapidly.

In such online-based product purchase and banking services, user authentication must be included in payment or remittance, and there are a number of security technologies for authenticating users in such an online way.

Recently, as the spread of mobile devices such as smartphones and tablet PCs increases, a number of user authentication technologies that can be utilized in these mobile devices are also emerging.

As interest in Virtual Reality (VR), which is rapidly emerging as one of the future technologies, is increasing, many VR-related devices and software that can utilize such VR in games and the like are appearing.

A representative device capable of implementing VR is a head mounted display (HMD) device, which is a device that allows a user to see a game screen through a display in front of his eyes after wearing it on his head like glasses.

As the HMD device is widely spread, it is expected that in the near future, the user will develop various forms such as purchasing game items or performing online shopping while wearing the HMD device.

However, since the distribution of HMD devices has not been actively made yet, when a user purchases an item online while wearing the HMD device, an appropriate technology for performing user authentication has not been introduced. .

There are various methods of authenticating a user using a desktop PC or a mobile device, such as a password input method and a touch gesture pattern recognition method. However, these user authentication technologies use a predetermined input device to obtain a password from a user. In terms of an input receiving method, it is difficult to introduce such a user authentication scheme in an HMD device that does not provide a separate input device.

Therefore, when a user authentication is required in the HMD device, such as purchase of goods or money transfer, a study on a new type of user authentication technology that supports proper user authentication in consideration of the characteristics of the HMD device is required.

The HMD (Head Mounted Display) device capable of gesture-based user authentication and the gesture-based user authentication method of the HMD device according to the present invention store predetermined first figure information previously designated for user authentication and then store the information on the front of the HMD device. By mounting a camera capable of shooting the external environment, the camera captures the movement of the user's hand and the external environment through the camera, and then performs object tracking on the movement of the user's hand from the captured image captured by the camera. After generating the shape information corresponding to the movement trajectory of the user's hand, the figure information corresponding to the movement trajectory of the user's hand and the predetermined first figure information are compared with each other so that both figure information is the same. If it is determined that the user can complete authentication, the HMD device does not have a separate input device. To support the user's authentication process easily.

Gesture-based user authentication capable HMD (Head Mounted Display) device according to an embodiment of the present invention is attached to the front of the HMD device, a camera capable of shooting in the external environment, n (n is a predetermined for the user authentication) If a user authentication command is applied under the control of an application installed in the HMD device and the figure information storage unit storing the number of natural figures), the camera is driven to obtain a captured image of the external environment. As the user moves his / her hand, when the external environment and the user's hand are photographed together through the camera, object tracking of the user's hand movement is performed from the photographed image captured by the camera. A figure information generation unit for generating figure information corresponding to the movement trajectory of the user's hand and the teeth of the user's hand When n pieces of figure information corresponding to a locus are generated, when n figure information corresponding to the movement trajectory of the user's hand is compared with the n first figure information, it is determined that both figure information correspond to each other. It includes a user authentication unit for completing the user authentication.

In addition, a gesture-based user authentication method of an HMD device having a camera attached to a front surface according to an embodiment of the present invention stores n-type first figure information in which n (n is a natural number) previously designated for user authentication is stored. If the user authentication command is applied by the control of the application installed in the HMD device, maintaining the wealth, the camera is driven to obtain a captured image of the external environment, and as the user moves his hand, When the external environment and the user's hand are photographed together, object tracking of the movement of the user's hand is performed from the captured image captured by the camera to generate figure information corresponding to the movement trajectory of the user's hand. And n pieces of figure information corresponding to the movement trajectory of the user's hand, are generated. When compared to the generated corresponding to the movement path of the hand of n shape information and said n number of first shape information determined to match, the amount of graphic information, and a step to complete the user authentication.

The HMD (Head Mounted Display) device capable of gesture-based user authentication and the gesture-based user authentication method of the HMD device according to the present invention store predetermined first figure information previously designated for user authentication and then store the information on the front of the HMD device. By mounting a camera capable of shooting the external environment, the camera captures the movement of the user's hand and the external environment through the camera, and then performs object tracking on the movement of the user's hand from the captured image captured by the camera. After generating the shape information corresponding to the movement trajectory of the user's hand, the figure information corresponding to the movement trajectory of the user's hand and the predetermined first figure information are compared with each other so that both figure information is the same. If it is determined that the user can complete authentication, the HMD device does not have a separate input device. Can also assist the user in the authentication process.

1 is a diagram illustrating a structure of a head mounted display (HMD) device capable of gesture-based user authentication according to an embodiment of the present invention.

2 and 3 are views for explaining the operation of the HMD device capable of gesture-based user authentication according to an embodiment of the present invention.

4 is a flowchart illustrating a gesture-based user authentication method of an HMD device according to an embodiment of the present invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the drawings, similar reference numerals are used for similar elements.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, the HMD device 110 capable of gesture-based user authentication according to an embodiment of the present invention may include a camera 111, a figure information storage unit 112, a figure information generation unit 113, and a user authentication. Part 114 is included.

The camera 111 is attached to the front of the HMD device 110, it is configured to allow the shooting of the external environment.

The figure information storage unit 112 stores n pieces of first figure information (n is a natural number) previously designated for user authentication.

When the user authentication command is applied under the control of an application installed in the HMD device 110, the figure information generator 113 drives the camera 111 to obtain a captured image of the external environment, and the user makes a hand. In response to the movement, when the external environment and the user's hand are photographed together through the camera 111, the user tracks the movement of the user's hand from the captured image captured by the camera 111 to perform the user's hand. Shape information corresponding to the movement trajectory of the hand is generated.

When n pieces of figure information corresponding to the movement trajectory of the user's hand are generated, the user authentication unit 114 may generate n figure information and the n first figure information generated corresponding to the movement trajectory of the user's hand. In comparison, when it is determined that both figures of information match, the user authentication is completed.

In this case, according to an embodiment of the present invention, the HMD device 110 capable of gesture-based user authentication may further include an image display control unit 115 and a gesture pattern recognition area display unit 116.

The image display controller 115 displays a captured image of the external environment captured by the camera 111 in real time on the display.

The gesture pattern recognition area display unit 116 displays a plurality of predetermined gesture pattern recognition areas at a predetermined point on the display.

In this case, the figure information generating unit 113 may include a figure information table holding unit 117, a gesture pattern recognition region extracting unit 118, and a figure information extracting unit 119.

The figure information table holding unit 117 stores and maintains a figure information table in which different figure information predetermined in advance with respect to combinations of a plurality of different gesture pattern recognition areas are recorded.

The gesture pattern recognition area extractor 118 performs object tracking on the movement of the user's hand from the captured image photographed by the camera 111 and photographs through the camera among the selected gesture pattern recognition areas. At least one gesture pattern recognition region overlapping the point where the user's hand is located is extracted from the captured image.

When the at least one gesture pattern recognition region is extracted, the figure information extracting unit 119 may combine the at least one gesture pattern recognition region and correspond to the combination of the at least one gesture pattern recognition region from the figure information table. The recorded figure information is extracted to generate the extracted figure information as figure information corresponding to a movement trajectory of the user's hand.

In this case, according to an embodiment of the present invention, different index values may be pre-assigned to the plurality of predetermined gesture pattern recognition regions.

In this case, the predetermined different figure information may be recorded in the figure information table corresponding to combinations of index values for a plurality of different gesture pattern recognition areas, and the figure information extracting unit 119 may be configured to include at least the figure. When one gesture pattern recognition region is extracted, the index value assigned to the at least one gesture pattern recognition region is checked, the identified index values are combined, and the combination of the identified index values is determined from the figure information table. Correspondingly recorded figure information may be extracted to generate the extracted figure information as figure information corresponding to a movement trajectory of the user's hand.

In addition, according to an embodiment of the present invention, the gesture pattern recognition region extractor 118 may include a hand region detector 120 and a hand region position tracker 121.

The hand region detector 120 checks color values of a plurality of pixels constituting each of the plurality of image frames with respect to the plurality of image frames constituting the captured image photographed by the camera 111, and thus, the plurality of pixels. Among them, a set of at least one pixel having a color value corresponding to a predetermined first color value range associated with a skin color is detected as the user's hand region.

The hand region position tracking unit 121 tracks a change in position of each of the plurality of image frames with respect to the set of at least one pixel detected by the hand region of the user, and thus changes the position of the user's hand. Perform object tracking for

In addition, according to an embodiment of the present invention, the gesture pattern recognition region extraction unit 118 overlaps the point where the user's hand is located on the captured image photographed by the camera among the plurality of selected gesture pattern recognition regions. The portion may be extracted as the at least one gesture pattern recognition only for a gesture pattern recognition area that exceeds a selected area size ratio in comparison with the size of each of the plurality of selected gesture pattern recognition areas.

Hereinafter, the operation of the HMD device 110 capable of gesture-based user authentication according to the present invention will be described in detail with reference to the drawings illustrated in FIGS. 2 and 3.

2 and 3 are views for explaining the operation of the HMD device 110 capable of gesture-based user authentication according to an embodiment of the present invention.

First, the HMD device 110 capable of gesture-based user authentication according to the present invention is worn on the user's head, as shown in FIG. 2, so that a display exists in front of both eyes of the user and the user through the display. To show the screen to the user.

In this case, the HMD device 110 according to the present invention may include a camera 111 attached to the front of the HMD device 110 and capable of photographing an external environment.

In addition, the figure information storage unit 112 may store n pieces of first figure information specified in advance for user authentication.

In relation to this, assuming that n is 5, and as shown by reference numeral 310 of FIG. 3, figure information previously designated for the user authentication is figure information for shape "3", figure information for shape "4". , Geometry information for "5" shape, "

"Shape information about the shape,"

In the case of the figure information on the shape, the information may be stored in the figure information storage unit 112 as shown in Table 1 below.

순번turn	제1 도형 정보First figure information
1One	"3" 모양에 대한 도형 정보Shape info for shape "3"
22	"4" 모양에 대한 도형 정보Shape info for shape "4"
33	"5" 모양에 대한 도형 정보Geometry information for "5" shape
44	" " 모양에 대한 도형 정보" "About Shapes for Shapes
55	" " 모양에 대한 도형 정보" "About Shapes for Shapes

If a user needs a predetermined user authentication such as purchasing a game item or purchasing an item in an online shopping mall while playing a game while wearing the HMD device 110, the HMD device 110 is installed in the HMD device 110. A user authentication command may be applied to the HMD device 110 by the control of a predetermined application such as a payment related application.

In this case, when the user authentication command is applied, the figure information generator 113 may drive the camera 111 to acquire a captured image of an external environment.

In this case, the image display control unit 115 may display a captured image of the external environment captured by the camera 111 in real time on a display, so that a user may control the external environment through the display. .

In addition, the gesture pattern recognition area display unit 116 displays a plurality of predetermined gesture

pattern recognition areas

211, 212, 213, 214, 215, and 216 at a predetermined point on the display. , 217).

In this regard, the gesture pattern recognition area display unit 116 may include a plurality of gesture pattern recognition areas as shown at 210 in the center of the display when a captured image of an external environment captured by the camera 111 is being displayed on the display. The selected gesture

pattern recognition areas

211, 212, 213, 214, 215, 216, and 217 may be displayed.

At this time, the user looks at the plurality of predetermined gesture

pattern recognition areas

211, 212, 213, 214, 215, 216, 217 displayed on the display, and uses his or her hand to select the plurality of selected gestures. Gesture in the form of drawing the same figure information as the five first figure informations stored in the figure information storage unit 112 on the

pattern recognition areas

211, 212, 213, 214, 215, 216, 217. By doing so, the user authentication process can be performed.

In this case, since the user's hand is photographed and displayed together with the captured image of the external environment captured by the camera 111, the user looks at his or her hand displayed on the display, It may be controlled to move on the selected gesture

pattern recognition areas

211, 212, 213, 214, 215, 216, 217.

When the user moves his or her hand, the figure information generation unit 113 captures the external environment and the user's hand together through the camera 111, and then the user's hand from the captured image captured by the camera 111. The object tracking of the movement of the user may be performed to generate figure information corresponding to the movement trajectory of the user's hand.

In this regard, the figure information generating unit 113 may include a figure information table holding unit 117, a gesture pattern recognition region extracting unit 118, and a figure information extracting unit 119.

The figure information table holding unit 117 may store and maintain a figure information table in which different figure information predetermined in advance with respect to combinations of a plurality of different gesture pattern recognition areas are recorded.

In this case, according to an embodiment of the present invention, different index values are pre-assigned to the plurality of predetermined gesture

pattern recognition areas

211, 212, 213, 214, 215, 216, 217, and the figure information table The predetermined different figure information corresponding to combinations of index values for a plurality of different gesture pattern recognition areas may be recorded.

In relation to this, an index value of “1” may be assigned to the gesture pattern recognition region 1 211 among the plurality of predetermined gesture

pattern recognition regions

211, 212, 213, 214, 215, 216, and 217. The index value "2" may be assigned to the gesture pattern recognition area 2212, and the index value "3" may be assigned to the gesture pattern recognition area 3213, and the gesture pattern recognition may be performed. An index value of "4" may be assigned to the area 4 214, an index value of "5" may be assigned to the gesture pattern recognition area 5 215, and a gesture pattern recognition area 6 (216). ), An index value of "6" may be assigned, and an index value of "7" may be assigned to the gesture pattern recognition area 7 217.

In addition, information may be recorded in the figure information table as shown in Table 2 below.

순번turn	제스처 패턴 인식 영역에 대한 인덱스 값의 조합Combination of index values for gesture pattern recognition area	미리 지정된 도형 정보Predefined Geometry Information
1One	7, 67, 6	"1" 모양에 대한 도형 정보Shape info for shape "1"
22	1, 2, 3, 6, 51, 2, 3, 6, 5	"2" 모양에 대한 도형 정보Shape info for shape "2"
33	1, 2, 3, 4, 51 2 3 4 5	"3" 모양에 대한 도형 정보Shape info for shape "3"
44	7, 3, 2, 47, 3, 2, 4	"4" 모양에 대한 도형 정보Shape info for shape "4"
55	1, 7, 3, 4, 51, 7, 3, 4, 5	"5" 모양에 대한 도형 정보Geometry information for "5" shape
66	1, 2, 3, 71, 2, 3, 7	" " 모양에 대한 도형 정보" "About Shapes for Shapes
77	3, 4, 5, 63, 4, 5, 6	" " 모양에 대한 도형 정보" "About Shapes for Shapes
......	......	......

In this case, the gesture pattern recognition region extracting unit 118 performs object tracking on the movement of the user's hand from the captured image photographed by the camera 111, thereby selecting a plurality of predetermined gesture

pattern recognition regions

211, 212,. At least one gesture pattern recognition region overlapping a point where the user's hand is located may be extracted from the captured images captured by the camera 111 among the 213, 214, 215, 216, and 217.

In this case, the gesture pattern recognition region extraction unit 118 may include a hand region detection unit 120 and a hand region position tracking unit 121 to perform object tracking on the movement of the user's hand.

The hand region detector 120 checks color values of a plurality of pixels constituting each of the plurality of image frames with respect to the plurality of image frames constituting the captured image photographed by the camera 111, and thus, the plurality of pixels. Among them, a set of at least one pixel having a color value corresponding to a predetermined first color value range associated with a skin color may be detected as the user's hand region.

In addition, the hand region position tracking unit 121 tracks a change in position of each of the plurality of image frames with respect to the set of at least one pixel detected by the hand region of the user, thereby controlling the position of the user's hand. Object tracking for movement can be performed.

For example, assume that "picture frame 1", "picture frame 2", and "picture frame 3" exist in the chronological order of the plurality of picture frames.

At this time, the hand region detector 120 checks the color values of the plurality of pixels constituting the "image frame 1" and then the color value of the plurality of pixels corresponding to the selected first color value range associated with the skin color. At least one set of pixels having a may be detected as the user's hand region in the "image frame 1".

The hand region detector 120 checks the color values of the plurality of pixels constituting “image frame 2”, and then the color value among the plurality of pixels corresponds to the selected first color value range associated with the skin color. At least one pixel set having a value may be detected as the user's hand region in the "image frame 2".

The hand region detector 120 checks the color values of the plurality of pixels constituting “image frame 3”, and then the color value of the plurality of pixels corresponds to the selected first color value range associated with the skin color. At least one pixel set having a value may be detected as the user's hand region in the "image frame 3".

At this time, the hand region position tracking unit 121 according to the time order in the "image frame 1", "image frame 2", "image frame 3" for the set of at least one pixel detected by the user's hand region. By tracking the position change, the object tracking for the movement of the user's hand can be performed.

That is, the hand region position tracking unit 121 tracks the position change of the user's hand region extracted from each image frame in the order of "image frame 1", "image frame 2", and "image frame 3". Object tracking of the movement of the user's hand may be performed.

In this manner, the gesture pattern recognition region extracting unit 118 performs object tracking on the movement of the user's hand from the captured image photographed by the camera 111, thereby selecting a plurality of predetermined gesture pattern recognition regions 211. At least one gesture pattern recognition region overlapping a point where the user's hand is located may be extracted from the captured images captured by the camera 111 among 212, 213, 214, 215, 216, and 217.

For example, as shown by reference numeral 321 of FIG. 3, the user may touch his or her hand with a gesture pattern recognition area 1 among a plurality of predetermined gesture

pattern recognition areas

211, 212, 213, 214, 215, 216, and 217. 211, gesture pattern recognition region 2 (212), gesture pattern recognition region 3 (213), gesture pattern recognition region 4 (214), gesture pattern recognition region 5 (215) The area extractor 118 performs object tracking on the movement of the user's hand from the captured image captured by the camera 111, thereby selecting a plurality of predetermined gesture

pattern recognition areas

211, 212, 213, 214, and 215. , At least one gesture pattern recognition region overlapping with the point where the user's hand is located on the captured image captured by the camera 111 among the 216, 217, gesture pattern recognition region 1 211 and gesture pattern recognition region 2 ( 212), gesture pattern recognition Station 3 213, it is possible to extract gesture pattern recognition area 4 214, the gesture pattern recognition area 5 215.

At this time, according to an embodiment of the present invention, the gesture pattern recognition region extraction unit 118 is the camera 111 of the plurality of predetermined gesture pattern recognition regions (211, 212, 213, 214, 215, 216, 217) When extracting the at least one gesture pattern recognition region overlapping with the point where the user's hand is located on the captured image photographed by using, a plurality of predetermined gesture

pattern recognition regions

211, 212, 213, 214, 215, 216 , 217, a portion overlapping with the point where the user's hand is located on the captured image captured by the camera 111 includes a plurality of predetermined gesture

pattern recognition regions

211, 212, 213, 214, 215, 216, and 217. Only the gesture pattern recognition area that exceeds the selected area size ratio in comparison to each size may be extracted as the at least one gesture pattern recognition area.

For example, when the predetermined area size ratio is 60% of the size of each gesture pattern recognition area, the gesture pattern recognition area extraction unit 118 may include the plurality of selected gesture

pattern recognition areas

211, 212, 213, and 214. , 215, 216, and 217, which overlap the point where the user's hand is located, may be extracted as the at least one gesture pattern recognition area only for a gesture pattern recognition area that exceeds 60% of the size of each gesture pattern recognition area. Can be.

When the at least one gesture pattern recognition region is extracted by the gesture pattern recognition region extraction unit 118, the figure information extraction unit 119 checks an index value assigned to the at least one gesture pattern recognition region. After combining the identified index values, the figure information corresponding to the combination of the identified index values is extracted from the figure information table, and the figure information corresponding to the movement trajectory of the user's hand is extracted. Can be generated as

In this regard, as in the above-described example, the gesture pattern recognition area extractor 118 may include the gesture pattern recognition area 1 211, the gesture pattern recognition area 2 212, the gesture pattern recognition area 3 213, and the gesture pattern recognition area. If 4 (214) and gesture pattern recognition area 5 (215) are extracted, the figure information extracting unit 119 checks the index value "1" assigned to gesture pattern recognition area 1 (211), and gesture pattern recognition area. Confirms the index value "2" assigned to 2 (212), checks the index value "3" assigned to gesture pattern recognition area 3 (213), and assigns it to gesture pattern recognition area 4 (214). Check the index value "4", check the index value "5" assigned to the gesture pattern recognition area 5 (215), and then, based on the combination "1, 2, 3, 4, 5" of each index value, Index values of "1, 2, 3, 4, 5" from the figure information table as shown in Table 2. The figure information for the shape "3" which is the figure information recorded corresponding to the combination can be extracted.

In this manner, as shown in FIG. 3, the user's hand movement as indicated by reference numeral 321 was applied by the user, the hand movement as indicated by reference numeral 322 was applied, and the hand movement as indicated by reference numeral 323 was applied. If a hand movement such as 324 is applied, and a hand movement such as 325 is applied, the figure information generating unit 113 of the user's hand such as 321, 322, 323, 324, and 325 is applied. Shape information for shape "3", five shape information corresponding to movement trajectory, shape information for shape "4", shape information for shape "5", "

"Shape information about the shape and"

"You can create the shape information about the shape.

Then, when five figure information corresponding to the movement trajectory of the user's hand is generated, the user authentication unit 114 generates five figure information generated corresponding to the movement trajectory of the user's hand and the table 1 as shown in Table 1 below. When comparing the five pieces of first figure information stored in the figure information storage unit 112 and determining that both figure information matches, the user authentication may be completed.

As a result, the HMD device 110 capable of gesture-based user authentication according to the present invention stores predetermined first figure information previously designated for user authentication, and then captures an external environment in front of the HMD device 110. 111, by recording the movement of the user's hand and the external environment through the camera 111 and performing object tracking of the movement of the user's hand from the captured image taken by the camera 111 After generating figure information corresponding to the movement trajectory of the user's hand, the figure information corresponding to the movement trajectory of the user's hand and the predetermined first figure information are compared with each other to determine that both figure information is identical. When the user authentication can be completed, the user can easily proceed with the authentication process even in the HMD device 110 without a separate input device I can support them.

4 is a flowchart illustrating a gesture-based user authentication method of an HMD device having a camera attached to a front surface according to an embodiment of the present invention.

In operation S410, a figure information storage unit, in which n pieces of first figure information (n is a natural number) designated in advance for user authentication, is stored.

In operation S420, when a user authentication command is applied under the control of an application installed in the HMD device, the camera attached to the front of the HMD device is driven to acquire a captured image of an external environment. As the hand moves, when the external environment and the user's hand are photographed together through the camera, object tracking of the movement of the user's hand is performed from the captured image photographed through the camera, so that the user's hand Graphic information corresponding to the movement trajectory is generated.

In step S430, when n pieces of figure information corresponding to the movement trajectory of the user's hand are generated, n shape information corresponding to the movement trajectory of the user's hand is compared with the n first figure information. If it is determined that both figure information match, the user authentication is completed.

At this time, according to an embodiment of the present invention, the gesture-based user authentication method of the HMD device displaying a captured image of the external environment taken by the camera in real time on a display and selected on the display The method may further include displaying a plurality of predetermined gesture pattern recognition regions at the point.

At this time, in step S420, storing and maintaining a figure information table in which different figure information predetermined in advance is recorded corresponding to combinations of a plurality of different gesture pattern recognition areas, and photographing images photographed by the camera. At least one gesture pattern recognition region overlapping a point where the user's hand is located on the captured image photographed by the camera among the plurality of selected gesture pattern recognition regions by performing object tracking on the movement of the user's hand from Extracting and extracting the at least one gesture pattern recognition area, the combination of the at least one gesture pattern recognition area is recorded corresponding to the combination of the at least one gesture pattern recognition area from the figure information table. Extracted figure information Corresponding to the type information on a movement path of the hand of the user may include the step of generating a figure information.

In this case, according to an embodiment of the present invention, different index values are pre-assigned to the plurality of predetermined gesture pattern recognition regions, and the index information for the plurality of different gesture pattern recognition regions is assigned to the figure information table. The predetermined different figure information for the combinations may be recorded correspondingly.

In this case, the generating of the extracted figure information as figure information corresponding to a movement trajectory of the user's hand may include: an index allocated to the at least one gesture pattern recognition region when the at least one gesture pattern recognition region is extracted; Determine a value, combine the checked index values, extract figure information recorded corresponding to the identified index value combination from the figure information table, and move the extracted figure information to the user's hand; It may be generated by the figure information corresponding to the.

In addition, according to an embodiment of the present disclosure, the extracting of the at least one gesture pattern recognition area may include each of the plurality of image frames with respect to a plurality of image frames constituting a captured image photographed by the camera. Identifying a color value of a plurality of pixels constituting a plurality of pixels, the set of at least one pixel having a color value corresponding to a predetermined first color value range associated with a skin color among the plurality of pixels as a user's hand region Detecting an object and tracking a change in position in each of the plurality of image frames with respect to the set of at least one pixel detected by the user's hand region to track the movement of the user's hand. It may include the step.

In addition, according to an embodiment of the present disclosure, the extracting of the at least one gesture pattern recognition region may include positioning the user's hand on a captured image photographed by the camera among the plurality of predetermined gesture pattern recognition regions. A portion overlapping with a point may be extracted as the at least one gesture pattern recognition region only for a gesture pattern recognition region in which a portion overlapping with the size of each of the selected gesture pattern recognition regions exceeds a predetermined area size ratio.

The gesture-based user authentication method of the HMD device according to the embodiment of the present invention has been described above with reference to FIG. 4. Here, the gesture-based user authentication method of the HMD device according to an embodiment of the present invention may correspond to the configuration of the operation of the HMD device 110 capable of gesture-based user authentication described with reference to FIGS. 1 to 3. Therefore, a detailed description thereof will be omitted.

The gesture-based user authentication method of the HMD device according to an embodiment of the present invention may be implemented as a computer program stored in a storage medium for execution by combining with a computer.

In addition, the gesture-based user authentication method of the HMD device according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In the present invention as described above has been described by the specific embodiments, such as specific components and limited embodiments and drawings, but this is provided to help a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations are possible from these descriptions.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and all the things that are equivalent to or equivalent to the claims as well as the following claims will belong to the scope of the present invention. .

Claims

In a head mounted display (HMD) device capable of gesture-based user authentication,

A camera attached to the front of the HMD device and capable of photographing an external environment;

A figure information storage unit storing n first figure information (n is a natural number) previously designated for user authentication;

When a user authentication command is applied under the control of an application installed in the HMD device, the camera is driven to obtain a captured image of the external environment, and as the user moves his / her hand, the external environment through the camera. And the user's hand are photographed together, generating figure information for generating figure information corresponding to the movement trajectory of the user's hand by performing object tracking on the movement of the user's hand from the captured image photographed by the camera. part; And

When n pieces of figure information corresponding to the movement trajectory of the user's hand are generated, both of the figure information is compared by comparing the n first figure information and the n figure information generated corresponding to the movement trajectory of the user's hand. If it is determined to match, the user authentication unit for completing the user authentication

HMD device capable of gesture-based user authentication comprising a.
The method of claim 1,

An image display controller which displays a captured image of the external environment captured by the camera in real time on a display; And

Gesture pattern recognition area display unit for displaying a plurality of predetermined gesture pattern recognition areas at a predetermined point on the display.

More,

The figure information generation unit

A figure information table holding unit which stores and holds a figure information table in which different figure information predetermined in advance for each combination of a plurality of different gesture pattern recognition areas is recorded;

A point where the user's hand is located on the captured image photographed by the camera among the plurality of predetermined gesture pattern recognition regions by performing object tracking on the movement of the user's hand from the captured image photographed by the camera; A gesture pattern recognition region extraction unit which extracts at least one overlapping gesture pattern recognition region; And

When the at least one gesture pattern recognition region is extracted, the at least one gesture pattern recognition region is combined and the figure information recorded corresponding to the combination of the at least one gesture pattern recognition region is extracted from the figure information table. A figure information extracting unit configured to generate the extracted figure information as figure information corresponding to a movement trajectory of the user's hand

HMD device capable of gesture-based user authentication comprising a.
The method of claim 2,

Different index values are pre-assigned to the plurality of predetermined gesture pattern recognition regions,

In the figure information table

The predetermined different figure information is correspondingly recorded for combinations of index values for a plurality of different gesture pattern recognition areas,

The figure information extracting unit

When the at least one gesture pattern recognition area is extracted, the index value assigned to the at least one gesture pattern recognition area is checked, the identified index values are combined, and the index value is determined from the figure information table. A HMD device capable of gesture-based user authentication which extracts figure information recorded corresponding to a combination and generates the extracted figure information as figure information corresponding to a movement trajectory of the user's hand.
The method of claim 2,

The gesture pattern recognition region extraction unit

The color value of the plurality of pixels constituting each of the plurality of image frames is checked with respect to the plurality of image frames constituting the captured image photographed by the camera, and the color value among the plurality of pixels is selected to be associated with the skin color. A hand region detector for detecting a set of at least one pixel having a color value corresponding to the first color value range as the user's hand region; And

Hand region location for tracking the position change in each of the plurality of image frames with respect to the set of at least one pixel detected by the hand region of the user to track the movement of the user's hand Tracking

HMD device capable of gesture-based user authentication comprising a.
The method of claim 2,

The gesture pattern recognition region extraction unit

A portion of the plurality of selected gesture pattern recognition regions overlapping with a location where the user's hand is located on the captured image photographed by the camera is selected in comparison with the size of each of the plurality of selected gesture pattern recognition regions. A gesture-based user authentication capable of extracting only the gesture pattern recognition area exceeding a size ratio into the at least one gesture pattern recognition area.
In the gesture-based user authentication method of the head mounted display (HMD) device with a camera attached to the front,

Maintaining a figure information storage unit storing n first figure information (n is a natural number) previously designated for user authentication;

When a user authentication command is applied under the control of an application installed in the HMD device, the camera is driven to obtain a captured image of an external environment, and as the user moves his or her hand, Generating the figure information corresponding to the movement trajectory of the user's hand by performing object tracking on the movement of the user's hand from the captured image photographed by the camera when the user's hand is photographed together; And

When n pieces of figure information corresponding to the movement trajectory of the user's hand are generated, both of the figure information is compared by comparing the n first figure information and the n figure information generated corresponding to the movement trajectory of the user's hand. If it is determined to match, completing the user authentication;

Gesture-based user authentication method of the HMD device comprising a.
The method of claim 6,

Displaying a captured image of the external environment captured by the camera on a display in real time; And

Displaying a plurality of predetermined gesture pattern recognition regions at a predetermined point on the display.

More,

Generating the figure information

Storing and maintaining a figure information table in which different figure information predetermined in advance for respective combinations of a plurality of different gesture pattern recognition areas are correspondingly recorded;

A point where the user's hand is located on the captured image photographed by the camera among the plurality of predetermined gesture pattern recognition regions by performing object tracking on the movement of the user's hand from the captured image photographed by the camera; Extracting at least one overlapping gesture pattern recognition region; And

When the at least one gesture pattern recognition region is extracted, the at least one gesture pattern recognition region is combined and the figure information recorded corresponding to the combination of the at least one gesture pattern recognition region is extracted from the figure information table. Generating the extracted figure information as figure information corresponding to a movement trajectory of the user's hand;

Gesture-based user authentication method of the HMD device comprising a.
The method of claim 7, wherein

Different index values are pre-assigned to the plurality of predetermined gesture pattern recognition regions,

In the figure information table

The predetermined different figure information is correspondingly recorded for combinations of index values for a plurality of different gesture pattern recognition areas,

The step of generating the extracted figure information as figure information corresponding to the movement trajectory of the user's hand

When the at least one gesture pattern recognition area is extracted, the index value assigned to the at least one gesture pattern recognition area is checked, the identified index values are combined, and the index value is determined from the figure information table. And extracting the figure information corresponding to the combination and generating the extracted figure information as figure information corresponding to a movement trajectory of the user's hand.
The method of claim 7, wherein

Extracting the at least one gesture pattern recognition region

The color value of the plurality of pixels constituting each of the plurality of image frames is checked with respect to the plurality of image frames constituting the captured image photographed by the camera, and the color value among the plurality of pixels is selected to be associated with the skin color. Detecting a set of at least one pixel having a color value corresponding to a first color value range as the user's hand region; And

Performing object tracking on the movement of the user's hand by tracking a change in position in each of the plurality of image frames with respect to the set of at least one pixel detected by the user's hand region in time order;

Gesture-based user authentication method of the HMD device comprising a.
The method of claim 7, wherein

Extracting the at least one gesture pattern recognition region

A portion of the plurality of selected gesture pattern recognition regions overlapping with a location where the user's hand is located on the captured image photographed by the camera is selected in comparison with the size of each of the plurality of selected gesture pattern recognition regions. A gesture-based user authentication method of an HMD device extracting only the gesture pattern recognition area exceeding a size ratio to the at least one gesture pattern recognition area.
A computer-readable recording medium having recorded thereon a program for performing the method of any one of claims 6 to 10.
A computer program stored in a storage medium for executing the method of any one of claims 6 to 10 in combination with a computer.