KR102542641B1 - Apparatus and operation method for rehabilitation training using hand tracking - Google Patents

Abstract

A training device is provided. The training device includes a sensing unit for sensing a user's hand; It may include a; tracking unit for tracking the first position of the hand by using the detection result of the sensing unit.

Description

Apparatus and operation method for rehabilitation training using hand tracking {Apparatus and operation method for rehabilitation training using hand tracking}

The present invention uses XR (eXtended Reality) technology to perform missions provided through cognitive rehabilitation training contents using a user's actual hand represented on Virtual Reality or Augmented Reality. It relates to a rehabilitation training device and operation method.

The effects of motor and cognitive function training using virtual reality have been proven through various research results.

However, it is inconvenient and difficult for people with reduced cognitive function and the elderly to use it, so it is not used even though it is an advanced system for cognitive function evaluation and training.

Korean Registered Patent Publication No. 1881986 discloses a VR cognitive rehabilitation training system and method for improving cognitive abilities of elderly people with mild cognitive impairment.

Korean Registered Patent Publication No. 1881986

An object of the present invention is to provide a training device and a method of operating the training device that allow a user to perform cognitive rehabilitation training in a virtual space using his or her actual hands.

The training apparatus of the present invention includes a sensing unit for detecting a user's hand; and a tracking unit that tracks a first position of the hand by using a detection result of the sensing unit, wherein the tracking unit obtains a second position of an object displayed on a screen viewed by the user, and the tracking unit obtains a second position of an object displayed on a screen viewed by the user. It is possible to determine whether the hand can interact with the object using the first position and the second position.

The operating method of the training apparatus of the present invention includes an acquisition step of acquiring a first position corresponding to the position of a user's hand, a second position corresponding to the position of an object displayed in a virtual space, and a reference position of the virtual space; determining whether the first location interacts with the second location by arranging the first location in the virtual space using the reference location; and a providing step of providing a result of determining whether or not the interaction has occurred to a media unit generating the virtual space.

The present invention is a technology in which one's hand is directly projected on virtual reality and can interact with objects in virtual reality without using a separate tool. An effective cognitive rehabilitation training environment can be provided by enabling various and flexible experiences.

The training apparatus of the present invention can project the user's real hand onto a virtual space. Hands in the real world and disparate coordinates in virtual space can be synchronized. In order to solve errors generated in the coordinate synchronization process between different dimensions, the present invention may additionally use the actual position of the user's eyes.

According to the present invention, it is possible to experience a rehabilitation training environment similar to touching a real object in the real world in a virtual space.

1 is a block diagram showing a training device of the present invention.
2 is a schematic diagram showing a tracking unit.
Figure 3 is a schematic diagram showing the training device of the present invention.
4 is a schematic diagram showing another training device of the present invention.
5 is a schematic diagram showing a screen displayed to a user in the training apparatus of the present invention.
Figure 6 is a flow chart showing the operating method of the training device of the present invention.
7 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In this specification, redundant descriptions of the same components are omitted.

In addition, in this specification, when a component is referred to as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but another component in the middle It should be understood that may exist. On the other hand, in this specification, when a component is referred to as 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

In addition, terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention.

Also, in this specification, a singular expression may include a plurality of expressions unless the context clearly indicates otherwise.

In addition, in this specification, terms such as 'include' or 'having' are only intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more It should be understood that the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Also in this specification, the term 'and/or' includes a combination of a plurality of listed items or any item among a plurality of listed items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the subject matter of the present invention will be omitted.

1 is a block diagram showing a training device of the present invention. 2 is a schematic diagram showing the tracking unit 130. Figure 3 is a schematic diagram showing the training device of the present invention. 4 is a schematic diagram showing another training device of the present invention.

Depth information photographed by a depth sensor attached to the top of the chroma key environment booth 30 or the front of the HMD may be transmitted to the tracking module 100 .

When using VR, images of hands and arms are extracted from the images taken by the stereoscopic camera attached to the front of the HMD and synthesized with images in virtual reality (virtual space 99) so that the hand 11 of the user 10 is in the virtual space. (99). In the case of VR, since it is difficult to display the actual hand of the user 10 on the screen 90, the hand of the user 10 is photographed using a separate stereoscopic camera, vision, etc. A graphic simulating a hand may be projected onto the virtual space 99 .

In the case of using AR of a see-through type in which a real object is displayed on the screen 90, some objects 91 of training content may be projected and displayed on the HMD.

The media unit 210 may transmit coordinates (second position) and size information of the interactive object 91 on the virtual space 99 to the tracking module 100 .

The tracking module 100 may detect the coordinates (first position) of the hand and arm from the depth information and determine whether or not interaction with the interactive object 91 in the virtual space 99 is possible. The tracking module 100 may transmit the determination result to the media unit 210 again.

The user 10 directly performs the mission provided by the media unit 210 with his or her own hands using the hand tracking technology implemented as a series of motions, and performs this continuously and repeatedly to perform cognitive function rehabilitation training. there is.

The virtual space 99 of the present invention may be implemented in XR, VR, MR, AR, and the like.

XR (eXtended Reality) is a concept that encompasses VR (Virtual Reality), MR (Mixed Reality), AR (Augmented Reality), etc. can mean

The cognitive rehabilitation training content presented in this patent can be expressed as rehabilitation training using XR because it can be implemented in VR, MR, and AR.

The training device shown in the figure may include a tracking module 100, a media unit 210, and a display unit 230.

The tracking module 100 may track the real hand of the user 10 existing in the real world. In addition, the tracking module 100 may determine whether there is an interaction between the object 91 disposed in the virtual space 99 and the user's 10 real hand.

A sensing unit 110 and a tracking unit 130 may be provided in the tracking module 100 .

The sensing unit 110 may detect the hand of the user 10 . For example, the sensing unit 110 may include a distance sensor that obtains depth information of the user's 10 hand. The sensing unit 110 may be installed in a chromakey environment booth 30 or installed in an HMD worn by the user 10 . A Head Mount Display (HMD) is a display device for viewing the virtual object 91 and reality together, and may be worn on the head of the user 10 similarly to glasses.

The tracking unit 130 may track a first position corresponding to the position of the hand of the user 10 existing in the real world by using the sensing result of the sensing unit 110 .

The tracking unit 130 may obtain the second position of the object 91 displayed on the screen 90 viewed by the user 10 from the media unit 210 .

The tracking unit 130 may determine whether an interaction between the hand and the object 91 is possible using the first position and the second position.

The media unit 210 may display rehabilitation content including an object 91 on the screen 90 .

The media unit 210 may provide training content for cognitive function improvement developed through the 3D Engine. The media unit 210 expresses 3D virtual reality (virtual space 99) through the HMD and provides responsive content through hand tracking performed by the tracking module 100 or interaction with virtual reality through the controller. can do.

Depending on the operating environment, it is divided into 100% virtual reality VR, MR including some reality information, and AR operating based on the real environment. Contents and missions suitable for each environment are displayed through the HMD, and the user 10 performs them can do.

The media unit 210 may obtain a result of determining whether or not the interaction is possible from the tracking unit 130 . The media unit 210 may change the object 91 according to the determination result. For example, when the object 91 is a valve and it is determined that an interaction exists, the media unit 210 may rotate the valve.

The screen 90 may include a Liquid Crystal Display (LCD) screen 90 , an Organic Light Emitting Diode (OLED) screen 90 , and the like provided by the display unit 230 . In other words, the media unit 210 may display rehabilitation content through the display unit 230 equipped with various displays. Rehabilitation content including the object 91 may be placed in the virtual space 99 created by the media unit 210 .

The display unit 230 may include an LCD, OLED, head mounted display (HMD), and the like.

The sensing unit 110 may include a first communication unit 131 , a first tracking unit 132 , a second communication unit 137 , a second tracking unit 137 , and a determination unit 135 .

The first communication unit 131 may receive depth information from the sensing unit 110 through a wired or wireless communication network.

The first tracking unit 132 may track the first location using depth information. The first location tracked by the first tracking means 132 may indicate a location in the real world.

The second communication unit 137 may acquire the object 91 . For example, the second communication unit 137 may communicate with the media unit 210 that provides the object 91 through various wired or wireless communication networks, and may receive the object 91 from the media unit 210 .

The second tracking unit 137 may track a second position corresponding to the position of the object 91 . In this case, the second location may represent a location in the virtual space 99 .

The determination unit 135 may determine whether the interaction is possible using the first location and the second location.

The first position may indicate a position of the hand, and the second position may indicate a position of the object 91 .

The position of the hand may be a position in the real world, and the position of the object 91 may be a position in the virtual space 99 . A method for determining whether or not an interaction exists by using two locations of different dimensions may be additionally prepared.

In the cognitive rehabilitation training of the present invention, an object 91 that is manipulated by hand may be provided in a virtual space 99 that cannot actually be manipulated by hand. Therefore, the cognitive rehabilitation training of the present invention can be performed by relying on eye vision rather than hand tactile sense.

Therefore, the object 91 placed in the virtual space 99 is recognized by the eyes, and at this time, the shape of the object 91 recognized by the user 10 is related to the position of the user 10's actual eyes. there is.

Therefore, for normal interaction, it is recommended that the user 10 directly look at his/her hand through the screen 90 .

Unlike augmented reality where the user's 10's real hand is displayed on the screen 90, in a VR environment where it is difficult to project the user's 10's real hand on the screen 90, the user's 10's real hand is displayed in a virtual space (99). ), it is necessary to provide a separate means for rendering on.

For example, the sensing unit 110 may include a vision sensor or a stereoscopic camera that 3D photographs the hand of the user 10 .

In summary, the determining unit 135 may determine an interaction with the object 91 of the XR content by using hand tracking as a method for interacting with the XR content.

Hand tracking performed by the tracking module 100 may include a method of detecting the position of the hand of the user 10 and interacting with the object 91 of the XR content.

In the case of VR where the real surrounding environment cannot be seen, a stereoscopic camera can be attached to the front of the HMD to capture the hand of the user 10 in 2D or 3D and project it in 2D or 3D to virtual reality.

Specifically, the tracking unit 130 may obtain image data of a hand photographed from the sensing unit 110 .

The tracking unit 130 may provide image data representing the hand along with the first position to the media unit 210 that displays rehabilitation content including the object 91 on the screen 90 .

The second position of the object 91 viewed in the virtual space 99 may be set normally over the surface of the object 91 .

On the other hand, the first position representing the coordinates of the hand of the user 10 is based on a value measured by the sensing unit 110 such as a depth sensor, and some errors may exist in the corresponding value, which may cause a visual illusion. can For example, the user 10 can expect to be able to touch the object 91 when he brings his hand to this position. However, in the case of the object 91, which is easy to ignore various light effects, it can easily cause an optical illusion of the user 10, and this optical illusion effect can negatively affect the performance of the user 10's cognitive rehabilitation training.

The optical illusion effect shows a great difference according to the visual position of the user 10, and a method for reducing an error due to the corresponding visual position may be additionally prepared.

For example, the tracking unit 130 may additionally obtain reference coordinates from the virtual space 99 formed on the screen 90 .

The tracking unit 130 may specify the first position in the virtual space 99 using reference coordinates. In addition, the tracking unit 130 may determine whether the interaction is possible using the distance between the first position and the second position based on the reference coordinates.

The tracking unit 130 may additionally obtain a third position corresponding to the position of the eyes of the user 10 from the sensing unit 110 .

The tracking unit 130 may use the third position to correct the determination result of mutual availability. For example, the tracking unit 130 may adjust the second position within the range of the hand region of the user 10 using the third position. The position of the surface of the hand may be partially changed due to various errors, and even when the actual position is designated accurately, in the case of the object 91 in which various shades are difficult to be normally reflected, various optical illusions may be caused to the user 10 . Due to the error and optical illusion, an interaction expected value perceived by the eyes of the user 10 and an interaction result value determined by the tracking unit 130 may be different from each other. According to the present embodiment, since the interaction result value is corrected based on the eyes of the user 10 looking at the object 91 and the hand, the expected value expected by the user 10 and the discrimination value of the tracking unit 130 difference can be reduced. The corresponding correction value may be obtained through experimental data on the eye position.

5 is a schematic diagram showing a screen 90 displayed to the user 10 in the training device of the present invention.

5 may represent a VR environment. In VR, the actual hand of the user 10 is not directly projected onto the virtual space 99 . In this case, a virtual hand 93 simulating the hand of the user 10 may be rendered in the corresponding virtual space 99 .

Figure 6 is a flow chart showing the operating method of the training device of the present invention.

The operation method of the training device of FIG. 6 may be performed by the training device shown in FIG. 1 .

The operation method of the training device may include an acquisition step (S 510), a determination step (S 520), and a provision step (S 530).

In the obtaining step (S510), the first position corresponding to the position of the hand of the user 10, the second position corresponding to the position of the object 91 displayed in the virtual space 99, and the virtual space 99 A reference position 97 is available. As an operation performed by the tracking unit 130 , the tracking unit 130 may acquire the first position from the sensing unit 110 . The tracking unit 130 may acquire the second location and the reference location 97 from the media unit 210 .

In the determination step (S520), the first location may be placed on the virtual space 99 using the reference location 97, and it may be determined whether the first location interacts with the second location. The determination step (S520) may be performed by the tracking unit 130.

In the providing step (S530), a determination result of whether or not an interaction may be provided may be provided to the media unit 210 that creates the virtual space 99. The providing step (S520) may be performed by the tracking unit 130.

In the acquiring step ( S510 ), a third position corresponding to the position of the user's 10 eyes may be additionally obtained.

In the determination step (S520), the second position may be corrected using the third position, and it may be determined whether or not the interaction occurs using the corrected second position.

7 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 7 may be a device described in this specification (eg, a training device, a tracking unit 130, etc.).

In the embodiment of FIG. 7 , the computing device TN100 may include at least one processor TN110, a transceiver TN120, and a memory TN130. In addition, the computing device TN100 may further include a storage device TN140, an input interface device TN150, and an output interface device TN160. Elements included in the computing device TN100 may communicate with each other by being connected by a bus TN170.

The processor TN110 may execute program commands stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Processor TN110 may be configured to implement procedures, functions, methods, and the like described in relation to embodiments of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of read only memory (ROM) and random access memory (RAM).

The transmitting/receiving device TN120 may transmit or receive a wired signal or a wireless signal. The transmitting/receiving device TN120 may perform communication by being connected to a network.

Meanwhile, the embodiments of the present invention are not implemented only through the devices and/or methods described so far, and may be implemented through a program that realizes functions corresponding to the configuration of the embodiments of the present invention or a recording medium in which the program is recorded. And, such implementation can be easily implemented by those skilled in the art from the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. belong to the scope of the invention.

10...user 11...hand
30...chroma key environment booth 90...screen
91 ... object 93 ... virtual hand
97 ... relative position 99 ... virtual space
100 ... tracking module 110 ... sensing unit
130 ... tracking unit 131 ... first communication means
132 First Tracking Means 135 Determination Means
137... Second communication means 138... Second tracking means
210 ... media unit 230 ... display unit

Claims (7)

a sensing unit that detects a user's hand;
A tracking unit tracking the first position of the hand by using a detection result of the sensing unit;
The tracking unit obtains a second position of an object displayed on a screen viewed by the user,
The tracking unit determines whether an interaction between the hand and the object is possible using the first position and the second position,
The tracking unit obtains image data of the hand from the sensing unit,
The tracking unit provides the image data along with the first location to a media unit displaying rehabilitation content including the object on the screen,
The tracking unit additionally obtains reference coordinates in the virtual space formed on the screen,
The tracking unit specifies the first position in the virtual space using the reference coordinates,
The tracking unit determines whether the interaction is possible using a distance between the first position and the second position based on the reference coordinates;
The tracking unit additionally obtains a third position corresponding to the position of the user's eyes from the sensing unit,
The tracking unit corrects a result of determining whether the interaction is possible using the third position,
The tracking unit adjusts the second position within a range of the user's hand using the third position,
A training device in which a virtual hand simulating the user's hand is rendered in the virtual space formed on the screen.
According to claim 1,
The media unit for displaying rehabilitation content including the object on the screen is provided,
The media unit obtains a result of determining whether the interaction is possible from the tracking unit,
The training device for changing the object according to the determination result of the media unit.
According to claim 1,
A first communication means, a first tracking means, a second communication means, a second tracking means, and a determining means are provided;
The first communication unit receives depth information from the sensing unit,
The first tracking means tracks the first position using the depth information;
the second communication means acquires the object;
the second tracking means tracks the second location;
wherein the determining unit determines whether the interaction is possible using the first location and the second location.
delete delete delete In the operation method performed by the training device,
obtaining a first position corresponding to a position of a user's hand, a second position corresponding to a position of an object displayed in the virtual space, and a reference position in the virtual space;
a determination step of arranging the first location on the virtual space using the reference location and determining whether the first location interacts with the second location;
A providing step of providing a result of determining whether or not the interaction has occurred to a media unit generating the virtual space;
Obtaining image data of the user's hand,
Providing the image data together with the first location to a media unit displaying rehabilitation content including the object on a screen viewed by the user;
additionally obtaining a third position corresponding to the position of the user's eyes;
Correcting a result of determining whether the interaction is possible using the third position,
Adjusting the second position within a range of the user's hand area using the third position;
A virtual hand simulating the user's hand is rendered in the virtual space formed on the screen.

Images