KR20230031765A - Hand tracking system capable of sensory transmission - Google Patents

Abstract

The present invention relates to a hand tracking system that can transmit a sense for an object in a virtual space to a user. The hand tracking system according to an embodiment of the present invention comprises: a camera photographing the user's arms and hands within a photographing area to acquire depth image information including the user's arms and hands; a wearable terminal attached to the user's arm to measure the user's electromyography signals, update the electromyography signals in real time, and transmit a sense for an object in a virtual space to the user; a hand tracking module generating a first object capable of performing an interaction in a virtual space, generating a second object which arrives at the first object to generate event information while being capable of performing an interaction with the user's arms and hands in the virtual space based on the depth image information and the electromyography signals, determining whether the first object and the second object are capable of performing an interaction in the virtual space, and if the event information occurs, generating sense information mapped corresponding to the event information to transmit the sense information to the wearable terminal; and an HMD terminal generating first creation coordinates at which the first object is to be generated, and size information and shape information of the first object to transmit the same to the hand tracking module, and outputting, to the virtual space, the first object and the second object which are determined to perform an interaction in the virtual space from the hand tracking module.

Description

Hand tracking system capable of sensory transmission

The present invention relates to a hand tracking system, and more particularly, to a hand tracking system capable of communicating a sense of an object in a virtual space to a user while interacting with a user's arm and hand in a virtual space.

In the conventional general game, the user enjoys the game according to the game situation output on the screen using a simple operating device such as a joystick or keyboard while looking at the monitor, so the user could not experience the same as actually experiencing the game situation. . However, when VR (Virtual Reality) is applied to games, the user can experience the game situation using the five senses as if it were a real situation, so the degree of immersion in the game can be increased.

Virtual reality is information created to indirectly experience situations that are difficult to directly experience in the real world due to physical and spatial constraints through interaction with the human sensory system in a virtual environment built using a computer. field of activity When systems that implement virtual reality provide a virtual environment in which the user's viewpoint and motion are realized, generally, changes in the user's viewpoint or motion are actually reflected in the virtual space or actions are performed on behalf of the human body in the virtual space. You can experience the environment through being.

In addition, recently, computer games and multimedia applications have begun to provide a natural user interface (NUI) using a camera and a software gesture recognition engine. These natural user interfaces detect, interpret, and use raw joint data and user gestures to control virtual space objects (avatars) or other aspects of an application.

And, one of the tasks of the natural user interface is to identify a user in the field of view of a camera or sensor, and to accurately identify positions of body parts including arms and hands of the user within the field of view. Conventionally, a routine for tracking a user's arms, hands, legs, head, and upper body has been disclosed.

However, even though the above tracking provides subtle details and very diverse positions of the user's arms and hands, conventional hand tracking systems can sufficiently track the user's body, including the positions of the user's arms and hands. There was a problem that could not be recognized and tracked.

In addition, there is a problem in that a lively virtual reality cannot be provided because only the positional relationship between the hand and the object is simply recognized in the virtual space.

Republic of Korea Patent No. 10-1748126 Republic of Korea Patent Publication No. 10-2012-0010374

An object of the present invention is to provide a hand tracking system capable of transmitting senses capable of conveying a sense of an object in a virtual space to a user while being able to interact with a user's arm and hand in a virtual space.

The hand tracking system according to an embodiment of the present invention,

a camera for acquiring depth image information including the user's arm and hand by capturing the user's arm and hand within the photographing area; a wearable terminal attached to the user's arm to measure an EMG signal of the user, update the EMG signal in real time, and transmit a sense of an object in a virtual space to the user; A first object capable of interaction is created in virtual space, and based on the depth image information and the EMG signal, it reaches the first object while being able to interact with the user's arm and hand in the virtual space, thereby providing event information. generating a second object, determining whether the first object and the second object can interact in the virtual space, and when the event information occurs, sensory information mapped to correspond to the event information a hand tracking module that generates and transmits the generated data to the wearable terminal; and generating and transmitting first creation coordinates at which the first object is to be created, size information and shape information of the first object to the hand tracking module, and determining that interaction is possible in the virtual space from the hand tracking module. and an HMD terminal outputting the first object and the second object to the virtual space.

In the hand tracking system according to an embodiment of the present invention, the wearable terminal may include an electrode unit attached to the user's arm or hand to measure EMG signals of the user's arm and hand; an EMG collecting unit that collects EMG signals measured from the electrode unit; a tracking sensor unit installed in the electrode unit and transmitting a tracking signal to the camera so that the camera tracks the arm and hand of the user; a detection sensor unit installed on the electrode unit and receiving a photographing signal generated from the camera; a sense delivery unit that transmits senses to the user based on the sense information generated by the hand tracking module; a communication unit that transmits the EMG signals collected from the EMG collection unit to the hand tracking module; and a controller configured to measure the EMG signal from the electrode unit when the detection sensor unit receives a photographing signal.

In the hand tracking system according to an embodiment of the present invention, the sense transmission unit includes a vibration generator installed therein to provide vibration to the user by operating the vibrator when the event information is related to vibration; When a heater is installed and the event information is related to heat, a heat generator operates the heater to provide heat to the user, and a speaker is installed inside to operate the speaker to provide heat to the user when the event information is related to sound. It may include a sound generator that provides.

In the hand tracking system according to an embodiment of the present invention, the sense transmission unit,

Depending on the type of event information, at least two or more of the vibration generator, the heat generator, and the sound generator may be operated in combination to provide a complex sensation.

In the hand tracking system according to an embodiment of the present invention, the hand tracking module may include: a communication unit configured to receive the depth image information, the EMG signal, the first generated coordinates, and size information and shape information of the first object; a first object processor configured to determine the size and shape of the first object based on the size and shape information of the first object, and to create the first object having the determined size and shape at the first creation coordinates; a coordinate detector configured to detect second creation coordinates at which the second object is to be generated from the depth image information; an EMG signal pattern storage unit pre-stored at least one EMG signal pattern for the user's arm and hand; a second object processing unit that compares the EMG signal with the previously stored EMG signal pattern to determine the size and shape of the second object, and creates the second object having the determined size and shape at the second creation coordinates; a control unit that determines whether the event information occurs when the second object reaches the first object; and a sensory processing unit configured to generate sensory information mapped to the event information when the event information occurs.

In the hand tracking system according to an embodiment of the present invention, the sensory processing unit may display physical characteristics according to an event of the second object with respect to each other of different types of first objects or with respect to an arbitrary first object, together with sensory information. and a sensory information database stored by mapping, wherein the control unit, when the event information occurs, transmits information on the first object, the first generated coordinates, the second object, and the second generated coordinates to the HMD device. and control to transmit the sensory information to the wearable terminal.

In the hand tracking system according to an embodiment of the present invention, the sensory information database may store sensory information reflecting physical characteristics of the virtual space.

In the hand tracking system according to an embodiment of the present invention, the sensory processing unit is provided with an object detection model, and determines an object in the virtual space by itself to determine whether the first objects are related to each other or to an arbitrary first object. It is possible to process the event of the second object for

In the hand tracking system according to an embodiment of the present invention, when the event information does not occur, the control unit causes the first object to be deleted from the first created coordinates or the second created coordinates to be deleted. The object is deleted, and after the first created coordinates are adjusted from the HMD terminal, the first object processing unit regenerates the first object at the adjusted first created coordinates, or the coordinate detector detects the second created coordinates. After the coordinates are adjusted, the second object processing unit may regenerate the second object at the adjusted second creation coordinates.

In the hand tracking system according to an embodiment of the present invention, the wearable terminal is formed in the form of a glove, and the vibration generator and the heat generator may be installed on a palm surface or a back surface of the glove-type wearable terminal.

Other specific details of implementations according to various aspects of the present invention are included in the detailed description below.

According to an embodiment of the present invention, an object capable of interacting with the user's arm and hand is created in a virtual space based on depth image information and EMG signals of the user's arm and hand, and the user has a sense of the object in the virtual space. It is possible to deliver a more lively virtual reality.

1 is a schematic diagram showing a hand tracking system according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a camera.
3 is a perspective view illustrating a wearable terminal.
4 is a block diagram illustrating the configuration of a wearable terminal.
5 is a flowchart illustrating a process of tracking a user's arm and hand and a process of measuring an EMG signal.
6 is a block diagram showing a configuration in which the LED unit is interlocked to generate light.
7 is a block diagram showing the configuration of an HMD terminal.
8 is a block diagram showing the configuration of a hand tracking module.
9 is a flowchart illustrating a process of creating a first object and a second object.
10 is a flowchart illustrating an operation process of a control unit of a hand tracking module according to whether event information is generated.
11 is an image showing a virtual space output through an output unit of an HMD terminal.
12 is a schematic diagram showing an electrode unit according to another embodiment of the present invention.
13 is a block diagram illustrating a configuration in which a first LED unit and a second LED unit are interlocked to generate light.
14 is a diagram illustrating a computing device according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be exemplified and described in detail in the detailed description. However, it should be understood that this is not intended to limit the present invention to specific embodiments, and includes all transformations, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In the present invention, terms such as 'include' or 'having' are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded. Hereinafter, a hand tracking system capable of sensory transmission according to an embodiment of the present invention will be described with reference to the drawings.

1 is a schematic diagram showing a hand tracking system according to an embodiment of the present invention.

Referring to FIG. 1 , a hand tracking system according to an embodiment of the present invention includes a camera 100, a wearable terminal 200, an HMD terminal 300, and a hand tracking module 400.

In the hand tracking system according to an embodiment of the present invention, a user attaches a wearable terminal 200 that measures EMG signals of the arm and hand to an arm, and uses a camera 100 to obtain depth image information of the user's arm and hand. A first object that can be acquired and can be interacted with in the virtual space 331 and the virtual space 331 based on the depth image information and the EMG signal through interworking between the HMD terminal 300 and the hand tracking module 400 ( 332) and the second object 333 are created to provide hand tracking to the user. At this time, the hand tracking module 400 transmits sensory information about the user's object according to the interaction between the first object 332 and the second object 333 to the wearable terminal 200, and the wearable terminal 200 A sense corresponding to the sensory information is provided to the user.

2 is a block diagram showing the configuration of the camera 100.

Referring to FIG. 2 , a camera 100 is provided to obtain depth image information on a user's arm and hand, and includes a depth camera unit 110, a tracking unit 120, a capturing area adjusting unit 130, and a capturing image. It may include a signal generator 140, a communication unit 150 and a control unit 160.

The depth camera unit 110 is installed on top of a structure (not shown) to be positioned above the user's arms and hands so that the user is positioned below, and captures the user's arms and hands within the shooting area to capture the user's arms and hands. Depth image information including at least a hand is acquired. Here, the depth camera unit 110 acquires depth data for each pixel of the photographed image. Depth means the distance from the user's arm and hand, which are objects, to the depth camera unit 110, and the depth image information may be expressed in different colors for each depth.

In addition, the depth camera unit 110 may include time-of-flight (TOF) (scanning TOF or array TOF), structured light, radar speckle pattern technology, a stereo camera, and an active stereo camera to acquire depth image information. can be implemented as

The tracking unit 120 receives a tracking signal from the tracking sensor unit 230 of the wearable terminal 200, which will be described later, before photography for obtaining depth image information from the depth camera unit 110 is performed, and the user's arm and hand to track Although this tracking unit 120 is not shown in the drawing, it is installed on one side of the depth camera unit 110 to track the user's arms and hands.

The capturing area adjusting unit 130 adjusts the capturing area of the depth camera unit 110 so that the arm and hand of the user tracked by the tracking unit 120 are positioned within the capturing area. Here, the capturing area adjusting unit 130 adjusts the capturing area of the depth camera unit 110 using zoom-in or zoom-out. However, the adjustment of the capture area is not limited to zoom-in or zoom-out, and the capture area of the depth camera unit 110 may be adjusted through rotation of the camera lens.

When the depth camera 110 photographs the user's arm and hand, the photographing signal generating unit 140 generates a photographing signal and transmits the photographing signal to the detection sensor unit 240 of the wearable terminal 200 described below.

The communication unit 150 transmits depth image information obtained from the depth camera unit 110 to the hand tracking module 400 . Here, the communication unit 150 transmits depth image information to the hand tracking module 400 through communication with the communication unit 410 of the hand tracking module 400, which will be described later, and a communication method for this is Wi-Fi, Bluetooth, ZigBee, etc. A wireless method of can be used. However, the communication unit 150 does not limit the communication method to the wireless method, and may transmit depth image information using a wired method.

The control unit 160 not only controls components of the camera 100, but also controls the shooting area of the depth camera unit 110 by the shooting area control unit 130 so that the user's arms and hands are positioned within the shooting area. , The depth camera unit 110 is controlled so that photography is performed from the depth camera unit 110 .

3 is a perspective view showing a wearable terminal, FIG. 4 is a block diagram showing the configuration of the wearable terminal, and FIG. 5 is a flowchart showing a process of tracking a user's arm and hand and measuring an EMG signal, 6 is a block diagram showing a configuration in which the LED unit is interlocked to generate light.

Referring to FIGS. 3 to 6 , the wearable terminal 200 is provided to measure and collect EMG signals of the user's arm and hand and to convey the sense of objects in the virtual space to the user, and the electrode unit 210 , EMG collection unit 220, tracking sensor unit 230, detection sensor unit 240, LED unit 250, sensory transmission unit 260, communication unit 270, and control unit 280 may be included. . The wearable terminal 200 may be a band type as shown in FIG. 3 or a glove type as shown in FIG. 12 .

The electrode unit 210 is attached to the user's arm or hand to measure the EMG signal of the user's arm and hand, and includes a plurality of electrodes 210a and 210b coupled to a band or glove-type accessory to be attached to the user's arm. , 210c, 210d, ??). That is, the electrode unit 210 may measure EMG signals of the user's arms and hands using a plurality of electrodes 210a, 210b, 210c, 210d, ??.

The EMG collection unit 220 is configured in one electrode 210b of the electrode units 210 and collects EMG signals measured from the electrode unit 210 . In this way, the electrode units 210 may be connected to each other through a circuit capable of transmitting and receiving EMG signals so that the EMG collecting unit 220 collects the EMG signals.

The tracking sensor unit 230 is configured on the electrode 210a, which is one of the electrode units 210, and transmits a tracking signal to the tracking unit 120 so that the tracking unit 120 tracks the user's arm and hand. When receiving the tracking signal, the tracking unit 120 detects the tracking signal and tracks the user's arm and hand.

The detection sensor unit 240 is composed of an electrode 210a, which is one of the electrode units 210, and the electrode unit 210 takes pictures from the recording signal generator 140 so that the EMG signals for the user's arm and hand are measured. receive a signal That is, when the detection sensor unit 240 receives a photographing signal, the electrode unit 210 measures the EMG signal for the user's arm and hand.

The LED unit 250 is configured on the electrode 210a, which is one of the electrode units 210, and when the EMG signal is measured from the electrode unit 210, the tracking sensor unit 230 transmits the tracking signal to the tracking unit 120. Light is emitted when at least one of the time of transmitting to and when the detection sensor unit 240 receives a photographing signal from the photographing signal generator 140 .

Furthermore, the LED unit 250 emits light in different colors or flickers the light in different ways so that the user can determine three cases.

As a specific example, the LED unit 250 generates green light when the EMG signal is measured from the electrode unit 210, and emits red light when the tracking sensor unit 230 transmits the tracking signal to the tracking unit 120. Light is generated, and when the detection sensor unit 240 receives a photographing signal from the photographing signal generator 140, blue light may be generated.

As another example, when the EMG signal is measured from the electrode unit 210, the LED unit 250 blinks light at an interval of 1 second or less, and the tracking sensor unit 230 transmits the tracking signal to the tracking unit 120. When the light is blinking at intervals of 2 seconds or more and 3 seconds or less, and when the detection sensor unit 240 receives a photographing signal from the photographing signal generator 140, the light may blink at intervals of 4 seconds or more and 5 seconds or less. there is.

The sense transmission unit 260 conveys the sense of an object in the virtual space to the user. The sensory transmission unit 260 generates sound, vibration, heat, and the like based on the sensory information generated by the sensory processing unit 460 of the hand tracking module 400 and transmits it to the user.

When the second object (333, see FIG. 11) interacting with the user's arm and hand reaches the first object (332, see FIG. 11) and event information (hit, collision, throw, grip, etc.) occurs, senses The transmission unit 260 may provide a more lively virtual reality by transmitting senses (sound, vibration, heat, etc.) mapped to correspond to each event information to the user.

Specifically, the sense transmission unit 260 may include a vibration generator 261, a heat generator 262, and a sound generator 263. Of course, it is not limited thereto, and a configuration capable of transmitting senses may be added.

The vibration generator 261 has a small vibrator installed inside, and when the event information generated by the sensory processing unit 460 is related to vibration (eg, hit/collision), the vibrator operates to the user's wrist (band type). ) or a palm (glove type) to provide a user with a sense of vibration. At this time, the type and strength of vibration, the vibration time, etc. may be adjusted according to the type of event information.

The heat generator 262 has a small heater installed inside, and when the event information generated by the sensory processing unit 460 is related to heat (for example, gripping a first object with heat), the heater is operated to move the user's wrist. (band type) or the palm (glove type) to provide a sense of heat to the user. In this case, the intensity and duration of the heat may be adjusted according to the type of event information.

The sound generator 263 has a small speaker installed therein, and when the event information generated by the sensory processing unit 460 is related to sound (eg, hit/collision), the speaker is operated to provide sound to the user. can In this case, the type, intensity, and duration of sound may be adjusted according to the type of event information.

Of course, depending on the type of event information, at least two or more of the vibration generator 261, the heat generator 262, and the sound generator 263 may be operated in combination to provide a complex sensation.

For example, when gripping a first object with heat, vibration may be transmitted to provide the user with a feeling of grip, and heat may be transmitted to provide a thermal sensation. In addition, when hitting the first object hitting the baseball, vibration may be transmitted to provide a sense of impact to the user, and sound may be transmitted to provide a sense of sound.

The communication unit 270 is configured in the electrode 210c, which is one of the electrode units 210, and transmits the EMG signal collected from the EMG collecting unit 220 to the hand tracking module 400. Here, the communication unit 270 transmits the collected EMG signals to the hand tracking module 400 through communication with the communication unit 410 of the hand tracking module 400 . In addition, the communication unit 270 may use a wireless method such as Wi-Fi, Bluetooth, or ZigBee, and may use a wired method instead of a wireless method.

In addition, the communication unit 270 receives sensory information generated by the sensory processing unit 460 through communication with the communication unit 410 of the hand tracking module 400 and transfers it to the control unit 280 .

The control unit 280 is formed on the electrode 210d, which is one of the electrode units 210, and not only controls components of the wearable terminal 200, but also allows the detection sensor unit 240 to take pictures from the shooting signal generator 140. When the signal is received, the electrode unit 210 is controlled so that the EMG signal is measured from the electrode unit 210 .

In addition, the controller 280 controls the operation of at least one of the vibration generator 261, the heat generator 262, and the sound generator 263 based on sensory information generated by the sensory processing unit 460.

Among the components of the wearable terminal 200, the tracking sensor unit 230, the detection sensor unit 240, and the LED unit 250 may be installed on one electrode unit 210a, and the other component, the electromyogram collection unit 220 ), the communication unit 270, and the control unit 280 may be installed on different electrode units 210b, 210c, and 210d, respectively.

7 is a block diagram showing the configuration of an HMD terminal, FIG. 8 is a block diagram showing the configuration of a hand tracking module, and FIG. 9 is a flowchart showing a process of generating a first object and a second object. 10 is a flowchart showing the operation process of the control unit of the hand tracking module according to whether event information is generated, and FIG. 11 is an image showing a virtual space output through the output unit of the HMD terminal.

Referring to FIGS. 7 and 11 , the HMD terminal 300 is interlocked with the hand tracking module 400 to provide virtual reality content including a virtual space 331 to the user, and the first object generator 310 , It may include a communication unit 320 and an output unit 330.

The first object creation unit 310 determines and creates first creation coordinates at which the first object 332 capable of interaction in virtual space is to be created, size information and shape information of the first object 332 . Here, the first object 332 means an object in virtual space, and the first created coordinates are updated when the first object 332 moves based on event information such as hitting, collision, throwing, and gripping in virtual space. , Size information and shape information of the first object 332 are updated when the size and shape of the first object 332 is changed based on event information such as combining, absorbing, cutting, or deleting in virtual space.

The communication unit 320 transmits the first creation coordinates generated by the first object generator 330 and the size information and shape information of the first object 332 to the hand tracking module 400 . Here, the communication unit 320 transmits the first generated coordinates and size information and shape information of the first object 332 to the hand tracking module 400 through communication with the communication unit 410 of the hand tracking module 400. Also, the communication unit 320 may use a wireless method such as Wi-Fi, Bluetooth, or ZigBee, and may use a wired method instead of a wireless method.

Furthermore, the communication unit 320 receives information on the first object 332 and the second object 333 generated by the hand tracking module 400 from the communication unit 410 of the hand tracking module 400. do.

As shown in FIG. 11 , the output unit 330 outputs the first object 332 and the second object 333 received by the communication unit 320 to the virtual space 331 . At this time, the output unit 330 outputs the center coordinates 331a of the virtual space 331 to the virtual space 331 together with the first object 332 and the second object 333 . Here, the center coordinates 331a mean coordinates forming the center of the x, y, and z coordinate system of the virtual space 331, and may be output to guide the user to a position in the virtual space 331.

Although the HMD terminal 300 is not shown in the drawing, a virtual space generator (not shown) that creates a virtual space 331 output by the output unit 330 and a control unit that controls components of the HMD terminal 300 (not shown) City) is preferably further included.

8 to 10, the hand tracking module 400 interworks with the HMD terminal 300 to create a first object 332 and a second object 333 capable of interaction in a virtual space, and the communication unit 410, a first object processing unit 420, a coordinate detection unit 430, an EMG signal pattern storage unit 440, a second object processing unit 450, a control unit 460, and a sensory processing unit 470. there is.

The communication unit 410 receives depth image information from the communication unit 150 of the camera 100, receives EMG signals collected from the communication unit 270 of the wearable terminal 200, and receives the EMG signal collected from the communication unit 320 of the HMD terminal 300. ), first created coordinates, size information and shape information of the first object 332 are received.

The first object processing unit 420 determines the size and shape of the first object 332 based on the size and shape information of the first object 332 received by the communication unit 410, and the communication unit 410 receives the received information. A first object 332 is created at one first creation coordinate.

The coordinate detector 430 determines the depths of the user's arm and hand from the depth camera 110 through the depth image information received by the communication unit 410 and detects second generated coordinates.

The EMG signal pattern storage unit 440 pre-stores one or more EMG signal patterns (eg, a fist clenched state, a state in which all fingers are spread, a pattern in which only the thumb and forefinger are spread and the rest of the fingers are folded, etc.).

The second object processing unit 450 determines the size and shape of the second object 333 by comparing the collected EMG signals received by the communication unit 410 with the EMG signal patterns pre-stored in the EMG signal pattern storage unit 440, , The second object 333 is created at the second generated coordinates generated by the coordinate detector 430 . Here, the comparison of the collected EMG signals and the EMG signal patterns is intended to significantly reduce errors in the EMG signals with a large gap depending on the individual user or the physical condition of the individual user. And, the second object 333 refers to an object capable of interacting with the user's arm and hand in virtual space and generating event information by reaching the first object 332 according to the update of the second created coordinates. .

As shown in FIG. 10 , the control unit 460 controls when the first object 332 is created from the first object processing unit 420 and the second object 333 is created from the second object processing unit 450 (S10). ), it is determined whether event information occurs when the second object 333 reaches the first object 332 (S20).

At this time, if event information occurs (S20-YES), the controller 460 transfers the first object 332 of the first created coordinates generated by the first object processor 420 to the virtual space 331 as the final object to be applied. The second object 333 of the second created coordinates generated by the second object processing unit 450 is determined as the final second object 333 to be applied to the virtual space 331 and transmitted to the HMD terminal 300 through the communication unit 410 (S30).

At this time, the communication unit 320 receives information on the final first object 332 and information on the second object 333 from the communication unit 410, respectively, and the output unit 330 outputs the final first object ( 332) and the second object 333 are output to the virtual space 331.

In addition, along with information on the first object and the second object, sensory information about the user's object according to the interaction between the first object 332 and the second object 333 is transmitted to the wearable terminal 200 ( S40).

In contrast, if event information does not occur (S20-NO), the first object 332 of the first object processing unit 420 is controlled by controlling the first object processing unit 420 and the second object processing unit 450, respectively. and the second object 333 of the second object processing unit 450 are deleted (S50).

Thereafter, the controller 460 transmits the deletion of the first object 332 to the first object creation unit 310 through the communication unit 410, and the deletion of the second object 333 to the second coordinate detector. forwarded to (430).

Then, the first object creation unit 310 adjusts the first creation coordinates, and the first object processing unit 420 places the first object 332 at the adjusted first creation coordinates from the first object creation unit 310. Regenerate (S60).

In addition, the coordinate detector 430 adjusts the second created coordinates, and the second object processor 450 regenerates the second object 333 at the adjusted second created coordinates from the coordinate detector 430 (S60).

In this way, when event information does not occur (S20-NO), the control unit 460 regenerates the first object 332 and the second object 333 by adjusting the first creation coordinates and the second creation coordinates, This is not limited to, and when event information does not occur (S20-NO), first creation coordinates, size information and shape information of the first object 332, second creation coordinates, and size information of the second object 333 And shape information may be adjusted to regenerate the first object 332 and the second object 333 .

As a specific example, the first object generator 310 adjusts the first creation coordinates and size information and shape information of the first object 332, and the first object processor 420 controls the first object generator 310 The size and shape of the first object 332 may be determined based on the size information and shape information of the first object 332 adjusted from , and the first object 332 may be regenerated at the adjusted first creation coordinates. .

In addition, the coordinate detection unit 430 adjusts the second generated coordinates, and the second object processing unit 450 compares the collected EMG signals with the EMG signal pattern stored in the EMG signal pattern storage unit 440 again, so that the second object 333 ), and regenerate the second object 333 at the adjusted second creation coordinates.

That is, the control unit 460 finally creates the first object 332 and the second object 333 based on the process of determination, and the final first object 332 and the second object through the communication unit 410 (333) is transmitted to the HMD terminal (300). At this time, the HMD terminal 300 outputs the final first object 332 and the second object 333 to the virtual space 331 through the output unit 330 as shown in FIG. 11 .

When the second object 333 interacting with the user's arm and hand reaches the first object 332 and event information such as hitting, collision, throwing, and gripping occurs, the sensory processing unit 470 generates each event information. By generating sensory information (sound, vibration, heat, etc.) mapped to correspond to, a more lively virtual reality is provided. This is the same even when one first object reaches another first object and event information such as hitting, collision, throwing, and gripping occurs. (For example, when a user hits a baseball with a baseball bat)

The sensory processing unit 470 generates sensory information by referring to attribute information and event information of the first object 332 . Attribute information of the first object 332 is information about the physical characteristics of the first object 332 and may be provided based on the physical characteristics of the first object in the real world. The physical characteristics may include the temperature (cold/hot), surface roughness, texture (soft/hard), and sound of hitting/collision/gripping of the first object 332 .

The sensory processing unit 470 converts physical characteristics according to each event, such as hitting, collision, throwing, and gripping of a second object to each other or to any first object, with sensory information. It includes a database of sensory information stored by mapping. When a specific event occurs for a specific first object, the sensory processing unit 470 extracts sensory information corresponding thereto from the sensory information database. In addition, the sensory information database may store sensory information reflecting the physical characteristics of the virtual space 331 .

For example, when a user (second object) hits a baseball (first object_2) with a baseball bat (first object_1) in the virtual space 331, the sensory processing unit 470 sends the user a baseball The grip feeling when gripping the bat generates sensory information representing weak vibration of the first level, and the hitting feeling when hitting the baseball with the baseball bat represents vibration and hitting sound of the second level greater than the first level. generate sensory information.

In addition, for example, when a user conducts a fire drill in the virtual space 331, in a state in which sensory information reflecting the physical characteristics (heat) of the virtual space 331 in which a fire is in progress is generated (heat generation), The feeling of grip when the user grips the fire hose generates sensory information representing weak vibration of the first level, and the feeling of discharge when discharging water through the fire hose generates vibration and sound of the second level that is greater than the first level. generate sensory information that represents Then, as time passes, when a fire is extinguished in the virtual space 331, sensory information reflecting this is generated. That is, sensory information representing a temperature of a second level during a fire in progress and a temperature of a first level lower than the second level during a fire suppression process is generated.

In addition, the sensory processing unit 470 has an object detection model, determines an object within the virtual space 331 by itself, and processes an event of a second object between first objects or an arbitrary first object. can do. The object detection model may be YOLO (Yoo Only Look Once), a single-step object detection algorithm, or Faster RCNN, a two-step object detection algorithm. Of course, the object detection model is not limited thereto.

Sensory information generated by the sensory processing unit 470 is transmitted to the wearable terminal 200 together with information on the first object and the second object.

Another embodiment of the present invention has some of the same configuration and method as the above-described embodiment, but unlike the above-described embodiment, the electrode unit for measuring the EMG signal of the user's arm and the EMG signal of the user's hand It includes an electrode unit for measuring , and there is a difference in that the LED unit for guiding whether or not to measure the EMG signal for the user's arm and the EMG signal for the user's hand are divided separately.

In addition, the wearable terminal 200 is formed in the form of a glove, and the vibration generator 261 and the heat generator 262, which are components of the sense transmission unit 260, are on the palm surface or the back surface 213 of the glove-type wearable terminal 200. It can effectively transmit vibration or thermal sensation.

Hereinafter, the description will focus on the electrode unit 210 and the LED unit 250, which are different from the first embodiment of the present invention, and a detailed description of the configuration and method overlapping with the first embodiment of the present invention is provided for convenience. I will omit it.

12 is a schematic diagram showing an electrode unit according to another embodiment of the present invention, and FIG. 13 is a block diagram showing a configuration in which a first LED unit and a second LED unit are interlocked to generate light.

Referring to FIG. 12 , the electrode unit 210 may include a first electrode unit 211 and a second electrode unit 212 .

The first electrode unit 211 is attached to the user's arm to measure the EMG signal of the user's arm. A detection sensor unit 240 and a first LED unit 251 to be described later are installed on one side of the first electrode unit 211, respectively.

The second electrode unit 212 is attached to the user's hand and measures the EMG signal of the user's hand. A tracking sensor unit 230 and a second LED unit 252 to be described later are respectively installed on one side of the second electrode unit 212 .

On the other hand, dividing the electrode unit 210 into a first electrode unit 211 and a second electrode unit 212 is obtained by measuring the EMG signal for the user's arm and the EMG signal for the user's hand with the respective electrodes. This is to measure the EMG signals of the user's arm and hand more accurately than the first embodiment of the present invention.

In addition, the first electrode unit 211 and the second electrode unit 212 measure the EMG by at least one measurement method among RMS, power spectrum, Median Frequency, Mean Frequency, RMS-probability distribution, RMS-correlation function, and MVIC Normalzation. Signals may be measured, and EMG signals of the user's arm and the user's hand may be measured in different ways.

As a specific example, the first electrode unit 211 may measure the EMG signal of the user's arm using the RMS method, and the second electrode unit 212 may measure the EMG signal of the user's hand using the RMS-probability distribution. EMG signals can be measured. However, the first electrode unit 211 and the second electrode unit 212 may measure EMG signals of the user's arm and the user's hand in the same measurement method through a design change or a setting change.

Furthermore, the first electrode unit 211 and the second electrode unit 212 may be operated through separate circuits, but may be operated as one electrode device through accessories (eg gloves, etc.) in which the circuit is configured. The EMG signal of the user's arm and the EMG signal of the user's hand may be collected from the EMG collecting unit 220 .

Referring to FIG. 13 , the LED unit 250 may include a first LED unit 251 and a second LED unit 252 .

The first LED unit 251 is installed on one side of the first electrode unit 211, and when the first electrode unit 211 measures the EMG signal of the user's arm or the detection sensor unit 240 captures a signal When a photographing signal is received from the generation unit 140, a light is generated or the light blinks.

Furthermore, the first LED unit 251 emits light in different colors or blinks the light in different ways so that the user can determine the two cases.

As a specific example, the first LED unit 251 generates green light when the first electrode unit 211 measures the EMG signal of the user's arm, and the sensor unit 240 is a photographing signal generator ( 140) generates red light when receiving a photographing signal.

As another example, the first LED unit 251 blinks light at intervals of 1 second or less when the first electrode unit 211 measures the EMG signal of the user's arm, and the sensor unit 240 transmits a photographing signal. When receiving a photographing signal from the generation unit 140, the light flickers at an interval of 2 seconds or more and 3 seconds or less.

The second LED unit 252 is installed on one side of the second electrode unit 212, and when the second electrode unit 212 measures the EMG signal of the user's hand or the tracking sensor unit 230 detects the tracking signal When transmitting to the tracking unit 120, a light is generated or a light is blinked. At this time, the second LED unit 252 generates light or blinks light in the same way or in a different way.

As a specific example, the second LED unit 252 generates green light when the second electrode unit 212 measures the EMG signal of the user's hand, and the tracking sensor unit 230 transmits the tracking signal to the tracking unit. When transmitting to (120), a red light is generated.

As another example, the second LED unit 252 blinks light at intervals of 1 second or less when the second electrode unit 212 measures the EMG signal of the user's hand, and the tracking sensor unit 230 transmits the tracking signal. When transmitting to the tracking unit 120, the light flickers at intervals of 2 seconds or more and 3 seconds or less.

On the other hand, dividing the LED unit 250 into a first LED unit 251 and a second LED unit 252, the first divided to measure the user's arm and hand, respectively, through light generation or flickering of light. This is to inform the user whether the electrode unit 211 and the second electrode unit 212 are operating.

Furthermore, the first LED unit 251 and the second LED unit 252 may be operated through separate circuits, but may be operated as one LED through an accessory (eg, gloves) in which the circuit is configured. there is.

14 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 14 may be a hardware component of the hand tracking system described in this specification.

In the embodiment of FIG. 14 , the computing device TN100 may include at least one processor TN110, a transceiver TN120, and a memory TN130. In addition, the computing device TN100 may further include a storage device TN140, an input interface device TN150, an output interface device TN160, and the like. Elements included in the computing device TN100 may communicate with each other by being connected by a bus TN170.

The processor TN110 may execute program commands stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Processor TN110 may be configured to implement procedures, functions, methods, and the like described in relation to embodiments of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of read only memory (ROM) and random access memory (RAM).

The transmitting/receiving device TN120 may transmit or receive a wired signal or a wireless signal. The transmitting/receiving device TN120 may perform communication by being connected to a network.

Meanwhile, the present invention may be implemented as a computer program. The present invention can be combined with hardware and implemented as a computer program stored in a computer-readable recording medium.

Methods according to embodiments of the present invention may be implemented in the form of programs readable by various computer means and recorded on computer-readable recording media. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination.

Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software.

For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CDROMs and DVDs, and magneto-optical media such as floptical disks. optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of the program command may include a high-level language that can be executed by a computer using an interpreter, as well as a machine language generated by a compiler.

These hardware devices may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

In the above, one embodiment of the present invention has been described, but those skilled in the art can add, change, delete, or add components within the scope not departing from the spirit of the present invention described in the claims. The present invention can be variously modified and changed by the like, and this will also be said to be included within the scope of the present invention.

100: camera
110: depth camera unit 120: tracking unit
130: shooting area control unit 140: shooting signal generator
150: communication unit 160: control unit
200: wearable terminal
210: electrode unit 220: EMG collection unit
230: tracking sensor unit 240: detection sensor unit
250: LED unit 260: sensory transmission unit
270: communication unit 280: control unit
300: HMD terminal
310: first object creation unit 320: communication unit
330: output unit
400: hand tracking module
410: communication unit 420: first object processing unit
430: coordinate detection unit 440: EMG signal pattern storage unit
450: second object processing unit 460: control unit
470: sensory processing unit

Claims (10)

a camera for acquiring depth image information including the user's arm and hand by capturing the user's arm and hand within the photographing area;
a wearable terminal attached to the user's arm to measure an EMG signal of the user, update the EMG signal in real time, and transmit a sense of an object in a virtual space to the user;
A first object capable of interaction is created in virtual space, and based on the depth image information and the EMG signal, it reaches the first object while being able to interact with the user's arm and hand in the virtual space, thereby providing event information. generating a second object, determining whether the first object and the second object can interact in the virtual space, and when the event information occurs, sensory information mapped to correspond to the event information a hand tracking module that generates and transmits the generated data to the wearable terminal; and,
A first creation coordinate at which the first object is to be created, size information and shape information of the first object are generated and transmitted to the hand tracking module, and the hand tracking module determines that the interaction is possible in the virtual space. an HMD terminal outputting the first object and the second object to the virtual space;
A hand tracking system capable of transmitting senses including.
The method according to claim 1, wherein the wearable terminal,
an electrode unit attached to the user's arm or hand to measure EMG signals of the user's arm and hand;
an EMG collecting unit that collects EMG signals measured from the electrode unit;
a tracking sensor unit installed in the electrode unit and transmitting a tracking signal to the camera so that the camera tracks the arm and hand of the user;
a detection sensor unit installed on the electrode unit and receiving a photographing signal generated from the camera;
a sense delivery unit that transmits senses to the user based on the sense information generated by the hand tracking module;
a communication unit that transmits the EMG signals collected from the EMG collection unit to the hand tracking module; and
a control unit configured to measure the EMG signal from the electrode unit when the detection sensor unit receives a photographing signal;
A hand tracking system capable of transmitting senses including.
The method according to claim 2, wherein the sensory transfer unit,
A vibration generator having a vibrator installed therein to provide vibration to the user by operating the vibrator when the event information is related to vibration;
A heat generator having a heater installed therein to provide heat to the user by operating the heater when the event information is related to heat;
A sound generator having a speaker installed therein to provide sound to the user by operating the speaker when the event information is related to sound
A hand tracking system capable of transmitting senses including.
The method according to claim 3, wherein the sensory transfer unit,
A hand tracking system capable of transmitting senses by operating at least two or more of the vibration generator, the heat generator, and the sound generator in a complex manner according to the type of the event information to provide a complex sensation.
The method according to claim 1, wherein the hand tracking module,
a communication unit configured to receive the depth image information, the EMG signal, the first generated coordinates, and size information and shape information of the first object;
a first object processor configured to determine the size and shape of the first object based on the size and shape information of the first object, and to create the first object having the determined size and shape at the first creation coordinates;
a coordinate detector configured to detect second creation coordinates at which the second object is to be generated from the depth image information;
an EMG signal pattern storage unit pre-stored at least one EMG signal pattern for the user's arm and hand;
a second object processing unit that compares the EMG signal with the previously stored EMG signal pattern to determine the size and shape of the second object, and creates the second object having the determined size and shape at the second creation coordinates;
a control unit that determines whether the event information occurs when the second object reaches the first object; and,
a sensory processing unit generating sensory information mapped to correspond to the event information when the event information occurs;
A hand tracking system capable of transmitting senses including.
The method of claim 5,
The sensory processing unit includes a sensory information database in which physical characteristics according to events of the second object for each other of different types of first objects or for an arbitrary first object are mapped with sensory information and stored,
When the event information occurs, the control unit controls to transmit information on the first object, the first generated coordinates, the second object, and the second generated coordinates to the HMD terminal, and transmits the sensory information to the HMD terminal. A hand tracking system capable of sensory transmission that is controlled to be transmitted to a wearable terminal.
The method according to claim 6, wherein the sensory information database,
A hand tracking system capable of transmitting senses that stores sensory information reflecting the physical characteristics of the virtual space.
The method according to claim 5, wherein the sensory processing unit,
A hand tracking system capable of sensory transfer by having an object detection model and determining an object in the virtual space by itself and processing an event of a second object between first objects or an arbitrary first object.
The method according to claim 5, wherein the control unit,
When the event information does not occur, the first object is deleted from the first created coordinates or the second object is deleted from the second created coordinates,
After the first creation coordinates are adjusted from the HMD terminal, the first object processing unit regenerates the first object at the adjusted first creation coordinates, or after the coordinate detection unit adjusts the second creation coordinates, A hand tracking system capable of sensory transfer such that the second object is regenerated at the adjusted second creation coordinates from the second object processor.
The method of claim 3,
The wearable terminal is formed in the form of a glove, and the vibration generator and the heat generator are installed on the palm surface or the back of the hand of the glove-type wearable terminal.

Images