KR102368929B1 - Voice recognition-based shortcut key XR interactive interworking system and the method using it - Google Patents

Abstract

The present invention relates to a voice recognition-based shortcut key XR interactive interworking system. More specifically, the present invention relates to the voice recognition-based shortcut key XR interactive interworking system that provides a real-time voice recognition interactive experience through input and XR content output based on a motion voice. The present invention comprises: a voice recognition-based shortcut key for a realistic experience of content using a voice recognition-based shortcut key for a part of movement with a wide activity space; a voice recognition part that recognizes the voice recognition-based shortcut key that defines the part of the movement with the wide activity space from an inputted voice from a voice input part; and an XR content module that generates XR content through the motion-based information and the voice recognition-based shortcut key.

Description

Voice recognition-based shortcut key XR interactive interworking system and the method using it}

The present invention relates to a voice recognition-based shortcut key XR interactive interworking system and an interworking method using the same, and more particularly, to a voice recognition-based shortcut key that provides a real-time voice recognition interactive experience through an input based on a motion voice and output of XR content It relates to the XR interactive interlocking system.

Korean Patent Application Laid-Open No. 2014-0061009 discloses the steps of obtaining an image including an object; recognizing at least one or more objects included in the acquired image; detecting augmented reality information related to the recognized object; inputting voice information and comparing it with the augmented reality information; and generating a user interface that separates and displays augmented reality matching the voice information.

The conventional augmented reality system is a hybrid augmented reality system in which object information and voice are matched, and three-dimensional content matched by recognizing a photographed marker or voice is displayed, and narration can feel real.

However, the precision of moving the object according to the recorded marker or voice was very poor, and in particular, there were many insufficient points to control the instantaneous movement direction or speed of the object.

In addition, Korean Patent Registration No. 0432373 includes the steps of: receiving a voice signal from a user through a voice input device connected to a middleware system; accessing a voice database to search for and store string data corresponding to the received voice signal; retrieving message data corresponding to the stored character string data in a message database storing message data in which 3D data can be recognized. S30); controlling the behavior of the 3D object according to the retrieved message data; and updating and displaying the behavior of the 3D object whose behavior is controlled in real time, wherein the retrieving the message data includes: a first step of retrieving message data corresponding to the character string data in a message database; And after deleting one last character of the character string data, a second step of searching the message database for message data corresponding to the character string data from which one last character has been deleted; by performing the first step on the stored character string data, including If the corresponding message data is retrieved, the retrieved message data is returned. If the corresponding message data is not retrieved, the second step is performed until one character remains in the character string data.

However, the prior art has the inconvenience of not recognizing a fast rotation operation or taking a lot of time to analyze complex character string data, and having to receive an additional input if the analysis fails.

The present invention has been made to solve the above problems, and the present invention is a vision method using a camera to develop and implement motion-based XR content such as soccer or boxing that reflects the user's movement in front, back, left and right in front of the camera. It is difficult to apply right step, left/right rotation, running, etc., but as a solution to this, a technology that enhances the realistic experience of contents by using voice recognition-based voice shortcut keys for a part of movement with a wide active space is provided. purpose is to

In addition, an object of the present invention is to provide a technique for defining, learning, and registering a voice recognition-based shortcut key according to a user's specific operation, since an operation may be defined by voice and/or sound by user definition.

In order to solve the above problems, the present invention provides a voice recognition-based shortcut key for a sensory experience of content by using a voice recognition-based shortcut key for a part of a movement with a wide activity space; a voice recognition unit for recognizing a voice recognition-based shortcut key defining a part of a movement with a wide active space in the voice input from the voice input unit; and an XR content module for generating XR content through the motion-based information and voice recognition-based shortcut keys.

The voice recognition unit interlocks with the XR contents to perform forward (F), backward (B), left (L), right (R) direction movement among the actions to implement the same as in the real situation, respectively “F”, “B”, It is divided into “L” and “R” and is defined as a voice shortcut key and stored in the voice database 140 .

After being stored in the voice database, when moving forward (F), backward (B), left (L), right (R) direction and rotating from left to right (LR), when rotating from right to left (RL), Defines a shortcut based on speech recognition for motion required for fast movement (FF) and stores it in the database.

The voice recognition unit recognizes the movement in the future as FF, and is linked with the XR content to implement the same as the real situation to enhance the sensory experience of the content.

The voice is inputted through the voice recognition unit, and it is implemented in the same way as the actual situation by interworking with XR contents based on the volume of the voice, the accuracy of the voice, and the speed of the voice.

It sets a bundle of voice waveforms or texts in the voice database, stores it in the voice database, divides the voice waveform input through the voice input unit into specific voice waveforms or texts with the voice recognition unit, and works with XR contents to implement the same as the real situation. .

movement execution or state change) according to the message data received through the voice input unit, fast forward (FF), fast backward (BB), quickly left and right (RL), quickly left and right (LR), quickly left It can be executed by dividing it into forward (FF), fast left backward (BB), fast left forward (LL), and fast right forward (RR).

In the search for a voice waveform or text input through the voice input unit, if there is a character string matching the voice signal related to the voice waveform or text in the voice database, the XR content module interlocks the character string corresponding to the character signal with the XR content. Implement the same situation.

Based on the voice command through the voice input unit, the time between the user input before and after is calculated, and the user input with a small time difference and the voice command are combined, and the XR content module interworks with the XR content to execute control in the same way as in the real situation.

When a voice command is executed through the voice input unit, it is determined whether a minimum time t1 has elapsed after each command is generated, and when the voice command is executed, each command is executed.

After a specific gesture is input to the camera, when a voice command is input, a voice command word is determined by searching for a similar word from a voice database corresponding to the voice command, and the voice command word is executed in response to the user input.

and a command execution unit that extracts a group of similar words corresponding to an intuitive voice command through the voice input unit, determines the corresponding voice command, combines the determined voice command, and executes the execution corresponding to the final command.

A command determination unit that searches for a similar word for the voice input through the voice input unit from a voice database and determines the command by searching together a plurality of intuitive analogues related to the command for the desired input, thereby eliminating the need to memorize the command. include

The voice input unit includes one main microphone and one or more auxiliary microphones, and the voice recognition unit compares the voice information collected by the one or more auxiliary microphones with each other and filters the noise of the voice information to recognize the actually pronounced voice information do.

The XR content module learns a voice recognition-based shortcut key by voice and/or sound by user definition, and controls to display an operation according to the voice recognition-based shortcut key on a display device.

The present invention includes the steps of receiving a voice signal from a user through a voice input unit (S110); Recognizing a voice recognition-based voice shortcut key defining a part of movement in the voice input from the voice input unit through the voice recognition unit (S120); and generating XR content corresponding to the voice recognition-based voice shortcut key from the voice recognition unit through the XR content module (S130).

The voice recognition unit interlocks with the XR contents to perform forward (F), backward (B), left (L), right (R) direction movement among the actions to implement the same as in the real situation, respectively “F”, “B”, dividing into "L" and "Al", defining a voice shortcut key and storing it in a voice database; Required for the forward (F), backward (B), left (L), right (R) direction movement and for left-to-right rotation (LR), right-to-left rotation (RL), and fast movement (FF) Including; defining a shortcut key based on motion speech recognition and storing it in a database.

The present invention includes the steps of defining a voice recognition-based shortcut key by voice and/or sound by user definition; Learning a sound that can be expressed differently for each person; and displaying, by the XR content module, an operation by various sounds on a display device according to the voice recognition-based shortcut key.

The technology disclosed herein may have the following effects. However, this does not mean that a specific embodiment should include all of the following effects or only the following effects, so the scope of the disclosed technology should not be construed as being limited thereby.

The present invention, which is made as described above, uses a vision method using a camera to develop and implement motion-based XR content such as soccer or boxing that reflects the user's movement, and provides forward/backward/left/right steps in front of the camera, rotation of the direction, and running activities. Easy to apply

In addition, the present invention can enhance the sensory experience of the content by implementing the same as the actual situation by interworking with the XR content by speaking to the voice input unit in FF when stepping on the next two steps.

In addition, the present invention can determine a voice command word by searching for a similar word, combine the determined voice command, and execute corresponding to the final command, so that the voice can be recognized more precisely and implemented in the same way as the actual situation.

In addition, the present invention maximizes convenience by eliminating the need for a user to memorize a command by determining a command by searching for a plurality of related similar words together.

In addition, the present invention can define and learn the voice recognition-based shortcut key according to the user's specific voice and/or sound according to the user's specific operation, so that the voice and/or sound can be recognized more precisely and the same as the actual situation. Implemented and maximized convenience.

1 is a view showing the overall configuration according to an embodiment of the present invention.
2 is a view showing a configuration in which a camera is added according to an embodiment of the present invention.
3 is a diagram showing an internal structure of a voice database according to an embodiment of the present invention.
4 is a diagram showing a structure of combining voices in a voice database according to another embodiment of the present invention.
5 is a diagram showing a structure of combining voices in a voice database according to another embodiment of the present invention.
6 is a diagram showing a structure of combining voices in a voice database before a predetermined time according to another embodiment of the present invention.
7 is a diagram showing a structure of combining voices in a voice database after a predetermined time according to another embodiment of the present invention.
8 is a diagram showing a structure for combining voices in a voice database according to another embodiment of the present invention.
9 is a diagram showing an interworking method using a voice recognition-based shortcut key XR interactive interworking system according to another embodiment of the present invention.

In order to fully understand the present invention, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Embodiments of the present invention may be modified in various forms, and the scope of the present invention should not be construed as being limited to the embodiments described in detail below. This example is provided to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shapes of elements in the drawings may be exaggerated to emphasize a clearer description. It should be noted that the same members in each drawing are sometimes shown with the same reference numerals. In addition, detailed descriptions of well-known functions and configurations determined to unnecessarily obscure the gist of the present invention will be omitted.

As shown in FIG. 1, the present invention includes: a voice input unit 110 for receiving a voice signal from a user; a voice recognition unit 120 for recognizing a voice recognition-based voice shortcut key defining a part of movement in the voice input from the voice input unit; an XR content module 130 for generating XR content corresponding to the voice recognition-based voice shortcut key from the voice recognition unit 120; voice database 140; etc.

Here, in the XR content generated by the XR content module 130, extended reality (eXtended Reality) is called XR, augmented reality (AR, Augmented Reality), virtual reality (VR, Virtual Reality), mixed reality (MR, Mixed Reality) ) refers to content that encompasses all technologies.

Also, the voice input unit 110 may be implemented as a portable microphone or a wearable voice input device, but is not limited thereto, and may be implemented as various devices capable of detecting a user's voice.

The voice input unit 110 is preferably connected to the voice recognition unit 120 through a wireless communication network, and the wireless communication network can be implemented using at least one of a short-range communication technology and a long-distance communication technology.

As the short-range communication technology, for example, CAN communication, Wi-Fi, Wi-Fi Direct, Bluetooth, Zigbee, Bluetooth Low Energy ) or NFC (Near Field Communication) may be employed.

In addition, as the long-distance communication technology, a communication technology based on a mobile communication standard such as 3GPP, 3GPP2, or WiMAX series may be employed.

The voice input unit 110 may obtain voice data by receiving the user's voice and converting the received voice into an electrical signal.

The voice data is transmitted to the voice recognition unit through a wireless communication network.

According to an embodiment, the voice input unit may continuously or periodically receive voice according to a predefined setting and transmit it to the voice database 140 to continuously store voice data.

As shown in FIG. 2 , in the present invention, it is difficult to capture the left and right rotation with the camera 160, so a voice shortcut key may be linked to compensate for this.

The voice recognition-based voice shortcut key for part of the movement with a wide active space consists of a voice string, which is character string data input from the user, for example, forward (F), backward (B), left (L), right (R). The direction movement is divided into “F”, “B”, “L”, and “R” voices, respectively, and defined as voice shortcut keys and stored in the voice database 140 .

One of the forward (F), backward (B), left (L), and right (R) voices of the voice input unit 110 for the voice recognition unit 120 to be linked with the XR content to implement the same as the actual situation. The above voice shortcut keys are individually recognized.

For example, in the case of forward (F), a shortcut is defined based on the action voice consisting of the “F” pronunciation required to move forward.

The voice recognition unit 120 sets the voice shortcut keys as forward (F), backward (B), left (L), right (R) and stores them in the voice database 140, L when rotating left, R when rotating right , when two steps are taken in the future, if you speak to the voice input unit 110 in FF, the voice recognition unit 120 works in conjunction with the XR content to implement the same as the actual situation, thereby enhancing the sensory experience of the content.

If the user wants to control the operation by inputting his or her voice into the voice input unit 110, the voice recognition unit 120 sends a search to the voice database 140 storing the voice shortcut key, and XR content module 130 converts the corresponding movement (forward, backward, left, and right direction movement) into XR content and displays it on the display device 150 .

Therefore, the voice recognition-based shortcut key XR interactive interworking system according to the present invention recognizes the user's intention to move based on the user's voice and automatically executes the function of the XR content being executed, thereby providing various activity information suitable for the user's intention. can be provided to users.

The additional display device 150 is a device for displaying the above-described XR contents in various display methods such as a screen or a hologram.

As shown in FIG. 3, the present invention stores the voice shortcut keys composed of forward (F), backward (B), left (L), and right (R) input through the voice input unit 110 as voice waveforms or text. and a voice database 140 to

Alternatively, in the voice database 140 , when rotating from left to right (LR), when rotating from right to left (RL), etc. as analogous words are also stored as voice waveforms or text.

According to an embodiment, it is possible to enhance the sensory experience of the content by receiving a voice input through the voice recognition unit and linking with the XR content based on the size of the voice, the accuracy of the voice, and the speed of the voice to implement the same as the actual situation. there is.

For example, a voice waveform or text related to the volume, accuracy of voice, and speed of voice is stored in the voice database 140 in advance so that the voice recognition unit 120 can recognize it.

According to an embodiment, as shown in FIG. 4 , by setting a word list bundle in the voice database 140 and interlocking with the XR content, it is possible to implement the same as the actual situation to enhance the sensory experience of the content.

The word list bundle is a bundle of voice waveforms or text, and stores bundles such as fast forward (FF), fast backward (BB), quickly left (LL), and quickly right (RR) in the voice database 140, and the voice After dividing the voice waveform input through the input unit 110 into a specific voice waveform or text by the voice recognition unit 120, it can be implemented in the same way as the actual situation by interworking with XR contents.

5 and 6, according to the message data received through the voice input unit, related motion execution or state change) is executed, for example, fast forward (FF) or fast backward (BB). ), quickly left and right (RL), quickly left and right (LR), quickly left forward (FF), quickly left backward (BB), quickly left forward (LL), quickly right forward (RR), etc. By implementing the same as the situation, it is possible to enhance the immersive experience of the content.

According to an embodiment, in the search for a voice waveform or text input through the voice input unit, if there is a character string matching the voice signal related to the voice waveform or text in the voice database 140, the character string corresponding to the character signal is XR content The module 130 is implemented in the same way as the actual situation by interworking with the XR content.

According to an embodiment of the present invention, an operation may be defined by voice and/or sound by user definition.

In order to provide a voice recognition-based shortcut key according to sound that comprehensively encompasses vocalizations that can be expressed differently for each user (eg, dialect, specific sound, frequency of voice, etc.), the content learned from sound is provided to the XR content module 130 Therefore, the operation can be defined not only by the user's voice but also by various sounds.

Therefore, it is possible to more precisely identify the voice and/or sound to realize the same as the actual situation and maximize the convenience.

According to an embodiment, as shown in FIG. 7 , the time between the user input before and after is calculated based on the voice command through the voice input unit, and the user input and the voice command with a small time difference are combined to generate the XR content module 130. Control can be executed in the same way as in the real situation by linking with the content.

In the case of fast forward (FF), fast backward (BB), fast left (LL), and fast right (RR), the same word is repeatedly pronounced in conjunction with XR content to realize the same as the real situation. If 120 is pronounced before a certain time (t), the quick action may be executed in the XR content module 130, but if pronounced after a certain time (t), a single action may be duplicated and the quick action may not be Therefore, it is necessary to appropriately adjust the predetermined time t.

Alternatively, a voice waveform for quickly pronouncing duplicate words may be stored in the voice database 140 in advance, and the voice recognition unit 120 may determine it by matching the pattern.

According to an embodiment, when a voice command is executed through the voice input unit 110 as shown in FIG. 8 , it is determined whether a minimum time t1 has elapsed after each command is generated, and when the elapsed time, each command is is executed

If the minimum time t1 is set too small, the voice recognition unit 120 appropriately sets the same overlapping word in the case of fast forward (FF), fast backward (BB), fast left (LL), and fast right (RR). It cannot be recognized, and if it is set too large, it cannot be reproduced in the same way as the real situation by linking with the XR content, so there is a problem that the sense of reality is lowered, so an appropriate setting is necessary.

According to an embodiment, when a voice command is input after a specific gesture is input to the camera, a voice command word is determined by searching for a similar word from the voice database 140 corresponding to the voice command, and the voice command word is generated in response to the user input. run

As the analogous words, forward (F), backward (B), left (L), and right (R) can be replaced with other sounds convenient for the user to pronounce.

According to an embodiment, a command execution unit that extracts a group of similar words corresponding to an intuitive voice command through the voice input unit, determines the corresponding voice command, combines the determined voice command, and executes the execution corresponding to the final command; includes; can do.

For example, in order to extract a similar word group, the Jacquard similarity, cosine similarity, Eucladian similarity, etc. of each voice command may be used.

The present invention is a command determining unit that eliminates the need to memorize a command by searching for a similar word for a voice input through the voice input unit from a voice database and determining the command by searching together a plurality of intuitive synonyms related to a command for a desired input. ; may be further included.

According to an embodiment, the voice input unit 110 includes one main microphone and one or more auxiliary microphones, and the voice recognition unit 120 compares the voice information collected by the one or more auxiliary microphones with each other to obtain the voice information. It is also possible to recognize the actually pronounced voice information as closely as possible by filtering the noise.

Hereinafter, an interworking method using a voice recognition-based shortcut key XR interactive interworking system for carrying out the present invention will be described in detail with reference to FIG. 9 .

First, receiving a voice signal from the user through the voice input unit 110 (S110); Recognizing, through the voice recognition unit 120, a voice recognition-based voice shortcut key defining a part of movement in the voice input from the voice input unit (S120); A step (S130) of generating XR content corresponding to the voice recognition-based voice shortcut key from the voice recognition unit 120 through the XR content module 130 (S130).

Finally, the XR content module 130 converts the corresponding movement (forward, backward, left, right movement) into XR content and displays it on the display device 150 ( S140 ).

In an embodiment, defining a voice recognition-based shortcut key by voice and/or sound by user definition; Learning a sound that can be expressed differently for each person; The XR content module 130 displays operations by various sounds on the display device 150 according to the voice recognition-based shortcut key.

Example (Boxing)

When developing boxing XR contents, it is necessary to apply steps (left and right, front and rear, etc.) to realize a feeling similar to actual boxing due to the nature of boxing.

The voice recognition unit 120 sets the voice shortcut keys as forward (F), backward (B), left (L), right (R) and stores them in the voice database 140, L when rotating left, R when rotating right , when two steps are taken in the future, if you speak to the voice input unit 110 in FF, the voice recognition unit 120 works in conjunction with the XR content to implement the same as the actual situation, thereby enhancing the sensory experience of the content.

If the user wants to control the operation by inputting his or her voice into the voice input unit 110, the voice recognition unit 120 sends it to the voice database 140 that stores the voice shortcut key and searches it. , the XR content module 130 converts the corresponding movement (forward, backward, left, right movement, etc.) into XR content and displays it on the display device 150 .

Example (Soccer)

When developing soccer XR content, dribbling and stepping (left and right, back and forth, etc.) are required to realize soccer-like realism due to the nature of soccer.

Forward (F), backward (B), left (L), right (R) among the voices of the voice input unit 110 for the voice recognition unit 120 to be linked with XR content to implement the same as actual soccer action One or more of the voice shortcut keys are individually recognized.

Example (Baseball)

When developing XR content, it is necessary to apply running, etc. to realize a real feeling similar to actual baseball due to the characteristics.

Additionally, since it is difficult to capture the left and right rotation with the camera 160, a voice shortcut key may be linked to compensate for this.

For example, in order to make the XR content module 130 recognize the running similar to actual baseball with the voice shortener, speaking FF, it can be implemented in the same way as the real baseball running situation in conjunction with the XR content.

110: voice input unit
120: voice recognition unit
130: XR content module
140: voice database
150: display device

Claims (8)

Voice recognition-based shortcut keys for realistic experience of content by using voice recognition-based shortcut keys for a part of movement with a wide activity space;
a voice recognition unit for recognizing a voice recognition-based shortcut key defining a part of a movement with a wide active space in the voice input from the voice input unit;
An XR content module that generates XR content through motion-based information and voice recognition-based shortcut key; voice recognition-based shortcut key XR interactive interworking system including. The method according to claim 1,
The voice recognition unit performs forward (F), backward (B), left (L), right (R) direction movement among the actions to be implemented in the same way as in the real situation in conjunction with XR contents, respectively, “F”, “B”, After dividing into “L” and “R” voices, defining the voice shortcut keys and storing them in the voice database 140, forward (F), backward (B), left (L), right (R) direction movement and left to right A voice recognition-based shortcut key XR interactive interlocking system, characterized in that it defines a shortcut key based on the operation voice made up of the pronunciation required for rotation (LR) and right-to-left (RL) and stores it in a database. The method according to claim 1,
The voice recognition unit recognizes the fast forward movement as FF pronunciation, and implements the same as the actual situation in conjunction with the XR content, such as stepping forward two steps, to enhance the sensory experience of the content,
Voice recognition-based shortcut XR interactive interworking system, characterized in that it receives a voice input through the voice recognition unit, and implements the same as the actual situation by interworking with XR contents based on the size of the voice, the accuracy of the voice, and the speed of the voice. The method according to claim 1,
It sets a bundle of voice waveforms or texts in the voice database, stores it in the voice database, divides the voice waveform input through the voice input unit into specific voice waveforms or texts with the voice recognition unit, and works with XR content to implement the same as the actual situation. ,
According to the message data received through the voice input unit, a related movement execution or state change) is executed, fast forward (FF), fast backward (BB), quickly right and left (RL), quickly left and right (LR), quickly left forward (FF), fast left reverse (BB), fast left forward (LL), and fast right forward (RR) can be executed separately, so it is implemented in the same way as in the real situation. The method according to claim 1,
In the search for the voice waveform or text input through the voice input unit, if there is a character string matching the voice signal related to the voice waveform or text in the voice database, the XR content module interlocks the character string corresponding to the character signal with the XR content to actually Implement the same situation as
Based on the voice command through the voice input unit, the time between the user input before and after is calculated, and the user input with a small time difference and the voice command are combined, and the XR content module interworks with the XR content to execute control in the same way as in the real situation. A voice recognition-based shortcut key XR interactive interworking system. The method according to claim 1,
When a voice command is executed through the voice input unit, it is determined whether a minimum time (t1) has elapsed after each command is generated, and when the elapsed time, each command is executed,
The XR content module learns a voice recognition-based shortcut key by voice or/and sound by user definition, and controls the operation according to the voice recognition-based shortcut key to be displayed on a display device. . The method according to claim 1,
After a specific gesture is input to the camera, when a voice command is input, a similar word is searched from a voice database corresponding to the voice command to determine a voice command word, and the voice command word is executed in response to the user input,
and a command execution unit that extracts a group of similar words corresponding to an intuitive voice command through the voice input unit, determines the corresponding voice command, combines the determined voice command, and executes the execution corresponding to the final command. Voice recognition-based shortcut key XR interactive interworking system. receiving a voice signal from a user through a voice input unit (S110); Recognizing a voice recognition-based voice shortcut key defining a part of movement in the voice input from the voice input unit through a voice recognition unit (S120); Including; generating XR content corresponding to the voice recognition-based voice shortcut key from the voice recognition unit through the XR content module (S130);
The voice recognition unit performs forward (F), backward (B), left (L), right (R) direction movement among the actions to be implemented in the same way as in the real situation in conjunction with XR contents, respectively, “F”, “B”, dividing into "L" and "Al", defining a voice shortcut key and storing it in a database;
The forward (F), backward (B), left (L), right (R) direction movement, left to right rotation (LR), right to left rotation (RL), and fast movement (FF) are required Defining a shortcut key based on motion speech recognition and storing it in a database;
defining a voice recognition-based shortcut key by voice and/or sound by user definition; Learning a sound that can be expressed differently for each person; A method of using a voice recognition-based shortcut key XR interactive interlocking system, comprising: displaying, on a display device, an operation of various sounds by the XR content module according to the voice recognition-based shortcut key.

Images