KR102170978B1 - Apparatus for educating coding using voice command and augmented reality - Google Patents

Abstract

According to the present invention, a coding education device using a voice command and augmented reality comprises: a microphone unit receiving a voice command uttered from a user and outputting the same as a voice signal; a display unit displaying an augmented reality image in which a virtual robot image is superimposed on a background image photographed by a camera unit; and a processor analyzing the voice signal to generate voice recognition data as a result of recognizing the voice command, generating an operation control command block that controls the operation of the robot image based on the voice recognition data, controlling the display unit to control the operation of the robot image in response to the operation control command block, generating a command block image that represents the operation control command block, and controlling the display unit so that the command block image is superimposed on the background image. According to the present invention, a command block image for representing an operation control command block is displayed on the augmented reality together with a robot controlled in response to the operation control command block, and thus, coding education can be effectively performed.

Description

Coding education apparatus using voice command and augmented reality {Apparatus for educating coding using voice command and augmented reality}

The present invention relates to a coding education apparatus using voice commands and augmented reality, and more particularly, to generate a motion control command block that controls the motion of a robot based on a user's voice command, and responds to the generated motion control command block. Thus, it relates to a coding education apparatus using a voice command and augmented reality for displaying a robot whose motion is controlled as an incremental reality.

As interest in coding education increases, applications and services that enable beginners, children, and adolescents to learn coding by providing a collection of instructions in block form rather than directly dealing with programming languages, are being provided. In addition, a learning service using a robot that operates according to coding to visually show the coding result and induce interest is provided.

Augmented Reality is a technology that superimposes virtual objects on the real world. It is also called Mixed Reality (MR) because it combines the real world with the virtual world with additional information in real time and shows it as a single image.

Unexamined Patent Publication No. 10-2018-0013487, published on February 7, 2018

The present invention generates a motion control command block that controls the operation of the robot based on a user's voice command, and the operation is controlled in response to a command block image representing the generated motion control command block and the generated motion control command block. An object of the present invention is to provide a coding education apparatus using a voice command and augmented reality capable of displaying in augmented reality.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention that are not mentioned can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. In addition, it will be easily understood that the objects and advantages of the present invention can be realized by the means shown in the claims and combinations thereof.

A coding education apparatus using voice commands and augmented reality according to the present invention for solving the above technical problem includes: a microphone unit for receiving a voice command uttered from a user and outputting a voice signal; A display unit for displaying an augmented reality image in which a virtual robot image is superimposed on a background image captured by the camera unit; And generating voice recognition data as a result of recognizing the voice command by analyzing the voice signal, generating a motion control command block for controlling the motion of the robot image based on the voice recognition data, and in the motion control command block And a processor configured to control the display unit to correspondingly control the operation of the robot image, generate a command block image representing the operation control command block, and control the display unit so that the command block image is superimposed on a background image. .

Preferably, the operation control command block may include an operation type control statement.

Advantageously, the processor may search for motion type information from the speech recognition data, and when the motion type information is retrieved, generate an operation type control command corresponding to the searched motion type information.

Preferably, the motion control command block may include an operation speed control statement.

Advantageously, the processor may search for motion speed information from the speech recognition data, and when the motion speed information is retrieved, generate a motion speed control statement corresponding to the retrieved motion speed information.

Preferably, if the operation speed information is not retrieved, the processor generates syllable spacing information based on the speech recognition data, generates the motion speed information in response to the syllable spacing information, and the generated motion speed information You can create a corresponding motion speed control statement.

Preferably, the operation control command block may include an operation repetition number control statement.

Preferably, the processor may search for operation repetition number information from the speech recognition data, and when the operation repetition number information is retrieved, generate an operation repetition number control statement corresponding to the searched operation repetition number information.

Preferably, if the operation repetition number information is not retrieved, the processor generates signal strength information based on the voice signal, generates the operation repetition number information in response to the signal strength information, and the generated operation repetition number It is possible to generate an operation repetition control statement corresponding to the information.

Preferably, the processor may generate user characteristic information of the user based on the voice signal, and set a programming language type of the operation control command block in response to the user characteristic information.

According to the present invention, a command block image representing a motion control command block generated based on a user's voice command and a robot whose motion is controlled in response to the generated motion control command block are displayed together in augmented reality, thereby effectively providing coding education. Can be done.

1 is a diagram illustrating a coding education apparatus using voice commands and augmented reality according to various embodiments of the present disclosure.
2 is a block diagram showing components of a coding education apparatus using voice commands and augmented reality according to an embodiment of the present invention.
3 to 5 are views showing first to third examples of screens displayed by a display unit of a coding education apparatus using a voice command and augmented reality according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present invention. In connection with the description of the drawings, similar reference numerals may be used for similar elements.

In this document, expressions such as "have", "may have", "include", or "may contain" are the presence of corresponding features (eg, elements such as numbers, functions, actions, or parts). And does not exclude the presence of additional features.

In this document, expressions such as "A or B", "at least one of A or/and B", or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, “A or B”, “at least one of A and B”, or “at least one of A or B” includes (1) at least one A, (2) at least one B, Or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as "first", "second", "first", or "second" used in this document can modify various elements regardless of their order and/or importance, and It is used to distinguish it from other components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of the rights described in this document, a first component may be named a second component, and similarly, a second component may be renamed to a first component.

Some component (eg, a first component) is "(functionally or communicatively) coupled with/to)" to another component (eg, a second component) or " When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or may be connected through another component (eg, a third component). On the other hand, when a component (eg, a first component) is referred to as being “directly connected” or “directly connected” to another component (eg, a second component), the component and the It may be understood that no other component (eg, a third component) exists between the different components.

The expression "configured to" used in this document is, for example, "suitable for", "having the capacity to" depending on the situation. )", "designed to", "adapted to", "made to", or "capable of" . The term "configured to (or set) to" may not necessarily mean only "specifically designed to" in hardware. Instead, in some situations, the expression "a device configured to" may mean that the device "can" along with other devices or parts. For example, the phrase “a processor configured (or configured) to perform A, B, and C” means a dedicated processor (eg, an embedded processor) for performing the operation, or by executing one or more software programs stored in memory. , May mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

The terms used in this document are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless they have a clearly different meaning in the context. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in a general dictionary may be interpreted as having the same or similar meanings as those in the context of the related technology, and unless explicitly defined in this document, they may be interpreted in an ideal or excessively formal meaning. It is not interpreted. In some cases, even terms defined in this document cannot be interpreted to exclude embodiments of this document.

1 is a diagram illustrating a coding education apparatus 100 using voice commands and augmented reality according to various embodiments of the present disclosure.

1 is a diagram illustrating a coding education apparatus 100 using voice commands and augmented reality according to various embodiments of the present invention, and FIG. 2 is a diagram illustrating coding education using voice commands and augmented reality according to an embodiment of the present invention. A block diagram showing the components of the device 100, FIGS. 3 to 5 are screens displayed by the display unit 130 of the coding education apparatus 100 using voice commands and augmented reality according to an embodiment of the present invention It is a diagram showing the first to third examples of.

1 to 5, a coding education apparatus 100 using voice commands and augmented reality includes a camera unit 110, a processor 120, a display unit 130, an input unit 140, a microphone unit 150, and A speaker unit 160, a storage unit 170, and a communication unit 180 may be included.

The camera unit 110 may generate a background image by photographing a background on which the robot image is overlapped. The camera unit 110 may be coupled to one side of the coding education apparatus 100 using a voice command and augmented reality held by a user to take a background.

To this end, the camera unit 110 may include one or more of a macro lens, a telephoto lens, and a light source lens.

The processor 120 may control the operation of the camera unit 110, the display unit 130, the input unit 140, the microphone unit 150, the speaker unit 160, the storage unit 170, and the communication unit 180. .

The processor 120 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus) for transmitting and receiving signals with other components. I can.

The processor 120 according to an embodiment may be configured to perform an operation of the coding education apparatus 100 using voice commands and augmented reality by executing one or more instructions stored in the storage unit 170.

Specifically, the processor 120 may control the display unit 130 to display a background image captured by the camera unit 110.

In addition, the processor 120 may control the display unit 130 to display the robot image superimposed on the background image.

At this time, as shown in FIG. 1, the processor 120 recognizes a plane image from among the background images, calculates area information of the plane image on the background image, and the area ratio of the calculated area information to the image size information of the background image If the ratio is greater than or equal to the preset reference area, the display unit 130 may be controlled so that the robot image is superimposed on the flat image.

In addition, the processor 120 may control the display unit 130 so that the robot image is superimposed on the center of the planar image before a voice command from the user is input to the microphone unit 150.

Meanwhile, when a voice command uttered from a user is input to the microphone unit 150 and output as a voice signal, the processor 120 may analyze the voice signal and generate voice recognition data as a result of recognizing the voice command.

In this case, as shown in FIG. 3, the processor 120 may activate the microphone unit 150 to receive a voice command when a voice command input button I1 is input by a user.

Here, the speech recognition data may be data generated based on a result of preprocessing a speech signal by the processor 120, extracting a speech characteristic from the preprocessed speech signal, comparing the extracted speech characteristic with a speech model, and the comparison result. have. For example, the voice recognition data may be text corresponding to a voice command.

In this case, it should be noted that the type of speech recognition method used by the processor 120 to generate speech recognition data corresponding to a speech command is not limited as long as it generates speech recognition data.

As shown in FIG. 4, the processor 120 may control the display unit 130 to display a speech recognition data image D1 representing the recognized speech recognition data.

In addition, the processor 120 may control the display unit 130 to display a confirmation input area I2 through which a user can input whether or not the recognized speech recognition data corresponds to a speech command.

The processor 120 may perform a process of generating an operation control command block when a confirmation input indicating that the recognized speech recognition data corresponds to a speech command is input.

Conversely, the processor 120 may perform a process of re-inputting the voice command when an unconfirmed input indicating that the recognized voice recognition data does not correspond to the voice command is input.

Thereafter, the processor 120 may generate a motion control command block for controlling the motion of the robot image based on the voice recognition data, and control the display unit 130 to control the motion of the robot image in response to the motion control command block. have.

Also, the processor 120 may generate a command block image D2 representing an operation control command block, and control the display unit 130 so that the command block image D2 is superimposed on the background image.

Here, the motion control command block may include one or more of an operation type control statement, an operation speed control statement, and an operation repetition number control statement corresponding to the set programming language type.

At this time, the operation type control statement includes an operation type control command and an operation type control variable corresponding to the set programming language type, and the operation speed control statement includes an operation speed control command and operation speed control variable corresponding to the set programming language type. And, the operation repetition number control command may include an operation repetition number control command and an operation repetition number control variable corresponding to a set programming language type.

The processor 120 searches for the motion type information C1 from the speech recognition data, and when the motion type information C1 is retrieved, the processor 120 may generate a motion type control statement C1' corresponding to the searched motion type information C1. have.

To this end, the text corresponding to the operation type information C1 may be matched with the operation type control command C1 ′ for each programming language type and stored in the storage unit 170.

The processor 120 retrieves the motion speed information C2 from the speech recognition data, and when the motion speed information C2 is retrieved, the processor 120 generates a motion speed control statement C2' corresponding to the retrieved motion speed information C2. have.

To this end, a text corresponding to the operation speed information C2 may be matched with an operation speed control statement C2' for each programming language type and stored in the storage unit 170.

Conversely, if the motion speed information C2 is not retrieved from the voice recognition data, the processor 120 generates syllable spacing information based on the voice recognition data, generates motion speed information corresponding to the syllable spacing information, and generated A motion speed control statement corresponding to motion speed information can be generated.

Specifically, the processor 120 calculates a first time difference between the end time of one syllable and the start time of the next syllable among the syllables included in the speech recognition data, and the average time difference between the plurality of first time differences Can be generated as syllable spacing information.

Thereafter, the processor 120 may calculate a time difference ratio of the average time difference indicated by the syllable interval information relative to the reference time difference, and apply the time difference ratio to the basic operation speed to generate operation speed information.

For example, when the average time difference is “12msec”, the reference time difference is “10msec”, and the basic operation speed is “1x”, the processor 120 calculates the time difference ratio as “120%”, and the basic operation speed By applying the time difference ratio “120%” to “1x speed”, you can generate the operation speed information at “1.2x”.

On the other hand, the processor 120 retrieves the operation repetition number information C3 from the speech recognition data, and when the operation repetition number information C3 is retrieved, the operation repetition number control statement C3 corresponding to the retrieved operation repetition number information C3 ') can be created.

To this end, the text corresponding to the operation repetition number information C3 may be matched with the operation repetition number control statement C3 ′ for each programming language type and stored in the storage unit 170.

Conversely, if the motion repetition number information C3 is not retrieved from the speech recognition data, the processor 120 generates signal strength information based on the voice signal, generates motion repetition number information corresponding to the signal strength information, and generates It is possible to generate an operation repetition control statement corresponding to the operation repetition number information.

Specifically, the processor 120 may generate the maximum signal strength as signal strength information among signal strengths of the voice signal over time.

Thereafter, the processor 120 may calculate the signal strength ratio of the maximum signal strength indicated by the signal strength information to the reference signal strength, and apply the signal strength ratio to the basic operation repetition number to generate the operation repetition number information.

For example, if the maximum signal strength is “200 dB”, the reference signal strength is “100 dB”, and the number of basic operation repetitions is “one time”, the processor 120 calculates the signal strength ratio as “200%” , By applying the signal strength ratio “200%” to the basic operation repetition number “1 time”, the operation repetition number information can be generated as “2 times”. In this case, the operation repetition number information may be rounded up and generated as a natural number.

Meanwhile, the processor 120 may generate user characteristic information of the user based on the voice signal, and set a programming language type of the operation control command block in response to the user characteristic information.

Here, the user characteristic information may include user gender information and user age information.

Specifically, the processor 120 calculates the average frequency from the frequency of the voice signal over time, and reads the user age information of the user corresponding to the average frequency from the frequency-user characteristic table in which user age information is mapped for each voice frequency. I can.

Thereafter, the processor 120 may read programming language type information corresponding to the user age information read from the age group-programming type table in which programming language type information according to the age group information is set. In this case, the age group-programming type table may be information about an age group whose age is close to the 20s to 40s, and the programming language type information with high difficulty level may be set.

Finally, the processor 120 may set the read programming language type information as programming language type information of the job control command block.

The display unit 130 may display various images and images under the control of the processor 120.

Specifically, the display unit 130 may display a background image and display an augmented reality image in which one or more of a robot image, voice recognition data, and command block image are superimposed on the background image.

The input unit 140 may receive various information from a user through a touch screen coupled to the display unit 130. Specifically, the input unit 140 may receive a voice command recognition request through the voice input button I1, and may receive input through the confirmation input area I2 whether or not the recognized voice recognition data corresponds to the voice command.

The microphone unit 150 may receive a voice command uttered from a user, convert the input voice command, and output a voice signal.

When the motion control command block is generated, the speaker unit 160 may output an audio signal corresponding to the motion control command block. Through this, the user can visually check the motion control command block through the command block image, and audibly check the motion control command block through the audio signal.

The storage unit 170 may store programs (one or more instructions) for processing and controlling the processor 120. Programs stored in the storage unit 170 may be divided into a plurality of modules according to functions.

The communication unit 180 may be connected to a communication network and communicate with an education server using universal communication. Here, the education server may be a server that manages the coding education apparatus 100 using a plurality of voice commands and augmented reality.

To this end, the communication unit 180 may include a general-purpose communication module that performs general-purpose communication. Here, the general-purpose communication is communication using an Internet network or a cellular communication protocol, for example, LTE (Long-Term Evolution), LTE-A (LTE Advanced), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), At least one of Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), and Global System for Mobile Communications (GSM) may be used.

Specifically, the communication unit 180 matches the recorded data on which the voice command is recorded, the voice signal from which the voice command is converted, the voice recognition data that is a result of the voice command recognition, and the motion control command block generated from the voice command to the education server. Can send.

So far, we have looked at the center of the preferred embodiment for the present invention. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and all differences within the scope equivalent thereto should be interpreted as being included in the present invention.

As described above, although the present invention has been described by the limited embodiments and drawings, the present invention is not limited thereto, and the technical idea of the present invention and the following by those of ordinary skill in the art to which the present invention pertains. It goes without saying that various modifications and variations are possible within the equivalent range of the claims to be described.

100: Coding education device using voice commands and augmented reality
110: camera unit
120: processor
130: display
140: input unit
150: microphone unit
160: speaker unit
170: storage unit
180: communication department

Claims (7)

A microphone unit for receiving a voice command uttered from a user and outputting a voice signal;
A display unit for displaying an augmented reality image in which a virtual robot image is superimposed on a background image captured by the camera unit; And
Analyzes the voice signal to generate voice recognition data as a result of the recognition of the voice command, generates a motion control command block for controlling the motion of the robot image based on the voice recognition data, and corresponds to the motion control command block A processor for controlling the display unit to control the operation of the robot image, generating a command block image representing the operation control command block, and controlling the display unit so that the command block image is superimposed on a background image; and
The motion control command block is
Contains a motion speed control statement,
The processor is
Searching for motion speed information from the speech recognition data, and when the motion speed information is retrieved, generates a motion speed control statement corresponding to the searched motion speed information,
If the motion speed information is not retrieved, a syllable spacing information is generated based on the speech recognition data, the motion speed information is generated in response to the syllable spacing information, and an motion speed control statement corresponding to the generated motion speed information Characterized in that to generate
Coding education device using voice commands and augmented reality.
The method of claim 1,
The motion control command block is
Contains the action type control statement,
The processor is
Retrieving motion type information from the speech recognition data, and generating a motion type control statement corresponding to the searched motion type information when the motion type information is retrieved.
Coding education device using voice commands and augmented reality.
delete delete The method of claim 1,
The motion control command block is
Includes a control statement for the number of operation repetitions,
The processor is
And generating an operation repetition number control statement corresponding to the searched operation repetition number information when the operation repetition number information is retrieved from the speech recognition data.
Coding education device using voice commands and augmented reality.
The method of claim 5,
The processor is
If the operation repetition number information is not retrieved, signal strength information is generated based on the voice signal, the operation repetition number information is generated in response to the signal strength information, and the operation repetition corresponding to the generated operation repetition number information Characterized in that to generate a number control statement
Coding education device using voice commands and augmented reality.
The method of claim 1,
The processor is
And generating user characteristic information of the user based on the voice signal, and setting a programming language type of the operation control command block in response to the user characteristic information.
Coding education device using voice commands and augmented reality.

Images