KR20190133361A - An apparatus for data input based on user video, system and method thereof, computer readable storage medium - Google Patents

Abstract

Provided is a method for inputting information based on a user image. The method comprises the steps of: generating mouth shape image information for a specific user and generating a plurality of training data sets individually including text information corresponding to the mouth shape image information; generating an image recognition model for outputting the text information corresponding to the inputted mouth shape image information based on the training data sets; and determining input text information corresponding to an input image for specific user based on the image recognition model. Therefore, the method may input information by recognizing a mouth shape of the specific user with improved accuracy.

Description

Information input device, method, system and computer readable storage medium based on user image {AN APPARATUS FOR DATA INPUT BASED ON USER VIDEO, SYSTEM AND METHOD THEREOF, COMPUTER READABLE STORAGE MEDIUM}

The present invention relates to information input, and more particularly, to an apparatus, a method and a system for inputting information using an image of a user, and a computer readable storage medium.

With the development of information and communication technology, various computing devices have been utilized, and demand for various information input means other than the conventional keyboard and mouse is increasing. In particular, for example, a mobile device such as a smart phone or a tablet PC is mainly used to touch the keyboard displayed on the touch screen for input of information, but users may feel uncomfortable with such an input. In order to increase the degree of freedom of the user's hand in the way of inputting information and controlling the function of the mobile device has been actively developed. However, the voice recognition as described above has a problem that the accuracy may be significantly reduced depending on the presence of ambient noise, and there is a problem that cannot be used in an environment in which it is difficult to generate a voice of a user.

Korean Laid-Open Patent Publication No. 2001-0012024 ("Voice Recognition Portal Service System and Method Using Multi-Level Voice Recognition", Katie Co., Ltd.)

Speech recognition has a problem that the accuracy can be significantly reduced depending on the presence of ambient noise, there is a problem that can not be used in an environment difficult to generate a user's voice.

As a means for replacing voice recognition, a method of recognizing a user's mouth shape and inputting information may be considered. However, since the shape of the mouths of the plurality of users are different from each other, a method for improving the accuracy of mouth shape recognition is required.

Summary of the Invention An object of the present invention for solving the above problems is to generate an image recognition model based on training data including mouth shape image information and text information corresponding to a specific user, thereby improving the accuracy and accuracy of the specific user. It is to provide an information input method that can input information by recognizing the shape of the mouth.

Another object of the present invention for solving the above problems is to generate a specific image with improved accuracy by generating an image recognition model based on training data including mouth shape image information and corresponding text information for a specific user. The present invention provides an information input device capable of inputting information by recognizing a shape of a user's mouth.

However, the problem to be solved of the present invention is not limited thereto, and may be variously expanded within a range without departing from the spirit and scope of the present invention.

According to an embodiment of the present invention, an information input method based on a user image includes a plurality of mouth image information and text information corresponding to the mouth image information for a specific user. Generating training data sets of a; Generating an image recognition model outputting text information corresponding to the input mouth shape image information based on the training data sets; And determining input text information corresponding to the input image for the specific user based on the image recognition model.

According to an aspect, the determining of the input text information may include: receiving an input image for the specific user acquired by the image input unit; And determining text information corresponding to mouth shape image information included in the input image as the input text information based on the image recognition model.

According to an aspect of the present disclosure, the training data sets may include i) speech recognition image information included in the video call data of the specific user and ii) speech recognition text information that is a speech recognition result of a voice corresponding to the mouth shape image information. It may include.

According to one aspect, generating the training data sets comprises: obtaining video call data of the particular user, wherein the video call data includes call video and call voice; Associating first mouth image information that is at least a portion of the call video with first voice information that is at least a portion of the call voice based on time information; Acquiring first speech recognition text information that is text information corresponding to the first speech information based on a speech recognition model; And storing the first mouth shape image information and the first speech recognition text information as a first training data set.

According to an aspect, a time length of the first mouth shape image information and the first voice information may be determined as a predetermined time length.

According to an aspect, the first mouth shape image information and the first voice information have a length of time from a first time point to a second time point, and the first time point indicates a time point at which the call voice is less than or equal to a predetermined threshold size. The second time point may indicate a time point at which the call voice subsequent to the first time point is less than or equal to a predetermined threshold size.

According to an aspect, the first mouth shape image information and the first voice information have a length of time from a first time point to a second time point, and the first time point is the mouth shape of the specific user included in the call image. Indicates a point in time at which the image information coincides with the first predetermined triggering mouth shape information, and a second second point of view corresponds to the first triggering mouth shape information of the specific user, which is subsequent to the first time point. It may indicate a time point.

According to an aspect, the first mouth shape image information and the first voice information have a length of time from a first time point to a second time point, and the first time point is the mouth shape of the specific user included in the call image. Indicates a time point at which the image information coincides with the first predetermined triggering mouth shape information, and the second time point is a time point when the mouth shape image information of the specific user included in the call image matches the predetermined second triggering mouth shape information. Can be represented.

According to an aspect, the training data sets may include i) exemplary text information and ii) mouth image information from which the specific user reads the exemplary text information.

According to one aspect, generating the training data sets comprises: displaying example text information; Acquiring a read image which is an image of the specific user while the specific user reads the example text information; And storing the second mouth shape image information and the example text information included in the read image as the second training data set.

According to one aspect, generating the training data sets comprises: displaying a plurality of example text information; Acquiring a read image that is an image for the specific user and a read voice that is a voice for the specific user while the specific user reads the example text information; Associating third mouth shape image information, which is at least a part of the read image, with third voice information that is at least a part of the read voice, based on time information; Obtaining third speech recognition text information, which is text information corresponding to the third speech information, based on a speech recognition model; And in response to determining that the third voice recognition test information and the third example text information are the same as the third example text information, the third mouth image information and the third example text information are trained. Storing as a data set.

According to another aspect of the present invention, an information input apparatus based on a user image includes: an image input unit configured to acquire image information; A voice input unit for obtaining voice information; A memory for storing image information, audio information, and text information; And a processor, wherein the processor generates a plurality of training data sets each comprising mouth shape image information for a specific user and text information corresponding to the mouth shape image information, and based on the training data sets, Generating an image recognition model for outputting text information corresponding to the input mouth image information; The display apparatus may be configured to determine input text information corresponding to the input image for the specific user based on the image recognition model.

According to an aspect, the determining of the input text information may include: the processor receiving an input image for the specific user acquired by the image input unit; And determining, as the input text information, text information corresponding to the mouth shape image information included in the input image, based on the image recognition model.

According to an aspect of the present disclosure, the training data sets may include i) speech recognition image information included in the video call data of the specific user and ii) speech recognition text information that is a speech recognition result of a voice corresponding to the mouth shape image information. It may include.

According to one aspect, generating the training data sets includes: obtaining, by the processor, video call data of the particular user, wherein the video call data includes call video and call voice; Correlating first mouth image information that is at least a portion of the call video with first voice information that is at least a portion of the call voice based on time information; Acquire first speech recognition text information that is text information corresponding to the first speech information based on a speech recognition model; And storing the first mouth shape image information and the first voice recognition text information as the first training data set in the storage unit.

According to an aspect, a time length of the first mouth shape image information and the first voice information may have a predetermined time length.

According to an aspect, the first mouth shape image information and the first voice information have a length of time from a first time point to a second time point, and the first time point indicates a time point at which the call voice is less than or equal to a predetermined threshold size. The second time point may indicate a time point at which the call voice subsequent to the first time point is less than or equal to a predetermined threshold size.

According to an aspect, the first mouth shape image information and the first voice information have a length of time from a first time point to a second time point, and the first time point is the mouth shape of the specific user included in the call image. Indicates a time point at which the image information coincides with the first predetermined triggering mouth shape information, and the second time point corresponds to the first triggering mouth shape information, which is the mouth shape image information of the specific user, subsequent to the first time point. It may indicate a time point.

According to an aspect, the first mouth shape image information and the first voice information have a length of time from a first time point to a second time point, and the first time point is the mouth shape of the specific user included in the call image. Indicates a time point at which the image information coincides with the first predetermined triggering mouth shape information, and the second time point is a time point when the mouth shape image information of the specific user included in the call image matches the predetermined second triggering mouth shape information. Can be represented.

According to an aspect, the training data sets may include i) exemplary text information and ii) mouth image information from which the specific user reads the exemplary text information.

According to an aspect, generating the training data sets may include: displaying, by the processor, exemplary text information on a display unit included in an information input apparatus based on the user image; Using the image input unit, obtain a read image that is an image of the specific user while the specific user reads the example text information; And storing the second mouth shape image information and the example text information included in the read image as the second training data set in the storage unit.

According to an aspect, the generating of the training data sets may include: displaying, by the processor, a plurality of example text informations on a display included in an information input apparatus based on the user image; Using the image input unit and the audio input unit, obtain a read image that is a video for the specific user and a read voice that is a voice for the specific user while the specific user reads out the example text information; Correlating third mouth shape image information, which is at least a part of the read image, with third voice information that is at least a part of the read voice, based on time information; Acquire third speech recognition text information that is text information corresponding to the third speech information based on a speech recognition model; Correlating the third speech recognition test information with third example text information which is any one of the plurality of example text information; And in response to determining that the third voice recognition test information and the third example text information are the same as the third example text information, the third mouth shape image information and the third example text information are arranged in a third manner. Storing in the storage as a training data set.

According to another aspect of the present invention, an information input system based on a user image includes a plurality of mouth image information for a specific user and text information corresponding to the mouth image information. A server, configured to obtain training data sets of and to generate, based on the training data sets, an image recognition model that outputs text information corresponding to input mouth shape image information; And a terminal, configured to acquire at least one of image information and audio information for the specific user, and configured to determine input text information corresponding to the input image for the specific user based on the image recognition model. have.

A computer readable storage medium according to another embodiment of the present invention for solving the above-mentioned problems includes instructions executable by a processor, and when the instructions are executed by the processor, a mouth-shaped image of a specific user. Generate a plurality of training data sets each including information and text information corresponding to the mouth shape image information; Generating an image recognition model outputting text information corresponding to the input mouth shape image information based on the training data sets; The display apparatus may be configured to determine input text information corresponding to the input image for the specific user based on the image recognition model.

The disclosed technique can have the following effects. However, since a specific embodiment does not mean to include all of the following effects or only the following effects, it should not be understood that the scope of the disclosed technology is limited by this.

According to the information input method and apparatus based on a user image according to an embodiment of the present invention, an image recognition model is generated based on training data including mouth shape image information and text information corresponding to a mouth shape of a specific user. By generating, it is possible to input the information by recognizing the shape of the mouth of a specific user with more improved accuracy.

In addition, by inputting the information by recognizing the shape of the user's mouth, even in an environment in which there is noise around the user or in a quiet environment where the user's voice cannot be expressed, the user can easily input the information and control the function of the mobile device based on the information There is an advantage to this.

1 is a block diagram illustrating a configuration of an information input apparatus based on a user image according to an embodiment of the present invention.
2 is an illustration of a recognition model implemented on the processor of FIG. 1.
3 is a conceptual diagram of an image recognition model and a voice recognition model of FIG. 2.
4 is an exemplary diagram of information stored in a memory of FIG. 2.
5 is a conceptual diagram of an information input system based on a user image according to an embodiment of the present invention.
6 is a conceptual diagram of a training data set generation based on video call data.
7 is a flowchart illustrating an information input method based on a user image according to an embodiment of the present invention.
8 is a detailed flowchart of an input test information determination step of FIG. 7.
9 is a detailed flowchart of a training data set generation based on video call data.
10 is an exemplary view illustrating a method for setting a time length of a segment according to a preset.
11 is an exemplary view illustrating a method for setting a time length of a segment according to a threshold size of speech.
12 is an exemplary view illustrating a method for setting a time length of a segment according to a triggering mouth shape.
13 is a conceptual diagram of a first embodiment of generating example text information-based training data set.
14 is a conceptual diagram of a second embodiment of generating example text information-based training data set.
15 is a flowchart of a first embodiment of example text information based training data set generation.
16 is a flowchart of a second embodiment of example text information based training data set generation.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description.

However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. In the following description of the present invention, the same reference numerals are used for the same elements in the drawings and redundant descriptions of the same elements will be omitted.

Information input based on user image

As described above, speech recognition has a problem that its accuracy may be significantly reduced depending on the presence of surrounding noise, and there is a problem that it cannot be used in an environment in which it is difficult to generate a voice of a user. As a means for replacing speech recognition, a method of inputting information by recognizing a user's mouth may be considered. However, since the mouth shape varies according to a person, a method for improving recognition accuracy is required compared to speech recognition. . Information input based on a user image according to an embodiment of the present invention is to solve the above-described problem and can improve the accuracy of mouth shape recognition by generating a mouth shape recognition model specialized for a specific user. . More specifically, the information input based on the user image according to an embodiment of the present invention, acquires a plurality of training data, including a plurality of training data including mouth shape image information and a text information corresponding to a specific user, and obtains By generating an image recognition model based on training data sets for a specific user, the user may input information by recognizing a mouth shape of a specific user with improved accuracy.

1 is a block diagram illustrating a configuration of an information input apparatus based on a user image according to an embodiment of the present invention, and FIG. 7 is a flowchart of an information input method based on a user image according to an embodiment of the present invention. 8 is a detailed flowchart of the input test information determination step of FIG. 7. Hereinafter, an information input method and apparatus based on a user image according to an embodiment of the present invention will be described in detail with reference to FIGS. 1, 7, and 8.

First, as shown in FIG. 1, an information input apparatus 100 based on a user image according to an exemplary embodiment of the present invention may include an image input unit 110, an audio input unit 120, a processor 130, and a display unit ( 140, a memory 150, and a communicator 160.

The image input unit 110 is configured to acquire image information of a user of the information input apparatus 100 based on the user image, and may be, for example, a camera device. The voice input unit 120 may be configured to acquire voice information of a user of the information input apparatus 100 based on the user image, and may be, for example, a microphone device.

The processor 130 may be configured to perform data processing for an information input procedure based on a user image according to an embodiment of the present invention. FIG. 2 is an exemplary diagram of a recognition model implemented on the processor of FIG. 1, and FIG. 3 is a conceptual diagram of an image recognition model and a voice recognition model of FIG. 2. As shown in FIG. 2, the processor 130 of the information input apparatus 100 based on a user image according to an embodiment of the present invention may include an image recognition unit 131 and a voice recognition unit 132. Can be. The image recognition unit 131 and the voice recognition unit 132 may exist as a logical module in a single processor, or each may exist as a logical module on a separate processor, or an image recognition module or a voice recognition module implemented in hardware. It may also be included as.

Meanwhile, the image recognition model 132 may be executed on the image recognition unit 132. As shown in FIG. 3, the image recognition model 132 may be a logical model that outputs text information corresponding to the input mouth shape image information when inputting mouth shape image information for a specific user. According to an aspect, the image recognition model 132 trains an artificial neural network (ANN) based on a plurality of training data sets each including mouth shape image information and a corresponding text information for a specific user. It may be a generated model. Also, according to an aspect, the image recognition model 132 may include a mouth position identification model 133 and a mouth shape identification model 134 as shown in FIG. 2. For example, the mouth position identification model 133 may be configured to extract an area in which the user's mouth is located within the input image data, and the mouth shape identification model 134 may be applied to the mouth position identification model 133. It may be configured to output the corresponding text information based on the image information on the area where the mouth is output by the output. According to an aspect, the mouth position identification model 133 may use a known AI algorithm model, and train an AI model as a training data set using an image and a mouth shape region for a specific user according to an embodiment of the present invention. It may be the result.

Referring back to FIG. 2, the speech recognition model 136 may be executed on the speech recognizer 135. As shown in FIG. 3, the voice recognition model 136 may be a logical model that outputs text information corresponding to the input voice information when voice information about a specific user is input. According to an aspect, the speech recognition model 132 may be a model generated by training an artificial neural network based on a plurality of training data sets each including speech information and text information corresponding thereto. Alternatively, the speech recognition model 132 may select and apply any one of known speech recognition models for converting speech information into text.

4 is an exemplary diagram of information stored in the memory 150 of FIG. 2. Referring to FIG. 4, for example, a plurality of training data sets 410 and a specific user are included in the memory 150 included in the information input apparatus 100 based on a user image according to an exemplary embodiment of the present invention. At least one of the video call data 420 and the example text information 430 may be stored. Training data sets 420 may be used to train the image recognition model 132, and as described in detail herein below, for example, data processing or example textual information for the video call data 420. It can be obtained by the data acquisition and processing procedure using 430.

Referring back to FIG. 2, the display unit 140 may be included in an information input apparatus based on a user image to display at least one of image and text information, and the communication unit 160 may be based on the user image. The information input apparatus 100 may be configured to perform data transmission / reception with an external device. The communication unit 160 may include, for example, at least one of a long range wireless communication module for cellular network connection, a long range wireless communication module for communication with a server, a wireless communication module for short range communication with a peripheral device, and a wired communication module. Can be.

The information input device 100 based on the user image according to an embodiment of the present invention may be included in a mobile device such as, for example, a smart phone or a tablet PC, or may be implemented as the mobile device itself.

As shown in FIG. 7, an information input method based on a user image according to an exemplary embodiment of the present invention first includes mouth shape image information and text information corresponding to the mouth shape image information for a specific user, respectively. A plurality of training data sets can be generated (step 710). As described above, the mouth shape is different for a plurality of users, and even when the same word is pronounced, different mouth shapes may be expressed according to pronunciation characteristics or habits of the individual. Therefore, when the image recognition model is trained based on mouth shape data of a plurality of users without distinguishing a specific user, the accuracy of text conversion according to mouth shape recognition may be reduced.

Therefore, according to an embodiment of the present invention, for a specific user who wants to perform a mouth shape analysis, each of the mouth image information including the mouth shape image of the specific user and text information indicating a corresponding character string may be included. The accuracy of mouth shape recognition may be improved by generating a plurality of training data and training an image recognition model based thereon. Referring to FIG. 4, the generated plurality of training data sets 410 may be stored in the memory 150, and the plurality of training data sets 410 may be the first training data sets 410-1 to n-th training. Data set 410-n, wherein the first training data set 410-1 includes first textual image information 411-1 and corresponding text strings 412-n. 1), and the n th training data set 410-n may include the n th text information 412-n including the n th mouth image information 411-n and a string corresponding thereto. Can be. According to an exemplary embodiment, the mouth shape image information may be included in image data acquired through the image input unit 110 of the information input apparatus 100 based on the user image.

Referring to FIG. 7 again, based on the training data sets generated above, an image recognition model capable of outputting text information corresponding to the input mouth shape image information may be generated. As described above with reference to FIG. 3, the image recognition model 132 may be a software module that outputs text information including a character string corresponding thereto in response to inputting mouth shape image information of a specific user. It can be generated by training an artificial neural network using training data sets.

When the generation of the training data set and the training of the image recognition model accordingly are completed, the mouth shape recognition may be performed based on the current input image of the specific user. As shown in FIG. 7, input text information corresponding to an input image of a specific user is determined based on the image recognition model (step 730) to enable input of text information using the shape recognition for the specific user. do. As shown in more detail in FIG. 8, determining the input text information (step 730) is, for example, specified by the image input unit 110 included in the information input apparatus 100 based on the user image. Receiving an input image for the user (step 731), and determining text information corresponding to the mouth shape image information included in the received input image as input text information based on the previously generated image recognition model (step 733). It may include. According to an aspect, an image recognition model 132 driven on the processor 130 included in the information input apparatus 100 based on the user image may be used, and may be used in the input image through the mouth position identification model 133. Extracting an image including a mouth shape and inputting it to the mouth shape identification model 134 may be configured to output corresponding text information.

According to the information input device 100 based on the user image according to an embodiment of the present invention, the generation of the training data set as described above (step 710), the image recognition model generation (step 720) and the input image The corresponding input text information determination 730 may be performed by the processor 130. 5 is a conceptual diagram of an information input system based on a user image according to an embodiment of the present invention. As shown in FIG. 5, for example, in an information input system based on a user image constituted by a plurality of entities, such as the terminal 100 and the server 200 capable of communicating with the terminal 100 via a network. The generation of the training data set as described above (step 710), the generation of the image recognition model (step 720), and the determination of the input text information corresponding to the input image 730 may include the terminal 100, or a processor included in the terminal ( 130 and server 200 together. For example, the terminal 100 includes an image input unit 110 and an audio input unit 120, and is configured to acquire image information and / or audio information for a specific user, and use the server using the communication unit 160. The image information and / or the voice information itself may be transmitted to the 200, or the plurality of training data sets that are data processed based on the image information and / or the voice information may be transmitted. The server 200 obtains a plurality of training data sets each including mouth shape image information and text information corresponding to mouth shape image information for a specific user, and based on the training data sets, inputs the input mouth shape image information. And generate an image recognition model that outputs corresponding text information. The generated image recognition model may be stored in at least one of the server 200 and the memory 150 of the terminal 100. In performing mouth shape recognition for a specific user, the terminal 100 transmits the input image information of the specific user to the server 200 to obtain corresponding text information based on the image recognition model stored in the server 200. After receiving it, the corresponding text information may be acquired based on an image recognition model stored in the memory 150 of the terminal 100. In terms of the terminal 100, the steps of generation of a training data set (step 710), generation of an image recognition model (step 720) and determination of input text information (step 730) according to an embodiment of the present invention are performed as described above. It should be understood that not only does the processor 130 of the terminal 100 directly perform, but also transmit and / or receive related data to an external device such as the server 200 to perform the above steps. .

Further, in terms of a computing device such as server 200, for example, a training data set is received by receiving video and audio data for generation of the training data set and / or training data set from a mobile device such as a smartphone or tablet PC. To generate an image recognition model for outputting text information corresponding to the input mouth shape data

Create training data set based on video call data

As described above, in order to improve accuracy of mouth shape recognition for a specific user, it is necessary to generate an image recognition model based on training data sets for the specific user. However, a method of sufficiently securing training data sets including mouth shape image information of a specific user and text information corresponding thereto may be secured to enable training of the image recognition model.

6 is a conceptual diagram of a training data set generation based on video call data. As shown in FIG. 6, the information input apparatus 100 based on a user image according to an embodiment of the present invention may be a mobile device used by a specific user, and may include an image input unit 610 and an audio input unit ( 620 and a display unit 640. The specific user of the mobile device may perform a video call with any other person, and when the video call is performed, the video of the other party is displayed on the counterpart area 643 of the display unit 640 and the video input unit 110 of the mobile device. The image of the specific user acquired through) may be displayed on the user region 641 of the mobile device. The voice information during the video call of the specific user may be obtained through the voice input unit 620, and the video information and the voice information of the specific user may be transmitted to the device of the counterpart user through the communication unit.

As such, when a specific user makes a video call based on the mobile device, mouth shape image information and voice information corresponding to the specific user may be naturally obtained. Therefore, the mobile device may transmit the information on the video and the voice of the specific user of the mobile device to the counterpart device during the video call, and store the same as the video call data. According to an aspect, the video call data 420 may be stored in a memory 150 of the information input apparatus 100 based on a user image, and the video call data 420 includes user mouth shape information. The video data may include call video data 421, which is video data, and call voice data 423 including information about a voice of a user.

According to an embodiment of the present invention, training data sets for generating an image recognition model may be generated based on the video call data 420. Such training data sets may include i) video call data of a specific user. And mouth recognition image information included in and ii) speech recognition text information that is a voice recognition result of the voice corresponding to the mouth image information. That is, since the video information and the audio information of the user included in the video call data may each include time information, the video information and the audio information may be corresponded based on the time information, and the voice information may be a predetermined voice recognition model. Since it can be converted into text information on the basis of, the image information (or mouth shape image information included in the image information) for the specific user can be correlated with each other. Therefore, mouth shape image information and text information corresponding to each other can be obtained and stored as one training data set.

Video call is carried out by user's needs, and it has the advantage that natural data can be collected, and video call is mainly performed in an environment with low ambient noise in order to recognize the voice of the other party and the user. The information can perform speech recognition with high accuracy to obtain accurate text information. Therefore, there is an advantage that it is possible to generate a training data set for a specific user with high accuracy.

9 is a detailed flowchart of a training data set generation based on video call data. As shown in FIG. 9, according to a training data set generation procedure based on video call data according to an embodiment of the present invention, first, video call data of a specific user using a mobile device-video call data is a call video and a call. It may be obtained (including voice) (step 711). As described above, the mobile device includes a video input unit and an audio input unit, and may acquire a call image and a call voice for a specific user during a video call, for example, the call image data 421 may be stored in the memory 150. And video call data 420 including call voice data 423.

Subsequently, the first mouth image information, which is at least a part of the call image, and the first voice information, which is at least a part of the call voice, may be corresponded based on the time information (step 713). The obtained call image and call voice may be divided and processed as segments having a predetermined length of time. For example, the first mouth image information may be any one of a plurality of segments constituting a call video, and the first voice information may be any one of a plurality of segments constituting a call voice. According to an aspect, the first mouth shape image information may be an image extracted by detecting a mouth position from a call image. Since the call video and the call voice may include time information at the time of acquisition, even when divided into a plurality of segments, the mouth shape video information and the voice information may be corresponded based on the time information. According to an aspect, the first mouth image information may be one frame included in a call image, and in another aspect, the first mouth image information may include a plurality of frames.

Meanwhile, a criterion for dividing the call video data and / or the call voice data into a plurality of segments may be set. 10 is an exemplary view illustrating a method for setting a time length of a segment according to a preset. As illustrated in FIG. 10, the call voice and the call video included in the video call data may exist in time series with time information. According to one aspect, the first mouth shape image information, which may be any one of a plurality of segments of a call voice that may be used for generating training data, and the first may be any one of the plurality of segments of a call image. The time length of the voice information can be determined as the predetermined time length t. Accordingly, all voice segments 10a-1, 10a-2, 10a-3, 10a-4, 10a-5, 10a-6, 10a-7, 10a-8 are all the same time as shown in FIG. It may be set to have a length, and may be designated as the first to eighth voice information, respectively. In addition, all image segments 10b-1, 10b-2, 10b-3, 10b-4, 10b-5, 10b-6, 10b-7, 10b-8 are also set to have the same length of time. Each of the first mouth shape image information and the eighth mouth shape image information may be designated.

11 is an exemplary view illustrating a method for setting a time length of a segment according to a threshold size of speech. As shown in FIG. 11, according to an embodiment of the present invention, the first mouth shape image information and the first audio information are configured from a first time point (eg, t ₁ ) to a second time point (eg, t _2). It can be set to have a time length up to), wherein the first time point (e.g., t ₁ ) represents a time point at which the call voice is less than or equal to a predetermined threshold size, and the second time point (e.g., t ₂ ) A voice following a voice point of time (eg, t ₁ ) may be indicated as a point of time below a predetermined threshold size. In other words, the segmentation point of the call voice and the call voice can be such that the size of the call voice is smaller than the preset threshold size. Thus, for successive time points t ₁ , t ₂ , t ₃ , t ₄ , t ₅ , t ₆ , the voice segments 11a-1, 11a-2, 11a-1, 11a-4, 11a- 5) each have a different time length, and the plurality of image segments 11b-1, 11b-2, 11b-3, 11b-4, 11b-5 may also be determined to have different sizes, respectively.

Meanwhile, although not shown in FIG. 11, according to one aspect, the segment split point may be set such that a time length that satisfies a size smaller than a preset threshold size of the call voice is longer than a preset time. Can be. In this case, the segment may be extracted except for a section in which the call voice size is smaller than a preset threshold size for more than a preset time.

12 is an exemplary view illustrating a method for setting a time length of a segment according to a triggering mouth shape. As shown in FIG. 12, according to an exemplary embodiment of the present invention, the first mouth shape image information and the first audio information are configured from a first time point (eg, t ₁ ) to a second time point (eg, t _2). The first time point (for example, t ₁ ) is a first triggered mouth shape in which mouth shape image information 121-1 of a specific user included in a call video is predetermined. The second time point (eg, t ₂ ) indicates a time point coinciding with the information, and the second time point (eg, t ₂ ) corresponds to a time point when the mouth shape image information 122-1 of the specific user included in the call video matches the predetermined second triggering mouth shape information. Can be represented. Here, the first mouth shape image information 12b-1 may be set to include image information of the mouth shape image information 121-1 to the mouth shape image information 122-1. In other words, the segmentation point of the call voice and the call video into the segment is the segment triggered when the mouth shape is the first triggering mouth shape with the first specific shape, and the second triggering mouth shape with the mouth shape the second specific shape. When the segment may be set to end. Thus, for successive time points t ₁ , t ₂ , t ₃ , t ₄ , t ₅ , the voice segments 12a-1, 12a-2, 12a-1, 12a-4 each have a different length of time. In addition, the plurality of image segments 12b-1, 12b-2, 12b-3, and 12b-4 may also be determined to have different sizes. The segments are started by start segments 121-1, 121-2, 121-3, 121-4, respectively, and end by end segments 122-1, 122-2, 122-3, 122-4. Can be. Although not shown in FIG. 12, a cutting area that is not extracted as a segment may be included between each segment.

Also, although not shown in FIG. 12, one triggering mouth shape may be used for the start and end of the segment. That is, the first view point (for example, t ₁ ) indicates a view point at which mouth shape image information (eg, 121-1) of a specific user included in the call video coincides with the first predetermined triggering mouth shape information. The second time point (eg, t ₂ ) may indicate a time point at which mouth shape image information of a specific user coincides with the first triggering mouth shape information 121-2, which is subsequent to the first time point. In this case, the first segment 12b-1 may be started from the mouth shape image information 121-1 and end before the mouth shape image information 121-2.

According to an aspect, segmentation may be performed in response to the mouth shape image information of a specific user included in a call image matching the first triggering mouth shape information predetermined for a predetermined time length or more. According to one aspect, the first triggering mouth shape may be a mouthful shape. Alternatively, the first triggering mouth shape information may be mouth shape motion information including a plurality of mouth shape sequences.

Referring back to FIG. 9, first voice information that is at least a portion of a call voice may be converted into text information (step 715). That is, the first speech recognition text information, which is text information corresponding to the first speech information, may be acquired based on the speech recognition model. Here, any known speech recognition model may be used as the speech recognition model. The video call data can lead to a high rate of speech recognition success as mentioned above. However, according to an aspect, a speech recognition model generated by training an artificial neural network using the voice of the specific user and text information corresponding thereto as a training data set may be used.

As shown in FIG. 9, by generating the first mouth shape image information and the first voice recognition text information as a first training data set (step 717), a training data set is generated based on the video call data. Can be performed. As mentioned above, since it is possible to match the first mouth image information and the first voice information based on the time information, it is also possible to match the first mouth image information and the first voice recognition text information. Accordingly, the first mouth shape image information and the second voice recognition text information corresponding to each other may be stored as the first training data set. Thereafter, the first to n th training data sets may be generated and stored through repetitive data processing on the second to n th mouth image information and the second to n th audio information.

Thus, in the course of performing a video call by a specific user, it is possible to generate training data sets that are not cumbersome, natural, and have high accuracy.

Create Training Data Set Based on Example Text Information

According to an embodiment of the present invention, it is possible to generate training data sets by displaying exemplary text information including exemplary text strings through a predetermined application and having a specific user read such text information. That is, the training data sets according to an embodiment of the present invention may include i) exemplary text information and ii) mouth shape image information from which the specific user reads the exemplary text information.

13 is a conceptual diagram of a first embodiment of generating example text information-based training data set. As shown in FIG. 13, the information input apparatus 100 based on a user image according to an embodiment of the present invention may be a mobile device used by a specific user, and may include a display unit 1300. .

15 is a flowchart of a first embodiment of example text information based training data set generation. Hereinafter, a first embodiment of generating an exemplary text information-based training data set of the present invention will be described in more detail with reference to FIGS. 13 and 15.

As shown in FIG. 15, according to an example text information-based training data set generation procedure according to an aspect of the present invention, first, example text information may be displayed (step 1510). Example text information, for example, as shown in FIGS. 1 and 4, example text information 430 in the memory 150 of the information input apparatus 100 based on the user image according to an embodiment of the present invention. May be stored as). For example, the processor 130 of the information input apparatus 100 based on the user image may display the text information 430 on the display unit 140. As illustrated in FIG. 13, at least one of exemplary text information 1310, 1320, 1330, and 1340 may be displayed on the display unit 1300.

Thereafter, the specific user may acquire a read image that is an image of the specific user while reading at least one of the example text information 1310, 1320, 1330, and 1340 (step 1520). According to an aspect, an operation of the image input unit 110 is initiated in response to a specific user touching the play button 1311 for the example text information 1310, thereby reading the specific user while reading the example text information 1310. Image information about the image may be obtained and stored as a read image. At this time, the specific user does not necessarily read the example text 1310 aloud.

Since the image of the specific user is acquired while reading the example text information 1310, the second mouth image information and the example text information 1310 included in the read image correspond to each other, and the second mouth image information and the example The second training data set may be generated by storing text information 1310 as a second training data set (step 1530). Data processing may be sequentially performed on the example text information 1320, 1330, and 1340, and buttons 1312, 1313, and 1314 may be used, respectively.

14 is a conceptual diagram of a second embodiment of generating example text information-based training data set. As shown in FIG. 14, the information input apparatus 100 based on a user image according to an embodiment of the present invention may be a mobile device used by a specific user, and may include a display unit 1400. .

16 is a flowchart of a second embodiment of example text information based training data set generation. Hereinafter, a second embodiment of generating an exemplary text information-based training data set of the present invention will be described in more detail with reference to FIGS. 14 and 16.

As illustrated in FIG. 16, according to an example text information-based training data set generation procedure according to an aspect of the present invention, first, a plurality of example text information may be displayed (step 1610). Example text information, for example, as shown in FIGS. 1 and 4, example text information 430 in the memory 150 of the information input apparatus 100 based on the user image according to an embodiment of the present invention. ), And the example text information 430 may include a plurality of example text information of the first example text 430-1 to the nth example text 430-n. For example, the processor 130 of the information input apparatus 100 based on the user image may display the text information 431-1 to 431-n as described above on the display unit 140. As illustrated in FIG. 14, a plurality of example text information 1410 may be displayed on the display unit 1400. A scroll bar 1420 may be displayed on the display unit 1400 to allow a specific user to recognize the location of the currently displayed example text information among the plurality of text information.

Thereafter, while the specific user reads the displayed plurality of example text information 1410, a read image that is an image for a specific user and a read voice that is a voice for a specific user may be acquired (step 1620). According to one aspect, by using the image input unit 110 of the information input device 100 based on the user image to obtain a read image which is an image while a specific user reads a plurality of example text 1410, The input unit 1420 may be used to obtain a read voice that is a voice while a specific user reads the plurality of example texts 1420.

Subsequently, based on the time information, the third mouth shape image information that is at least a portion of the read image and the third voice information that is at least a portion of the read voice may be associated with each other (step 1630). Similar to the procedure of segmentation for video call data described above, the video data of the read video and the audio data of the read voice are divided into a plurality of segments, and the third mouth image information is any one of the plurality of video segments. The third voice information may be a segment that is matched with the third mouth shape image information based on time information as one of the plurality of voice segments.

The third voice information can be converted into text information. That is, third speech recognition text information, which is text information corresponding to the third speech information, may be acquired based on the speech recognition model (step 1640). Similar to the above processing of video telephony data, the speech recognition model may use any known model.

When the third speech recognition text information including the character string corresponding to the third speech information is determined, it is determined whether text information corresponding to the third speech recognition test information exists among the plurality of exemplary text information. Example text information that is an object of identity determination may include a plurality of divided example text information subdivided into different lengths for the same sentence. If it is determined that the third voice recognition text information is the same as the third example text information which is any one of the plurality of example text informations, the third mouth image information and the third example text information may be stored as the third training data set. (Step 1650).

Through such a procedure, a specific user may generate a training data set by simply reading a plurality of pieces of example text information without matching specific mouth shape information and example text information. In addition, the dual identification procedure, such as speech recognition and example text information matching, has the advantage of generating a more accurate training data set.

The method according to the present invention described above may be embodied as computer readable code on a computer readable recording medium. Computer-readable recording media include all kinds of recording media having data stored thereon that can be decrypted by a computer system. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like. The computer readable recording medium can also be distributed over computer systems connected over a computer network, stored and executed as readable code in a distributed fashion.

As described above with reference to the drawings and examples, it does not mean that the scope of protection of the present invention is limited by the above drawings or embodiments, and those skilled in the art are skilled in the art It will be understood that various modifications and variations can be made in the present invention without departing from the spirit and scope.

Specifically, the described features may be implemented within digital electronic circuitry, or computer hardware, firmware, or combinations thereof. The features may be executed in a computer program product implemented in storage in a machine readable storage device, for example, for execution by a programmable processor. And features may be performed by a programmable processor executing a program of instructions to perform functions of the described embodiments by operating on input data and generating output. The described features include at least one programmable processor, at least one input device, and at least one output device coupled to receive data and directives from a data storage system, and to transmit data and directives to a data storage system. It can be executed within one or more computer programs that can be executed on a programmable system comprising a. A computer program includes a set of directives that can be used directly or indirectly within a computer to perform a particular action on a given result. A computer program is written in any form of programming language, including compiled or interpreted languages, and included as a module, element, subroutine, or other unit suitable for use in another computer environment, or as a standalone program. Can be used in any form.

Suitable processors for the execution of a program of instructions include, for example, both general purpose and special purpose microprocessors, and one of a single processor or multiple processors of another kind of computer. Computer program instructions and data storage devices suitable for implementing the described features are, for example, magnetic memory such as semiconductor memory devices, internal hard disks and removable disks such as EPROM, EEPROM, and flash memory devices. Devices, magneto-optical disks and all forms of non-volatile memory including CD-ROM and DVD-ROM disks. The processor and memory may be integrated in application-specific integrated circuits (ASICs) or added by ASICs.

Although the present invention described above has been described based on a series of functional blocks, the present invention is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes without departing from the technical spirit of the present invention. It will be apparent to one of ordinary skill in the art that this is possible.

Combinations of the above-described embodiments are not limited to the above-described embodiments, and various types of combinations as well as the above-described embodiments may be provided according to implementation and / or need.

In the above-described embodiments, the methods are described based on a flowchart as a series of steps or blocks, but the present invention is not limited to the order of steps, and any steps may occur in a different order or at the same time from other steps as described above. have. Also, one of ordinary skill in the art appreciates that the steps shown in the flowcharts are not exclusive, that other steps may be included, or that one or more steps in the flowcharts may be deleted without affecting the scope of the present invention. I can understand.

The foregoing embodiments include examples of various aspects. While not all possible combinations may be described to represent the various aspects, one of ordinary skill in the art will recognize that other combinations are possible. Accordingly, the invention is intended to embrace all other replacements, modifications and variations that fall within the scope of the following claims.

Claims (24)

An information input method based on a user image,
Generating a plurality of training data sets each including mouth shape image information for a specific user and text information corresponding to the mouth shape image information;
Generating an image recognition model outputting text information corresponding to the input mouth shape image information based on the training data sets; And
And determining input text information corresponding to the input image for the specific user based on the image recognition model.
The method of claim 1,
Determining the input text information,
Receiving an input image for the specific user acquired by the image input unit; And
And determining, as the input text information, text information corresponding to the mouth shape image information included in the input image based on the image recognition model.
The method of claim 1,
The training data sets may include: i) mouth shape image information included in the video call data of the specific user; and ii) voice recognition text information that is a voice recognition result of a voice corresponding to the mouth shape image information. Information input method based on.
The method of claim 1,
Generating the training data sets,
Obtaining video call data of the specific user, wherein the video call data includes call video and call voice;
Associating first mouth image information that is at least a portion of the call video with first voice information that is at least a portion of the call voice based on time information;
Obtaining first speech recognition text information that is text information corresponding to the first speech information based on a speech recognition model; And
And storing the first mouth shape image information and the first speech recognition text information as a first training data set.
The method of claim 4, wherein
And a time length of the first mouth shape image information and the first voice information is determined as a predetermined time length.
The method of claim 4, wherein
The first mouth image information and the first audio information have a length of time from a first time point to a second time point,
The first time point indicates a time point at which the call voice is less than or equal to a predetermined threshold size, and the second time point is based on a user image indicating a time point after which the call voice is less than or equal to a predetermined threshold size. How to enter information.
The method of claim 4, wherein
The first mouth image information and the first audio information have a length of time from a first time point to a second time point,
The first time point represents a time point at which mouth shape image information of the specific user included in the call image coincides with predetermined first triggering mouth shape information.
And wherein the second view point represents a point in time at which mouth shape image information of the specific user coincides with predetermined first triggering mouth shape information subsequent to the first view point.
The method of claim 4, wherein
The first mouth image information and the first audio information have a length of time from a first time point to a second time point,
The first time point represents a time point at which mouth shape image information of the specific user included in the call image coincides with predetermined first triggering mouth shape information.
The second viewpoint is an information input method based on a user image, which represents a time point at which mouth shape image information of the specific user included in the call image matches predetermined second triggering mouth shape information.
The method of claim 1,
The training data sets include: i) exemplary text information and ii) mouth image information from which the specific user has read the exemplary text information.
The method of claim 1,
Generating the training data sets,
Displaying example text information;
Acquiring a read image which is an image of the specific user while the specific user reads the example text information; And
And storing the second mouth shape image information and the example text information included in the read image as a second training data set.
The method of claim 1,
Generating the training data sets,
Displaying a plurality of example text informations;
Acquiring a read image that is an image for the specific user and a read voice that is a voice for the specific user while the specific user reads the example text information;
Associating third mouth shape image information, which is at least a part of the read image, with third voice information that is at least a part of the read voice, based on time information;
Obtaining third speech recognition text information, which is text information corresponding to the third speech information, based on a speech recognition model;
In response to determining that the third speech recognition test information and the third exemplary text information are the same as the third exemplary text information, the third mouth shape image information and the third exemplary text information are trained in a third manner. And storing the data as a data set.
An information input device based on a user image,
An image input unit which acquires image information;
A voice input unit for obtaining voice information;
A memory for storing image information, audio information, and text information; And
Includes a processor,
The processor,
Generating a plurality of training data sets each including mouth shape image information for a specific user and text information corresponding to the mouth shape image information,
Generating an image recognition model outputting text information corresponding to the input mouth shape image information based on the training data sets; And
And determine input text information corresponding to the input image for the specific user based on the image recognition model.
The method of claim 12,
Determining the input text information, the processor,
Receive an input image for the specific user acquired by the image input unit; And
And determining text information corresponding to mouth shape image information included in the input image as the input text information based on the image recognition model.
The method of claim 12,
The training data sets may include: i) mouth shape image information included in the video call data of the specific user; and ii) voice recognition text information that is a voice recognition result of a voice corresponding to the mouth shape image information. Information input device based on.
The method of claim 12,
Generating the training data sets, the processor,
Obtain video call data of the specific user, wherein the video call data includes call video and call voice;
Correlating first mouth image information that is at least a portion of the call video with first voice information that is at least a portion of the call voice based on time information;
Acquire first speech recognition text information that is text information corresponding to the first speech information based on a speech recognition model; And
And storing the first mouth shape image information and the first voice recognition text information as the first training data set in the storage unit.
The method of claim 15,
And a time length of the first mouth shape image information and the first voice information has a predetermined time length.
The method of claim 15,
The first mouth image information and the first audio information have a length of time from a first time point to a second time point,
The first time point indicates a time point at which the call voice is less than or equal to a predetermined threshold size, and the second time point is based on a user image indicating a time point after which the call voice is less than or equal to a predetermined threshold size. Information input device.
The method of claim 15,
The first mouth image information and the first audio information have a length of time from a first time point to a second time point,
The first time point represents a time point at which mouth shape image information of the specific user included in the call image coincides with predetermined first triggering mouth shape information.
And the second viewpoint is a point of time when the mouth shape image information of the specific user coincides with the predetermined first triggering mouth shape information subsequent to the first viewpoint.
The method of claim 15,
The first mouth image information and the first audio information have a length of time from a first time point to a second time point,
The first time point represents a time point at which mouth shape image information of the specific user included in the call image coincides with first predetermined triggering mouth shape information.
And the second view indicates a time point at which mouth shape image information of the specific user included in the call image matches predetermined second triggering mouth shape information.
The method of claim 12,
And the training data sets comprise i) exemplary text information and ii) mouth image information from which the specific user has read the exemplary text information.
The method of claim 12,
Generating the training data sets, the processor,
Displaying exemplary text information on a display unit included in the information input device based on the user image;
Using the image input unit, obtain a read image that is an image of the specific user while the specific user reads the example text information; And
And storing the second mouth shape image information and the example text information included in the read image as the second training data set in the storage unit.
The method of claim 12,
Generating the training data sets, the processor,
Displaying a plurality of example text informations on a display unit included in the information input device based on the user image;
Using the image input unit and the audio input unit, obtain a read image that is a video for the specific user and a read voice that is a voice for the specific user while the specific user reads out the example text information;
Correlating third mouth image information, which is at least a part of the read image, with third voice information that is at least a part of the read voice, based on time information;
Acquire third speech recognition text information that is text information corresponding to the third speech information based on a speech recognition model;
Correlating the third speech recognition test information with third example text information which is any one of the plurality of example text information; And
In response to determining that the third speech recognition test information and the third exemplary text information are the same as the third exemplary text information, the third mouth shape image information and the third exemplary text information are trained in a third manner. And storing the data in the storage unit as a data set.
An information input system based on user images,
Acquiring a plurality of training data sets each including mouth shape image information for a specific user and text information corresponding to the mouth shape image information, and based on the training data sets, text corresponding to the input mouth shape image information. A server, configured to generate an image recognition model that outputs information; And
And a terminal, configured to acquire at least one of image information and audio information for the specific user, and configured to determine input text information corresponding to the input image for the specific user based on the image recognition model. Information input device based on video.
A computer readable storage medium comprising instructions executable by a processor, the instructions being executed by the processor,
Generate a plurality of training data sets each comprising mouth shape image information for a specific user and text information corresponding to the mouth shape image information;
Generating an image recognition model outputting text information corresponding to the input mouth shape image information based on the training data sets; And
And determine input text information corresponding to an input image for the particular user based on the image recognition model.

Images