KR101894422B1 - lip recognition mobile control terminal - Google Patents

Abstract

The present invention relates to a lip recognition mobile control terminal and, more particularly, to a lip recognition mobile control terminal that can control a control target device such as a smart phone by analyzing the lip shape of a user. The lip recognition mobile control terminal includes a camera module, an image processing part, an image extracting part, a lip shape selecting part, a lip shape reading part, a control command generating part, a communication part, a memory, and a voice recognition part.

Description

A lip recognition mobile control terminal

The present invention relates to a lip recognition mobile control terminal, and more particularly, to a lip recognition mobile control terminal capable of controlling a control target device such as a smart phone by analyzing a lip shape of a user.

Recent advances in computing technology have helped people with physical disabilities to analyze images through neural networks to explain what objects are in front of them, and a variety of assistive technology techniques such as the ability to assist the eyes of the visually impaired using VR Is being developed.

In particular, auxiliary devices such as the above are being developed to assist persons with physical disabilities who are not free to use the body, such as a hand or an arm, to freely manipulate controlled devices such as smart phones.

Korean Patent No. 10-0957194, for example, proposes a 'mouse interface system and a control method between a human and a terminal', Korean Patent Laid-Open Publication No. 2013-0123316 discloses a device for controlling a mobile terminal And method.

However, in the conventional technology as described above, since the terminal itself, which is the control target device, needs to be improved or upgraded, the price of the terminal itself must be increased, and each of the terminals must have an auxiliary function. Therefore, there is a problem that it is difficult to access the disabled people due to the price burden.

In addition, as disclosed in Korean Patent Laid-Open Publication No. 2013-0123316, the method of recognizing a face of a user and transmitting a command is difficult for a user to input an accurate command, and it is difficult to intuitively understand each command. Particularly, The number of controllable instructions is extremely small.

On the other hand, the aforementioned Korean Patent No. 10-0957194 has a relatively large number of usable commands in terms of controlling the terminal by analyzing the mouth shape of the user, but the accuracy of the mouth shape detection and analysis is low, There is a problem in that it is difficult to generate various commands using a combination of voice and face directions.

Korean Patent No. 10-0957194 Korean Patent Publication No. 2013-0123316

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide an apparatus and a method for implementing the present invention, The present invention provides a lip recognition mobile control terminal capable of remarkably improving the number of commands by analyzing changes, voices and facial directions together.

To this end, the lip recognition mobile control terminal according to the present invention comprises a camera module for photographing a user's face; An image processor for signal processing the face image photographed by the camera module into a readable image; An image extracting unit for extracting a lip image of a face image signal processed by the image processing unit; A maximum frequency lip shape selection unit that receives a plurality of lip images extracted by the image extraction unit as a continuous frame and selects a lip image having the highest frequency among the plurality of lip image frames; A mouth shape reading unit for analyzing a lip image selected by the maximum frequency lip shape selecting unit to read a mouth shape; A control command generator for generating a device control command for controlling the control target device according to a mouth shape read by the mouth shape reader; A communication module for transmitting the device control command generated by the control command generation unit to the control target device; A mouth shape check unit for informing the user of the current mouth shape read by the mouth shape reading unit through a display device; A memory for previously inputting and storing a voice setting command including a first voice setting command indicating a control preparation from a user and a second voice setting command indicating a start of control; And a voice recognition unit for receiving a voice from a user and analyzing whether the first voice setting command or the second voice setting command recorded in the memory matches the first voice setting command or the second voice setting command, Generating a preparation command for activating a control screen displayed on the control target apparatus and generating the apparatus control command for controlling the control target apparatus when the second voice setting command is input.

The control command generation unit may further include a face direction reading unit for analyzing a face image signal-processed by the image processing unit and analyzing a direction of a face of the current user, And the face direction read by the face direction reading unit are combined to generate the device control command word.

The control command generation unit may further include a consecutive analysis unit for analyzing two consecutive mouth shapes provided from the mouth shape reading unit and providing the analysis result to the control command generation unit, .

In addition, the image extracting unit recognizes a face landmark by calling a Histogram of Oriented Gradients (HOG) algorithm in an Open Source Computer Vision Library (Open CV), extracts a lip image of the face image, The shape reading unit reads the mouth shape using a CNN (Convolutional Neutral Network) deep drawing that extracts features of an image by repeating the convolution and the pulling process, and the mouth shape check unit reads' , 'E', 'o', 'o', and 'right' to inform the user of the current mouth shape indicating any one of 'e', 'e', 'o' It is preferable to include a plurality of light-emitting LEDs.

In addition, the memory may further include a voice type device control command for controlling the control target device by using voice sounded with different sounds, respectively, and further stores the voice type device control command, Wherein the control command generation unit provides a device control command for controlling the control target device in accordance with the voice type device control command provided by the voice recognition unit in addition to the device control command for the mouth shape, .

The smartphone control system according to the present invention further includes a lip recognition mobile control terminal as described above and a smartphone receiving a device control command according to the lip shape from the lip recognition mobile control terminal and performing a specific function But; When a first voice setting command is input, selectable commands are listed on the smartphone screen to be displayed to the user, and when a second voice setting command corresponding to a specific command is input, the specific function control of the smartphone starts When the first voice setting command is input, commands corresponding to the respective vowels of 'ah', 'e', 'i', 'o', and 'right' are displayed in a pop-up form on the smartphone screen; When a command corresponding to a specific set of commands is selected and input, the smartphone screen is divided into a plurality of sections, and a command for selecting the corresponding section is displayed for each divided section.

While the present invention has been described with reference to the preferred embodiments thereof, it is to be understood that the invention is not limited thereto and that various changes and modifications may be made without departing from the scope of the invention as defined by the appended claims. The number of instructions can be greatly improved.

1 is a use state diagram of a lip recognition mobile control terminal according to the present invention.
2 is a block diagram of a lip recognition mobile control terminal according to the present invention.
FIG. 3 is an example of a source code for extracting the lip shape of the lip recognition mobile control terminal according to the present invention.
4 is a diagram illustrating a lip shape extraction state of the lip recognition mobile control terminal according to the present invention.
FIG. 5 is a diagram illustrating a method of driving the mouth shape reader of the lip recognition mobile control terminal according to the present invention.
FIG. 6 is an example of a control target device control state of the lip recognition mobile control terminal according to the present invention.
7 is a diagram illustrating a control preparation state of the lip recognition mobile control terminal according to the present invention.

Hereinafter, a lip recognition mobile control terminal according to a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the present invention will be described as a representative example in which the present invention is used for a person with a physical disability.

However, it will be obvious that the present invention can be extended to workers, drivers, and medical personnel who can not directly operate a mobile device due to various tasks such as operation using both hands, driving the vehicle, cooking, and the like.

As shown in FIG. 1, the lip recognition mobile control terminal 100 according to the present invention selects a command in the form of a mouth of a user and controls another device or a terminal (hereinafter referred to as a control target device) with a command generated using the command Is used.

At this time, the control target device 10 is a mobile communication terminal device such as a smart phone as an embodiment, and a person with a physical disability can mount the terminal 100 of the present invention on a cradle and fix it on a wheelchair or a bed.

The present invention is an integrated control terminal for receiving a mouth shape captured through a camera through a user interface (UI) and generating and transmitting a command corresponding to the mouth shape to control the control target device (10) In addition to devices, various applications such as various home appliances capable of home networking are possible.

Particularly, the present invention can be implemented at a low price as described later, so that it is possible to easily access the person with a physical disability, accurately generate a command for the control target device 10 by dividing the mouth shape, The voice and face directions are analyzed together to greatly improve the number of commands.

2, the lip recognition mobile control terminal 100 according to the present invention includes a camera module 110, an image processing unit 111, an image extraction unit 120, a maximum frequency lip shape selection unit 121, A shape recognition unit 130, a control command generation unit 140, a communication module 150, a mouth shape check unit 160, a memory 170, and a voice recognition unit 180.

In addition, the present invention may further include a face direction reading unit 122 and a continuation analysis unit 131 as another preferred embodiment, and the above-described various technical constructions of the present invention can be applied to the center Is controlled through the control device (C).

The lip recognition mobile control terminal 100 of the present invention can be implemented, for example, in Raspberry Pi, which is an ultra-small and ultra low-priced PC manufactured by the Raspberry Foundation of England.

At this time, the camera module 110 may be integrally provided in a terminal such as a raspberry pie or may be assembled separately, thereby photographing a user's face.

The image processing unit 111 performs signal processing of a face image captured by the camera module 110 with an image readable by the camera module 110. For example, the image processing unit 111 performs filter processing such as image compression, interpolation, and noise processing, Alternatively, the RGB image may be changed to an image of the YCbCr color space.

The image extracting unit 120 extracts a lip image of the facial image processed by the image processing unit 111. The image extracting unit 120 extracts a lip image for selecting a command to be transmitted to the control target apparatus 10, Only the image (i.e. mouth shape) is extracted separately.

The image extracting unit 120 may be implemented by calling various algorithms such as Haar Cascade Classifier or Histogram of Oriented Gradients (SVM Classifier) in an Open Source Computer Vision Library (Open CV).

Open CV is a cross-platform library of open source libraries focused on image processing that enables a variety of image processing, including binarization, noise reduction, outline detection, image transformation, and filters.

In this case, Haar Cascade Classifier among the classification algorithms corresponds to the image extraction algorithm that classifies based on the Haar feature. Three filters including Edge feature, Line feature, and Four-rectangle feature are used to filter the corresponding values And extracts feature values and learns and classifies them in a multistage manner.

The Haar Cascade Classifier has a relatively fast recognition speed. However, it is preferable that the HOG algorithm be used because it is confirmed that the recognition rate is somewhat lowered in a specific situation, for example, when the mouth is opened small, but is widely recognized.

HOG SVM (Support Vector Machine) Classifier algorithm recognizes face landmarks and extracts lip image of face image. It calculates local gradient in image, generates Local Histogram, And learns the feature value created.

In this case, in order to classify using the HOG algorithm, first, face learning data is required. In the present invention, a face is extracted based on facial landmark data announced by iBUG as an embodiment. same.

Accordingly, as shown in FIG. 4, the marker can be generated in the face image, and extraction of the mouth portion can be performed at a high recognition rate when extracting a desired face region (i.e., mouth region). In particular, it enables stable and real-time extraction of the mouth area even if the face is rotated or a specific part other than the mouth is hidden or not recognized.

On the other hand, the maximum frequency lip shape selection unit 121 receives a plurality of lip images extracted by the image extraction unit 120 in successive temporally continuous frames, and extracts the lip image having the highest frequency among the plurality of lip image frames Select.

For example, a group of images having the highest frequency among five to ten consecutive frames photographed within a set time range is selected, and the selected maximum frequency frame is determined to correspond to a command to be input by the user. Therefore, the user can grasp the intended mouth shape accurately.

The mouth shape reading unit 130 analyzes the lip image selected by the maximum frequency lip shape selection unit 121 to read a mouth shape. The mouth shape that is read out selects a command to be transmitted to the control target apparatus 10 And is matched 1: 1 with the command.

The mouth shape basically includes 'ah', 'e', 'i', 'oh', and 'right', and these mouth shapes designate different commands.

Of course, as will be described later, the present invention can also use a combination of a mouth shape and a face direction, using two consecutive mouth shapes, or designating a larger number of commands by using a voice including a consonant.

As shown in FIG. 5, the mouth shape reading unit 130 may use a CNN (Convolutional Neutral Network) deep-processing which extracts features of an image by repeating a convolution and a pooling process And reads the mouth shape.

Specifically, the mouth-shaped image extracted through the Open CV is extracted by repeating the convolution-pooling process to extract the characteristics of the image, and then inputting the characteristics into a Fully-Connected Neural Network to obtain the probability of each lip Finally, classify the shape of the lips.

At this point, you need to learn about mouth-shaped images to determine mouth shape, preferably using Google ™'s inception model. The inceptive model provides an algorithm to classify mouth shapes by combining multiple convolution and pooling layers.

For example, when five mouth shapes are classified as 'ah', 'e', 'i', 'o', and 'right', at least 100 Uses mouth-shaped data.

Next, the control command generation unit 140 generates a 'device control command' for controlling the control target device 10 according to the mouth shape read by the mouth shape reading unit 130 as described above. That is, one instruction word is generated corresponding to one mouth shape.

The communication module 150 transmits the device control command generated by the control command generation unit 140 to the control target device 10 as described above, and transmits the command data according to a wired or wireless communication protocol.

As shown in FIG. 6, when the control target device 10 is a smart phone, it transmits command data to a smartphone using a short-range wireless communication protocol such as Bluetooth. Accordingly, various commands such as item selection, touch, and menu opening are executed on the smartphone. For example, when the mouth shape corresponding to the first [e] is detected using the 'switch control' function built in the iOS, the control command generation unit 140 generates a control command for 'item selection' The communication module 150 transmits the control command word to the control target device 10. The control target device 10 displays icons arranged in a specific position (for example, a specific row or column) among the icons arranged on the screen through a built-in 'switch control' function or a separate application based on the received control command word (To represent various image processing such as boundary display, highlighting, solidification or color change so as to be distinguishable from other icons), and at the same time, an auxiliary line composed of a vertical line and a horizontal line intersecting each other on the screen is outputted. When the intersection of the auxiliary lines reaches the specific icon position, when the mouth shape corresponding to [i] is sensed, the control command generation unit 140 generates a control signal, The control target device 10 generates a control command for 'touch' and the corresponding control command is transmitted to the control target device 10 by the communication module 150. The control target device 10 touches the corresponding icon by the received control command word . The corresponding application may be executed immediately when the corresponding icon is touched. Alternatively, as shown in FIG. 6, the menu opening function may be performed according to the mouth shape detection corresponding to [o]. After opening the menu, you can control to perform a specific function by touching the icon inside the menu in the same way.

The mouth check unit 160 provides an interface for notifying the user of the current mouth shape read by the mouth shape reader 130 through a display device. If the mouth shape checker 160 has a display panel such as an LCD, Characters or symbols or mouth-shaped images.

However, it is desirable to provide a mouth-shaped check unit 160 using LEDs since the present invention solves the problem of making it difficult for a person with a physical disability to access due to a price burden as it is possible to realize a low cost.

The mouth-shaped check unit 160 using LEDs may be classified into five mouth shapes including 'ah', 'e', 'i', 'o' and 'right' So that the LED corresponding to the current mouth shape is emitted.

Accordingly, the user can check the current command (i.e., the current mouth shape) in real time and then transmit the command to the control target device 10, thereby improving the operational reliability and ease of use at the same time.

However, it is preferable that the present invention further includes the memory 170 and the voice recognition unit 180 in addition to the above-described configurations. In the memory 170, a first voice setup command indicating 'preparation for control' And a second voice setting command for notifying the start of the control.

The voice recognition unit 180 includes a voice recognition sensor function and is used not only for receiving the first voice setting command and the second voice setting command stored in the memory 170, The second voice setting command recorded in the memory 170 or the second voice setting command.

At this time, the control command generation unit 140 generates a preparation command for activating the control screen in the control target device 10 when the first voice setting command is input through the voice recognition unit 180. [ The control target device 10 is ready to receive a command by the generated activation preparation command.

For example, when the first voice setting command indicating 'preparation for control' is input from the user as shown in FIG. 7A, a command word which can be selected is displayed on the smartphone as the control target device 10 and displayed to the user. A pop-up window can be used for the listed commands, and there are no special restrictions such as a screen auxiliary line (cross shape) format as shown in Fig. For example, as shown in (a) of FIG. 7, commands corresponding to each vowel such as 'ah', 'ei', 'i', 'oh', and 'right' may be displayed in a pop-up form. When the second voice setting command for notifying the start of control is inputted, the mouth shape is extracted and analyzed as described above, and the actual control of the smartphone is started. For example, when the user selects an instruction corresponding to 'A' in the screen of FIG. 7A, the selection screen as shown in FIG. 7B is activated. That is, the smartphone screen, which is the control target device 10, is divided into a plurality of zones to be displayed (a border line is displayed or rendered stereoscopic), and a command for selecting the zone is displayed for each divided zone. For example, as shown in FIG. 7 (b), the smartphone screen is divided into three areas, each of which is classified into three areas, and each area displays commands such as 'a', 'e', and 'i'. Then, when a command corresponding to 'E' is selected by the user, as shown in (c) of FIG. 7, the 'E' zone is further divided into a plurality of zones, The command is displayed together. By repeating this process, the application can be executed by touching a specific icon finally. The division of the area is performed by a unit provided by the OS of the smartphone or a separate application, so that the menu can be selected in a finer granularity.

Furthermore, it is preferable that the present invention further includes a face direction reading unit 122 for analyzing the face image signal processed by the image processing unit 111 and analyzing the direction of the face of the current user.

The control command generation unit 140 generates a control command for the mouth shape and the face direction reading unit 122 read by the mouth shape reading unit 130, ) To generate a device control command.

For example, if five mouths can be selected including 'ah', 'e', 'i', 'oh', and 'right', and the face direction is up to four, 20 (4 x 5) device control commands can be selected by combining the shape and the face direction.

The present invention further includes a continuous analysis unit 131 for analyzing two consecutive mouth shapes provided from the above-described mouth shape reading unit 130 and providing the same to the control command generation unit 140. Therefore, it is preferable that the control command generation unit 140 generates device control commands corresponding to two mouth shapes.

For example, it accepts consecutive mouth-shaped information such as 'ah', 'ei', 'i', 'o', 'ou' Instead of the instruction corresponding to ',' to ',' to ',' to ', and' right '.

In addition, the present invention can selectively generate and execute a command for a complex command by sequentially generating and executing commands corresponding to 'a' and 'i' when 'eye' is input.

In addition, the present invention may be configured such that the 'voice type device control command' for controlling the control target device 10 using the voice sounded in different sounds in the memory 170 described above is inputted in advance and further stored desirable.

At this time, the voice recognition unit 180 confirms and inputs the voice type device control command to the control command generation unit 140 when a voice type device control command similar to or similar to the one stored in the memory 170 is input from the user. Then, the control command generation unit 140 generates a device control command according to the voice type device control command.

Therefore, the control command generator 140 generates a device control command in accordance with the voice type device control command provided by the voice recognition unit 180 in addition to the mouth control device control command, thereby selecting a larger number of commands .

In particular, in the case of a person with physical disabilities who has difficulty in correctly pronouncing, it is possible to store voice type device control commands directly in the memory 170 using their voices, and to use them to generate commands.

In addition, it is possible to more accurately recognize various vowels, such as 'a', 'd', 'bar', or 'a', 'a', 'o' Allows you to create commands.

The specific embodiments of the present invention have been described above. It is to be understood, however, that the spirit and scope of the invention are not limited to these specific embodiments, but that various changes and modifications may be made without departing from the spirit of the invention, If you are a person, you will understand.

Therefore, it should be understood that the above-described embodiments are provided so that those skilled in the art can fully understand the scope of the present invention. Therefore, it should be understood that the embodiments are to be considered in all respects as illustrative and not restrictive, The invention is only defined by the scope of the claims.

110: camera module
111:
120: image extracting unit
121: maximum frequency lip shape selection unit
122: face direction reading section
130: mouth shape reading unit
131: Continuous analysis section
140: Control command generation unit
150: Communication module
160: mouth shape check part
170: memory
180:
C: central processing unit (CPU)
10: Controlled device (smartphone)

Claims (7)

A camera module 110 for photographing a user's face;
An image processing unit (111) for signal processing a face image photographed by the camera module (110) into a readable image;
An image extraction unit (120) for extracting a lip image of a face image signal processed by the image processing unit (111);
A maximum frequency lip shape selection unit 121 for receiving a plurality of lip images extracted by the image extraction unit 120 as continuous frames and selecting a lip image of the group having the highest frequency among the plurality of lip image frames, ;
A mouth shape reading unit 130 for analyzing a lip image selected by the maximum frequency lip shape selecting unit 121 and reading a mouth shape;
A control command generation unit 140 for generating a device control command for controlling the control target device 10 according to the mouth shape read by the mouth shape reading unit 130;
A communication module 150 for transmitting the device control command generated by the control command generation unit 140 to the control target device 10;
A mouth shape check unit 160 for informing the user of the current mouth shape read by the mouth shape reading unit 130 through a display device;
A memory (170) for previously receiving and storing a voice setting command including a first voice setting command indicating the preparation of control from the user and a second voice setting command informing the start of the control; And
And a voice recognition unit 180 for receiving a voice from a user and analyzing whether the voice is matched with a first voice setting command or a second voice setting command recorded in the memory 170,
The control command generating unit 140 generates a preparation command for activating a control screen displayed on the control target device 10 when the first voice setting command is input, 10, < / RTI >
And a continuous analysis unit 131 for analyzing two consecutive mouth shapes provided from the mouth shape reading unit 130 and providing the analysis result to the control command generation unit 140. The control command generation unit 140 Generates a device control command corresponding to the two mouth shapes,
The image extracting unit 120 recognizes a face landmark by calling a Histogram of Oriented Gradients (HOG) algorithm in an Open Source Computer Vision Library (Open CV), extracts a lip image from the face image, The mouth shape reading unit 130 reads the mouth shape by using CNN (Convolutional Neutral Network) deep drawing which extracts features of an image by repeating the convolution and the pulling process, The checking unit 160 may check whether the user has a current mouth shape indicating 'ah', 'e', 'i', 'oh' or ' , 'O', and 'W', respectively,
When the first voice setting command is input, the control command generator 140 displays selectable commands on the smartphone screen and is displayed to the user, and the second voice setting command corresponding to a specific command among the displayed commands is input Controls the smartphone to start controlling the specific function of the smartphone,
When the first voice setting command is input, the control command generation unit 140 generates a command corresponding to each word of 'a', 'e', 'a', 'o' Wherein when a command corresponding to a specific vowel of the displayed command is selected and inputted, the smartphone screen is divided into a plurality of sections and displayed, and a command for selecting the corresponding section is displayed for each divided section Wherein the control unit controls the smartphone. The method according to claim 1,
And a face direction reading unit (122) for analyzing a face image signal processed by the image processing unit (111) and analyzing a direction of a face of the current user,
Wherein the control command generation unit generates the device control command by combining the mouth shape read by the mouth shape reading unit (130) and the face direction read by the face direction reading unit (122) Mobile control terminal. delete delete The method according to claim 1,
In the memory 170, voice type device control commands for controlling the control target device 10 using voice sounded in different sounds are respectively input and stored in advance,
The speech recognition unit 180 provides the control command generation unit 140 with a voice type device control command stored in the memory 170,
The control command generator 140 generates a device control command for controlling the control target device 10 according to the voice type device control command provided by the voice recognition unit 180 in addition to the mouth control device control command Wherein the lip recognition mobile control terminal comprises: delete delete

Images