KR101227883B1 - Control device based on user motion/voice and control method applying the same - Google Patents

Abstract

A user motion based control apparatus and a user motion based control method applied thereto are provided. The user's motion-based control device recognizes a user's face using an image captured by a camera, extracts configuration information corresponding to the recognized user, and uses the data detected by the depth measurement sensor as the configuration information. Can be processed according to, the user can more easily control the electronic device.

Description

Control device based on user motion / voice and control method applied to it {Control device based on user motion / voice and control method applying the same}

The present invention relates to a user motion / voice based control device and a user motion based control method applied thereto, and more particularly, to a user motion based control device for controlling another device by recognizing a user's motion and a user motion based control method applied thereto. It is about.

Due to the improvement of the technical level and the economic level, the spread of the television award receiving and outputting video and audio signals transmitted from broadcasting stations has become popular. Is evolving into a form that can be implemented further.

In general, channel selection / volume adjustment or the like of the television receiver is controlled through input of a function button exposed on the outer surface of the television receiver or by using a remote controller at a short distance / distance.

In other words, the technology for controlling television has begun by mechanically adjusting channels and volumes to generate corresponding electric signals, and has developed into an electronic method of transmitting signals by touching a PCB circuit board to a television and pressing a button. Has evolved into a control method using a remote control, which is a generalized control method.

However, various types of user interfaces have been developed at present, and as a part thereof, user interfaces for recognizing a user's operation have been developed. Accordingly, there is a demand for a scheme for a user to more easily control an electronic device such as a television.

The present invention has been made to solve the above problems, an object of the present invention is to recognize the user's face using the image taken through the camera, extract the configuration information corresponding to the recognized user, A user motion based control apparatus for processing data sensed by a depth measuring sensor according to the configuration information and a user motion based control method applied thereto.

According to an embodiment of the present invention for achieving the above object, a user motion-based control device, a camera for photographing a subject; A depth measuring sensor for detecting a user's motion; And recognizing a user's face using the image photographed by the camera, extracting configuration information corresponding to the recognized user, and processing data detected by the depth measurement sensor according to the configuration information. It includes a control unit.

The environment setting information may include user interface information including commands according to user-specific operations, and the controller may apply the user interface information according to a user recognized using the face of the user.

The apparatus may further include a microphone configured to receive a voice of the user, and the controller may recognize the user by using the face of the user included in the image captured by the camera and the input voice of the user.

The apparatus may further include a microphone configured to receive a voice of a user, and the controller may generate a command corresponding to the input voice of the user.

The controller may receive a command corresponding to a specific operation by the user.

On the other hand, a user motion based control method according to an embodiment of the present invention, the step of photographing the user through the camera; Recognizing a face of a user using an image photographed by the camera; Applying environment setting information corresponding to the recognized user; Detecting a user's motion using the skeleton tracking data; And processing the data sensed by the depth measuring sensor according to the environment setting information.

The environment setting information may include user interface information including commands according to user-specific operations, and the applying of the environment setting information may include applying the user interface information according to a user recognized using a face of the user. It may be.

The method may further include receiving a voice of the user, wherein the recognizing of the user may recognize the user by using the face of the user included in the image captured by the camera and the input voice of the user. .

In addition, receiving a user's voice; And generating a command corresponding to the input voice of the user.

The method may further include receiving a command corresponding to a specific operation / voice by the user.

According to various embodiments of the present disclosure, a face of a user may be recognized by using an image photographed by a camera, an environment setting information corresponding to the recognized user may be extracted, and the data detected by a depth measurement sensor may be set. It is possible to provide a user motion based control apparatus for processing according to the information and a user motion based control method applied thereto, so that the user can more easily control the electronic device.

1 is a diagram illustrating a state in which a user motion based control apparatus is mounted on a TV according to an embodiment of the present invention;
2 is a diagram showing the configuration of a user motion based control apparatus according to an embodiment of the present invention;
3 is a flowchart provided to explain a user motion based control method according to an embodiment of the present invention, and
4 is a diagram illustrating a TV with a built-in user motion control device according to an embodiment of the present invention.

Hereinafter, with reference to the drawings will be described the present invention in more detail.

1 is a diagram illustrating a state in which a user motion based control apparatus 100 is mounted on a TV 200 according to an embodiment of the present invention. The user motion-based control device 100 detects a user's face and motion / voice, and the TV 200 is controlled according to the detected user's face and motion / voice.

2 is a diagram illustrating a configuration of a user operation based control apparatus 100 according to an embodiment of the present invention. As shown in FIG. 2, the user motion based control apparatus 100 includes a camera 110, a depth measurement sensor 120, a microphone 130, a controller 140, and an output unit 150. In addition, although the depth measuring sensor 120, the camera 110, and the microphone 130 are illustrated as separate in FIG. 2, the camera 110 and the microphone 130 are embedded in the depth measuring sensor 120. It may be implemented as.

The camera 110 photographs a subject and provides the photographed image signal to the controller 140. The camera 110 detects light passing through the lens and generates an image signal of an image of a subject. In the case of a digital camera, image sensors are arranged in the imaging area. In the present embodiment, a case in which the camera 110 is a digital camera will be described. Image sensors used in digital cameras are generally divided into charge coupled devices (CCD) and complementary metal oxide semi-conductors (CMOS). The method is different but the basic principle is the same. CCDs are sensitive to weak light and have high image quality and are used in most digital cameras. However, there are disadvantages in that the production process is difficult and the production cost is high. On the other hand, CMOS has been used as a sensor for low-cost digital cameras because it has a lower production cost and a relatively easy production process than CCDs. Recently, the development of image processing technology enables high-quality, high-quality images to be used in professional DSLRs. While being used a lot. The camera 110 photographs the user, generates an image signal for the photographed user, and transmits the image signal to the controller 140.

The depth measurement sensor 120 generates depth data by measuring a depth of a specific area, and generates skeleton tracking data and a depth measurement image of a user included in the specific area by using the measured depth data. Here, the depth measurement image represents image data obtained by expressing the size of the depth data for a specific region in a specific color, saturation, contrast, and the like.

Specifically, the depth measurement sensor 120 shoots a large amount of infrared light in a specific area to scan the user. Then, the infrared rays hitting the user's body are reflected back to the depth measurement sensor 120. The depth sensor 120 senses the reflected infrared rays for each pixel, measures the depth for each pixel, and generates depth data for the entire specific region using the depth for each pixel.

Thereafter, the depth measurement sensor 120 extracts the skeleton tracking data of the user using the depth data for the specific region. In detail, the depth measuring sensor 120 determines the area protruding in the shape of the user as the area where the user is located, and extracts a position corresponding to a predetermined skeleton and joint part. The depth sensor 120 calculates skeletal tracking data for the user by connecting the positions corresponding to the skeletal and the joint parts. Thereafter, the depth measurement sensor 120 continues to track the movement of the position corresponding to the predetermined skeleton and joint part to detect the skeleton tracking data corresponding to the user's motion. Thereafter, the depth measurement sensor 120 transmits the generated skeleton tracking data to the controller 140.

In addition, the depth measurement sensor 120 may measure depth data of a specific region and transmit the depth data to the controller 140, and the controller 140 may generate skeleton tracking data of the user.

Through this process, the depth measurement sensor 120 recognizes the user's motion.

The microphone 130 detects a user's voice and transmits the detected voice data to the controller 140.

The storage unit 160 stores various user information and environment setting information for each user. The user information includes personal information, face information, body shape information, and voice information of a user registered in the user motion based control apparatus 100.

The user's personal information includes a user name, ID, and the like.

The face information is information used for face recognition such as a face photograph of a user and feature points of a face. Face information is used to recognize a user by using a face included in a captured image.

In addition, the user body type information indicates shape information about a body type for each user. Since height and weight are different for each user, the user may be used as auxiliary information for recognizing the user by using the user's body type information. In addition, the user body type information may be used to increase the recognition rate of the user's motion.

The voice information of the user is stored as voice information for each user. Specifically, user-specific voice information may be stored for frequency bands and tones, respectively. The user-specific voice information is used to distinguish whose voice the voice input through the microphone 130 is. In addition, the voice information of the user may include commands corresponding to a specific user voice.

In addition, the environment setting information for each user is environment setting information for each user registered for each user. The user setting information for each user includes user interface information based on actions for each user. Here, the user interface information represents information on control commands of a specific device according to the user's operation. The same operation and voice input may also be set to different commands for each user. For example, the 'hurray operation' may be set as a volume up command in the user-specific environment setting information for the user A, or may be set as a channel up command for the user B. In addition, the user interface information may include control commands of a specific device according to the voice of the user and information about control commands of the specific device according to the user's operation and voice.

The controller 140 recognizes a user's face using the image photographed by the camera 110, extracts configuration information corresponding to the recognized user, and uses the data sensed by the depth measurement sensor 120. Process according to the environment setting information corresponding to.

In detail, the controller 140 distinguishes a face from an image photographed by the camera 110 or a depth measurement image extracted by the depth measurement sensor 120. The controller 140 compares the face included in the captured image with the face information stored in the storage 160 to recognize who the user includes the face in the captured image. Specific face recognition technology is as follows.

Face recognition technology is a very complex and variable technology and its software development has been made a lot recently. First, when the camera captures an image of a face, the face recognition technology is started by comparing the image with templates.

The technology underlying the face recognition is divided into two types. The first is to compare the feature values of each component of the face and its interrelationships, such as the nose length and the distance from the eye, and the second most important image data of the face. This is a way to compare and match the size of a nose-for example, with the data of a face stored in a database.

Face recognition technology has various methods as follows. First of all, there is a geometrical feature analysis using geometrical features of the face. Recognition technology using the facial features of the face is focused on the fact that each individual's face can be recognized only by geometric factors such as the position and size of the feature points of the face such as eyes, nose and mouth, or the distance between them. Geometric features correspond to the last remaining elements when the resolution of the input image is lowered. This is the most commonly used methodology for face recognition.

There is also an Eigenface approach. Eigenface means 'unique face', a method proposed by Sirovich and Kirby of MIT. It is a technique that enables the low-dimentional representation of a human face to analyze the principal components of the face image. In terms of features, mathematically, it means an Eigenvector of a covariance matrix representing a group of face images. First, the face images inputted to the above basic elements are projected, and the features are extracted by comparing or correlating them with the projections of the stored face images. The extracted features are called 'Eigenface' or 'Holon', which can be classified as weighted multi-template matching.

In addition, there are various face recognition techniques such as a template matching approach and a neural network mapping approach using an artificial neural network, and the controller 240 uses any one of such face recognition techniques. The user's face is recognized, and the position of the face can be calculated within a specific area.

Thereafter, the controller 140 applies the environment setting information corresponding to the recognized user among the stored user setting information for each user. The controller 140 determines the type of the user's motion by using the skeleton tracking data sensed by the depth measuring sensor 120. Therefore, the controller 140 performs a control command corresponding to the applied configuration information by using the type of the user's motion according to the bone tracking data sensed by the depth sensor 120.

In addition, the controller 140 may distinguish whose voice the voice signal input through the microphone 130 is. In detail, the controller 140 extracts voice information (eg, frequency band and tone information) of the voice signal input through the microphone 130. The controller 140 compares the user-specific voice information stored in the storage unit 160 with the voice information of the input voice signal to distinguish the user.

As such, the controller 140 may recognize the user by using face information and voice information. In addition, the controller 140 selects a control command for the TV 200 corresponding to the user's motion by using the skeleton tracking data which is data on the user's motion. In addition, the controller 140 may generate a control command for the TV 200 corresponding to the voice of the user using the voice of the user. The controller 140 may select a control command for the TV 200 by combining a user's voice and an operation.

In addition, the controller 140 may directly transmit to the TV 200 without processing at least one of the detected skeleton tracking data, voice data, and face information. In this case, the TV 200 directly processes at least one of skeleton tracking data, voice data, and face information to perform a corresponding control command.

In addition, the controller 140 may collect an operation pattern and related information according to a user and store the same in the storage 160. In addition, when calculating the skeleton tracking data of the user, the controller 140 may increase the accuracy of the skeleton tracking data calculation using the data collected for the user's operation pattern.

In addition, the controller 140 may directly set a command according to an operation by the user. In detail, the controller 140 may set a control command corresponding to the user's motion detected by the depth measurement sensor 120 according to the user's selection. That is, the user may set the hurray to be set to the channel up command.

The output unit 150 outputs a control command or various data output from the controller 140 to the TV 200. The output unit 150 outputs data by wire or wirelessly. The output unit 150 transmits data to the TV 200 through a Bluetooth or a wireless LAN when wireless.

The user's motion-based control device 100 having such a structure recognizes a user's motion and performs a command corresponding to the motion. In addition, the user motion based control apparatus 100 may execute different commands for each user even for the same operation and voice input. For example, the 'hurray operation' may be set as a volume up command in the user-specific environment setting information for the user A, or may be set as a channel up command for the user B. In addition, the user motion based control apparatus 100 may execute control commands of a specific device according to a user's voice and control commands of the specific device according to a user's motion and voice.

Hereinafter, referring to FIG. 3, a user operation based control method will be described in detail. 3 is a flowchart provided to explain a user motion based control method according to an embodiment of the present invention.

First, the user motion based control apparatus 100 photographs the user through the camera (S310). Then, the user's motion-based control device 100 recognizes the user's face in the image captured by the camera 110 (S320). In detail, the user motion-based control apparatus 100 distinguishes a face from an image captured by the camera 110 or a depth measurement image extracted by the depth measurement sensor 120. In addition, the user's motion-based control device 100 compares the face included in the captured image with the face information stored in the storage 160 to recognize who is the user whose face is included in the captured image.

Thereafter, the user operation based control apparatus 100 applies the environment setting information corresponding to the recognized user among the stored user setting information (S330).

Next, the user motion-based control device 100 detects the user's motion and receives the skeleton tracking data through the depth measurement sensor 120 (S340). In operation S350, the user motion based control apparatus 100 may determine the type of the user's motion using the skeleton tracking data.

Thereafter, the user motion-based control device 100 performs a control command according to the applied configuration information corresponding to the type of the user's motion according to the skeleton tracking data (S360). Accordingly, the user operation based control apparatus 100 may execute different commands for each user even for the same operation and voice input. For example, the 'hurray operation' may be set as a volume up command in the user-specific environment setting information for the user A, or may be set as a channel up command for the user B.

In addition, the user motion based control apparatus 100 may execute control commands of a specific device according to a user's voice and control commands of the specific device according to a user's motion and voice.

In addition, the user's motion-based control device 100 may distinguish whose voice the voice signal input through the microphone 130 is. In detail, the user motion based control apparatus 100 extracts voice information (eg, frequency band and tone information) of a voice signal input through the microphone 130. In addition, the user operation-based control apparatus 100 compares the user-specific voice information stored in the storage unit 160 with the voice information of the input voice signal to distinguish the user.

As such, the user motion-based control apparatus 100 may recognize the user by using face information and voice information. In addition, the user motion-based control apparatus 100 selects a control command for the TV 200 corresponding to the user's motion by using the skeleton tracking data which is data on the user's motion. In addition, the user operation based control apparatus 100 may select a control command for the TV 200 corresponding to the user's voice using the user's voice. The user motion based control apparatus 100 may select a control command for the TV 200 by combining the voice and the motion of the user.

In addition, the user's motion-based control device 100 may directly transmit to the TV 200 without processing at least one of the detected skeleton tracking data, voice data, and face information. In this case, the TV 200 directly processes at least one of skeleton tracking data, voice data, and face information to perform a corresponding control command.

In addition, the user motion based control apparatus 100 may collect an operation pattern and related information according to a user and store it in the storage 160. In addition, when calculating the user's skeleton tracking data, the user's motion-based control apparatus 100 may increase the accuracy of calculating the skeleton tracking data by using the data collected for the user's motion pattern.

In addition, the user operation based control apparatus 100 may directly set a command according to the operation by the user. In detail, the user motion based control apparatus 100 may set a control command corresponding to the user's motion detected by the depth measurement sensor 120 according to the user's selection. That is, the user may set the hurray to be set to the channel up command.

Through such a process, the user operation based control apparatus 100 may control the TV 200 according to the user's operation.

Meanwhile, in the present exemplary embodiment, the user motion based control apparatus 100 is described as a separate device from the TV 200, but this is only an example. Thus, as shown in FIG. Of course, the 100 may be embodied as being embedded in the TV 200.

Meanwhile, in the present embodiment, the device to be controlled is described as the TV 200, but other devices besides the TV may also be to be controlled. In addition, the user motion based control apparatus 100 may control a specific device not only by a wired device but also wirelessly.

On the other hand, the technical idea of the present invention can be applied to a computer-readable recording medium containing a computer program for performing a user operation based control method according to the present embodiment. In addition, the technical idea according to various embodiments of the present disclosure may be implemented in the form of computer readable codes recorded on a computer readable recording medium. The computer-readable recording medium is any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. In addition, the computer readable code or program stored in the computer readable recording medium may be transmitted through a network connected between the computers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

100: user motion based control device 110: camera
120: depth measuring sensor 130; MIC
140: control unit 150: output unit
160:

Claims (10)

A camera for photographing a subject;
A depth measuring sensor for detecting a user's motion; And
Recognizing a face of a user using an image photographed by the camera, extracting preference information registered to the recognized user from among preference information registered for each user, and of the user detected by the depth measurement sensor And a controller configured to process according to the configuration information extracted from the operation.
The environment setting information,
And a control command indicated by the user's operation is information set separately for each user. delete The method of claim 1,
The microphone further receives a user's voice;
The control unit,
And a user's face based on the user's face included in the image captured by the camera and the input user's voice. The method of claim 1,
The microphone further receives a user's voice;
The control unit,
And generating a command corresponding to the input voice of the user. The method of claim 1,
The control unit,
And a command corresponding to a specific operation by the user. Photographing the user through a camera;
Recognizing a face of a user using an image photographed by the camera;
Extracting environment setting information registered to a user recognized in the recognizing step, from among environment setting information registered for each user;
Detecting a user's motion;
And processing the operation of the user according to the configuration information extracted in the extracting step.
The environment setting information,
And a control command indicated by the user's operation is information set separately for each user. delete The method according to claim 6,
Receiving a user's voice;
The user recognition step,
And a user's face based on a user's face included in the image captured by the camera and the input user's voice. The method according to claim 6,
Receiving a voice of a user; And
And generating a command corresponding to the input voice of the user. The method according to claim 6,
Receiving a command corresponding to a specific operation by the user; User-based control method further comprising a.

Images