KR20170067398A

KR20170067398A - User interface control method and system using triangular mesh model according to the change in facial motion

Info

Publication number: KR20170067398A
Application number: KR1020150174033A
Authority: KR
Inventors: 김영섭; 박인호; 이용환; 한우리
Original assignee: 단국대학교 천안캠퍼스 산학협력단
Priority date: 2015-12-08
Filing date: 2015-12-08
Publication date: 2017-06-16
Also published as: KR101909326B1

Abstract

A user interface control method and system utilizing a triangular mesh model according to a change in a face motion are disclosed. The user interface control method includes: detecting a face region in each of a first frame input and a second frame input after the first frame, extracting feature points from the face region, generating a triangle mesh based on the feature points, Comparing the first frame and the second frame to track the change of the triangle mesh, determining whether the change of the triangle mesh is above the threshold, and generating a control event according to the face motion corresponding to the change of the triangle mesh above the threshold .

Description

TECHNICAL FIELD [0001] The present invention relates to a user interface control method and system using a triangle mesh model according to a face motion change,

Embodiments of the present invention relate to a user interface control technique, and more particularly, to a user interface control method and system using a triangular mesh model according to a change in a face motion.

The user interface using the conventional face recognition mainly refers to the control through pupil recognition of the face. Such conventional control through face recognition can be used only when both eyes of both faces of the user are recognized by the input device.

However, the conventional control through face recognition is a system for grasping the movement of the pupil of the user's face, and is advantageous in that both hands can be used freely. However, in a mobile device such as a mobile terminal, .

The object of the present invention is to solve the problem of the control through the conventional face recognition as described above, and it is an object of the present invention to provide a method and apparatus for improving face motion recognition, And an interface control method and system.

According to an aspect of the present invention, there is provided an interface for controlling a system operation through a triangular mesh model according to a user's face motion for a mobile device and a desktop device equipped with a camera.

That is, in one aspect of the present invention, a face region of a user is detected with respect to an image input through an external input device (e.g., a camera), the feature points of the user's face are extracted from the region, And controlling the operation of the system by tracking the amount of change of the triangular mesh that changes according to the movement of the user's face.

According to another aspect of the present invention, there is provided an image processing method comprising the steps of: (a) extracting, from the input image, user's facial contours, eyebrows, eyes, eyes, nose, mouth, A triangle mesh is generated through a line segment connecting the distances between the points, and a vector component of the generated triangle mesh is traced to determine a gradient that varies according to the mesh model And a user interface control method capable of outputting a user interface signal for a device to be controlled or a signal for supporting user interaction by calculating a trigonometric function and a moving distance of the distance between the points do.

According to another aspect of the present invention, there is provided a method of detecting a face region in a face region, the method comprising the steps of: detecting a face region in each of an input first frame and a second frame input after the first frame; Generating a triangle mesh based on the feature points, comparing the first frame with the second frame to track the change of the triangle mesh, determining whether the change of the triangle mesh is above the threshold, and changing the triangle mesh above the threshold And generating a control event in accordance with the corresponding face motion.

According to another aspect of the present invention, there is provided an image processing apparatus including a detection unit detecting a face region in each of a first frame input and a second frame input after a first frame, an extraction unit extracting feature points from the face region, A generating unit for generating a triangle mesh based on the feature points, a tracking unit for tracking the change of the triangle mesh by comparing the first frame and the second frame, a determining unit for determining whether the change of the triangle mesh is above a threshold, And an output unit for generating a control event according to the face motion corresponding to the change of the face motion.

The user interface control method and apparatus using the triangular mesh model according to the present invention can provide an improved recognition rate because it includes various feature point distributions in addition to eye pupil and eyes of the user's face. , And other strong elements (glasses, sunglasses, hats, etc.), and more robust user motion control is possible.

In addition, when the user interface control method and apparatus of the present invention is used, since the triangular mesh model is generated based on the feature points in tracking the user's face motion, , The triangular mesh change amount can be tracked regardless of the user's face direction and the camera position direction. The exclusion of such constraints can provide a wider range of applications. For example, when tracking eyes or lips, etc., there is a limitation that the position of the camera is limited to the front, but the method and apparatus of the present invention are greatly free from such limitation.

In addition, when the user interface control method and apparatus of the present invention is used, a control system (e.g., Kinect) through conventional user's body motion recognition supports only input from a fixed camera device in order to track movement of two hands In comparison, limitations can be overcome in mobile devices with convenient mobility. That is, according to the present invention, it is possible to analyze the face recognition by tracking the movement of the face of the camera while moving it in the hand, so that the user can freely use the two hands Can be provided.

In addition, when the user interface control method and apparatus of the present invention is used, since the behavior recognition object is the user's recognition through the facial expression and the tracking, not the gesture, the gesture, the degree of freedom of the user is high and the physical weakness, There is an advantage that it can be utilized also as a user interface (particularly, a user interface for a mobile device) that can easily control an inconvenient user and an elderly user.

1 is a hardware block diagram of a user interface control apparatus according to an embodiment of the present invention.
2 is a block diagram of a program block that can be employed in a user interface control apparatus according to another embodiment of the present invention.
3 is a flowchart of a user interface control method according to another embodiment of the present invention.
4 is a schematic block diagram of a computing device capable of employing the user interface control apparatus of FIG.
5 is a configuration diagram of a user interface control device implemented in the computing device of FIG.
6 is a configuration diagram of a tracking unit of the user interface control apparatus of FIG.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless otherwise defined herein, all terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a hardware block diagram of a user interface control apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a user interface control apparatus according to an embodiment of the present invention includes a processing device 100 and a camera module 110. Here, the processing apparatus 100 may be connected to a triangular mash model 200, and the triangle mesh model 200 may be connected to training data 400.

The processing apparatus 100 is an image processing apparatus that includes at least some functional units of a processor capable of processing an input stream corresponding to a still image or a moving image and at least some constituent units of a processor performing functions corresponding to the functional units Can be implemented. The processing device 100 may extract the face region from the input stream including the face image of the user. Further, the processing apparatus 100 can extract feature points from the face region.

Such a processing apparatus 100 may be implemented by a processor mounted on a mobile terminal, a notebook computer, a desktop computer, or the like, but is not limited thereto. For example, the processing apparatus 100 is connected to the camera module 110 and is mounted on an apparatus connected to an internet of the network (IOT), and outputs a control event signal to an external device such as a mobile terminal, a notebook computer, And to output a control signal for operation.

The camera module 110 obtains or generates an input stream including the face information of the user, and transfers the input stream to the processing device 100. [ The camera module 110 may be integrally coupled to the processing apparatus 100, but is not limited thereto. The camera module 110 may be installed in a separate camera device and connected to the processing device 100 through a wired or wireless network.

The triangle mesh model 200 connected to the processing apparatus 100 in this embodiment generates a triangle mesh based on the facial feature points. The triangular mesh model 200 may be implemented by being stored in the memory of the processing apparatus 100 and executed by the processor of the processing apparatus 100, but is not limited thereto. The triangle mesh model 200 may be implemented such that it is stored in a memory of another apparatus to which the processing apparatus 100 is connected or is executed by a processor of another apparatus and returns only the result to the processing apparatus. In that case, the user interface control apparatus according to the present embodiment may be implemented to include other apparatuses including the triangle mesh model 200 connected to the processing apparatus 100 and the processing apparatus 100 in a broad sense.

The triangle mesh model 200 can generate triangle meshes based on facial feature points in various ways.

The triangular mesh model 200 applicable to the present embodiment sets one of the minutiae obtained in the face region as the main axis, divides the face region into a plurality of portions based on the set main axis, And a method of generating a triangular mesh using a 2D Delaunay triangulation.

The triangular mesh model 200 according to the present embodiment can be implemented by using Delaunay triangulation that connects the facial feature points with triangles to divide a plane or a space so that the internal angle of the triangles becomes a minimum value have.

The triangle mesh model 200 according to the present embodiment can be implemented by generating a triangle mesh based on a tree structure in which the facial feature points can reach each other and the sum of the total distances is minimized.

It should be understood that the triangular mesh model 200 according to the present embodiment can be implemented by a combination of at least some of the above-described various methods.

The training data 400 is coupled to the triangle mesh model 200. Training data 400 may support generating an optimal triangulation of facial feature points based on previously learned known properties in the triangle mesh formation of triangle mesh model 200. [ For example, the training data 400 may be implemented to select a triangle mesh generation procedure according to characteristics and classification of data of various face images and facial image data.

Such training data 400 may be provided to a computing device on which the processing apparatus 100 is mounted, but is not limited thereto. The training data 400 may be implemented in a computing device that supports machine learning to interact with the triangle mesh model 200 through a network.

According to the present embodiment, the user interface control apparatus acquires an input stream including a user's face image through the camera module 110, extracts a face region from the face image connected to the camera module 110, The feature points are extracted from the region and a triangle mesh is generated based on the feature points through the triangle mesh model 200 or a combination of the triangle mesh model 200 and the training data 400 connected to the triangle mesh model 200, The control event can be generated based on the control event. Here, the control event may be used for interactive control with the apparatus 500 connected to the processing apparatus 100 or mounting the processing apparatus 100. The apparatus 500 may include an actuator or may include a processor as an apparatus controlled according to a control event signal of the user interface control apparatus. Such a device 500 may be a simple-to-operate controlled target in response to a control event signal, or may be a computing device (interoperable device) capable of interacting with a user interface control device. The computing device includes a desktop computer, a notebook computer, and the like, and may include a mobile terminal such as Apple's iPhone or iPad, Samsung Galaxy Note, LG Electronics' G pad, and the like.

2 is a block diagram of a program block that can be employed in a user interface control apparatus according to another embodiment of the present invention.

Referring to FIG. 2, the user interface control apparatus according to the present embodiment includes input video sequences 112, a frame filter 114, a face area detector 120, A facial feature points extractor 130, a triangular mash creator 140, a triangular mash variation tracker 210, a variance estimator 220, A threshold checker 230, a nearest neighbor classifier 310, a weighted decision evaluator 320, and a user control provider 330. The threshold control unit 330 includes a threshold checker 230, a nearest neighbor classifier 310, a weighted decision evaluator 320, and a user control provider 330.

The input video sequence providing unit 112 may correspond to an input terminal of a processing unit connected to the camera module or the camera module. The input video sequence providing unit 112 may provide an input stream including a user's face image to the processing apparatus. The input terminal may have the form of a connection port, a connection terminal, and the like, and may have a resolution and a format such as an input stream data size, a data type (RGB, YCbCr, black and white), a data format (JPEG, MPEG-1 and MPEG-2) And thus can be implemented to support the necessary protocols.

The frame filter 114 may be implemented as a unit for selecting a desired one of a plurality of frames of the input stream received from the input video sequence providing unit 112 or a unit for performing a function corresponding to such a unit. The frame filter 114 basically has a function of extracting a desired frame from an input stream, and may further include a function of removing noise from an extracted frame according to an implementation.

The face region detection unit 120 may be implemented as a means for detecting a user's face region in a frame transmitted from the frame filter 114 or a component for performing a function corresponding to the means. When there are two or more detected face areas, the face area detecting unit 120 can select any one of a plurality of face areas according to a preset reference. For example, the face area detection unit 120 may select a face area having the largest size among a plurality of face areas, select a face area located at the center of the frame, or determine a face area recognized by a predetermined marker As shown in FIG.

The facial feature point extraction unit 130 extracts feature points from the facial region of the frame transmitted from the facial region detection unit 120. [ The facial feature point extracting unit 130 may use a wavelet-based feature point extracting method that processes the frame data by decomposing the two-dimensional image data to have frequency and time locality.

The triangle mesh generation unit 140 generates a triangle mesh by connecting at least some of the minutiae points of the facial region transmitted from the facial feature point extraction unit 130. The triangle mesh generation unit 140 may include a triangle mesh model. Techniques that can be employed in the triangle mesh generation unit 140 are as described above with reference to FIG.

The triangle mesh change tracking unit 210 compares the first triangle mesh of the first frame received from the triangle mesh generator 140 with the second triangle mesh of the second frame and outputs the first triangle mesh from the first triangle mesh to the second triangle mesh. Track changes. If the triangle mesh change is not tracked, the triangle mesh change tracking unit 210 may transmit a signal requesting the face area detection unit 120 to re-detect the face area.

The change estimating unit 220 estimates the shape or intensity of the change of the triangle mesh in the first frame and the second frame transmitted from the triangle mesh change tracking unit 210. The change estimating unit 220 can calculate the position of the facial feature, the direction in which the facial feature is located, the size or the amount of change, and the like, whether the change of the triangular mesh is plural or complex.

The threshold value determination unit 230 determines whether the change type or the change intensity of the triangular mesh estimated by the change estimation unit 220 exceeds a preset threshold value. In the confirmation of exceeding the threshold value, the threshold value determination unit 230 may use training data (400). The threshold value determining unit 230 may transmit a signal for re-detecting the face region to the face region detecting unit 120 if the change type or the change intensity of the triangle mesh is below the threshold value.

Upon receiving the frame from the threshold value determining unit 230, the nearest neighbors classifier 310 detects the subregion or feature having the largest triangular mesh change in the face region based on the training data 400. The nearest neighbors classifier 310 is used to generate a desired control event in response to a change in the traced triangle mesh.

The weighting decision evaluating unit 320 may assign predetermined weights according to the type or position of the feature region or the sub region of the face region classified by the closest neighbor classifying unit 310. [ The weighting decision evaluating unit 320 detects an event in the face region of the user based on the change of the triangle mesh and then assigns a weight reflecting the level of reliability previously stored according to the face feature or region generated in the event to the control event generation Can be implemented.

The user control unit 330 receives the signal from the threshold value determiner 230, the nearest neighbors classifier 310 or the weight determination evaluator 320, To the control target apparatus. The user control unit 330 may be implemented as an output terminal of the processing apparatus. In this case, the user control unit 330 may have a buffer or be implemented to support a protocol necessary for network connection with the controlled device.

3 is a flowchart of a user interface control method according to another embodiment of the present invention.

The present embodiment relates to a portion characterized in an interface method for controlling a system according to a change in a face motion using hardware and software schematically illustrated in FIGS.

Referring to FIG. 3, the user interface control method according to the present embodiment checks whether a camera module is connected to the processing apparatus (S31). The connection success of the camera module can be confirmed a predetermined number of times, and if the connection success is not confirmed (No in S32), this procedure can be terminated. When the connection success is confirmed (Yes in S32), the video stream of the camera module is transmitted to the processing device.

Next, the processing device receives the video stream from the camera module or the camera equipped with the camera module (S33). The camera module or camera may be integrally coupled to the processing device, but is not limited thereto.

Next, the processing apparatus filters video frames (S34). The filtering may include filtering the frames other than the specific frame according to a predefined preference.

Next, the processing apparatus detects a face region of the user in one filtered frame (S35). The face region may include one or more sub-regions of the face. The detected user face region may correspond to a facial sub-area for one frame.

Various methods can be used for face area detection. For example, the processing apparatus can detect the face region using region segmentation and position information. The face region detection method detects a skin color of a user based on the RGB distribution of the skin color of the RGB space and detects a second region having a color other than the skin color with a first color (e.g., white) And binarizing it into a second color (e.g., black). In this case, a median filter may be used for edge preservation, and a final face region may be detected by removing a first color region having a relatively small size in addition to a face estimation region having a predetermined size or more.

As another example of the face region detection, the processing apparatus can detect unevenness in skin tone, age, pose change (front, left, right, top, or bottom direction), illumination (JPEG, MPEG-1, and MPEG-2) conditions, depending on the environment, the distance from the camera module, the number of faces, the size of the image (resolution of the image), data type (RGB, YCbCr, And a face area detecting unit for detecting a face of the user or detecting a face area. The face region detection unit can be implemented to distinguish the face region from the background region by comparing the geometrically and / or statistically learned model with the model detected in the current image (single frame), in addition to the method using the skin color. Euclidean distance and absolute reference can be used for comparison of models. Of course, in addition to the above-described methods, the face region detecting unit may detect at least one of face region detection using an ellipse mask, face region detection using HSI color information, face region detection using a HUE and a difference image, The face region can be detected using the method of FIG.

Next, the processing apparatus determines that the detection of the face area is successful (S36). If the detection is not successful, the processing unit may return to the previous step S35 to again detect the face region of the user in one filtered frame. If the detection is not successful, the processing apparatus can return to step S33 to receive the video stream from the camera module. When the detection is not successful, a predetermined determination step S36a may be performed to return to any one of the previous steps S33 and S35. For example, the predetermined determination step may be set to a repetition number (for example, three times), the face region detection step may be returned to the face region detection step for three times, and the video stream may be received after three times.

Next, the processing apparatus extracts facial feature points from the face region (S37). The facial feature points extracted from the facial feature point extracting unit can be output in the form of coordinate points of feature points in a predetermined dimension.

Facial feature point detection is necessary for detecting changes requiring more than two frames of image processing. The facial feature point detection can be performed using or based on the edge component and the geometric position information of the face feature (eyes, nose, mouth, etc.), but it is not limited thereto and other suitable other feature point detection Technique. &Lt; / RTI >

Next, the processing device determines whether the facial feature point extraction is successful (S38). If it is not an extraction success, the processing apparatus can return to the facial feature point extraction step. Of course, if the facial feature point extraction is unsuccessful for a predetermined number of times or more, the processing apparatus may return to the step of receiving the video stream, or may output an error and terminate the process for the current frame.

Next, the processing apparatus forms a triangular mesh through the triangular mesh model (S39). In this step, the triangle mesh generation unit includes generating triangle meshes by connecting at least some of the feature points of the face region. The triangle mesh generation method may be as described above with reference to FIG. The triangular mesh model can generate triangular meshes in the face region by referring to or using training data (see 200 in FIG. 1).

Next, the processing apparatus can determine whether to perform buffering for triangular mesh tracking (S40). If it is necessary to perform buffering (Yes in S40), the processing apparatus can perform buffering for the frame (the previous frame or the first frame) in which the triangle mesh is generated (S41). In that case, the processing apparatus may include a buffering unit or a buffer. The buffering unit may store a triangle mesh (first triangle mesh) for the face region of the first frame. On the other hand, similar to the case of the first frame, the processing apparatus performs facial region extraction, facial feature point extraction, and triangular mesh (second triangular mesh) for the second frame (current frame) received after the first frame ) Generating steps.

On the other hand, if it is not necessary to perform buffering (No in S40), the processing device compares or tracks the first triangle mesh for the face area of the first frame and the second triangle mesh for the face area of the second frame You can proceed. In this case, the processing apparatus sets the parallel processing for the first frame and the second frame through the parallel processing unit, and thereby generates the triangular mesh generation for the plurality of frames in parallel through the plurality of processing processes included in the processing apparatus It can be processed as an enemy.

When the first triangular mesh of the first frame and the second triangular mesh of the second frame are prepared, the processing unit converts the second triangular mesh of the second frame into the first triangular mesh of the first frame and the second triangular mesh of the second frame, And the change from the first triangle mesh to the second triangle mesh can be tracked (S42).

Next, the processing device compares the change of the triangular mesh with the threshold value to determine whether the change of the triangular mesh exceeds the error range or within the tolerance range (S43). If the determination result is within an allowable range error, the processing device may return to the step of comparing and tracking the triangular meshes, or may terminate the present process for the first frame. If the determination result indicates that the allowable range error is exceeded, a preset control event may be generated in response to the change of the triangular mesh (S44).

On the other hand, when the result of the determination exceeds the allowable range error, the processing apparatus selects any one of the preset control events corresponding to the start position, range, shape, direction, amount of change, And a step of selecting the step. According to such a configuration, the user interface control device uses a plurality of facial features such as eyebrows, eyes, eyes, mouth, lips, face, cheek, jaw line, Various control events can be set in correspondence with the size, position, direction, symmetry, or a change of the combinations of the features.

4 is a schematic block diagram of a computing device capable of employing the user interface control apparatus of FIG.

Referring to FIG. 4, the computing device 500 employing the user interface control apparatus according to the present embodiment may receive a control event signal from the user interface control apparatus or receive an operation control signal. Such a computing device 500 may include a processor 510, a memory 520, an input / output device 530, a network interface 540 and a storage device 550. The above-described components of the computing device 500 may be interconnected by wires, conductive patterns or buses 560 for transmitting or receiving signals or data.

The processor 510 may include one or more cores and a cache memory. When the processor 510 has a multi-core structure, a multi-core may refer to integrating two or more independent cores into one package of a single integrated circuit. When the processor 510 has a single core architecture, the processor 510 may be referred to as a central processing unit. The central processing unit (CPU) may be implemented as a system on chip (SOC) in which a micro control unit (MCU) and a peripheral device (integrated circuit for external expansion device) are disposed together, but the present invention is not limited thereto. The core consists of a register that stores instructions to process, an arithmetic logical unit (ALU) that performs comparisons, judgments, and operations, an internal control unit that internally controls the CPU to interpret and execute instructions a control unit, and an internal bus.

The processor 510 described above may include, but is not limited to, one or more data processors, image processors, or CODECs. The data processor, image processor or codec may be configured separately.

In addition, the processor 510 may further include a peripheral interface and a memory interface. The peripheral device interface connects the processor 510 and the input / output device 530 and / or the processor and other peripherals, and the memory interface can couple the processor 510 and the memory 520.

The processor 510 may execute a specific software module (instruction set) stored in the memory 520 to perform various specific functions corresponding to the module.

The memory 520 may store a software module implementing the user interface control method according to the present embodiment. The software module may include a parallel processing module, a detection module, an extraction module, a generation module, a buffering module, a tracking module, a determination module, a nearest neighbor classification module, a weight determination evaluation module, and a user control provision module. And the tracking module may include a classification submodule, a selection submodule, and a comparison submodule.

The memory 520 may include high-speed random access memory and / or nonvolatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or a flash memory. The memory 520 may store software, programs, a set of instructions, or a combination thereof. In this embodiment, the memory 520 may at least temporarily store the triangular mesh model and / or training data.

The input / output device 530 transfers the input stream to the processor 510 or the user interface control device in the processor. The input / output device 530 may basically include a camera module or a camera, and may include, but is not limited to, an analog to digital converter, a latch, a sample and holder, or a combination thereof. The input / output device 530 externally outputs a signal including control event information based on the detection of the user's face event from the user interface control device or the processor 510 including the control device. The input / output device 530 may include a digital to analog converter, a buffer, an amplifier, or a combination thereof. The input / output device 530 may include various input devices such as a keyboard, a mouse, a speaker, a microphone, a touch screen, an output device, or a combination thereof.

The network interface 540 connects at least one of the processor 510, the memory 520, the input / output device 530, the network interface 540 and the storage device 550 to an external device via a network. The network interface 540 may support the protocols required by the network or the transmit and receive data formats. The network interface 540 may be implemented to enable the computing device 500 to be connected to the controlled device or to allow the user interface control device in the computing device 500 to access or interoperate with the triangular mesh model and / . In this case, the training data may be stored in a server device, a database server, a cloud server, or the like, to which the user interface control device is connected via a network.

The storage device 550 may store training data according to an implementation. The storage device 550 may be a hard disk, an optical disk, or a flash memory.

5 is a configuration diagram of a user interface control device implemented in the computing device of FIG.

5, the user interface control apparatus 100 according to the present embodiment includes a detection unit 20, an extraction unit 30, a generation unit 40, a tracking unit 60, and a determination unit 70 do. In addition, the user interface control apparatus 100 may further include a buffering unit 50 and / or a parallel processing unit 10 according to an implementation.

The parallel processing unit 10 may determine or set whether to process the first frame and the second frame including the face image of the user simultaneously in the stream data input to the UI controller 100. If the parallel processing environment is set to be active in the parallel processing unit 10, the user interface control apparatus 100 performs the following processes for detecting the face area and extracting the feature points, for the first frame and the second frame It can be performed independently or in parallel.

The detection unit 20 includes means or components for detecting the face region in one frame provided from the camera module or the input video sequence providing unit and filtered by the frame filter. The detection unit 20 may have a detection module (first module) type stored in a memory and may be operated by a processor to operate as a face region detection unit.

The extraction unit 30 includes means for extracting the feature points from the face region detected by the detection unit 20 and a constituent unit. The extracting unit 30 may have a form of an extraction module (second module) stored in a memory, and may be executed by a processor to operate as a facial feature point extracting unit.

The generating unit 40 includes means and a configuration unit for generating a triangular mesh based on the facial feature points extracted by the extracting unit 30. [ The generation unit 40 may have a form of a generation module (third module) stored in a memory, and may be executed by a processor to operate as a triangular mesh generation unit.

The buffering unit 50 includes means and a configuration unit for storing the processing result of the first processed frame when the generating unit 40 sequentially processes the first frame and the second frame. The buffering unit 50 may have the form of a buffering module (fourth module) stored in a memory, and may be executed by a processor to operate as a buffer.

The tracking unit 60 compares the first triangle mesh of the first frame generated by the generating unit 40 and the second triangle mesh of the second frame to track the change from the first triangle mesh to the second triangle mesh Means or component. The tracking unit 60 may have the form of a tracking module (fifth module) stored in a memory and may be operated by a processor to operate as a triangular mesh change tracking unit.

The determination unit 70 includes means for determining whether a control event for user interface control is generated based on the change of the triangle mesh tracked by the tracking unit 60 or a configuration unit. The determination unit 70 may include a determination module (sixth module) stored in a memory, and may be performed by a processor to operate as a threshold determination unit.

Although not shown in FIG. 5, the user interface control apparatus 100 according to the present embodiment may further include a classifying unit, an evaluating unit, and a providing unit.

The classification unit may include a means or a component for classifying the largest change or the change in the ROI within the triangular mesh change determined by the determination unit 70 using the nearest neighbor classification technique or the like. The classifier may have the form of a classification module (seventh module) stored in a memory and may be performed by a processor to act as a nearest neighbor classifier.

The evaluating unit may include means for determining a predetermined weight based on a position or a feature in the face region with respect to the largest triangular mesh change classified by the classifying unit or the triangular mesh change in the ROI. The evaluation unit may have the form of an evaluation module (eighth module) stored in the memory, and may be performed by the processor to operate as a weight determination evaluation unit.

The providing unit may include means for generating a preset control event corresponding to the triangular mesh change obtained by at least one of the judging unit 70, the classifying unit and the evaluating unit, and a configuration unit. The providing unit may have a form of a providing module (ninth module) stored in a memory, and may be performed by a processor and operated as a user control unit.

The first to ninth modules may be software modules stored in a memory and may be implemented by a processor connected to a memory to implement a user interface control method according to the present embodiment.

6 is a configuration diagram of a tracking unit of the user interface control apparatus of FIG.

Referring to FIG. 6, in the user interface control apparatus according to the present embodiment, the tracking unit 60 may include a classifying unit 62, a selecting unit 64, and a comparing unit 66.

The classifying unit 62 may include a first frame including the first triangle mesh generated by the generating unit or the triangle mesh generating unit and a second frame including the second triangle mesh, Sub-regions of the < / RTI > The classifying unit 62 can output information (first information) about a plurality of sub-areas. The classifying unit 62 may have a type of a classification module (first sub-module) stored in a memory, and may be performed by a processor to operate as a part of the triangle mesh change tracking unit.

The selecting unit 64 can select the sub region of interest having the largest change among the sub regions divided by the dividing unit 62. [ The selection unit 64 may output information on the selected sub-region of interest. The selection unit 64 may be a selection module (second submodule) stored in the memory, and may be performed by the processor to operate as a part of the function of the triangular mesh change tracking unit.

The comparing unit 66 compares the triangle meshes between the first frame and the second frame in the selected subarea selected by the selecting unit 64 to track changes in the triangle mesh. The comparator 66 may output information on the shape, distribution, volume (amount of change, size, intensity, etc.) of the triangular mesh, or a combination thereof. The comparator 66 may be a comparison module (third sub-module) stored in the memory, and may be executed by the processor to operate as a part of the function of the triangular mesh change tracking unit.

The division unit 62, the selection unit 64 and the comparison unit 66 described above are stored in the memory in the form of a submodule of the module corresponding to the tracking unit 60 and are executed by the processor to operate as a triangular- can do.

According to the above-described embodiment, a plurality of feature points are extracted in a sub-area of an input face image, and a variation event of a triangle mesh connecting feature points is traced to generate a control event according to a preset triangle mesh model. This means that the target of the action recognizes the user's intention or command through facial expression and tracking, rather than the user's hand motion or gesture, and outputs a control event, control command, or control signal accordingly, A notebook computer, an actuator mounting device, etc.) can be controlled. In addition, this has the advantage that the degree of freedom of the user can be greatly increased, and the utilization for the physical weak can be greatly improved.

Meanwhile, in the present embodiment, the components of the user interface control device may be, but not limited to, a functional block or a module mounted on a user's mobile terminal or a computer device. The above-described components may be stored in a computer-readable medium (recording medium) in the form of software for implementing a series of functions that they perform, or may be transmitted to a remote location in the form of a carrier to be implemented to operate in various computer devices. Herein, the computer readable medium may be disposed in a plurality of computer devices or cloud systems connected via a network, and at least one of the plurality of computer devices or the cloud system may be connected to a memory system for performing the user interface control method of the present embodiment You can store programs and source code.

That is, the computer-readable medium may be embodied in the form of a program command, a data file, a data structure, or the like, alone or in combination. Programs recorded on a computer-readable medium may include those specifically designed and constructed for the present invention or those known and available to those skilled in the computer software arts.

The computer-readable medium may also include a hardware device specifically configured to store and execute program instructions, such as a ROM, a RAM, a flash memory, and the like. Program instructions may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. The hardware device may be configured to operate with at least one software module to perform the user interface control method of the present embodiment, and vice versa.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

Claims

Detecting a face region in each of a first frame to be input and a second frame to be inputted next to the first frame;
Extracting feature points from the face region;
Generating a triangle mesh based on the feature points;
Comparing the first frame and the second frame to track changes in the triangle mesh;
Determining whether the change of the triangle mesh is above a threshold value; And
Generating a control event in accordance with a face motion corresponding to a change in the triangle mesh above the threshold.

The method according to claim 1,
Wherein the generating the triangle mesh comprises:
A user interface control method for setting one of the minutiae as a main axis and dividing the face area into a plurality of areas based on the main axis and generating the triangle mesh by using two- .

The method according to claim 1,
Wherein the generating the triangle mesh comprises:
And using the Delaunay triangulation to connect the feature points to the triangles so that the interior angles of the triangles are minimized when dividing the plane or space.

The method of claim 3,
Wherein the generating the triangle mesh comprises:
Wherein the feature points are generated according to a tree structure connected so that the feature points can reach each other and the sum of the total distances is minimized.

The method according to claim 1,
Prior to tracking the change in the triangle mesh,
Further comprising buffering the first frame including the triangle mesh.

The method according to claim 1,
Wherein the first frame and the second frame are processed in different processes of the image processing apparatus.

The method according to claim 1,
Wherein tracking the change in the triangle mesh comprises:
Wherein the face region is divided into a plurality of predetermined sub regions, a region of interest having the largest change among the sub regions is selected, and a change of the triangle mesh in the region of interest is compared.

The method of claim 7,
Wherein the region of interest comprises an eyebrow, an eye, a pupil, a mouth, a lip, a cheek, a jaw line, a face orientation, or a combination thereof.

The method according to claim 1,
Wherein the step of determining whether the change of the triangle mesh is equal to or greater than a threshold value,
And comparing the variation of the triangle mesh with the training data.

The method of claim 9,
The training data may be stored in a storage unit of a control target device receiving a signal according to the control event or in a storage unit of an interoperable device or stored in a server device or a cloud server connected to the control target device or an interoperable device through a network, Interface control method.

The method of claim 10,
Wherein the control target device or the interoperable device comprises a mobile terminal and the first frame and the second frame are obtained from a camera module of the mobile terminal.

A detector for detecting a face region in each of a first frame to be input and a second frame to be inputted next to the first frame;
An extracting unit for extracting feature points from the face region;
A generator for generating a triangle mesh based on the feature points;
A tracking unit for tracking a change of the triangular mesh by comparing the first frame and the second frame;
A determining unit determining whether the change of the triangle mesh is equal to or greater than a threshold value; And
And generating a control event in accordance with a face motion corresponding to the change of the triangle mesh above the threshold value.

The method of claim 12,
Wherein the generation unit comprises:
Wherein one of the minutiae is set as a main axis, the face region is divided into a plurality of regions based on the principal axis, the triangle mesh is generated using two-dimensional Dilnyi triangulation for each divided region,
A Delaunay triangulation is used to connect the minutiae to the triangles to divide the plane or the space so that the internal angles of the triangles are minimized,
Wherein the triangle meshes are generated according to a concatenated tree structure such that the feature points can reach each other and the sum of the total distances is minimized.

The method of claim 12,
Further comprising a buffering unit for buffering the first frame including the triangle mesh and disposed between the generating unit and the tracking unit.

The method of claim 12,
Further comprising a parallel processing unit for supporting independently performing a first process of processing the first frame and a second process of processing the second frame.

The method of claim 12,
The tracking unit includes:
A dividing unit dividing the face area into a plurality of preset sub-areas;
A selector for selecting a region of interest having the largest change among the sub regions; And
And a comparator for tracking a change of the triangle mesh in the area of interest.

18. The method of claim 16,
Wherein the region of interest comprises an eyebrow, an eye, a pupil, a mouth, a lip, a chin, a face orientation, or a combination thereof.

The method of claim 12,
Wherein the determination unit compares the change of the triangle mesh with pre-stored training data.

The method of claim 9,
The proximity classifier may further include a closest neighbor classifier between the determination unit and the user control unit, wherein the closest neighbor classifier classifies the user Interface control device.

The method of claim 19,
And a weight determination evaluation unit between the nearest neighbor classification unit and the user control unit, wherein the weight determination evaluation unit assigns a weight according to the type or position of the feature or the sub-area.