KR101835531B1 - A display device for providing augmented reality image based on face recognition and method for controlling the display device thereof - Google Patents

Abstract

A display device for providing an augmented reality image based on face recognition and a control method thereof are provided. The display device includes an input part for receiving a user command for selecting a first AR theme including at least one AR element, a camera that photographs an image including the face of a user, a display part for displaying the augmented reality image to which the AR theme is applied, and a control part for recognizing the face of the user in the photographed image, synthesizing the first AR element with the recognized face to generate an augmented reality image, and controlling the display part to display the generated augmented reality image.

Description

TECHNICAL FIELD [0001] The present invention relates to a display device for providing augmented reality image based on face recognition, and a control method thereof. [0002]

The present invention relates to a display device for providing augmented reality image based on face recognition and a control method thereof, and more particularly to a display device for recognizing a face of a user in a photographed image, transforming the recognized face, To a display device for providing augmented reality image based on face recognition for providing an image and a control method thereof.

Recently, as the technology for the augmented reality has developed, opportunities for accessing the augmented reality service through applications have increased. For example, when an image is taken by a user, augmented reality service that applies an image such as an animal to a user's face is provided.

Most of the users who can easily access these augmented reality services are those who use smartphones and tablet PCs. Although young / infants can be interested in augmented reality services, there is little augmented reality service targeting infants and young children, although they can form new markets.

Accordingly, there is a need for a face recognition-based augmented reality service that applies a theme that is likely to be of interest to infants and young children through a display device that can be installed in a place such as a kids' café .

An object of the present invention is to provide a face recognition-based augmented reality image providing augmented reality image applying an AR theme, A display device for providing an image, and a control method thereof.

According to an aspect of the present invention, there is provided a display device for providing a face recognition-based augmented reality image, the AR device comprising: an input unit for receiving a user command for selecting a first AR theme including at least one AR element; ; A camera that captures an image including a user's face; A display unit configured to display an AR image to which the AR theme is applied; And a control unit for recognizing a face of the user in the photographed image to generate an augmented reality image by synthesizing a first AR element with the recognized face, and controlling the display unit to display the generated augmented reality image .

The control unit may include a face recognition module that detects a plurality of feature points from the recognized face and determines at least one of a facial expression or a facial motion by analyzing a correlation between the detected plurality of feature points; And an AR theme synthesis module for modifying the first AR element to correspond to the detected minutiae points and compositing the modified first AR element to the recognized user face.

In addition, the AR theme synthesis module may modify the user face in which the first AR element is synthesized to correspond to the first AR element.

The AR theme synthesis module modifies a second AR element corresponding to the determined facial motion when the facial recognition module judges a facial motion, and transmits the modified second AR element to the recognized user face Can be synthesized.

The image processing apparatus may further include an output unit configured to output an image including at least one frame of the augmented reality image, wherein the control unit controls the output unit to print a frame corresponding to the print event when a print event occurs , And the print event is set to print a captured print in a predetermined time unit when a user command for printing an image is input.

A storage unit for storing the generated augmented reality image; And a communication unit for transmitting the stored augmented reality image to an external device.

According to another aspect of the present invention, there is provided a method of controlling a display device for providing a face recognition-based augmented reality image, the method comprising: receiving a first AR theme including at least one AR element, Receiving an instruction; A step of photographing an image including a user's face; Generating augmented reality image by synthesizing a first AR element with the recognized face by recognizing a user's face in the captured image; And displaying the generated augmented reality image; .

As described above, according to the various embodiments of the present invention, the user can receive the AR-themed augmented reality image on the user's face only by inputting the AR theme.

1 is a block diagram schematically illustrating a configuration of a display device for providing augmented reality image based on face recognition according to an embodiment of the present invention;
FIG. 2 is a block diagram illustrating a detailed configuration of a control unit according to an embodiment of the present invention;
3 is a diagram for explaining an embodiment for detecting facial feature points according to an embodiment of the present invention,
4 is a flowchart illustrating a control method of a display device for providing a face recognition-based augmented reality image according to an embodiment of the present invention.

The terms used in the embodiments of the present disclosure will be briefly described, and the embodiments will be described in detail.

The terms used in the embodiments of the present disclosure have selected the currently widely used generic terms possible in light of the functions in this disclosure, but these may vary depending on the intentions or precedents of those skilled in the art, the emergence of new technologies, and the like . Also, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, the meaning thereof will be described in detail in the description of the corresponding embodiments. Therefore, the terms used in the embodiments should be defined based on the meaning of the term, not on the name of a simple term, and on the contents of the embodiments throughout.

In the embodiments of the present disclosure, terms including ordinal numbers such as first, second, and so on can be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Moreover, in the embodiments of the present disclosure, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise.

Also, in the embodiments of the present disclosure, terms such as "comprise" or "have ", etc. are intended to specify that there is a feature, number, step, operation, component, Steps, operations, elements, components, or combinations of elements, numbers, steps, operations, components, parts, or combinations thereof.

Further, in the embodiments of the present disclosure, "module" or "module" performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. Also, a plurality of "modules" or a plurality of "parts" may be embodied as at least one processor integrated into at least one module, except "modules" or "modules "

Also, in the embodiments of the present disclosure, when a portion is referred to as being "connected" with another portion, it is not only the case where it is "directly connected" And the like.

Further, in the embodiments of the present disclosure, the user input may include at least one of a touch input, a bending input, a voice input, a button input, and a multimodal input, but is not limited thereto.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, the present disclosure will be described with reference to the drawings. 1 is a diagram illustrating a configuration of a display device (hereinafter, a display device) for providing augmented reality image based on face recognition, according to an embodiment of the present invention. 1, a display device 100 includes a camera 110, an input unit 120, a display device 130, an output unit 140, and a control unit 150. [ Meanwhile, the display device 100 according to an embodiment of the present invention may be implemented by various electronic devices such as a smart TV, a kiosk, a desktop PC, a notebook PC, and the like.

The camera 110 is configured to shoot an image. In particular, the camera 110 may capture an image including the user's face.

The input unit 120 receives a user command for controlling the display device 100. In particular, the input unit 120 may receive a user command for selecting an AR theme. The input unit 120 may receive a user command for printing an image. However, the present invention is not limited to this, and user commands can be received through various methods such as recognition of an operation using the camera 110, user recognition, and the like.

The display unit 130 displays at least one of a video frame processed by a video processor (not shown) and various screens generated by a graphics processor (not shown), the video data received from an image receiver (not shown). In particular, the display unit 130 may display an augmented reality image to which an AR theme is applied according to the control of the control unit. Also, the display unit 130 may display an AR theme selection UI including a plurality of AR theme icons in order to select an AR theme.

The output unit 140 is configured to output an image. In particular, the output unit 140 may output an image in which some of the images to which the AR theme is applied are rearranged according to the control of the control unit.

The control unit 150 controls the overall operation of the display device 100 using various programs stored in a storage unit (not shown). The controller 150 may include a RAM, a ROM, a graphics processor, a main CPU, first to n-th interfaces, a bus, and the like. The RAM, the ROM, the graphics processor, the main CPU, Lt; / RTI >

In particular, when a user command for selecting an AR theme is input through the input unit 120, the control unit 150 recognizes the user's face from the image captured through the camera 110. [ Then, the controller 150 generates the augmented reality image by synthesizing the first AR element included in the AR theme in the face of the user recognized according to the inputted user command. The control unit 150 controls the display unit 130 to display the generated augmented reality image. At this time, the AR theme may be various themes such as a dinosaur park theme, a birthday party theme, an underwater exploration theme, a princess costume room theme, and a plurality of AR elements corresponding to an AR theme may be stored in a storage unit . For example, underwater exploration themes can be composed of AR elements in various shapes such as an octopus shape, a scuba diver shape, and the like.

When a user command to print an image is input through the input unit 120, the controller 150 may control the output unit 140 to print a frame corresponding to the input user command. The controller 150 may control the output unit 140 to rearrange the frame according to the input post-order command and to print the post-frame if the post-command is input. At this time, the post-processing may be various functions such as filter application, sticker layering, and graffiti.

Also, the control unit 150 may control the output unit 140 to print the advertisement received through the communication unit (not shown) or the advertisement stored in the storage unit (not shown) together with the frame that has been finalized. The control unit 150 may control the output unit 140 to print an advertisement corresponding to any one of the age group of the user determined based on the recognized face of the user or the selected AR theme together with the frame of interest. For example, if the user command is a dinosaur park theme, the control unit 150 may control the output unit 140 to print the dinosaur park toy advertisement, which is an advertisement corresponding to the dinosaur park theme, together with the frame of interest.

Also, the controller 150 may store the generated augmented reality image in a storage unit (not shown). The control unit 150 may transmit the stored augmented reality image to an external device.

Hereinafter, a method of transforming a user's face and applying an AR theme according to a user command will be described in detail with reference to FIG. 2 and FIG. 3. FIG. As shown in FIG. 2, the controller 150 includes a face recognition module 151 and an AR theme synthesis module 153.

The face recognition module 151 recognizes the face of the user and can detect a plurality of feature points from the recognized face. In addition, the face recognition module 151 can determine at least one of a facial expression or a facial motion by analyzing the correlation of the detected plurality of feature points.

Specifically, the face recognition module 151 can recognize the face of the user by distinguishing the face and the background of the user based on information such as brightness, motion, and color eye position estimation of the photographed image. For example, the face recognition module 151 may perform a separate learning for the data that designate each face region (generally, a rectangular region) for a person included in a plurality of images having different brightness, face shape, age, The user's face can be recognized through the result. As the number of image data for learning increases, the accuracy of the learning result increases, and as a result, the user's face can be detected more accurately in the captured image. 3, the face recognition module 151 recognizes the face shape 310, the eyebrow 320, the eyes 330, the nose 340, and the face 340 included in the face region, Data representing the minutiae points for the mouth 350 can be generated. These feature points can be detected by using separate learning results for data showing the face shape, eyebrow, eye, nose, and mouth positions of a plurality of faces, and as the data for learning increases, It is possible to increase the accuracy of detecting feature points in the face.

Then, the face recognition module 151 can determine at least one of facial expression or facial motion by analyzing the correlation between the detected plurality of feature points. For example, the face recognition module 151 analyzes the positional correlation between the minutiae allocated to the eyes, determines whether the eyes are closed, analyzes the positional relationship between the minutiae allocated to the mouth, and opens the mouth And determine at least one of a facial expression or a facial motion based on the shape of the eye, eyebrow, mouth, and the like.

The AR theme synthesis module 153 may transform the first AR element based on the detected location of the minutiae points and synthesize the modified first AR element with the recognized user face.

Then, the AR theme synthesis module 153 modifies the second AR element so as to correspond to the expression or motion of the AR theme, and synthesizes the second AR element to the user's face.

For example, if the AR theme selected by the user is the underwater exploration theme, the AR theme synthesis module 153 transforms the first AR element, the scuba diving hood, based on the location of the feature points detected by the face recognition module 151 . If the face recognizing module 151 determines that there is a movement of the user to sniff the mouth, the AR theme synthesizing module 153 may synthesize the octopus shape, which is the second AR element corresponding to the mouth motion, .

In another embodiment, when the AR theme selected by the user is a princess dressing room theme, the AR theme synthesis module 153 generates a snowy prince face, which is a first AR element, based on the location of the feature points detected by the face recognition module 151, It can be synthesized on the face. If the face recognition module 151 determines that there is a blinking motion of the user, the AR theme synthesis module 153 may synthesize the Cinderella face user face, which is the second AR element.

When a predetermined period elapses or when a user command for changing the AR element is input, the AR theme synthesizing module 153 may modify the second AR element and synthesize it on the user's face. For example, if the AR theme selected by the user is the underwater exploration theme, the AR theme synthesis module 153 transforms the first AR element, the scuba diving hood, based on the location of the feature points detected by the face recognition module 151 . Then, when a predetermined time has elapsed, the AR theme combination module 153 can synthesize the octopus shape, which is the second AR element, on the face of the user. Then, when the second AR element is synthesized and the predetermined time has elapsed, the AR theme synthesizing module 153 can synthesize the third AR element on the face of the user by modifying the fish shape.

Hereinafter, a control method of the display apparatus 100 according to an embodiment of the present invention will be described with reference to FIG.

First, the display device 100 receives a user command for selecting an AR theme (S410). For example, the display device 100 selects one of the AR theme among various AR themes such as a dinosaur park theme, a birthday party theme, an underwater exploration theme, a princess costume room theme through various inputs such as a touch input and a button input User commands can be input.

Then, the display device 100 captures an image including the user's face (S420). On the other hand, the display device 100 may photograph not only an image including a single user's face but also an image including a plurality of users' faces.

Then, the display apparatus 100 recognizes the user's face on the photographed image, and synthesizes the first AR element with the recognized face to generate an augmented reality image (S430). Specifically, the display apparatus 100 can recognize a user's face in the photographed image and detect a plurality of feature points in the recognized user's face. Then, the display apparatus 100 can generate an augmented reality image by modifying the AR element based on the detected positions of the plurality of minutiae points and synthesizing the AR element with the user's face.

Then, the display device 100 displays the generated augmented reality image (S440).

Meanwhile, the above-described method can be implemented in a general-purpose digital computer that can be created as a program that can be executed by a computer and operates the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM,

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, disclosure methods should be considered from an illustrative point of view, not from a restrictive point of view. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: Display device providing augmented reality image based on face recognition
110: camera
120: Input unit
130:
140:
150:

Claims (7)

A display device for providing a face recognition-based augmented reality image,
An input unit for receiving a user command for selecting an AR theme including at least one AR element;
A camera that captures an image including a user's face;
A display unit configured to display an AR image to which the AR theme is applied; And
And a control unit for controlling the display unit to display the generated augmented reality image by combining the first AR element with the recognized face by recognizing the user's face in the captured image,
Wherein,
The method comprising the steps of: detecting a plurality of feature points from the recognized face of the user to scale learning data; deriving feature points from a user's face photographed based on the accumulated learning data to increase the accuracy of deriving feature points; A facial recognition module for determining at least one of a facial expression and a facial motion by analyzing a correlation between the face and the facial motion; And
Modifying the first AR element to correspond to the position of the detected minutiae points, synthesizing the recognized face of the user with the modified first AR element, When an AR element replacement event is generated, in which a user command for inputting a user command for recognizing motion, a predetermined period of time elapsing, or an AR element replacement at the input section is generated, a second AR element And an AR theme synthesis module for transforming and compositing the modified second AR element to the recognized face of the user
Wherein the augmented reality image is a face image. delete The method according to claim 1,
The AR theme synthesis module includes:
Wherein the first AR element transforms the combined user face to correspond to the first AR element. delete The method according to claim 1,
And an output unit for printing an image including at least one of the augmented reality images,
Wherein,
Controls the output unit to print a frame corresponding to the print event when a print event occurs,
Wherein the print event is any one of a case where a user command for printing an image is input and a case where a print captured on a predetermined time unit basis is printed. Device. The method according to claim 1,
A storage unit for storing the generated augmented reality image; And
And a communication unit for transmitting the stored augmented reality image to an external device. A method of controlling a display device for providing augmented reality image based on face recognition,
(a) receiving a user command for selecting an AR theme including at least one AR element;
(b) capturing an image including a user's face;
(c) detecting a feature point in the photographed face of the user and synthesizing the recognized face into the AR theme to generate a first AR element;
(d) displaying the augmented reality image generated by synthesizing the AR theme on the face of the user; , ≪ / RTI &
In the step (c)
The method comprising the steps of: generating learning data by detecting a plurality of minutiae from the photographed face of a user, performing learning by scaling the minutia data to increase the accuracy of extracting minutiae from the photographed face, And determining at least one of a facial expression and a facial motion by analyzing the correlation of the facial motion,
After the step (d)
If a replacement event is generated in which a preset facial motion or a predetermined facial movement is judged in the user's face recognized in the first AR element, a preset period elapses, or an instruction of the user is inputted to replace the AR, Modifying the second AR element and compositing and displaying the modified second AR element on the recognized user face
And a face recognition-based augmented reality image, the method comprising:

Images