WO2016167504A1

WO2016167504A1 - Smart mirror device and user-customized image providing method

Info

Publication number: WO2016167504A1
Application number: PCT/KR2016/003481
Authority: WO
Inventors: 김성한
Original assignee: (주)사름
Priority date: 2015-04-16
Filing date: 2016-04-05
Publication date: 2016-10-20
Also published as: KR20160123644A

Abstract

A smart mirror device is disclosed. The smart mirror device comprises: a mirror disposed in the front side thereof; and a display module for transmitting a reproduction image through at least a partial area of the mirror so as to display the same to the outside, thereby providing a mirror function and an image display function. In addition, the smart mirror device further comprises: a camera which is disposed in the front side thereof so as to photograph a user; a storage unit for storing a plurality of image contents; and a control unit for reading, from the storage unit, an image content corresponding to a user's age and gender which are acquired by analyzing an image photographed by the camera and reproducing the image content.

Description

Smart mirror device and how to provide customized video

The present invention relates to a smart mirror.

Smart mirrors are well known. A half mirror with translucent function and reflection function is placed in front and a display panel behind it. When an image is output from the display panel, the image is transmitted through the half mirror and the image is displayed externally. If not, mirror function is provided.

Meanwhile, Korean Laid-Open Patent Publication No. 10-2013-0003384 discloses an image and mirror fusion type digital multimedia data display system.

Provided is a mirror function and an image display function, when providing an image display function is disclosed a smart display device that can provide a customized image content according to the user.

According to an aspect of the present invention, a smart mirror device that provides a mirror function and an image display function includes a mirror disposed on the front surface and a display module that transmits a reproduced image to at least a portion of the mirror to be displayed to the outside. The apparatus may further include a camera photographing, a storage unit storing a plurality of image contents, and a controller configured to read and reproduce image contents corresponding to the age and gender of the user obtained by analyzing the photographed images of the camera, from the storage unit.

According to an aspect, the smart mirror device further includes a proximity sensor that detects a user's approach, and the controller may control to start capturing the camera when the user's approach is detected.

According to an aspect, the controller transmits user's age and gender information to the management server, receives command data to play a specific image from the management server, and reads and reproduces the image content corresponding to the received command data from the storage unit. Can be.

According to an aspect, the controller may include a frame obtaining unit obtaining a frame from a photographed image signal of a camera, a face detecting unit detecting a face region in the obtained frame, and a feature point pattern of the detected face region, and a reference face image pattern by gender and age. The comparison may include an age / gender estimator for estimating the age and gender of the user.

According to an aspect, the face detector may extract a representative frame from the obtained frames, and detect a face image from the extracted representative frame.

According to an aspect, the mirror may include a first mirror area that provides only a mirror function, and a second mirror area that provides a mirror function or an image display function.

On the other hand, according to one aspect of the present invention provides a user-customized image of the smart mirror device that provides a mirror function and the image display function, including a mirror disposed on the front surface, and a display module for transmitting the playback image to at least a portion of the mirror to the outside The method includes: obtaining an image frame photographing a user, extracting a representative frame from the image frames, detecting a face region from the extracted representative frame, a feature point pattern of the detected face region, and a reference face by gender and age Estimating the age and gender of the user by comparing the image patterns, and reproducing the image content corresponding to the estimated age / gender.

According to the present invention, by analyzing the user's face photographed in the camera to provide a customized advertisement according to age / gender, it is possible to create an effect to maximize the advertising effect. Furthermore, it is possible to provide a service that saves time and costs by efficiently preparing advertising plans by database of the floating population.

1 is a block diagram of a smart mirror system according to an embodiment.

2 is a diagram schematically illustrating a front surface of a smart mirror device according to an embodiment.

3 is a block diagram of a smart mirror device according to an embodiment.

4 is a table of image content information management corresponding to age / gender according to an embodiment.

5 is a block diagram of a controller according to an exemplary embodiment.

6 is a flowchart of a method of providing a user-customized image, according to an exemplary embodiment.

The foregoing and further aspects of the present invention will become more apparent through the preferred embodiments described with reference to the accompanying drawings. Hereinafter, the present invention will be described in detail to enable those skilled in the art to easily understand and reproduce the present invention.

1 is a block diagram of a smart mirror system according to an embodiment. As shown in FIG. 1, the smart mirror system includes a smart mirror device (hereinafter referred to as a “smart mirror”) 100 and a management server 200. The smart mirror 100 and the management server 200 are connected through a communication network, the smart mirror 100 may be a plurality. The smart mirror 100 may be installed in a work space of a member, for example, a store or a beauty salon, which is a work space of a self-employed person engaged in sales or service businesses. The smart mirror 100 provides a mirror function and an image display function, and may provide a mirror function and an image display function selectively or simultaneously. Furthermore, when providing an image display function, the smart mirror 100 may recognize a user's face, estimate age and gender therefrom, and display customized image content. Here, the image content may be advertisement content.

The management server 200 may identify the smart mirrors 100 by an equipment identifier uniquely assigned to the plurality of smart mirrors 100. In addition, the management server 200 holds image contents corresponding to age / gender. In this case, video contents corresponding to age / gender may be different for each device and video contents may be uploaded to the management server 200 by a store manager. The smart mirror 100 may download image content to be reproduced in real time from the management server 200, but may also download and hold image contents corresponding to age / gender in advance. In this case, the download may be performed periodically, or may be made at the request of the user.

The management server 200 receives an age / gender estimation result from the smart mirror 100, identifies the image content suitable for the result, and sends a command to the smart mirror 100 to play the identified image content. Can be. In addition, the smart mirror 100 may detect the floating population and report the detection result to the management server 200. In addition, the management server 200 may construct an advertising plan / scenario by making a database of the floating population detection results.

2 is a diagram schematically illustrating a front surface of a smart mirror device according to an embodiment. The smart mirror 100 includes a mirror 140 disposed at the front of the main body 110, a camera 160 disposed at the upper center of the mirror 140, and a proximity sensor 170 disposed at the lower center of the mirror 140. It may include. In addition, although not shown, a touch panel for a user interface may be provided. The touch panel may be configured to correspond to the mirror 140 so that the user may touch the mirror 140 to make a desired input.

3 is a block diagram of a smart mirror device according to an embodiment. As shown in FIG. 3, the smart mirror 100 may include a communication unit 120, a storage unit 130, a mirror 140, a display module 150, a camera 160, and a controller 180. It may further include a proximity sensor 170. The communication unit 120 may be a communication module for communicating with the management server 200 and may be a communication module using a communication network such as WiFi, 3G, LTE, or LAN. The storage unit 130 may be a database that stores operation programs and control data for the overall operation of the smart mirror 100, application programs and data and image contents for performing a process for providing a customized image. The storage unit 130 may include a system memory for storing an operation program and control data for operating the device, and an external memory such as a Micor SD Card for storing image contents.

The mirror 140 may be disposed at the front surface to provide a mirror function that reflects light from the outside to illuminate an object. In addition, the mirror 140 may provide an image display function for transmitting the image signal from the display module 150 to be displayed to the outside. In one embodiment, the mirror 140 may be a half mirror. When the light source of the display module 150 is turned off, the half mirror 140 has a mirror-like reflection effect. When the light source is turned on, the half mirror 140 transmits an image of the display module 150 to be displayed externally. In another embodiment, the mirror 140 is coated on a transparent glass substrate layer capable of transmitting light and a rear surface of the transmissive glass substrate to transmit the image light from the display module 150 to the outside to display the image light output. When suspended, it may be composed of a titanium deposition coating layer that acts as a mirror to project the object.

The display module 150 is a display module for a digital information display (DID). The display module 150 may include a backlight unit for irradiating light with a display panel in response to the mirror 140 and a driver for driving the display unit. Here, the display panel may be an LCD panel. The display module 150 may be the same size as the mirror 140, but may have a smaller size than the mirror 140 so that light is transmitted through only a portion of the mirror 140. In the latter case, the mirror 140 may be divided into a first mirror area providing only a mirror function and a second mirror area providing a mirror function or an image display function. In this case, the mirror 140 may be two mirrors. That is, the first mirror may be a first mirror area, and the second mirror may be a second mirror area. The first mirror area always provides a mirror function, and the second mirror area may provide a mirror function or a display function depending on whether the display module 150 emits light.

The camera 160 is disposed at the front and photographs the user. In one embodiment, the camera 160 may be disposed at the upper center of the mirror 140. In addition, the proximity sensor 170 may be further included in the smart mirror 100 to detect proximity of the user. The proximity sensor 170 may detect whether the user is close to the mirror 140, and if detected, the camera 160 may start capturing by the controller 180. The controller 180 may be configured with one or more processors that control the overall operation of the smart mirror 100. The controller 180 may download image contents from the management server 200 in advance or periodically or at the time of a user's command. The controller 180 estimates the age and gender of the user based on the captured image of the camera 160. The image content corresponding to the estimated age and gender is read from the storage 130 and processed. To this end, the image contents stored in the storage 130 may be managed according to age / gender. The reproduced image is displayed on the display module 150 to be transmitted to the outside through the mirror 140.

According to an aspect, the controller 180 may receive an instruction from the management server 200 instead of directly inquiring about the image 130 corresponding to the estimated age and gender and checking the storage 130. To this end, the controller 180 transmits the estimated age / gender information to the management server 200. The management server 200 receives age / gender information and checks command data related to image content corresponding to the received age / gender information. 4 illustrates a table of command data corresponding to image content according to age / gender managed by the management server 200. The management server 200 may check the command data corresponding to the age / gender received from the smart mirror 100 by querying the table as illustrated in FIG. 4. The management server 200 transmits the confirmed command data to the smart mirror 100. Accordingly, the smart mirror 100 receives the command data, and reads the image content corresponding to the received command data from the storage unit 130 to reproduce the image data. Meanwhile, although only one table is illustrated in FIG. 2, such a table may be generated and managed separately for each smart mirror 100. That is, the image content to be displayed according to age / gender for each of the smart mirrors 100 may be different. The management server 200 may distinguish a table to be queried through the identifier of the smart mirror 100 received along with age / gender information.

5 is a block diagram of a controller according to an exemplary embodiment. As illustrated in FIG. 5, the controller 180 may include a frame acquirer 181, a face detector 182, an age / gender estimator 183, a playback image selector 184, and an image playback unit 185. It may include. The frame acquirer 181 is a frame grabber that acquires a frame by digitizing an analog image signal input from the camera 160 into bits defined per sample. The face detector 182 detects a face by analyzing an image frame acquired by the frame acquirer 181. According to an embodiment, the face detector 182 may extract one frame from among the image frames acquired by the frame acquirer 181 as a representative frame, and detect the face by analyzing the extracted representative frame. Here, the representative frame may be any one of frames after the first frame, for example, the sixth frame may be the representative frame.

As a face detection algorithm, a knowledge based method, a feature invariant method, a template matching method, a training based method, and the like are known. Knowledge-based methods use rules for the location and distance of the eyes, nose, and mouth that make up the face. That is, in general, the eyes are determined symmetrically at the top of the face, and the areas satisfying the same are determined as faces based on the rule that the eyes exist vertically and horizontally at regular intervals below the nose and mouth. The feature invariant method extracts features that do not change even with the rotation and illumination of the face area and detects the face area using them. That is, feature values for eyebrows, eyes, nose, and mouth are extracted in advance, and the faces are detected by comparing them in the input image.

The template mapping method detects a face region by pre-configuring a face composition pattern as a template and comparing a correlation coefficient obtained by matching it with an input image to a specific threshold value. The training-based method is to compensate for the shortcomings of the template matching scheme. Instead of a specific template produced in advance, a training image is composed of various faces, the features are trained, and then used in the matching process. Training-based methods are also supported in OpenCV, an open library widely used in image processing and pattern recognition. Here, the window is composed of Haar features having orthogonal characteristics and then trained on various training images to detect faces. Extract only these excellent Haar features and use them in the Haar matching process.

The age / gender estimator 183 compares the feature point pattern of the face area detected by the face detector 182 with the reference face image patterns by gender / age stored in the reference data storage 131 to compare the user's age and gender. Estimate. The reference data storage unit 131 stores reference face image pattern values for each gender and age, and the reference face image pattern values for each gender and age may be plural instead of one. For example, the reference face image pattern value for male / 20 (or twenties) may be plural. The plural can increase confidence in gender and age estimates.

The playback image selector 184 reads the image content corresponding to the age / gender estimated by the age / gender estimator 183 from the image content storage 132. In one embodiment, the playback image selector 184 may search for and read the video content directly from the video content storage 132 with the estimated age / gender information. In another embodiment, the playback image selector 184 transmits age / gender information to the management server 200, and accordingly, the video content storage unit transmits the video content corresponding to the command data received from the management server 200. Read at 132. The image reproducing unit 185 reproduces the image content selected by the reproducing image selecting unit 184, that is, read.

On the other hand, the controller 180 may further pass the floating population information to the management server 200. In this case, the floating population information may be gender-divided information or may be combined information. The floating population information may coincide with the number of times the proximity of the user is detected by the proximity sensor 170, and may coincide with the estimated number of times of face detection and age / gender. The controller 180 transmits the floating population information to the management server 200, and may periodically transmit the floating population information. For example, deliver in 24 hours. Alternatively, the playback image selector 184 may deliver the real time along with age / gender information. The management server 200 makes a database of the transferred floating population information, and thus a member accessing the managing server 200 can design an advertisement plan / scenario based on the male / female floating population.

6 is a flowchart of a method of providing a user-customized image, according to an exemplary embodiment. When the user's approach is detected, the smart mirror 100 starts capturing the user to obtain an image frame (S100). The smart mirror 100 extracts a representative frame from the image frames obtained through the step S100 (S200). The smart mirror 100 detects the face region in the representative frame using the face detection algorithm (S300). When the face region is detected, the user's age and gender are estimated by comparing the feature point pattern of the detected face region with previously stored reference face image patterns by gender / age (S400). When the estimation is completed, the estimated age / gender information is transmitted to the management server 200 (S500) and, accordingly, command data is received from the management server 200 (S600). The smart mirror 100 reproduces the image content indicated by the received command data (S700).

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

Claims

A mirror disposed at the front; And a display module configured to transmit a reproduced image to at least a portion of the mirror and to display the image to the outside, wherein the smart mirror device provides a mirror function and an image display function.

A camera disposed on the front surface to photograph the user;

A storage unit storing a plurality of image contents; And

A controller configured to read and reproduce image content corresponding to an age and a gender of a user obtained by analyzing a photographed image of a camera, from a storage unit;

Smart mirror device further comprising.
The method of claim 1,

Proximity sensor for detecting the approach of the user; further comprising,

The control unit is a smart mirror device for controlling the start of shooting of the camera when the user's approach detection.
The method of claim 1,

The control unit transmits user's age and gender information to the management server, receives command data to play a specific image from the management server, and reads and reproduces the image content corresponding to the received command data from the storage unit.
The method of claim 1, wherein the control unit:

A frame obtaining unit obtaining a frame from a photographed video signal of a camera;

A face detector detecting a face region in the acquired frame; And

An age / gender estimator for estimating the age and gender of the user by comparing the detected feature point pattern of the face region and the reference face image patterns by gender and age;

Smart mirror device comprising a.
The method of claim 4, wherein

The face detector extracts a representative frame from the obtained frames, and detects a face image from the extracted representative frame.
The mirror of claim 1 wherein the mirror is:

A first mirror area providing only a mirror function; And

A second mirror area providing a mirror function or an image display function;

Smart mirror device comprising a.
A mirror disposed at the front; And a display module configured to transmit a reproduced image to at least a portion of the mirror and to display the image to the outside, wherein the smart mirror device provides a customized function for providing a mirror function and an image display function.

Obtaining an image frame photographing the user;

Extracting a representative frame from the image frames;

Detecting a face region in the extracted representative frame;

Estimating the age and gender of the user by comparing the detected feature point pattern of the face region with the reference face image patterns by gender and age; And

Playing the video content corresponding to the estimated age / gender;

User-customized image providing method comprising a.