KR102420100B1 - Electronic apparatus for providing health status information, method for controlling the same, and computer-readable storage medium - Google Patents

Abstract

Health that obtains a user's face image, normalizes the user's face image, extracts facial state information indicating the user's face state from the normalized user's face image, and uses the facial state information to indicate the user's health state Disclosed is a device according to an embodiment including a control unit for obtaining status information and a display unit for displaying health status information.

Description

ELECTRONIC APPARATUS FOR PROVIDING HEALTH STATUS INFORMATION, METHOD FOR CONTROLLING THE SAME, AND COMPUTER-READABLE STORAGE MEDIUM

Embodiments of the present invention relate to an electronic device that provides health state information, a method for controlling the electronic device, and a computer-readable storage medium storing computer program codes for performing the electronic device control method.

Recently, various electronic devices have provided health-related functions to users. Various devices are needed to measure health-related information. For example, special devices for measuring health-related information such as photographing a subject using ultrasound, CT, MRI, measuring blood pressure using a blood pressure monitor, or measuring body weight using a scale are required. However, since such devices are difficult for general users to use, it is difficult for general users to obtain health status information.

In addition, technologies for detecting a face region in an image of a user's face and identifying the detected face have recently been developed.

Accordingly, if the user's health status can be checked based on the image in which the user's face is captured, health status information can be provided to the user without a special device.

Various embodiments are provided for providing health state information to a user using a face image and enabling the user to easily obtain health state information.

Various embodiments are provided for easily collecting a face image to obtain health state information.

Various embodiments are provided for conveniently providing a user with a change in health state information over time.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure is to obtain a user's face image, normalize the user's face image, and represent the state of the user's face from the normalized user's face image It is possible to provide a device including a controller for extracting facial state information and obtaining health state information indicating a health state of a user by using the facial state information, and a display unit for displaying the health state information.

In addition, the device further includes an imaging unit for photographing the user's face, the display unit displays photographing guide information for guiding the user's face photographing according to a preset face image acquisition condition, and the control unit includes the imaging unit for the face image It is determined whether or not the user's face is photographed according to the acquisition condition, and when the imaging unit captures the user's face according to the face image acquisition condition, the image obtained by photographing may be determined as the user's face image.

Also, the controller may normalize the face image by enlarging or reducing the size of the face image to a preset size.

In addition, the controller may normalize the face image by obtaining a color temperature of light illuminating the user's face at the time the face image is captured and adjusting a color tone value of the face image based on the obtained color temperature.

In addition, the device further includes a sensing unit that detects the user's biometric information at the time the face image is captured, the display unit displays a user interface for selecting the detected user's biometric information, and the controller displays the biometric information Upon receiving the selected user input, the user's health state information may be acquired by using the selected biometric information and facial state information together.

In addition, the control unit extracts facial state information from a plurality of pre-obtained facial images of the user, and the device stores the health state information obtained from the extracted facial state information in response to a photographing date of the plurality of facial images. The storage unit further includes, wherein the control unit obtains the date the face was captured from the file of the face image of the user, and considers the user's health status information related to the face image captured before the date among the plurality of pre-acquired face images Thus, it is possible to obtain health state information from the face state information.

In addition, the display unit displays the face image and health state information corresponding to the face image together, and displays an image indicating that the displayed health state information is a region related to the displayed health state information on a region related to the displayed health state information among the entire region of the face image. can be displayed.

In addition, the face image of the user includes a plurality of face images of the user, and the display unit includes a plurality of face images and a plurality of face images in the order of the shooting date and shooting time, based on the shooting date and shooting time of the plurality of face images. The user's health state information corresponding to the face image may be displayed.

In addition, the health status information of the user includes the user's disease, and the display unit displays a user interface for selecting at least one of a plurality of diseases related to a plurality of face images, and receives a user input for selecting the at least one disease. Upon reception, a face image related to a disease selected from among a plurality of face images may be displayed according to a photographing date and photographing time order.

In addition, the display unit may display health status information corresponding to a face image captured on each date among a plurality of face images in a calendar indicating dates and days of the week for a certain period and an area corresponding to each date in the calendar. have.

In addition, a second aspect of the present disclosure includes the steps of obtaining, by the device, a face image of a user, normalizing the face image of the user, and extracting face state information indicating the state of the user's face from the normalized face image of the user. It is possible to provide a method for providing health state information, comprising the steps of: obtaining health state information indicating a user's health state by using the face state information; and displaying the health state information.

In addition, the step of obtaining the user's face image may include, according to a preset face image acquisition condition, displaying photographing guide information guiding photographing of the user's face, photographing the user's face using an image capturing unit provided in the device. Step, determining whether the image capturing unit has captured the user's face according to the face image obtaining condition, and when the capturing unit has captured the user's face according to the face image obtaining condition, the image obtained by the capturing is the user's face image It may include the step of determining

Also, the normalizing of the face image may include enlarging or reducing the size of the face image to a preset size.

In addition, the step of normalizing the face image may include obtaining the color temperature of the light illuminating the user's face at the time the face image was taken, and adjusting the color tone value of the face image based on the obtained color temperature. have.

In addition, the step of obtaining the user's face image includes obtaining the user's biometric information sensed at the time the face image is captured, and obtaining health condition information related to the user's health by using the face condition information. The steps include displaying a user interface for selecting the detected biometric information of the user, and upon receiving a user input for selecting the biometric information, use the selected biometric information and facial state information together to obtain the user's health state information It may include the step of obtaining.

In addition, the step of acquiring health status information related to the user's health by using the face status information includes: acquiring the date the face image was captured from the file of the user's face image; obtaining health state information of a user related to a previously photographed face image, and obtaining health state information by using the user's health state information and facial state information related to a face image photographed earlier than the date may include

In addition, the displaying of the health state information may include displaying a face image and health state information corresponding to the face image, and the displayed health state information on a region related to the displayed health state information among the entire region of the face image. and displaying an image indicating that the region is related to the .

In addition, the face image of the user includes a plurality of face images of the user, and the step of displaying the health state information includes: based on the shooting date and shooting time of the plurality of face images, the plurality of face images in the order of the shooting date and shooting time and displaying the user's health state information corresponding to the face image and the plurality of face images.

In addition, the health state information of the user includes the user's disease, and the displaying of the health state information includes: displaying a user interface for selecting at least one of a plurality of diseases related to a plurality of face images; In response to a user input for selecting a disease is received, the method may include displaying a face image related to a disease selected from among a plurality of face images according to a shooting date and shooting time order.

In addition, the displaying of the user's health status information may include displaying a calendar indicating dates and days of the week together for a certain period, and in an area corresponding to each date in the calendar, on each date of the plurality of face images. The method may include displaying health state information corresponding to the photographed face image.

1 is a diagram for explaining a method of extracting health state information of a user from an image including a user's face, according to an exemplary embodiment.
2 is a flowchart illustrating a method in which a device obtains health state information of a user based on a face image of the user, according to an exemplary embodiment.
3 is a diagram for describing a method for registering a user's face, according to an embodiment.
4 is a view for explaining a method for registering information about a user, according to an embodiment.
FIG. 5 is a diagram for describing a method for a device to set information to be considered during a health diagnosis based on a user's selection, according to an exemplary embodiment.
6 is a flowchart illustrating a method for a device to acquire a face image, according to an embodiment.
7 is a flowchart illustrating a method for a device to acquire a face image from an input image, according to an embodiment of the present invention.
8 is a flowchart illustrating a method for a device to acquire a face image through template capturing, according to an embodiment.
9 is a view for explaining a method for a device to provide guide information when capturing a template, according to an embodiment.
10 is a view for explaining a method for a device to provide guide information when capturing a template, according to another embodiment.
11 is a diagram for describing a method for a device to photograph a face within a predetermined distance from a camera when photographing a template, according to another exemplary embodiment.
12 is a diagram for explaining a method for photographing a face by a device so that a direction of a face in an input image is a direction facing a camera in the front, when photographing a template, according to an embodiment.
13 is a diagram for explaining a method of acquiring an input image based on whether a device has applied makeup on a user's face when a template is photographed, according to another embodiment.
FIG. 14 is a diagram for describing a method for a device to acquire an input image according to a predetermined illuminance value when capturing a template, according to another exemplary embodiment.
15A to 15B are diagrams illustrating a method in which a device acquires a face image in a face authentication mode when a template is captured, according to an exemplary embodiment.
16 is a diagram illustrating a method for a device to acquire a user's face image while performing a video call, according to an embodiment.
17 is a diagram illustrating a method for a device to acquire a user's face image while an application is running, according to an exemplary embodiment.
18 is a diagram illustrating a method for a device to acquire a face image of a user when the device is worn on a wrist and used, according to an embodiment.
19 is a diagram illustrating a method for a device to acquire a user's face image when the device is in the form of glasses, according to another embodiment.
20 is a diagram for describing photographing situation information that a device acquires together when acquiring a face image through an imaging unit, according to an embodiment.
21 is a flowchart illustrating a method for a device to acquire a face image from an image received from the outside, according to an embodiment.
22 is a diagram for describing a method for a device to acquire a face image from an external server, according to an embodiment.
23 is a diagram illustrating a method for a device to obtain a user's face image from an image selected by the user, according to an exemplary embodiment.
24 is a diagram for describing a process in which a device acquires a face image from an image stored in the device, according to an exemplary embodiment.
25 is a diagram for describing a method for a device to extract a face image from an image selected by a user, according to an exemplary embodiment.
26 is a diagram for explaining a method for a device to extract a face image from a video, according to an embodiment.
27 is a diagram illustrating a process in which a device allocates an identifier to a face image, according to an embodiment.
28 is a diagram illustrating a method for a device to record information indicating that health state information is an acquired image in an image file, according to an exemplary embodiment.
29 is a diagram illustrating a method for a device to store a face image according to an embodiment.
30 is a flowchart illustrating a method for a device to normalize a face image, according to an embodiment.
31 is a diagram illustrating a method for a device to normalize a size of a face image, according to an embodiment.
32 is a flowchart of a method for a device to adjust an effect of a color temperature of light from a face image, according to an embodiment.
33 is a flowchart illustrating a method for a device to extract facial state information indicating a state of a user's face from a normalized user's face image, according to an embodiment.
34 is a flowchart illustrating a method for a device to extract face state information appearing on a face from a normalized face image, according to an embodiment.
35 is a diagram for describing a method for a device to determine a location of a diagnostic area, according to an exemplary embodiment.
36 is a diagram illustrating a method for a device to extract facial state information from a diagnosis area, according to an exemplary embodiment.
37 is a flowchart illustrating a method for a device to obtain health state information related to a user's health based on facial state information, according to an embodiment.
38 is a diagram illustrating a method in which a device extracts health state information of a user based on facial state information extracted from a face image, according to an embodiment.
39 is a flowchart illustrating a method for a device to obtain health state information from facial state information in consideration of photographing situation information obtained when photographing a face, according to an embodiment.
40 is a diagram illustrating a method of displaying, by a device, photographing situation information obtained when capturing an image together with health state information of a user, according to an exemplary embodiment.
41 is a diagram illustrating a method of providing a function for selecting, by the device, photographing situation information to be considered by the device from among a plurality of photographing situation information, when the device obtains health state information from facial state information, according to an embodiment.
42 is a flowchart illustrating a method for a device to obtain health state information from facial state information based on a time point at which a face image is captured, according to an embodiment.
43 is a diagram for describing a process in which a device acquires health state information according to an embodiment.
44 is a diagram illustrating a process in which a device acquires health state information using a service server according to another embodiment.
45 is a flowchart illustrating a method for a device to display health state information of a user, according to an embodiment.
46 is a diagram illustrating a method of providing, by a device, a user interface for providing health state information calculated from a displayed user's face when displaying an image stored in a device, according to an embodiment.
47 is a diagram illustrating a method for a device to display health state information on a displayed image, according to an embodiment.
48A is a diagram illustrating a method of providing, by a device, a user interface for selecting a person to be displayed on a screen from among health status information about a plurality of persons, according to an exemplary embodiment;
48B is a diagram illustrating a method of displaying, by a device, an input image of a person selected by a user and health state information corresponding to the input image, according to an exemplary embodiment.
49A to 49C are diagrams illustrating a method for a device to provide health status information for a period or disease selected by a user, according to an exemplary embodiment.
50A is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.
50B is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.
51 is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.
52 is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.
53A and 53B are diagrams illustrating a method for a device to provide health status information of a user in a calendar format, according to an exemplary embodiment.
54 is a diagram for describing a method of displaying, by a device, health state information of a user when a social network application is executed, according to an embodiment.
55 shows a block diagram for a device, according to one embodiment.
56 is a block diagram illustrating a configuration of a device according to another embodiment of the present disclosure.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

In the entire specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. Also, as used herein, the term “unit” refers to a hardware component such as software, FPGA, or ASIC, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors. Thus, by way of example, “part” refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

Throughout the specification, “normalization” may mean adjusting an attribute value of a face image based on a preset standard. The attribute value of the face image may include the size of the face image, the brightness of the face image, and a color tone change value due to the influence of lighting.

Throughout the specification, a “facial image acquisition condition” may refer to a condition that a face region in an input image must satisfy in order for a device to extract face state information from the face region of the input image.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the embodiments of the present invention. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description will be omitted.

1 is a diagram for explaining a method of extracting health state information of a user from an image including a user's face, according to an exemplary embodiment.

Referring to FIG. 1 , the device 100 may obtain a face image 20 from an input image 10 . The input image 10 may be an image obtained by a user's general picture taking. Also, the input image 10 may be an image received from the external device 100 . Also, the input image 10 may be an image obtained by photographing a template for photographing a face (eg, unlocking face recognition).

The device 100 may acquire the face image 20 from the input image 10 that satisfies a preset face image acquisition condition among the input images 10 . The preset face image acquisition conditions include, for example, a condition for whether a face is detected, a condition for a face direction, a condition for illuminance during shooting, a condition for shaking during shooting, a condition for whether the eyes are open Conditions, conditions for facial expressions, conditions for whether ears are visible, conditions for whether the face is located in the center, conditions for face size, and the like may be included.

The face image 20 may mean an image including only the face region among the entire region of the input image 10 . For example, the face image 20 includes a rectangular area in which the area from the beginning of the forehead to the end of the chin among the face of the subject in the input image 10 is the vertical area and the area between the ears is the horizontal area. can mean

The device 100 may detect a face region from the input image 10 , and store the detected image of the face region as the face image 20 . An embodiment in which the device 100 acquires the input image 10 and acquires the face image 20 from the input image 10 will be described in detail below with reference to FIGS. 8 to 29 .

As the face image 20 is acquired, the device 100 may normalize the face image 20 based on a preset criterion. For example, the device 100 may change the size of the face image 20 to a preset size. Also, the device 100 may adjust the effect of the color temperature of light from the face image 20 . Also, the device 100 may change the brightness of the face image 20 to a preset brightness value. An embodiment in which the device 100 normalizes the face image 20 based on a preset criterion will be described in detail below with reference to FIGS. 30 to 32 .

As the face image 20 is normalized, the device 100 may extract face state information from the normalized face image 20 . An embodiment in which the device 100 extracts face state information from the normalized face image 20 will be described in detail below with reference to FIGS. 33 to 36 .

As the facial state information is acquired, the device 100 may acquire the health state information 40 indicating the health state of the user by using the extracted facial state information. An embodiment in which the device 100 obtains the health state information 40 indicating the health state of the user by using the extracted facial state information will be described in detail below with reference to FIGS. 37 to 44 .

Also, the device 100 may provide the acquired health state information to the user in various ways. An embodiment in which the device 100 provides health state information will be described in detail below with reference to FIGS. 45 to 54 .

2 is a flowchart illustrating a method for the device 100 to obtain health state information of a user based on a face image of the user, according to an exemplary embodiment.

In step S210, the device 100 may obtain a face image of the user.

The device may acquire an input image and acquire a face image from the acquired input image.

For example, the device 100 may acquire the input image by taking a general picture of the user. The device 100 may receive an input image from an external device.

Also, for example, the device 100 may acquire the input image by using a template photographing (eg, unlocking face recognition) for photographing a face. In template photographing, the device 100 may display photographing guide information for guiding the user to photograph a face according to a preset face image acquisition condition.

As the input image is acquired, the device 100 may determine whether the input image satisfies a preset face image acquisition condition. For example, the device 100 may determine whether the illuminance when capturing the input image is within a reference range. Also, the device 100 may determine whether the camera shakes when capturing the input image.

Also, the device 100 may determine whether a face in the face region of the input image satisfies a preset face image acquisition condition. For example, the device 100 may determine whether the angle of the face is located within a reference angle range from the front face. Also, the device 100 may determine whether the face in the input image has opened eyes. Also, the device 100 may determine whether a face in the input image represents an expression. Also, the device 100 may determine whether a face in the input image has an ear. Also, the device 100 may determine whether the face in the input image is larger than or equal to a reference size.

When the input image satisfies a preset face image acquisition condition, the device 100 may acquire an image of the face region from the input image as a face image.

In step S220 , the device 100 may normalize the user's face image.

For example, the device 100 may normalize the face image of the user by enlarging or reducing the size of the face image to a preset size. Also, the device 100 may exclude the effect of the color temperature of light from the face image. For example, the face image of the user may be normalized by acquiring the color temperature of light illuminating the user's face at the time the face image is captured, and adjusting the hue value of the face image based on the acquired color temperature. Also, the device 100 may normalize the user's face image by changing the brightness of the face image to a preset brightness value.

In step S230 , the device 100 may extract face state information indicating the state of the user's face from the normalized face image.

The face state information may refer to information regarding whether the user's face exhibits symptoms. The facial state information may include, for example, the color of the face, the number and size of blemishes or acne, whether the eyes are red, the color of the eyes, the size of the pupil, the movement of the pupil, the size of the face, the shape of the face outline, the color of the lips, the lips It may include whether or not the skin is split, the position of each organ (eyes, nose, mouth, ears, eyebrows) of the face, the color of the hair, the gloss of the hair, whether the hair is split, the movement of the facial muscles, and the like.

As the face image is normalized, the device 100 may determine a region from which face state information is extracted from the normalized face image. For example, the device 100 may determine positions such as an under-eye region, a nose region, and an area around the mouth in the face image.

As the region from which the facial state information is to be extracted is determined, the device 100 may extract facial state information from each region. For example, the device 100 may determine whether the area under the eyes is inflated. Also, the device 100 may determine whether the color of the area under the eyes has changed to a darker color than the previous color.

In operation S240 , the device 100 may obtain health state information indicating the health state of the user by using the face state information.

The health state information may include information about a user's disease, a function-degraded organ, or a user's condition predicted from the facial state information.

For example, when it is determined that the area under the eyes is swollen in the face image, the device 100 may determine that the user has hyperthyroidism or an allergic disease. In addition, when it is determined that the area under the eyes has turned black in the face image, the device 100 may determine that the user has allergic rhinitis.

In addition to the extracted face state information, the device 100 may acquire health state information in consideration of the user's biometric information obtained during photographing. In this case, the device 100 may display a user interface for selecting at least one of the plurality of obtained biometric information. Upon receiving the user input for selecting biometric information, the device 100 may obtain health state information related to the user's health by considering the extracted facial state information and the selected biometric information together.

In addition, when obtaining health state information from the face state information, the device 100 may obtain the health state information of the user by using the health state information of the user prior to the point in time when the face image is captured together with the face state information of the user. .

For example, when obtaining health state information from arbitrary face state information, the device 100 may obtain a date when the face was captured from metadata of an arbitrary face image. The device 100 may extract the user's previous face state information from a plurality of face images captured before the date the face image was captured. The device 100 may obtain the user's previous health state information based on the previous facial state information. The device 100 may acquire health state information by using the obtained previous health state information and the extracted facial state information together.

In step S250 , the device 100 may provide the obtained health state information.

The device 100 may display the acquired health state information on the screen in various ways.

For example, the device 100 may display a face image and health state information corresponding to the face image together. In this case, the device 100 may display an image indicating that the displayed health state information is a region related to the displayed health state information on a region from which the facial state information obtained from among the entire region of the face image is extracted.

In addition, when displaying a plurality of face images, the device 100 determines the plurality of face images and the user corresponding to the plurality of face images according to the photographing date and photographing time order based on the photographing date and photographing time of the face image. health status information can be displayed.

Also, the device 100 may display a user interface for selecting at least one disease obtained from the user's face state information extracted from a plurality of face images. Upon receiving a user input for selecting at least one disease, the device 100 may display an image related to the selected disease from among a plurality of face images according to the shooting date and shooting time order.

Also, the device 100 may display a calendar indicating a date and a day of the week for a predetermined period. Also, the device 100 may display health state information obtained from facial state information extracted from a face image captured on a date among a plurality of face images in an area corresponding to the date in the calendar.

3 is a diagram for describing a method for registering a user's face, according to an embodiment.

Referring to FIG. 3 , the device 100 may provide a user face registration interface 310 capable of photographing a user's face and registering the photographed face as the user's face.

The user face registration interface 310 may include an interface for capturing face images in various directions, such as the front and side surfaces of the user's face.

Also, the user face registration interface 310 may include a guide image 320 for photographing the user's face according to a predetermined size.

Upon receiving a user input for registering a photographed face image, the device 100 may store the photographed face image corresponding to the user's identification information. As the user's face image is registered, the device 100 may extract facial features based on the registered face image. The device 100 may store the extracted feature in response to the user's identification information. Accordingly, the device 100 extracts facial features from an arbitrary image, compares the extracted facial features with pre-stored features of the user's face, and, when the similarity is greater than or equal to a threshold, sets the face in the arbitrary image to the user's You can decide by face.

According to an embodiment, the device 100 may register the user's face based on an image selected by the user from among the images stored in the device 100 .

4 is a view for explaining a method for registering information about a user, according to an embodiment.

Referring to FIG. 4 , the device 100 may provide an interface for registering information about a user.

The information about the user may include information about the user's body and the user's medical history. For example, the information about the user may include the user's age, weight, height, gender, information about a current disease, information about a past disease, and information about a face color.

The device 100 may receive information about the user from the user. Also, the device 100 may store information about the user in response to the user's identification information. The device 100 may more accurately obtain the user's health state from the user's face state information in consideration of the user information.

FIG. 5 is a diagram for explaining a method of setting, by the device 100, information to be considered during a health diagnosis based on a user's selection, according to an exemplary embodiment.

Referring to FIG. 5 , the device 100 may provide a user interface for selecting information to be considered during a health diagnosis.

The information that may be considered during the health diagnosis may include a photographing time, a photographing date, a photographing location, biometric information, and information about a user. The biometric information may include heart rate, blood pressure, blood sugar level, sleep amount, and EEG information. The biometric information may be obtained from a sensor provided in the device 100 when photographing. Also, the biometric information may be recorded in the form of metadata in the input image.

As information that can be considered during health diagnosis is selected, the device 100 may obtain health state information from the user's face state information in consideration of the selected information.

6 is a flowchart illustrating a method for the device 100 to acquire a face image, according to an embodiment.

In step S610, the device 100 may obtain a face image of the user.

The device 100 may acquire an input image. As the input image is acquired, the device 100 may detect a face region in the input image. As the face region in the input image is detected, the device 100 may determine whether the face region satisfies a face image acquisition condition. When the face region satisfies the face image acquisition condition, the device 100 may store data of the face region as a face image.

7 is a flowchart illustrating a method for the device 100 to obtain a face image from an input image according to an embodiment of the present invention.

In step S710 , the device 100 may acquire an input image.

The device 100 may acquire the input image in various ways.

For example, the device 100 may receive an input image including a face from an imaging unit provided in the device 100 . In this case, the device 100 may provide a template capturing interface for obtaining an input image that satisfies a preset face image obtaining condition. This embodiment will be described in detail below with reference to FIGS. 8 to 14 .

Also, when the device 100 satisfies a condition for photographing the user's face, the device 100 may photograph the user's face even if there is no user input. This embodiment will be described in detail below with reference to FIGS. 15 to 19 .

Also, the device 100 may acquire a picture selected by the user from among the images stored in the device 100 as an input image. This embodiment will be described in detail below with reference to FIGS. 21 to 25 . Also, the device 100 may acquire an image downloaded from the external device 100 as an input image. This embodiment will be described in detail below with reference to FIG.

In operation S720 , the device 100 may detect a face region in the input image.

The device 100 may detect the face region in the input image according to various algorithms. For example, the device may input an input based on Knowledge-based Methods, Feature-based Methods, Template-matching Methods, and Appearance-based Methods. The position of the face region in the image may be determined.

In operation S730, the device 100 may determine whether the face region satisfies a face image acquisition condition.

The face image acquisition condition may mean a condition that a face region in the input image must satisfy in order for the device 100 to extract face state information from the face region of the input image. The face image acquisition conditions include, for example, a condition for whether a face is detected, a condition for a face direction, a condition for illuminance at the time of shooting, a condition for whether the camera is shaken during shooting, a condition for whether the eyes are open, a face condition It may be determined by a combination of various criteria, such as a condition for an expression, a condition for whether the ears are visible, a condition for whether the face is located in the center, and a condition for the size of the face.

The condition for whether or not a face is detected is a condition that, when a face is detected from the input image, is adopted as the face image, and when no face is detected, the corresponding input image is not adopted as the face image. The device 100 may use various face detection algorithms, such as the Adaboost algorithm, to detect the face image from the input image.

The condition for the face direction is a condition for adopting the input image as a face image when a face in a predetermined direction is detected from the input image. For example, when an image obtained by photographing a front face is requested, the device 100 adopts the input image as the face image when a front face is detected from the input image. As another example, when an image of the side of the face is requested, the device 100 adopts the input image as the face image when the side of the face is detected from the input image.

The condition for illuminance during photographing may be, for example, a condition in which the illuminance during photographing is equal to or greater than the first illuminance and less than or equal to the second illuminance. When the input image does not satisfy the condition for illuminance during the photographing, the device 100 may not adopt the input image as the face image. The condition for the illuminance during the photographing may be measured using an illuminance sensor provided in the device 100 during the photographing. If the input image is a stored image, an illuminance condition at the time of photographing stored in an image file of the input image may be used.

The condition for whether the image is shaken during photographing is a condition in which the input image is not used as a face image if the input image shakes more than a predetermined level. According to an embodiment, whether the input image is shaken may be measured using a shake sensor when photographing. According to another exemplary embodiment, whether the input image is shaken may be determined from the input image.

The condition for whether the eyes are open is a condition in which a face is detected in the input image, but when the eyes are closed, the corresponding input image is not adopted as the face image. For example, the eyes may be detected from the input image, the whites and the black regions of the eyes are detected, and when the areas of the whites and the black regions are equal to or less than a predetermined value, it may be determined that the eyes are closed.

The condition for the facial expression is a condition for adopting a corresponding input image as a facial image when a predetermined facial expression is detected according to health state information to be measured. The device 100 may detect a face from the input image and perform facial expression recognition to determine whether a predetermined facial expression is detected in the corresponding input image. The predetermined facial expression may be set as a smiling face, an expressionless face, a face with eyes closed, and the like.

The condition for whether the ears are visible is a condition in which a face is detected in the input image and, when the ears are visible, the corresponding input image is adopted as the face image.

In operation S740 , when the face region satisfies the face image acquisition condition, the device 100 may store data of the face region as a face image.

When the face region satisfies the face image acquisition condition, the device 100 may extract data of the face region in the input image and store the extracted data as a face image.

8 is a flowchart illustrating a method in which the device 100 acquires a face image through template capturing, according to an embodiment.

In operation S810 , the device 100 may display photographing guide information for guiding the user to photograph a face according to a preset face image acquisition condition on the screen.

For example, the device 100 may provide an application for diagnosing a user's health through a template photographing. Upon receiving a user input for selecting an application for photographing a template, the device 100 may enter a health examination mode. In the health diagnosis mode, the device 100 may display photographing guide information for guiding photographing a face according to a preset face image acquisition condition.

According to an embodiment, when a user enters a health examination mode through a menu operation while a photo album function capable of reproducing images stored in the device 100 is being executed, the device 100 performs an operation of photographing a face image. can make it happen As another example, when a user executes an application for health diagnosis installed on the device 100 , the device 100 may perform an operation of photographing a face image.

According to an embodiment, the device 100 may provide a photographing guide in order to photograph a face image in the health examination mode. The shooting guide may be provided in the form of a screen, a sound, a blinking LED, or the like.

According to an embodiment, the photographing guide may be provided while a preview image is displayed in the health examination mode. The device 100 may update the shooting guide by continuously determining whether a condition for obtaining a face image is satisfied from the preview image.

According to another embodiment, the photographing guide may be provided when a control signal requesting provision of the photographing guide is input from the user. For example, the user may request the device 100 to provide a photographing guide by selecting a specific menu or button in the medical examination mode.

According to another embodiment, when the device 100 includes a shutter button (not shown) for receiving a shutter release signal, and the shutter button can be pressed in two steps of a half-pressed state and a full-pressed state, the device ( 100) may calculate and provide the shooting guide when the shutter button is pressed halfway. Here, when the shutter button is pressed halfway, an S1 signal for commanding AF may be generated, and when the shutter button is fully pressed, an S2 signal corresponding to a shutter release signal may be generated.

The photographing guide information may include a guide image or guide phrase for photographing a face according to a face image acquisition condition. The guide information will be described in detail below with reference to FIGS. 9 to 14 .

In operation S820 , the device 100 may receive an image including the user's face from the imaging unit provided in the device 100 .

As the image including the user's face is received from the imaging unit, the device 100 may display the received image.

In operation S830, the device 100 may determine whether a face region in the received image satisfies a face image acquisition condition.

For example, the device 100 may determine whether a face is detected in the received image based on the face detection algorithm.

Also, for example, the device 100 may detect a “‡ direction of a face in the received image, and determine whether the detected direction of the face satisfies a reference direction. For example, the device 100 may calculate a left-right rotation direction (Yaw Direction) of the face, a vertical rotation direction (Tilt Direction), and a roll direction of the face itself from the received image. The reference direction may be a direction in which the face displayed in the received image is photographed when the face is viewed from the front.

Upon detecting the “‡ direction of the face in the received image, the device 100 may determine that the detected direction of the face satisfies the reference direction when the calculated direction is within a preset range from the reference direction.

Also, for example, the device 100 may detect an illuminance value during photographing. The device 100 may obtain an illuminance value at the time of photographing from an illuminance sensor provided in the device 100 . As the illuminance value during photographing is obtained, the device 100 may determine whether the illuminance value during photographing is within a preset range.

Also, for example, the device 100 may determine whether a subject or a camera is shaken during photographing. For example, the device 100 may obtain a value indicating the degree of shaking from a shake sensor provided in the device 100 . The device 100 may determine whether a value indicating the degree of shaking is within a preset range.

Also, for example, the device 100 may determine whether the size of the face in the received image is greater than or equal to a preset reference size.

When the face region in the image received in operation S830 satisfies the face image acquisition condition, in operation S840 , the device 100 may store the received image as an input image.

The device 100 may store only the detected face region among the input images as the face image. Also, the device 100 may store the entire input image as a face image.

When the image received in step S830 does not satisfy the face image acquisition condition, the device 100 may display guide information for indicating the unsatisfied face image acquisition condition on the screen.

For example, when the angle of the face is not located within the preset angle range, the device 100 may indicate the direction and angle in which the user should turn the face on the screen.

9 is a diagram illustrating a method of providing guide information by the device 100 when a template is photographed, according to an exemplary embodiment.

When the photographing of the template for the health diagnosis is started, a preview image may be displayed as shown in FIG. 9 . According to the present embodiment, in the preview image, it is possible to provide a guide for photographing so that a condition for obtaining a face image is satisfied.

The device 100 determines from the preview image whether a condition for obtaining a face image is satisfied, a shooting guide for whether the condition for obtaining a face image is satisfied, or how a user should take a picture in order to satisfy the condition for obtaining a face image A shooting guide can be provided.

The shooting guide may be displayed on the screen as shown in FIG. 9 . In another embodiment, the photographing guide may be output as a sound.

The photographing guide may include, for example, an instruction to bring the face closer or an instruction to move the face away in order to adjust the face size as shown in FIG. 9 . Also, the photographing guide may include, for example, a guide to brighten or darken the lighting according to the measured illuminance. Also, the photographing guide may include, for example, a guide to make the ears visible when the ears are not visible in the photographed face. In addition, the photographing guide may include a guide for facial expressions, a guide for facial positions, and the like.

According to an embodiment, the device 100 provides a photographing guide for unmet conditions among the face image acquisition conditions, and whenever a face image acquisition condition is satisfied, a guide for the corresponding face image acquisition condition is provided can be terminated. According to another embodiment, the device 100 may display all face image acquisition conditions on the display unit 120 and provide information on whether each face image acquisition condition is satisfied.

10 is a view for explaining a method for the device 100 to provide guide information when capturing a template according to another embodiment.

According to the present embodiment, the device 100 may display a face box 1010 for informing the size and location of the face according to the face image acquisition condition on the preview screen. The user can easily take a face image by placing the face on the face box 1010 on the preview screen.

According to an embodiment, the device 100 detects the position and size of the face in the preview image, compares it with the face image acquisition condition, and provides information on whether the face image acquisition condition is satisfied through the face box 1010 . can be printed out. For example, the device 100 changes the color, line type, existence, etc. of the face box 1010 according to whether the face image acquisition condition for the position and size of the face is satisfied, so that the face image acquisition condition is satisfied It is possible to provide the user with information on whether or not

When a face image is obtained through photographing, the device 100 may generate an image file for storing the photographed face image and store the image file in the device 100 . The device 100 may also store information that the content of the image file is a face image used for health diagnosis in the image file.

According to an embodiment, information that the image stored in the image file is the face image may be stored in the header of the image file.

According to another embodiment, the device 100 may store and manage information that the image file is a face image as separate data. For example, the device 100 may store a file name and a storage path of an image file corresponding to the face image as a separate file.

According to an embodiment, the device 100 may store information related to the condition for obtaining the face image together in an image file. For example, the device 100 may store, in the image file, information on the position of the face, the size of the face, the facial expression, the illuminance at the time of shooting, the direction of the face, the shaking at the time of shooting, etc. in the image file together with the face image in the image file. have. Information that the stored image is a face image and information related to a condition for obtaining the face image may be stored, for example, in a header portion of the image file.

11 is a diagram for describing a method for photographing a face by the device 100 within a predetermined distance from a camera when photographing a template, according to another exemplary embodiment.

Referring to FIG. 11 , the device 100 may display a user interface for photographing a face within a predetermined distance from the camera.

For example, the device 100 may display a guide image 1110 for photographing a face at a location separated by a preset distance from the camera on the screen. The guide image 1110 may be a square or an elliptical image of a face shape.

Also, for example, the device 100 may display a guide phrase 1120 for requesting to match the face to the guide image 1110 together with the guide image 1110 .

The device 100 may determine that the user's face is located at a location separated from the camera by a preset distance based on whether the user's face is located in the area of the guide image 1110 . For example, the device 100 may detect the contour of the user's face from the image received from the imaging unit, and determine whether the detected contour is located in the area of the guide image 1110 . When the contour of the user's face is located in the area of the guide image 1110 , the device 100 may determine that the user's face is located at a location separated by a preset distance from the camera.

When the user's face region is located within the guide image 1110 region, the device 100 may determine the received image as the input image. When the user's face area is not located within the guide image 1110 area, the device 100 may display a guide phrase for guiding to move the face in a specific direction based on the location of the user's face.

12 is a diagram for explaining a method for photographing a face by the device 100 so that the direction of the face in the input image is the direction facing the camera in the front, when photographing the template, according to an embodiment.

Referring to FIG. 12 , the device 100 may determine whether the direction of the face is a direction to face the camera without rotating left or right.

For example, the device 100 may detect a face region from an image received from the imaging unit. As the face region is detected, the device 100 may detect the position of the eyes, nose, or mouth in the face region. As the positions of the two eyes are detected, the device 100 may determine a center point between the two eyes. At this time, if the center between the two eyes is located within a preset range in the left and right directions from the vertical center line 1220 that divides the box image into two equal parts, the device 100 does not rotate the user's face left or right. It can be determined that the direction is facing the camera directly.

When the center between the two eyes is not located within a preset range on the vertical center line 1220 that divides the box image into left and right halves, the device 100 guides the face to rotate left or right A guide message may be displayed.

13 is a diagram for explaining a method of acquiring an input image based on whether or not the device 100 has makeup on a user's face when photographing a template, according to another embodiment.

Referring to FIG. 13 , the device 100 may detect a face region in an image received from the imaging unit, and determine whether artificial makeup has been applied to the user's face based on color data of the detected face region. .

The device 100 may determine whether or not makeup is applied to the user's face based on the color, brightness, or saturation indicated by the data of the face region.

For example, the eyelid region may be detected from the detected face region. When the detected color data of the eyelid region differs from the skin color of other parts of the face by more than a reference value, it may be determined that makeup is applied to the eyelid region.

Also, for example, the device 100 may detect the lip region from the face region. When the color data of the lip region indicates brightness or saturation exceeding the reference range, it may be determined that the lip color is not the original lip color of the user.

Also, the device 100 may store a face image representing the user's original face without makeup as a reference image. For example, the device 100 may store, as a reference image, a face image having the largest number of blemishes, pimples, or dots among recently photographed face images of the user.

As the device 100 detects the facial region from the input image, the device 100 may detect the location of blemishes, pimples, or dots within the detected facial region. And, when the number of blemishes, pimples, or dots in the reference image differs from the number of blemishes, pimples or dots in the detected face region by more than the reference number, the device 100 determines that makeup is applied to the user's face. may be In this case, the device 100 may compare with the reference image for each part of the detected face region. Also, the device 100 may compare only a predetermined specific region with the reference image.

When it is determined that the user's face has makeup applied, the device 100 may display a guide phrase guiding the user to remove the makeup.

Also, when it is determined that the user's face is not made-up, the device 100 may determine the image received from the imaging unit as the input image. In addition, when it is determined that only a specific part is made-up, the device 100 determines the image received from the imaging unit as an input image, acquires a face image from the input image, and excludes only the makeup part from among the acquired facial images. Health state information may be obtained from the remaining face regions.

14 is a diagram for explaining a method of acquiring an input image according to a predetermined illuminance value by the device 100 when capturing a template, according to another exemplary embodiment.

Referring to FIG. 14 , the device 100 may determine whether the illuminance value is within a reference range when the user's face is photographed.

The device 100 may measure an illuminance value at the time of photographing by using an illuminance sensor provided in the device 100 . The device 100 may determine whether the measured illuminance value is within a reference range.

When the measured illuminance value is less than or greater than the reference range, the device 100 may display a guide message requesting the user to move to a brighter place or a darker place.

15A to 15B are diagrams illustrating a method in which the device 100 acquires a face image in a face authentication mode when capturing a template, according to an exemplary embodiment.

Referring to FIG. 15A , in the case of the device 100 that unlocks the lock state using face authentication, an image captured in a mode for photographing a user's face for face authentication may be used as a face image for health diagnosis. .

For example, in the locked state of the device 100 , when a user input for selecting to perform unlocking using face authentication is received, the device 100 may enter the face authentication mode. As the face authentication mode is entered, the device 100 may receive an image including the user's face from the image sensor. The device 100 may detect a face region from the received image and determine whether the detected face region satisfies a face image acquisition condition. When the detected face region satisfies the face image acquisition condition, the device 100 may acquire data of the detected face region as a face image.

According to the present embodiment, it is possible to easily collect a face image without additional manipulation by the user. In addition, since the face image captured for face authentication is captured under conditions similar to those of the face image obtaining condition, it is possible to easily collect a face image that satisfies the face image obtaining condition. Furthermore, when shooting for face authentication, face authentication processing is performed on the captured image, and since identification information of a photographed face can be obtained as a result of the authentication process, it is easy to perform face image and identification information can be obtained.

According to the present embodiment, as shown in FIG. 15B , when face authentication is successful in the face authentication mode, health state information extracted from a face image used for face authentication may be provided on the first screen after unlocking.

16 is a diagram illustrating a method in which the device 100 acquires a user's face image while executing a video call, according to an embodiment.

Referring to FIG. 16 , the device 100 may acquire a face image of a user while executing a video call application.

For example, the device 100 may determine whether a video call function provided in the device 100 is being executed. When the video call function provided in the device 100 is executed, the device 100 may receive an image including the user's face from the image sensor. The device 100 may detect the face region from the received image. Upon detecting the face region, the device 100 may determine whether the face region satisfies a face image acquisition condition. When the detected face region satisfies the face image acquisition condition, the device 100 may acquire data of the detected face region as a face image.

The device 100 may continuously receive an image including the user's face during a video call. The device 100 may select an image received at a predetermined time interval among continuously received images, and determine whether a face region in the selected image satisfies a face image acquisition condition.

17 is a diagram illustrating a method in which the device 100 acquires a user's face image while executing an application, according to an exemplary embodiment.

Referring to FIG. 17 , the device 100 may acquire a face image of a user at a preset time interval when an application is executed.

For example, the device 100 may determine whether a game, a video, or a web browser provided in the device 100 is running. When a game, video, or web browser provided in the device 100 is executed, the device 100 may capture an image by driving an imaging unit provided in the device 100 . When the execution of the application starts, the device 100 may drive the imaging unit to capture an image, and may periodically drive the imaging unit to capture an image.

Then, the device 100 may determine whether a face region exists in the captured image. When the face region is detected, the device 100 may determine whether the face region satisfies a face image acquisition condition. When the detected face region satisfies the face image acquisition condition, the device 100 may acquire data of the detected face region as a face image.

18 is a diagram illustrating a method in which the device 100 acquires a user's face image when the device 100 is worn on the wrist and used according to an embodiment.

Referring to FIG. 18 , as the user raises the wrist to look at the device 100 , the device 100 worn on the user's wrist may acquire a face image of the user.

The device 100 may determine whether the device 100 is close to the user's face. For example, the device 100 may measure a moving direction or a moving distance of the device 100 by using a motion sensor provided in the device 100 . The measured movement direction may include height from the ground, up, down, left, right, and rotational directions of the wrist.

Based on the measured movement direction or movement distance, the device 100 may determine that the device 100 is located within a reference distance from the user's face.

When it is determined that the device 100 is located within a reference distance from the user's face, the device 100 may capture an image by driving an imaging unit provided in the device 100 . In addition, the device 100 may determine whether a face region exists in the captured image. When the face region is detected, the device 100 may determine whether the face region satisfies a face image acquisition condition. When the detected face region satisfies the face image acquisition condition, the device 100 may acquire data of the detected face region as a face image.

In addition, when it is determined that the device 100 is located within a reference distance from the user's face, and receives a user's input for operating the device 100 , the device 100 captures images provided in the device 100 . You can also capture an image by driving the unit.

19 is a diagram illustrating a method in which the device 100 acquires a user's face image when the device 100 is in the form of glasses, according to another embodiment.

Referring to FIG. 19 , as the user looks at the mirror while wearing the device 100 configured in the form of glasses on the face, the device 100 may obtain an image of the user's face.

The device 100 may photograph an environment located in the user's field of view by using the imaging unit provided in the device 100 . When the user looks at the mirror while wearing the device 100 , the device 100 may obtain an image in which the user's face is reflected in the mirror and received.

The device 100 may determine whether a face is present in the image received through the imaging unit. When a face is present, the device 100 may detect a face region from the received image and determine whether the detected face region is the user's face. When the detected face area is the user's face, the device 100 may determine whether the detected face area satisfies a face image acquisition condition. When the detected face region satisfies the face image acquisition condition, the device 100 may acquire data of the detected face region as a face image.

20 is a diagram illustrating photographing situation information that the device 100 acquires together when acquiring a face image through an imaging unit, according to an embodiment.

Referring to FIG. 20 , the device 100 may acquire and store shooting situation information when acquiring a face image of a user through an imaging unit provided in the device 100 .

The photographing situation information may include illuminance during photographing, photographing time, photographing location, and biometric information obtained from a biometric sensor.

For example, the device 100 may obtain illuminance information when photographing using an illuminance sensor attached to the device 100 . Also, the device 100 may obtain information about a time when the face image was captured. In addition, the device 100 may obtain information about the shooting location by using a device such as GPS.

In addition, the device 100 may receive information from a biometric sensor attached to the user's body, and may acquire the user's biometric information when photographing. For example, the device 100 may receive information about the amount of movement of the user from a pedometer attached to the user's body. Also, the device 100 may receive information about the user's body temperature from a thermometer attached to the user's body. Also, the device 100 may receive information about the user's heart rate from an electrocardiogram attached to the user's body.

Also, the device 100 may acquire information about an activity that the user is doing at the time of shooting.

For example, the user's activity information may be estimated based on the shooting location or the shooting time. For example, when the shooting location is a school and the shooting time is 10 am, the user's activity may be estimated as school life.

Also, for example, the user's activity information may be estimated by analyzing a photo in which a face image is detected. For example, when food is captured in a photo in which a face image is detected, the user's activity may be estimated as a meal.

In addition, the device 100 may estimate the user's activity information by further considering not only the photo in which the face image is detected but also images taken before and after the time when the face image is captured.

The shooting situation information may be stored corresponding to the input image. For example, the shooting situation information may be recorded in the image file in the form of metadata. In addition, the shooting situation information may be stored in a separate storage space in response to the identification information of the input image. The photographing context information may be used when the device 100 obtains the user's health state information from the face state information.

21 is a flowchart illustrating a method for the device 100 to obtain a face image from an image received from the outside, according to an embodiment.

In step S2110 , the device 100 may receive an input image from an external network connected to the device 100 .

As the device 100 receives a user input for selecting a photo, the device 100 may download the selected photo from the external server. The external server may include a social network service (SNS) server, a cloud storage server, and another device 100 connected to the device 100 by short-distance communication or by wire.

In operation S2120, the device 100 may detect a face region in the input image.

As the device 100 receives the input image from the external network, the device 100 may detect the face region in the input image. The device 100 may detect a human face region from the received input image and extract a location of the detected face region.

In operation S2130, the device 100 may extract a face region representing the user's face from among the face regions detected in the input image, based on the pre-registered features of the user's face.

For example, the device 100 may extract facial features from the detected face region in the received image. Various methods such as Gabor filter or LBP (Local Binary Pattern) may be used as a method of extracting facial features from the face region.

The device 100 may compare the facial features extracted from the facial regions in the input image with the pre-registered facial features of the user, and determine a facial region having a similarity within a preset range as the user's face region.

In operation S2140, the device 100 may determine whether the user's face region satisfies a face image acquisition condition. In operation S2150 , when the face region of the user satisfies the face image acquisition condition, the device 100 may acquire the face region of the user as a face image.

22 is a diagram illustrating a method for the device 100 to obtain a face image from an external server, according to an embodiment.

Referring to FIG. 22 , the device 100 may exchange data such as a face image, face state information, and health state information with an external device or server.

The device 100 may use the received data to acquire health state information, or transmit a face image, face state information, health state information, etc. to an external device or server.

According to an embodiment, the device 100 may download a face image from an external server.

According to an embodiment, the device 100 may download a face image from a cloud server.

As another example, whenever an image including information that the corresponding image is a face image is added to a predefined cloud account, the device 100 may download the corresponding image and use it as a face image. Whether a new image has been added to the cloud server may be known by, for example, a notification pushed from the corresponding cloud server or periodic communication with the cloud server.

According to an embodiment, the device 100 may download a face image from a social network service (SNS) server. For example, whenever an image is added to a predetermined SNS account, the device 100 determines whether the corresponding image meets the face image acquisition condition, and when the face image acquisition condition is satisfied, the device 100 converts the image to the face. Available as a video.

As another example, whenever an image including information that the corresponding image is a face image is added to a predetermined SNS account, the device 100 may download the corresponding image and use it as a face image.

According to another embodiment, the device 100 may extract a face image from viewable images using a pre-specified SNS account. For example, the device 100 may include a photo registered to another account registered as a friend in a pre-specified SNS account, a photo shared by a user of another account and viewable in the pre-specified SNS account, a photo tagged with my account, etc. It is possible to extract the face image from The device 100 may determine whether images that can be viewed using the predetermined SNS account satisfy a condition for obtaining a face image, and when the condition for obtaining a face image is satisfied, the device 100 may use the image as the face image.

Whether a new image is registered in a pre-specified SNS account or another account registered as a friend can be known by, for example, a notification pushed from the corresponding SNS server or periodic communication with the SNS server.

According to another embodiment, the device 100 may download the face image from a service server that provides a function of providing health state information. For example, a face image is stored for each user account in the service server, and the device 100 may access a predetermined account and download the face image from the corresponding account. According to an embodiment, the service server may provide the device 100 with face state information and/or health state information related to the face image together with the face image.

According to an embodiment, after correcting the face image according to a predetermined algorithm, the device 100 may extract face state information from the corrected face image and obtain health state information from the extracted face state information.

23 is a diagram illustrating a method in which the device 100 obtains a user's face image from an image selected by the user, according to an exemplary embodiment.

Referring to FIG. 23 , the device 100 may obtain a user's face image from an image selected by the user.

The device 100 may display an image 2310 selected by the user from among a plurality of images stored in the device 100 on the screen. Also, the device 100 may display a menu 2315 for providing health status information of the user from the user's face in the image 2310 displayed on the screen.

Upon receiving the user input for selecting the menu 2315 providing the user's health status information, the device 100 may determine the image 2310 displayed on the screen as the input image. As the image 2310 displayed on the screen is determined as the input image, face regions 2320 to 2340 may be detected in the input image 2310 displayed on the screen. The device 100 may detect human face regions 2320 to 2340 from the input image 2310 and extract positions of the detected face regions 2320 to 2340 .

As the positions of the face regions 2320 to 2340 are extracted, the device 100 may extract facial features from the detected face regions 2320 to 2340 . The device 100 may compare the facial features extracted from each of the face regions 2320 to 2340 with the facial features of a previously registered user to determine the user represented by each of the face regions 2320 to 2340 . For example, when the degree of similarity between a facial feature extracted from one face region and a previously registered facial feature of a user is within a preset range, the device 100 sets the corresponding face region as the user's face region 2330 . can decide

As the user indicated by each of the face regions 2320 to 2340 is determined, the device 100 may display user identification information corresponding to each of the face regions 2320 to 2340 .

Also, as a user input for selecting one of the face regions 2320 to 2340 is selected, the device 100 may determine whether the face region 2330 of the user satisfies a face image acquisition condition. When the face region 2330 of the user satisfies the face image acquisition condition, the device 100 may acquire the face region 2330 of the user as a face image.

Also, the device 100 may normalize the obtained face image based on a preset criterion. The device 100 may extract face state information appearing on the face from the normalized face image. The device 100 may obtain health state information for the selected user by using the face state information, and may provide the obtained health state information.

24 is a diagram for describing a process in which the device 100 acquires a face image from an image stored in the device 100 according to an exemplary embodiment.

Referring to FIG. 24 , the device 100 may extract a face image from images stored in the device 100 . The device 100 may determine whether each of the images stored in the device 100 satisfies the face image acquisition condition, and may extract images that meet the face image acquisition condition as the face image.

According to an embodiment, whenever an image is captured and stored in the device 100 , or whenever an image is input from an external device and stored in the device 100 , the device 100 displays a newly stored image of the face. It may be determined whether an image acquisition condition is satisfied.

According to another embodiment, the device 100 periodically checks whether the images stored in the device 100 meet the face image acquisition conditions for images that have not yet been determined whether or not they meet the face image acquisition conditions. can determine whether For example, the device 100 may determine whether a condition for acquiring a face image is satisfied with respect to newly stored images every week or for one week.

According to another embodiment, whenever the user inputs a request signal for requesting health state information, the device 100 may perform a face image acquisition condition for images for which it has not yet been determined whether or not they meet the face image acquisition condition. It can be determined whether or not For example, in a state in which 100 images are stored in the electronic device and it is determined whether or not a condition for obtaining a face image is satisfied with respect to 90 images, when a user inputs a request signal for requesting health state information, the device 100 may extract a face image from the remaining 10 images, extract face state information, and then provide health state information.

25 is a diagram for explaining a method of extracting, by the device 100, a face image from an image selected by a user, according to an embodiment.

Referring to FIG. 25 , a user may directly select an image to be used as a face image among images stored in the device 100 . For example, the user may select an image to be used as a face image while a function such as a photo album or a gallery is executed. In this case, as shown in FIG. 25 , in a state in which thumbnail images of images stored in the device 100 are displayed, the user may select an image to be used as a face image.

According to an embodiment, the device 100 may use all images selected by the user as face images.

According to another embodiment, the device 100 may determine whether a condition for obtaining a face image is satisfied with respect to images selected by a user, and extract images that satisfy the condition for obtaining a face image as a face image.

According to an embodiment, as shown in FIG. 25 , when a user inputs a control signal requesting health state information while a plurality of images are selected, the device 100 extracts facial state information using the selected images. and extract and provide health status information. A control signal for requesting health state information by the user may be generated and input by, for example, pressing a health care button as shown in FIG. 25 .

According to another embodiment, when a user inputs a control signal requesting health status information while a function such as a photo album or a gallery is executed, the device 100 automatically displays the image included in the photo album or gallery on the face. After determining whether an image acquisition condition is satisfied and extracting a face image, it is possible to obtain and provide face state information and health state information from the face image. In this case, even if the user does not select an image to be used as the face image, health status information can be obtained with only one selection signal.

26 is a diagram illustrating a method of extracting a face image from a video by the device 100, according to an embodiment.

Referring to FIG. 26 , the device 100 may extract a face image from a video. In this case, it is determined whether a condition for obtaining a face image is satisfied for each of the video frames, and a video frame that satisfies the condition for obtaining a face image can be extracted and used as a face image. According to an embodiment, when a condition for obtaining a face image is satisfied in successive video frames, one of the successive video frames may be extracted as the face image.

According to an embodiment, the user may select a health diagnosis function while playing the video and extract health status information from the video. When an input for selecting a health diagnosis function from the user is detected during video playback, the device 100 determines whether a face image acquisition condition is satisfied for video frames included in a video being reproduced, extracts a face image, and After extracting the state information, the health state information may be extracted and provided.

According to another embodiment, the device 100 may obtain face state information and health state information using a video. For example, the device 100 may extract facial state information, such as movement of a facial muscle, a degree of blinking of an eye, and flexibility of a neck, using the video. In addition, the device 100 may determine the user's condition by using the movement of the facial muscles, the degree of blinking of the eyes, the flexibility of the neck, and the like.

27 is a diagram illustrating a process in which the device 100 allocates an identifier to a face image, according to an embodiment.

According to the present embodiment, when a face image is input, identification information on the input face image may be determined by comparing information on the face of the registered identifiers with the input face image. For example, the device 100 may store a representative face image for each identifier or facial feature point information for each identifier. In this case, the device 100 may determine which identifier corresponds to the input face image by comparing the input face image with the representative face image or facial feature point information stored in the device 100 .

According to the present embodiment, classification information and an identifier for the face image may be stored together with the face image. The classification information is information indicating that the face image corresponds to a face image used for diagnosing health state information. Also, according to an embodiment, additional information may be stored together with the face image and the identifier.

28 is a diagram illustrating a method for the device 100 to record information indicating that the health state information is an acquired image in an image file, according to an embodiment.

According to the present embodiment, the device 100 may record at least one of classification information, an identifier, and additional information or a combination thereof as a pixel value in a visible form in the face image. For example, as shown in FIG. 28 , a mark 2810 “HEALTH CARE” may be inserted as a pixel value on the face image. What is recorded as a pixel value in a visible form may be a character, an icon, a barcode, a quick response (QR) code, or the like. According to the present embodiment, there is an effect that the user can intuitively recognize that the corresponding image is a face image for health diagnosis. In addition, by inserting an icon, barcode, QR code, etc. into the image, it is possible to provide a function such as requesting more detailed face state information or health state information or entering into a health diagnosis mode using them.

According to an embodiment, the device 100 may record at least one or a combination of the classification information, the identifier, and the additional information as a pixel value in the face image in a non-visible form. For example, a code or a mark of a face image used for a health diagnosis function may be recorded in the face image as a pixel value in an invisible form. What is recorded as a pixel value in an invisible form may be a character, an icon, a barcode, a quick response (QR) code, or the like. Information in invisible form can be read using an algorithm for reading it.

According to the present embodiment in which at least one or a combination of classification information, identifier, and additional information is recorded as a pixel value in a visible or invisible form, it is possible to store predetermined information in the face image without additional storage space. have. In addition, according to the present embodiment, there is an effect that classification information, identifiers, additional information, etc. can be stored regardless of the file format. According to the present embodiment, it is also possible to record information or a link to a storage path in which the classification information, the identifier, and the additional information are stored, without recording the classification information, the identifier, and the additional information themselves as pixel values.

According to an embodiment, the device 100 may store at least one or a combination thereof, such as the classification information, the identifier, and the additional information, in the header part of the file of the face image. For example, when the image file is stored in an Exchangeable Image File Format (Exif) format, at least one or a combination thereof, such as the classification information, an identifier, and additional information, may be stored in a header part of the Exif file.

29 is a diagram illustrating a method in which the device 100 stores a face image according to an exemplary embodiment.

According to an embodiment, the device 100 may store and manage at least one or a combination thereof as separate data, such as classification information, identifiers, and additional information for files of face images used for a health diagnosis function. For example, the device 100 may store at least one or a combination thereof, such as classification information, identifiers, and additional information for files of face images, as separate management files. According to an embodiment, the management file may be stored in a storage space such as a photo album or gallery in which the face image is stored. According to another embodiment, the management file may be stored in a storage space for an application providing the health management function. As shown in FIG. 29 , the management file may store information such as a storage path and identifier of each file, respectively.

30 is a flowchart illustrating a method in which the device 100 normalizes a face image, according to an embodiment.

In operation S3010, the device 100 may normalize the face image.

Normalization may mean adjusting an attribute value of a face image based on a preset standard. The attribute value of the face image may include the size of the face image, the brightness of the face image, and a color tone change value due to the influence of lighting.

Since a photographing environment or photographing method is different for each input image, an attribute value may be different for each face image obtained from the input image. Accordingly, the device 100 may normalize the face image to obtain health state information for the same user under the same condition.

In addition, since the input images are photographed under different photographing conditions, face state information may be erroneously extracted depending on the photographing conditions, and thus the accuracy of the health state information may be deteriorated. In order to maximally exclude a variable according to the shooting condition from the face image, the device 100 may correct the face image to improve the accuracy of the acquired health state information.

According to an embodiment, the device 100 may correct the face image according to a shooting condition. For example, the brightness of the face image may be adjusted by using information on illuminance at the time of photographing. The information on the illuminance may be detected by an illuminance sensor provided in the electronic device. As another example, the white balance of the face image may be adjusted by using information about the white balance at the time of photographing.

According to an embodiment, the device 100 may correct the color of the face image based on the color of a specific organ or region in the face image. For example, since the color of the neck in the face image does not change significantly depending on the health condition, the color of the face image may be corrected based on the color of the neck. In this case, neck color information for the user corresponding to the face image is stored in advance, and when the user's face image is obtained, the color of the face image is corrected based on the previously stored neck color information. The color correction of the face image may correct the color of the entire face image so that the color of the neck in the face image is corrected with the color of the neck stored in advance.

According to an embodiment, when the red-eye phenomenon appears in the face image, the device 100 may perform red-eye correction.

According to an exemplary embodiment, when shake occurs in the face image, the device 100 may perform shake correction.

According to an embodiment of the present disclosure, when entering the health examination mode and photographing a face image, in the photographing step, a request may be made to photograph a white subject in order to obtain information on white balance. The white subject may be, for example, white paper.

The method of correcting the face image according to the shooting conditions, the method of correcting the color of the face image based on the color of a specific organ or region, the method of red-eye correction, and the method of stabilizing the shake are some of these methods depending on the embodiment. Only or all of them can be used.

When correction is performed on the face image, the device 100 may extract face state information from the corrected face image, obtain health state information, and then provide health state information.

31 is a diagram illustrating a method in which the device 100 normalizes a size of a face image, according to an embodiment.

Referring to FIG. 31 , the device 100 may obtain a face image from an input image, and normalize the size of the obtained face image to a reference size.

The device 100 may store a reference size of the face image. For example, in the device 100 , the reference size of the face image may be set to a resolution of 200 * 300 .

When the size of the face image is smaller than the reference image, the device 100 may increase the face image to the reference size by applying an upsampling or interpolation method to pixel data of the face image. Also, when the size of the face image is larger than the reference image, the device 100 may reduce the size of the face image to the reference size by subsampling or downsampling pixel data of the face image.

As the size of the face image is normalized to the reference size, the size of the face region in the face image may also be normalized. Also, since the face image is an image in which only the face region is detected from the input image, the size of the face image may be regarded as the size of the face region. As the size of the face image is adjusted to the reference size, face state information displayed on the user's face may be extracted from the face image under the same face size condition for the same user.

32 is a flowchart of a method for the device 100 to adjust an effect of a color temperature of light from a face image, according to an embodiment.

Even in the face image of the same user, when the types of illumination illuminating the face are different, the color of the face in the image may be different. The color temperature may refer to a numerical value expressing the color of the light of an object by using the temperature of a black body that emits the same light as that of the object when the object is lit.

For example, a fluorescent lamp having a color temperature of 4000K may make a white object slightly reddish, sunlight having a color temperature of 5200K may represent a white object as it is white, and a shade having a color temperature of 7000K may make a white object slightly blue.

Accordingly, even the same object may have a red color under a fluorescent lamp but a blue color under the sun. Accordingly, even for the same user's face, the color of the face in the face image may be measured differently depending on the illumination that illuminates the face during photographing. The device 100 may obtain the original color tone value of the user's face by excluding the effect of the color temperature of light from the face image.

In operation S3210 , the device 100 may acquire a color temperature of light illuminating the user's face when the user's face is photographed.

For example, in the input image, the color temperature of light obtained from a color temperature sensor during photographing may be recorded as metadata. The device 100 may obtain the color temperature of light recorded in the input image from metadata of the input image.

Also, the device 100 may extract the color temperature of light at the point in time when the input image is captured from the input image itself. For example, the device 100 may determine the brightest area in the input image as the area in which the original white color is photographed. As the brightest region is detected in the input image, the device 100 may detect an average color tone value of the detected region. In addition, the device 100 may acquire the color temperature of the light at the time of photographing, based on the detected color tone value. The color temperature of the light corresponding to the hue value may be previously stored in the device.

In operation S3220, the device 100 may adjust the color tone value of the face image based on the obtained color temperature.

As the color temperature of the light is obtained, the device 100 may obtain a hue value indicated when the light having the obtained color temperature illuminates white, based on pre-stored information about the color temperature. Since the hue values may be determined by the gains of the red, green, and blue colors, the device 100 determines the gains of the red, green, and blue colors corresponding to the hue values represented when the light having the obtained color temperature illuminates white. can

The device 100 may exclude the effect of the color temperature of light when photographing the face image from the face image based on the determined gains of the red, green, and blue colors. For example, the device 100 may adjust gains of the determined red, green, and blue colors so that the original white is expressed as white. In addition, the device 100 may exclude the effect of the color temperature of light by adjusting the gain of the red, green, and blue colors for all pixel values of the face image by the adjusted gain.

Accordingly, the original color tone value of the user's face can be obtained by excluding the effect of the color temperature of light from the face image. According to an embodiment, this method may be referred to as a white balance adjustment method.

33 is a flowchart illustrating a method of extracting, by the device 100, face state information indicating a state of a user's face from a normalized face image of the user, according to an embodiment.

In operation S3310, the device 100 may extract face state information indicating the state of the user's face from the normalized face image of the user.

The face state information may refer to information regarding whether the user's face exhibits symptoms. The facial state information may include, for example, the color of the face, the number and size of blemishes or acne, whether the eyes are red, the color of the eyes, the size of the pupil, the movement of the pupil, the size of the face, the shape of the face outline, the color of the lips, the lips It may include whether or not the skin is split, the position of each organ (eyes, nose, mouth, ears, eyebrows) of the face, the color of the hair, the gloss of the hair, whether the hair is split, the movement of the facial muscles, and the like.

As the face image is normalized, the device 100 may determine a region from which face state information is extracted from the normalized face image. For example, the device 100 may determine positions such as an under-eye region, a nose region, and an area around the mouth in the face image.

As the region from which the facial state information is to be extracted is determined, the device 100 may extract facial state information from each region. For example, the device 100 may determine whether the area under the eyes is inflated. Also, the device 100 may determine whether the color of the area under the eyes has changed to a darker color than the previous color.

A method of determining a location of a region from which face state information is to be extracted corresponding to a region from which face state information is to be extracted, a symptom to be confirmed, and a method of identifying a symptom may be stored in the device 100 . Accordingly, the device 100 may determine whether a symptom appears from the region from which each face state information is to be extracted.

When an abnormality occurs in the human body, the symptoms appearing on the face may appear on a specific part of the face. For example, if you have hyperthyroidism or an allergic disease, you may have more fat under your eyes than before. Also, for example, when allergic rhinitis occurs, dark circles under the eyes may form.

The device 100 may store symptom information corresponding to a specific part of the face. For example, the device 100 may store, as symptom information, whether the area under the eyes is swollen or the color under the eyes has turned black in response to the area under the eyes.

Accordingly, the device 100 determines an area under the eyes among the entire face area as an area from which face state information is to be extracted, and from the area under the eyes, whether the area under the eyes is inflated or whether the color under the eyes has changed to black. can be extracted as face state information.

34 is a flowchart illustrating a method for the device 100 to extract face state information appearing on a face from a normalized face image, according to an embodiment.

In operation S3410 , the device 100 may determine a position of a face component in the face region in the face image.

Facial components may include eyes, nose and mouth, and the like. The device 100 may determine the position of the face component in the face image in various ways.

For example, the device 100 binarizes the face image by using a characteristic in which the brightness of the eyes, eyebrows, and mouth is darker than that of the facial skin in the human face, and sets the positions of the dark areas to the eyes, eyebrows, mouth, etc. can be determined by the location of

Also, the device 100 may extract a skin color region from the face image and determine positions of eyes, eyebrows, and mouth in the extracted region.

In addition, the device 100 uses an Active Appearance Models (AAM) method for determining the positions of facial components based on the pattern of the face, since the positions of the eyes, eyebrows, nose, and mouth on the face represent a certain pattern. to determine the position of the face component.

In operation S3420, the device 100 may determine the location of the diagnosis area, which is the area from which the face state information is to be extracted, based on the location of the face component.

The diagnosis area may be pre-stored in the device 100 . For example, in the device 100 , a forehead, an area under the eyes, a nose, a periphery of a mouth, and a chin area may be set as the diagnosis area.

Also, a method of determining the location of the diagnostic area corresponding to the diagnostic area may also be previously stored in the device 100 . For example, the nose region may be determined as a triangle region connecting both ends of the nasal bridge and the starting point of the nose. A method of determining the location of the diagnostic area will be described in detail below with reference to FIG. 35(b).

In operation S3430, the device 100 may extract face state information from the determined diagnosis area.

The extraction of the face state information may be performed by analyzing the face image. For example, the device 100 may extract the face state information using color information of a face region of the face image, a face recognition algorithm, a face expression recognition algorithm, and the like. The type of facial state information that can be extracted in response to the diagnosis area and a method of extracting the facial state information may be pre-stored in the device 100 .

The device 100 may extract information such as the color of the face, the number of blemishes or acne, the color of the eyes, and the color of the lips, by using the color information of the face region. Also, for example, the device 100 uses a face recognition algorithm or a facial expression recognition algorithm to determine whether the eyes are red, the size of the pupil, the movement of the pupil, the size of the face, the shape of the face outline, and whether the lips are cracked. , information such as the position of each organ of the face and the movement of facial muscles can be extracted. In addition, various algorithms and methods may be used to extract the face state information.

The skin condition may be determined using the color of the face, the number and size of blemishes or acne, the shape of the facial contour, and the like. For example, if it is determined that the color of the face is dark, the number of blemishes or acne is large, the number of blemishes or acne is large, and the outline of the face is sagging, it may be determined that the skin condition is bad.

The state of the eye may be determined using whether the eye is red, the color of the eye, the size of the pupil, the movement of the pupil, and the like. For example, the device 100 may determine that the condition of the eyes is bad when the eyes are red, the whites of the eyes are dark, and the reaction rate of the pupils is slow. Also, the device 100 may determine a case in which the liver is not good, a case in which an eye disease occurs, etc. according to the color of the white of the eye.

Whether or not the weight has changed may be determined using the face size and the shape of the face outline. For example, when the size of the face increases and the contour of the cheek portion in the face contour moves outward compared to a previously captured face image, it may be determined that the body weight has increased.

The state of the lips may be determined using the color of the lips and whether the lips are cracked. For example, if the color of the lips is red and saturated and there is no cracking of the lips, the condition of the lips is judged to be good. may be considered bad. In addition, the device 100 may determine the abnormality of the body according to whether the lip color is dark red, blue, flesh, or low saturation.

The state of the hair may be determined using the color of the hair, the gloss of the hair, whether the hair is split or not. For example, the device 100 may determine that if the color of the hair is dark, the hair is shiny, and the hair is not split, the condition of the hair is good, otherwise, the condition of the hair is not good. The luster of the hair may be determined by detecting a bright color region appearing in a curved region of the hair among the hair regions.

The physical condition may be determined by synthesizing the skin condition, the eye condition, the lips condition, the hair condition, the movement of the facial muscles, and the like. For example, the device 100 may determine that the better the condition of the eyes, the lips, the hair, and the movement of the facial muscles, the better.

35 is a diagram for describing a method of determining a location of a diagnosis area by the device 100, according to an exemplary embodiment.

Referring to FIG. 35A , the device 100 may determine a position of a face component in a face image.

For example, the device 100 includes both ends between the eyes (3420 to 3426), between the eyebrows (3410 to 3416), the nostrils (3430 to 3434), and both ends of the lips (3440 to 3442). ) and the nose end point 3434 and the nose start point 3436 may be determined.

Referring to FIG. 35B , the device 100 may determine the location of the diagnosis area based on the location of the face component.

For example, in the device 100 , a forehead, an area under the eyes, a nose, a periphery of a mouth, and a chin area may be set as the diagnosis area.

Also, a method for determining the location of each diagnostic area may be set in the device 100 . For example, the nose region may be determined as a triangular leather region 3450 connecting both end points 3430 to 3434 of the nasal bridge and the nose start point 3436 . Also, the upper boundary of the mouth peripheral region 3460 may be determined as the Y-axis coordinate of the midpoint 3446 between the nose tip 3434 and the lip center point 3444 .

36 is a diagram illustrating a method for the device 100 to extract face state information from a diagnosis area, according to an exemplary embodiment.

Referring to FIG. 36 , the device 100 may extract face state information corresponding to the diagnosis region from the diagnosis region.

In the device 100 , corresponding to the mouth area, the color of the lips and whether pimples have occurred in the area around the mouth may be set as symptoms.

Referring to FIG. 36A , the device 100 may determine a position of a lip region and extract a lip hue value or a lip saturation value from the lip region as face state information. For example, the device 100 may extract the color tone value of the lips by averaging the color tone values of pixels constituting the lip region.

Also, the device 100 may determine the position of the area around the mouth and extract the degree of skin trouble from the area around the mouth as face state information.

Referring to FIG. 36B , the device 100 may use various image processing methods to detect pimples appearing on the skin.

For example, the device 100 may increase the contrast of the mouth region to highlight the pimple in contrast to the skin. In addition, since the area where pimples or skin troubles occur appears darker than normal skin, the device 100 may binarize the mouth area and determine the location of the site where pimples or skin troubles occur in the binarized mouth area. In addition, since the portion where the pimple or skin trouble occurs is red, the position of the pixel representing the red color may be determined as the location of the portion where the pimple or skin trouble occurs.

The device 100 may determine whether the pimples or skin troubles increase or decrease according to a change in the area of the pimples or skin troubles.

37 is a flowchart illustrating a method for the device 100 to obtain health state information related to a user's health based on facial state information, according to an embodiment.

In operation S3710, the device 100 may obtain health state information indicating the health state of the user by using the face state information.

The device 100 may obtain health state information indicating the user's health state by using the face state information extracted from the diagnosis area in the face image.

The health state information may include information about a user's disease, a function-degraded organ, or a user's condition predicted from the facial state information. Health state information indicating the user's health state may be stored in the device 100 .

The device 100 may obtain health state information from the face state information in consideration of the photographing situation information obtained when photographing the face. The photographing situation information may include a photographing time, a photographing location, a user's activity during photographing, and user's biometric information obtained during photographing.

In addition, the device 100 may obtain health state information from the facial state information in consideration of the user's physical condition such as height, weight, age, and gender, or a medical history such as a current disease or a past disease.

38 is a diagram illustrating a method of extracting, by the device 100, health state information of a user based on facial state information extracted from a face image, according to an embodiment.

Referring to FIG. 38A , the device 100 may store information on a predicted disease according to a face color.

Accordingly, the device 100 may determine that there is an abnormality in the liver when the face is dark blue, the kidneys when the face is black, the lungs when the face is white, and the heart when the face is red.

Referring to (b) of FIG. 38 , the device 100 may store information on a cause of an expected disease or symptom in response to a symptom extracted from a diagnosis area.

Accordingly, the device 100 may determine the cause of the expected disease or symptom based on the symptoms extracted from each diagnosis area. For example, when the facial state information extracted from the eyes is in a state in which the eyes are congested, the device 100 may determine that there is an abnormality in the user's liver and heart.

39 is a flowchart illustrating a method for the device 100 to obtain health state information from facial state information in consideration of photographing situation information obtained when photographing a face, according to an embodiment.

In step S3910 , the device 100 may obtain photographing situation information obtained when photographing a face.

The shooting situation information may be stored in a file of the input image in the form of metadata of the input image. Also, when the device 100 generates the input image, the device 100 may store the shooting situation information in response to the identification information of the input image.

The photographing situation information may include a photographing time, a photographing location, a user's activity during photographing, and user's biometric information obtained during photographing.

In operation S3920, the device 100 may obtain health state information related to the user's health by considering the face state information and the photographing context information together.

For example, when the face photographing time is 3 am, the device 100 may determine that the condition of the face extracted from the face image is more serious than a preset standard, but may be determined as deterioration due to temporary overwork. In this case, when the device 100 obtains the health state information from the facial state information, it may be determined by reducing the degree of the extracted facial state.

Also, for example, when the photographing time is dawn, the device 100 may determine that the user's face is not made-up, and may acquire health state information by weighting the user's face as a non-make-up state.

Also, for example, when the user's pulse rate is a higher value than usual, the device 100 may determine that the user took the picture immediately after exercising, and may exclude the user's face color and the like from the face state information.

Also, for example, when the amount of sleep of the user is 2 hours when photographing the face, the device 100 may determine that the degree of the facial condition extracted from the face image is worse than the reference, but may be determined as aggravation due to temporary overwork. In this case, when the device 100 obtains the health state information from the facial state information, it may be determined by reducing the degree of the extracted facial state.

In addition, for example, when the location of the face photographing is a bar, and the symptom extracted from the face image is redness, the device 100 determines that it is redness due to temporary drinking, and excludes the symptom of flushing of the face extracted from the face image. , health status information can be obtained.

Also, for example, when the user's activity level is high, the heart rate is high, and the symptom extracted from the face image is redness when photographing the face, the device 100 determines that the face is flushed due to a temporary motion, and Except for the extracted facial flushing symptom, health status information may be obtained.

40 is a diagram illustrating a method of displaying, by the device 100, photographing situation information obtained when capturing an image together with health state information of a user, according to an exemplary embodiment.

Referring to FIG. 40 , the device 100 may display a face image on a screen and display health state information of the user extracted from the displayed face image. Also, along with the user's health state information, shooting situation information taken into consideration when acquiring the health state information from the face image may be displayed.

When extracting health state information from a face image, the device 100 may consider a shooting location, a shooting time, a distance from the camera to the face, a heart rate during shooting, an amount of activity during shooting, an amount of sleep during shooting, and the like.

Also, the device 100 may display an image processing method applied to the original input image in order to extract the user's health state information from the original image. For example, the device 100 may display an enlarged image ratio and whether or not to adjust the illuminance.

41 is a diagram illustrating a method of providing, by the device 100, a function of selecting photographing condition information to be considered by the device 100 from among a plurality of photographing condition information when the device 100 obtains health condition information from facial condition information, according to an embodiment. It is a drawing.

The device 100 may display a user interface for selecting photographing context information to be considered when acquiring health state information from face state information. Upon receiving a user input for selecting at least one of the photographing situation information displayed on the screen, the device 100 may obtain health state information from the face state information based on the selected photographing situation information.

Also, the device 100 may display a user interface for selecting an image processing method to be applied to a face image when obtaining health state information from the face state information. Upon receiving a user input for selecting at least one of the image processing methods displayed on the screen, the device 100 corrects the face image based on the selected image processing method, and obtains face state information from the corrected face image. may be

Referring to FIG. 41A , the device 100 may extract skin troubles around the mouth from the face image as face state information. Also, the device 100 may determine the color of the face in the face image as black. The device 100 may acquire health state information as the user's kidney function is deteriorated based on skin troubles around the mouth and dark face color.

Referring to FIG. 41B , the device 100 may display a user interface for adjusting the color temperature of light in a face image, and a user interface for considering a heart rate, an activity amount, and a sleep amount when obtaining health state information.

Upon receiving a user input for adjusting the color temperature of light and selecting a heart rate, an activity amount, and an amount of sleep, the device 100 may adjust a hue value in the face image in consideration of the color temperature of the light.

As the color tone value in the face image is adjusted, the device 100 may change the face color in the face image from black to yellow. In addition, in consideration of the yellow face color and the amount of sleep of the user at the time of photographing, the device 100 provides the health status information of the user as the skin trouble around the mouth is due to the imbalance of hormones due to lack of sleep, not the deterioration of kidney function. can be obtained

42 is a flowchart illustrating a method for the device 100 to obtain health state information from facial state information based on a time point at which a face image is captured, according to an embodiment.

In operation S4210, the device 100 may obtain the date the face was captured from the file of the face image.

For example, in the face image, the date the face was captured may be recorded in the form of metadata. Accordingly, the device 100 may obtain the date the face was captured from the metadata of the face image.

In operation S4220 , the device 100 may obtain health state information from the facial state information in consideration of health state information related to a face image captured before the acquired date.

The device 100 may extract face state information from a plurality of pre-obtained face images of the user. As the facial state information is extracted from the plurality of pre-obtained face images, the device 100 may store the health state information obtained from the extracted facial state information in correspondence with the capture dates of the plurality of face images. Accordingly, the device 100 may acquire the health status of the user according to the date.

The device 100 may obtain health status information of the user related to the face image captured before the capturing date of the face image, based on the capturing date obtained in step S4210. The device 100 may acquire health state information prior to the date of capturing the face image, based on the user's health state according to the date.

Accordingly, the device 100 may acquire a disease or abnormality that the user experienced before the time the face image was captured.

As health state information before the photographing date of the face image is obtained, the device 100 obtains the user's health state information from the facial state information of the face image in consideration of the health state information before the photographing date of the face image can do.

The device 100 may obtain health state information from the facial state information based on a disease or abnormality that the user has before the point in time when the face image is captured.

When one symptom extracted from the face image can be determined to be a plurality of diseases or abnormal findings, the device 100 returns to the health status information of the user extracted from face images captured before the input image was captured. Based on the facial state information, health state information may be determined.

For example, a disease or abnormality extracted from the user's face images taken during the month before the face image was taken is gastroenteritis, and gastroenteritis, overwork or drinking is extracted from one facial condition information extracted from the face image. If possible, the device 100 may determine the cause of the symptom as gastroenteritis.

In addition, when the frequency of the acquired disease or abnormality appearing before the time of photographing is less than the reference number, the device 100 may not determine the acquired disease or abnormality as the user's health status information.

Also, the device 100 may determine whether a specific disease has worsened or improved based on the user's health state information extracted from the face images captured before the time the face image is captured.

Health state information extracted from temporally continuous face images may be similar to each other. For example, health state information extracted from the face of the same person photographed for a month may indicate similar diseases or abnormalities. Accordingly, the device 100 may obtain more accurate health state information from the facial state information by using the disease or abnormality that the user has before the time of capturing the face image.

43 is a diagram for describing a process in which the device 100 acquires health state information according to an embodiment.

According to an embodiment, the device 100 may obtain health state information by comparing the usual state with the current state. For example, the device 100 may extract face state information from face images captured and stored in the past of a specific user, calculate an average value of the extracted face state information, and calculate the user's usual state.

In addition, the device 100 may extract face state information from the captured image of the user to calculate the current state of the user. The usual state and the current state may be calculated for each type of face state information. The type of the face state information may include, for example, the color of the face, the number and size of blemishes or acne, whether the eyes are red, the color of the eyes, the size of the pupil, the movement of the pupil, the size of the face, the shape of the face outline, and the shape of the lips. It may include color, whether the lips are cracked, the position of each organ (eyes, nose, mouth, ear, eyebrows) of the face, hair color, hair gloss, whether the hair is cracked, movement of facial muscles, and the like.

The device 100 may evaluate a health state for each type of face state information according to a comparison result between the usual state and the current state, or may evaluate the health state by synthesizing the comparison results for the face state information.

According to another embodiment, the device 100 may obtain the health state information by calculating a difference between the pre-stored normal state and the current state. For example, a normal state for each face state information may be defined according to the user's age, gender, height, weight, time zone, season, and the like.

The device 100 may calculate the difference between the normal state and the current state for each type of the face state information. The device 100 may evaluate a health state for each type of face state information according to the comparison result between the normal state and the current state, or may evaluate the health state by synthesizing the comparison results for the face state information.

According to an embodiment, the operation of obtaining and providing the health state information may be performed while a function such as a photo album or a gallery of the device 100 is executed. For example, when a user selects images to be used as a face image among images stored in a photo album and executes a health diagnosis function, health state information may be calculated and provided from the selected face images.

As another example, when the user executes the health diagnosis function while the photo album is running, a face image satisfying the face image acquisition condition is extracted from among the stored images, and health status information is calculated using the extracted face images. , can be provided. In this case, by classifying the face images for each user, health status information for each user may be provided.

As another example, the device 100 may provide a list of users included in the face images, and obtain and provide health state information for a user selected by the user.

According to another embodiment of the present invention, the operation of obtaining and providing the health state information may be performed while a specific application is executed in the device 100 . For example, the user may obtain health state information by performing an application of a health diagnosis function of the device 100 .

According to an embodiment of the present invention, the health state information may be calculated by the device 100 . In this case, the device 100 may use a predetermined algorithm for obtaining health state information.

When the health state information is calculated, the device 100 provides the calculated health state information to the user. The method of providing the health state information to the user may be performed in various ways, such as displaying on the device 100, outputting the sound, outputting it to an external device, and storing it in a memory.

The health state information device 100 may display the face image and the health state information together on one screen, or display only the health state information, according to an embodiment.

44 is a diagram illustrating a process in which the device 100 acquires health state information using a service server according to another embodiment.

According to the present embodiment, the process of acquiring health state information may be performed by the service server 2100 communicating with the device 100 . The service server 2100 is, for example, a server that provides a service for obtaining health state information, and may be a medical service server, an application server, a website server, or the like.

First, the device 100 acquires a face image (S4410). Next, the device 100 transmits the acquired face image to the service server 2100 (S4420). The device 100 may access the service server 2100 through the Internet, a telephone network, a wireless communication network, or the like. For example, when the device 100 is a portable communication terminal, the device 100 may include Wi-Fi, third generation mobile communication, fourth generation mobile communication, and long term evolution (LTE). , long term evolution-advanced (LTE-A), etc. may be used to access the network and to the service server 2100 . In addition, according to an embodiment, the device 100 transmits a face image to the service server 2100 within the photo album function, transmits a face image to the service server 2100 within an application, or provides a face image service through an SMS message. The face image is transferred to the service server 2100 by various methods, such as transmitting the face image to the server 2100, transmitting the face image to the service server 2100 using a messenger application, or transmitting the face image to the service server 2100 via email. ) can be transmitted.

According to an embodiment, the device 100 may transmit an image file of the obtained face image to the service server 2100 . In this case, the service server 2100 may extract face identification information included in the image file. The service server 2100 may retain facial feature point information corresponding to each piece of identification information, and extract identification information of the face image by using the retained facial feature point information. Also, according to an embodiment, the service server 2100 may search for information stored in the service server 2100 for the corresponding identification information by using the extracted identification information. For example, the service server 2100 may retain facial state information, health state information, personal information, medical record information, other facial images, etc. for the corresponding identification information, and retrieve these information using the identification information. .

According to another embodiment, the device 100 may transmit an image file and identification information of the obtained face image to the service server 2100 . The identification information may include, for example, at least one of an ID, a mobile phone number, an IP address, and a MAC address, or a combination thereof. The service server 2100 may use the identification information to search for facial state information, health state information, personal information, medical record information, other facial images, etc. that correspond to the identification information and are stored in the service server 2100 . .

According to another embodiment, the device 100 may transmit an image file of the obtained face image, identification information, and facial state information and health state information for the corresponding identification information to the service server 2100 . According to another embodiment, the device 100 may transmit personal information and additional information on the identification information together with the image file to the service server 2100 . The personal information may be, for example, information such as a name, contact information, and occupation. The additional information may be, for example, medical records, medical history, height, weight, age, blood pressure, blood sugar, waist circumference, hip circumference, chest circumference, and the like.

When the service server 2100 receives the face image (S4420), the service server 2100 extracts face state information from the corresponding face image (S4430). The extraction of the face state information may be performed by analyzing the face image. For example, the service server 2100 may extract the face state information using color information of a face region, a face recognition algorithm, a face expression recognition algorithm, and the like. For example, the service server 2100 may extract information such as the color of the face, the number of blemishes or acne, the color of the eyes, the color of the lips, etc. by using the color information of the face region. In addition, for example, the service server 2100 uses a face recognition algorithm or a facial expression recognition algorithm, etc., whether the eyes are red, the size of the pupil, the movement of the pupil, the size of the face, the shape of the face outline, and the lips are split. It is possible to extract information such as whether or not there is, the position of each organ of the face, and the movement of the facial muscles. In addition, various algorithms and methods may be used to extract the face state information.

Next, the service server 2100 extracts health state information by using the face state information (S4440). For example, the service server 2100 may extract health state information in a manner of determining skin state, eye state, weight change, lip state, hair state, physical condition, etc. from the electronic device described above. .

The facial condition information extraction method and the health condition information extraction method of the service server 2100 may be updated according to an administrator or medical staff of the corresponding server.

According to an embodiment, the service server 2100 may provide facial state information and/or health state information directly extracted from a facial image by a medical professional. In this case, the face state information and/or health state information may be provided to the device 100 several hours or days after the face image is transmitted.

When the health state information is calculated by the service server 2100 ( S4440 ), the service server 2100 transmits the face state information and/or health state information to the device 100 ( S4450 ). The face state information and/or health state information may be transmitted in various ways, such as an application message, an SMS message, a message of a messenger application, or an email (S4450).

The device 100 provides the face state information and/or health state information transmitted from the service server 2100 to the user (S4460). For example, the device 100 may display health state information on the screen of the device 100 .

45 is a flowchart illustrating a method for the device 100 to display health state information of a user, according to an embodiment.

In step S4510, the device 100 may display the user's health status information.

The device 100 may display health state information obtained by using the face state information. In addition, the device 100 provides not only health status information, but also information about the symptoms themselves extracted from the face image, information about the causes of the symptoms, information about other symptoms that may appear along with the extracted symptoms, and information needed to improve the symptoms. Information about the action and the like can be displayed.

46 is a diagram illustrating a method of providing a user interface for providing, by the device 100, health state information calculated from a displayed user's face when displaying an image stored in the device 100, according to an embodiment; to be.

Referring to FIG. 46 , the device 100 may display an image selected by the user from among the images stored in the device 100 .

Upon receiving a user input for selecting one of the stored images, the device 100 may determine whether the selected image is an image from which face state information is extracted. When the selected image is an image from which face state information is extracted, the device 100 may display a user interface 4610 for providing health state information corresponding to the face image.

Upon receiving a user input for selecting the button 4610 for providing health state information, the device 100 may display the health state information stored in response to the selected image.

47 is a diagram illustrating a method in which the device 100 displays health state information on a displayed image, according to an embodiment.

Referring to FIG. 47 , the device 100 may display health state information on a displayed image.

The device 100 may obtain identification information of the displayed image. The device 100 may acquire stored health state information in response to the acquired identification information. The device 100 may store the predicted disease and the location of the diagnosis area from which the predicted disease is extracted in response to the identification information of the image. Accordingly, the device 100 may display a phrase 4720 indicating a predicted disease and an image 4710 indicating a diagnosis region from which the predicted disease is extracted, on the image displayed on the screen.

Also, the device 100 may provide a user interface 4730 for not extracting health state information from an image displayed on the screen. Upon receiving a user input for selecting the user interface 4730 for not extracting health state information from the image displayed on the screen, the device 100 may delete the health state information stored in response to the selected image.

Accordingly, when the user's health status information is distorted because the user's face is not clearly photographed due to an external environment during shooting, the distorted health status information may be deleted.

48A is a diagram illustrating a method of providing, by the device 100, a user interface for selecting a person to be displayed on a screen from among health status information about a plurality of persons, according to an exemplary embodiment.

Referring to FIG. 48A , the device 100 may display a user interface 4810 for selecting a person to be displayed on the screen from among health status information of a plurality of persons. The device 100 may display a face image from a face image. It is possible to extract health status information about the plurality of people who have appeared. Also, the device 100 may store identification information of the input image from which the facial state information is extracted and health state information obtained from the facial state information in correspondence to the identification information of the person. Upon receiving a user input for selecting one of a plurality of persons, the device 100 may display an input image and health state information of the selected person.

48B is a diagram illustrating a method for the device 100 to display an input image of a person selected by a user and health state information corresponding to the input image, according to an embodiment.

Referring to FIG. 48B , the device 100 may display an input image of a person selected by the user in FIG. 48B and health state information corresponding to the input image.

The device 100 may display the input image 4820 according to the time sequence in which the input image was captured. Also, the device 100 may display information 4830 on the date when the input image 4820 is generated together with the input image 4820 . Also, the device 100 may display health state information 4840 corresponding to each input image 4820 .

Accordingly, the user can check the health status information of the selected person in chronological order.

49A to 49C are diagrams illustrating a method for the device 100 to provide health state information for a period or disease selected by a user, according to an exemplary embodiment.

Referring to FIG. 49A , the device 100 may display a user interface 4910 for selecting a period. The user interface 4910 for selecting a period may be a user interface for selecting a unit period from the present time to the past.

Upon receiving a user input for selecting a unit period from the current time to the past, the device 100 may display health state information extracted from an image captured during the selected unit period.

Referring to FIG. 49B , the device 100 may display a user interface 4920 for selecting a disease. The device 100 may display a user's disease related to an image stored in the device 100 . Also, from among the images stored in the device 100 , a user's disease related to an image captured during a unit period selected by the user may be displayed.

Referring to FIG. 49C , upon receiving a user input for selecting a period and a user input for selecting a disease, the device 100 may display input images related to diseases occurring during the selected period in chronological order.

50A is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.

According to an embodiment, the health state information calculated by the device 100 may be displayed on the screen of the device 100 . Also, when there is a change in the health state, the device 100 may provide information on the change in health state information to the user. For example, as shown in FIG. 50 , a message indicating that the skin condition has deteriorated due to an increase in the number of blemishes may be displayed on the screen.

50B is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.

According to an embodiment, the health state information may be provided in a form of notifying a change in the health state information over time. When a face image is extracted using a captured image, the time may be known using the capture time or the current time. When a face image is extracted using a stored image, the time can be known by using information about a photographing date and time stored in an image file of the stored image. For example, as shown in FIG. 50B , it is possible to show how the number of blemishes changes according to the date, thereby indicating the skin condition. As another example, the device 100 may provide blood pressure, skin color, eye condition, skin condition change, and the like over time. According to the present embodiment, there is an effect of allowing a user to easily grasp a change in a health state over time.

51 is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.

According to an embodiment, the device 100 may display health state information together with a face image. For example, as shown in FIG. 51 , a change in the number of blemishes a week ago and today may be displayed using a face image. According to the present embodiment, there is an effect of notifying a change in a health state intuitively and visually.

Also, according to the present embodiment, the device 100 may provide a face image and health state information together by comparing the best state with the current state. For example, a face image representing the best skin condition and a face image representing the current condition may be displayed together. According to this embodiment, there is an effect that the user can intuitively grasp the current state by comparing the current state with the best state.

Also, according to the present embodiment, the device 100 may provide health state information by displaying a reference face image indicating a normal state and a face image indicating the current state of the user even if it is not a face image of the corresponding user. According to the present embodiment, the user can intuitively grasp his or her health condition.

52 is a diagram illustrating a screen for providing health state information according to an exemplary embodiment.

According to an embodiment, the device 100 may provide advice based on health state information to the user. For example, if the user's skin condition is not good, advice to supplement vitamin C may be provided. In this case, in order to induce the user to replenish vitamin C twice a day, a notification to supplement vitamin C twice a day at a predetermined time may be provided in the form of a pop-up message or the like. Also, for example, the device 100 may provide a message or a notification for inducing an exercise to the user at a predetermined time.

In addition, the device 100 may provide information informing the user of necessary food, lifestyle, exercise, etc., based on the user's health state information.

53A and 53B are diagrams illustrating a method for the device 100 to provide health status information of a user in a calendar format, according to an exemplary embodiment.

Referring to FIG. 53A , the device 100 may display a calendar displaying dates for each week or month on one screen. Also, the device 100 may display a user interface 5310 for displaying health state information corresponding to the date.

Referring to FIG. 53B , as the device 100 receives a user input for selecting a user interface 5310 for displaying health state information corresponding to a date, the device 100 displays an area corresponding to each date. may display health state information corresponding to the face image captured on each date among the plurality of face images.

The device 100 may store health state information corresponding to the face image corresponding to the date and identification information of the face image captured on the corresponding date. In this case, the health state information corresponding to each date may be health state information obtained from facial state information extracted from a face image captured on the corresponding date.

In addition, upon receiving the user input for selecting one date, the device 100 provides health status information obtained from the face image 5340 photographed on the selected date and facial state information extracted from the face image photographed on the selected date. 5330 may be displayed together.

54 is a diagram illustrating a method of displaying, by the device 100, health state information of a user when a social network application is executed, according to an embodiment.

Referring to FIG. 54 , the device 100 may display health status information of the user when the social network application is executed.

For example, the device 100 may store the user's health state information corresponding to the face image. The device 100 may store health state information corresponding to the most recently captured user's face image as the user's current health state information. In this case, the user's current health status information may be stored in a predetermined location. Accordingly, the application running on the device 100 may extract the user's current health state information from the predetermined location.

Also, the application executed on the device 100 may display a different image 5410 indicating the user's state according to the user's current health state information. For example, an application running on the device 100 may display an image representing a sick person as an image representing the user when the health status of the user is less than or equal to the standard.

55 shows a block diagram of device 100, according to one embodiment.

As shown in FIG. 55 , according to an embodiment, the device 100 includes a display unit 110 , a sensing unit 190 , a communication unit 130 , an imaging unit 155 , a memory 120 , and a control unit 170 . ) may be included. However, not all illustrated components are essential components. The device 100 may be implemented by more elements than the illustrated elements, and the device 100 may be implemented by fewer elements than that.

The device 100 may acquire an input image in which the user's face appears. For example, the device 100 may acquire an input image through a template photographing in which the user captures the user's face in accordance with a preset face image acquisition condition. In this case, the display 110 may display guide information for photographing a face on the screen according to a preset face image acquisition condition. As the user's face approaches the imaging unit 155 , the device 100 may receive an input image including the user's face through the imaging unit 155 .

The imaging unit imaging unit 155 photoelectrically converts incident light to generate an electrical imaging signal. The imaging unit 155 may include a lens, an iris, and an imaging device. In addition, the imaging unit 155 may be implemented as a mechanical shutter or an electronic shutter method.

According to the present embodiment, the input image may be an image photographed using the imaging unit 155 . The input image may include, for example, at least one of a captured image captured by the imaging unit 155 , a preview image, and a moving image.

According to an embodiment, when a user takes an image by using the imaging unit 155 and the captured image satisfies the face image acquisition condition, the controller 170 may use the captured image as a face image. As another example, when the captured image satisfies the condition for obtaining the face image, a message inquiring to the user whether to use the captured image as a face image for health diagnosis is output, and the captured image is selected according to the user's selection. It can be used as the face image. According to the present embodiment, even when photographing in the normal mode, a face image for a health diagnosis may be collected without a special operation.

According to another embodiment, a face image may be captured in an existing specific mode provided by the device 100 . The specific mode includes, for example, a face authentication mode in which the lock state of the device 100 is released through face authentication, a self-photography mode, a person photographing mode, and the like.

A predetermined shooting parameter may be set in the imaging unit 155 . The predetermined shooting parameter may include, for example, an aperture value, whether a flash is emitted, a white balance condition, and the like.

When the condition for obtaining the face image is satisfied in the preview image of the health diagnosis mode, the controller 170 may automatically capture an image and use the captured image as a face image.

When photographing is performed by the input of a shutter release signal in the health diagnosis mode, the controller 170 determines whether the captured image satisfies the face image acquisition condition, and information on whether the face image acquisition condition is satisfied may be provided to the user. Also, when the captured image satisfies a condition for obtaining a face image, the controller 170 may use the captured image as the face image.

Also, the device 100 may receive an input image from an external network connected to the device 100 through the communication unit 130 .

Upon receiving the input image, the controller 180 may store the input image in the memory 120 . Also, the controller 180 may obtain a face image from the input image.

The controller 180 may obtain a face image from the input image through the image processing unit 185 . The image processing unit 185 may include a user face detection unit 181 , a face image acquisition condition determination unit 182 , and a face normalization unit 183 .

The user face detection unit 181 may detect a human face region from the input image and extract a position of the detected face region. As the position of the face region is detected, the user face detector 181 may extract facial features from the detected face region. Various methods such as Gabor filter or LBP may be used as a method of extracting facial features from the face region.

The user face detection unit 181 may compare facial features extracted from each face region with previously registered facial features of the user, and determine a facial region having a similarity within a preset range as the user's face region.

Upon detecting the user's face region, the face image acquisition condition determining unit 182 may determine whether the input image satisfies a preset face image acquisition condition. For example, the device 100 may determine whether the illuminance when capturing the input image is within a reference range. Also, the device 100 may determine whether the camera shakes when capturing the input image.

Also, the face image acquisition condition determiner 182 may determine whether a face in the face region of the input image satisfies a preset face image acquisition condition.

For example, the face image acquisition condition determining unit 182 may determine whether the angle of the face is located within a reference angle range from the front face. Also, the face image acquisition condition determining unit 182 may determine whether a face in the input image has opened eyes. Also, the face image acquisition condition determining unit 182 may determine whether a face in the input image represents an expression. Also, the face image acquisition condition determining unit 182 may determine whether a face in the input image has an ear. Also, the face image acquisition condition determining unit 182 may determine whether a face in the input image is larger than or equal to a reference size.

When the input image satisfies a preset face image acquisition condition, the controller 186 may acquire an image of the face region from the input image as a face image.

As the image of the face region is acquired from the input image as the face image, the face normalizer 183 may normalize the face image based on a preset criterion.

For example, the face normalizer 183 may change the size of the face image to a preset size. Also, the face normalizer 183 may adjust the effect of the color temperature of light from the face image. Also, the face normalizer 183 may change the brightness of the face image to a preset brightness value.

As the face image is normalized, the controller 180 may determine the health state information based on the face image through the health state information determiner 189 . The health state information determiner 189 may include a diagnosis region extractor 186 , a face state information extractor 187 , and a health state information extractor 188 .

The diagnosis region extractor 186 may determine a diagnosis region from which preset face state information is to be extracted from the face image.

For example, the diagnosis region extractor 186 may determine the position of the face component in the face image. Facial components may include eyes, nose and mouth, and the like. The diagnostic region extraction unit 186 binarizes the face image by using a characteristic in which the brightness of the eyes, eyebrows, and mouth is darker than that of the facial skin in the human face, and determines the positions of the dark regions such as the eyes, eyebrows, and mouth. location can be determined. Also, the diagnosis region extractor 186 may extract a non-flesh color region from the face image using skin color information of the face, and determine the extracted region as positions of eyes, eyebrows, and mouth. In addition, since the positions of the eyes, eyebrows, nose, and mouth on the face represent a constant pattern, the diagnostic area extractor 186 uses the AAM method to determine the positions of the face components based on the face pattern to construct the face. It is also possible to determine the position of an element.

As the location of the facial component is determined, the diagnostic area extractor 186 may determine the location of the diagnostic area based on the location of the facial component.

As the location of the diagnosis region is determined, the facial state information extractor 178 may extract facial state information from the determined diagnosis region.

The face state information may be information about a symptom appearing on the face when an abnormality occurs in the human body. The facial state information may include, for example, the color of the face, the number and size of blemishes or acne, whether the eyes are red, the color of the eyes, the size of the pupil, the movement of the pupil, the size of the face, the shape of the face outline, the color of the lips, the lips It may include whether or not the skin is split, the position of each organ (eyes, nose, mouth, ears, eyebrows) of the face, the color of the hair, the gloss of the hair, whether the hair is split, the movement of the facial muscles, and the like.

A type of facial state information that can be extracted corresponding to the diagnosis area and a method of extracting the facial state information may be pre-stored in the memory 120 .

Extraction of face state information may be performed by analyzing the face image. For example, the face state information extractor 178 may extract face state information using color information of a face region of a face image, a face recognition algorithm, a face expression recognition algorithm, and the like.

As the face state information is extracted from the face image, the health state information extractor 188 may obtain health state information related to the user's health by using the face state information.

The health state information may refer to information about a user's disease or lifestyle predicted from the face state information. For example, when it is determined that the area under the eyes is swollen in the face image, the device 100 may determine that the user has hyperthyroidism or an allergic disease. In addition, when it is determined that the area under the eyes has turned black in the face image, the device 100 may determine that the user has allergic rhinitis.

The controller 180 may provide the calculated health state information through the display 110 .

The controller 170 may acquire the face image from an image file stored in the memory 120 . In addition, the controller 170 may provide an interface for viewing the image files stored in the memory 120 through functions such as a photo album and a gallery.

The memory 120 may store one or more image files. The image file includes a still image file and a moving image file. The memory 120 may store an image file generated from a captured image and/or an image file received from an external device. The image file may have a format such as joint photographic coding experts group (JPEG), moving picture expert group (MPEG), MP4, audio visual interleaving (AVI), and advanced streaming format (ASF).

According to an embodiment, the device 100 may include a sensing unit 190 . Also, the device 100 may acquire facial state information and health state information by using the biometric parameters detected by the sensing unit 190 . The bio-parameters may include, for example, blood pressure, heart rate, blood sugar, acidity, blood hemoglobin concentration, oxygen saturation, etc. The sensing unit 190 may include, for example, a heart rate sensor, a blood pressure measurement sensor, and acidity It may be implemented as a sensor that measures

According to an embodiment, the control unit 170 receives the facial state information and health state information from the user through a user input unit (not shown), and uses the received facial state information and health state information together with a face image, New face state information and health state information may be acquired. The face state information that may be input from the user may include, for example, height, weight, age, blood pressure, and the like. The health status information that can be input from the user may include, for example, chronic disease, past medical history, family history, current condition information, and the like.

The facial state information and health state information input by the user may be stored and managed as separate files, managed in an application, or managed in a service server. In addition, face state information and health state information input from a user may be managed for each user.

The communication unit 130 may communicate with an external device. The communication unit 130 may communicate with an external device by wire or wirelessly. According to an embodiment, the communication unit 130 may communicate with a cloud server, a social network service (SNS) server, a service server that provides a health diagnosis service, and the like. Also, the communication unit 130 may communicate with other electronic devices such as a smartphone, a camera, a tablet PC, a PDA, a notebook computer, and a mobile phone.

The device 100 may be implemented in the form of a smart phone, a tablet PC, a mobile phone, a camera, a notebook computer, personal digital assistants (PDA), or the like.

56 is a block diagram illustrating the configuration of the device 100 according to another embodiment of the present disclosure.

As shown in FIG. 56 , the configuration of the device 100 is, for example, a camera, a mobile phone, a tablet PC, a PDA, an MP3 player, a kiosk, an electronic picture frame, a navigation device, a digital TV, a wrist watch, or It may be applied to various types of devices such as a wearable device 100 such as a head-mounted display (HMD).

Referring to FIG. 56 , the device 100 includes a display unit 110 , a control unit 170 , a memory 120 , a GPS chip 125 , a communication unit 130 , a video processor 135 , an audio processor 140 , and a user. At least one of an input unit 145 , a microphone unit 150 , an imaging unit 155 , a speaker unit 160 , and a motion sensing unit 165 may be included.

Also, the display unit 110 may include a display panel 111 and a controller (not shown) for controlling the display panel 111 . The display panel 111 includes various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode (AM-OLED), and a plasma display panel (PDP). can be The display panel 111 may be implemented to be flexible, transparent, or wearable. The display unit 110 may be provided as a touch screen (not shown) by being coupled to the touch panel 147 of the user input unit 145 . For example, the touch screen (not shown) may include an integrated module in which the display panel 111 and the touch panel 147 are coupled in a stacked structure.

The memory 120 may include at least one of an internal memory (not shown) and an external memory (not shown).

The built-in memory includes, for example, a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.), a non-volatile memory (eg, One Time Programmable ROM (OTPROM)). ), Programmable ROM (PROM), Erasable and Programmable ROM (EPROM), Electrically Erasable and Programmable ROM (EEPROM), Mask ROM, Flash ROM, etc.), hard disk drive (HDD), or solid state drive (SSD). may include According to an embodiment, the controller 170 may load a command or data received from at least one of the non-volatile memory or other components into the volatile memory and process it. Also, the controller 170 may store data received or generated from other components in a non-volatile memory.

The external memory includes, for example, at least one of CF (Compact Flash), SD (Secure Digital), Micro-SD (Micro Secure Digital), Mini-SD (Mini Secure Digital), xD (extreme Digital), and Memory Stick. may include

The memory 120 may store various programs and data used for the operation of the device 100 . For example, the memory 120 may temporarily or semi-permanently store at least a portion of content to be displayed on the lock screen.

The controller 170 may control the display 110 so that a part of the content stored in the memory 120 is displayed on the display 110 . In other words, the controller 170 may display a part of the content stored in the memory 120 on the display 110 . Alternatively, when a user gesture is made in one area of the display unit 110 , the controller 170 may perform a control operation corresponding to the user's gesture.

The controller 170 may include at least one of a RAM 171 , a ROM 172 , a CPU 173 , a graphic processing unit (GPU) 174 , and a bus 175 . The RAM 171 , the ROM 172 , the CPU 173 , and the GPU 174 may be connected to each other through the bus 175 .

The CPU 173 accesses the memory 120 and performs booting using the O/S stored in the memory 120 . In addition, various operations are performed using various programs, contents, data, etc. stored in the memory 120 .

The ROM 172 stores an instruction set for system booting and the like. For example, in the mobile terminal 100, when a turn-on command is input and power is supplied, the CPU 173 copies the O/S stored in the memory 120 to the RAM 171 according to the command stored in the ROM 172, You can boot the system by running O/S. When booting is completed, the CPU 173 copies various programs stored in the memory 120 to the RAM 171 , and executes the programs copied to the RAM 171 to perform various operations. The GPU 174 displays the UI screen in the area of the display 110 when the booting of the device 100 is completed. Specifically, the GPU 174 may generate a screen on which an electronic document including various objects such as content, icons, and menus is displayed. The GPU 174 calculates property values such as coordinate values, shape, size, color, etc., at which each object is to be displayed according to the layout of the screen. In addition, the GPU 174 may generate screens of various layouts including objects based on the calculated attribute values. The screen generated by the GPU 174 may be provided to the display unit 110 and displayed on each area of the display unit 110 .

The GPS chip 125 may receive a GPS signal from a Global Positioning System (GPS) satellite and calculate the current location of the device 100 . The controller 170 may calculate the user's location by using the GPS chip 125 when using a navigation program or when the user's current location is required.

The communication unit 130 may communicate with various types of external devices 100 according to various types of communication methods. The communication unit 130 may include at least one of a Wi-Fi chip 131 , a Bluetooth chip 132 , a wireless communication chip 133 , and an NFC chip 134 . The control unit 170 may communicate with various external devices 100 using the communication unit 130 .

The Wi-Fi chip 131 and the Bluetooth chip 132 may perform communication using a WiFi method and a Bluetooth method, respectively. In the case of using the Wi-Fi chip 131 or the Bluetooth chip 132, various connection information such as an SSID and a session key are first transmitted and received, and then various types of information can be transmitted and received after communication connection using the same. The wireless communication chip 133 refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evoloution (LTE). The NFC chip 134 refers to a chip operating in a Near Field Communication (NFC) method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz, and 2.45 GHz.

The video processor 135 may process video data included in content received through the communication unit 130 or content stored in the memory 120 . The video processor 135 may perform various image processing, such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion, on video data.

The audio processor 140 may process audio data included in the content received through the communication unit 130 or the content stored in the memory 120 . In the audio processor 140 , various processes such as decoding, amplification, and noise filtering on audio data may be performed.

When a playback program for multimedia content is executed, the controller 170 drives the video processor 135 and the audio processor 140 to play the corresponding content. The speaker unit 160 may output audio data generated by the audio processor 140 .

The user input unit 145 may receive various commands from the user. The user input unit 145 may include at least one of a key 146 , a touch panel 147 , and a pen recognition panel 148 .

The key 146 may include various types of keys, such as mechanical buttons and wheels, which are formed in various areas, such as the front, side, or rear of the main body of the device 100 .

The touch panel 147 may detect a user's touch input and output a touch event value corresponding to the sensed touch signal. When the touch panel 147 is combined with the display panel 111 to form a touch screen (not shown), the touch screen may be implemented with various types of touch sensors such as capacitive type, pressure sensitive type, and piezoelectric type. The capacitive type is a method of calculating touch coordinates by sensing micro-electricity generated by the user's body when a part of the user's body is touched on the touch screen surface using a dielectric coated on the touch screen surface. The decompression type includes two electrode plates built into the touch screen, and when the user touches the screen, the touch coordinates are calculated by detecting that the upper and lower plates of the touched point come into contact and current flows. A touch event occurring on a touch screen may be mainly generated by a human finger, but may also be generated by an object made of a conductive material capable of applying a change in capacitance.

The pen recognition panel 148 detects a pen proximity input or a touch input according to the operation of the user's touch pen (eg, a stylus pen, a digitizer pen), and detects a pen proximity event or pen A touch event can be output. The pen recognition panel 148 may be implemented, for example, in an EMR method, and may sense a touch or proximity input according to the proximity of the pen or a change in the intensity of an electromagnetic field due to the touch. In detail, the pen recognition panel 148 includes an electromagnetic induction coil sensor (not shown) having a grid structure and an electronic signal processing unit (not shown) that sequentially provides an AC signal having a predetermined frequency to each loop coil of the electromagnetic induction coil sensor. ) may be included. When a pen having a built-in resonance circuit is present in the vicinity of the loop coil of the pen recognition panel 148, a magnetic field transmitted from the loop coil generates a current based on mutual electromagnetic induction in the resonance circuit in the pen. Based on this current, an induced magnetic field is generated from the coil constituting the resonance circuit in the pen, and the pen recognition panel 148 detects the induced magnetic field from the loop coil in the signal receiving state, so that the pen approaching position or A touch position may be sensed. The pen recognition panel 148 may be provided with a predetermined area under the display panel 111 , for example, an area capable of covering the display area of the display panel 111 .

The microphone unit 150 may receive a user's voice or other sound and convert it into audio data. The control unit 170 may use the user's voice input through the microphone unit 150 in a call operation or convert it into audio data and store it in the memory 120 .

The imaging unit 155 may capture a still image or a moving image according to a user's control. The imaging unit 155 may be implemented in plurality, such as a front camera or a rear camera.

When the imaging unit 155 and the microphone unit 150 are provided, the control unit 170 performs a control operation according to a user voice input through the microphone unit 150 or a user motion recognized by the imaging unit 155 . may be For example, the device 100 may operate in a motion control mode or a voice control mode. When operating in the motion control mode, the controller 170 may activate the imaging unit 155 to capture an image of the user, track a change in the user's motion, and perform a control operation corresponding thereto. When operating in the voice control mode, the controller 170 may operate in the voice recognition mode in which the user's voice input through the microphone unit 150 is analyzed and a control operation is performed according to the analyzed user's voice.

The motion detection unit 165 may detect a movement of the body of the device 100 . The device 100 may be rotated or tilted in various directions. In this case, the motion detection unit 165 may detect motion characteristics such as a rotation direction, an angle, and a tilt using at least one of various sensors such as a geomagnetic sensor, a gyro sensor, and an acceleration sensor.

In addition, although not shown in FIG. 68, in the embodiment, a USB port to which a USB connector can be connected in the device 100, various external input ports for connection with various external terminals such as a headset, a mouse, and a LAN, DMB ( A DMB chip for receiving and processing (Digital Multimedia Broadcasting) signals, various sensors, and the like may be further included.

Names of components of the above-described device 100 may vary. In addition, the device 100 according to the present disclosure may be configured to include at least one of the above-described components, and some components may be omitted or may further include additional other components.

On the other hand, the present invention can be implemented by storing a computer-readable code in a computer-readable storage medium. The computer readable storage medium includes all kinds of storage devices in which data readable by a computer system is stored.

When the computer readable code is read from the computer readable storage medium by a processor and executed, the computer readable code is configured to perform the steps of implementing the electronic device control method according to the present invention. The computer readable code may be implemented in various programming languages. And functional programs, codes, and code segments for implementing the embodiments of the present invention may be easily programmed by those skilled in the art to which the present invention pertains.

Examples of the computer-readable storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like, and includes implementation in the form of a carrier wave (eg, transmission over the Internet). In addition, the computer-readable storage medium may be distributed in network-connected computer systems, so that the computer-readable code is stored and executed in a distributed manner.

Although embodiments of the present invention have been described with reference to the above and the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can implement the present invention in other specific forms without changing the technical spirit or essential features. You will understand that there is Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (20)

camera;
display;
GPS sensor;
light sensor;
a biometric sensor that detects the user's biometric information;
a memory storing instructions; and
at least one processor configured to execute the instructions;
including,
The at least one processor executes the instructions to:
Obtaining a plurality of first face images of the user through the camera,
obtaining an average value of a plurality of first face state information obtained from the plurality of first face images,
obtaining a second face image by photographing the user's face through the camera, and obtaining second face state information indicating the state of the user's face from the second face image;
Obtaining illuminance information in a situation in which the second face image is acquired using the illuminance sensor,
Obtaining photographing location information and photographing time information for obtaining the second face image by using the GPS sensor,
Obtaining the user's biometric information detected at the time of acquiring the second face image from the biometric sensor,
A comparison result of comparing the obtained photographing situation information including at least one of the obtained illuminance information, photographing location information, photographing time information, and biometric information and the second facial condition information with the average value of the plurality of first facial condition information to obtain health state information indicating the health state of the user,
A device for displaying the obtained health state information through the display.
The method of claim 1,
The at least one processor executes the instructions to:
according to a preset face image acquisition condition, displaying shooting guide information for guiding the user's face shooting through the display,
It is determined whether the camera has photographed the user's face according to the face image acquisition condition, and when the camera has photographed the user's face according to the face image acquisition condition, the image obtained by the photographing is used as the Determining the user's second face image, the device.
The method of claim 1,
The at least one processor executes the instructions to:
and normalize the second face image by enlarging or reducing the size of the second face image to a preset size.
The method of claim 1,
The at least one processor executes the instructions to:
The second face image is obtained by obtaining a color temperature of illumination on the user's face at a time point at which the second face image is captured, and adjusting a hue value of the second face image based on the obtained color temperature. A device configured to normalize .
The method of claim 1,
The at least one processor executes the instructions to:
displaying a user interface for selecting the obtained biometric information through the display;
and display, through the display, the health state information of the user further based on the selected biometric information, upon receiving a user input for selecting the biometric information.
The method of claim 1,
The device, wherein the plurality of first face images are images captured on dates before the second face image is captured.
The method of claim 1,
The at least one processor executes the instructions to:
Through the display, the second face image and the health status information are displayed together, and on an area related to the displayed health status information among the entire area of the second face image, an area related to the displayed health status information A device configured to display an image indicating that
The method of claim 1,
and the average value of the plurality of first facial state information and the second facial state information are compared according to a face part.
The method of claim 1,
The device, wherein the average value of the plurality of first facial state information and the second facial state information are compared according to at least one of a face color, an eye color, a lip color, and a hair color.
The method of claim 1,
The average value of the plurality of first facial condition information and the second facial condition information may include a face size, a size of a pupil, a size of a blemish, a size of a pimple, a number of blemishes, a number of acne, a shape of a facial contour, and a cracked lip. , and splitting of the hair.
acquiring, by a device, a plurality of first face images of a user through a camera of the device;
obtaining an average value of a plurality of first face state information obtained from the plurality of first face images;
obtaining a second face image by photographing the user's face through the camera; obtaining second face state information indicating the state of the user's face from the second face image;
obtaining photographing situation information including at least one of illuminance information, photographing location information, photographing time information, and biometric information of the user at the time of acquiring the second face image;
obtaining health state information indicating the health state of the user based on a comparison result of comparing the second facial state information with the average value of the plurality of first facial state information and the obtained photographing situation information; and
displaying the acquired health state information through a display of the device;
including,
The step of obtaining the shooting situation information includes:
acquiring illuminance information at the time of acquiring the second face image using an illuminance sensor;
obtaining photographing location information and photographing time information for obtaining the second face image by using a GPS sensor; and
acquiring the user's biometric information detected at the time of acquiring the second face image from the biometric sensor attached to the user's body;
A method of providing health condition information, comprising:
12. The method of claim 11,
The step of obtaining a second face image of the user comprises:
displaying, through the display, photographing guide information for guiding photographing of the user's face according to a preset face image acquisition condition; and
It is determined whether the camera has photographed the user's face according to the face image acquisition condition, and when the camera has photographed the user's face according to the face image acquisition condition, the image obtained by the photographing is used as the and determining the second face image of the user.
12. The method of claim 11,
Normalizing the second face image by enlarging or reducing the size of the second face image to a preset size.
12. The method of claim 11,
obtaining a color temperature of light illuminating the user's face at the time the second face image is captured; and
Normalizing the second face image by adjusting a hue value of the second face image based on the obtained color temperature.
12. The method of claim 11,
The step of displaying the health status information includes:
displaying a user interface for selecting the obtained biometric information through the display; and
in response to receiving a user input for selecting the biometric information, displaying the health state information of the user through the display further based on the selected biometric information;
A method comprising
The method of claim 11 , wherein the plurality of first face images are images captured on dates before the second face image is captured.
12. The method of claim 11,
The step of displaying the health status information includes:
displaying the second face image and the health state information together through the display; and
and displaying an image indicating the area related to the displayed health state information on a region related to the displayed health state information among the entire region of the second face image.
12. The method of claim 11,
and the average value of the plurality of first facial condition information and the second facial condition information are compared according to a face part.
12. The method of claim 11,
and the average value of the plurality of first facial state information and the second facial state information are compared according to at least one of a face color, an eye color, a lip color, and a hair color. 12. The method of claim 11,
The average value of the plurality of first facial condition information and the second facial condition information may include a face size, a size of a pupil, a size of a blemish, a size of acne, the number of blemishes, the number of acne, a shape of a facial contour, and a shape of a lip. and splitting according to at least one of splitting and splitting of the hair.

Images