KR20190097494A - Electronic devices and methods thereof - Google Patents

Abstract

Disclosed is an electronic device for estimating a head posture. The electronic device of the present invention comprises: a photographing unit; a memory storing a weight corresponding to a landmark representing a previously defined feature point; and a processor identifying a face of a user from an image photographed by the photographing unit, identifying the landmark among face feature points of the identified user, and identifying a head posture of the user by obtaining the weight corresponding to the identified landmark from the memory and applying the obtained weight to the landmark. Relatively small weight among the weights stored in the memory is given to a landmark with relatively little movement.

Description

ELECTRICAL DEVICES AND METHODS THEREOF

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method for estimating a head posture by setting a landmark on a user's face.

Recently, many studies have been conducted to detect, track, and recognize faces in video images. Technology for identifying a user's head posture is very important in various computer vision fields such as face recognition, facial expression recognition, and image modeling.

In order to estimate the head posture of the user, a comparison is made with a plurality of samples stored in advance. The user's head posture was estimated by finding the sample most similar to the current image. This approach may have a relatively high accuracy in general situations where the user's facial expression does not change. However, there is a problem in that the user's face change due to various situations such as when the user talks or expresses emotions cannot be reflected.

For example, the conventional method has a problem in that the head posture of the user is differently identified when the facial expression is changed, such as the jaw moving or closing the eyes, while the head posture of the user is kept constant.

In addition, the conventional method required a processor having a large memory capacity and a high computational speed because a variety of samples must be stored. Therefore, there is a problem that a high cost and high performance computing system is required.

The present disclosure is devised to improve the above-described problem, and an object of the present disclosure is to provide an electronic device and a control method thereof for estimating a head posture by setting a plurality of landmarks and giving different weights.

According to an embodiment of the present invention, an electronic device is provided that includes: an imager, a memory storing a weight corresponding to a landmark representing a predetermined feature point, a face of a user in an image photographed by the imager, A processor for identifying a landmark among the identified facial feature points of the user, obtaining a weight corresponding to the identified landmark from the memory, and applying the obtained weight to the landmark to identify the head posture of the user And a weight stored in the memory is given a relatively small weight to a landmark having a lot of movement.

In this case, the processor may identify the head posture of the user by applying the weight to a difference value between the landmark of the user's face and the landmark of the reference face data.

In addition, the processor may apply a weight that is relatively smaller than other landmarks to the landmarks constituting the outside of the face among the identified landmarks of the user.

In addition, the processor assigns a first weight to a landmark corresponding to at least one of eyes, an eyebrow, and a nose with less movement, and a second weight less than the first weight to a landmark corresponding to at least one of a mouth and a jaw. Can be given.

The landmark may include three-dimensional location information, and the processor may identify a head posture of the user by applying a Perspective n Point (PnP) method to the three-dimensional location information.

The electronic device may be a device provided in a vehicle, and the processor may track at least one of a driver's state and a driver's gaze based on the identified head posture of the user.

The processor may acquire a degree of movement of the landmark identified on the face of the user, and map a relatively large weight to a landmark having a relatively high degree of movement and store the weight.

Meanwhile, a control method of an electronic device for storing a weight corresponding to a landmark representing a predefined feature point according to an embodiment of the present disclosure may include identifying a face of a user in a captured image, and identifying the face feature point of the identified user. Identifying a landmark and obtaining a weight corresponding to the identified landmark, and applying the obtained weight to the landmark to identify a head posture of the user. Relatively small landmarks are given relatively small weights.

In this case, the step of identifying the head pose of the user may identify the head pose of the user by applying the weight to a difference value between the landmark of the user's face and the landmark of the reference face data.

In the identifying of the user's head posture, a weight that is relatively smaller than other landmarks may be applied to a landmark constituting the outside of the face among the identified landmarks of the user.

In addition, the identifying of the user's head posture may include assigning a first weight to a landmark corresponding to at least one of eyes, eyebrows, and nose having less movement, and applying the first weight to a landmark corresponding to at least one of a mouth and a chin. A second weight less than one weight can be given.

In addition, the landmark includes three-dimensional location information, and the step of identifying the head pose of the user may identify the head pose of the user by applying a perspective n point (PnP) method to the three-dimensional location information. .

In addition, the electronic device is a device provided in a vehicle, and may further include tracking at least one of a driver's state and a driver's gaze based on the identified head posture of the user.

In the identifying of the user's head posture, the movement degree of the landmark identified in the face of the user may be obtained, and the weight may be mapped to a relatively large landmark and stored.

Meanwhile, when executed by a processor of an electronic device that stores a weight corresponding to a landmark representing a predefined feature point, a non-transitory storing computer instructions for causing the electronic device to perform an operation. The computer readable medium may further include identifying a face of the user in the captured image, identifying a landmark among the identified facial feature points of the user, and obtaining a weight corresponding to the identified landmark, and obtaining the weighted value. And identifying the head posture of the user by applying to the landmark, wherein the weight is given a relatively small weight to a landmark having a relatively high movement.

1 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure;
2 is a view for explaining a method of identifying a face of a user;
3 is a view for explaining a method for setting a landmark in a user face image;
4 is a view for explaining a method of grouping landmarks;
5 is a view for explaining a method of estimating a head posture according to an embodiment of the present disclosure;
6 is a view for explaining a method for weighting a landmark;
7 is a view for explaining a perspective n point (PnP), and
8 is a diagram illustrating a control method of an electronic device according to an embodiment of the present disclosure.

Before describing this disclosure in detail, the description method of this specification and drawings is demonstrated.

First, terms used in the present specification and claims have been selected from the general terms in consideration of functions in various embodiments of the present disclosure, but these terms are defined by intention or legal or technical interpretation of those skilled in the art and It may vary depending on the emergence of new technologies. In addition, some terms are terms arbitrarily selected by the applicant. Such terms may be interpreted in the meanings defined herein, and may be interpreted based on the general contents of the present specification and common technical knowledge in the art without specific term definitions.

In addition, the same reference numerals or symbols described in each drawing attached to the present specification represent parts or components that perform substantially the same function. For convenience of explanation and understanding, different embodiments will be described using the same reference numerals or symbols. That is, although all the components having the same reference numerals are shown in the plurality of drawings, the plurality of drawings does not mean an embodiment.

Also, in the present specification and claims, terms including ordinal numbers such as “first”, “second”, and the like may be used to distinguish between components. These ordinal numbers are used to distinguish the same or similar components from each other, and the meaning of the terms should not be construed as limited by the use of these ordinal numbers. For example, the components combined with these ordinal numbers should not be limited in order of use or arrangement by the number. If necessary, the ordinal numbers may be used interchangeably.

As used herein, the singular forms "a", "an" and "the" include plural forms unless the context clearly indicates otherwise. In this application, "includes." Or "made up." And the like are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, and that one or more other features or numbers, step, action, component, part, or It should be understood that they do not preclude the presence or possibility of adding these in advance.

The embodiments may be variously modified and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the scope to the specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the scope of the disclosed spirit and technology. In describing the embodiments, when it is determined that the detailed description of the related known technology may obscure the gist, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the scope. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, "includes." Or "made up." And the like are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, and that one or more other features or numbers, step, action, component, part, or It should be understood that they do not preclude the presence or possibility of adding these in advance.

In the embodiments of the present disclosure, terms such as “module”, “unit”, “part”, and the like are terms for referring to a component that performs at least one function or operation, and such components are referred to as hardware or software. It may be implemented or a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except that each needs to be implemented with a particular specific hardware, and is at least one processor. It can be implemented as.

In addition, in an embodiment of the present disclosure, when a part is connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a part includes a certain component means that it may further include other components, without excluding other components, unless specifically stated otherwise.

1 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

1 may include an imaging unit 101, a memory 102, and a processor 103.

The image capturing unit 101 is a configuration for generating a captured image by capturing a subject, where the captured image includes both a moving image and a still image.

The imaging unit 101 may acquire an image of at least one external device, and may be implemented as a camera, a lens, an infrared sensor, or the like.

The imaging unit 101 may include a lens and an image sensor. The type of lens includes a general-purpose lens, a wide-angle lens, a zoom lens, and the like, and may be determined according to the type, characteristics, and use environment of the electronic device 100. As an image sensor, a complementary metal oxide semiconductor (CMOS), a charge coupled device (CCD), or the like may be used.

The imaging unit 101 outputs incident light as an image signal. In detail, the imaging unit 101 may include a lens, a pixel, and an AD converter. The lens collects the light of the subject to form an optical image in the image capturing area, and the pixel may output an image formed in the form of an analog signal through the lens. The AD converter may convert an analog video signal into a digital video signal and output the converted video signal.

In particular, the image capturing unit 101 may be arranged to capture a front direction of the electronic device 100, and may generate a captured image by capturing a user existing in the front of the electronic device 100.

Meanwhile, in describing the electronic device 100 according to an embodiment of the present disclosure, the imaging unit 101 is described as one person, but in actual implementation, a plurality of imaging units may be disposed. The electronic device 100 may include a plurality of imaging units, and may combine the images received through the plurality of imaging units to identify a user's head posture. Using a plurality of imaging units 101 can analyze the three-dimensional movements more precisely than using one imaging unit 101, which can be effective in identifying a user's head posture.

The memory 102 may be implemented as a nonvolatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), a solid state drive (SSD), or the like. The memory may be implemented not only as a storage medium in the electronic device but also as an external storage medium, for example, a micro SD card, a USB memory, or a web server through a network.

In detail, the memory 102 may store data used for the operation of the electronic device 100, and the memory 102 may store weights corresponding to landmarks representing predefined feature points.

The processor 103 may perform an overall control operation of the electronic device 100. In detail, the processor 103 may identify the head posture of the user using the feature points of the face.

According to an embodiment, the processor 103 may include a digital signal processor (DSP), a microprocessor, a time controller (TCON), a central processing unit (CPU), and a micro controller unit (MCU). ), Microprocessor unit, controller, application processor (AP), or communication processor (CP), or one or more of ARM processors, or defined in these terms In addition, the processor 103 may be implemented as a system on chip (SoC), a large scale integration (LSI) in which a processing algorithm is embedded, or a field programmable gate array (FPGA).

The processor 103 may identify the head posture of the user. Here, the head posture of the user may mean estimating the position and state of the user's head. The user's head may be determined, and the user's gaze detection, motion recognition, and drowsiness detection may be identified according to the determined state.

In order to identify the posture of the user's head, the processor 103 should be able to detect the face part of the user. In detail, the face of the person should be identified from the image photographed by the imaging unit 101.

The processor 103 may set a feature point in the captured image to identify the face of the person in the image captured by the imager 101. Assuming that the input image is an image of a human head, the processor 103 may detect a facial feature point as a preset feature point.

The processor 103 may detect a facial feature point by applying a pre-stored face model of an arbitrary human face to an image input through the imaging unit 101. The facial feature point means a feature point for eyes, eyebrows, nose, upper lip, lower lip, jawline, and the like. The face model inputs an image of an arbitrary human face, manually or automatically selects feature points for eyes, eyebrows, nose, upper lip, lower lip, jawline, and the like in the face area indicated by the selected point. A result of repeating an operation of inputting information about a plurality of images.

As described above, the face model includes three-dimensional information and color information of the facial feature point based on information that can be obtained from the face image, that is, information about the position of the facial feature point, information about the color, information about the texture, and information about the image pickup position. It is to include.

The processor 103 may automatically detect facial feature points in the input image through various algorithms using the face model. For example, Active Appearance Models (AAM) or Active Shape Models (ASM) may be used.

The ASM detects the position of the eye and determines the position using shape information of other parts of the generic model, such as nose, mouth, and eyebrows, based on the detected eye position. to be. AAM is a model that detects feature points that can effectively describe facial features by considering texture information as well as shape information.

AAM applies principal component analysis (PCA) to face model vectors and face surface texture vectors to warn sample face models created using a variety of human face statistics to normalize the sample face data. The error square of the face data of the captured image 2D is minimized. Use this data to find facial feature points. AAM has the advantage of speed calculation and tracking.

When using the AAM, the processor 103 performs a fitting process of detecting feature points for the eyes, eyebrows, nose, upper lip, lower lip, and jaw line of the face included in the input image.

The processor 103 may identify the facial feature point of the user from the image received through the above process.

The processor 103 may set a specific facial feature point as a landmark among the plurality of identified facial feature points. When there are a plurality of facial feature points identified in one image, some of the feature points may be set as landmarks.

The landmark may be a facial feature point selected from a plurality of facial feature points in a preset manner.

The preset method according to an embodiment of the present disclosure may be a method of setting a landmark at a predetermined interval. For example, suppose that 100 facial features that make up the jawline have been identified. In this case, 20 of the 100 facial feature points can be set as a landmark.

The processor 103 may set one facial feature point per five as a landmark from the 100 facial feature points listed in order.

It is assumed that 50 out of 100 facial feature points are facial feature points forming a jaw line. The 50 facial features that make up the jawline will be arranged in order. The fifty facial features that form exactly the jawline may be a set of facial features that are arranged in a U shape. In this case, the processor 103 may set the facial feature points corresponding to multiples of 5, such as the fifth, tenth, and fifteenth, to the landmarks in the ordered facial feature points.

Setting some of the 50 identified jawline feature points (20 jawline feature points) as landmarks can reduce data analysis and processing time. To identify the user's head posture, the user's facial feature points must be analyzed. If the user analyzes 20 jawline features rather than 50 jawline features, the processing time for the processor 103 to estimate the user's head pose is Can be shortened.

The numbers 100, 20, and 5 described above are only examples for description and may be changed according to a user's setting or the performance of the electronic device 100.

In addition, the processor 103 may identify a landmark among the identified facial feature points of the user, and obtain a weight corresponding to the identified landmark from the memory 102.

The weight corresponding to the landmark may be a predetermined value. The weight corresponding to the landmark may have a different value depending on the position of the facial feature point. For example, even if the landmark represents the same jawline, a smaller weight may be applied to the landmark at the tip of the jaw.

Referring to FIG. 3, a landmark (No. 10) at the tip of the jaw, a landmark (49 and 55) corresponding to the corner of the mouth, and a landmark (No. 37, 40, 43) corresponding to the eyes are shown. And 46).

The above-mentioned landmarks 10, 37, 40, 43, 46, 49, and 55 may correspond to relatively high landmarks. The processor 103 may apply a small weight to a relatively moving landmark. The reason why small weights are applied to landmarks that are relatively in motion is to identify the user's head posture more accurately. There may be a case where the movement is misidentified by many landmarks even though the head posture of the user has not changed.

For example, suppose that the user maintains the motion of looking at the front. The user is looking straight ahead, but if he smiles loudly during the conversation, his mouth, chin, and eyes will move more than the rest. In this case, the processor 103 which grasps the user's landmark is likely to misidentify the head, even though the user's head posture is looking straight ahead. Since the processor 103 uses the landmark to identify the head posture of the user, the identification result is likely to be influenced by the movement of the specific landmark.

In order to solve such a problem, the processor 103 may assign a small weight to a relatively high landmark. In addition, the processor 103 may identify the user's head posture by applying the obtained weight to the landmark.

In addition, the processor 103 may apply a weight that is relatively smaller than other landmarks to the landmarks constituting the outside of the face among the identified landmarks of the user.

In addition, the processor 103 assigns a first weight to a landmark corresponding to at least one of eyes, an eyebrow, and a nose with less movement, and a second smaller than the first weight to a landmark corresponding to at least one of a mouth and a jaw. Can be weighted.

In this case, the processor 103 may identify the user's head posture by applying a weight to a difference value between the landmark of the user's face and the landmark of the reference face data.

The reference face data may refer to data which is a reference for identifying a posture of a user. In detail, the reference face data may store landmarks (or facial feature points) according to a user's head posture and may be compared with landmarks identified by the processor 103. The reference face data may include at least one data.

The processor 103 may identify the reference face data having the smallest difference by comparing the information included in the plurality of stored reference face data with the landmark identified by the processor 103. For example, when identifying the reference face data having the smallest difference in the position information of the compared landmarks, the processor 103 may infer that the user is currently taking a head posture corresponding to the reference face data.

In addition, the processor 103 may determine a weight to be applied to the landmark using the variance value. In the above-described example, the reference face data has been described as meaning one representative data, but there may be a plurality of reference face data. For example, the electronic device 100 may analyze all of the landmarks included in the reference face data to obtain a dispersion value of the corresponding landmark. The obtained variance may be a measure of how much the deformation of the face shape is affected. Since the processor 103 may be regarded as more distortion occurs as the variance value increases, the processor 103 may give a small weight to a landmark having a large variance value.

In addition, the processor 103 may compare landmarks of the same portion of the reference face data to measure the variance value of the landmarks. For example, the plurality of reference face data may include landmark information included in each person's face. Here, the processor 103 may include the landmark of the tip of the jaw included in the first reference face data, the landmark of the tip of the jaw included in the second reference face data, and the land of the tip of the jaw included in the third reference face data. The marks can be compared. The processor 103 may compare the landmarks of the three identical parts to determine the variance value and the weight of the jaw end. The processor 103 may determine the variance value and the weight by comparing the landmarks of the corresponding portions for each reference face data in the above manner.

In detail, the landmark includes three-dimensional location information, and the processor 103 may identify a user's head posture by applying a Perspective n Point (PnP) method to the three-dimensional location information.

The processor 103 may identify at least one of the user's state and the user's gaze by using the identified landmark.

In detail, the processor 103 may detect a user's line of sight from an image of the user captured by the imaging unit 101, and the detected line of sight may be located at an image (that is, an image of the user) displayed on the display unit. You can determine if it is located. The processor 103 may acquire, as the ROI, an area gazed at the user for a predetermined time from the image displayed on the display unit.

The processor 103 may detect a user's gaze using various algorithms. For example, the imaging unit 101 may be a device having a light emitting unit for irradiating infrared rays, and may irradiate infrared rays to the eyes of a user during imaging. In the captured image, infrared rays reflected from the cornea appear, and the processor 103 may detect the pupil by dividing the iris and the pupil by using the position of the reflected infrared rays, and obtain eye gaze information therefrom.

In addition, the processor 103 detects the position of the eye using a technique (eg, Haar-like features) of finding an object feature in the captured image, and uses the edge detection technique to detect the position of the eye. The pupil may be detected, and the gaze of the user may be detected based on the relative positions of the iris and the pupil.

The processor 103 projects the detected user gaze on the captured image to determine which part of the image displayed on the display unit by the user. In detail, the memory 102 stores data about a correlation between the position of the image capturing unit 101 and the position of the image in the display unit, and by using this, the processor 103 may determine the user's gaze detected in the captured image. By converting the coordinates based on the image displayed on the display unit, the user can detect a gaze gaze in the captured image displayed on the display unit.

Meanwhile, the processor 103 may assign different weights to each user. The processor 103 may identify the specific user by analyzing the face of the user identified through the imaging unit 101, and may apply differently stored weights for each user.

For example, assuming that the weight is applied to the same jawline portion, the weight may be differently applied according to the user.

Meanwhile, if previously stored user information is not retrieved, the processor 103 may generate new user information. Specifically, the processor 103 detects a user's face from the image captured by the image capturing unit 101, and acquires facial feature points of eyes, the middle of the eye, the nose, the nostrils, the lips, and the outline from the detected face. As described above, the processor 103 generates user information based on the facial feature points acquired in the captured image.

On the other hand, in the above description of the weighting operation, it has been described as giving a predetermined weight. However, the processor 103 may acquire a degree of movement of the landmark identified on the face of the user, and map and store a relatively large weight to a landmark having a relatively high degree of movement.

In this case, the processor 103 may photograph and analyze the face of the user for a predetermined time. In detail, the processor 103 may photograph the face of the user for a predetermined time and analyze the change of the facial feature point of the user. The processor 103 may determine which part of the user's face feature point is changed the most by changing the face feature point.

The processor 103 may identify a portion where the facial feature points move relatively much even though the head posture of the user is maintained. For example, even though it is determined that the user is in a head posture looking at the front, the analyzed region may determine the chin tip, mouth, eye area, and the like where the facial feature points change most.

In the above-described example, only the frontal posture has been described, but this is only a limitation for description, and the processor 103 may determine a change in facial feature points in various postures.

In addition, the electronic device 100 described above may be a device provided in a vehicle. In this case, the processor 103 may track at least one of the driver's state and the driver's gaze based on the identified head posture of the user.

2 is a diagram for describing a method of identifying a face of a user.

Referring to FIG. 2, various methods may exist for automatically identifying a face of a user.

The electronic device 100 may identify the location of the face of the user in the image received through the imaging unit.

Specifically, referring to FIG. 2A, all faces 201, 202, 203, and 204 of the user included in the image may be identified based on the eyes, eyebrows, nose, and mouth of the user.

In addition, referring to FIG. 2B, all of the faces 211, 212, 213, 214, and 215 of the user included in the image may be identified, including the shape of the user's eyes, eyebrows, nose, mouth, and face. Here, in order to determine the shape of the entire face, the user's chin, ear, etc. can be determined, and the outline of the entire face can be determined. The outline of the entire face may refer to information representing the outline of the entire face.

In Figure 2 (a) does not include the shape of the entire face of the user can not determine the degree of movement of the user's jaw, but in Figure 2 (b) includes the shape of the entire face can determine the degree of movement of the user's chin. have.

The electronic device 100 may set a method of automatically identifying the face of the user according to the user's setting or optimal selection. For example, the electronic device 100 may identify the face of the user according to the method of FIG. 2 (a) or FIG. 2 (b), and each method may vary according to the user's setting. In order to analyze and process the minimal data, the method according to FIG. 2 (a) may be desirable. In addition, it may be desirable to select the method according to FIG. 2 (b) to accurately identify the head posture of the user. Therefore, the electronic device 100 may automatically identify the user's face by applying another method according to the user's setting or optimal selection.

Here, the optimal selection may be related to the mode in which the user operates the electronic device 100. For example, if the user sets the operation mode of the electronic device 100 to a mode for the purpose of power saving or minimum processing time, the method of FIG. 2A using relatively little data may be selected. On the other hand, if the user is set to the mode for the purpose of accurate identification can choose the method according to Figure 2 (b) using the overall feature points of the face.

3 is a diagram for describing a method of setting a landmark in a user face image.

Referring to FIG. 3, the electronic device 100 may identify the user's face 301 from the image received through the imaging unit, and may identify a plurality of facial feature points. Here, the facial feature point may mean an outline representing a face of a person.

The electronic device 100 may set the landmarks (No. 1 to No. 68) in a preset manner at the plurality of facial feature points.

The landmark may be a facial feature point selected in a preset manner among the plurality of facial feature points, and the landmark may mean a part of the facial feature points.

Meanwhile, the preset method according to an embodiment of the present disclosure may be a method of setting a landmark at a predetermined interval.

The electronic device 100 may set a specific portion as a landmark on the identified face portion of the user. In order to determine a specific part, location information of a predefined feature point may be set.

The electronic device 100 may use facial feature points to identify the user's face. Referring to FIG. 3, the outermost facial feature point may be identified as the face part of the user. For example, the electronic device 100 may identify a rectangle connecting the face feature points 4, 19, 26, and 10 as the face of the user.

Here, the electronic device 100 may assign a weight to each landmark, and may assign different weights to each landmark.

In FIG. 3, each of the landmarks is weighted, but in actual implementation, landmarks may be grouped to give the same weight for each group.

4 is a diagram for describing a method of grouping landmarks.

Referring to FIG. 4, the electronic device 100 may set some of the plurality of facial feature points as landmarks (numbers 1 to 68) and identify the user's face 401.

Here, the electronic device 100 may group the landmarks set on the face of the user. For example, the user's face may be divided into a face outer portion including the jaw line, eyes, a nose, a mouth, and the like, and the electronic device 100 may mark a landmark in each part of the face (face outer portion, eyes, nose, Group).

Referring to FIG. 4, the electronic device 100 may define a landmark (No. 1 to No. 27) of the outer portion of the face as a group 4, a landmark (No. 28 to 36) of the nose portion as a group 2, and The landmarks (No. 37 to No. 48) can be set to Group 3, and the landmarks (No. 49 to No. 68) of the mouth can be set to Group 1. In addition, different weights may be assigned to each group. In more detail, this may mean a method in which group 1 is given the same weight but group 2 is smaller or larger than group 1. The electronic device 100 may assign different weights to each group in this manner.

On the other hand, assigning different weights to each group may mean that different weights are assigned to each group. However, this does not necessarily mean that different weights are applied to each group. According to the user's setting or optimal selection, the same weight may be applied to Group 1 and Group 2, and different weights may be applied to Group 3 and Group 4 than Group 1.

Meanwhile, in the description of FIG. 4, the outer part of the face is displayed as a group 4 and landmarks 1 to 27 are included, but in actual implementation, the outer part of the face is divided into the jaw line part (No. 1 to No. 17) and the head part. (Nos. 18-27). Generally, the head part does not move well during the conversation of the user, and the jaw line part may move relatively much. Accordingly, the electronic device 100 may give a smaller weight to the jaw line portion (No. 1 to No. 17).

5 is a diagram for describing a method of estimating a head posture according to an exemplary embodiment.

Referring to FIG. 5A, the electronic device 100 may identify facial feature points on a face of a user in real time through a landmark detection algorithm. In addition, a landmark may be set at the identified facial feature to identify a face part of the user. In FIG. 5 (a), a method of detecting only the jaw line part of the outer face part is applied.

The electronic device 100 may calculate location information included in the landmark by using the detected landmark. Here, the location information may be two-dimensional or three-dimensional information.

Referring to FIG. 5B, the electronic device 100 may apply a Perspective n Point (PnP) method for obtaining a projection matrix between two pieces of information using both two-dimensional and three-dimensional location information. In addition, the electronic device 100 may obtain information for eventually estimating the head posture by calculating the R (Roation) component and the t (Translation) component which are parameters in the projection matrix. The electronic device 100 may estimate the head posture of the user using the calculated R component of PnP. The R component may be composed of (Roll, Pitch, Yaw) and may have a value of (0,0,0) when looking at the front.

6 is a diagram for describing a method of assigning a weight to a landmark.

In the process of calculating the projection matrix, the electronic device 100 may assign a weight to reduce the influence of deformation of the face shape. In detail, the electronic device 100 may reduce the distortion of the face shape by weighting the landmark, and may accurately identify the head posture when the distortion of the face shape is reduced. In this case, the electronic device 100 may identify the user's head posture by applying a weight to a difference value between the landmark of the user's face and the landmark of the reference face data.

The reference face data may refer to data which is a reference for identifying a posture of a user. In detail, the reference face data may store a landmark (or facial feature point) according to a user's head posture and may be compared with a landmark identified by the electronic device 100.

The electronic device 100 may identify the reference face data having the smallest difference by comparing the information included in the plurality of previously stored reference face data with a landmark identified by the electronic device 100. For example, when identifying the reference face data having the smallest difference in the location information of the compared landmarks, the electronic device 100 may infer that the user is currently taking a head posture corresponding to the reference face data.

In addition, the electronic device 100 may determine a weight to be applied to the landmark by using the variance value. For example, all the landmarks included in the reference face data may be analyzed to obtain a variance value of the corresponding landmark. The obtained variance may be a measure of how much the deformation of the face shape is affected. Since the electronic device 100 may be regarded as more distortion occurs as the dispersion value increases, the electronic device 100 may give a small weight to a landmark having a large dispersion value.

The electronic device 100 may compare landmarks of the same portion of the reference face data to measure dispersion values of the landmarks. For example, the plurality of reference face data may include landmark information included in each person's face. Here, the electronic device 100 may include the landmark of the tip of the jaw included in the first reference face data, the landmark of the tip of the jaw included in the second reference face data, and the tip of the jaw included in the third reference face data. Compare landmarks. The electronic device 100 may compare the landmarks of the three same parts to determine the variance value and the weight of the tip of the jaw. The electronic device 100 may determine the variance value and the weight by comparing the landmarks of the portions corresponding to the respective reference face data in the above manner.

 Referring to FIG. 6, after the electronic device 100 sets some of the facial feature points of the user as a landmark, the electronic device 100 may assign different weights to the landmarks. As described above, smaller weights may be assigned to landmarks of relatively moving parts. Whether there is a lot of movement can be determined by the relative position of the landmark. This may be determined by a predetermined value, or may be determined through user movement for a predetermined time.

In FIG. 6, the larger the landmark, the larger the circle. The tip of the jaw, the mouth, the tip of the nose, the edge of the eyes can be judged as the most moving parts, and in FIG. 6, these parts are indicated by large circles to indicate that the moving parts are relatively high. In addition, the electronic device 100 may accurately identify the head posture by assigning a small weight to the chin tip, the mouth, the tip of the nose, and the corner of the eye which are relatively in motion.

7 is a diagram for describing a perspective n point (PnP).

The Perspective n Point (PnP) method may be a method of acquiring an R (Roation) component and a t (Translation) component under a condition in which two-dimensional location information and three-dimensional location information are mapped to each other 1: 1. When the electronic device 100 cannot have a closed form solution in the calculation process, the electronic device 100 may use an iterative calculation method. In addition, the electronic device 100 may calculate PnP in a direction of reducing the reprojection error. The electronic device 100 may estimate the head posture of the user using the calculated R component of PnP. The R component may be composed of (Roll, Pitch, Yaw) and may have a value of (0,0,0) when looking at the front.

8 is a diagram illustrating a control method of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 8, the control method of the electronic device 100 that stores a weight corresponding to a landmark representing a predefined feature point according to an embodiment of the present disclosure may include identifying a face of a user in a captured image ( 801, identifying a landmark among the identified facial feature points of the user (802), obtaining a weight corresponding to the identified landmark, and applying the obtained weight to the landmark (803) to identify the user's head posture. Step 804, where the weight is given a relatively small weight to a landmark with relatively little movement.

In operation 804 of identifying the posture of the user, the electronic device 100 may apply a weight that is relatively smaller than other landmarks to the landmarks that constitute the outline of the face among the identified landmarks of the user.

In operation 804, the head position of the user may be identified by assigning a first weight to a landmark corresponding to at least one of eyes, eyebrows, and nose having less movement, and applying a first weight to a landmark corresponding to at least one of a mouth and a chin. A second weight less than the weight may be given.

In addition, the landmark includes three-dimensional location information, and in step 804 of identifying a head posture of the user, the facial posture of the user may be identified by applying a Perspective n Point (PnP) method to the three-dimensional location information.

In operation 804, the head position of the user may be identified by acquiring a degree of movement of the landmark identified on the face of the user, and may be stored by mapping a relatively large weight to a landmark having a relatively high degree of movement.

Meanwhile, the electronic device 100 is a device provided in the vehicle and may further include tracking at least one of the driver's state and the driver's gaze based on the identified head posture of the user.

Meanwhile, the above-described methods according to various embodiments of the present disclosure may be implemented in an application form that can be installed in an existing electronic device.

In addition, the above-described methods according to various embodiments of the present disclosure may be implemented only by software upgrade or hardware upgrade of an existing electronic device.

In addition, the above-described various embodiments of the present disclosure may be performed through an embedded server provided in the electronic device or an external server of the electronic device.

Meanwhile, the electronic device control method according to the above-described embodiment may be implemented as a program and provided to the electronic device. In particular, the program including the electronic device control method may be stored and provided in a non-transitory computer readable medium.

In addition, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. According to a hardware implementation, the embodiments described in the present disclosure may include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). ), Processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions. In some cases, the embodiments described herein may be implemented in the processor 120 itself. According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing a processing operation in the electronic device according to various embodiments of the present disclosure may be stored in a non-transitory computer-readable medium. The computer instructions stored in the non-transitory computer readable medium allow the specific device to perform processing operations in the former according to the above-described various embodiments when executed by the processor of the specific device.

A non-transitory computer readable medium refers to a medium that stores data semi-permanently and is readable by a device, not a medium storing data for a short time such as a register, a cache, a memory, and the like. Specific examples of non-transitory computer readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Specifically, when executed by the processor 103 of the electronic device 100 that stores a weight corresponding to a landmark representing a predefined feature point, the electronic device 100 performs an operation. A non-transitory computer readable medium storing computer instructions for acquiring, may include identifying a face of a user in a captured image, identifying a landmark among identified facial feature points of the user, and obtaining a weight corresponding to the identified landmark. And identifying the head posture of the user by applying the obtained weight to the landmark, wherein the weight is given a relatively small weight to the landmark with relatively little movement.

While the above has been illustrated and described with respect to preferred embodiments of the present disclosure, the present disclosure is not limited to the above-described specific embodiments, and is normally made in the art without departing from the gist of the present disclosure as claimed in the claims. Various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present disclosure.

100: electronic device 101: imaging unit
102: memory 103: processor

Claims (15)

In an electronic device,
An imaging unit;
A memory in which weights corresponding to landmarks representing predefined feature points are stored;
Identify the face of the user in the image taken by the image pickup unit,
Identifying a landmark among facial feature points of the identified user;
And a processor configured to obtain a weight corresponding to the identified landmark from the memory, and apply the obtained weight to the landmark to identify a head posture of the user.
The weight stored in the memory is,
An electronic device, wherein a relatively small weight is given to a landmark having a lot of movement.
The method of claim 1,
The processor,
And applying the weight to a difference value between a landmark of the user's face and a landmark of reference face data to identify the posture of the user's head.
The method of claim 1,
The processor,
The electronic device of claim 1, wherein a weight that is relatively smaller than other landmarks is applied to a landmark constituting the outer edge of the face.
The method of claim 1,
The processor,
The first weight is given to a landmark corresponding to at least one of the less-moving eyes, the eyebrows, and the nose, and the second weight is less than the first weight to the landmark corresponding to at least one of the mouth and the jaw. Device.
The method of claim 1,
The landmark includes three-dimensional location information,
The processor,
And applying a perspective n point (PnP) method to the 3D location information to identify the posture of the user's head.
The method of claim 1,
The electronic device is a device provided in the vehicle,
The processor,
And track at least one of a driver's state and a driver's gaze based on the identified user's head posture.
The method of claim 1,
The processor,
And obtaining a movement degree of the landmark identified on the face of the user and mapping and storing a relatively large weight to a landmark having a relatively large movement degree.
A control method of an electronic device for storing a weight corresponding to a landmark representing a predefined feature point,
Identifying a face of the user in the captured image;
Identifying a landmark among the identified facial feature points of the user; And
Acquiring a weight corresponding to the identified landmark and applying the obtained weight to the landmark to identify a head posture of the user.
The weight is,
A relatively small weight is given to a landmark with a lot of movement.
The method of claim 8,
Identifying the head posture of the user,
And applying the weight to a difference value between a landmark of the user's face and a landmark of reference face data to identify the posture of the user's head.
The method of claim 8,
Identifying the head posture of the user,
And applying a weight that is relatively smaller than other landmarks to the landmarks constituting the outside of the face among the identified landmarks of the user.
The method of claim 8,
Identifying the head posture of the user,
A first weight is assigned to a landmark corresponding to at least one of the less-moving eye, an eyebrow, and a nose, and a second weight is less than the first weight to a landmark corresponding to at least one of the mouth and the chin. Control method of the device.
The method of claim 8,
The landmark includes three-dimensional location information,
Identifying the head posture of the user,
And controlling the posture of the user's head by applying a Perspective n Point (PnP) method to the 3D location information.
The method of claim 8,
The electronic device is a device provided in the vehicle,
And tracking at least one of a driver's state and a driver's gaze based on the identified head posture of the user.
The method of claim 8,
Identifying the head posture of the user,
And obtaining a degree of movement of the landmark identified on the face of the user and mapping and storing a relatively large weight to a landmark having a relatively high degree of movement.
A non-transitory computer readable medium for storing computer instructions for causing an electronic device to perform an operation when executed by a processor of an electronic device that stores a weight corresponding to a landmark representing a predefined feature point. ,
Identifying a face of the user in the captured image;
Identifying a landmark among the identified facial feature points of the user; And
Acquiring a weight corresponding to the identified landmark, and applying the obtained weight to the landmark to identify a head posture of the user.
The weight is,
A computer readable recording medium in which relatively small weights are given to landmarks having a lot of movement.

Images