KR20160110070A - Electronic apparatus, and control method therof - Google Patents

Abstract

Disclosed is an electronic apparatus. The electronic apparatus includes: a storage which stores a standard three-dimensional body model; a photographing unit which photographs a user by using a depth camera; and a processor which generates a three-dimensional body model of the user by applying a depth image, captured by the depth camera, to the standard three-dimensional body model, and extracts body information of the user from the three-dimensional body model of the user.

Description

ELECTRONIC APPARATUS AND CONTROL METHOD THEROF BACKGROUND OF THE INVENTION 1. Field of the Invention [0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an electronic apparatus and a control method thereof, and more particularly to an electronic apparatus having a photographing function and a control method thereof.

Various types of electronic devices are being developed and distributed due to the development of electronic technologies. In particular, electronic devices such as smartphones, one of the most widely used electronic products, have been developing in recent years.

[0003] As the performance of electronic devices has become higher, it has become common for an electronic device to have a camera and to perform various functions through an image obtained through a camera.

In particular, the user creates a three-dimensional body model using an image taken by the user and provides various information (for example, body information) using the generated three-dimensional body model.

However, there is a problem in that the user's three-dimensional body model generated by the existing electronic device can not be provided with the optimal service because the error is large compared with the actual user's body.

SUMMARY OF THE INVENTION The present invention has been conceived to solve the above-described problems, and it is an object of the present invention to provide a three-dimensional body model of a user that matches a body of an actual user through simple shooting, thereby providing an optimal service.

According to an aspect of the present invention, there is provided an electronic device including a storage for storing a standard three-dimensional body model, a photographing unit for photographing a user using a depth camera, And a processor for applying the depth image to the standard three-dimensional body model to generate the three-dimensional body model of the user and extracting the user's body information from the three-dimensional body model of the user.

In addition, the processor may generate a three-dimensional body model reflecting the user's body structure by matching a pose of the standard three-dimensional body model with a pose of a user included in the depth image, Dimensional body model surface based on at least a portion of the three-dimensional body model, and compensating a portion not obtained in the depth image to generate the three-dimensional body model of the user.

The processor is further configured to map the joint of the user traced from the depth image to the joint of the standard three-dimensional body model, to deform the shape of the rigid body part between the joint and the joint, Can be matched with the pose of the user included in the depth image.

Also, the processor may generate a three-dimensional body model of the user by performing resizing according to at least one of length adjustment and ratio adjustment for each rigid body part of the generated three-dimensional body model.

The processor may further include a vertex having a low reliability based on the reliability of a vertex constituting the three-dimensional body model when the three-dimensional body model is deformed along the surface of the user based on the depth information Dimensional body model of the user by moving to a position adjacent to the vertex having high reliability and compensating for parts not obtained in the depth image.

In addition, the processor may extract the kidney information of the user by applying a linear regression method to the inter-joint distance information tracked in the user's three-dimensional body model.

In addition, the processor may extract at least one of the waist circumference information and the hip circumference information of the user by applying the ASM (Active Shape Model) circumferential sectional shape fitting and the distance measurement to the user's three-dimensional body model can do.

In addition, the processor may be configured to apply mesh information to the user's three-dimensional body model to calculate volume information corresponding to the user's three-dimensional body model, and to apply volume and weight relationships to the calculated volume information And calculate at least one of weight information and body mass index information.

In addition, the processor may generate the three-dimensional body model of the user based on one depth image photographed at a single viewpoint or a plurality of depth images photographed at multiple viewpoints.

In addition, when using the plurality of depth images, the processor may generate a three-dimensional body model reflecting the user's body structure by matching a pose of the standard three-dimensional body model with a pose of a user included in the depth image , And a shape transformation is performed on the generated three-dimensional body model using an iterative closest point (ICP) based local tracking method to generate the three-dimensional body model of the user.

The display may further include a display, and the processor may display a UI screen including the extracted user's body information on the display based on the user's three-dimensional body model.

According to another aspect of the present invention, there is provided a method of controlling an electronic device, the method comprising: photographing a user using a depth camera; applying a depth image photographed by the depth camera to a previously stored standard three- Generating a three-dimensional body model, and extracting the user's body information from the three-dimensional body model of the user.

In addition, the step of generating the user's three-dimensional body model includes matching a pose of the standard three-dimensional body model with a pose of a user included in the depth image to generate a three-dimensional body model reflecting the user's body structure And modifying at least a part of the surface of the three-dimensional body model based on the depth information obtained in the depth image and generating a three-dimensional body model of the user by compensating the part not obtained in the depth image .

In addition, the step of generating the user's three-dimensional body model may include mapping the joint of the user traced from the depth image to joints of the standard three-dimensional body model, forming a rigid body part shape between the joints and joints To match the pose of the standard three-dimensional body model with the pose of the user included in the depth image.

In addition, the step of generating the user's three-dimensional body model may include performing resizing according to at least one of length adjustment and ratio adjustment for each rigid body part of the generated three-dimensional body model, Can be generated.

In addition, the step of generating the 3D body model of the user may further include: when the 3D body model is deformed along the surface of the user based on the depth information, the reliability of the vertex constituting the 3D body model Dimensional body model of the user by moving a vertex having low reliability to a position adjacent to the vertex having high reliability based on the vertex of the vertex and compensating the part not obtained in the depth image.

In addition, the step of extracting the user's body information may extract the kidney information of the user by applying a linear regression method to the inter-joint distance information tracked in the user's three-dimensional body model.

In addition, the step of extracting the user's body information may include applying an ASM (Active Shape Model) circumferential sectional shape fitting and distance measurement to the user's three-dimensional body model to calculate waist circumference information of the user and hip At least one of the circumference information can be extracted.

In addition, the step of extracting the user's body information may include the steps of: applying mesh information to the user's three-dimensional body model to calculate volume information corresponding to the user's three-dimensional body model; Weight, and body mass index information to calculate at least one of weight information and body mass index information.

The step of generating the three-dimensional body model of the user may include the step of matching the pose of the standard three-dimensional body model with the pose of the user included in the depth image when using a plurality of depth images photographed at a plurality of viewpoints, Dimensional body model reflecting the body structure of the user, and performing a shape transformation using an iterative closest point (ICP) based local tracking method on the generated three-dimensional body model, thereby obtaining the three-dimensional body model of the user And a step of generating the data.

As described above, according to the various embodiments of the present invention, the user can be provided with the optimum service by generating the three-dimensional body model of the user corresponding to the body of the actual user with only a simple photographing.

1 is a diagram showing an embodiment of an electronic device according to an embodiment of the present invention.
2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present invention.
3 is a block diagram showing a specific configuration of the electronic device shown in Fig.
4 is a block diagram illustrating a configuration stored in a storage according to an embodiment of the present invention.
5 is a diagram illustrating a process of matching a pose of a standard three-dimensional body model with a pose of a user according to an embodiment of the present invention.
6 and 7 are views for explaining a method of performing a surface modification operation based on a depth image of a user according to an embodiment of the present invention.
FIG. 8 and FIG. 9 are diagrams for explaining a compensation process for a portion where data is not acquired in the depth sensor according to an embodiment of the present invention.
10 and 11 are views for explaining a method of generating a three-dimensional body model of a user according to another embodiment of the present invention.
12 is a diagram illustrating an example of a three-dimensional body model of a user generated according to an embodiment of the present invention.
13 is a diagram for explaining a method of extracting extension information according to an embodiment of the present invention.
FIG. 14 is a diagram for explaining a method of extracting information on the waist circumference and the hip circumference according to an embodiment of the present invention.
15 is a view for explaining a method of extracting body weight information according to an embodiment of the present invention.
16 is a view showing an example of a UI screen displayed according to an embodiment of the present invention.
17 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention.

Various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing an embodiment of an electronic device according to an embodiment of the present invention.

As shown in FIG. 1, the electronic device 100 may be implemented as a digital TV having a depth camera 10, but is not limited thereto as long as it has a depth camera such as a PC, a kiosk, a tablet PC, a mobile phone, .

The electronic device 100 can construct a three-dimensional model of the user's body using one depth image taken at a single viewpoint 20 or a plurality of depth images photographed at multiple viewpoints 30 to 50. [ For this purpose, the depth camera can capture the user and acquire the depth information for the photographed user.

The electronic device 100 records various types of services such as recording the body shape that changes daily in the life log system using the user's three-dimensional body model, recommending the clothes size considering the body shape of the individual by estimating the waist circumference, leg length, .

In particular, the electronic device 100 according to the present invention generates a three-dimensional body model of a user by applying a depth image of a user to a standard three-dimensional body model, and generates a three-dimensional body model of the user, Hereinafter, the present invention will be described in more detail with reference to the drawings.

2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present invention.

2, the electronic device 100 includes a storage 110, a photographing unit 120, and a processor 130. [

The storage 110 stores a standard three-dimensional body model.

Here, the standard three-dimensional body model is a conversion of a standard body image of a person into three-dimensional modeling data according to various values. The standard three-dimensional body model may be received from the outside, It is also possible. On the other hand, the standard three-dimensional body model may be stored classified by sex, age, and race. For example, Korean adult male model, Korean adult female model, Western adult male model, and the like may be stored separately. However, the present invention is not limited to this, and may be data obtained by averaging standard three-dimensional body models of each sex, age, and race.

In addition, a standard three-dimensional body model may take an arbitrary pose, for example, taking a pose with both hands down and legs down, a pose with both hands raised, a pose with legs open, etc. .

In addition, the storage 110 may store the user's body shape change data based on the result measured by the processor 130, which will be described later.

The storage 110 may include at least one of an internal memory or an external memory. The internal memory may be, for example, a volatile memory (e.g., dynamic RAM, SRAM, or synchronous dynamic RAM (SDRAM)), a non-volatile memory (e.g., an OTPROM time programmable ROM (ROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (such as NAND flash or NOR flash) Or a solid state drive (SSD).

The external memory may be a flash drive such as a compact flash (CF), a secure digital (SD), a micro secure digital (SD), a mini secure digital (SD), an extreme digital A multi-media card (MMC), a memory stick, or the like. The external memory may be functionally and / or physically connected to the electronic device 100 via various interfaces.

The photographing unit 120 photographs a user using a depth camera. The photographing unit 120 is disposed in an outer area of the electronic device 100. [ For example, the photographing unit 120 may be disposed at an upper center, a left upper center, or a right upper central bezel region of the electronic device 100, but is not limited thereto.

The photographing unit 120 includes a lens module including a lens and an image sensor. The shape input through the lens is input as an optical signal to an image sensor serving as a film. The image sensor converts the input optical signal into an electrical signal and transmits the electrical signal to the processor 130.

In particular, the photographing unit 120 may include a depth sensor that outputs distance data between the camera and the pixel as an output value. The depth sensor can generate the depth information by detecting the depth like a human eye. The depth information is a depth value assigned to each pixel of the image, and may be a depth map type. Depth map is a table containing depth information for each area of an image. The area may be divided into a pixel unit or a predetermined area larger than a pixel unit. According to one example, the depth map may be represented by a grayscale value from 0 to 216. [

Processor 130 controls the overall operation of electronic device 100. The processor 130 may include one or more of a central processing unit (CPU), a controller, an application processor (AP), or a communication processor (CP) .

In particular, the processor 130 may generate a three-dimensional body model of the user by applying the depth image photographed by the depth camera to the standard three-dimensional body model.

To this end, the processor 130 may generate a three-dimensional body model reflecting the user's body structure by matching the pose of the standard three-dimensional body model with the pose of the user included in the depth image.

Specifically, the processor 130 maps the joint of the user traced from the depth image to the joint of the standard three-dimensional body model, and deforms the shape of the rigid body part between the joint and the joint, The pose can be matched to the pose of the user included in the depth image. In this case, the processor 130 may perform resizing by the rigid body part of the generated three-dimensional body model. For example, you can adjust the length of the rigid body part between the joints and the joints, or adjust the ratio between each rigid body part.

In addition, the processor 130 may transform at least a portion of the surface of the three-dimensional body model based on the depth data obtained in the depth image, and may generate a three-dimensional body model of the user by compensating for parts not obtained in the depth image have.

Specifically, when the three-dimensional body model is deformed along the user's surface based on the depth data from the depth image, the processor 130 determines whether the three-dimensional body model has low reliability based on the reliability of the vertex constituting the three- By moving the vertex to a position adjacent to the vertex with high reliability, it is possible to generate a three-dimensional body model of the user by compensating parts not acquired in the depth image.

Meanwhile, the processor 130 may extract the user's body information from the user's three-dimensional body model generated in the manner described above.

For example, the processor 130 may extract a user's kidney information by applying a linear regression method to the inter-joint distance information tracked in the user's three-dimensional body model. Specifically, the processor 130 may measure the height information of the user by measuring the distance between the upper body and the lower body joint, and then applying the linear regression method to the measured inter-joint distance.

As another example, the processor 130 may apply an ASM (Active Shape Model) type circumferential section shape fitting method to the user's three-dimensional body model to extract at least one of the user's waist circumference information and hip circumference information .

As another example, the processor 130 may apply the Mesh and Normal information to the user's three-dimensional body model to calculate volume information corresponding to the user's three-dimensional body model, And body weight information to calculate at least one of weight information and body mass index information.

3 is a block diagram showing a specific configuration of the electronic device shown in Fig.

Referring to FIG. 3, an electronic device 100 'includes a storage 110, a photographing unit 120, a processor 130, and a display 140. 3 will not be described in detail. As shown in FIG.

Processor 130 generally controls the operation of electronic device 100 '.

Specifically, the processor 130 includes a RAM 131, a ROM 132, a main CPU 133, a graphics processing unit 134, first through n interfaces 135-1 through 135-n, .

The RAM 131, the ROM 132, the main CPU 133, the graphics processing unit 134, the first to n interfaces 135-1 to 135-n, etc. may be connected to each other via the bus 136. [

The first to n-th interfaces 135-1 to 135-n are connected to the various components described above. One of the interfaces may be a network interface connected to an external device via a network.

The main CPU 133 accesses the storage 110 and performs booting using the O / S stored in the storage 110. [ Then, various operations are performed using various programs stored in the storage 110, contents, data, and the like.

The ROM 132 stores a command set for booting the system and the like. The main CPU 133 copies the O / S stored in the storage 110 to the RAM 131 according to the instruction stored in the ROM 132, executes the O / S, . When the booting is completed, the main CPU 133 copies various application programs stored in the storage 110 to the RAM 131, executes the application programs copied to the RAM 131, and performs various operations.

The graphic processing unit 134 generates a graphic image based on a screen including various objects such as an icon, an image, text, and the like, for example, a three-dimensional body model of the user, using an operation unit (not shown) and a rendering unit And generates a screen including the user's body information. The operation unit (not shown) calculates an attribute value such as a coordinate value, a shape, a size, and a color to be displayed by each object according to the layout of the screen based on the received control command. The rendering unit (not shown) creates screens of various layouts including the objects based on the attribute values calculated by the operation unit (not shown). A screen generated in the rendering unit (not shown) is displayed on the display 140.

Display 140 displays various information.

In particular, the display 140 may display a UI screen including body information of the extracted user based on the user's three-dimensional body model.

Meanwhile, the display 140 may be implemented as various types of displays such as a liquid crystal display, an organic light-emitting diode, a liquid crystal on silicon (LCoS), a digital light processing (DLP) have. Also, the display 140 may be implemented as a transparent display that is implemented with a transparent material and displays information.

The communication unit 150 may be connected to the communication unit 150 through a wired or wireless LAN, WAN, Ethernet, Wireless Fidelity, BT, Zigbee, Infrared, Serial Interface, Universal Serial Bus (USB) And the like, through a variety of communication methods. For example, when the standard three-dimensional body model data is not stored in the electronic device 100 ', the data is received from the external device, or when the depth camera is not provided in the electronic device 100' It is possible to receive a user photographed image.

In addition, the electronic device 100 'may further include various components such as an audio processing unit (not shown), a video processing unit (not shown), and the like according to the embodiment.

Meanwhile, the operation of the processor 130 described above may be performed by a program stored in the storage 110. [

The storage 110 may store various data such as O / S (Operating System) software modules for driving the electronic device 100 'and various multimedia contents in addition to the data including the standard three-dimensional body model as described in FIG. .

4, the storage 110 includes a communication module 111, a sensing module 112, a body model generation module 113, a body information extraction module 114) may be stored.

The communication module 111 is a module for managing communication with an external device and an external server. In particular, the communication module 111 may include a program used to receive data, such as a standard three-dimensional body model, from an external server, or to receive a user captured image from an external device.

The sensing module 112 is a module for collecting information from various sensors and analyzing and managing the collected information. In particular, the sensing module 112 may include a program for performing a function of processing a related value sensed by a depth sensor of the photographing unit 120.

The body model generation module 113 is a module for generating a user's three-dimensional body model by applying the algorithm according to the present invention to a standard three-dimensional body model and a user captured image. For example, the body model generation module 113 generates a three-dimensional body model by pose matching of a user's image with a standard three-dimensional body model through user joint tracking, reflection of depth obtained through depth information, compensation of parts not obtained in a depth image, To generate a three-dimensional body model of the user.

The body information extraction module 114 is a module for extracting the user's body information from the three-dimensional body model of the user generated through the user's three-dimensional body model generation module 123. For example, the body information extraction module 124 may include an algorithm for applying a linear regression method to a user's three-dimensional body model to extract a user's height information, an algorithm for calculating a user's three- And may include an algorithm that applies an ASM (Active Shape Model) circumferential section shape fitting to the body model.

As such, the processor 130 may utilize various software modules stored in the storage 110 to provide functionality in accordance with the present invention.

Hereinafter, a three-dimensional body model generation method and a body information extraction method using the same according to the present invention will be described in detail with reference to the drawings.

5 is a diagram illustrating a process of matching a pose of a standard three-dimensional body model with a pose of a user according to an embodiment of the present invention.

As shown in FIG. 5, joint points corresponding to joints of the user to be traced are designated by using a depth sensor in the standard three-dimensional body model 530. In this case, the joint mapping can be performed based on the image 520 shown by simplifying the joint positions and the rigid body parts between the joints obtained from the depth image 510 of the user.

The rigid body part is then indexed according to the relationship between the joints and the joints, and each point on the surface of the standard three-dimensional body model is indexed according to the rigid body part.

Then, the joint of the actual user is tracked by using the depth sensor. Calculate the transformation matrix of the rigid body part using the actual joints being tracked and the joints inside the standard three-dimensional body model. In this case, different matrices can be calculated for each rigid part.

The vertices of the surface of the standard three-dimensional body model are then transformed using a matrix according to the index. Once each rigid body part is transformed according to a matrix, a matched model 540 is created that matches the pose of the actual user being tracked.

FIGS. 6 and 7 are views for explaining a method of performing a surface deforming operation based on a depth image of a user according to an embodiment of the present invention.

When the process described with reference to FIG. 5 is completed, the surface 540 is partially transformed based on the depth image 510 as shown in FIG. 6 to transform the model 540 generated in FIG. 5 into a real person's shape 610 Perform the operation. In this case, it is assumed that the 2D depth image space and the calibration of the 3D camera space have already been performed correctly.

의 위치로 움직여야 한다.

의 위치는,

와 가장 가까운 표준 모델상의 Joint

와

를 지나는 라인

과 뎁스 이미지상의 surface가 교점을 이루는 점을 찾으면 구할 수 있다. 그러나 이 방법으로 교점을 계산하는 데 많은 계산량이 소요되므로 근사 방법을 사용한다.The method of deforming the surface of the standard model using the obtained depth image is as follows. As shown in the image 710 of FIG. 7, each Point on the standard model is associated with a Point on the depth image

To the position of.

Lt; / RTI >

Joint on the nearest standard model

Wow

Line passing through

And the surface on the depth image intersect each other. However, since this method requires a lot of computation to calculate the intersection, we use the approximation method.

를 초기 위치로 하여, 수학식 1을 이용하여

라인 상의 특정 위치로

의 위치를 이동시킨다.이를 반복적으로 수행하여, 수렴되는 위치를

가 이동해야 할

의 위치로 정한다. 매 반복마다, 모델 상의 포인트

은 다음 근사 위치

로 이동된다. 여기서

은 포인트

과 매핑되는 뎁스 이미지상 위치에서의 뎁스 값이며 α는 라인

의 기울기를 나타낸다. 반복의 종료 여부는 수학식 2로 평가된다. 도 7의 720 이미지는 반복에 의해

의 위치가 근사화되는 과정을 보여준다.The approximate method is as follows.

Is defined as an initial position, using Equation (1)

To a specific location on the line

Is moved repeatedly, and the position converged

Should go

. For each iteration,

The next approximate position

. here

Point

Is a depth value at a position on a depth image mapped to a line

. The end of the iteration is evaluated by the following equation (2). The image 720 of FIG. 7 is repeated

And the position of the target is approximated.

8 and 9 are diagrams for explaining a compensation process for a portion where data is not acquired in the depth sensor according to an embodiment of the present invention.

As shown in Fig. 8, the compensation process is performed by using a portion (data portion) obtained by obtaining data for a portion where the data can not be acquired by the depth sensor and is not deformed (gray region in the image 810 in Fig. 8, 8), it is possible to completely generate the user's three-dimensional body model by performing the transformation using the information of the white area (hereinafter, seen part) in the image 810 of FIG. Hereinafter, a method of compensating for the unseen part will be described with reference to FIG.

First, all the vertices on the model have a confidence value that indicates the reliability of the position in the three-dimensional space that determines the shape. The initial value of the confidentiality can be set to 1, and the unseen part can be set to approximately 0, as shown in Fig. That is, the vertex obtained by the camera has the highest reliability, and the vertex, which is invisible by the camera and maintains the shape of the initial model, has the lowest reliability. Using this confidence, an energy function is created to generate motion momentum so that all the vertices on the model can move to the optimum position in the three-dimensional space. Confidence propagation is performed so that energy is minimized to move the vertices To deform the model surface.

The energy function E for a particular vertex p can be composed of two energy functions Ec and Es as shown in Equation 3 below.

First, Ec is the energy for the specific vertex p and Es is the energy related to the smoothness of the model surface shape for a particular vertex p.

The energy function Ec can be expressed by the following equation (4).

는 p의 8개의 이웃 포인트들 중i번째 이웃 포인트이며,

는 그 이웃 포인트의 컨피던스를 나타낸다. 모든 포인트들의 컨피던스는

이라고 가정한다. 즉 본 계산식은 이웃한 버텍스까지의 거리에 반비례하는 가중치를 곱한 컨피던스값들의 합을 계산하여 이를 최소화 한다. 다시 말하여, 이웃한 버텍스들중 높은 컨피던스를 가지는 점들과의 거리를 줄여나가는 방향으로 현재 버텍스를 움직이도록 한다.here,

Is the i < th > neighboring point of the eight neighboring points of p,

Represents the confidence of the neighbor point. The confidence of all points is

. In other words, this formula minimizes the sum of the confidence values multiplied by the weight inversely proportional to the distance to the neighboring vertex. In other words, it moves the current vertex in a direction that reduces the distance from the neighboring vertices with high confidence.

Further, the energy function Es can be expressed by the following equation (5).

에서 이웃한 두 버텍스와 형성하는 사잇각도 값 대한 2차 미분값의 합을 모두 더한 값이 에너지에 해당한다. 이는 각 이웃 버텍스에서 다시 주변 점들과의 곡률을 계산하고 이를 모두 더한 값에 해당한다. 즉 현재 버텍스가 이웃한 버텍스들과 만드는 사잇각의 변화가 가장 적은 위치, 즉 주변 버텍스들이 가지는 곡률을 유지하는 위치로 현재 버텍스를 이동 하도록 한다. 에너지 함수는 신뢰도(Confidence)를 가중치로 사용하고 있는데, 반복해서 신뢰도를 전파할 때 마다 신뢰도가 새롭게 갱신되어 에너지 함수에도 영향을 끼친다. 이 경우, 신뢰도의 갱신은 다음 수학식 5와 같이 이루어진다. Where < RTI ID = 0.0 > is a weight for the confidence of the neighboring point.

The sum of the values of the secondary differential values of the angle of incidence formed with the two adjacent vertices is added to the energy. This computes the curvature with the neighboring vertices again at each neighbor vertex and adds up to the sum. That is, the current vertex moves the current vertex to the position where the change of the angle of making with the adjacent vertices is the lowest, that is, the position maintaining the curvature of the surrounding vertices. The energy function uses the confidence as a weight, and each time the reliability is repeatedly propagated, the reliability is renewed to affect the energy function as well. In this case, the reliability is updated according to the following equation (5).

All vertices with low confidence are moved repeatedly to minimize this energy function. In this case, iteration can be restricted to perform only when the sum of the confidences of the vertex's neighboring points is less than?.

10 and 11 are views for explaining a method of generating a three-dimensional body model of a user according to another embodiment of the present invention.

Referring to FIGS. 10 and 11, a three-dimensional body model of a user can be generated by using a plurality of depth-depth images, unlike in FIGS. Here, the multi-view image can be obtained by photographing a moving user in real time without taking a fixed posture and continuously taking an arbitrary posture.

When a plurality of depth-depth images are used, some processes are similar to the case of using one depth image described in FIGS. 5 to 9, but some processes may be different.

Specifically, the pose matching process described with reference to FIG. 5 is applied to the same embodiment, so a detailed description will be omitted.

In the case of using a plurality of depth-depth images, as shown in FIG. 10, a pose matching (the same as shown in FIG. 5) is performed in one step and then an ICP-based local shape matching method is further used in a two- A three-dimensional body model can be created.

Specifically, using the ICP-based local shape matching method for each body part, the surface according to the depth image of the user can be applied to the model generated through the procedure of FIG. 5 to modify the model shape and restore the joint position . In addition, the process shown in Fig. 11 can be performed for each of a plurality of photographed multi-view images.

Here, an iterative closest point (ICP) is used to estimate the displacement between the three-dimensional points corresponding to the human body obtained from the depth image and the vertex points of the model. Dimensional vertex points of the model for each body part are matched so as to have a minimum distance difference from the 3D points of the corresponding part obtained from the depth image. Through this process, the body movement can be tracked by reflecting the shape change due to the more detailed body movement which is not reflected in the pose registration step.

12 is a view showing an example of a three-dimensional body model of a user generated according to another embodiment of the present invention.

According to another embodiment of the present invention, the body information of the user can be measured using the nude model of the user matched using the predetermined deformation parameter.

Specifically, a three-dimensional body model of the user obtained through the above-described method may be used. However, in order to solve the problem that a shape other than a body such as a clothes or a bag is reflected, a new nude model is matched May be used. That is, a nude model capable of only limited modification by the change parameter of the human body part is additionally matched to the three-dimensional body model obtained through the pose matching step, and then the body information is measured using the matched nude model Can be used. Here, the nude model is not deformed by the depth information of the input image as described above, but is deformed by the deformation parameters of each body as shown in Fig. 12, and by this deformation, shapes other than the body such as clothes, It is possible to extract only pure body shape information that does not affect the body shape information.

When the user's three-dimensional body model is generated as shown in FIG. 12, various body information can be extracted based on the generated three-dimensional body model. For example, body information such as height, waist circumference, hip circumference, abdominal obesity, body weight, body mass index, chest circumference, shoulder width, thigh circumference, and arm circumference can be extracted.

Hereinafter, some body information extraction methods will be described with reference to the drawings.

13 is a diagram for explaining a method of extracting extension information according to an embodiment of the present invention.

As shown in FIG. 13, it is possible to determine the upper and lower body joint groups through joint tracking, and to extract extension and partial extension information by measuring the distance between the upper body and lower joint, and to extract the body information such as the upper / can do.

However, it is possible to extract the kidney information of the user by applying a linear regression method to the inter-joint distance measured according to an embodiment of the present invention. That is, since the joint distance may be inaccurate due to the noise, the y-axis sum of the measured inter-joint distance is provided as an input to the linear regression method, thereby enabling more accurate measurement of the elongation.

In this case, extension and partial extension information extraction is possible, and body information such as upper / lower body ratio information can be extracted.

FIG. 14 is a diagram for explaining a method of extracting information on the waist circumference and the hip circumference according to an embodiment of the present invention.

As shown in FIG. 14, various body circumference information can be calculated through a circumferential sectional shape fitting of an active scissor system and distance measurement.

Specifically, as shown in the figure, a circular model is created at a portion of the user's three-dimensional body model where the circumference is to be measured, for example, around the waist, and the points constituting the corresponding circular model are set as the shortest After moving to the distance surface, the waist circumference can be calculated by calculating the distance between the circular model points.

In this way, various body circumference information such as waist circumference A (for WHR (Waist Hip Ratio)), waist circumference B (bottom size), hip circumference (for WHR), abdominal obesity can be calculated. In this case, the abdominal obesity degree can be calculated by a method such as (waist circumference) / (hip circumference). In addition, various body information such as the chest circumference, shoulder width, thigh circumference, and arm circumference can be calculated by a similar method.

15 is a view for explaining a method of extracting body weight information according to an embodiment of the present invention.

As shown in FIG. 15, the volume of the three-dimensional body model of the user can be calculated using the tetrahedral volume accumulation method using the Mesh and Normal information. Thereafter, body and partial body weight can be calculated using the volume and weight relationship. In addition, the body mass index (BMI) can be calculated through an expression such as "weight / (height ^ 2)".

16 is a view showing an example of a UI screen displayed according to an embodiment of the present invention.

As shown in FIG. 16, a UI screen including a user's three-dimensional body model and various body information extracted based thereon may be provided through a display.

17 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention.

According to the method of controlling an electronic device according to an embodiment of the present invention shown in Fig. 17, first, a user is photographed using a depth camera (S1710).

Then, the depth image photographed by the depth camera is applied to the pre-stored standard three-dimensional body model to generate the user's three-dimensional body model (S1720).

Thereafter, the user's body information is extracted from the user's three-dimensional body model (S1730).

The step S1720 of generating the user's three-dimensional body model includes the steps of generating a three-dimensional body model reflecting the user's body structure by matching the pose of the standard three-dimensional body model with the pose of the user included in the depth image, Modifying at least a part of the surface of the three-dimensional body model based on the depth information obtained from the depth image, and generating a three-dimensional body model of the user by compensating the part not obtained in the depth image.

In addition, in step S1720 of generating the user's three-dimensional body model, the joint of the user traced from the depth image is mapped with the joint of the standard three-dimensional body model, and the shape of the rigid body part between the joints and joints is transformed So that the pose of the standard three-dimensional body model can be matched with the pose of the user included in the depth image.

In addition, the step of generating the user's three-dimensional body model may generate a three-dimensional body model of the user by performing resizing for each rigid body part of the generated three-dimensional body model.

If the three-dimensional body model is deformed along the surface of the user based on the depth information, based on the reliability of the vertex constituting the three-dimensional body model, Moving a vertex with low reliability to a location adjacent to a vertex with high reliability can compensate for parts that are not acquired in the depth image.

In addition, in step S1730 of extracting the body information of the user, the kidney information of the user can be extracted by applying a linear regression method to the inter-joint distance information tracked in the user's three-dimensional body model.

In step S1730 of extracting the user's body information, a peripheral section shape fitting and a distance measurement of the ASM (Active Shape Model) method are applied to the user's three-dimensional body model to calculate the waist circumference information and the hip circumference information At least one of them can be extracted.

In addition, in step S1730 of extracting the user's body information, mesh information corresponding to the user's three-dimensional body model is calculated to calculate volume information corresponding to the user's three-dimensional body model, Weight relationship and weight information and body mass index information can be calculated.

If a plurality of depth images photographed at a plurality of points are used, the pose of the standard three-dimensional body model is matched with the pose of the user included in the depth image to generate the three-dimensional body model of the user, A step of generating a three-dimensional body model reflecting the body structure, and a step of deforming the generated three-dimensional body model using an iterative closest point (ICP) based local tracking method to generate a three-dimensional body model of the user .

As described above, according to various embodiments of the present invention, it is possible to provide an optimal service by creating a three-dimensional body model of a user that matches the body of an actual user with only a simple photographing operation.

The three-dimensional body model of the user generated according to the above-described method is a service for recording a body shape that changes every day, a body size estimation such as a waist circumference or a leg length, and a service for recommending clothes dimensions considering an individual body shape .

The control method of the electronic device according to the above-described various embodiments may be implemented by a program and stored in various recording media. That is, a computer program, which is processed by various processors and can execute the various control methods described above, may be stored in the recording medium.

For example, a step of generating a three-dimensional body model of a user by applying a depth image photographed by a depth camera to a pre-stored standard three-dimensional body model, and extracting a user's body information from the user's three- A non-transitory computer readable medium may be provided.

A non-transitory readable medium is a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is readable by the apparatus. In particular, the various applications or programs described above may be stored on non-volatile readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM,

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

100: electronic device 110: storage
120: photographing unit 130: processor

Claims (20)

Storage to store standard three-dimensional body models;
A photographing unit photographing a user using a depth camera; And
And a processor for applying the depth image photographed by the depth camera to the standard three-dimensional body model to generate the three-dimensional body model of the user and extracting the user's body information from the three-dimensional body model of the user Device. The method according to claim 1,
The processor comprising:
Dimensional body model reflecting the body structure of the user by matching a pose of the standard three-dimensional body model with a pose of a user included in the depth image, and generating a three-dimensional body model based on the depth information obtained in the depth image, Deforms at least a portion of the body model surface and compensates for parts not obtained in the depth image to generate the three-dimensional body model of the user. 3. The method of claim 2,
The processor comprising:
Mapping a joint of the user traced from the depth image to a joint of the standard three-dimensional body model, and deforming a shape of a rigid body part between the joint and the joint, Is matched with a pose of a user included in the electronic device. The method of claim 3,
The processor comprising:
Wherein the controller generates the three-dimensional body model of the user by performing resizing according to at least one of length adjustment and ratio adjustment for each rigid body part of the generated three-dimensional body model. 3. The method of claim 2,
The processor comprising:
If the three-dimensional body model is deformed along the surface of the user based on the depth information, the vertex having low reliability based on the reliability of the vertex constituting the three-dimensional body model is referred to as a vertex having high reliability Dimensional body model of the user by moving to an adjacent position and compensating for parts not obtained in the depth image. The method according to claim 1,
The processor comprising:
And extracting the kidney information of the user by applying a linear regression method to the inter-joint distance information tracked in the user's three-dimensional body model. The method according to claim 1,
The processor comprising:
Wherein at least one of the waist circumference information and the hip circumference information of the user is extracted by applying an ASM (Active Shape Model) circumferential sectional shape fitting and distance measurement to the user's three-dimensional body model Device. The method according to claim 1,
The processor comprising:
The mesh information is applied to the user's three-dimensional body model to calculate volume information corresponding to the user's three-dimensional body model, and volume and weight relationship is applied to the calculated volume information to calculate weight information and body mass index Information of at least one of the plurality of devices. The method according to claim 1,
The processor comprising:
Dimensional body model of the user based on a depth image taken at a single viewpoint or a plurality of depth images taken at multiple viewpoints. 10. The method of claim 9,
The processor comprising:
Dimensional body model reflecting the user's body structure by matching a pose of the standard three-dimensional body model with a pose of a user included in the depth image when the plurality of depth images are used, Wherein the shape transformation is performed on the body model using an iterative closest point (ICP) based local tracking method to generate the three-dimensional body model of the user. The method according to claim 1,
Further comprising a display,
The processor comprising:
And displays the UI screen including the extracted user's body information on the display based on the three-dimensional body model of the user. A method of controlling an electronic device,
Capturing a user using a depth camera;
Generating a three-dimensional body model of the user by applying a depth image photographed by the depth camera to a pre-stored standard three-dimensional body model; And
And extracting the user's body information from the three-dimensional body model of the user. 13. The method of claim 12,
Wherein the generating the three-dimensional body model of the user comprises:
Matching a pose of the standard three-dimensional body model with a pose of a user included in the depth image to generate a three-dimensional body model reflecting the body structure of the user; And
Modifying at least a part of the surface of the three-dimensional body model based on the depth information obtained in the depth image and generating a three-dimensional body model of the user by compensating the part not obtained in the depth image . 14. The method of claim 13,
Wherein the generating the three-dimensional body model of the user comprises:
Mapping a joint of the user traced from the depth image to a joint of the standard three-dimensional body model, and deforming a shape of a rigid body part between the joint and the joint, Is matched with a pose of a user included in the control unit. 15. The method of claim 14,
Wherein the generating the three-dimensional body model of the user comprises:
Wherein the three-dimensional body model of the user is generated by performing resizing according to at least one of length adjustment and ratio adjustment for each rigid body part of the generated three-dimensional body model. 14. The method of claim 13,
Wherein the generating the three-dimensional body model of the user comprises:
If the three-dimensional body model is deformed along the surface of the user based on the depth information, the vertex having low reliability based on the reliability of the vertex constituting the three-dimensional body model is referred to as a vertex having high reliability Dimensional body model of the user is generated by compensating a part that is not acquired in the depth image by moving to an adjacent position. 13. The method of claim 12,
The step of extracting the user's body information comprises:
And extracting the kidney information of the user by applying a linear regression method to the inter-joint distance information tracked in the user's three-dimensional body model. 13. The method of claim 12,
The step of extracting the user's body information comprises:
Wherein at least one of the waist circumference information and the hip circumference information of the user is extracted by applying an ASM (Active Shape Model) circumferential sectional shape fitting and distance measurement to the user's three-dimensional body model Way. 13. The method of claim 12,
The step of extracting the user's body information comprises:
The mesh information is applied to the user's three-dimensional body model to calculate volume information corresponding to the user's three-dimensional body model, and volume and weight relationship is applied to the calculated volume information to calculate weight information and body mass index Wherein the control unit calculates at least one of the information. 12. The method of claim 11,
Wherein the generating the three-dimensional body model of the user comprises:
Generating a three-dimensional body model in which a body structure of the user is reflected by matching a pose of the standard three-dimensional body model with a pose of a user included in the depth image, when using a plurality of depth images photographed at multiple viewpoints; And
And generating a three-dimensional body model of the user by performing shape transformation using an iterative closest point (ICP) based local tracking method on the generated three-dimensional body model.

Images