KR20220063847A - Electronic device for identifying human gait pattern and method there of - Google Patents

Abstract

The present invention relates to a method for identifying the walking pattern of a target and an electronic device performing the same. In particular, a method for a robot device to identify the walking pattern of a target according to one embodiment of the present invention may comprise: a step of filming a target by means of a camera connected to the robot device to acquire an image; a step of generating human body skeleton information from the acquired image; a step of identifying the filming angle of the camera with respect to the target based on the connecting relationship of at least one link in the generated human body skeleton information; a step of determining the joint information of the robot model based on the identified filming angle of the camera; and a step of identifying the walking pattern of the target by performing a posture simulation of a humanoid robot based on the determined joint information. Therefore, the method for identifying the walking pattern of a target by using the humanoid robot model and the electronic device performing the same may be provided.

Description

A method for identifying a subject's gait pattern and an electronic device for performing the same

The present disclosure relates to a method for identifying a gait pattern of a subject and an electronic device for performing the same. More particularly, it relates to a method of identifying a gait pattern of a subject using a humanoid robot model, and an electronic device for performing the same.

In hospitals or nursing homes, gait pattern technologies for automatic health measurement and monitoring necessary for medical care are being developed. In particular, human motion recognition or posture estimation techniques are being performed by analyzing images or images, and are being actively studied based on various fusion techniques.

In addition, by analyzing images or images, research on 2D-dimensional posture analysis and 3D-dimensional posture analysis is being actively conducted. This is done by estimating my joint angle, etc.

On the other hand, artificial intelligence technology is used in various fields such as image processing and voice processing as a system that learns by itself and becomes smart unlike the conventional rule-based system. Recently, a deep neural network (DNN) algorithm is used in various machine learning fields, and performance is greatly improved.

In order to analyze a person's posture, techniques for analyzing a person's gait pattern, posture, etc. have been studied using a human body model that imitates a human body model. Accordingly, there is a demand for technology development for more accurately identifying a person's gait pattern or posture.

Korean Patent Publication No. 2020-0084567

According to an embodiment, a method for an electronic device to identify a gait pattern of a subject and an electronic device for performing the same may be provided.

According to an embodiment, a method for identifying a gait pattern of a subject using a humanoid robot model and an electronic device for performing the same may be provided.

According to an embodiment of the present disclosure for achieving the above-described technical problem, a method for an electronic device to identify a gait pattern of a subject includes: acquiring an image by photographing the subject through a camera connected to the electronic device; generating human skeleton information from the acquired image; identifying a photographing angle of the camera with respect to the subject based on a connection relationship of at least one link in the generated human skeleton information; determining joint information of the robot model based on the photographing angle of the camera; and identifying a gait pattern of the subject by executing a humanoid robot posture simulation based on the determined joint information; may include

In addition, according to another embodiment of the present disclosure for solving the above technical problem, an electronic device for identifying a gait pattern of a subject includes: a network interface; a memory storing one or more instructions; and at least one processor; including, wherein the at least one processor obtains an image by photographing the subject through a camera connected to the electronic device by executing the one or more instructions, generates human skeleton information obtained from the image, and generates the Identifies the photographing angle of the camera for the subject based on the connection relationship of at least one link in the human skeleton information, determines the joint information of the robot model based on the photographing angle of the camera, and the determined joint information It is possible to identify the subject's gait pattern by executing a posture simulation of the humanoid robot based on the

In addition, according to another embodiment of the present disclosure for solving the technical problem, in a method for an electronic device to identify a subject's gait pattern, acquiring an image by photographing the subject through a camera connected to the electronic device step; generating human skeleton information from the acquired image; identifying a photographing angle of the camera with respect to the subject based on a connection relationship of at least one link in the generated human skeleton information; determining joint information of the robot model based on the photographing angle of the camera; and identifying a gait pattern of the subject by executing a humanoid robot posture simulation based on the determined joint information; A computer-readable recording medium in which a program for performing a method including a method is stored may be provided.

1 is a diagram schematically illustrating a process in which an electronic device identifies a gait pattern of a subject according to an exemplary embodiment.
2 is a diagram illustrating a geometric relationship between adjacent coordinate systems using a DH method according to an exemplary embodiment.
3 is a view for explaining a joint-link model of the lower body of the robot model according to an embodiment.
4 is a view for explaining three planes of the human body according to an embodiment.
5 is a view for explaining a joint-link model of the upper body of the robot model according to an embodiment.
6 is a diagram for explaining a joint recognized by a neural network model and a link connecting the joints according to an embodiment.
7 is a diagram for a relationship when a three-dimensional object is projected on a coronal plane and a sagittal plane, according to an embodiment.
8 is a diagram for explaining an angular relationship between a person and a camera.
9 is a view for explaining an angle between two links according to an embodiment.
10 is a view for explaining an angle between two links according to another embodiment.
11 is a view for explaining an angle between two links according to another embodiment.
12 is a diagram for describing a skeletal model recognized by a neural network model when a person faces the front or the right, according to an embodiment.
13 is a diagram for describing a skeletal model recognized by a neural network model when a person faces right, front, and left according to an embodiment.
14 is a diagram for describing a neural network skeletal model recognized when a person faces right and left, according to an embodiment.
15 is a view for explaining an angle between a face link and a body link, according to an embodiment.
16 is a diagram illustrating a skeletal model of a human body displayed in pixel coordinates, according to an exemplary embodiment.
17 is a view showing a joint-link model of the robot model lower body standing according to an embodiment.
18 is a diagram for calculating a right hip coronal angle according to an embodiment.
19 is a diagram for calculating the angle of the coronal plane of the left hip joint according to an embodiment.
20 is a view for explaining a joint-link model of the upper body of the robot model according to an embodiment.
21 is a diagram for calculating right and left shoulder joint angles according to an embodiment.
22 is a diagram illustrating a skeletal model of a human body indicated by pixel coordinates observed from the left, according to an exemplary embodiment.
23 is a diagram illustrating a neural network skeletal model and joint angles of right and left legs recognized by a neural network according to an exemplary embodiment.
24 is a diagram for calculating a right hip joint angle according to an exemplary embodiment.
25 is a diagram for calculating a right knee joint angle according to an embodiment.
26 is a diagram for calculating a left hip joint angle according to an embodiment.
27 is a diagram for calculating a left knee joint angle according to an embodiment.
28 is a view for explaining a link and a joint of a right arm, a link and a joint of a right shoulder, and a link and a joint of a right elbow, according to an embodiment.
29 is a diagram for describing a link and joints of a left arm, according to an exemplary embodiment.
30 is a diagram illustrating a skeletal model of a human body displayed in pixel coordinates when a person walks toward the left, according to an exemplary embodiment.
31 is a view for explaining skeletal models and joint angles of a right leg and a left leg, according to an exemplary embodiment.
32 is a diagram for describing links and joints of a right arm and a left arm, according to an exemplary embodiment.
33 is a diagram for describing a relationship between an image size and a distance to a camera when a subject faces a camera, according to an embodiment.
34 is a flowchart of a method for an electronic device to identify a gait pattern of a subject, according to an embodiment.
35 is a diagram illustrating a process in which an electronic device generates a robot model, according to an exemplary embodiment.
36 is a diagram illustrating a process in which an electronic device identifies a photographing angle of a camera with respect to a subject, according to an exemplary embodiment.
37 is a diagram for describing a process in which an electronic device identifies a gait pattern of a subject according to this embodiment.
38 is a block diagram of an electronic device according to an embodiment.
39 is a block diagram of an electronic device according to another embodiment.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the present disclosure have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In the entire specification, when a part "includes" a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

In addition, throughout the specification, the abbreviation hp means hip, kn means knee, an means ankle, sh means shoulder, and el means elbow. ), hd means head, l means left, and r means right.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram schematically illustrating a process in which a job management system manages a target job according to an embodiment.

According to an embodiment, the electronic device 1000 may acquire the images 102 , 104 , and 106 by photographing the subject, and may identify the subject's gait pattern included in the acquired image. According to an embodiment, the electronic device 1000 includes a processor 120 and a memory 140 , and the memory 140 includes information on the pre-built humanoid robot model 142 and information on the neural network model 144 . may contain information.

For example, the electronic device 1000 may acquire images 102 , 104 , and 106 of the subject. According to an embodiment, the electronic device 1000 may acquire images 102 , 104 , and 106 from the camera device by being connected to the camera device. However, according to another embodiment, the electronic device 1000 may include a camera device, and may acquire images 102 , 104 , and 106 of a subject through the camera device. For example, the electronic device 1000 obtains human skeleton information (including joint information and link information) by inputting an image into a pre-stored neural network model, and based on the obtained human skeleton information, a camera shooting angle for the subject , joint angle, and information on the length of the link connecting the joint can be identified. The electronic device 1000 may determine various gait information 162 such as a gait speed and step length of the subject based on the simulation of the robot model based on the obtained human skeleton information. The electronic device 1000 according to the present disclosure may identify the subject's gait pattern 164 based on at least one of joint information, link information, gait speed, and step length.

According to an embodiment, the electronic device 1000 according to the present disclosure may identify the gait pattern of the subject by interworking with the server 2000 . According to an embodiment, the electronic device 1000 may interwork with the server 2000 through a network, and the network includes a local area network (LAN), a wide area network (WAN), and a value-added communication network ( It may include a combination of at least one of a Value Added Network (VAN) and a mobile radio communication network. In addition to the above-described server 2000, the electronic device 1000 according to the present disclosure may include a data communication network in a comprehensive sense for communicating with various network constituent entities, and a wired Internet, a wireless Internet, and a mobile wireless communication network itself. .

According to an embodiment, the neural network model 144 used by the electronic device 1000 may be an OpenPose model. The electronic device 1000 may display the 3D human body motion by overlaying it on the 2D camera image plane using the open pose model. According to an embodiment, the human skeleton information may include joint information and link information about a link connecting the joints. According to an embodiment, the joint information may include information on joint angles and joint identification numbers, and the link information may further include information on the length of the link. According to an embodiment, the information on the angle between the links may be included in the joint information.

Also, according to an embodiment, the electronic device 1000 may identify the posture of the subject in the image by further using other artificial intelligence models in addition to the open pose model. According to an embodiment, the artificial intelligence model used by the electronic device 1000 may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a recurrent neural network (RNN), It may include, but is not limited to, a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Networks.

2 is a diagram illustrating a geometric relationship between adjacent coordinate systems using a DH method according to an exemplary embodiment.

동차변환행렬(homogeneous transformation matrix)의 곱을 이용해서 로봇의 링크나 몸체에 부착된 좌표계를 이전 링크의 좌표계로부터 체계적으로 구축하는 방법이다. DH법은 링크와 링크가 서로 연결되어 있으면 대부분의 로봇에 적용할 수 있으므로, 기준좌표계

를 기준으로 임의의 링크 좌표계

나 말단부 좌표계

의 원점과 방향도 순차적으로 구할 수 있다.The DH method is a method proposed by Denavit and Hartenberg in 1955 to express the relationship between rotation and movement between adjacent coordinate systems of a robot.

It is a method to systematically build the coordinate system attached to the link or body of the robot from the coordinate system of the previous link by using the product of a homogeneous transformation matrix. Since the DH method can be applied to most robots if the link and the link are connected to each other, the reference coordinate system

Any link coordinate system based on

I distal coordinate system

The origin and direction of can also be found sequentially.

번째 관절이

번째 링크와

번째 링크를 연결하고, 각 링크에

좌표계와

좌표계가 부착되어 있다고 가정한다.

번째 관절의 액추에이터가 회전이나 이동을 하면 후단에 연결된 i번째 링크와

좌표계가 이에 반응해서 움직인다. 도 2에서 네 개의 DH 파라미터

(202, 204, 206, 208)는 DH법에서 규정한 파라미터로서 각각 관절각(joint angle), 링크 비틀림(link twist), 링크 오프셋(link offset), 링크 길이(link length)를 나타낸다. DH변환행렬

는 후행승산(post-multiply) 기법을 적용하여 하기 수학식 1과 같이 나타날 수 있다.As shown in Figure 2 to understand the DH method,

the second joint

second link and

Connect the second link, and in each link

coordinate system and

Assume that a coordinate system is attached.

When the actuator of the th joint rotates or moves, the i-th link connected to the rear end

The coordinate system moves in response to this. The four DH parameters in Figure 2

(202, 204, 206, 208) are parameters stipulated by the DH method and indicate a joint angle, a link twist, a link offset, and a link length, respectively. DH transformation matrix

can be expressed as in Equation 1 below by applying a post-multiply technique.

와

는 sin 함수와 cos 함수를 각각 나타내며, 위첨자와 아래첨자는 DH 파라미터와 그 인덱스를 나타낸다. 즉,

는

를 나타내고

는

를 의미한다. 그리고 Rot은 좌표계 회전 행렬을, Trans는 좌표계 이동 행렬을 의미한다. Capital letters in Equation 1 above

Wow

denotes the sin function and the cos function, respectively, and the superscript and subscript denote the DH parameter and its index. in other words,

Is

to indicate

Is

means And Rot means a coordinate system rotation matrix, and Trans means a coordinate system movement matrix.

3 is a view for explaining a joint-link model of the lower body of the robot model according to an embodiment.

좌표계는 왼쪽 고관절에 위치하고, 세계좌표계(world coordinate frame)인

좌표계는 통상적인 방식대로

축은 로봇의 앞쪽 방향을,

축은 로봇의 오른쪽에서 왼쪽을 향하는 측면 방향을,

축은 위쪽을 향하는 방향으로 설정될 수 있다. 도 3에서 휴머노이드 로봇의 하체는 2개의 발바닥면과 7개의 링크(

), 6개의 시상면(sagittal plane) 관절(

), 4개의 관상면(coronal plane) 관절, 2개의 횡평면(transversal plane) 관절로 구성되어 있음을 알 수 있다.3 shows the lower body when a robot model (eg, a humanoid robot) is standing with links and joints, and each link coordinate system follows the parameter regulation method of the DH method. a reference coordinate frame

The coordinate system is located in the left hip joint, and the world coordinate frame is

The coordinate system is in the usual way

The axis is the front direction of the robot,

The axis is the lateral direction from right to left of the robot,

The axis may be set in an upward direction. 3, the lower body of the humanoid robot has two soles and seven links (

), six sagittal plane joints (

), four coronal plane joints, and two transversal plane joints.

4 is a view for explaining three planes of the human body according to an embodiment.

According to an embodiment, the human body may be divided into three planes (eg, a movement surface of the human body). For example, the coronal plane 404 may be a plane that divides the human body into anterior and posterior by a coronal plane. In addition, the sagittal plane 402 may be a sagittal plane or a median plane that divides the body into left and right sides based on the center. In addition, the Transverse Plane 406 divides the body based on the upper/lower plane, and may be a horizontal plane or a transverse plane. The electronic device 1000 may determine joint information or link information for three planes of the human body.

에서 하체 링크 상의 임의의 점을 측정할 때의 좌표를 계산하는 과정을 설명한다. 먼저 왼쪽 고관절에서 횡평면 관절인

회전에 의해 변환된

좌표계 상의 한 점

을

좌표계에서 보았을 때의 확장벡터

은 수학식 1의 DH행렬

를 이용하여 다음과 같이 표현할 수 있다.Now with reference to FIGS. 2 to 4, the reference coordinate system located in the left hip joint of the lower body

Describes the process of calculating the coordinates when measuring any point on the lower body link. First, the transverse plane joint in the left hip joint

transformed by rotation

a point on the coordinate system

second

Expansion vector as seen in the coordinate system

is the DH matrix of Equation 1

can be expressed as follows using

벡터에서 위첨자 0은 기준좌표계가 0번째(

) 좌표계이고 아래첨자 1은 대상좌표계가 첫 번째(

) 좌표계라는 의미를 갖는다.

은 DH변환행렬로서, 도 3을 참조하면 네 개의 DH 파라미터는

의 관계가 있음을 알 수 있다. 여기서 주의할 점은 고관절 횡평면 회전각

은 오른손 법칙에 의해 상향축(

축)을 중심으로 뒤에서 앞으로 회전하는 것이 양의 방향이라는 사실이다. 그 다음, 왼쪽 고관절에서 관상면 관절인

회전에 의해 변환된

좌표계 상의 한 점

를

좌표계에서 보았을 때의 확장벡터

는 DH변환행렬과 후행승산(post-multiply) 규칙을 이용하여 다음의 수학식 3으로 표현할 수 있다.of Equation 2 above

In the vector, the superscript 0 is the 0th reference coordinate system (

) coordinate system, and subscript 1 indicates that the target coordinate system is the first (

) has the meaning of a coordinate system.

is a DH transformation matrix, and referring to FIG. 3 , the four DH parameters are

It can be seen that there is a relationship between The point to note here is the hip joint transverse plane rotation angle

is the upward axis (

The fact that rotation about an axis from back to front is in the positive direction. Then, the coronal joint in the left hip joint

transformed by rotation

a point on the coordinate system

cast

Expansion vector as seen in the coordinate system

can be expressed by the following Equation 3 using the DH transformation matrix and the post-multiply rule.

의 네 개 DH 파라미터는

의 관계가 있음을 알 수 있다. 여기서 주의할 점은 고관절 관상면 회전각

은 오른손 법칙에 의해 전향축(

축)을 중심으로 왼쪽으로 회전하는 것이 양의 방향이라는 사실이다. 이제 왼쪽 허벅지 링크에 위치한

좌표계 상의 한 점

, 쪽 정강이 링크에 위치한

좌표계 상의 한 점

, 왼쪽 발목에 위치한

좌표계 상의 한 점

를

좌표계에서 보았을 때의 확장벡터

,

,

를 다음과 같은 관계식으로 표현할 수 있다.In Equation 3 above

The four DH parameters of

It can be seen that there is a relationship between It should be noted here that the angle of rotation of the coronal plane of the hip joint is

is the forward axis ( by the right-hand rule)

The fact that rotation about the axis to the left is in the positive direction. Now located on the left thigh link

a point on the coordinate system

, located on the side shin link

a point on the coordinate system

, located on the left ankle

a point on the coordinate system

cast

Expansion vector as seen in the coordinate system

,

,

can be expressed in the following relational expression:

의 DH 파라미터는

이고,

의 DH 파라미터는

이고,

의 DH 파라미터는

임을 알 수 있다. 여기서 주의할 점은 시상면 관절 회전각

은 오른손 법칙에 의해 측면을 향하는 축(

축)을 중심으로 위쪽으로 회전시킬 경우는 음의 값, 아래쪽으로 회전시키는 경우 양의 값이 된다는 사실이다. 그 다음으로 왼쪽 발바닥에 위치한

좌표계 상의 한 점

를

좌표계에서 보았을 때의 확장벡터

는 다음 관계식으로 표현된다.In Equations 4 to 6,

The DH parameter of

ego,

The DH parameter of

ego,

The DH parameter of

it can be seen that It should be noted here that the angle of rotation of the sagittal joint

is the axis pointing to the side by the right-hand rule (

It is a fact that a negative value when rotating upwards around the axis), and a positive value when rotating downwards. Then on the left foot

a point on the coordinate system

cast

Expansion vector as seen in the coordinate system

is expressed by the following relation.

에 대한 네 개의 DH 파라미터는

이다. 이제 하체의 오른쪽 고관절에 위치한

좌표계 상의 한 점

를

좌표계에서 보았을 때의 확장벡터

를 계산하기 위해 도 3을 참조하면

좌표계를

축 방향으로

(

: 골반 링크길이)만큼 이동하고

축을 중심으로

축을

회전시키면

축이 되므로 DH행렬

에

파라미터를 입력하여 다음과 같이 표현할 수 있다.In Equation 7 above

The four DH parameters for

am. Now located in the right hip joint of the lower body

a point on the coordinate system

cast

Expansion vector as seen in the coordinate system

Referring to Figure 3 to calculate

coordinate system

axially

(

: move as much as the length of the pelvic link)

around the axis

axis

if you rotate

DH matrix because it is an axis

to

By entering parameters, it can be expressed as follows.

회전에 의해 변환된

좌표계 상의 한 점

을

좌표계에서 보았을 때의 확장벡터

은 상기 수학식 8과 DH행렬을 이용하여 다음과 같이 표현할 수 있다.Transverse plane joint in the right hip joint

transformed by rotation

a point on the coordinate system

second

Expansion vector as seen in the coordinate system

can be expressed as follows using Equation 8 and the DH matrix.

의 네 개 DH 파라미터는

의 관계가 있음을 알 수 있다. 그 다음, 오른쪽 고관절에서 관상면 관절인

회전에 의해 변환된

좌표계 상의 한 점

를

좌표계에서 보았을 때의 확장벡터

는 DH변환행렬과 후행승산 규칙을 이용하여 다음과 같이 표현할 수 있다.3 ,

The four DH parameters of

It can be seen that there is a relationship between Then, the coronal joint in the right hip joint

transformed by rotation

a point on the coordinate system

cast

Expansion vector as seen in the coordinate system

can be expressed as follows using the DH transformation matrix and the post-multiplication rule.

의 DH 파라미터는

의 관계가 성립함을 알 수 있다. 이제 오른쪽 허벅지 링크에 위치한

좌표계 상의 한 점

, 오른쪽 정강이 링크에 위치한

좌표계 상의 한 점

, 오른쪽 발목에 위치한

좌표계 상의 한 점

를

좌표계에서 보았을 때의 확장벡터

,

,

를 다음과 같은 관계식으로 표현할 수 있다.In Equation 10 above

The DH parameter of

It can be seen that the relationship of Now located on the right thigh link

a point on the coordinate system

, located on the right shin link

a point on the coordinate system

, located on the right ankle

a point on the coordinate system

cast

Expansion vector as seen in the coordinate system

,

,

can be expressed in the following relational expression:

의 DH 파라미터는

이고,

의 DH 파라미터는

이고,

의 DH 파라미터는

임을 알 수 있다. 그 다음으로 오른쪽 발바닥에 위치한

좌표계 상의 한 점

을

좌표계에서 보았을 때의 확장벡터

는 다음 관계식으로 표현된다.In Equations 11 to 13,

The DH parameter of

ego,

The DH parameter of

ego,

The DH parameter of

it can be seen that Then on the right foot

a point on the coordinate system

second

Expansion vector as seen in the coordinate system

is expressed by the following relation.

에 대한 네 개의 DH 파라미터는

이다. 하기 표 1에 따라 모든 링크 좌표계의 DH 행렬의 파라미터를 나타낼 수 있다.In Equation 14 above

The four DH parameters for

am. According to Table 1 below, the parameters of the DH matrix of all link coordinate systems can be represented.

DH transformation matrix

0 0

0 0

0

0

0

0

0 0

0

0

0 0

0 0

0 0

0

0

0

0

0 0

0

0

Table 1 shows the DH transformation matrix of the link coordinate system of the lower body of the standing humanoid robot model determined according to Equations 1 to 14 described above.

5 is a view for explaining a joint-link model of the upper body of the robot model according to an embodiment.

좌표계는 하체의 오른쪽 고관절에 위치한

좌표계에서

축을 따라

만큼 이동시키고, 새로운

축을 따라

(

: 어깨 링크)만큼 이동시킨 후,

축을 중심으로

축을

만큼 회전시켜

축을 얻음으로써 구할 수 있다. 그러므로 DH행렬

의 파라미터는

이며,

좌표계 상의 한 점

을

좌표계에서 보았을 때의 확장벡터

는 DH행렬을 이용하여 다음과 같이 표현할 수 있다.5 is a figure expressing the upper body with links and joints when the humanoid robot is standing. Each link coordinate system follows the parameter regulation method of the DH method. Located in the left coronal plane of the upper body at the shoulder joint

The coordinate system is located in the right hip joint of the lower body.

in the coordinate system

along the axis

move as much as possible, and

along the axis

(

: After moving as much as shoulder link),

around the axis

axis

rotate as much

It can be obtained by obtaining the axis. Therefore, the DH matrix

the parameters of

is,

a point on the coordinate system

second

Expansion vector as seen in the coordinate system

can be expressed as follows using the DH matrix.

좌표계는

축을 중심으로

축을

만큼 회전시켜서

축을 얻고,

축을 중심으로

축을

만큼 회전시켜

축을 얻는다. 이를 DH행렬

의 파라미터로 표현하면

이며 수식으로 표현하면 다음과 같다.Located in the left sagittal plane of the upper body at the shoulder joint

the coordinate system

around the axis

axis

rotate it as much

get an axis,

around the axis

axis

rotate as much

get the axis This is the DH matrix

If expressed as a parameter of

and expressed as a formula:

좌표계 상의 한 점

과 왼쪽 아래팔에 위치한

좌표계 상의 한 점

을

좌표계에서 보았을 때의 확장벡터

와

은 다음 수학식에 따른 관계식으로 표현된다.Next, located on the left upper arm link

a point on the coordinate system

and located on the left forearm

a point on the coordinate system

second

Expansion vector as seen in the coordinate system

Wow

is expressed as a relational expression according to the following equation.

에 대한 네 개의 DH 파라미터는 앞에서 설명한 회전각 부호설정 원리에 의해

이고,

에 대한 네 개의 DH 파라미터는

이다. 상체의 오른쪽 관상면 어깨관절에 위치한

좌표계는

좌표계를

축을 따라

만큼 이동하여 얻은 것이므로,

의 DH 파라미터는

이 되고

좌표계 상의 한 점

을

좌표계에서 보았을 때의 확장벡터

는 DH행렬을 이용하여 하기 수학식 19로 표현될 수 있다.In Equations 17 to 18,

The four DH parameters for

ego,

The four DH parameters for

am. Located in the right coronal plane of the upper body at the shoulder joint

the coordinate system

coordinate system

along the axis

Since it is obtained by moving as much as

The DH parameter of

become

a point on the coordinate system

second

Expansion vector as seen in the coordinate system

can be expressed by Equation 19 below using the DH matrix.

좌표계는

축을 중심으로

축을

만큼 회전시켜서

축을 얻고,

축을 중심으로

축을

만큼 회전시켜

축을 얻는다. 이를 DH행렬

의 파라미터로 표현하면

이며 수식으로 표현하면 다음과 같다.Located in the right sagittal plane of the upper body at the shoulder joint

the coordinate system

around the axis

axis

rotate it as much

get an axis,

around the axis

axis

rotate as much

get the axis This is the DH matrix

If expressed as a parameter of

and expressed as a formula:

좌표계 상의 한 점

과 오른쪽 아래팔에 위치한

좌표계 상의 한 점

을

좌표계에서 보았을 때의 확장벡터

과

은 다음 관계식으로 표현된다.Next, located on the right upper arm link

a point on the coordinate system

and located on the right lower arm

a point on the coordinate system

second

Expansion vector as seen in the coordinate system

class

is expressed by the following relation.

에 대한 네 개의 DH 파라미터는

이고,

에 대한 네 개의 DH 파라미터는

이다. 도 5에서 머리와 목에 위치한 관절들의 좌표계를 구하기 위해 각 좌표계 변환관계를 알아보자. 목에 위치한

좌표계는

좌표계를

축을 따라

만큼 이동하고, 새로운

축을 중심으로

축을

만큼 회전하여

축을 얻으므로,

의 DH 파라미터는

가 된다. 확장벡터

는 DH행렬을 이용하여 다음과 같이 표현할 수 있다.In Equations 21 to 22,

The four DH parameters for

ego,

The four DH parameters for

am. In Fig. 5, let's find out the transformation relationship of each coordinate system in order to obtain the coordinate system of the joints located on the head and neck. located on the neck

the coordinate system

coordinate system

along the axis

As long as you move, new

around the axis

axis

rotate as much

Since we get the axis,

The DH parameter of

becomes extension vector

can be expressed as follows using the DH matrix.

의 회전에 의해 변환되는

좌표계는

축을 중심으로

축을

만큼 회전하고,

축을 따라서

만큼 이동하고,

축을 중심으로

축을

만큼 회전하여

축을 얻게 되므로

의 DH 파라미터는

가 되고 다음과 같이 표현된다.transverse plane articular angle of the neck

is transformed by the rotation of

the coordinate system

around the axis

axis

rotate as much as

along the axis

move as much

around the axis

axis

rotate as much

to get an axis

The DH parameter of

and is expressed as

의 회전에 의해 변환되는

좌표계는

축을 중심으로

축을

만큼 회전하여

축을 얻고,

축을 따라서

만큼 이동하여 얻게 되므로

의 DH 파라미터는

가 되고

과 같이 표현된다.sagittal articular angle of the head

is transformed by the rotation of

the coordinate system

around the axis

axis

rotate as much

get an axis,

along the axis

Because you get it by moving

The DH parameter of

become

is expressed as

Table 2 below summarizes the DH transformation matrix of all link coordinate systems of the upper body.

DH transformation matrix

0

0 0

0

0

0

0

0 0

0

0 0

0

0

0

0

0 0

0

0

0

를 왼쪽 고관절에 위치한

좌표계 기준으로 측정했을 때의 좌표 관계식을 구했는데, 이를 도 3 또는 도 5의 신체 중심점에 놓인 휴먼좌표계인

좌표계를 기준으로 측정한 좌표값으로 변환하기 위해서는 다음 식과 같이 좌표계 변환행렬을 추가적으로 곱해줘야 한다.So far, the coordinates of a point in the link coordinate system

located in the left hip joint

A coordinate relational expression was obtained when measured based on the coordinate system, which is a human coordinate system placed at the center point of the body in FIG. 3 or FIG.

In order to convert the coordinate values measured based on the coordinate system, it is necessary to additionally multiply the coordinate system transformation matrix as shown in the following equation.

는

좌표계의 각 축 단위벡터를,

는

좌표계의 각 축 단위벡터를 각각 나타낸다.In Equation 25 above

Is

The unit vector of each axis of the coordinate system,

Is

Each axis unit vector of the coordinate system is indicated.

6 is a diagram for explaining a joint recognized by a neural network model and a link connecting the joints according to an embodiment.

According to an embodiment, the electronic device 1000 inputs an image obtained by photographing a subject into a predetermined neural network model (eg, an open pose model), thereby providing a joint point for the subject and a joint-link model for a link connecting the joints. can be obtained. The open pose model can recognize each position of a human joint, face, and foot as numbers 0 to 24 as shown in FIG. 6 .

Since Open Pose projects a three-dimensional human skeleton onto a two-dimensional image, it is necessary to use the projection method well. However, if you consider when a person walks when viewed from the front and from the side, it can be seen that the projected skeletal model is different even for the same motion. In other words, when one leg is raised to walk, when viewed from the front, the thigh length appears to have decreased due to the effect of the projection method. joint angles appear. On the other hand, if you look at the thigh from the side, the length is the same, so you just need to calculate the angle value. Therefore, the method of calculating the same joint angle varies depending on the angle from which the person is viewed. Hereinafter, a process of calculating the joint angle according to the viewing angle of a person will be described with reference to FIG. 7 to be described later.

7 is a diagram for a relationship when a three-dimensional object is projected on a coronal plane and a sagittal plane, according to an embodiment.

8 is a diagram for explaining an angular relationship between a person and a camera.

인 역진자를 시상면에서

만큼 회전시키고 관상면(coronalplnae, y-z 평면)에서

만큼 회전시킨 것을 3차원과 2차원에서 표현한 것을 나타낸다. 도 8의 그림 (704)를 참조하면 카메라가 사람을 촬영하는 각도를

라고 할 때 사람과 카메라의 각도 관계를 정의한 것이다. 즉, 카메라가 사람을 정면으로 마주보고 있을 때는

는

, 카메라가 시계 반대방향으로 움직여서 마주보는 사람의 왼쪽을 찍으면 양의 각도, 시계 방향으로 움직여서 마주보는 사람의 오른쪽을 찍으면 음의 각도라고 정의한다. 그러므로 카메라가 정확하게 좌측에서 사람을 찍으면(혹은 화면에서 사람이 정확하게 오른쪽을 향하면)

값은

, 정확하게 우측에서 사람을 찍으면(혹은 화면에서 사람이 정확하게 왼쪽을 향하면)

가 된다. Referring to figure 702 of FIG. 7 , the length is

Inverted pendulum in the sagittal plane

rotated by and in the coronal plnae (yz plane)

It represents the representation in 3D and 2D that rotated as much as possible. Referring to the figure 704 of FIG. 8, the angle at which the camera shoots a person is

It defines the angular relationship between the person and the camera. That is, when the camera is facing the person

Is

, is defined as a positive angle when the camera moves counterclockwise to take a picture on the left side of the person facing it, and a negative angle when the camera moves clockwise to take a picture on the right side of the person facing it. So if the camera is shooting a person exactly on the left (or if the person on the screen is pointing exactly right)

value is

, when you photograph a person precisely from the right (or when the person is correctly pointed to the left on the screen).

becomes

에서

으로 변화하는 것을 이용해서 계산한 각도 값에는 아래첨자

을, 연결된 두 링크 간의 각도인

를 이용해서 계산한 각도 값에는 아래첨자

를 붙인다. Since OpenPose provides two-dimensional image coordinates for each joint, the length and angle of the link can be calculated from them. For each joint, the associated link length is

at

The angle value calculated using the change to

, which is the angle between the two connected links

The angle value calculated using

paste it

와 시상면 관절

, 관상면 관절

와의 관계를 보자. 도 7의그림 (702)의 아래 왼쪽 그림은 역진자의 움직임이 관상면에 투영된 것이다. 이때

이고,

에 의해서 링크의 길이가

에서

으로 바뀌고,

에 의해서 역진자의 회전이 일어났으므로 하기 수학식에 따른 관계식이 성립한다.Now with reference to the figure 702 of FIG. 7 and the figure 704 of FIG. 8, the shooting angle

and sagittal joint

, coronal joint

Let's see the relationship with The lower left figure of the figure 702 of FIG. 7 is that the motion of the inverted pendulum is projected onto the coronal plane. At this time

ego,

the length of the link by

at

is changed to

Since the rotation of the inverted pendulum occurred by

이고,

에 의해서 역진자의 회전이 일어나고,

에 의해서 링크의 길이가

에서

으로 바뀌었으므로 다음의 관계식이 성립한다.The lower right figure of the figure 702 of FIG. 7 is that the motion of the inverted pendulum is projected on the sagittal plane, at this time

ego,

The rotation of the inverted pendulum is caused by

the length of the link by

at

, so the following relation is established:

는

가

일 때(카메라가 사람을 정면으로 촬영할 때)는 오픈포즈 골격모델의 링크 길이 변화를 사용하고,

가

일 때(카메라가 사람을 좌측에서 촬영할 때)는 회전각을 사용해서 계산할 필요성이 있다. 그러므로 카메라의 촬영각

가 다양하게 변할 때 오픈포즈 골격모델로부터 계산되는 시상면 각도는 다음 수학식 28과 같이

에 따른 가중치 합으로 계산해야 한다. 카메라가 사람을 좌측면에서 촬영할 때는

가 음수이므로 식에서 절대값을 취했다.Referring to Equations 26 to 27, the sagittal joint angle

Is

go

When (when the camera shoots a person in front), the link length change of the open pose skeletal model is used,

go

, it is necessary to calculate using the angle of rotation (when the camera is photographing a person from the left). Therefore, the angle of view of the camera

When is varied, the sagittal angle calculated from the open pose skeletal model is as shown in Equation 28 below.

should be calculated as the sum of weights according to When the camera is shooting a person from the left side

Since is a negative number, the absolute value is taken from the expression.

는

가

일 때는 회전각을 사용하고,

가

일 때는 오픈포즈 골격모델의 링크 길이 변화를 사용해서 계산해야 한다. 그러므로 오픈포즈 골격 정보로부터 계산되는 관상면 각도는 다음 수학식 29 및 30과 같이

에 따른 가중치 합으로 계산해야 한다.Conversely, the coronal joint angle

Is

go

When the angle of rotation is used,

go

In some cases, it should be calculated using the link length change of the open pose skeletal model. Therefore, the coronal angle calculated from the open pose skeleton information is as shown in Equations 29 and 30 below.

should be calculated as the sum of weights according to

A method of calculating an angle between two links when there are two links will now be described with reference to FIGS. 9 to 11, which will be described later.

9 is a view for explaining an angle between two points, according to an exemplary embodiment.

은 앞서가는 링크의 끝점,

는 뒤따르는 링크의 끝점을 의미하고, 점

의 각도가

, 점

의 각도가

라고 하자. 두 점이 둘 다 1사분면에 있을 때, 두 점의 사이각

는 다음과 같이 계산할 수 있다.As shown in figure 902 of FIG. 9, the point

is the endpoint of the preceding link,

denotes the endpoint of the link that follows, and the dot

the angle of

, dot

the angle of

let's say When two points are in the first quadrant, the angle between the two points

can be calculated as

값을

에서

범위에서 계산하는 함수를 의미한다. In Equations 31 to 32, the atan2 function is the inverse function of the tan function.

value

at

A function that calculates in a range.

10 is a view for explaining an angle formed by two points according to another embodiment.

를 더해야 한다.Referring to the figure 906 of FIG. 10, when the leading point is in the third or fourth quadrant, the result value of the atan2 function of the leading point becomes a negative angle.

should be added

를 더하고 뒤따르는 점은 그냥 음의 각도값을 빼주면 된다.As shown in figure 908 of FIG. 10 , when the leading point is in the third quadrant and the following point is in the fourth quadrant, the leading point is in the fourth quadrant.

To add and follow the dot, simply subtract the negative angle value.

11 is a view for explaining an angle formed by two points according to another embodiment.

As shown in FIG. 11 , if the leading point and the following point are in the same quadrant even in the 3rd or 4th quadrant, the angle of the following point is simply subtracted from the angle of the leading point.

은 앞서가는 점,

는 뒤따르는 점을 의미하고,

이다.

과

가 이루는 각도 차이는 하기 표 3의 보정된 각도 값 및 상기 보정된 각도 값을 하기 수학식 35에 적용함으로써 계산될 수 있다.The following is a table by summarizing the various cases so far. in the table

is the leading point,

means the following point,

am.

class

The angle difference formed by can be calculated by applying the corrected angle value of Table 3 below and the corrected angle value to Equation 35 below.

Quadrant 1 quadrant 2 quadrant 3 quadrant 4 Quadrant 1

quadrant 2

quadrant 3

quadrant 4

를 계산하는 방법에 대해 설명한다.Now, using the open pose, the angle at which the camera shoots the person with only the joint-link model of the target person expressed in the two-dimensional image coordinate system.

How to calculate .

돌아서 있을 때와 오른쪽으로

돌아서 있을 때 투영되어 보이는 골반의 길이가 같아서 이 길이 변화만으로는 왼쪽인지 오른쪽인지 구분할 수가 없다. If you stand upright in place and turn left or right, your shoulder links or pelvic links will appear that short of projection. Among them, since the shoulder link is also affected by the posture and leaning angle of the upper body, it is better to measure the change in the length of the pelvic link. However, since the projection method has a cosine function relation, the same positive and negative angles are obtained. That is, the person moves to the left in place

when turning and to the right

Since the projected pelvis is the same length when turned around, it is impossible to tell whether the pelvis is left or right just by changing this length.

를 계산할 수 있다. 참고로 도 6에서 얼굴 부분의 특징점 번호가 코는 0, 오른쪽 눈은 15, 오른쪽 귀는 17, 왼쪽 눈은 16, 왼쪽 귀는 18이다.As a countermeasure against this, check the direction the head is facing, check whether it is twisted to the left or the right with respect to the trunk link, and compare the length of the pelvic link to see how many degrees it is twisted by comparing the camera angle.

can be calculated. For reference, in FIG. 6 , the feature point numbers of the face part are 0 for the nose, 15 for the right eye, 17 for the right ear, 16 for the left eye, and 18 for the left ear.

12 is a diagram for explaining a model recognized by a neural network model when a person faces the front or the right, according to an embodiment.

Referring to FIG. 12 , it can be seen the comparison result of the link-joint model recognized by the open pose when the camera is fixed and the person faces the front and the right side. As shown in the figure 922 of FIG. 12 , when the person faces the front, the links 0-1 corresponding to the face are vertical, but as shown in the figure 924 of FIG. 12 , the person faces to the right. At this time, it can be seen that the links 0-1 are inclined to the left compared to the torso link (even in the image coordinates where the up and down inversion occurs). This is a phenomenon that occurs when the face is protruded forward relative to the shoulder and viewed from the side. Conversely, it can be seen that when a person turns left, links 0-1 will lean to the right of the body link (right in the image coordinates).

13 is a diagram for describing a recognition model recognized by a neural network model when a person faces right, front, and left, according to an embodiment.

A skeletal model recognized in an open pose when a person faces right, front, or left when the camera is fixed, or when the left, front, or right side of a person is photographed when the camera is moving will be described with reference to FIG. 13 . When expressing the movement of human skeleton data recognized by an open pose in a three-dimensional space, it is more intuitive to model based on a person regardless of the position of the camera because the reference coordinate system is assumed to be a posture floating in the air at the center of the body.

좌표계는 인체 움직임과는 상관없이 인체 중심부에 고정되어 있는 절대좌표계(absolute coordinate frame)이고,

좌표계는 인체의 중심에 붙어있으며 사람이 바라보는 방향을 틀거나 카메라가 회전하면 그에 따라 회전하는 상대좌표계(relative coordinate frame)이다.

는

축을 중심으로

축을 회전시켜서

축을 얻을 때 그 회전각도이다. 오른손 법칙에 의하면 도 13의 그림 (932)에 도시된 바와 같이 사람이 카메라에 대해 오른쪽으로 도는 경우

는 음의 각도가 되고, 도 13의 그림 (936)에 도시된 바와 같이 카메라에 대해 왼쪽으로 도는 경우

는 양의 각도가 되며, 도 13의 그림 (934)에 도시된 바와 같이 카메라에 대해 정면인 경우 각도

는 0이 된다. 그럼 오픈포즈 영상으로부터 사람이 향하는 각도

를 계산하기 위해서는 아래의 두 단계가 필요하다. 먼저 2차원 영상좌표계에서 0번-1번 링크(얼굴 부분)가 1번-8번 링크의 왼쪽에 있는지 오른쪽에 있는지를 확인한다. 앞서 설명한 대로 영상좌표계에서 얼굴 링크가 몸통 링크의 왼쪽에 있으면 사람이 오른쪽을 향하는 것이고

, 반대로 얼굴 링크가 몸통 링크의 오른쪽에 있으면 왼쪽을 향하는 것이다

. 두번째로 대상인이 정면을 향하고 차려 자세로 있을 때 골반의 픽셀 길이를

, 걸어가는 중에 측정된 골반의 픽셀 길이를

라고 하면, 그 비율의 acos 함수 값을 이용하면

의 크기를 알 수 있다. 그런데 대상인이 카메라로 다가오거나 멀어질 경우에도

가 달라질 수 있으므로, 몸통 대 골반의 픽셀 길이의 비율 변화를 계산하는 것이 보다 안정적인 측정결과를 얻을 수 있는 방법이다. 첫 번째 단계에서 0번-1번 링크가 왼쪽과 오른쪽 중 어디에 있는지는 두 링크(0번-1번 링크와 1번-8번 링크)간 사이각 계산 방법을 적용하면 된다.13 in FIG.

The coordinate system is an absolute coordinate frame that is fixed at the center of the human body regardless of human body movement.

The coordinate system is attached to the center of the human body and is a relative coordinate frame that rotates according to the direction of the person's gaze or the camera rotates.

Is

around the axis

by rotating the axis

It is the rotation angle when the axis is obtained. According to the right-hand rule, when a person turns to the right with respect to the camera as shown in figure 932 of FIG.

is a negative angle, and when turning left with respect to the camera as shown in figure 936 of FIG.

is a positive angle, the angle when facing the camera as shown in figure 934 of FIG.

becomes 0. Then, the angle the person faces from the open pose image

The following two steps are required to calculate . First, it is checked whether the 0-1 link (face part) is on the left or right side of the 1-8 link in the 2D image coordinate system. As described above, if the face link is to the left of the body link in the image coordinate system, the person is facing to the right.

, conversely, if the face link is to the right of the body link, it is pointing to the left.

. Second, measure the pixel length of the pelvis when the subject is in a front-facing pose.

, the pixel length of the pelvis measured during walking

, if we use the acos function value of that ratio,

size can be found. However, even if the subject approaches or moves away from the camera,

Since ? can vary, calculating the ratio change of the pixel length of the torso to the pelvis is a way to obtain a more stable measurement result. In the first step, the method of calculating the angle between the two links (Links 0-1 and Links 1-8) is applied to determine whether the 0-1 link is on the left or the right side.

14 is a diagram for explaining a skeletal model recognized by a neural network when a person faces right and left, according to an embodiment.

Referring to FIG. 14 , figures 942 and 944 of FIG. 14 show the open pose skeletal model recognized in the image coordinate system when a person faces right and left with respect to the camera, respectively.

15 is a view for explaining an angle between a face link and a body link, according to an embodiment.

를 계산하는 과정을 나타낸다. 영상좌표계에서 얼굴 링크의 각도를

, 몸통 링크의 각도를

, 그 사이각을

라고 하면 다음의 관계가 성립한다.15 is an angle at which the face link is inclined to the left or right compared to the body link in the image coordinate system.

shows the process of calculating The angle of the face link in the image coordinate system

, the angle of the body link

, the angle between

, then the following relationship is established:

,

이며, 얼굴 링크가 몸통 링크에 비해 왼쪽에 있으면(사람은 오른쪽을 향함)

,

, 얼굴 링크가 몸통 링크에 비해 오른쪽에 있으면(사람은 왼쪽을 향함)

,

이다. 두 번째 단계에서 몸통의 픽셀 길이(

) 대 골반의 픽셀 길이(

)의 비율을

라고 하면, 정면을 보고 서있을 때의 골반 길이 비율(

) 값을 미리 확보해 둔다. 그리고 현재 자세에서의 몸통 대 골반 길이의 비율인

를 계산하여 현재 골반 길이가 얼마나 변했는지를 계산할 수 있다. 이를 식으로 표현하면 다음과 같다.15, when the facial link is on a continuous line of the body link

,

, if the face link is to the left relative to the body link (people face right)

,

, if the face link is to the right compared to the torso link (people face left)

,

am. In the second step, the pixel length of the body (

) versus pelvis length in pixels (

) to the ratio of

, the ratio of pelvic length when standing facing the front (

) to reserve the value in advance. And the ratio of the torso to pelvis length in the current posture is

can be calculated to calculate how much the current pelvic length has changed. Expressing this in an expression is as follows.

를 계산하면 다음과 같다.The angle the person faces with respect to the camera according to the above

is calculated as follows.

가 이렇게

(정면을 향할 때)가 아닌 값이 되면 신체 중심점 휴먼좌표계인

좌표계 식을 세계좌표계인

좌표계 식으로 변환해야 한다. 이를 위해 아래와 같이 회전변환행렬을 곱해줘야 한다.And

go like this

If it becomes a value other than (when facing forward), the human coordinate system of the body center

Coordinate system expression is world coordinate system

It must be converted to a coordinate system expression. To do this, we need to multiply the rotation transformation matrix as follows.

는

좌표계의 각 축 단위벡터를

는

좌표계의 각 축 단위벡터를 각각 나타낸다.in the expression

Is

The unit vector of each axis of the coordinate system

Is

Each axis unit vector of the coordinate system is indicated.

16 is a diagram illustrating a skeletal model of a human body displayed in pixel coordinates, according to an exemplary embodiment.

Open pose recognizes the main joints, face, and feet of a human from the camera pixel image and calculates pixel coordinate values. Based on these coordinates, if the human body is modeled in two dimensions, the posture as shown in figure 1632 of FIG. 16 is obtained. comes out When displaying body parts in the figure, red indicates the right and blue indicates the left, so you can see that the top and bottom are inverted. The human skeleton model of the figure 1632 of FIG. 16 may include a joint 1634 and a link 1636 connected to the joint according to pixel coordinates. First, a method for calculating the coronal and sagittal joint angles of the lower body will be described with reference to FIG. 17 to be described later.

17 is a view showing a joint-link model of the robot model lower body standing according to an embodiment.

은 차려자세로 있을 때는

, 오른쪽에서 왼쪽으로 회전할 때는 양의 각도일 수 있다.FIG. 17 is a diagram for comparison with FIG. 16 as a humanoid robot model defined in a three-dimensional space. The right coronal joint angle shown in FIG. 17

When you are in the pose

, can be a positive angle when rotating from right to left.

18 is a diagram for calculating a right hip coronal angle according to an embodiment.

을 계산하기 위한 개념도이고, 그림 (964)는 그림 (962)의 9번 노드를 원점으로 이동함에 따른 그림이다. 보다 상세하게는 그림 (964)는 골반 링크와 오른쪽 허벅지 링크 간 각도 계산을 하기 위해 9번 노드의 위치를 원점으로 이동한 모습이다. 이제 atan2 함수에서 계산한 각도값을 표 3에서 보정한 함수

를 이용해서 하기 수학식 41과 같이 두 각도 값을 계산할 수 있다.In more detail, the figure 962 of FIG. 18 shows the right hip joint coronal angle.

It is a conceptual diagram for calculating . In more detail, Figure 964 shows that the position of node 9 is moved to the origin to calculate the angle between the pelvic link and the right thigh link. Now, the angle value calculated by the atan2 function is corrected in Table 3

Two angle values can be calculated using Equation 41 below.

을 계산하면 다음과 같다.From this angle value, referring to figure 962 of FIG. 18,

is calculated as follows.

은 두 링크의 사이각(

)에

를 뺀 값이 각도 값이 되고, 각도의 방향은 반대 방향이므로 앞에 마이너스를 곱해준다.in other words,

is the angle between the two links (

)to

The subtracted value becomes the angle value, and since the direction of the angle is in the opposite direction, it is multiplied by a minus in front.

19 is a diagram for calculating a left hip coronal angle according to an embodiment.

를 계산하기 위한 개념도이고, 그림 (968)은 골반 링크와 왼쪽 허벅지 링크 간 각도 계산을 용이하게 하기 위해 12번 노드의 위치를 원점으로 이동한 그림이다. 마찬가지로

를 이용해서 하기 수학식 43과 같이 두 각도 값을 계산할 수 있다.In more detail, the figure 966 of FIG. 19 shows the left hip coronal angle.

It is a conceptual diagram for calculating . Likewise

Two angle values can be calculated using Equation 43 below.

을 계산하면 하기 수학식과 같다.from this angle

is calculated as the following equation.

은 두 링크간 사이각(

)에

를 더한 값이 각도의 크기가 되고 회전축의 방향은 반대이므로 앞에 마이너스를 곱해준다. 다음으로 하체의 시상면 관절각 계산법에 대해 알아보자. 사람이 걸어가는 것을 정면에서 보면 다리를 앞으로 들어올리는 경우 길이가 짧아져 보이는데 이는 투영법에 의한 것임을 상술한 바 있다. 도 17에서 하체의 시상면 관절각

은 다리를 앞으로 들어올릴 때는 음의 각도, 뒤로 뻗을 때는 양의 각도로 정의되어 있음에 주의한다. 오른쪽 골반의 시상면 관절각

은 다음 식으로 계산할 수 있다.in other words,

is the angle between the two links (

)to

The added value becomes the size of the angle, and the direction of the axis of rotation is opposite, so multiply it by minus in front. Next, let's learn how to calculate the sagittal joint angle of the lower body. If you see a person walking from the front, the length appears shorter when the legs are raised forward, but it has been described above that this is due to the projection method. 17, the sagittal joint angle of the lower body

Note that is defined as a negative angle for lifting the leg forward and a positive angle for extending the leg backward. Sagittal articular angle of the right pelvis

can be calculated in the following way.

는 사람이 차려자세로 있을 때 측정한 허벅지뼈의 픽셀길이를,

은 사람이 걸어갈 때 측정한 오른쪽 허벅지뼈의 픽셀길이를 의미한다. 식의 부호는 허벅지를 위로 들어올리는 경우 음의 값, 뒤로 내젖는 경우 양의 값이 되므로

로 표시했다. 투영법의 특성상 각도의 부호는 알 수 없으므로 다른 정보를 사용해야 한다. 지금처럼 사람이 앞으로 걸어가는 경우에는 제일 먼저 들어올리는 다리를 인식해서 그 다리의 골반 시상면 관절각의 부호를 음수로 잡으면 된다. in the expression

is the pixel length of the thighbone measured when the person is in a standing position,

is the pixel length of the right thighbone measured when a person walks. The sign of the expression is a negative value when the thigh is lifted upwards, and a positive value when the thigh is extended backwards.

marked with Due to the nature of the projection method, the sign of the angle is unknown, so other information must be used. When a person walks forward like now, the first leg is recognized and the sign of the joint angle of the pelvic sagittal plane of that leg is taken as a negative number.

에 대해서 생각해보자. 사람이 걸어갈 때 정강이의 투영된 길이는

값에 의해 결정되며, 인체 골격 구조의 특징 상 슬관절은 앞으로 꺾이지 않기 때문에 다음 식으로 계산할 수 있다.Sagittal articular angle of the right knee

Let's think about When a person walks, the projected length of the shin is

It is determined by the value, and since the knee joint does not bend forward due to the characteristics of the human skeletal structure, it can be calculated using the following formula.

는 사람이 차려자세로 있을 때 측정한 정강이뼈의 픽셀길이를,

은 사람이 걸어갈 때 측정한 오른쪽 정강이뼈의 픽셀길이를 의미한다. 왼쪽 다리의 골반과 무릎의 시상면 관절각들은 동일한 원리로 수식화할 수 있다. 즉, 왼쪽 골반의 시상면 관절각

은 다음 식으로 계산할 수 있다.in the above formula

is the pixel length of the shin bone measured when the person is in a standing position,

is the pixel length of the right shinbone measured when a person walks. The sagittal joint angles of the pelvis and knee of the left leg can be formulated using the same principle. That is, the sagittal joint angle of the left pelvis

can be calculated in the following way.

은 사람이 걸어갈 때 측정한 왼쪽 허벅지뼈의 픽셀길이를 의미한다.

는 왼쪽과 오른쪽 동일한 값으로 두는데, 차려 자세에서 오픈포즈로 측정한 왼쪽과 오른쪽 허벅지의 픽셀 길이의 평균값으로 계산하면 된다. 왼쪽 무릎의 시상면 관절각

은 다음 식으로 계산할 수 있다.in the above formula

is the pixel length of the left thighbone measured when a person walks.

is set as the same value for the left and right sides, and it can be calculated as the average value of the pixel lengths of the left and right thighs measured in an open pose in a dressing up posture. Sagittal articular angle of the left knee

can be calculated in the following way.

은 사람이 걸어갈 때 측정한 왼쪽 정강이뼈의 픽셀길이를 의미한다.in the above formula

is the pixel length of the left shinbone measured when a person walks.

20 is a view for explaining a joint-link model of the upper body of the robot model according to an embodiment.

은 차려자세로 있을 때는

, 오른쪽에서 왼쪽으로 회전할때는 양의 각도라는 것을 기억하자.With reference to FIG. 20, let's learn about the method of calculating the coronal and sagittal joint angles of the upper body. FIG. 20 is a diagram for comparison with the figure of FIG. 16 as a humanoid robot model defined in a three-dimensional space. Referring to Figure 20, the coronal joint angles on both shoulders

When you are in the pose

, remember that a right-to-left rotation is a positive angle.

21 is a diagram for calculating right and left shoulder joint angles according to an embodiment.

을 계산하기 위한 개념도이고, 도 21의 그림 (974)는 왼쪽 어깨관절 관상면 각도인

을 계산하기 위한 개념도이다. 상술한 하체 관절에 대해 적용한 방법을 그대로 적용하면 다음과 같이 두 각도 값을 계산할 수 있다.In more detail, the figure 972 of FIG. 21 is the right shoulder joint coronal angle.

is a conceptual diagram for calculating

It is a conceptual diagram for calculating . If the method applied to the lower body joint described above is applied as it is, two angle values can be calculated as follows.

을 계산하면 다음과 같다.From this angle value, referring to figure 972 of FIG.

is calculated as follows.

을 계산하는 과정을 설명하기로 한다. 두 각도 값은 다음과 같이 계산할 수 있다.With reference to the figure 974 of FIG. 21, the angle of the coronal plane of the left shoulder joint is

The process of calculating . The two angle values can be calculated as follows.

을 계산하면 다음과 같다.From this angle value, referring to the figure 974 of Fig. 21,

is calculated as follows.

은

와 같이 표현된다.The shoulder and wrist sagittal joint angle calculation method of the upper body is similar to the principle calculated in the lower body. First, the sagittal angle of the right shoulder joint is

silver

is expressed as

는 사람이 차려자세로 있을 때 측정한 위팔뼈의 픽셀길이를,

은 사람이 걸어갈 때 측정한 오른쪽 위팔뼈의 픽셀길이를 의미한다. 식의 부호는 팔을 앞으로 흔드는 경우 음의 값, 뒤로 흔드는 경우 양의 값이 되므로

로 표시했다. 투영법의 특성상 각도의 부호는 알 수 없으므로 다른 정보를 사용해야 한다. 지금처럼 사람이 앞으로 걸어가는 경우에는 왼쪽, 오른쪽의 다리와 팔이 서로 반대방향으로 움직이는 원리를 이용해서 다리와 팔의 시상면 관절각 부호를 반대로 설정하면 된다. 오른쪽 팔꿈치 시상면 관절각

에 대해 생각해보자. 사람이 오른팔을 앞뒤로 흔들 때 아래팔의 투영된 길이는

값에 의해 결정되며, 인체 골격 구조의 특징 상 팔꿈치 관절은 뒤로 꺾이지 않기 때문에 다음 식으로 계산할 수 있다.in the above formula

is the pixel length of the upper arm bone measured when the person is in a standing position,

is the pixel length of the right upper arm bone measured when a person walks. The sign of the expression is negative when the arm is waved forward, and a positive value when the arm is waved backward.

marked with Due to the nature of the projection method, the sign of the angle is unknown, so other information must be used. When a person walks forward as it is now, using the principle that the left and right legs and arms move in opposite directions, set the sagittal joint angle sign of the leg and arm in reverse. right elbow sagittal joint angle

Let's think about When a person swings their right arm back and forth, the projected length of the forearm is

It is determined by the value, and since the elbow joint does not bend backwards due to the characteristics of the human skeletal structure, it can be calculated using the following formula.

는 사람이 차려자세로 있을 때 측정한 아래팔뼈의 픽셀길이를,

은 사람이 걸어갈 때 측정한 오른쪽 아래팔뼈의 픽셀길이를 의미한다. 왼쪽 팔의 시상면 관절각들은 동일한 원리로 수식화할 수 있다. 즉, 왼쪽 어깨의 시상면 관절각

은 다음 식으로 계산할 수 있다.in the above formula

is the pixel length of the forearm measured when the person is in a standing position,

is the pixel length of the right forearm measured when a person walks. The sagittal joint angles of the left arm can be formulated using the same principle. That is, the sagittal joint angle of the left shoulder

can be calculated in the following way.

은 사람이 걸어갈 때 측정한 왼쪽 위팔뼈의 픽셀길이를 의미한다. in the above formula

is the pixel length of the left upper arm bone measured when a person walks.

에 대해 생각해보자. 사람이 왼팔을 앞뒤로 흔들 때 아래팔의 투영된 길이는

값에 의해 결정되며, 인체 골격 구조의 특징 상 팔꿈치 관절은 뒤로 꺾이지 않기 때문에 다음 식으로 계산할 수 있다.left elbow sagittal joint angle

Let's think about When a person swings their left arm back and forth, the projected length of the forearm is

It is determined by the value, and since the elbow joint does not bend backwards due to the characteristics of the human skeletal structure, it can be calculated using the following formula.

은 사람이 걸어갈 때 측정한 오른쪽 왼팔뼈의 픽셀길이를 의미한다.in the above formula

is the pixel length of the right left arm bone measured when a person walks.

22 is a diagram illustrating a skeletal model of a human body indicated by pixel coordinates observed from the left, according to an exemplary embodiment.

A process of calculating the joint angle when a person turns to the right will be described with reference to FIG. 22 . 22 is a screen obtained by extracting an open pose skeletal model when a person walks from a place to the right (the camera is taken from the person's left side). In the same way, be aware that the red color represents the right side of the body and the blue color represents the left side of the body due to the upside-down inversion in the image coordinate system. Referring to the open pose skeletal model screen 2210, it can be observed that a human joint 2202 is displayed as a point, and a link 2204 connecting the joint 2202 is shown.

은 차려자세로 있을 때는

, 다리를 앞으로 들어 올리면 음의 각도라는 것을 기억하자. First, let's learn how to calculate the sagittal and coronal joint angles of the lower body. 17, the right hip sagittal joint angle

When you are in the pose

, remember that lifting the leg forward is a negative angle.

23 is a diagram illustrating a neural network skeletal model and joint angles of right and left legs recognized by a neural network according to an exemplary embodiment.

24 is a diagram for calculating a right hip joint angle according to an exemplary embodiment.

을 계산하기 위한 개념도이다. 이제 앞 절에서 사용한 방식을 그대로 적용해서, 표 3의 보정함수

를 이용하여 다음과 같이 두 각도 값을 계산할 수 있다.In more detail, Figure 24 is a right hip joint sagittal angle

It is a conceptual diagram for calculating . Now, applying the method used in the previous section as it is, the correction function in Table 3

can be used to calculate the two angle values as follows.

을 계산하면 다음과 같다.Using this angle value and referring to the figure on the right of FIG. 24,

is calculated as follows.

25 is a diagram for calculating a right knee joint angle according to an embodiment.

을 계산하는 과정을 설명하기로 한다. 마찬가지로 보정함수

를 이용해서 다음과 같이 두 각도 값을 계산할 수 있다.Referring to the figure 2226 of FIG. 25, the right knee joint sagittal angle is

The process of calculating . Similarly, the correction function

can be used to calculate the two angle values as follows.

을 계산하면 다음과 같다.using this angle

is calculated as follows.

를 이용해서 다음과 같이 두 각도 값을 계산할 수 있다.correction function

can be used to calculate the two angle values as follows.

을 계산하기로 한다.Using this angle value and referring to FIG. 26 to be described later,

to calculate

26 is a diagram for calculating a left hip joint angle according to an embodiment.

하면 다음과 같다. 하기 수학식

은 수학식 57과 같은 형태임을 알 수 있다.Referring to FIG. 26 using the angle value of Equation 60,

If you do: the following formula

It can be seen that is of the same form as in Equation 57.

27 is a diagram for calculating a left knee joint angle according to an embodiment.

을 계산하기 위한 개념도이다. 보정함수

를 이용해서 두 각도 값은 각각

와 같이 계산될 수 있다. 이렇게 계산된 각도 값을 이용하여

을 계산하면

와 같이 표현된다.In more detail, Figure 27 is the left knee joint sagittal angle

It is a conceptual diagram for calculating . correction function

Each of the two angle values using

can be calculated as Using this calculated angle value,

If you calculate

is expressed as

은 다리를 왼쪽으로 들어올릴 때는 양의 각도, 오른쪽으로 들어올릴 때는 음의 각도로 정의되어 있음을 기억하자. 오른쪽 골반의 관상면 관절각

과 왼쪽 골반의 관상면 관절각

은 다음 식으로 계산할 수 있다. Next, let's learn how to calculate the coronal joint angle of the lower body. If you see a person walking from the left or right side, the length appears shorter when the legs are raised to the side. In Figure 17, the coronal joint angle of the lower body

Note that is defined as a positive angle for lifting the leg to the left and a negative angle for lifting the leg to the right. Coronal articular angle of the right pelvis

and coronal articular angles of the left pelvis

can be calculated in the following way.

는 사람이 차려자세로 있을 때 측정한 허벅지뼈의 픽셀길이를,

과

은 사람이 걸어갈 때 측정한 오른쪽과 왼쪽 허벅지뼈의 픽셀길이를 각각 의미한다. 식의 부호는 다리를 왼쪽으로 들어올릴 때는 양의 각도, 오른쪽으로 들어올릴 때는 음의 각도이고, 사람이 앞으로 걸어가는 경우에는 일반적으로 다리를 바깥쪽(왼쪽 다리는 왼쪽, 오른쪽 다리는 오른쪽)으로 들어올리는 경향이 있으므로,

에는 음의 부호,

에는 양의 부호를 부여하면 된다. 이제 상체의 시상면, 관상면 관절각을 상기 방법대로 적용해 보자. 도 20을 참조하면 양쪽 어깨에 있는 시상면 관절각

과 팔꿈치에 있는 시상면 관절각

은 차려자세로 있을 때는

, 팔을 앞쪽으로 들어올릴 때는 음의 각도, 뒤쪽으로 내저을 때는 양의 각도라는 것을 기억하자.In Equations 61 to 62,

is the pixel length of the thighbone measured when the person is in a standing position,

class

is the pixel length of the right and left thighbones, respectively, measured when a person walks. The sign of the equation is a positive angle when the leg is lifted to the left and a negative angle when it is lifted to the right, and when a person is walking forward, the leg is usually turned outward (left leg left, right leg right). As there is a tendency to lift,

is a negative sign,

should be given a positive sign. Now, let's apply the sagittal and coronal joint angles of the upper body as above. Referring to Figure 20, the sagittal joint angles in both shoulders

and sagittal articular angles in the elbow

When you are in the pose

, Remember that when you raise your arm forward, it is a negative angle, and when you swing it backwards, it is a positive angle.

28 is a view for explaining a link and a joint of a right arm, a link and a joint of a right shoulder, and a link and a joint of a right elbow, according to an embodiment.

과

을 계산하기 위한 개념도이다. 도 28의 그림 (2244)는 어깨 관절 부분을 확대해서 그린 그림으로서, 보정함수

를 이용해서 다음과 같이 두 각도 값과

을 계산할 수 있다. Figure 2242 of Figure 28 is a joint in the right arm.

class

It is a conceptual diagram for calculating . Figure 2244 of Figure 28 is an enlarged drawing of the shoulder joint portion, the correction function

using the two angle values and

can be calculated.

을 이용해서 다음과 같이 두 각도 값과

을 계산할 수 있다.Figure 2246 of Figure 28 is an enlarged drawing of the elbow joint of the right arm, and the correction function

using the two angle values and

can be calculated.

29 is a diagram for describing a link and joints of a left arm, according to an exemplary embodiment.

과

을 계산하기 위한 개념도이다. 앞에서 계산한 방식대로 왼쪽 어깨 관절 부분을 확대해서 그린 그림으로서, 보정함수

를 이용해서 다음과 같이 두 각도 값과

을 계산할 수 있다.In more detail, referring to the figure 2248 of FIG. 29, the joint in the left arm

class

It is a conceptual diagram for calculating . As a figure drawn by enlarged the left shoulder joint in the same way as previously calculated, the correction function

using the two angle values and

can be calculated.

를 이용해서 다음과 같이 두 각도 값과

을 계산할 수 있다.The left elbow joint part is a correction function

using the two angle values and

can be calculated.

은 팔을 왼쪽으로 들어올릴 때는 양의 각도, 오른쪽으로 들어올릴 때는 음의 각도로 정의되어 있음을 기억하자. 이 원리에 따라 오른쪽 어깨의 관상면 관절각

은 다음 식으로 계산할 수 있다.Now let's learn how to calculate the coronal joint angle of the upper body and both shoulders. In Fig. 20, the coronal joint angle of the shoulder

Note that the angle is defined as a positive angle when the arm is raised to the left and a negative angle when the arm is raised to the right. According to this principle, the coronal joint angle of the right shoulder

can be calculated in the following way.

는 사람이 차려자세로 있을 때 측정한 위팔뼈의 픽셀길이를,

은 사람이 걸어갈 때 측정한 오른쪽 위팔뼈의 픽셀길이를 의미한다. 팔을 왼쪽으로 올릴 때는 양의 각도, 오른쪽으로 올릴 때는 음의 각도이고, 사람이 앞으로 걸어갈 때에는 일반적으로 팔을 바깥쪽(왼쪽 팔은 왼쪽, 오른쪽 팔은 오른쪽)으로 올리는 경향이 있으므로,

에는 음의 부호,

에는 양의 부호를 부여하면 된다. 마찬가지로 왼쪽 어깨의 관상면 관절각

은 다음 식으로 계산할 수 있다.In Equation 71 above

is the pixel length of the upper arm bone measured when the person is in a standing position,

is the pixel length of the right upper arm bone measured when a person walks. When the arm is raised to the left, it is a positive angle, and when it is raised to the right, it is a negative angle, and when a person walks forward, the arm generally tends to be raised outward (left arm left, right arm right).

is a negative sign,

should be given a positive sign. Similarly, the coronal joint angle of the left shoulder

can be calculated in the following way.

은 사람이 걸어갈 때 측정한 왼쪽 위팔뼈의 픽셀길이를 의미한다.in the above formula

is the pixel length of the left upper arm bone measured when a person walks.

30 is a diagram illustrating a skeletal model of a human body displayed in pixel coordinates when a person walks toward the left, according to an exemplary embodiment.

A process of calculating the joint angle when a person turns to the left will be described with reference to FIG. 30 . Figure 2250 of FIG. 30 is a screen extracting an open pose skeleton model when a person walks from a place to the left (the camera is taken from the person's right side).

31 is a view for explaining skeletal models and joint angles of a right leg and a left leg, according to an exemplary embodiment.

의 관계식을 유도하면 다음과 같다.In more detail, figure 2262 of FIG. 31 is a view defining the open pose skeletal model and joint angle of the right leg, and figure 2264 of FIG. 31 is a view showing the open pose skeletal model and joint angle definition of the left leg am. Applying the above-mentioned principle as it is and formulating only the result is as follows. Note that the positive rotation direction of the sagittal joint angle is clockwise when viewed from the left, but counterclockwise when viewed from the right. First, the right hip sagittal angle

If we derive the relational expression of

의 관계식을 유도하면 다음과 같다.Right knee sagittal angle

If we derive the relational expression of

의 관계식을 유도하면 다음과 같다.Left knee sagittal angle

If we derive the relational expression of

은 다리를 왼쪽으로 들어올릴 때는 양의 각도, 오른쪽으로 들어올릴 때는 음의 각도로 정의되어 있음을 기억하자.Next, let's learn how to calculate the coronal joint angle of the lower body. In Figure 17, the coronal joint angle of the lower body

Note that is defined as a positive angle for lifting the leg to the left and a negative angle for lifting the leg to the right.

과 왼쪽 골반의 관상면 관절각

은 다음 식으로 계산할 수 있다.Coronal articular angle of the right pelvis

and coronal articular angles of the left pelvis

can be calculated in the following way.

는 사람이 차려자세로 있을 때 측정한 허벅지뼈의 픽셀길이를,

과

은 사람이 걸어갈 때 측정한 오른쪽과 왼쪽 허벅지뼈의 픽셀길이를 각각 의미한다. 상술한 바와 마찬가지로 사람이 앞으로 걸어갈 때 일반적으로 다리를 바깥쪽(왼쪽 다리는 왼쪽, 오른쪽 다리는 오른쪽)으로 들어올리는 경향이 있으므로,

에는 음의 부호,

에는 양의 부호를 부여하면 된다.in the expression

is the pixel length of the thighbone measured when the person is in a standing position,

class

is the pixel length of the right and left thighbones, respectively, measured when a person walks. As mentioned above, as people generally tend to lift their legs outward (left leg left, right leg right) when walking forward,

is a negative sign,

should be given a positive sign.

32 is a diagram for describing links and joints of a right arm and a left arm, according to an exemplary embodiment.

Now, with reference to FIG. 32 in the same principle, let's calculate the upper body sagittal joint angle. The angular relational expression of the right shoulder sagittal joint is derived from FIG. 32 as follows.

The angular relational expression of the right elbow sagittal joint is derived as follows.

The angular relational expression of the left shoulder sagittal joint is derived from the figure as follows.

From the figure, the angular relational expression of the sagittal joint of the left elbow is derived as follows.

,

은 다음 식으로 계산할 수 있다.Now let's learn how to calculate the coronal joint angle of the upper body and both shoulders. Coronary articular angles of the right and left shoulder according to the principle of projection

,

can be calculated in the following way.

과

은 사람이 걸어갈 때 측정한 오른쪽 위팔뼈와 왼쪽 위팔뼈의 픽셀길이를 각각 의미한다. 앞에서와 마찬가지로 사람이 앞으로 걸어가는 경우에는 일반적으로 팔을 바깥쪽(왼쪽 팔은 왼쪽, 오른쪽 팔은 오른쪽)으로 뻗는 경향이 있으므로,

에는 음의 부호,

에는 양의 부호를 부여했다. in the expression

class

denotes the pixel length of the right upper arm and the left upper arm, respectively, measured when a person walks. As before, when a person walks forward, they generally tend to extend their arms outward (left arm left, right arm right), so

is a negative sign,

is given a positive sign.

33 is a diagram for describing a relationship between an image size and a distance to a camera when a subject faces a camera, according to an embodiment.

According to an embodiment, the subject monitored by the robot device may walk in place, or may walk while moving its position toward the camera. According to an embodiment, the length of the entire body of a person recognized by the open pose model is recognized as being longer as the camera approaches the camera by perspective. Referring to FIG. 33, it can be seen that the position of the subject changes as shown in Fig. 2272 b as the subject walks toward the camera at v speed using the position of the front end of the camera where the subject is standing in Fig. 2272 a as the reference position. In Figure 2272 b, the subject's body length in the image taken by the camera may be longer because the subject is located closer to the camera than in Figure 2272 a.

Referring to Figure 2272 c, in Figure 2272 a, the distance the subject is away from the camera is x1, and the position reached by the subject walking towards the camera for t time. In Figure 2272 b, the distance the subject is away from the camera is x2 , and the distance relationship is shown when the focal length of the camera is f. Based on the relationship in Fig. 2272c, the process of identifying the walking speed and step length of the robot device will be described.

및 대상자를 촬영함으로써 획득된 이미지상 몸통의 픽셀 길이

를 측정한다. 또한, 현재 이미지 상 대상자의 몸통 픽셀 길이

를 측정하고, 카메라로부터 떨어진 대상자의 현재 위치(또는 거리)를 x₀로 측정한다. 로봇 장치는 하기 수학식에 따라 비율 관계를 결정할 수 있다.First, the robotic device identifies the distance between two points and the time spent walking between the distances to determine the walking speed. Actual torso length before subject moves

and the pixel length of the torso on the image obtained by photographing the subject.

measure In addition, the pixel length of the subject's body in the current image

is measured, and the current position (or distance) of the subject away from the camera is measured as x ₀ . The robot device may determine the ratio relationship according to the following equation.

은 픽셀당 길이(미터 단위)를 나타내고, 카메라에 고유한 특성 값이다. 상기 수학식 91에 따른 관계는 하기 수학식과 같이 표현될 수 있다.in the above formula

represents the length per pixel (in meters) and is a characteristic value unique to the camera. The relationship according to Equation 91 can be expressed as the following Equation.

The robot device may derive the measurement position x of the current subject from Equation 92. From this equation, x can be calculated using the current body pixel length as shown in Equation 93 below.

에서

까지 카메라를 향하여 v 속도로 이동시 걸음 속도는 하기 수학식에 따라 로봇 장치에 의해 결정될 수 있다.As shown in Equations 91 to 93, the distance of the subject from the camera may decrease as the length of the body measured on the current image increases. Distance from subject to camera

at

When moving at speed v toward the camera, the walking speed may be determined by the robot device according to the following equation.

및

는 거리

및

에서 사람 몸통 길이를 나타낸다. 로봇 장치(1000)는 대상자가 이동하는 방향의 x축에 대해, 휴머노이드 로봇 시뮬레이션 시 두 발의 중심점 사이의 거리를 측정하고, 측정된 두 발의 중심점 사이의 거리에 기초하여 스텝 길이를 식별할 수 있다. 도 17에서 나타나는 세계 좌표계에서 오른발 및 왼쪽 발의 중심 좌표는 각각

및

에 해당된다. 로봇 장치(1000)는 하기 수학식에 기초하여 스텝 길이를 결정할 수 있다.in the above formula

and

is the distance

and

represents the length of the human body. The robot device 1000 may measure the distance between the center points of the two feet during the humanoid robot simulation with respect to the x-axis in the direction in which the subject moves, and may identify the step length based on the measured distance between the center points of the two feet. The center coordinates of the right foot and the left foot in the world coordinate system shown in FIG. 17 are respectively

and

applies to The robot apparatus 1000 may determine the step length based on the following equation.

In the above equation, sl denotes a step length. The variables in the max and min functions are variables related to the center point coordinates of the right and left feet.

34 is a flowchart of a method for an electronic device to identify a gait pattern of a subject, according to an embodiment.

In S210, the robot device 1000 acquires an image by photographing the subject. According to an embodiment, the robot apparatus 1000 may acquire an image including the subject from an external device connected to the robot apparatus.

More specifically, the robot apparatus 1000 may acquire human skeleton information from the neural network model by inputting the acquired image to the neural network model.

According to an embodiment, the neural network model used by the robot device may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a recurrent neural network (RNN), and a restricted boltzmann (RBM). Machine), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a deep Q-networks, but is not limited thereto.

Also, according to an embodiment, the neural network model used by the robot apparatus 1000 is an OpenPose model, and may be a neural network model that outputs information about joints and links of the human body. The human skeleton information according to the present disclosure may include coordinate information of the subject's joint in the image and length information of a link connecting the joints.

In S230 , the robot device 1000 may identify a photographing angle of the camera with respect to the subject based on a connection relationship of at least one link in the robot model. A process in which the robot device 1000 identifies a photographing angle of a camera for a subject may be performed by Equation 39 described above.

In S240 , the robot device 1000 may determine joint information of the robot model based on the photographing angle of the camera. For example, the robot device 1000 may determine the link length and joint angle of a specific link in the robot model in different ways based on the identified camera angle.

For example, the robot device 1000 identifies two links adjacent to a predetermined joint based on joint information and link information in the robot model, determines an angle between the identified two links, and determines between the determined links. Joint information of the robot model may be determined based on the angle.

In addition, according to an embodiment, the robot device 1000 corrects the angle between the two links determined based on the photographing angle of the camera, and based on the corrected angle between the two links, the robot model Joint information can be determined. In addition, according to an embodiment, the robot device 1000 identifies the quadrant information in which the endpoints of the two adjacent links are located in order to determine the angle between the two links, and based on the identified quadrant information, the two generating a correction function for correcting the link angle of the link, and generating a link correction angle by applying the generated correction function to the angles of the two adjacent links, respectively, using the difference between the generated link correction angles The angle between the two links can be determined.

In S250 , the robot apparatus 1000 may identify the gait pattern of the subject by executing a posture simulation of the humanoid robot based on the joint information determined in S240 . For example, the robot apparatus 1000 may determine the information on each joint in the robot model corresponding to the body skeleton of the subject, and identify the gait pattern of the subject by simulating the determined joint information using the robot model.

In addition, although not shown in FIG. 34 , the robotic device 1000 identifies the length of the subject's torso in the image and the distance between the center points of the subject's two feet in the image, and based on the identified torso length and the distance between the center points of the two feet To determine the walking speed and the step length of the subject, it is also possible to identify the walking pattern based on the joint information, the walking speed and the step length.

According to an embodiment, the robot device 1000 determines the walking speed of the subject based on the change in the subject's torso length between two predetermined points, and based on the distance between the center points of the head of the subject image in the image The step length can be identified.

According to another embodiment, although not shown in FIG. 34 , the robot apparatus 1000 may estimate the walking direction of the subject based on joint information determined from an image obtained by photographing the subject. The robot apparatus 1000 may identify the subject's gait pattern based on at least one of the determined joint information, the estimated subject's gait direction, gait speed, and step length.

35 is a diagram illustrating a process in which an electronic device generates a robot model, according to an exemplary embodiment.

In S310, the robot device 1000 based on the left hip joint by applying the DH transformation matrix to the joint-link model including a preset number of joints disposed based on the skeletal structure of the human body and links connecting the joints. A link coordinate system according to the reference coordinate system can be created. In S320 , the robot device 1000 may generate a human coordinate system based on a body center point by additionally applying a coordinate system transformation matrix to the link coordinate system.

In S330 , the robot apparatus 1000 may generate a world coordinate system by adaptively applying a rotation transformation matrix to the human coordinate system based on a camera photographing angle. In S340 , the robot device 1000 may generate the robot model according to the human coordinate system or the world coordinate system based on the camera photographing angle. According to an embodiment, the robot model may be a model generated based on information about a joint corresponding to a body skeleton of a subject and a link connecting the joint. That is, the robot device 1000 according to the present disclosure determines the camera shooting angle, and when the determined shooting angle is not 0 degrees, converts the robot coordinate system from the human coordinate system to the world coordinate system, and generates a robot model according to the converted world coordinate system can do. In S350 , the robot device 1000 may identify the subject's gait pattern by executing a posture simulation of the humanoid robot using the robot model.

36 is a diagram illustrating a process in which an electronic device identifies a photographing angle of a camera with respect to a subject, according to an exemplary embodiment.

In S410 , the robot device 1000 may identify a face link and a torso link in the robot model. In S420 , the robot device 1000 may identify a direction in which the face link is inclined with respect to the body link based on the identified angle between the face link and the neck link. In S430 , the robot device 1000 may identify a ratio of the length of the torso and the length of the pelvis in the robot model. In S440 , the robot device 1000 may identify a photographing angle of the camera with respect to the subject based on the identified direction and length ratio.

37 is a diagram for describing a process in which an electronic device identifies a gait pattern of a subject according to this embodiment.

In S502, the robot device 1000 measures the arm length of the subject and the distance between the camera and the subject. In S504 , the robot apparatus 1000 may launch an open pose neural network model for neural network calculation of the open pose neural network model. In S506 , the robot apparatus 1000 may acquire initial human skeleton information about the subject's resting posture from the open pose neural network model.

In S508 , the robotic device 1000 may determine a photographing angle of the camera for the subject. A process in which the robot device 1000 identifies a photographing angle of a camera for a subject is the same as described above, and thus will be omitted.

In S510 , the robot device 1000 determines sagittal joint variables and coronal joint variables from each link length change in the robot model. In S512 , the robot device 1000 determines an angle between two adjacent links for each joint. In S514, the robot device 1000 determines the weighted average of the above-described link length and the above-described joint variable based on the photographing angle. In S516 , the robot device 1000 may simulate the generated robot model based on the determined joint variable values. In S518, the robot device 1000 may identify the walking speed and the step length of the subject based on the simulation result. In S520 , the robot device 1000 may determine whether to continue the process of identifying the subject's gait pattern. For example, when the gait pattern of the subject is not identified, the robot apparatus 1000 may perform step S508 again, but when the gait pattern of the subject is identified, the above-described method may be terminated.

38 is a block diagram of an electronic device according to an embodiment.

39 is a block diagram of an electronic device according to another embodiment.

As shown in FIG. 38 , the robot apparatus 1000 according to an embodiment may include a processor 1300 , a network interface 1500 , and a memory 1700 . However, not all illustrated components are essential components. The robot apparatus 1000 may be implemented by more components than the illustrated components, or the robot apparatus 1000 may be implemented by fewer components.

For example, as shown in FIG. 39 , the robot device 1000 includes a processor 1300 , a network interface 1500 and a memory 1700 , a user input interface 1100 , an output unit 1200 , and a sensing unit ( 1400 ), a network interface 1500 , and an A/V input unit 1600 may be further included.

The user input interface 1100 means a means for a user to input data for controlling the robot apparatus 1000 . For example, the user input interface 1100 includes a key pad, a dome switch, and a touch pad (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, red There may be a mechanical tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto. The user input interface 1100 may obtain a keyboard input, voice input, and other touch input for controlling the robot device from the subject.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210 , a sound output unit 1220 , and a vibration motor 1230 . there is. For example, the robot apparatus 1000 may output content according to a user input using the output unit 1200 .

The display unit 1210 includes a screen for displaying and outputting information or content processed by the robot device 1000 . Also, the screen may display an image. For example, the display unit 1210 may display information on a target object recognized by the robot apparatus 1000 and contents matching a user's voice input.

The sound output unit 1220 outputs audio data received from the network interface 1500 or stored in the memory 1700 . Also, the sound output unit 1220 may output response audio data according to a result of voice recognition for a user's voice input. Also, the sound output unit 1220 may output a sound signal related to a function (eg, a call signal reception sound, a message reception sound, and a notification sound) performed by the electronic device 2000 .

The processor 1300 typically controls the overall operation of the robot device 1000 . For example, the processor 1300 executes programs stored in the memory 1700 , so that the user input interface 1100 , the output unit 1200 , the sensing unit 1400 , the network interface 1500 , and the A/V input unit (1600) and the like can be controlled in general. In addition, the processor 1300 may execute the programs stored in the memory 1700 to perform the functions of the robot apparatus 1000 described in FIGS. 1 to 37 .

According to an embodiment, the processor 1300 acquires an image by photographing the subject through a camera connected to the robot device by executing the one or more instructions, and preset based on the human skeleton information obtained from the image. Generate a robot model according to a coordinate system, identify a shooting angle of the camera with respect to the subject based on a connection relationship of at least one link in the generated robot model, and based on the shooting angle of the camera, the robot model It is possible to determine the joint information, and identify the gait pattern of the subject based on the determined joint information.

According to an embodiment, the processor 1300 identifies the length of the subject's torso in the image and the distance between the center points of the subject's two feet in the image, and based on the identified length of the torso and the distance between the center points of the two feet It is possible to identify the walking speed and step length of the subject, and identify the walking pattern based on the joint information, the walking speed and the step length.

According to an embodiment, the processor 1300 may acquire human skeleton information from the neural network model by inputting the acquired image to the neural network model.

According to an embodiment, the at least one processor applies a DH transformation matrix to a joint-link model including a preset number of joints disposed based on the skeletal structure of the human body and links connecting the joints to the left hip joint. Generates a link coordinate system according to a reference coordinate system based on , and additionally applies a coordinate system transformation matrix to the link coordinate system to generate a human coordinate system based on a body center point, and adaptively based on the camera shooting angle, the human coordinate system A world coordinate system may be generated by applying a rotation transformation matrix to , and the robot model according to the human coordinate system or the world coordinate system may be generated based on the camera shooting angle.

According to an embodiment, the at least one processor identifies a facial link and a torso link in the robot model, and identifies a direction in which the facial link is inclined with respect to the torso link based on an angle between the identified facial link and the neck link. and identify a length ratio of a torso and a pelvis in the robot model, and identify a shooting angle of the camera for the subject based on the identified direction and the length ratio.

According to an embodiment, the processor 1300 identifies two links adjacent to a predetermined joint based on joint information and link information in the robot model, determines an angle between the identified two links, and Joint information of the robot model may be determined on the basis of the determined interstitial angle.

According to an embodiment, the processor 1300 corrects the angle between the two links determined based on the angle taken by the camera, and calculates the joint information of the robot model based on the corrected angle between the two links. can decide

According to an embodiment, the processor 1300 identifies quadrant information in which the endpoints of the two adjacent links are located, and generates a correction function for correcting the link angle of the two links based on the identified quadrant information, and , a link correction angle may be generated by applying the generated correction function to the angles of the two adjacent links, respectively, and an angle between the two links may be determined using a difference between the generated link correction angles.

According to an embodiment, the at least one processor determines the walking speed of the subject based on the change in the subject's torso length between two predetermined points, and the distance between the center points of the subject's two feet of the subject's three-dimensional simulation in the image. Based on the step length can be identified.

The sensing unit 1400 may detect a state of the robot apparatus 1000 or a state around the robot apparatus 1000 , and transmit the sensed information to the processor 1300 . The sensing unit 1400 may be used to sense specification information of the robot device 1000, status information of the robot device 1000, surrounding environment information of the robot device 1000, user status information, user location information, and the like. there is.

The sensing unit 1400 includes a magnetic sensor 1410 , an acceleration sensor 1420 , a temperature/humidity sensor 1430 , an infrared sensor 1440 , a gyroscope sensor 1450 , and a position sensor. (eg, GPS) 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , and at least one of an illuminance sensor 1490 , but is not limited thereto. Since a function of each sensor can be intuitively inferred from a person skilled in the art from the name, a detailed description thereof will be omitted.

The network interface 1500 may include one or more components that allow the robot device 1000 to communicate with other devices (not shown) or the server 2000 . The other device (not shown) may be a computing device or a sensing device capable of transmitting/receiving data to and from the robot device 1000 , but is not limited thereto. For example, the network interface 1500 may include a short-distance communication unit 1510 , a mobile communication unit 1520 , and a broadcast receiving unit 1530 .

Short-range wireless communication unit 1510, Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, short-range wireless communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared ( It may include an IrDA, infrared Data Association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, and the like, but is not limited thereto.

The mobile communication unit 1520 transmits/receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text/multimedia message.

The broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an embodiment, the robot device 1000 may not include the broadcast receiver 1530 .

The A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 , a microphone 1620 , and the like. The camera 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a shooting mode. For example, the camera 1610 may acquire an image of the subject by photographing the subject. Also, the camera 1610 may acquire a moving picture by acquiring an image of a subject according to a frame. An image captured through an image sensor in the camera may be processed through the processor 1300 or a separate image processing unit (not shown).

The microphone 1620 receives an external sound signal and processes it as electrical voice data. For example, the microphone 1620 may receive an acoustic signal or a voice signal from an external device or a user. The microphone 1620 may receive a user's voice input. The microphone 1620 may use various noise removal algorithms for removing noise generated in the process of receiving an external sound signal. The microphone 1620 transmits the acquired user voice signal to the processor 1300 . The processor 1300 may identify the user's voice by analyzing the user's voice signal using a predetermined neural network model or a model for voice recognition.

The memory 1700 may store a program for processing and control of the processor 1300 , and may store data input to the robot device 1000 or output from the electronic device 2000 . Also, the memory 1700 may store information on the robot model, information on links or joints in the robot model, information on the neural network model, information on the voice recognition model, and the like. Also, the memory 1700 may further include information on various contents that may be displayed according to a user's voice.

According to an embodiment, the memory 1700 may further store parameter information on at least one artificial intelligence model used by the robot apparatus 1000 . For example, the memory 1700 may store layers in at least one neural network model, nodes, and weight values related to connection strengths of the layers. In addition, the robot apparatus 1000 may further store various learning data for learning the artificial intelligence model.

The memory 1700 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks.

Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions, for example, may be classified into a UI module 1710 , a touch screen module 1720 , a notification module 1730 , and the like. .

The UI module 1710 may provide a specialized UI, GUI, etc. interworking with the robot device 1000 for each application. The touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the processor 1300 . The touch screen module 1720 according to some embodiments may recognize and analyze a touch code. The touch screen module 1720 may be configured as separate hardware including a controller.

The notification module 1730 may generate a signal for notifying the occurrence of an event in the robot device 1000 . Examples of events generated in the robot device 1000 include call signal reception, message reception, key signal input, schedule notification, and the like. The notification module 1730 may output a notification signal in the form of a video signal through the display unit 1210 , may output a notification signal in the form of an audio signal through the sound output unit 1220 , and the vibration motor 1230 . It is also possible to output a notification signal in the form of a vibration signal through

The method for identifying a subject's gait pattern according to an embodiment may be implemented in the form of program instructions that may be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present disclosure, or may be known and available to those skilled in the art of computer software.

In addition, according to the embodiment, a computer program apparatus including a recording medium storing a program for performing another method may be provided. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improved forms of the present disclosure are also provided by those skilled in the art using the basic concept of the present disclosure as defined in the following claims. belong to the scope of the right.

Claims (19)

A method for a robot device to identify a gait pattern of a subject, the method comprising:
acquiring an image by photographing the subject through a camera connected to the robot device;
generating human skeleton information from the acquired image;
identifying a photographing angle of the camera with respect to the subject based on a connection relationship of at least one link in the generated human skeleton information;
determining joint information of the robot model based on the photographing angle of the camera; and
identifying a gait pattern of the subject by executing a posture simulation of a humanoid robot based on the determined joint information; A method comprising The method of claim 1, wherein the method
identifying a distance between the subject's torso length in the image and the center point of the subject's two feet in the three-dimensional model;
identifying a walking speed and a step length of the subject based on the identified trunk length and the distance between the center points of the two feet;
estimating the walking direction of the subject; and
identifying the gait pattern based on the joint information, the gait speed, the step length, and the estimated gait direction; A method comprising The method of claim 1, wherein the method
acquiring human skeleton information from the neural network model by inputting the acquired image into a neural network model; further comprising,
The human skeleton information is characterized in that it comprises coordinate information of the joint of the subject in the image and length information of a link connecting the joints. The method of claim 1, wherein the method
Link coordinate system according to the reference coordinate system based on the left hip joint by applying the DH transformation matrix to the joint-link model including a preset number of joints arranged based on the skeletal structure of the human body and links connecting between the joints. generating;
generating a human coordinate system located at a body center point by additionally applying a coordinate system transformation matrix to the link coordinate system;
generating a world coordinate system by adaptively applying a rotation transformation matrix to the human coordinate system based on the camera shooting angle;
generating the robot model according to a human coordinate system or a world coordinate system based on the photographing angle of the camera; and
Recognizing a gait pattern of the subject by executing a posture simulation of the humanoid robot using the robot model. The method of claim 1, wherein the step of identifying the shooting angle comprises:
identifying a facial link and a torso link in the robot model;
identifying a direction in which the facial link is inclined with respect to the body link based on the angle between the identified facial link and the neck link;
identifying a ratio of the length of the torso and the length of the pelvis in the robot model;
identifying a shooting angle of the camera for the subject based on the identified direction and the length ratio; A method comprising According to claim 1, wherein the step of determining the joint information of the robot model
identifying two adjacent links with respect to a predetermined joint based on joint information and link information in the robot model;
determining an angle between the identified two links; and
determining joint information of the robot model based on the determined angle; A method comprising According to claim 6, The step of determining the joint information of the robot model
correcting the angle between the two links determined based on the photographing angle of the camera; and
determining joint information of the robot model based on the corrected angle between the two links; A method comprising The method of claim 6, wherein determining the angle between the two links comprises:
identifying information about a quadrant in which endpoints of the two adjacent links are located;
generating a correction function for correcting a link angle of the two links based on the identified quadrant information;
generating a link correction angle by applying the generated correction function to the angles of the two adjacent links, respectively; and
determining an angle between the two links using a difference between the generated link correction angles; A method comprising The method of claim 2, wherein the step of identifying the walking speed and the step length comprises:
determining the walking speed of the subject based on the change in the length of the subject's torso at two predetermined time points; and
identifying the step length based on the distance between the center points of the two feet during the subject simulation in the image; A method comprising A robot device for identifying a subject's gait pattern, comprising:
network interface;
a memory storing one or more instructions; and
at least one processor; including,
The at least one processor by executing the one or more instructions,
Obtaining an image by photographing the subject through a camera connected to the robot device,
Obtain human skeleton information from the image,
Identifies a shooting angle of the camera with respect to the subject based on a connection relationship of at least one link in the obtained human skeleton information,
Determine the joint information of the robot model based on the shooting angle of the camera,
A robot device that identifies the subject's gait pattern by executing a posture simulation of the humanoid robot based on the determined joint information. 11. The method of claim 10, wherein the at least one processor comprises:
identifying the length of the subject's torso in the image and the distance between the center points of the subject's two feet in the three-dimensional model,
identifying the walking speed and step length of the subject based on the identified trunk length and the distance between the center points of the two feet;
Estimate the walking direction of the subject,
A robot device that identifies the gait pattern based on the joint information, the gait speed, the step length, and the estimated gait direction. 12. The method of claim 11, wherein the at least one processor comprises:
Obtaining the human skeleton information from the neural network model by inputting the obtained image into a neural network model,
The human skeleton information is a robot device, characterized in that it includes coordinate information of the joint of the subject in the image and length information of a link connecting the joints. 12. The method of claim 11, wherein the at least one processor comprises:
Link coordinate system according to the reference coordinate system based on the left hip joint by applying the DH transformation matrix to the joint-link model including a preset number of joints arranged based on the skeletal structure of the human body and links connecting between the joints. create,
By additionally applying a coordinate system transformation matrix to the link coordinate system, a human coordinate system located at the center of the body is generated,
A world coordinate system is generated by adaptively applying a rotation transformation matrix to the human coordinate system based on the shooting angle of the camera,
Generating the robot model according to the human coordinate system or the world coordinate system based on the shooting angle of the camera,
A robot device that identifies a gait pattern of the subject by executing a posture simulation of the humanoid robot using the robot model. 12. The method of claim 11, wherein the at least one processor comprises:
identifying a facial link and a torso link in the robot model;
identify a direction in which the facial link is inclined with respect to the body link based on the angle between the identified facial link and the neck link;
Identifies the ratio of the length of the torso and the length of the pelvis in the robot model,
identifying a shooting angle of the camera with respect to the subject based on the identified direction and the length ratio. 12. The method of claim 11, wherein the at least one processor comprises:
Identifies two adjacent links with respect to a predetermined joint based on joint information and link information in the robot model,
Determining the angle between the identified two links,
A robot device for determining joint information of the robot model on the basis of the determined in-between angle. 16. The method of claim 15, wherein the at least one processor comprises:
Correcting the angle between the two links determined on the basis of the shooting angle of the camera,
For determining the joint information of the robot model based on the corrected angle between the two links, the robot device. 16. The method of claim 15, wherein the at least one processor comprises:
Identifies the quadrant information in which the endpoints of the two adjacent links are located,
generating a correction function for correcting a link angle of the two links based on the identified quadrant information;
generating a link correction angle by applying the generated correction function to the angles of the two adjacent links, respectively;
Using the difference of the generated link correction angle to determine the angle between the two links, the robot device. 13. The method of claim 12, wherein the at least one processor comprises:
Determine the subject's walking speed based on the change in the subject's trunk length between two predetermined points,
identifying the step length based on the distance between the center points of the subject's three-dimensional model of the subject in the image. A method for a robot device to identify a gait pattern of a subject, the method comprising:
acquiring an image by photographing the subject through a camera connected to the robot device;
generating human skeleton information from the acquired image;
identifying a photographing angle of the camera with respect to the subject based on a connection relationship of at least one link in the generated human skeleton information;
determining joint information of the robot model based on the photographing angle of the camera; and
identifying a gait pattern of the subject by executing a posture simulation of a humanoid robot based on the determined joint information; A computer-readable recording medium having a program stored thereon for performing a method comprising a.

Images