KR20230043653A - Hand joint 3d pose estimation method and hand joint 3d pose estimation system based on stereo camera input - Google Patents

Abstract

A joint position estimation method based on stereo camera input according to an embodiment of the disclosed invention comprises: (a) a step of receiving a first image acquired by a first camera and a second image acquired by a second camera by a stereo image input unit; (b) a step of extracting a first feature from the first image and a second feature from the second image using a deep learning model by an image feature extraction unit; (c) a step of generating first joint 2D position information and second joint 2D position information based on the first feature and the second feature by a stereo attention block unit; and (d) a step of determining the 3D position of the hand joint using a stereo deep learning model based on the first joint 2D position information and the second joint 2D position information, by a hand joint 3D position determination unit. Accordingly, the present invention can reduce the cost of data collection.

Description

Hand joint 3D position estimation method and hand joint 3D position estimation system based on stereo camera input

The present invention relates to a method for estimating a 3D position of a hand joint and a system for estimating a 3D position of a hand joint capable of estimating a 3D position of a hand joint based on an image received from a stereo camera.

As various types of wearable cameras are sold, various VR/AR-based technologies such as rehabilitation treatment using them, medical simulation education, life pattern analysis, and AR applications have been introduced. This wearable camera-based application technology interacts based on the user-centered gaze because the user is wearing the device. The most useful input tool for interaction is the hand, and hand posture estimation is essential for free interaction.

There are two types of hand posture estimation: absolute position estimation and relative position estimation. In absolute position estimation, the origin is the camera and the position of the hand joint is estimated based on the camera position. On the other hand, in the relative position estimation, the origin is one of the hand joints, and the relative position of the hand joint is estimated from the position of the corresponding joint. In other words, since absolute position estimation knows the position between the user's hand and the camera, it has the advantage of providing various interactions by utilizing the position of the hand.

For absolute position estimation, it is essential to obtain depth information. In order to obtain depth information, there is a method of using an RGBD camera or a stereo camera. The RGBD camera emits an infrared pattern and based on it, depth information is obtained. However, this method has a disadvantage in that inaccurate information may be obtained due to interference with other light, and a lot of power is consumed because infrared patterns are emitted. On the other hand, a stereo camera has the advantage of no light interference or additional power consumption because it obtains depth information using the parallax of the two cameras.

However, the current technique for estimating the absolute position of the hand joint based on a stereo camera has limitations. First, the existing deep learning-based technology does not use the geometric information of the stereo camera for learning. Although a stereo camera-based deep learning algorithm is proposed, it only learns information about hand joint positions and does not use geometric information for learning. Second, the current technology estimates the absolute position of the hand joint based on several assumptions. We use the assumption that the hand is in front of other objects. This assumption limits free use. Third, a dataset (a pair of images and an answer sheet) is required to train a deep learning model, but currently, the answer sheet is created manually. This method has limitations in that it takes a lot of time as the number of datasets increases and the accuracy may be inaccurate.

An object of the present invention is to provide a hand joint 3D position estimation method and a hand joint 3D position estimation system capable of estimating the 3D position of a hand joint more accurately than conventional hand joint position estimation methods.

In addition, the present invention provides a hand joint 3D position estimation method and a hand joint 3D position estimation system that enables a user to perform various tasks regardless of the type of task in a wearable environment based on a stereo camera by performing accurate hand posture estimation without special assumptions. is to provide

In addition, the present invention provides a hand joint 3D position estimation method and hand joint 3D position estimation method that can accurately and quickly collect data required for learning a machine learning model used for hand joint 3D position estimation and reduce the cost of data collection. It is to provide a position estimation system.

A joint position estimation method based on a stereo camera input according to an aspect of the disclosed invention includes: (a) inputting a first image obtained by a first camera and a second image obtained by a second camera through a stereo image input unit; receiving step; (b) extracting a first feature from a first image and a second feature from a second image using a deep learning model by an image feature extractor; (c) generating first joint 2D position information and second joint 2D position information based on the first feature and the second feature by the stereo attention block unit; and (d) using a stereo deep learning model based on the first joint 2D position information and the second joint 2D position information to determine the hand joint 3D position by the hand joint 3D position determiner.

The method may further include outputting, by the heat map block unit, a first joint 2D location map based on the first feature and outputting a second joint 2D location map based on the second feature.

Also, the step (c) may include: generating a combined feature by combining the first feature and the second feature by the stereo attention block unit; and generating, by the stereo attention block unit, a mask map in which components have values greater than or equal to 0 and less than or equal to 1, based on the combined feature.

In the step (c), the stereo attention block unit generates the first joint 2D location information based on the mask map and the first joint 2D location map, and generates the mask map and the second joint 2D location information. The method may include generating 2D location information of the second joint based on a 2D location map.

In addition, the step (c) includes generating the same joint parallax information based on the coupling feature by the stereo attention block unit, and the step (d) includes the step of generating the same joint disparity information by the hand joint 3D positioning unit and determining the 3D position of the hand joint based on the 2D position information of the first joint, the 2D position information of the second joint, and the parallax information of the same joint.

In addition, the generating of the same-joint parallax information may include coordinates in the first image and coordinates in the second image of the same object of the first image and the second image by the stereo attention block unit. It may include generating the same-joint parallax information based on.

In addition, the step of receiving a plurality of image pairs for learning by the stereo image input unit; generating joint 2D position information and same-joint parallax information of the training image pair by the stereo attention block unit; and learning, by an artificial intelligence learning unit, the stereo deep learning model based on 2D joint position information and parallax information of the same joint of the training image pair.

In addition, the step of learning the stereo deep learning model: a first loss function based on the 2D joint position information of the training image pair and the reference joint 2D position information of the training image pair generated by the artificial intelligence learning unit. calculating; calculating, by the artificial intelligence learning unit, a second loss function based on the same-joint disparity information of the training image pair and the reference same-joint disparity information of the training image pair; Calculating, by the artificial intelligence learning unit, a third loss function based on 2D joint position information of the training image pair and projected position information generated by projecting the 2D joint position of the training image pair to an opposite image generated by the artificial intelligence learning unit. ; and calculating an integration loss function based on the first loss function, the second loss function, and the third loss function by the artificial intelligence learning unit, and learning the stereo deep learning model so that the integration loss function decreases. steps may be included.

In addition, the step of changing the brightness of the input learning image by the change learning image generation unit to generate a plurality of changed learning images; And a learning data generator generates joint position information for each of the changed learning images, compares the joint position information of a plurality of changed learning images, and reduces the difference between the joint position information of the changed learning images. Calculating joint parallax information may be further included.

A computer program according to an aspect of the disclosed invention may be stored in a computer-readable recording medium to execute the joint position estimation method based on the stereo camera input.

A joint position estimation system based on a stereo camera input according to an aspect of the present invention includes a stereo image input unit configured to receive a first image obtained by a first camera and a second image obtained by a second camera; an image feature extraction unit configured to extract a first feature from a first image and a second feature from a second image by using a deep learning model; a stereo attention block unit configured to generate first joint 2D position information and second joint 2D position information based on the first and second characteristics; and a hand joint 3D position determiner configured to determine a hand joint 3D position using a stereo deep learning model based on the first joint 2D position information and the second joint 2D position information.

It may further include a heat map block unit configured to output a first joint 2D location map based on the first feature and output a second joint 2D location map based on the second feature.

In addition, the stereo attention block unit: generates a combined feature by combining the first feature and the second feature; And it may be configured to generate a mask map in which components have values of 0 or more and 1 or less based on the combination feature.

The stereo attention block unit may: generate the first joint 2D position information based on the mask map and the first joint 2D position map; And it may be configured to generate the second joint 2D location information based on the mask map and the second joint 2D location map.

In addition, the stereo attention block unit is configured to generate identical joint parallax information based on the coupling feature, and the hand joint 3D position determining unit includes the first joint 2D position information, the second joint 2D position information, and the same joint 3D position information. It may be configured to determine the hand joint 3D position based on joint parallax information.

The stereo attention block unit may be configured to generate the same joint parallax information based on coordinates in the first image and coordinates in the second image of the same object in the first image and the second image. can

In addition, the stereo image input unit is configured to receive a plurality of image pairs for learning, and the stereo attention block unit is configured to generate joint 2D position information and parallax information of the same joint of the image pair for learning, An artificial intelligence learning unit configured to learn the stereo deep learning model based on joint 2D position information and the same joint parallax information may be further included.

In addition, the artificial intelligence learning unit: calculates a first loss function based on the generated joint 2D position information of the image pair for learning and the reference joint 2D position information of the image pair for training; calculating a second loss function based on the generated same-joint disparity information of the training image pair and the reference joint disparity information of the training image pair; Calculate a third loss function based on the generated 2D joint position information of the training image pair and projected position information generated by projecting the 2D joint position of the training image pair to an opposite image; An integrated loss function may be calculated based on the first loss function, the second loss function, and the third loss function, and the stereo deep learning model may be learned to reduce the integrated loss function.

In addition, a change learning image generator configured to generate a plurality of changed learning images by changing the brightness of the input learning image; and generating joint position information for each image for change learning, and comparing joint position information of a plurality of images for change learning, and calculating reference same-joint parallax information by reducing a difference between joint position information of the image for change learning. It may further include a data generation unit for learning.

According to one aspect of the disclosed invention, it is possible to estimate the 3D position of the hand joint more accurately than the conventional method for estimating the position of the hand joint.

In addition, according to an embodiment of the present invention, accurate hand posture estimation is performed without special assumptions, so that a user can perform various tasks regardless of the type of task in a wearable environment based on a stereo camera.

Finally, according to an embodiment of the present invention, data collection necessary for learning a machine learning model used for hand joint 3D position estimation can be performed accurately and quickly, and the cost of data collection can be reduced.

1 is a configuration diagram of a joint estimation system according to an embodiment.
2 is a diagram for explaining a joint position estimation process according to an embodiment.
3 is a diagram illustrating a process of determining a 3D position of a hand joint according to an embodiment.
4 is a diagram for explaining an operation of a stereo attention block unit according to an exemplary embodiment.
5 is a diagram for explaining same-joint parallax information according to an exemplary embodiment.
6 is a diagram for explaining a process of generating learning data according to an exemplary embodiment.
7 is a flowchart of a joint position estimation method according to an embodiment.
8 is a graph showing the degree of improvement of the joint position estimation method according to an embodiment compared to the conventional joint position estimation method.

Like reference numbers designate like elements throughout the specification. This specification does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the disclosed invention belongs is omitted. The term '~unit' used in the specification may be implemented in software or hardware, and according to embodiments, a plurality of '~units' may be implemented as one component, or one '~unit' may constitute a plurality of components. It is also possible to include elements.

In addition, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated.

Terms such as first and second are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In each step, the identification code is used for convenience of description, and the identification code does not explain the order of each step, and each step may be performed in a different order from the specified order unless a specific order is clearly described in context. there is.

Hereinafter, the working principle and embodiments of the disclosed invention will be described with reference to the accompanying drawings.

1 is a configuration diagram of a joint estimation system according to an embodiment, and FIG. 2 is a diagram for explaining a joint position estimation process according to an embodiment.

1 and 2, the joint position estimation system 100 according to an embodiment of the present invention includes a stereo image input unit 110, an image feature extraction unit 120, a stereo attention block unit 130, a hand joint 3D It may include a location determination unit 140, a heat map block unit 150, an artificial intelligence learning unit 160, and a memory 170.

The wearable device 200 may be a device that can be worn by a person like glasses. The wearable device 200 may include a first camera 201 and a second camera 202 . The first camera 201 and the second camera 202 may be located near the user's left eyeball and right eyeball, respectively. That is, the first camera 201 and the second camera 202 may obtain image information from positions where the left and right eyeballs of the person are looking, respectively.

The stereo image input unit 110 may receive a first image 301 obtained by the first camera 201 and a second image 302 obtained by the second camera 202 .

The first image 301 is a left image acquired by the first camera 201 near the user's left eyeball, and the second image 302 is acquired by the second camera 202 near the user's right eyeball. It may be one image (Right image), but is not limited thereto. That is, conversely, the first image 301 may be an image acquired near the user's right eyeball and the second image 302 may be an image acquired near the user's left eyeball. In addition, when the first camera 201 and the second camera 202 are arranged up and down instead of left and right, the first image 301 and the second image 302 are relatively images acquired from the left side and images acquired from the right side. It may be an image obtained from a relatively top or bottom, respectively.

In summary, the first image 301 and the second image 302 may be images obtained by taking pictures of a specific location, a specific area, or the same object at different locations by two cameras at a specific time point.

The stereo image input unit 110 may wirelessly receive the first image 301 and the second image 302 from the first camera 201 and the second camera 202 . At this time, the stereo image input unit 110 communicates wirelessly with the wearable device 200 using any method such as Wi-Fi, LTE, 4G, or 5G. Also, the stereo image input unit 110 may receive the first image 301 and the second image 302 from the wearable device 200 through a wired communication method.

Meanwhile, the stereo image input unit 110 may extract a first area image from the first image 301 and a second area image from the second image 302 . The first region image and the second region image may be cropped images extracted in the form of a bounding box in which the user's hand is located in the first image 301 or the second image 302, respectively. there is. Detecting the region where the user's hand is located may be a method using an object detection deep learning model, for example, YOLOv3, but is not limited thereto.

The image feature extractor 120 may extract features from an image. A feature of a specific image may be information indicating various characteristics of the image. For example, the feature of a specific image may be information about color, brightness, boundary, etc. in each pixel unit of the image, but is not limited thereto.

The image feature extractor 120 extracts a first feature 401 from the first image 301 and a second feature 402 from the second image 302 using the deep learning model 171. can In this case, the image feature extractor 120 may extract a first feature 401 from the first area image and a second feature 402 from the second image 302 . That is, the first feature 401 may be a feature of the first image 301 , and the second feature 402 may be a feature of the second image 302 .

The deep learning-based object detection technology may use a deep learning model 171 pre-learned based on data of features extracted from an image. In this case, a convolutional neural networks (CNN) structure in which several convolutional layers are stacked may be used to learn a method of extracting features from an image, but is not limited thereto.

The stereo attention block unit 130 may generate first joint 2D position information 601 and second joint 2D position information 602 based on the first feature 401 and the second feature 402 .

The first joint 2D position information 601 may be information related to plane coordinate values where the joints of the photographed hand are located on the planar first image 301 . The second joint 2D position information 602 may be information related to plane coordinate values where the joints of the photographed hand are located on the planar second image 302 . At this time, even if the joint position information is generated for the same hand, the plane coordinate values of the joints on the first image 301 and the second image 302 are due to the position difference between the first camera 201 and the second camera 202. may be different from each other.

The hand joint 3D position determining unit 140 determines the absolute 3D hand position (Absolute 3D hand) position by using the stereo deep learning model 172 based on the first joint 2D position information 601 and the second joint 2D position information 602. pose) can be determined.

The stereo deep learning model 172 may be an artificial intelligence model learned through a machine learning method. The 3D positions of the hand joints may be information related to three-dimensional coordinate values in which the joints of the photographed hand are located in a three-dimensional 3D space.

The stereo image input unit 110, the image feature extraction unit 120, the stereo attention block unit 130, the hand joint 3D position determination unit 140, the heat map block unit 150, and the artificial intelligence learning unit 160 are It may include any one processor among a plurality of processors included in the position estimation system 100 . In addition, the joint position estimation method according to the embodiments of the present invention described so far and the embodiments to be described in the future may be implemented in the form of a program that can be driven by a processor.

Here, the program may include program commands, data files, and data structures alone or in combination. The program may be designed and manufactured using machine language codes or high-level language codes. The program may be specially designed to implement the above-described code correction method, or may be implemented using various functions or definitions that are known and usable to those skilled in the art in the field of computer software. A program for implementing the above information display method may be recorded on a recording medium readable by a processor. At this time, the recording medium may be the memory 170 .

The memory 170 may store a program for performing the above-described operation and an operation to be described later, and the memory 170 may execute the stored program. When there are a plurality of processors and memories 170, they may be integrated into a single chip, or may be provided in physically separate locations. The memory 170 may include volatile memory such as static random access memory (S-RAM) and dynamic random access memory (D-lab) for temporarily storing data. In addition, the memory 170 may include a read only memory (ROM), an erasable programmable read only memory (EPROM), and an electrically erasable programmable read only memory (EEPROM) for long-term storage of control programs and control data. It may contain non-volatile memory.

The processor may include various logic circuits and arithmetic circuits, process data according to programs provided from the memory 170, and generate control signals according to processing results.

3 is a diagram illustrating a process of determining a 3D position of a hand joint according to an exemplary embodiment, and FIG. 4 is a diagram for explaining an operation of a stereo attention block unit according to an exemplary embodiment.

Referring to FIG. 3 , a first feature (XL) 401 and a second feature (XR) 402 are based on a first image (Left image) 301 and a second image (Right image) 302. extracted, a first joint 2D position map (M2,L) 501 is generated based on the first feature 401, and a second joint 2D position map (M2,R) based on the second feature 402 It can be confirmed that 502 is created.

In addition, based on the first feature 401, the second feature 402, the first joint 2D location map 501 and the second joint 2D location map 502, first joint 2D location information (2D left Keypoints) ( 601), second joint 2D position information (2D right Keypoints) 602, and same joint disparity information (Disparities) 603 are generated.

The heat map block unit 150 outputs a first joint 2D position map 501 based on the first feature 401 and outputs a second joint 2D position map 502 based on the second feature 402. can do.

The first joint 2D location map 501 may be a feature map representing coordinate information of joints of the hand captured in the first image 301 . The second joint 2D location map 502 may be a feature map representing coordinate information of joints of the hand captured in the second image 302 .

In summary, the heat map block unit 150 outputs a feature map related to the position where the joint of the hand is expected to be located in the first image 301 based on the first feature 401, and the second feature 402 Based on this, it is possible to output a feature map related to a position where the joint of the hand is expected to be located in the second image 302 .

Referring to FIGS. 3 and 4 , the stereo attention block unit 130 may generate a combined feature 403 by combining a first feature 401 and a second feature 402 . That is, the combination feature 403 may be a feature including both information related to the first image 301 and information related to the second image 302 . At this time, the process of generating the combined feature 403 may include a process of increasing the size of the feature due to a deconvolution process.

The stereo attention block unit 130 may generate mask maps (A(XL) and A(XR)) 503 in which components have values greater than 0 and less than 1 based on the combination feature 403 . The mask map 503 may be a map generated by scaling components of the combined features 403 to have a value greater than or equal to 0 and less than or equal to 1. In summary, each component of the combination feature 403 may have a value exceeding 1, but the stereo attention block unit 130 scales the combination feature 403 so that each component has a value between 0 and 1 (attention mask). mask) can be created.

The stereo attention block unit 130 generates first joint 2D position information 601 based on the mask map 503 and the first joint 2D position map 501, and the mask map 503 and the second joint 2D position Second joint 2D position information 602 may be generated based on the map 502 .

Specifically, the stereo attention block unit 130 calculates a value obtained by adding the feature map of the first image 301 to a value obtained by multiplying the feature map of the first image 301 by the mask map 503 as the first joint 2D position. The second joint 2D position information (602 ) can be determined.

[Equation 1]

Referring to [Equation 1], M'2,R is the second joint 2D position information 602, A(C(XR,XL)) is a mask map, and M2,R(XR) is the second joint 2D It may be a location map 502 . At this time, the second joint 2D location map 502 is added to the value obtained by multiplying the second joint 2D location map 502, which is a feature map of the second image 302, by the mask map 503, and the second joint 2D location information 602 ) can be calculated, and the first joint 2D position information 601 can be calculated in the same way.

The first joint position 2D information and the second joint position information may be a heat map. That is, in the above-described manner, the stereo attention block unit 130 may generate two types of heat maps. The heat map may be a map for estimating relative positions of hand joints.

Specifically, the stereo attention block unit 130 is a heat map (M') for the first image 301 based on the mask map 503 and the first joint 2D position map 501 for the first image 301 . 2,L), and generate a heat map (M′2,R) for the second image 302 based on the mask map 503 and the second joint 2D position map 502 for the second image 302 ) can be created.

The heat map may be expressed as a 2D map. The stereo attention block unit 130 may generate as many heat maps for the first image 301 as the number of joints, and generate a heat map for the second image 302 as many as the joints. For example, if there are 21 joints to be estimated, 21 2D maps for the first image 301 and 21 2D maps for the second image 302 may be created. Each two-dimensional map has information on the probability that a joint exists at a corresponding location.

The stereo attention block unit 130 may generate the same-joint parallax map D based on the coupling feature 403 . Same-joint parallax maps can also be created as many maps as the number of joints.

According to an embodiment, a calculation method such as soft-argmax may be applied to the heat map obtained in the above-described manner, and as a result of the calculation, an actual estimated value rather than a probability value may be obtained. In this case, the 2D coordinates of the hand joint may be expressed as the same location on the heat map.

The stereo attention block unit 130 may generate the same joint parallax information (D) 603 based on the coupling feature 403 . The same-joint disparity information 603 is information used for estimating the disparity between two stereo images, and may be used when generating depth information. The same joint parallax information 603 may also generate a map as many as the number of joints.

5 is a diagram for explaining same-joint parallax information according to an exemplary embodiment.

Referring to FIG. 5 , another 2D disparity map may be used to estimate the disparity between two stereo images.

The stereo attention block unit 130 is the same joint based on the coordinates in the first image 301 and the coordinates in the second image 302 of the first image 301 and the second image 302 for the same object. Time difference information 603 may be generated. That is, the stereo attention block unit 130 may generate a 2D disparity map using binocular disparity (positional differences of objects seen in two images).

[Equation 2]

)를 계산한다. 상대 시차는 상대적으로 크기를 줄인 시차를 의미하고, [방정식 2]를 통하여 계산될 수 있다. 여기에서 wb와

는 임의의 값일 수 있다.Specifically, uL position values may be used for the x-axis and uR values may be used for the y-axis. For example, if (uL, vL) is (2,3) and (uR, vR) is (4,5), it is expressed as (2,4) in the 2D disparity map. Based on this calculated value, the relative parallax (

) is calculated. The relative disparity means a disparity that has been relatively reduced in size, and can be calculated through [Equation 2]. wb here

can be any value.

Referring back to FIGS. 3 and 4 , the hand joint 3D position determining unit 140 based on the first joint 2D position information 601 , the second joint 2D position information 602 and the same joint parallax information 603 The 3D position of the hand joint can be determined.

Specifically, the hand joint 3D position determination unit 140 determines the relative position of the joint based on the first joint 2D position information 601 and the second joint 2D position information 602, [Equation 3], [Equation 3] 4] and [Equation 5], information on the absolute 3D joint position can be generated.

[Equation 3]

는 원점이 원 이미지의 오른쪽 상단인 관절 위치이고,

는 원점이 손 영역탐지로 인해 잘린 이미지 오른쪽 상단으로 조정된 위치일 수 있다.Referring to [Equation 3],

is the joint position where the origin is the upper right corner of the original image,

may be a position where the origin is adjusted to the top right of the cropped image due to hand region detection.

[Equation 4]

) 스테레오 이미지의 상대 시차 계산에 이용된 wb와

값을 그대로 사용하여 절대 시차(

)를 계산할 수 있다.Referring to [Equation 4], the relative parallax (

) wb used for calculating the relative parallax of the stereo image and

Using the value as it is, the absolute parallax (

) can be calculated.

Thereafter, the hand joint 3D position determining unit 140 may obtain an absolute joint position, that is, a hand joint 3D position, using the generated absolute 3D joint position information and an absolute parallax value. At this time, the dimension of the absolute hand joint 3D position is three-dimensional, and may be a depth dimension added to the existing (x, y). The above-described process of obtaining the 3D position of the hand joint may use a 3D position restoration method generally used in stereo vision.

[Equation 5]

와

는 초점 거리(focal length)이고

와

는 주점(pricipal point), B는 제1 카메라(201)와 제2 카메라(202) 사이의 거리 차일 수 있다.Referring to [Equation 5], where

and

is the focal length and

and

may be a principal point, and B may be a distance difference between the first camera 201 and the second camera 202.

After going through all the processes described above, the hand joint 3D position determining unit 140 can finally determine the 3D absolute joint positions for the right and left images. At this time, since the hand joint 3D position determiner 140 obtains a total of two values, it is possible to find out one 3D hand joint 3D position by calculating an average value of the two values.

Meanwhile, a method of finding the 3D position of the hand joint may be that the 3D hand joint position determiner 140 determines the 3D position of the hand joint using the stereo deep learning model 172 . Therefore, in order to determine the 3D position of the hand joint according to an embodiment, a process of learning the stereo deep learning model 172 through machine learning based on the learning image in advance is required.

Referring back to FIG. 1 , the stereo image input unit 110 may receive a plurality of learning images. At this time, the stereo image input unit 110 may receive a pair of images for learning, each of which is captured by the first camera 201 and the second camera 202 of the same object or the same background.

The stereo attention block unit 130 may generate joint 2D position information and same-joint parallax information 603 of an image pair for learning. At this time, the stereo attention block unit 130 is not a learning step, but a joint for a pair of training images in the same way as the method for finding out the 3D positions of the hand joints for the first image 301 and the second image 302 in general. 2D location information and the same joint parallax information 603 may be generated.

The artificial intelligence learning unit 160 may learn the stereo deep learning model 172 based on the joint 2D position information and the same joint parallax information 603 of the training image pair.

The artificial intelligence learning unit 160 may calculate the first loss function, the second loss function, and the third loss function, and learn the stereo deep learning model 172 based on the three loss functions.

The artificial intelligence learning unit 160 may calculate a first loss function based on 2D joint position information of the generated training image pair and reference joint 2D position information of the training image pair.

[Equation 6]

)는 추정된 학습용 이미지쌍의 상대적인 2D 관절 위치와 미리 정해진 학습용 이미지쌍의 정답 상대 2D 관절 위치의 차이를 최소화하는데 이용되는 손실 함수임을 알 수 있다. 이때, 2D 관절 위치는 2D 픽셀 좌표일 수 있다. 여기에서

는 추정된 j번째 관절의 상대적인 위치이고,

는 그와 대응되는 j번째 관절의 상대적인 위치의 정답 값일 수 있다.

은

로 구성될 수 있다.Referring to [Equation 6], the first loss function (

) is a loss function used to minimize the difference between the relative 2D joint position of the estimated training image pair and the correct answer relative 2D joint position of the predetermined training image pair. In this case, the 2D joint positions may be 2D pixel coordinates. From here

is the relative position of the estimated j-th joint,

may be the correct answer value of the relative position of the j-th joint corresponding thereto.

silver

may consist of

The artificial intelligence learning unit 160 may calculate a second loss function based on the generated same-joint disparity information 603 of the image pair for training and the reference joint disparity information 603 of the image pair for training.

[Equation 7]

)는 추정된 상대 시차와 정답 상대 시차의 차이를 최소화하는데 이용되는 손실 함수임을 알 수 있다.

은 j번째 상대 관절의 시차이고,

은 그에 대응되는 정답 시차일 수 있다. Referring to [Equation 7], the second loss function (

) is a loss function used to minimize the difference between the estimated relative lag and the correct answer relative lag.

is the parallax of the j-th relative joint,

may be a corresponding answer lag.

The artificial intelligence learning unit 160 may calculate a third loss function based on 2D joint position information of the generated training image pair and projected position information generated by projecting the 2D joint position of the training image pair to the opposite image.

[Equation 8]

)는 좌측 이미지에서 추정한 3D 관절 위치를 우측 이미지로 투영했을 때의 차이를 최소화는데 이용되는 손실 함수일 수 있다. 만약 제대로 3D 관절 위치의 추정을 했다면 좌측 이미지에서 추정한 관절 위치를 우측 이미지로 투영했을 때 차이는 0이 되어야 한다. 여기에서

와

은 투영 행렬로서 각각 좌측 이미지에서 우측으로 투영을 하고, 우측 이미지에서 좌측으로 투영을 하는데 이용되는 행렬일 수 있다.

와

는 j번째 왼쪽, 오른쪽 절대 손 관절 위치일 수 있다.Referring to [Equation 8], the third loss function (

) may be a loss function used to minimize a difference when the 3D joint position estimated from the left image is projected onto the right image. If the 3D joint position was correctly estimated, the difference should be zero when the joint position estimated from the left image is projected onto the right image. From here

and

is a projection matrix, and may be a matrix used for projection from the left image to the right and projection from the right image to the left, respectively.

and

may be the jth left and right absolute hand joint positions.

The artificial intelligence learning unit 160 may calculate an integrated loss function based on the first loss function, the second loss function, and the third loss function.

[Equation 9]

)는 임의의 값(α,β 또는 γ)이 각각 곱해진 제1 손실 함수, 제2 손실 함수 및 제3 손실 함수를 합해서 구해질 수 있다.Referring to [Equation 9], the integrated loss function (

) can be obtained by summing the first loss function, the second loss function, and the third loss function multiplied by an arbitrary value (α, β or γ), respectively.

The artificial intelligence learning unit 160 may learn the stereo deep learning model 172 so that the integrated loss function decreases through repetitive machine learning. The pretrained stereo deep learning model 172 may be stored in the memory 170 .

Machine learning may mean using a model composed of multiple parameters and optimizing the parameters with given data. Machine learning may include supervised learning, unsupervised learning, and reinforcement learning depending on the form of a learning problem. Supervised learning is to learn the mapping between inputs and outputs, and can be applied when input and output pairs are given as data. Unsupervised learning is applied when there are only inputs and no outputs, and regularities between inputs can be found. However, machine learning according to an embodiment is not necessarily limited to the aforementioned learning method.

The machine learning unit may learn the stereo deep learning model 172 in various ways. For example, the machine learning unit may learn features extracted from a plurality of training images using a deep learning-based learning method. At this time, a CNN (Convolutional Neural Networks) structure in which several convolution layers are stacked can be used to learn how to extract features related to the 3D hand joint position from the image, but the machine learning unit's learning method It is not necessarily limited to a method using a CNN structure.

Meanwhile, in order to perform machine learning in the above-described manner, it is necessary to pre-determine the correct answer value of the relative position of the joint and the correct answer parallax for each image pair for learning.

6 is a diagram for explaining a process of generating learning data according to an exemplary embodiment.

Referring to FIG. 6 , when an image for learning is input, 3D hand joint position candidate data may be first generated. In order to generate 3D hand joint position candidate data, an operation of estimating 2D hand joint positions of the left image and the right image and then converting them into 3D may be performed. For 2D hand joint position estimation, a method of generating multiple candidate groups while controlling light can be used. In this case, the number of candidate groups may be arbitrarily determined.

The joint position estimating system 100 may further include an image generating unit for change learning and a data generating unit for learning.

The alteration learning image generation unit may generate a plurality of alteration learning images by changing the brightness of the input training image. That is, a plurality of changed learning images that differ only in brightness may be generated for any one training image pair.

The learning data generator may generate joint position information for each change learning image.

The training data generation unit may estimate an optimal joint position based on joint position information of a plurality of modified learning images. The training data generator may calculate the difference between the joint position information of each change image and the joint position information of the reference change image as shown in [Equation 10] in order to estimate the optimal joint position. In this case, a change image serving as a reference may be arbitrarily selected from among a plurality of change images.

[Equation 10]

[Equation 11]

)을 결정할 수 있다.The learning data generator determines the optimal joint positions of any one training image pair through an optimization algorithm as shown in [Equation 11] based on the calculated joint position information.

) can be determined.

That is, the learning data generator compares the joint position information of a plurality of change learning images and reduces the difference between the joint position information of the plurality of change learning images. Time difference information 603 can be calculated.

7 is a flowchart of a joint position estimation method according to an embodiment. This is only a preferred embodiment for achieving the object of the present invention, and it goes without saying that some components may be added or deleted as needed.

Referring to FIG. 7 , the stereo image input unit 110 may receive a first image 301 obtained by a first camera 201 and a second image 302 obtained by a second camera 202. Yes (1001).

The image feature extractor 120 extracts a first feature 401 from the first image 301 and a second feature 402 from the second image 302 using the deep learning model 171. can (1002).

The heat map block unit 150 outputs a first joint 2D position map 501 based on the first feature 401 and outputs a second joint 2D position map 502 based on the second feature 402. can (1003).

The stereo attention block unit 130 may generate a combined feature 403 based on the first feature 401 and the second feature 402, and generate a mask map 503 based on the combined feature 403. Yes (1004).

The stereo attention block unit 130 generates first joint 2D position information 601 based on the mask map 503 and the first joint 2D position map 501, and the mask map 503 and the second joint 2D position Second joint 2D position information 602 may be generated based on the map 502 (1005).

The stereo attention block unit 130 may generate the same joint parallax information 603 based on the coupling feature 403 (1006).

The hand joint 3D position determination unit 140 may determine the hand joint 3D position based on the first joint 2D position information 601, the second joint 2D position information 602, and the same joint parallax information 603 (1007 ).

In order to verify the performance of the joint position estimation method according to an embodiment of the present invention, an experiment was conducted based on images acquired using a pair of stereo cameras.

8 is a graph showing the degree of improvement of the joint position estimation method according to an embodiment compared to the conventional joint position estimation method.

Referring to FIG. 8 , it can be seen that the joint position estimation method (StereoNet, StereoDMap) according to an embodiment generates fewer errors than other conventional methods (ResNet, AttentionNet, baseline, etc.).

Specifically, the x-axis value (error threshold) of each graph shown represents the difference between the 3D hand joint position of the actual correct answer and the estimated 3D hand joint position. In addition, the y-axis represents the frequency of difference between each 3D hand joint position, that is, the frequency of occurrence for each error degree. For example, according to the joint position estimation method (StereoNet), the frequency of occurrence of errors of 10 mm or less is about 30%, but in the conventional method, it can be confirmed that the frequency of occurrence of errors of 10 mm or less is lower than 15%.

In summary, it can be confirmed that the joint position estimation method (StereoNet, StereoDMap) according to an embodiment generates a smaller difference between the correct answer and the estimated 3D hand joint position than the conventional method.

As above, the disclosed embodiments have been described with reference to the accompanying drawings. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a form different from the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

100: joint position estimation system
110: stereo image input unit
120: image feature extraction unit
130: stereo attention block unit
140: hand joint 3D positioning unit
150: heat map block part
160: artificial intelligence learning unit
170: memory
171: deep learning model
172: stereo deep learning model
200: wearable device
201: first camera
202: second camera
301: first image
302: second image
401 First characteristic
402 Second feature
403: Combination feature
501: first joint 2D position map
502: second joint 2D position map
503: mask map
601: first joint 2D position information
602: second joint 2D position information
603: co-joint parallax information

Claims (19)

(a) receiving a first image obtained by a first camera and a second image obtained by a second camera by a stereo image input unit;
(b) extracting a first feature from a first image and a second feature from a second image using a deep learning model by an image feature extractor;
(c) generating first joint 2D position information and second joint 2D position information based on the first feature and the second feature by the stereo attention block unit; and
(d) determining the 3D position of the hand joint using a stereo deep learning model based on the 2D position information of the first joint and the 2D position information of the second joint by the 3D hand joint position determiner; A method for estimating joint position. According to claim 1,
Outputting a first joint 2D position map based on the first feature and outputting a second joint 2D position map based on the second feature by a heat map block unit based on stereo camera input Joint position estimation method. According to claim 2,
The step (c) is:
generating a combined feature by combining the first feature and the second feature by the stereo attention block unit; and
and generating, by the stereo attention block unit, a mask map in which components have values of 0 or more and 1 or less based on the combined feature. According to claim 3,
In step (c),
The stereo attention block unit generates the first joint 2D position information based on the mask map and the first joint 2D position map, and the second joint 2D position information based on the mask map and the second joint 2D position map. A method for estimating joint positions based on stereo camera input, comprising generating joint 2D position information. According to claim 4,
In step (c),
Generating, by the stereo attention block unit, same-joint parallax information based on the combining feature;
In step (d),
determining the 3D position of the hand joint based on the 2D position information of the first joint, the 2D position information of the second joint, and the parallax information of the same joint by the 3D hand joint position determination unit; A method for estimating joint position. According to claim 5,
The step of generating the same-joint parallax information,
generating the same joint parallax information based on coordinates in the first image and coordinates in the second image of the same object in the first image and the second image, by the stereo attention block unit; A joint position estimation method based on a stereo camera input including: According to claim 6,
receiving a plurality of image pairs for learning by the stereo image input unit;
generating joint 2D position information and same-joint parallax information of the training image pair by the stereo attention block unit; and
A method of estimating joint positions based on stereo camera input, further comprising the step of learning the stereo deep learning model based on the joint 2D position information and the same joint parallax information of the training image pair by an artificial intelligence learning unit. According to claim 7,
The step of learning the stereo deep learning model is:
calculating, by the artificial intelligence learning unit, a first loss function based on 2D joint position information of the training image pair and reference joint 2D position information of the training image pair;
calculating, by the artificial intelligence learning unit, a second loss function based on the same-joint disparity information of the training image pair and the reference same-joint disparity information of the training image pair;
Calculating, by the artificial intelligence learning unit, a third loss function based on 2D joint position information of the training image pair and projected position information generated by projecting the 2D joint position of the training image pair to an opposite image generated by the artificial intelligence learning unit. ; and
By the artificial intelligence learning unit, calculating an integration loss function based on the first loss function, the second loss function, and the third loss function, and learning the stereo deep learning model so that the integration loss function decreases A joint position estimation method based on a stereo camera input comprising steps. According to claim 8,
generating a plurality of changed learning images by changing the brightness of the input learning image by a change learning image generation unit; and
The learning data generation unit generates joint position information for each of the changed learning images, and compares the joint position information of the plurality of changed learning images while reducing the difference between the joint position information of the plurality of changed learning images. A method for estimating joint positions based on stereo camera input, further comprising calculating joint 2D position information and reference same-joint parallax information. A computer program stored in a computer-readable recording medium to execute the joint position estimation method based on any one of claims 1 to 9 of a stereo camera. a stereo image input unit configured to receive a first image acquired by a first camera and a second image acquired by a second camera;
an image feature extraction unit configured to extract a first feature from a first image and a second feature from a second image by using a deep learning model;
a stereo attention block unit configured to generate first joint 2D position information and second joint 2D position information based on the first and second characteristics; and
A joint position estimation system based on a stereo camera input including a hand joint 3D position determiner configured to determine a hand joint 3D position using a stereo deep learning model based on the first joint 2D position information and the second joint 2D position information. According to claim 11,
A heat map block unit configured to output a first joint 2D position map based on the first feature and a second joint 2D position map based on the second feature; joint position estimation based on stereo camera input system. According to claim 12,
The stereo attention block unit:
combine the first feature and the second feature to create a combined feature; and
A joint position estimation system based on stereo camera input, configured to generate a mask map in which components have values of 0 or more and 1 or less based on the combined feature. According to claim 13,
The stereo attention block unit:
generating the first joint 2D position information based on the mask map and the first joint 2D position map; and
A joint position estimation system based on a stereo camera input configured to generate the second joint 2D position information based on the mask map and the second joint 2D position map. According to claim 14,
The stereo attention block unit is configured to generate same-joint parallax information based on the combining feature;
The hand joint 3D position determining unit,
A joint position estimation system based on stereo camera input configured to determine the 3D position of the hand joint based on the first joint 2D position information, the second joint 2D position information, and the same joint parallax information. According to claim 15,
The stereo attention block unit,
Joint position estimation based on a stereo camera input configured to generate the same-joint parallax information based on coordinates in the first image and coordinates in the second image for the same object in the first image and the second image system. According to claim 16,
The stereo image input unit,
It is configured to receive a plurality of image pairs for learning,
The stereo attention block unit,
configured to generate joint 2D position information and same-joint parallax information of the pair of images for learning;
Stereo camera input-based joint position estimation system further comprising an artificial intelligence learning unit configured to learn the stereo deep learning model based on the joint 2D position information and the same joint parallax information of the training image pair. According to claim 17,
The artificial intelligence learning unit:
calculating a first loss function based on the generated 2D joint position information of the training image pair and reference joint 2D position information of the training image pair;
calculating a second loss function based on the generated same-joint disparity information of the training image pair and the reference joint disparity information of the training image pair;
Calculate a third loss function based on the generated 2D joint position information of the training image pair and projected position information generated by projecting the 2D joint position of the training image pair to an opposite image; and
Based on a stereo camera input, configured to calculate an integrated loss function based on the first loss function, the second loss function, and the third loss function, and learn the stereo deep learning model so that the integrated loss function decreases. Joint localization system. According to claim 18,
a change learning image generator configured to generate a plurality of change learning images by changing the brightness of the input learning image; and
The reference joint 2D position information and the reference are the same by generating joint position information for each of the change learning images and reducing the difference between the joint position information of the plurality of change learning images while comparing the joint position information of the plurality of change learning images. A system for estimating joint position based on stereo camera input, further comprising a learning data generator configured to calculate joint parallax information.

Images