KR102181828B1 - 4d rig reconstructing device and a method thereof - Google Patents

Abstract

Embodiments of the present invention include an image acquisition unit that receives an image sequence photographing one or more objects from a monocular camera, and 2D rigging that detects an object for each frame of the image sequence and estimates 2D rigging information based on the 2D joint position of the object. Information estimation unit, 3D rigging information conversion unit that converts 2D rigging information into 3D rigging information, 2D rigging information conversion unit that converts 3D rigging information into 2D rigging information, and 2D rigging information estimated from subsequent frames It provides a 4D rigging information restoration apparatus including a data linking unit that complements 2D rigging information by comparing 2D rigging information.
Accordingly, embodiments of the present invention can accurately restore the rigging information of an object from an image captured by a monocular camera.

Description

4D rigging information restoration device and method {4D RIG RECONSTRUCTING DEVICE AND A METHOD THEREOF}

Embodiments of the present invention relate to an apparatus and method for restoring 4D rigging information, and more particularly, estimating 2D rigging information of one or more objects from an image sequence photographed from a monocular camera, and using the estimated 2D rigging information as 3D rigging information. It relates to a 4D rigging information restoration apparatus and method for supplementing newly estimated 2D rigging information by converting the converted 3D rigging information into 2D rigging information after conversion.

Rigging may mean attaching a skeleton to data modeled in 2D or 3D, and rigging information may mean joint position information, joint distance information, or skeleton information.

Here, the joint position information may refer to the coordinates at which the joint is located for each joint, and the inter-joint distance information may refer to a distance from another joint for each joint, and the skeletal information refers to the length or length of each bone connecting adjacent joints. It can mean an angle, etc.

Reconstruction of rigging information may mean calculating rigging information of an object that changes over time from an image sequence.

On the other hand, since the posture of the object can be known through the rigging information, the rigging information can be used in the same meaning as the posture information, and the posture change of the object can be known through the restoration of the rigging information. It can be used with the same meaning as.

Rigging information restoration is used in the field of motion recognition technology through image analysis, and in this field, it is mainly used for rehabilitation treatment, tangible game, or sports motion practice.

At this time, most of the prior art restores rigging information by using depth information on the object along with the image captured by the object. In order to restore the rigging information by measuring depth information, a multi-view image or a depth measurement sensor is required. .

However, in order to obtain a multi-view image, a plurality of cameras arranged so that objects can be photographed from various angles are required, which requires space for camera placement and increases cost. In the case of using a depth sensor, the depth measurement distance or measurement There is a problem in that the range is limited and the depth measurement error increases due to noise caused by lighting or the like.

Meanwhile, depth information may be measured through a stereo image rather than a multi-view image. In this case, since a binocular (stereo) camera is required, there is a problem that a general monocular (mono) camera cannot be used.

In this regard, the prior art is as follows.

Korean Patent Registration No. 10-1784410 relates to a method for recognizing an exercise posture, and in more detail, the step of obtaining location information of a joint part of a user's body in a specific exercise posture presented, and a vector in a three-dimensional space using the location information. It relates to an exercise posture recognition method comprising the step of obtaining, calculating an angle of a body joint part from the vector, and calculating a match rate of the actual posture of the user with respect to the specific exercise posture using the angle.

However, since the prior art uses a depth measurement sensor (IR projector) to calculate a three-dimensional posture, it cannot propose a method of restoring rigging information from an image sequence captured by a monocular camera without depth information.

In order to solve the above-described problem, embodiments of the present invention provide an image acquisition unit that receives an image sequence photographing one or more objects from a monocular camera, and detects an object for each frame of the image sequence, and based on the 2D joint position of the object. From the 2D rigging information estimation unit that estimates 2D rigging information, the 3D rigging information conversion unit that converts 2D rigging information to 3D rigging information, the 2D rigging information conversion unit that converts the converted 3D rigging information into 2D rigging information, and the subsequent frames. An object of the present invention is to provide a 4D rigging information restoration apparatus including a data linking unit that complements the 2D rigging information by comparing the estimated 2D rigging information and the converted 2D rigging information.

In addition, embodiments of the present invention include a 3D rigging information prediction unit that predicts 3D rigging information after a specific time elapses from the converted 3D rigging information, and the 2D rigging information conversion unit converts the predicted 3D rigging information into 2D rigging information. Its purpose is to provide a device for restoring rigging information.

In addition, an object of the present invention is to provide a 4D rigging information restoration apparatus in which a data association unit compares 2D rigging information estimated from a subsequent frame with converted 2D rigging information to supplement 2D rigging information for each object.

In addition, in the embodiments of the present invention, the 2D rigging information includes 2D joint distance information or 2D joint position information, and the data linking unit includes 2D rigging information estimated from a subsequent frame or an image patch around the 2D joint position of a subsequent frame. An object of the present invention is to provide a 4D rigging information restoration apparatus that supplements 2D rigging information for each object by comparing the converted 2D rigging information or the image patch around the 2D joint position related to the converted 2D rigging information.

In addition, an object of the present invention is to provide a 4D rigging information restoration apparatus in which a data association unit supplements 2D rigging information using a Kalman filter or LSTM.

In addition, embodiments of the present invention are 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information, or predicts 3D rigging information from the converted 3D rigging information Its purpose is to provide a 4D rigging information restoration apparatus using a deep learning model to convert rigging information into 2D rigging information.

In addition, in embodiments of the present invention, a deep learning model is learned using 2D joint distance information or 3D joint distance information, and a 3D rigging information conversion unit, a 3D rigging information prediction unit, or a 2D rigging information conversion unit is 2D rigging information or 3D rigging information. An object thereof is to provide a 4D rigging information restoration apparatus that encodes information into 2D joint distance information or 3D joint distance information and inputs it into a learned deep learning model.

In addition, embodiments of the present invention are 2D rigging information using a 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit encoded 2D joint distance information or 3D joint distance information using a multidimensional scaling method (MDS). Another object is to provide a 4D rigging information recovery apparatus that decodes 3D rigging information.

Further, an object of the present invention is to provide an apparatus for restoring 4D rigging information including a deep learning model unit for learning a deep learning model.

In addition, embodiments of the present invention include an image acquisition step of receiving an image sequence photographing one or more objects from a monocular camera, detecting an object for each frame of the image sequence, and estimating 2D rigging information based on the 2D joint position of the object. 2D rigging information estimation step, 3D rigging information conversion step for converting 2D rigging information to 3D rigging information, 2D rigging information conversion step for converting the converted 3D rigging information to 2D rigging information, and 2D rigging information estimated from subsequent frames. The purpose of this is to provide a method for restoring 4D rigging information including a data association step of compensating 2D rigging information by comparing the converted 2D rigging information.

An embodiment of the present invention for achieving the above object is an image acquisition unit for receiving an image sequence of one or more objects from a monocular camera; A 2D rigging information estimation unit for detecting an object for each frame of the image sequence and estimating 2D rigging information based on the 2D joint position of the object; A 3D rigging information conversion unit converting the 2D rigging information into 3D rigging information; A 2D rigging information conversion unit converting the converted 3D rigging information into 2D rigging information; And it provides a 4D rigging information restoration apparatus including a data linking unit for compensating the 2D rigging information by comparing the 2D rigging information estimated from a subsequent frame and the converted 2D rigging information.

In one embodiment, the 4D rigging information restoration apparatus includes a 3D rigging information prediction unit for predicting 3D rigging information after a specific time elapses from the converted 3D rigging information, and the 2D rigging information conversion unit Can be converted into 2D rigging information.

In an embodiment, the data linking unit may supplement the 2D rigging information for each object by comparing the 2D rigging information estimated from a subsequent frame with the converted 2D rigging information.

In one embodiment, the 2D rigging information includes 2D joint distance information or 2D joint position information, and the data linking unit converts 2D rigging information estimated from a later frame or an image patch around the 2D joint position of a later frame. The 2D rigging information may be supplemented for each object by comparing the image patch around the 2D joint position related to the converted 2D rigging information or the converted 2D rigging information.

In one embodiment, the data linking unit may supplement the 2D rigging information using a Kalman filter or an LSTM.

In one embodiment, the 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information, or predicts 3D rigging information from the converted 3D rigging information or 3D rigging information. A deep learning model can be used to convert the data into 2D rigging information.

In one embodiment, the deep learning model is learned using 2D joint distance information or 3D joint distance information, and the 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit is 2D rigging information or 3D The rigging information may be encoded into 2D joint distance information or 3D joint distance information and input into the learned deep learning model.

In one embodiment, the 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit converts the encoded 2D joint distance information or 3D joint distance information into 2D rigging information or 3D using a multidimensional scaling method (MDS). It can be decoded with rigging information.

In one embodiment, the 4D rigging information restoration apparatus may include a deep learning model unit that trains the deep learning model.

In addition, another embodiment of the present invention for achieving the above object is an image acquisition step of receiving an image sequence photographing one or more objects from a monocular camera; A 2D rigging information estimation step of detecting an object for each frame of the image sequence and estimating 2D rigging information based on the 2D joint position of the object; 3D rigging information conversion step of converting the 2D rigging information into 3D rigging information; 2D rigging information conversion step of converting the converted 3D rigging information into 2D rigging information; And a data association step of compensating the 2D rigging information by comparing the 2D rigging information estimated from a subsequent frame with the converted 2D rigging information.

As described above, embodiments of the present invention include an image acquisition unit that receives an image sequence photographing one or more objects from a monocular camera, detects an object for each frame of the image sequence, and provides 2D rigging information based on the 2D joint position of the object. Estimated 2D rigging information estimation unit, 3D rigging information conversion unit that converts 2D rigging information into 3D rigging information, 2D rigging information conversion unit that converts 3D rigging information into 2D rigging information, and 2D rigging estimated from subsequent frames By providing a 4D rigging information restoration apparatus including a data linking unit that complements the 2D rigging information by comparing the information and the converted 2D rigging information, it is possible to accurately restore the rigging information of the object from the image captured by the monocular camera.

In addition, embodiments of the present invention include a 3D rigging information prediction unit that predicts 3D rigging information after a specific time elapses from the converted 3D rigging information, and the 2D rigging information conversion unit converts the predicted 3D rigging information into 2D rigging information. By providing the rigging information restoration device, it is possible to accurately restore the rigging information of the object.

In addition, embodiments of the present invention provide a 4D rigging information restoration apparatus that complements 2D rigging information for each object by comparing the 2D rigging information estimated from the subsequent frame with the converted 2D rigging information by the data association unit. Can be accurately restored.

In addition, in the embodiments of the present invention, the 2D rigging information includes 2D joint distance information or 2D joint position information, and the data linking unit includes 2D rigging information estimated from a subsequent frame or an image patch around the 2D joint position of a subsequent frame. By providing a 4D rigging information restoration device that complements the 2D rigging information for each object by comparing the converted 2D rigging information or the image patch around the 2D joint position related to the converted 2D rigging information, it is possible to accurately restore the rigging information for each object. .

In addition, embodiments of the present invention provide a 4D rigging information restoration device in which the data linking unit complements 2D rigging information using a Kalman filter or LSTM, so that the rigging information of an object can be accurately restored from an image captured by a monocular camera. have.

In addition, embodiments of the present invention are 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information, or predicts 3D rigging information from the converted 3D rigging information By providing a 4D rigging information restoration apparatus using a deep learning model to convert rigging information into 2D rigging information, rigging information of an object can be accurately restored from an image captured by a monocular camera.

In addition, in embodiments of the present invention, a deep learning model is learned using 2D joint distance information or 3D joint distance information, and a 3D rigging information conversion unit, a 3D rigging information prediction unit, or a 2D rigging information conversion unit is 2D rigging information or 3D rigging information. By providing a 4D rigging information restoration device that encodes the information into 2D joint distance information or 3D joint distance information and inputs it into the learned deep learning model, deep learning learning can be efficiently performed and objects can be retrieved from images captured by a monocular camera. Rigging information can be accurately restored.

In addition, embodiments of the present invention are 2D rigging information using a 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit encoded 2D joint distance information or 3D joint distance information using a multidimensional scaling method (MDS). Alternatively, by providing a 4D rigging information restoration device that decodes into 3D rigging information, deep learning learning can be efficiently performed, and rigging information of an object can be accurately restored from an image captured by a monocular camera.

In addition, embodiments of the present invention provide a 4D rigging information restoration apparatus including a deep learning model unit for learning a deep learning model, so that rigging information of an object can be accurately restored from an image captured by a monocular camera.

In addition, embodiments of the present invention include an image acquisition step of receiving an image sequence photographing one or more objects from a monocular camera, detecting an object for each frame of the image sequence, and estimating 2D rigging information based on the 2D joint position of the object. 2D rigging information estimation step, 3D rigging information conversion step for converting 2D rigging information to 3D rigging information, 2D rigging information conversion step for converting the converted 3D rigging information to 2D rigging information, and 2D rigging information estimated from subsequent frames. By providing a 4D rigging information restoration method including a data association step of compensating the 2D rigging information by comparing the converted 2D rigging information, it is possible to accurately restore the rigging information of the object from the image captured by the monocular camera.

It is not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art from the following description.

1 is a diagram showing a 4D rigging information restoration system according to an embodiment of the present invention.
2 is a block diagram showing a 4D rigging information restoration apparatus according to an embodiment of the present invention.
3 is a view showing a joint and skeleton according to an embodiment of the present invention.
4 is a view showing 2D joint position information according to an embodiment of the present invention.
5 is a view showing distance information between joints according to an embodiment of the present invention.
6 is a diagram showing a deep learning model according to an embodiment of the present invention.
7 and 8 are diagrams showing a 4D rigging information restoration apparatus according to an embodiment of the present invention.
9 is a flowchart illustrating a method of restoring 4D rigging information according to an embodiment of the present invention.

Since the description of the present invention is merely examples for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the present embodiments can be variously changed and have various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only those effects, the scope of the present invention should not be understood as being limited thereto.

In addition, the accompanying drawings are provided to aid in the understanding of the present invention, and provide embodiments together with a detailed description. However, the technical features of the present invention are not limited to a specific drawing, and features disclosed in each drawing may be combined with each other to constitute a new embodiment.

The apparatus and method for restoring 4D rigging information disclosed in the following embodiments will be described in more detail with reference to each drawing.

1 is a diagram showing a 4D rigging information restoration system according to an embodiment of the present invention.

Referring to FIG. 1, a 4D rigging information restoration system 100 according to an embodiment may include an object 200, a camera 300, and a 4D rigging information restoration apparatus 400.

The object 200 is photographed by the camera 300 and can move while changing a posture, and may be a sports player. In addition, the number of objects 200 may be plural.

The camera 300 may be a monocular (mono) camera and may be connected to the 4D rigging information restoration device 400 through a communication network. The camera 300 may transmit an image sequence photographing one or more objects 200 to the 4D rigging information restoration apparatus 400 through a communication network.

Here, the communication network may mean a network of a wide concept including a wired or wireless communication network.

Also, the camera 300 may be a PTZ camera capable of photographing while tracking the movement of the object 200.

Here, the PTZ camera may mean a camera capable of adjusting a rotation (PAN), a vertical tilt (Tilt), and a zoom (Zoom).

The 4D rigging information restoration device 400 is connected to the camera 300 through a communication network and is connected to a server, PC, notebook, mobile phone, smart phone (2G/3G/4G/LET, smart phone), PMP (Portable Media Player), It may be any one of PDA (Personal Digital Assistant) and Tablet PC (Tablet PC). A detailed configuration of the 4D rigging information restoration device 400 will be described with reference to FIG. 2.

2 is a block diagram showing a 4D rigging information restoration apparatus according to an embodiment of the present invention.

2, a 4D rigging information restoration apparatus 400 according to an embodiment includes an image acquisition unit 410, a 2D rigging information estimation unit 420, a 3D rigging information conversion unit 430, and a 3D rigging information prediction unit. 440, 2D rigging information conversion unit 450, data association unit 460, deep learning model unit 470, transmission and reception unit 480, database 490, and may be configured to include a control unit 495. .

The image acquisition unit 410 receives the image sequence captured by the monocular (mono) camera 300 from the camera 300 through the transmission/reception unit 480 or inputs the image sequence from the database 490 in which the image sequence is stored. I can receive it.

Here, the image sequence may refer to a series of frames, and the frame may refer to one of a plurality of images (frames) constituting a moving picture (series of frames). The image sequence may include a current frame and a subsequent frame.

The 2D rigging information estimation unit 420 receives the image sequence from the image acquisition unit 410, detects the object 200 for each frame of the image sequence, and calculates 2D rigging information based on the 2D joint position of the object 200. Can be estimated.

Here, detecting the object 200 for each frame means that the 2D rigging information estimation unit 420 detects the object 200 for all frames transmitted from the image acquisition unit 410 or the database 490 It does not mean that the object 200 is detected for each frame of a part of the transmitted image sequence.

In addition, here, the 2D joint position may mean two-dimensional position coordinates for each type of joint. For example, it may mean a two-dimensional position coordinate of a left knee joint or a two-dimensional position coordinate of a right elbow joint.

In addition, here, rigging may mean attaching a skeleton to data modeled in 2D or 3D.

In addition, here, the rigging information may mean joint position information, joint distance information, or skeleton information, 2D rigging information may mean rigging information in two dimensions, and 3D rigging information may mean rigging information in three dimensions. .

Here, the joint position information may refer to the coordinates at which the joint is located for each type of joint, and the inter-joint distance information may refer to the distance from other joints for each joint, and the bone information refers to the length of each bone connecting adjacent joints. Or angle.

However, since inter-joint distance information or skeleton information may be generated based on joint position information, the rigging information may be basically estimated based on joint position information regardless of its meaning.

3 is a view showing a joint and skeleton according to an embodiment of the present invention.

3 is a diagram illustrating an example of a joint and a skeleton according to an embodiment.

Referring to FIG. 3, the object 200 may include joints such as shoulder joints J1 and J4, elbow joints J2 and J5, knee joints J8 and J11, and ankle joints J9 and J12. By connecting adjacent joints such as knee joints (J8, J11) and ankle joints (J9, J12), you can obtain a skeleton such as the lower leg bones (B1, B2). In other words, it is possible to obtain the skeleton from the joint.

4 is a view showing 2D joint position information according to an embodiment of the present invention.

Referring to FIG. 4, 2D joint position information may correspond to two-dimensional position coordinates (X, Y) for each type of joints J1 to J14. Here, X and Y may be coordinate values of the X-axis and Y-axis of a pixel in which each joint is located in one frame.

As described above, since the rigging information may refer to joint position information, the 2D rigging information may refer to the 2D joint position information of FIG. 4, and the 3D rigging information is joints (J1 to J1 to which the coordinates of the Z-axis are added in FIG. J14) may mean three-dimensional position coordinates (coordinates of the X-axis, Y-axis, and Z-axis) for each type.

Meanwhile, the rigging information may mean skeleton information indicating the length or angle of each bone, and the skeleton information can be easily calculated based on joint position information of adjacent joints as shown in FIG. 3.

That is, the length of a straight line connecting two adjacent joints may correspond to the length of the bone, and the slope of the straight line may correspond to the angle of the bone.

5 is a view showing distance information between joints according to an embodiment of the present invention.

Referring to FIG. 5, the inter-joint distance information may correspond to distances from each joint (J1 to J14) to all other joints (J1 to J14), and the distance between two joints may be calculated based on two joint position information. . Here, the distance may be an Euclidean distance.

As described above, the rigging information may mean the inter-joint distance information of FIG. 5 and the format of the inter-joint distance information of FIG. 5 regardless of whether the rigging information is 2D rigging information or 3D rigging information (ie, regardless of the dimension). (n rows and n columns, n is the number of joints).

However, the values of the inter-joint distance information in 2D are calculated based on the 2D joint position information, and the values of the inter-joint distance information in 3D are calculated based on the 3D joint position information. It may be less than or equal to values of the inter-joint distance information in 3D.

Returning to FIG. 2 again, the 2D rigging information estimation unit 420 uses, for example, a method of separately detecting a pedestrian's head, arms, legs, upper body, etc. among the pedestrian detection methods. After detecting the part, the connection point between the body parts can be estimated as a 2D joint position.

However, the 2D rigging information estimation unit 420 is not limited to the above method and may use various other methods to estimate the 2D joint position.

In addition, the 2D rigging information estimation unit 420 may use the 2D joint position estimated by the above method as 2D rigging information when the rigging information is joint position information, and when the rigging information is skeleton information or inter-joint distance information, the Based on the 2D joint position estimated by the method, skeleton information or inter-joint distance information can be calculated and used as 2D rigging information.

The 3D rigging information conversion unit 430 may receive 2D rigging information estimated by the 2D rigging information estimating unit 420 or supplemented by the data linking unit 460 to be described later, and convert it into 3D rigging information.

The 3D rigging information prediction unit 440 may predict 3D rigging information after a specific time has elapsed from the 3D rigging information converted by the 3D rigging information conversion unit 430. Here, the specific time may mean a difference between a current frame capture time and a frame capture time after the rigging information is restored.

The 2D rigging information converting unit 450 may receive 3D rigging information converted by the 3D rigging information converting unit 430 or predicted by the 3D rigging information predicting unit 440 and convert it into 2D rigging information.

At this time, loss may occur in information related to depth (Z axis) among the rigging information. The lost depth information may be determined by the photographing direction of the camera, and the front direction of the camera may correspond to the depth (Z axis).

The 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440 or the 2D rigging information conversion unit 450 converts 2D rigging information into 3D rigging information, or predicts 3D rigging information from the converted 3D rigging information. Deep learning models can be used to convert rigging information into 2D rigging information.

Regarding the deep learning model, it will be described in detail in the deep learning model unit 470 to be described later.

In this way, the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, and the 2D rigging information conversion unit 450 convert 2D rigging information into 3D rigging information and predict 3D rigging information after a specific time. By converting the converted or predicted 3D rigging information into 2D rigging information, the accuracy of the rigging information from a subsequent frame can be improved.

The data linking unit 460 is connected to the 2D rigging information estimating unit 420 and the 2D rigging information converting unit 450, and is estimated from the subsequent frames from the 2D rigging information estimating unit 420 and the 2D rigging information converting unit 450. The 2D rigging information can be supplemented by receiving the converted 2D rigging information and the converted 2D rigging information and comparing the two 2D rigging information.

Here, the 2D rigging information estimated from a subsequent frame means that the 2D rigging information estimation unit 420 detects the object 200 from the subsequent frame and estimates the 2D rigging information based on the 2D joint position of the object 200. It can mean.

In addition, here, the converted 2D rigging information refers to the 2D rigging information estimated from the current frame by the 2D rigging information estimating unit 420, the data linking unit 460, the 3D rigging information converting unit 430, and the 2D rigging information converting unit. It may mean 2D rigging information converted through 450. At this time, the 3D rigging information prediction unit 440 may be further passed between the 3D rigging information conversion unit 430 and the 2D rigging information conversion unit 450.

In addition, here, the 2D rigging information supplemented by the data linking unit 460 may be 2D rigging information estimated from a later frame or converted 2D rigging information. In either case, the 2D rigging information supplemented by the data linking unit 460 It is common in that the 2D rigging information can be transferred to the 3D rigging information conversion unit 430 and converted into 3D rigging information.

In addition, the data linking unit 460 may supplement 2D rigging information using a Kalman filter or LSTM.

First, a method of supplementing 2D rigging information using a Kalman filter by the data association unit 460 will be described.

Here, the Kalman filter is an algorithm based on a series of measurements observed over time, and can calculate an accurate estimate of a state from measurements including noise.

The Kalman filter algorithm can be divided into two stages: a prediction stage and an update stage. In the prediction stage, the predicted state is calculated from the state estimated at the previous time, and in the update stage, the accurate state is determined based on the previously calculated predicted state and the actually measured state. Can be calculated.

Specifically, the data linking unit 460 uses the Kalman filter to predict future 2D rigging information based on the 2D rigging information obtained from the current frame in the prediction step, and 2D rigging information estimated from the subsequent frames in the update step. 2D rigging information can be supplemented by comparing the converted 2D rigging information.

At this time, the data linking unit 460 may use the speed of each joint point to predict 2D rigging information.

In addition, the data linking unit 460 performs transformation or prediction in the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or the 2D rigging information conversion unit 450 in the prediction step or supplementation step of the Kalman filter. It may be performed independently, or may be performed in conjunction with transformation or prediction in the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or the 2D rigging information conversion unit 450.

Second, it looks at how the data linking unit 460 supplements the 2D rigging information using LSTM.

Here, the LSTM (Long Short-Term Memory) is a unit of a cyclic neural network (RNN) and may be composed of a cell, an input gate, an output gate, and a forgetting gate. The cell stores values over arbitrary time intervals, and the three gates can control the flow of information into and out of the cell.

Specifically, the data linking unit 460 can predict the 2D rigging information expected later based on the 2D rigging information obtained from the current frame using a deep learning (RNN) model with LSTM as a constituent unit. 2D rigging information can be supplemented by comparing the estimated 2D rigging information with the converted 2D rigging information.

Here, the deep learning model and the RNN will be described in the deep learning model unit 470.

Meanwhile, the data linking unit 460 may use a Kalman filter and an LSTM. Specifically, in the prediction step, the data linking unit 460 uses a deep learning model with LSTM as a constituent unit to obtain 2D rigging information from the current frame. It is possible to predict the 2D rigging information expected as a basis and the variables to be used in the Kalman filter, and in the update step, the 2D rigging information can be supplemented by comparing the 2D rigging information estimated from the subsequent frames using the Kalman filter and the converted 2D rigging information. have.

In addition, the data linking unit 460 converts or predicts the deep learning model using LSTM as a constituent unit in the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or the 2D rigging information conversion unit 450. It may be used independently of the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or may be used in conjunction with the transformation or prediction in the 2D rigging information conversion unit 450.

In this way, the data linking unit 460 compares the 2D rigging information estimated from a subsequent frame with the converted 2D rigging information based on the temporal change of the 2D rigging information using a Kalman filter or LSTM to compare the 2D rigging information. By supplementing, the accuracy of the rigging information can be improved.

On the other hand, when a plurality of objects 200 are photographed in the image information, the rigging information must be restored by distinguishing each object 200, and a method related thereto will be described.

The data linking unit 460 may supplement the 2D rigging information for each object 200 by comparing the 2D rigging information estimated from a subsequent frame with the converted 2D rigging information.

At this time, the data linking unit 460 includes the 2D rigging information estimated from the subsequent frame or the image patch around the 2D joint position of the subsequent frame and the converted 2D rigging information or the 2D joint position related to the converted 2D rigging information. 2D rigging information for each object 200 may be supplemented by comparing image patches.

Specifically, the data linking unit 460 compares the 2D rigging information estimated from the subsequent frame with the 2D rigging information that has been distinguished by object 200 from the current frame and converted by the 2D rigging information conversion unit 450 200), 2D rigging information can be supplemented.

That is, when the 2D rigging information estimation unit 420 incorrectly distinguishes the rigging information of the two objects 200 from a subsequent frame and estimates the 2D rigging information with the joints of the two objects 200 changed, the data linking unit ( 460 may supplement 2D rigging information of a subsequent frame by comparing it with 2D rigging information that has been correctly distinguished and converted for each object 200 from the current frame.

At this time, the data linking unit 460 may compare 2D joint position information, 2D joint distance information, or image patches around 2D joints to differentiate the object 200 to supplement 2D rigging information. Here, the image patch may mean a 2D image of a specific size.

In this way, the data linking unit 460 may differentiate the object 200 and supplement the 2D rigging information, thereby increasing the accuracy of the rigging information. In particular, the data linking unit 460 can quickly and accurately supplement the rigging information by comparing the 2D joint distance information to distinguish the object 200.

The deep learning model unit 470 may be connected to the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450, or the data linking unit 460, and a deep learning model to be described later. Can provide.

The deep learning model unit 470 may generate a deep learning model and train the deep learning model using training data previously stored in the database 490. Here, deep learning is a type of machine learning method for learning data representation, and representatively, DNN, CNN, RNN, etc. may correspond to this. However, other deep learning methods such as DBN can also be used.

Here, the DNN (Deep Neural Network) is an artificial neural network made up of several hidden layers between an input layer and an output layer.

In addition, here, CNN (Convolutional Neural Network) is a kind of multilayer perceptrons designed to use minimal preprocessing, and one or several convolutional layers and general artificial It consists of neural network layers, and weights and pooling layers can be additionally used.

In addition, here, a recurrent neural network (RNN) is a neural network in which a connection between units constituting an artificial neural network constitutes a directed cycle, and the memory inside the neural network can be used to process an arbitrary input. As a structure of a recurrent neural network, various methods such as a Fully Recurrent Network and a Long Short Term Memory Network (LSTM) can be used.

The deep learning model unit 470 generates a deep learning model suitable for the functions of the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450 or the data linking unit 460. It can be provided by learning and learning, but we will look at it below.

First, it looks in relation to the 3D rigging information conversion unit 430.

The deep learning model unit 470 may generate a 3D transformed deep learning model for converting 2D rigging information into 3D rigging information. At this time, the deep learning model unit 470 can generate a 3D transformed deep learning model using 2D rigging information as an input value and 3D rigging information as an output value. The input value and the output value are relatively simple, and the current output value is Since the input/output value may not be affected, the DNN model may be preferable for the 3D transformed deep learning model. However, other methods such as RNN and CNN may also be used.

In addition, the deep learning model unit 470 may receive training data composed of various 2D rigging information and 3D rigging information corresponding to each 2D rigging information from the database 490 to train a 3D transformed deep learning model.

In addition, the deep learning model unit 470 may provide the 3D transformed deep learning model in which the training has been completed to the 3D rigging information conversion unit 430.

Second, it looks in relation to the 3D rigging information prediction unit 440.

The deep learning model unit 470 may generate a 3D predictive deep learning model for predicting 3D rigging information after a specific time elapses from the 3D rigging information. At this time, the deep learning model unit 470 can generate a 3D predictive deep learning model using 3D rigging information as an input value and 3D rigging information as an output value. The input value and the output value are relatively simple, and the current predicted output value is Since it may be affected by the previous output value, the RNN model may be preferable for the 3D prediction deep learning model. However, other methods such as DNN and CNN may also be used. In this case, the 3D prediction deep learning model may need to input additional information such as previous joint position information or joint speed in order to calculate the joint speed. .

In addition, the deep learning model unit 470 may receive training data composed of various 3D rigging information and predicted 3D rigging information corresponding to each 3D rigging information from the database 490 to train a 3D predictive deep learning model.

In addition, the deep learning model unit 470 may provide the 3D predictive deep learning model in which the learning has been completed to the 3D rigging information predictor 440.

Third, it looks in relation to the 2D rigging information conversion unit 450.

The deep learning model unit 470 may generate a 2D transformed deep learning model for converting 3D rigging information into 2D rigging information. At this time, the deep learning model unit 470 can generate a 2D transformed deep learning model using 3D rigging information as an input value and 2D rigging information as an output value. The input value and output value are relatively simple, and the current output value is Since the input/output value may not be affected, the DNN model may be preferable for the 2D transformed deep learning model. However, other methods such as RNN and CNN may also be used.

In addition, the deep learning model unit 470 may receive training data composed of various 3D rigging information and 2D rigging information corresponding to each 3D rigging information from the database 490 to train a 2D transformed deep learning model.

In addition, the deep learning model unit 470 may provide the 2D transformed deep learning model in which the training has been completed to the 2D rigging information conversion unit 450.

Fourth, it looks in relation to the data association unit 460.

The deep learning model unit 470 may generate a 2D predictive deep learning model for predicting future 2D rigging information based on the 2D rigging information obtained from the current frame. At this time, the deep learning model unit 470 can generate a 2D prediction deep learning model using 2D rigging information as an input value and 2D rigging information as an output value. The input value and the output value are relatively simple, and the predicted output value is Since it may be affected by the current output value, the RNN model may be preferable for the 2D prediction deep learning model. However, other methods such as DNN and CNN may also be used. In this case, the 2D prediction deep learning model may need to input additional information such as previous joint position information or joint speed to calculate the joint speed. .

In addition, the deep learning model unit 470 may receive training data consisting of various 2D rigging information and predicted 2D rigging information corresponding to each 2D rigging information from the database 490 to train the 2D predictive deep learning model.

In addition, the deep learning model unit 470 may provide the 2D predictive deep learning model in which the training has been completed to the data association unit 460.

In this way, the deep learning model unit 470 trains and provides a deep learning model, and the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450 or the data linking unit ( 460 can increase the accuracy of the rigging information by using the deep learning model provided by the deep learning model unit 470.

Meanwhile, the deep learning model unit 470 may use 2D joint distance information or 3D joint distance information when learning a 3D transformed deep learning model, a 3D predictive deep learning model, a 2D transformed deep learning model, or a 2D predictive deep learning model. have.

That is, the input and output values of the 3D transformed deep learning model, 3D predicted deep learning model, 2D transformed deep learning model, or 2D predicted deep learning model may correspond to 2D joint distance information or 3D joint distance information.

6 is a diagram showing a deep learning model according to an embodiment of the present invention.

6, the deep learning model can be learned by the deep learning model unit 470 using inter-joint distance information as input values and output values, and after the learning is completed, the 3D rigging information conversion unit 430, the 3D rigging information prediction It may be provided to the unit 440, the 2D rigging information conversion unit 450, or the data association unit 460.

The matrix of 14 rows and 14 columns corresponding to the input values and output values of FIG. 6 may mean the Euclidean distance between the joints for the 14 joints of J1 to J14 of FIG. 3, and has the same meaning as the inter-joint distance information of FIG. Can be

In this way, the deep learning model unit 470 learns and provides a deep learning model using the inter-joint distance information, so that the 2D and 3D rigging information are not affected by rotation and translation of the image. It can be expressed in dimensions (n rows and n columns, where n is the number of joints) and does not use 2D or 3D coordinates to reduce the solution space during deep learning learning, so deep learning learning can be performed efficiently.

On the other hand, when the deep learning model is learned using the joint distance information, the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450 or the data linking unit 460 In order to use the learned deep learning model, it may be necessary to encode 2D rigging information or 3D rigging information into 2D joint distance information or 3D joint distance information.

In addition, the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450, or the data linking unit 460 uses the encoded 2D joint distance information or 3D joint distance information. After using the deep learning model, it may be necessary to decode the encoded 2D joint distance information or 3D joint distance information into 2D rigging information or 3D rigging information, and at this time, multi-dimensional scaling can be used.

Here, the multidimensional scaling method may be used to visualize the Euclidean distance matrix as a means to visualize the degree of similarity of data. Visualization mainly uses a method of representing points in a two-dimensional or three-dimensional space.

However, when 2D rigging information or 3D rigging information is 2D joint distance information or 3D joint distance information, it can be used as it is without encoding or decoding.

The database 490 may store an image sequence received from the camera 300 and training data for learning a 3D transformed deep learning model, a 3D predictive deep learning model, a 2D transformed deep learning model or a 2D predictive deep learning model.

This database 490 is a concept including a computer-readable recording medium, and refers to not only a database of consultations, but also a database in a broad sense including data recording based on a file system, and searches for a simple set of logs. Thus, if data can be extracted, it is included within the scope of the database referred to in the present invention.

Finally, the control unit 495 is an image acquisition unit 410, 2D rigging information estimation unit 420, 3D rigging information conversion unit 430, 3D rigging information prediction unit 440, 2D rigging information conversion unit 450 , Control flow or data flow between the data linking unit 460, the deep learning model unit 470, the transmission/reception unit 480, and the database 490 may be controlled.

7 and 8 are diagrams showing a 4D rigging information restoration apparatus according to an embodiment of the present invention.

7 and 8, the 4D rigging information restoration apparatus 400 according to an embodiment includes an image acquisition unit 410, a 2D rigging information estimation unit 420, a 3D rigging information conversion unit 430, and a 3D rigging. It may include an information prediction unit 440, a 2D rigging information conversion unit 450, and a data association unit 460.

In FIG. 7, A is a later frame, B is a current frame, and the data linking unit 460 receives 2D rigging information estimated from a later frame (A), and 2D rigging information converted from the current frame (B) Compared with, 2D rigging information can be supplemented for each object.

At this time, the data linking unit 460 includes 2D rigging information estimated from a subsequent frame (A) or an image patch around the 2D joint position of a subsequent frame (A) and 2D rigging information converted from the current frame (B). Alternatively, the 2D rigging information converted from the current frame B may be compared with the image patches around the 2D joint position related to the 2D rigging information to find a matching point in A and B, and 2D rigging information may be supplemented for each object 200.

In FIG. 8, a 3D rigging information conversion unit 430, a 3D rigging information prediction unit 440, and a 2D rigging information conversion unit 450 can use a deep learning model learned from joint distance information. 430) encodes the 2D rigging information into 2D joint distance information (C), and the 2D rigging information converter 450 may decode (D) the 3D joint distance information into 3D rigging information.

9 is a flowchart illustrating a method of restoring 4D rigging information according to an embodiment of the present invention.

9, the 4D rigging information restoration method 500 according to an embodiment includes an image acquisition step (S510), 2D rigging information estimation step (S520), 3D rigging information conversion step (S530), and 3D rigging information prediction step. It may be configured to include (S540), 2D rigging information conversion step (S550), and data association step (S560).

In the image acquisition step (S510), the 4D rigging information restoration device 400 receives an image sequence captured by the monocular (mono) camera 300 from the camera 300 or from the database 490 in which the image sequence is stored. Sequence can be input.

In the 2D rigging information estimation step (S520), the 4D rigging information restoration device 400 receives the image sequence, detects the object 200 for each frame of the image sequence, and detects the object 200 based on the 2D joint position of the object 200. Rigging information can be estimated.

In the 3D rigging information conversion step (S530), the 4D rigging information restoration device 400 receives the 2D rigging information estimated in the 2D rigging information estimating step (S520) or supplemented in the data association step (S560) to be described later, and the 3D rigging information Can be converted to

In the 3D rigging information prediction step (S540), the 4D rigging information restoration device 400 may predict 3D rigging information after a specific time has elapsed from the converted 3D rigging information.

In the 2D rigging information conversion step (S550), the 4D rigging information restoration device 400 receives the 3D rigging information converted in the 3D rigging information conversion step (S530) or predicted in the 3D rigging information prediction step (S540) to receive 2D rigging information. Can be converted to

In the data association step (S560), the 4D rigging information restoration device 400 receives the estimated 2D rigging information and the converted 2D rigging information from a subsequent frame, and compares the two 2D rigging information to supplement the 2D rigging information. have.

In addition, the 4D rigging information restoration device 400 may supplement 2D rigging information using a Kalman filter or LSTM.

In addition, the 4D rigging information restoration apparatus 400 may supplement the 2D rigging information for each object 200 by comparing the 2D rigging information estimated from a subsequent frame with the converted 2D rigging information.

At this time, the 4D rigging information restoration device 400 includes 2D rigging information estimated from a subsequent frame or an image patch around the 2D joint position of a subsequent frame and converted 2D rigging information or 2D joint position related to the converted 2D rigging information. 2D rigging information for each object 200 may be supplemented by comparing surrounding image patches.

In addition, the 4D rigging information restoration device 400 returns to the 3D rigging information conversion step (S530) and converts the 2D rigging information supplemented in the data association step (S560) into 3D rigging information, thereby cyclically recovering the rigging information. have.

Meanwhile, the 4D rigging information restoration device 400 converts 2D rigging information into 3D in the 3D rigging information conversion step (S530), the 3D rigging information prediction step (S540), the 2D rigging information conversion step (S550), or the data association step (S560). A deep learning model can be used to convert 3D rigging information into rigging information or predict 3D rigging information from the converted 3D rigging information, convert 3D rigging information into 2D rigging information, or predict 2D rigging information from 2D rigging information.

In addition, the 4D rigging information restoration device 400, when the deep learning model is learned using the joint distance information, 3D rigging information conversion step (S530), 3D rigging information prediction step (S540), 2D rigging information conversion step (S550). ) Or, in order to use the deep learning model learned in the data association step (S560), it may be necessary to encode or decode 2D rigging information or 3D rigging information into 2D joint distance information or 3D joint distance information.

As described above, although it has been described with reference to the preferred embodiments of the present application, those skilled in the art will variously modify the present application within the scope not departing from the spirit and scope of the present invention described in the following claims. And it will be appreciated that it can be changed.

100: 4D rigging information restoration system
200: object 300: camera
400: 4D rigging information restoration device
410: image acquisition unit 420: 2D rigging information estimation
430: 3D rigging information conversion unit 440: 3D rigging information prediction unit
450: 2D rigging information conversion unit 460: Data connection unit
470: deep learning model unit 480: transceiver unit
490: database 495: control unit
500: 4D rigging information restoration method

Claims (10)

An image acquisition unit receiving an image sequence photographing one or more objects from a monocular camera;
A 2D rigging information estimation unit for detecting an object for each frame of the image sequence and estimating 2D rigging information based on the 2D joint position of the object;
3D rigging information conversion unit for converting 2D rigging information into 3D rigging information;
A 3D rigging information prediction unit for predicting 3D rigging information after a specific time has elapsed from the converted 3D rigging information;
A 2D rigging information conversion unit converting the predicted 3D rigging information into 2D rigging information; And
Comprising a data linking unit for compensating the estimated 2D rigging information by comparing the 2D rigging information estimated from a later frame by the 2D rigging information estimating unit and the 2D rigging information converted by the 2D rigging information conversion unit. 4D rigging information restoration device.
delete The method of claim 1,
The data linking unit compares the 2D rigging information estimated from a subsequent frame with the converted 2D rigging information to supplement the 2D rigging information for each object.
The method of claim 3,
The 2D rigging information includes 2D joint distance information or 2D joint position information,
The data linking unit compares the 2D rigging information estimated from a later frame or an image patch around the 2D joint position of a later frame and the converted 2D rigging information or the image patch around the 2D joint position related to the converted 2D rigging information. Thus, a 4D rigging information restoration device that supplements the 2D rigging information for each object.
The method of claim 1,
The data linking unit is a 4D rigging information restoration device that supplements the 2D rigging information using a Kalman filter or LSTM.
The method of claim 1,
The 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information, or predicts 3D rigging information from the converted 3D rigging information, or converts 3D rigging information into 2D rigging information. 4D rigging information restoration device that uses a deep learning model to transform.
The method of claim 6,
The deep learning model is learned using 2D joint distance information or 3D joint distance information,
The 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit is a 4D input to the deep learning model that is learned by encoding 2D rigging information or 3D rigging information into 2D joint distance information or 3D joint distance information. Rigging information restoration device.
The method of claim 7,
The 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit decodes the encoded 2D joint distance information or 3D joint distance information into 2D rigging information or 3D rigging information using a multidimensional scaling method (MDS). 4D rigging information restoration device.
The method of claim 6,
The 4D rigging information restoration device,
4D rigging information restoration apparatus comprising a deep learning model unit for training the deep learning model.
delete

Images