KR20200030947A - 4d rig reconstructing device and a method thereof - Google Patents

Abstract

Provided is a 4D rigging information restoration device. According to embodiments of the present invention, the 4D rigging information restoration device comprises: an image acquisition part receiving an image sequence of one or more objects from a monocular camera; a 2D rigging information estimation part detecting an object for each frame of the image sequence and estimating 2D rigging information based on a 2D joint position of the object; a 3D rigging information conversion part converting 2D rigging information into 3D rigging information; a 2D rigging information conversion part converting the converted 3D rigging information into 2D rigging information; and a data association part complementing the 2D rigging information by comparing 2D rigging information estimated from a subsequent frame with the converted 2D rigging information. According to embodiments of the present invention, the rigging information of the object can be accurately restored from the image photographed by the monocular camera.

Description

4D Rigging Information Restoration Device and Method {4D RIG RECONSTRUCTING DEVICE AND A METHOD THEREOF}

Embodiments of the present invention relates to an apparatus and method for restoring 4D rigging information, and more specifically, estimating 2D rigging information of one or more objects from an image sequence taken from a monocular camera and converting the estimated 2D rigging information to 3D rigging information. After conversion, the converted 3D rigging information is converted into 2D rigging information, and a 4D rigging information restoration apparatus and method for supplementing newly estimated 2D rigging information.

Rigging may mean attaching a skeleton to data modeled in 2D or 3D, and rigging information may mean joint location information, inter-articular distance information, or skeleton information.

Here, the joint position information may refer to the coordinates where the joint is located for each joint, and the inter-joint distance information may mean a distance from another joint for each joint, and the skeleton information may refer to the length of each bone connecting adjacent joints or the like. It may mean an angle or the like.

Rigging information restoration may mean calculating rigging information of an object that changes over time from an image sequence.

On the other hand, since the posture of the object can be known through the rigging information, the rigging information can be used in the same sense as the posture information, and the change in the posture of the object can be known by restoring the rigging information, so that the rigging information restoration is posture recognition, posture estimation, or motion recognition. It can be used in the same sense as.

Rigging information restoration is used in the field of motion recognition technology through image analysis, and is mainly used in rehabilitation therapy, haptic games, or sports movement exercises in these fields.

At this time, most of the prior art restores the rigging information using the depth information on the object together with the image of the object, but to measure the depth information and restore the rigging information, a multi-view image or depth measurement sensor is required. .

However, in order to obtain a multi-view image, a plurality of cameras arranged to shoot an object from various angles is required, so there is a problem that a space for the camera is required and the cost increases, and when using a depth measurement sensor, depth measurement distance or measurement There is a problem that the range is limited and the depth measurement error is increased due to noise caused by lighting.

On the other hand, depth information may be measured through a stereo image rather than a multi-view image. In this case, since a binocular (stereo) camera is required, a general monocular (mono) camera cannot be used.

In this regard, the prior art is as follows.

Korean Registered Patent No. 10-1784410 relates to a method for recognizing an exercise posture, and more specifically, obtaining position information of a user's body joint part taking a specific exercise posture, and using the position information, a vector in a 3D space It is related to a method of recognizing an exercise posture, comprising: obtaining, calculating an angle of a body joint part from the vector, and calculating a matching rate of the actual posture of the user with respect to the specific exercise posture using the angle.

However, the prior art uses a depth measurement sensor (IR projector) to calculate a three-dimensional attitude, and therefore does not propose a method of restoring rigging information from an image sequence taken with a monocular camera without depth information.

In order to solve the above-described problems, embodiments of the present invention is an image acquisition unit that receives an image sequence obtained by capturing one or more objects from a monocular camera, detects an object for each frame of the image sequence, and based on the 2D joint position of the object. From the 2D rigging information estimator for estimating 2D rigging information, the 3D rigging information conversion unit for converting 2D rigging information into 3D rigging information, the 2D rigging information conversion unit for converting the converted 3D rigging information into 2D rigging information, and subsequent frames. An object of the present invention is to provide a 4D rigging information restoration apparatus including a data connection unit that complements 2D rigging information by comparing the estimated 2D rigging information with the converted 2D rigging information.

In addition, embodiments of the present invention includes a 3D rigging information prediction unit that predicts 3D rigging information after a specific time from the converted 3D rigging information, and the 2D rigging information conversion unit converts the predicted 3D rigging information into 2D rigging information. The purpose is to provide a device for restoring rigging information.

In addition, embodiments of the present invention has an object to provide a 4D rigging information restoration apparatus that complements 2D rigging information for each object by comparing the 2D rigging information estimated from the subsequent frame with the converted 2D rigging information.

In addition, in the embodiments of the present invention, the 2D rigging information includes 2D joint distance information or 2D joint position information, and the data connection unit includes 2D rigging information estimated from a subsequent frame or a video patch around a 2D joint position of a subsequent frame. An object of the present invention is to provide a 4D rigging information restoration device that complements 2D rigging information for each object by comparing image patches around 2D joint positions related to the converted 2D rigging information or the converted 2D rigging information.

In addition, embodiments of the present invention has an object to provide a 4D rigging information restoration apparatus that complements 2D rigging information using a Kalman filter or LSTM by the data connection unit.

In addition, embodiments of the present invention, a 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information or predicts 3D rigging information from the converted 3D rigging information or 3D The purpose of the present invention is to provide a 4D rigging information restoration apparatus using a deep learning model to convert rigging information into 2D rigging information.

In addition, in the embodiments of the present invention, a deep learning model is learned using 2D joint distance information or 3D joint distance information, and the 3D rigging information conversion unit, 3D rigging information prediction unit, or 2D rigging information conversion unit 2D rigging information or 3D rigging An object of the present invention is to provide a 4D rigging information restoration apparatus that encodes information into 2D joint distance information or 3D joint distance information and inputs it into a learned deep learning model.

In addition, embodiments of the present invention is a 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit encoding the 2D joint distance information or 3D joint distance information using the multidimensional scaling method (MDS) 2D rigging information Another object is to provide a 4D rigging information restoration apparatus that decodes 3D rigging information.

In addition, embodiments of the present invention has an object to provide a 4D rigging information restoration apparatus including a deep learning model unit for learning a deep learning model.

In addition, embodiments of the present invention, an image acquisition step of receiving an image sequence of one or more objects taken from a monocular camera, detecting an object for each frame of the image sequence, and estimating 2D rigging information based on the 2D joint position of the object 2D rigging information estimation step, 3D rigging information conversion step to convert 2D rigging information to 3D rigging information, 2D rigging information conversion step to convert the converted 3D rigging information to 2D rigging information, and 2D rigging information estimated from the subsequent frames. An object of the present invention is to provide a method for restoring 4D rigging information, including a data linking step to complement the 2D rigging information by comparing the converted 2D rigging information.

One embodiment of the present invention for achieving the above object is an image acquisition unit for receiving an image sequence of one or more objects taken from a monocular camera; 2D rigging information estimation for detecting an object for each frame of the image sequence and estimating 2D rigging information based on the 2D joint position of the object; A 3D rigging information conversion unit converting the 2D rigging information into 3D rigging information; A 2D rigging information conversion unit converting the converted 3D rigging information into 2D rigging information; And it provides a 4D rigging information recovery apparatus including a data connection unit that complements the 2D rigging information by comparing the converted 2D rigging information with the 2D rigging information estimated from a subsequent frame.

In one embodiment, the apparatus for restoring 4D rigging information includes a 3D rigging information prediction unit for predicting 3D rigging information after a specific time has elapsed from the converted 3D rigging information, and the 2D rigging information conversion unit is the predicted 3D rigging information Can be converted into 2D rigging information.

In one embodiment, the data connection unit may complement the 2D rigging information for each object by comparing 2D rigging information estimated with 2D rigging information estimated from a subsequent frame.

In one embodiment, the 2D rigging information includes 2D joint distance information or 2D joint position information, and the data connection unit converts 2D rigging information estimated from a subsequent frame or an image patch around a 2D joint position of a subsequent frame. The 2D rigging information may be supplemented for each object by comparing an image patch around a 2D joint position related to the 2D rigging information or the converted 2D rigging information.

In one embodiment, the data connection unit may supplement the 2D rigging information using a Kalman filter or LSTM.

In one embodiment, the 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information or predicts 3D rigging information from the converted 3D rigging information or 3D rigging information A deep learning model can be used to convert the data into 2D rigging information.

In one embodiment, the deep learning model is learned using 2D joint distance information or 3D joint distance information, and the 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit is 2D rigging information or 3D The rigging information may be encoded into 2D joint distance information or 3D joint distance information and input into the learned deep learning model.

In one embodiment, the 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit encodes the encoded 2D joint distance information or 3D joint distance information using 2D rigging information (MDS) or 2D rigging information. It can be decoded into rigging information.

In one embodiment, the apparatus for restoring 4D rigging information may include a deep learning model unit for learning the deep learning model.

In addition, another embodiment of the present invention for achieving the above object is an image acquisition step of receiving an image sequence of one or more objects taken from a monocular camera; A 2D rigging information estimation step of detecting an object for each frame of the image sequence and estimating 2D rigging information based on the 2D joint position of the object; A 3D rigging information conversion step of converting the 2D rigging information into 3D rigging information; A 2D rigging information conversion step of converting the converted 3D rigging information into 2D rigging information; And a data linking step of compensating for the 2D rigging information by comparing the 2D rigging information estimated with the 2D rigging information estimated from a subsequent frame.

As described above, embodiments of the present invention, an image acquisition unit that receives an image sequence obtained by capturing one or more objects from a monocular camera, detects an object for each frame of the image sequence, and provides 2D rigging information based on the 2D joint position of the object. Estimation of 2D rigging information, 3D rigging information conversion unit for converting 2D rigging information into 3D rigging information, 2D rigging information conversion unit for converting converted 3D rigging information into 2D rigging information, and 2D rigging estimated from subsequent frames By providing the 4D rigging information restoration apparatus including a data connection unit that complements the 2D rigging information by comparing the information with the converted 2D rigging information, it is possible to accurately restore the rigging information of an object from an image photographed by a monocular camera.

In addition, embodiments of the present invention includes a 3D rigging information prediction unit that predicts 3D rigging information after a specific time from the converted 3D rigging information, and the 2D rigging information conversion unit converts the predicted 3D rigging information into 2D rigging information. By providing the rigging information restoring apparatus, the rigging information of the object can be accurately restored.

In addition, embodiments of the present invention provides a 4D rigging information restoration apparatus that complements 2D rigging information for each object by comparing the 2D rigging information estimated from a subsequent frame and the converted 2D rigging information by the data association unit. Can be accurately restored.

In addition, in the embodiments of the present invention, the 2D rigging information includes 2D joint distance information or 2D joint position information, and the data connection unit includes 2D rigging information estimated from a subsequent frame or a video patch around a 2D joint position of a subsequent frame. By providing a 4D rigging information restoration apparatus that complements 2D rigging information for each object by comparing image patches around 2D joint positions related to the converted 2D rigging information or the converted 2D rigging information, rigging information can be accurately restored for each object. .

In addition, embodiments of the present invention by providing a 4D rigging information recovery device that complements the 2D rigging information using the Kalman filter or LSTM by the data connection unit, it is possible to accurately restore the rigging information of the object from the image taken by the monocular camera. have.

In addition, embodiments of the present invention, a 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information or predicts 3D rigging information from the converted 3D rigging information or 3D By providing a 4D rigging information restoration apparatus using a deep learning model to convert rigging information into 2D rigging information, it is possible to accurately restore rigging information of an object from an image photographed by a monocular camera.

In addition, in the embodiments of the present invention, a deep learning model is learned using 2D joint distance information or 3D joint distance information, and the 3D rigging information conversion unit, 3D rigging information prediction unit, or 2D rigging information conversion unit 2D rigging information or 3D rigging By providing a 4D rigging information restoring device that encodes information into 2D joint distance information or 3D joint distance information and inputs it into the learned deep learning model, deep learning can be efficiently performed, and the object can be recorded from the image captured by the monocular camera. Rigging information can be accurately restored.

In addition, embodiments of the present invention is a 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit encoding the 2D joint distance information or 3D joint distance information using the multidimensional scaling method (MDS) 2D rigging information Alternatively, by providing a 4D rigging information restoration apparatus for decoding into 3D rigging information, deep learning can be efficiently performed and object rigging information can be accurately restored from an image captured by a monocular camera.

In addition, embodiments of the present invention by providing a 4D rigging information recovery apparatus including a deep learning model unit for learning a deep learning model, it is possible to accurately restore the rigging information of the object from the image taken by the monocular camera.

In addition, embodiments of the present invention, an image acquisition step of receiving an image sequence of one or more objects taken from a monocular camera, detecting an object for each frame of the image sequence, and estimating 2D rigging information based on the 2D joint position of the object 2D rigging information estimation step, 3D rigging information conversion step to convert 2D rigging information to 3D rigging information, 2D rigging information conversion step to convert the converted 3D rigging information to 2D rigging information, and 2D rigging information estimated from the subsequent frames. By providing a method for restoring 4D rigging information including a data linking step to complement the 2D rigging information by comparing the converted 2D rigging information, it is possible to accurately restore the rigging information of an object from an image photographed by a monocular camera.

It is not limited to the above-mentioned effects, other effects are not mentioned can be clearly understood by those skilled in the art from the following description.

1 is a view showing a 4D rigging information restoration system according to an embodiment of the present invention.
2 is a block diagram showing a 4D rigging information restoration apparatus according to an embodiment of the present invention.
3 is a view showing a joint and a skeleton according to an embodiment of the present invention.
4 is a view showing 2D joint position information according to an embodiment of the present invention.
5 is a view showing inter-articular distance information according to an embodiment of the present invention.
6 is a diagram illustrating a deep learning model according to an embodiment of the present invention.
7 and 8 are views illustrating a 4D rigging information restoration apparatus according to an embodiment of the present invention.
9 is a flowchart illustrating a method for restoring 4D rigging information according to an embodiment of the present invention.

Since the description of the present invention is merely embodiments for structural or functional description, the scope of the present invention should not be interpreted as being limited by the embodiments described in the text. That is, since the present embodiments can be variously changed and have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing technical ideas. In addition, the purpose or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such an effect, and the scope of the present invention should not be understood as being limited thereby.

In addition, the accompanying drawings are provided to help understanding of the present invention, and provide embodiments with detailed description. However, the technical features of the present invention are not limited to specific drawings, and the features disclosed in each drawing may be combined with each other to form a new embodiment.

The 4D rigging information restoration apparatus and method disclosed in the following embodiments will be described in more detail with reference to each drawing.

1 is a view showing a 4D rigging information restoration system according to an embodiment of the present invention.

Referring to FIG. 1, the 4D rigging information restoration system 100 according to an embodiment may include an object 200, a camera 300, and a 4D rigging information restoration device 400.

The object 200 is photographed by the camera 300 and can be moved while changing posture, and may be a sports player. Also, the object 200 may be plural.

The camera 300 may be a monocular (mono) camera and may be connected to the 4D rigging information restoration device 400 through a communication network. The camera 300 may transmit an image sequence photographing one or more objects 200 to the 4D rigging information restoration apparatus 400 through a communication network.

Here, the communication network may mean a wide-concept network including a wired or wireless communication network.

Further, the camera 300 may be a PTZ camera capable of photographing while tracking the movement of the object 200.

Here, the PTZ camera may mean a camera capable of rotation (PAN), vertical tilt (Tilt), and zoom (Zoom) adjustment.

The 4D rigging information restoration device 400 is connected to the camera 300 through a communication network, and a server, PC, notebook, mobile phone, smart phone (2G / 3G / 4G / LET, smart phone), PMP (Portable Media Player), It may be any one of a PDA (Personal Digital Assistant) and a tablet PC (Tablet PC). The detailed configuration of the 4D rigging information restoration apparatus 400 will be described with reference to FIG. 2.

2 is a block diagram showing a 4D rigging information restoration apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the 4D rigging information restoration apparatus 400 according to an embodiment includes an image acquisition unit 410, a 2D rigging information estimation unit 420, a 3D rigging information conversion unit 430, and a 3D rigging information prediction unit. 440, a 2D rigging information conversion unit 450, a data connection unit 460, a deep learning model unit 470, a transceiver 480, a database 490 and a control unit 495. .

The image acquisition unit 410 receives the image sequence photographed by the monocular (mono) camera 300 from the camera 300 through the transmission / reception unit 480 or inputs the image sequence from the database 490 in which the image sequence is stored. Can receive

Here, the video sequence may mean a series of frames, and the frame may mean one of a plurality of images (frames) constituting a video (a series of frames). The video sequence may include the current frame and subsequent frames.

The 2D rigging information estimator 420 receives the image sequence from the image acquisition unit 410, detects the object 200 for each frame of the image sequence, and provides 2D rigging information based on the 2D joint position of the object 200. Can be estimated.

Here, detecting the object 200 for each frame means that the 2D rigging information estimator 420 detects the object 200 for all frames received from the image acquisition unit 410 or the database 490. It may mean that the object 200 is detected for each frame of a part of the received image sequence.

In addition, here, 2D joint position may mean two-dimensional position coordinates for each type of joint. For example, it may mean a 2D position coordinate of a left knee joint or a 2D position coordinate of a right elbow joint.

Also, here, rigging may mean attaching a skeleton to data modeled in 2D or 3D.

Also, here, the rigging information may mean joint position information, inter-articular distance information, or skeleton information, 2D rigging information may mean rigging information in two dimensions, and 3D rigging information may mean rigging information in three dimensions. .

Here, the joint position information may refer to the coordinates where the joint is located for each type of joint, and the inter-joint distance information may mean the distance from each joint to each joint, and the skeleton information is the length of each bone connecting adjacent joints. It can mean or angle.

However, since the inter-articular distance information or the skeleton information may be generated based on the joint position information, the rigging information may be estimated based on the joint position information basically regardless of its meaning.

3 is a view showing a joint and a skeleton according to an embodiment of the present invention.

3 is a view showing an example of a joint and a skeleton according to an embodiment.

Referring to FIG. 3, the object 200 may include joints such as shoulder joints J1 and J4, elbow joints J2 and J5, knee joints J8 and J11, and ankle joints J9 and J12. By connecting adjacent joints such as the knee joints (J8, J11) and ankle joints (J9, J12), a skeleton such as the lower leg skeletons (B1, B2) can be obtained. That is, the skeleton can be obtained from the joint.

4 is a view showing 2D joint position information according to an embodiment of the present invention.

Referring to FIG. 4, the 2D joint position information may correspond to two-dimensional position coordinates (X, Y) for each type of joints J1 to J14. Here, X and Y may be coordinate values of the X-axis and the Y-axis of the pixel where each joint is located in one frame.

As described above, since the rigging information may mean joint position information, the 2D rigging information may mean the 2D joint position information in FIG. 4, and the 3D rigging information may include joints (J1 to J1) in FIG. J14) may mean three-dimensional position coordinates (coordinates of the X-axis, Y-axis, and Z-axis).

On the other hand, the rigging information may refer to bone information indicating the length or angle of each bone, and the bone information can be easily calculated based on the joint position information of adjacent joints as shown in FIG. 3.

That is, the length of the straight line connecting two adjacent joints may correspond to the length of the bone, and the slope of the straight line may correspond to the angle of the bone.

5 is a view showing inter-articular distance information according to an embodiment of the present invention.

Referring to FIG. 5, the inter-articular distance information may correspond to a distance from each joint (J1 to J14) to all other joints (J1 to J14), and the distance between the two joints may be calculated based on two joint position information. . Here, the distance may be the Euclidean distance.

As described above, the rigging information may mean inter-articular distance information in FIG. 5 and the format of the inter-articular distance information in FIG. 5 regardless of whether the rigging information is 2D rigging information or 3D rigging information (that is, regardless of dimensions). (n rows n columns, n is the number of joints).

However, the values of the inter-articular distance information in 2D are calculated based on the 2D joint position information, and the values of the inter-articular distance information in 3D are calculated based on the 3D joint position information. It may be less than or equal to the values of the inter-articular distance information in 3D.

Returning to FIG. 2 again, the 2D rigging information estimator 420 uses, for example, a method of separately detecting a pedestrian's head, arms, legs, upper body, etc., among the pedestrian detection methods, each body of the object 200 in the image information. After detecting the site, the connection point between body parts can be estimated as the 2D joint position.

However, the 2D rigging information estimator 420 is not limited to the above method and may use various other methods to estimate the 2D joint position.

In addition, the 2D rigging information estimator 420 may use the 2D joint position estimated by the above method as 2D rigging information when the rigging information is joint position information, and when the rigging information is skeleton information or inter-articular distance information, Based on the 2D joint position estimated by the method, bone information or inter-articular distance information can be calculated and used as 2D rigging information.

The 3D rigging information conversion unit 430 may receive 2D rigging information estimated by the 2D rigging information estimator 420 or supplemented by the data connection unit 460 to be described later, and convert the 3D rigging information into 3D rigging information.

The 3D rigging information prediction unit 440 may predict 3D rigging information after a specific time has elapsed from the 3D rigging information converted by the 3D rigging information conversion unit 430. Here, the specific time may mean a difference between the shooting time of the current frame and the shooting time of the frame after restoring rigging information.

The 2D rigging information conversion unit 450 may be converted by the 3D rigging information conversion unit 430 or 3D rigging information predicted by the 3D rigging information prediction unit 440 to be converted into 2D rigging information.

At this time, a loss may occur in the information related to the depth (Z-axis) among the rigging information. The lost depth information may be determined by the photographing direction of the camera, and the front direction of the camera may correspond to the depth (Z-axis).

The 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or the 2D rigging information conversion unit 450 converts 2D rigging information into 3D rigging information or predicts 3D rigging information from the converted 3D rigging information or 3D A deep learning model can be used to convert rigging information to 2D rigging information.

In relation to the deep learning model, the deep learning model unit 470 will be described in detail later.

As such, the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, and the 2D rigging information conversion unit 450 convert 2D rigging information into 3D rigging information and predict 3D rigging information after a specific time, By converting the converted or predicted 3D rigging information into 2D rigging information, the accuracy of the rigging information can be increased from a subsequent frame.

The data association unit 460 is connected to the 2D rigging information estimation unit 420 and the 2D rigging information conversion unit 450 and estimated from the subsequent frames from the 2D rigging information estimation unit 420 and the 2D rigging information conversion unit 450. 2D rigging information can be supplemented by receiving the converted 2D rigging information and the converted 2D rigging information.

Here, the 2D rigging information estimated from the subsequent frame is the 2D rigging information estimator 420 detects the object 200 from the subsequent frame and estimates the 2D rigging information based on the 2D joint position of the object 200. Can mean

In addition, here, the converted 2D rigging information includes 2D rigging information estimated from the current frame by the 2D rigging information estimator 420, a data association unit 460, a 3D rigging information conversion unit 430, and a 2D rigging information conversion unit. It may mean 2D rigging information converted through 450. At this time, the 3D rigging information prediction unit 440 may be further passed between the 3D rigging information conversion unit 430 and the 2D rigging information conversion unit 450.

In addition, here, the 2D rigging information supplemented by the data association unit 460 may be 2D rigging information estimated from a subsequent frame or may be converted 2D rigging information. In any case, it is supplemented by the data association unit 460 It is common that the 2D rigging information is transmitted to the 3D rigging information conversion unit 430 and converted into 3D rigging information.

In addition, the data connection unit 460 may supplement 2D rigging information using a Kalman filter or LSTM.

First, a method of complementing the 2D rigging information using the Kalman filter by the data connection unit 460 will be described.

Here, the Kalman filter is an algorithm based on a series of measurements observed over time, and can calculate an accurate estimate of the state from measurements that include noise.

The Kalman filter algorithm can be divided into two phases: the prediction phase and the update phase. The prediction phase calculates the expected state from the estimated state at the previous time, and the update phase calculates the correct state based on the predicted state and the actual measured state. Can be calculated.

Specifically, the data association unit 460 uses the Kalman filter to predict 2D rigging information to be expected later based on the 2D rigging information obtained from the current frame in the prediction step, and in the update step, the 2D rigging information estimated from the subsequent frame. 2D rigging information can be supplemented by comparing and converted 2D rigging information.

At this time, the data connection unit 460 may use the speed of each joint point to predict 2D rigging information.

In addition, the data connection unit 460 converts or predicts the prediction or supplementary steps of the Kalman filter with the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or the 2D rigging information conversion unit 450. It may be performed independently or may be performed in conjunction with transformation or prediction by the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or the 2D rigging information conversion unit 450.

Second, a method of supplementing 2D rigging information using LSTM by the data association unit 460 will be described.

Here, a long short-term memory (LSTM) is a unit of a cyclic neural network (RNN) and may be composed of a cell, an input gate, an output gate, and an oblivion gate. The cell remembers values over an arbitrary time interval, and the three gates can control the flow of information into and out of the cell.

Specifically, the data connection unit 460 can predict the predicted 2D rigging information based on the 2D rigging information obtained from the current frame using a deep learning (RNN) model using LSTM as a structural unit, and from the subsequent frames The 2D rigging information may be supplemented by comparing the estimated 2D rigging information with the converted 2D rigging information.

Here, the deep learning model and the RNN will be described in the deep learning model unit 470.

On the other hand, the data connection unit 460 may use a Kalman filter and LSTM. Specifically, the data connection unit 460 uses the deep learning model using LSTM as a structural unit in the prediction stage to obtain 2D rigging information obtained from the current frame. The predicted 2D rigging information and the variables to be used for the Kalman filter can be predicted, and in the update step, the 2D rigging information can be supplemented by comparing the 2D rigging information estimated from the subsequent frame with the converted 2D rigging information using the Kalman filter. have.

In addition, the data connection unit 460 converts or predicts the deep learning model using LSTM as a unit in the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, or the 2D rigging information conversion unit 450. It may be used independently of the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440 or 2D rigging information conversion unit 450, or may be used in conjunction with the prediction.

As described above, the data connection unit 460 compares the 2D rigging information estimated from the subsequent frame and the converted 2D rigging information based on the temporal change of the 2D rigging information using a Kalman filter or LSTM to obtain the 2D rigging information. By supplementing, the accuracy of the rigging information can be increased.

On the other hand, when a plurality of objects 200 are photographed in the image information, the rigging information should be restored by distinguishing each object 200, and a method related to this will be described.

The data connection unit 460 may supplement 2D rigging information for each object 200 by comparing 2D rigging information estimated from a subsequent frame with converted 2D rigging information.

At this time, the data association unit 460 is around the 2D joint position related to the 2D rigging information estimated from the subsequent frame or the image patch around the 2D joint position of the subsequent frame and the converted 2D rigging information or the converted 2D rigging information. 2D rigging information may be supplemented for each object 200 by comparing image patches.

Specifically, the data association unit 460 compares the 2D rigging information estimated by the 2D rigging information conversion unit 450 by distinguishing the 2D rigging information estimated from the subsequent frame for each object 200 from the current frame. 200) can be used to supplement 2D rigging information.

That is, when the 2D rigging information estimator 420 incorrectly distinguishes the rigging information of the two objects 200 from the subsequent frame and estimates the 2D rigging information while the joints of the two objects 200 are interchanged, the data connection unit ( 460) compares the 2D rigging information that is correctly classified and converted for each object 200 from the current frame to supplement the 2D rigging information of the subsequent frame.

At this time, the data connection unit 460 may complement the 2D rigging information by distinguishing the object 200 by comparing the 2D joint position information, the 2D joint distance information, or the image patch around the 2D joint position. Here, the image patch may mean a 2D image of a specific size.

As described above, the data association unit 460 distinguishes the object 200 and supplements 2D rigging information, thereby increasing the accuracy of the rigging information. In particular, the data connection unit 460 can complement the rigging information quickly and accurately by distinguishing the object 200 by comparing 2D joint distance information.

The deep learning model unit 470 may be connected to the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450, or the data connection unit 460, and the deep learning model described later. Can provide

The deep learning model unit 470 may generate a deep learning model and train a deep learning model using learning data previously stored in the database 490. Here, deep learning is a type of machine learning method for learning data representation, and typically DNN, CNN, RNN, etc. may be applicable to this. However, other deep learning methods such as DBN may also be used.

Here, the deep neural network (DNN) is an artificial neural network composed of several hidden layers between an input layer and an output layer.

Also here, the Convolutional Neural Network (CNN) is a kind of multi-layer perceptrons designed to use minimal preprocessing, and one or more convolutional layers and common artificial objects on it. It consists of neural network layers, and weights and pooling layers can be additionally used.

Also, here, the recurrent neural network (RNN) is a neural network in which connections between units constituting an artificial neural network constitute a Directed cycle, and can utilize memory inside the neural network to process arbitrary input. As a structure constituting a cyclic neural network, various methods such as a fully recurrent network (LSTM) and a long short term memory network (LSTM) can be used.

The deep learning model unit 470 generates a deep learning model suitable for the functions of the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450, or the data connection unit 460. It can be provided by learning and see below.

First, the 3D rigging information conversion unit 430 will be described.

The deep learning model unit 470 may generate a 3D-converted deep learning model for converting 2D rigging information into 3D rigging information. At this time, the deep learning model unit 470 can generate a 3D-converted deep learning model that uses 2D rigging information as an input value and 3D rigging information as an output value. The input and output values are relatively simple and the current output value is Since the input / output value may not be affected, the DNN model may be preferable for the 3D transformed deep learning model. However, other methods such as RNN and CNN may also be used.

In addition, the deep learning model unit 470 may receive training data consisting of various 2D rigging information and 3D rigging information corresponding to each 2D rigging information from the database 490 to train a 3D transformed deep learning model.

Also, the deep learning model unit 470 may provide the 3D rigging information conversion unit 430 with a 3D transformed deep learning model for which learning is completed.

Second, look at the 3D rigging information prediction unit 440.

The deep learning model unit 470 may generate a 3D predictive deep learning model for predicting 3D rigging information after a specific time has elapsed from the 3D rigging information. At this time, the deep learning model unit 470 may generate a 3D predictive deep learning model using 3D rigging information as an input value and 3D rigging information as an output value. The input and output values are relatively simple and the current predicted output value is The RNN model may be preferable for the 3D predictive deep learning model because it may be affected by the previous output value. However, other methods such as DNN and CNN may also be used. In this case, the 3D predictive deep learning model may need to input additional information such as previous joint position information or joint speed in order to calculate the joint speed. .

In addition, the deep learning model unit 470 may receive training data consisting of various 3D rigging information and predicted 3D rigging information corresponding to each 3D rigging information from the database 490 to train a 3D predictive deep learning model.

In addition, the deep learning model unit 470 may provide the 3D rigging information prediction unit 440 with a 3D prediction deep learning model that has been completed.

Third, look at the 2D rigging information conversion unit 450.

The deep learning model unit 470 may generate a 2D transformed deep learning model for converting 3D rigging information into 2D rigging information. At this time, the deep learning model unit 470 may generate a 2D-converted deep learning model using 3D rigging information as an input value and 2D rigging information as an output value. The input and output values are relatively simple and the current output value is Since the input / output value may not be affected, the DNN model may be preferable for the 2D transformed deep learning model. However, other methods such as RNN and CNN may also be used.

In addition, the deep learning model unit 470 may receive training data consisting of various 3D rigging information and 2D rigging information corresponding to each 3D rigging information from the database 490 to train a 2D transformed deep learning model.

Also, the deep learning model unit 470 may provide the 2D rigging information conversion unit 450 with a 2D transformed deep learning model that has completed learning.

Fourth, it looks at in relation to the data connection unit 460.

The deep learning model unit 470 may generate a 2D predictive deep learning model for predicting expected 2D rigging information based on 2D rigging information obtained from the current frame. At this time, the deep learning model unit 470 may generate a 2D predictive deep learning model that uses 2D rigging information as an input value and 2D rigging information as an output value. The input and output values are relatively simple and the predicted output value afterwards The RNN model may be preferable for the 2D predictive deep learning model because it may be affected by the current output value. However, other methods such as DNN and CNN may also be used. In this case, the 2D predictive deep learning model may need to input additional information such as previous joint position information or joint speed in order to calculate joint speed. .

In addition, the deep learning model unit 470 may receive a training data composed of various 2D rigging information and predicted 2D rigging information corresponding to each 2D rigging information from the database 490 to train a 2D predictive deep learning model.

In addition, the deep learning model unit 470 may provide the 2D predictive deep learning model, which has been completed, to the data association unit 460.

As such, the deep learning model unit 470 learns and provides a deep learning model and provides a 3D rigging information conversion unit 430, a 3D rigging information prediction unit 440, a 2D rigging information conversion unit 450, or a data connection unit ( 460) can increase the accuracy of the rigging information by using the deep learning model provided by the deep learning model unit 470.

Meanwhile, the deep learning model unit 470 may use 2D joint distance information or 3D joint distance information when training a 3D transformed deep learning model, a 3D predicted deep learning model, a 2D transformed deep learning model, or a 2D predicted deep learning model. have.

That is, the input and output values of the 3D transformed deep learning model, 3D predicted deep learning model, 2D transformed deep learning model, or 2D predicted deep learning model may correspond to 2D joint distance information or 3D joint distance information.

6 is a diagram illustrating a deep learning model according to an embodiment of the present invention.

Referring to FIG. 6, the deep learning model may be trained by the deep learning model unit 470 using inter-joint distance information as input and output values, and after learning is completed, the 3D rigging information conversion unit 430 and 3D rigging information prediction The unit 440 may be provided to the 2D rigging information conversion unit 450 or the data connection unit 460.

The matrix of 14 rows and 14 columns corresponding to the input value and the output value of FIG. 6 may mean an inter-Euclidean distance for 14 joints of J1 to J14 of FIG. 3 and the same meaning as the inter-articular distance information of FIG. 5 Can be

As described above, the deep learning model unit 470 trains and provides a deep learning model using inter-articular distance information, so that the 2D and 3D rigging information is the same without being affected by rotation and translation of the image. It can be expressed as a dimension (n rows, n columns, and n is the number of joints). Since it does not use 2D or 3D coordinates, the solution space is reduced when learning deep learning, so deep learning can be performed efficiently.

On the other hand, when the deep learning model is learned using the inter-articular distance information, the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450, or the data connection unit 460 In order to use the learned deep learning model, it may be necessary to encode 2D rigging information or 3D rigging information into 2D joint distance information or 3D joint distance information.

In addition, the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, the 2D rigging information conversion unit 450 or the data connection unit 460 uses the encoded 2D joint distance information or 3D joint distance information. After using the deep learning model, it may be necessary to decode the encoded 2D joint distance information or 3D joint distance information into 2D rigging information or 3D rigging information. In this case, multi-dimensional scaling may be used.

Here, the multi-dimensional scaling method can be used to visualize the Euclidean distance matrix as a means of visualizing the similarity of data. In the visualization, a method of expressing a point in a 2D or 3D space is mainly used.

However, when 2D rigging information or 3D rigging information is 2D joint distance information or 3D joint distance information, it may be used without encoding or decoding.

The database 490 may store image sequences received from the camera 300 and training data for training a 3D transformed deep learning model, a 3D predictive deep learning model, a 2D transformed deep learning model, or a 2D predictive deep learning model.

The database 490 is a concept including a computer-readable recording medium, and includes not only a consultative database, but also a database having a broad meaning including data recording based on a file system. Therefore, if data can be extracted, it is included in the scope of the database referred to in the present invention.

Finally, the control unit 495, the image acquisition unit 410, 2D rigging information estimation unit 420, 3D rigging information conversion unit 430, 3D rigging information prediction unit 440, 2D rigging information conversion unit 450 , It can control the control flow or data flow between the data connection unit 460, the deep learning model unit 470, the transmission / reception unit 480, and the database 490.

7 and 8 are views illustrating a 4D rigging information restoration apparatus according to an embodiment of the present invention.

7 and 8, the 4D rigging information restoring apparatus 400 according to an embodiment includes an image acquisition unit 410, a 2D rigging information estimation unit 420, a 3D rigging information conversion unit 430, and a 3D rigging. It may be configured to include an information prediction unit 440, a 2D rigging information conversion unit 450, a data connection unit 460.

In FIG. 7, A is a subsequent frame, B is a current frame, and the data connection unit 460 receives the estimated 2D rigging information from the subsequent frame A, and converts the 2D rigging information from the current frame B. Compare with, 2D rigging information can be supplemented for each object.

At this time, the data association unit 460 estimates the 2D rigging information from the subsequent frame A or the image patch around the 2D joint position of the subsequent frame A and the 2D rigging information converted from the current frame B. Alternatively, the image patch around the 2D joint position related to the 2D rigging information converted from the current frame B may be compared to find a matching point in A and B, and 2D rigging information may be supplemented for each object 200.

In FIG. 8, the 3D rigging information conversion unit 430, the 3D rigging information prediction unit 440, and the 2D rigging information conversion unit 450 enable the 3D rigging information conversion unit to use the deep learning model learned as the inter-articular distance information. 430) encodes (C) the 2D rigging information into 2D joint distance information, and the 2D rigging information conversion unit 450 can decode (D) the 3D joint distance information into 3D rigging information.

9 is a flowchart illustrating a method for restoring 4D rigging information according to an embodiment of the present invention.

Referring to FIG. 9, the method for restoring 4D rigging information 500 according to an embodiment includes an image acquisition step (S510), a 2D rigging information estimation step (S520), a 3D rigging information conversion step (S530), and a 3D rigging information prediction step. (S540), 2D rigging information conversion step (S550) and may be configured to include a data association step (S560).

In the image acquisition step (S510), the 4D rigging information restoration apparatus 400 receives an image sequence photographed by the monocular (mono) camera 300 from the camera 300 or an image from a database 490 in which the image sequence is stored. The sequence can be input.

In the 2D rigging information estimation step (S520), the 4D rigging information restoration apparatus 400 receives the image sequence, detects the object 200 for each frame of the image sequence, and based on the 2D joint position of the object 200 Rigging information can be estimated.

In the 3D rigging information conversion step (S530), the 4D rigging information restoration apparatus 400 receives 3D rigging information by receiving the 2D rigging information estimated in the 2D rigging information estimation step (S520) or supplemented in the data association step (S560) to be described later. Can be converted to

In the 3D rigging information prediction step (S540), the 4D rigging information restoration apparatus 400 may predict 3D rigging information after a specific time has elapsed from the converted 3D rigging information.

In the 2D rigging information conversion step (S550), the 4D rigging information restoration apparatus 400 receives the 3D rigging information converted in the 3D rigging information conversion step (S530) or the 3D rigging information prediction step (S540) and receives the 2D rigging information. Can be converted to

In the data linking step (S560), the 4D rigging information restoring device 400 receives the estimated 2D rigging information and the converted 2D rigging information from a subsequent frame, and compares the two 2D rigging information to supplement the 2D rigging information. have.

In addition, the 4D rigging information restoration apparatus 400 may supplement 2D rigging information using a Kalman filter or LSTM.

In addition, the 4D rigging information restoring apparatus 400 may supplement 2D rigging information for each object 200 by comparing 2D rigging information estimated from a subsequent frame and converted 2D rigging information.

At this time, the 4D rigging information restoring apparatus 400 may estimate the 2D rigging information estimated from the subsequent frame or the image patch around the 2D joint position of the subsequent frame and the 2D joint position related to the converted 2D rigging information or the converted 2D rigging information. 2D rigging information may be supplemented for each object 200 by comparing surrounding image patches.

In addition, the 4D rigging information restoring device 400 may return to the 3D rigging information conversion step (S530) and convert the 2D rigging information supplemented in the data association step (S560) into 3D rigging information to restore rigging information recursively. have.

On the other hand, the 4D rigging information restoration apparatus 400 3D rigging information in the 3D rigging information conversion step (S530), 3D rigging information prediction step (S540), 2D rigging information conversion step (S550) or data association step (S560). A deep learning model can be used to predict 3D rigging information from converted 3D rigging information or to convert 3D rigging information to 2D rigging information or to predict 2D rigging information from 2D rigging information.

In addition, the 4D rigging information restoration apparatus 400, when the deep learning model is learned using the inter-articular distance information, the 3D rigging information conversion step (S530), the 3D rigging information prediction step (S540), and the 2D rigging information conversion step (S550) ) Or, in order to use the deep learning model learned in the data connection step (S560), it may be necessary to encode or decode 2D rigging information or 3D rigging information into 2D joint distance information or 3D joint distance information.

As described above, although described with reference to preferred embodiments of the present application, those skilled in the art variously modify the present application without departing from the spirit and scope of the present invention as set forth in the claims below. And can be changed.

100: 4D rigging information restoration system
200: object 300: camera
400: 4D rigging information restoration device
410: image acquisition unit 420: 2D rigging information estimation
430: 3D rigging information conversion unit 440: 3D rigging information prediction unit
450: 2D rigging information conversion unit 460: Data connection unit
470: deep learning model unit 480: transceiver
490: database 495: control unit
500: 4D rigging information restoration method

Claims (10)

An image acquisition unit that receives an image sequence of one or more objects taken from a monocular camera;
2D rigging information estimation for detecting an object for each frame of the image sequence and estimating 2D rigging information based on the 2D joint position of the object;
A 3D rigging information conversion unit converting the 2D rigging information into 3D rigging information;
A 2D rigging information conversion unit converting the converted 3D rigging information into 2D rigging information; And
A 4D rigging information reconstruction device comprising a data linkage unit that complements the 2D rigging information by comparing the 2D rigging information estimated with the 2D rigging information estimated from a subsequent frame.
According to claim 1,
The 4D rigging information recovery device,
And a 3D rigging information prediction unit for predicting 3D rigging information after a specific time has elapsed from the converted 3D rigging information,
The 2D rigging information conversion unit is a 4D rigging information restoration apparatus for converting the predicted 3D rigging information into 2D rigging information.
According to claim 1,
The data association unit compares the 2D rigging information estimated from a subsequent frame with the converted 2D rigging information, and a 4D rigging information restoration apparatus that complements the 2D rigging information for each object.
According to claim 3,
The 2D rigging information includes 2D joint distance information or 2D joint location information,
The data connection unit compares the 2D rigging information estimated from the subsequent frame or the image patch around the 2D joint position of the subsequent frame with the image patch around the 2D joint position related to the converted 2D rigging information or the converted 2D rigging information. 4D rigging information restoration device to complement the 2D rigging information for each object.
According to claim 1,
The data connection unit is a 4D rigging information recovery device that supplements the 2D rigging information using a Kalman filter or LSTM.
The method according to claim 1 or 2,
The 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit converts 2D rigging information into 3D rigging information or predicts 3D rigging information from the converted 3D rigging information or converts 3D rigging information into 2D rigging information. 4D rigging information restoration device using deep learning model to convert.
The method of claim 6,
The deep learning model is learned using 2D joint distance information or 3D joint distance information,
The 3D rigging information conversion unit, the 3D rigging information prediction unit or the 2D rigging information conversion unit encodes 2D rigging information or 3D rigging information into 2D joint distance information or 3D joint distance information, and inputs it to the learned deep learning model. Rigging information restoration device.
The method of claim 7,
The 3D rigging information conversion unit, the 3D rigging information prediction unit, or the 2D rigging information conversion unit decodes the encoded 2D joint distance information or 3D joint distance information into 2D rigging information or 3D rigging information using a multidimensional scaling method (MDS). 4D rigging information restoration device.
The method of claim 6,
The 4D rigging information recovery device,
4D rigging information restoration apparatus including a deep learning model unit for learning the deep learning model.
An image acquisition step of receiving an image sequence of one or more objects taken from a monocular camera;
A 2D rigging information estimation step of detecting an object for each frame of the image sequence and estimating 2D rigging information based on the 2D joint position of the object;
A 3D rigging information conversion step of converting the 2D rigging information into 3D rigging information;
A 2D rigging information conversion step of converting the converted 3D rigging information into 2D rigging information; And
A method of restoring 4D rigging information comprising a data linking step of compensating for the 2D rigging information by comparing the 2D rigging information estimated with the 2D rigging information estimated from a subsequent frame.

Images