KR20230122551A - Motion data annotation method and system for ai learning - Google Patents

Abstract

A motion data annotation method for artificial intelligence (AI) learning according to an embodiment of the present invention, which can quickly and accurately acquire motion data for AI learning using 2D images and also 3D images, comprises the steps of: matching a coordinate system of a first sensor system and a coordinate system of a second sensor system; taking a three-dimensional image including joint position information of a skeleton using the first sensor system; taking a two-dimensional image using the second sensor system; projecting the joint position information of the skeleton to a virtual camera applying internal and external variables of the second sensor system; and by using the joint position information of the skeleton projected to the virtual camera and the two-dimensional image taken by the second sensor system, automatically annotating joints in the two-dimensional image taken by the second sensor system.

Description

Motion data annotation method and system for AI learning {MOTION DATA ANNOTATION METHOD AND SYSTEM FOR AI LEARNING}

An embodiment of the present invention relates to a motion data annotation method for AI (Artificial Intelligence) learning, and more particularly, to a motion data annotation method for AI learning through matching of heterogeneous sensor systems.

Motion capture refers to recording the movement of an object in digital form, and high-quality motion capture is required as the game industry, animation industry, and VR (Virtual Reality) technology develop.

When converting an object in a 2D image into a 3D model, a data set is required to infer the pose and shape of the object. This data set includes object joint information, posture information, and shape information, and can be used for AI learning.

In general, an annotation tool may be used to estimate at least one of a joint, posture, and shape of an object from a 2D image. According to this, 2D location information of joints of objects in a 2D image may be manually annotated.

1-2 are examples of general annotation tools.

Referring to FIGS. 1 and 2 , a user may manually annotate joints of an object. As shown in FIG. 1 , in the case of a model wearing clothes revealing the shape of the body, the annotation process is easy and the annotation accuracy is high. On the other hand, as shown in FIG. 2 , in the case of a model wearing various types of equipment or clothes that cover body parts such as a skirt, an error in posture estimation may occur.

As such, when manually performing annotation using an annotation tool, it may be difficult to obtain an accurate joint position. When AI learning is performed using manually performed annotation results, a skeleton with a large error may be generated.

An embodiment of the present invention is to provide a method and apparatus for annotating motion data for AI (Artificial Intelligence) learning.

According to an embodiment of the present invention, it is intended to provide a method and apparatus for annotating motion data for AI learning necessary for predicting a posture of an object in a 2D image.

The problem to be solved in the embodiments of the present invention is not limited thereto, and it will be said that the purpose or effect that can be grasped from the solution or embodiment of the problem described below is also included.

A motion data annotation method for AI (Artificial Intelligence) learning according to an embodiment of the present invention includes the steps of matching a coordinate system of a first sensor system and a coordinate system of a second sensor system, joint position information of a skeleton using the first sensor system. photographing a 3D image including, photographing a 2D image through the second sensor system, and projecting the joint position information of the skeleton to a virtual camera to which internal and external variables of the second sensor system are applied. and automatically annotating joints in the 2D image captured through the second sensor system using joint position information of the skeleton projected on the virtual camera and the 2D image captured through the second sensor system. It includes steps to

In the projecting step, the joint position information of the skeleton may be projected as a 2D image on the virtual camera.

In the annotating, the joint position information of the skeleton projected as a 2D image by the virtual camera may be overlapped with the 2D image captured through the second sensor system.

In the annotating step, the joint position information of the skeleton projected on the virtual camera is converted into 2D image coordinates, and the 2D image coordinates are automatically matched to the 2D image captured through the second sensor system. can

The method may further include estimating the internal variable of the second sensor system before the matching.

The first sensor system includes: an optical motion capture module that outputs infrared light and receives the infrared light reflected from the object to estimate the position of the object; a sensor-type motion capture module that senses the motion of the object; And it may include an integration module for integrating the information estimated from the optical motion capture module and the information sensed from the sensor-type motion capture module.

In the matching step, origins of the correction plate of the optical motion capture module and the marker of the second sensor system may be matched.

External variables of the second sensor system may include position and attitude information between the marker and the second sensor system.

The capturing of the 3D image and the capturing of the 2D image may be performed simultaneously.

A motion data annotation system for artificial intelligence (AI) learning according to an embodiment of the present invention includes a first sensor system for capturing a 3D image including joint position information of a skeleton, a second sensor system for capturing a 2D image, and The coordinate system of the first sensor system and the coordinate system of the second sensor system are matched, the joint position information of the skeleton is projected on a virtual camera to which the internal and external variables of the second sensor system are applied, and projected on the virtual camera. and an annotation device that automatically annotates joints in the 2D image captured through the second sensor system using the joint position information of the skeleton and the 2D image captured through the second sensor system.

According to an embodiment of the present invention, motion data for AI (Artificial Intelligence) learning can be quickly and accurately obtained using a 3D image as well as a 2D image.

According to an embodiment of the present invention, it is possible to effectively annotate body parts hidden by accessories or clothes in a 2D image.

Various advantageous advantages and effects of the present invention are not limited to the above description, and will be more easily understood in the process of describing specific embodiments of the present invention.

1-2 are examples of general annotation tools.
3 is a schematic diagram of a system for annotating motion data for AI learning according to an embodiment of the present invention.
4 is a flowchart of a method of annotating motion data for AI learning according to an embodiment of the present invention.
5 is a block diagram of a first sensor system according to an embodiment of the present invention.
6 is a flowchart of a motion capture method of the first sensor system according to an embodiment of the present invention.
7 is a block diagram of a system performing annotation of a human body pose through matching of heterogeneous sensors according to an embodiment of the present invention.
8 is an example of a method for calibrating the second sensor system.
9 is a diagram for explaining a method of matching a coordinate system of a first sensor system and a coordinate system of a second sensor system according to one embodiment of the present invention.
10 is an example of using the origin matching jig of FIG. 9(c) to match the coordinate system of the first sensor system and the coordinate system of the second sensor system.
11 is an image captured by the second sensor system.
12 is a diagram for explaining external variables between a marker and a second sensor system.
13 is an example in which the first sensor system 100 and the second sensor system 200 simultaneously photograph an object.
14 is an example of an image captured through the second sensor system 200 .
15 to 17 are diagrams for explaining a process of projecting joint position information of a skeleton included in a 3D image captured by the first sensor system to a virtual camera.
18 is skeleton data motion-captured through a virtual camera.
19 is position and posture information of individual joints of a human skeleton.
20 is an automatically annotated result according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated and described in the drawings. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms including ordinal numbers such as second and first may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a second element may be termed a first element, and similarly, a first element may be termed a second element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings, but the same or corresponding components regardless of reference numerals are given the same reference numerals, and overlapping descriptions thereof will be omitted.

3 is a schematic diagram of a system for annotating motion data for AI learning according to an embodiment of the present invention, FIG. 4 is a flowchart of a method for annotating motion data for AI learning according to an embodiment of the present invention, and FIG. 5 is a flowchart of a method for annotating motion data for AI learning according to an embodiment of the present invention. 6 is a flowchart of a motion capture method of the first sensor system according to an embodiment of the present invention, and FIG. 7 is a block diagram of a first sensor system according to an embodiment of the present invention. It is a block diagram of a system that performs annotation of human body poses through registration.

Referring to FIG. 3 , a system 10 for annotating motion data for AI learning includes a first sensor system 100 , a second sensor system 200 and an annotation device 300 . Here, the first sensor system 100 may be a hybrid motion capture system, and the second sensor system 200 may be a general photographing camera for photographing a live person. According to an embodiment of the present invention, the annotation device 300 uses the data obtained from the first sensor system 100 and the data obtained from the second sensor system 200 to take pictures by the second sensor system 200. Annotation is performed on the 2D image. That is, according to an embodiment of the present invention, the annotation device 300 uses the motion capture data obtained from the first sensor system 100 to capture a 2-dimensional image captured by the second sensor system 200. Annotate on

Referring to FIGS. 5 and 6 , the hybrid motion capture system 100 includes an optical motion capture module 110 , a sensor motion capture module 120 and an integrated module 1300 .

The optical motion capture module 110 includes at least two optical cameras. At least two optical cameras project the same point, and the three-dimensional coordinates of the object can be inverted through triangulation. Here, the object means an object of motion capture, and may be referred to as a rigid body, a target object, or a human body.

In order to increase the accuracy of optical motion capture, the optical motion capture module 110 may further include at least one reflective marker, for example, three reflective markers, and the reflective markers may be attached to an object.

The reflective marker may be a passive marker, for example. A passive marker may reflect light of a specific wavelength, for example infrared light.

An optical camera outputs infrared light, receives infrared light reflected from a reflective marker attached to an object, tracks the position of the reflective marker, and accordingly, at least among the absolute position, posture, speed, and movement of the object in three dimensions. Estimate one (S600).

The sensor-type motion capture module 120 includes a plurality of sensors attached to an object. The plurality of sensors may include, for example, an acceleration sensor and a gyro sensor. According to this, the sensor-based motion capture module 120 may sense at least one of the position, posture, speed, and movement of the object (S610).

The integration module 130 integrates the information estimated from the optical motion capture module 110 and the information sensed from the sensor motion capture module 120 to capture motion of the object (S620). For example, the integration module 130 determines at least one of the position, posture, speed, and motion of an object sensed by the sensor-type motion capture module 120 according to the absolute coordinates of the object estimated by the optical motion capture module 110. It can be corrected using at least one of the detailed position, posture, speed, and movement of the . The integration module 130 may integrate information estimated from the optical motion capture module 110 and information sensed from the sensor motion capture module 120 using a known location estimation integration algorithm. For example, the integration module 130 fuses the error values of the information estimated from the optical motion capture module 110 and the information sensed from the sensor-type motion capture module 120 through an integrated Kalman filter to determine the integrated position of the object, It is possible to estimate the exact position of an object by calculating the velocity and posture estimation values. The integration module 130 may be a computing device that integrates information estimated from the optical motion capture module 110 and information sensed from the sensor motion capture module 120 . Accordingly, in this specification, the integration module 130 may be referred to as a motion capture computer.

Referring back to FIGS. 3 and 4 , the annotation device 300 matches the coordinate system of the first sensor system 100 and the coordinate system of the second sensor system 200 (S400). To this end, as shown in FIG. 7, the coordinate system of the first sensor system 100, which is a hybrid motion capture system, is set in advance (S700), and the coordinate system of the second sensor system 200, which is a general photographing camera for photographing a real person, is set in advance (S700). After internal variables are estimated (S702) and external variables of the second sensor system 200 are estimated (S704), the coordinate system of the first sensor system 100 and the coordinate system of the second sensor system 200 may be matched. Yes (S706).

More specifically, in the second sensor system 200, which is a general camera for capturing real-life people, internal variables of the second sensor system 200, for example, a lens, a distance between a lens and an image sensor, and a lens and an image sensor The degree of distortion may vary depending on the angle between the screens and the like, and a process of correcting the distortion is required. In order to correct the distortion of the second sensor system 200, a method using a chess board based on a pin-hole camera model may be used. The pinhole camera model is a commonly used camera model, and is a model in which a ray coming from an object is projected onto a normalized image plane having a focal length of 1 and converted into image coordinates. In general, radial distortion and tangential distortion can be considered. 8 is an example of a method for calibrating the second sensor system. Parameters of the camera model estimated through calibration of the second sensor system are referred to as camera internal variables, and a process of obtaining parameter values of camera internal variables is referred to as camera calibration. Camera calibration in the pinhole camera model can be modeled by Equation 1 below.

Here, (X, Y, Z) are three-dimensional coordinates on the world coordinate system, [R|t] is a rotation/movement transformation matrix for converting the world coordinate system to the camera coordinate system, and A is the internal variable camera matrix (intrinsic camera matrix).

9 is a diagram for explaining a method of matching a coordinate system of a first sensor system and a coordinate system of a second sensor system according to one embodiment of the present invention. The coordinate system of the first sensor system 100, which is a hybrid motion capture module, can be set through the correction plate of the optical position tracking device shown in FIG. The coordinate system of (100) can be set through the marker shown in FIG. 9(b). Here, the marker may be referred to as an AR marker or an active marker. Using the marker shown in FIG. 9( b ), it is possible to estimate the position and posture of the second sensor system 200 in a 3D space.

According to an embodiment of the present invention, in order to match the coordinate system of the first sensor system 100 and the coordinate system of the second sensor system 200, the correction for the first sensor system 100 shown in FIG. 9 (a) An origin matching jig incorporating a plate and a marker for the second sensor system 200 shown in FIG. 9(b) may be used. 9(c) is an example of an origin alignment jig in which a correction plate for the first sensor system 100 and a marker for the second sensor system 200 shown in FIG. 9(b) are integrated. If the origin matching jig shown in FIG. 9(c) is used, the origin point viewed by the first sensor system 100 and the origin point viewed by the second sensor system 200 can be matched. The coordinate system of the sensor system 100 and the coordinate system of the second sensor system 200 may be matched.

10 is an example of using the origin matching jig of FIG. 9(c) to match the coordinate system of the first sensor system and the coordinate system of the second sensor system, and FIG. 11 is an image captured by the second sensor system, 12 is a diagram for explaining external variables between a marker and a second sensor system.

Referring to FIGS. 10 to 12 , the origin matching jig 1000 is disposed together with the first sensor system 100 and the second sensor system 200 . At this time, the second sensor system 200 determines the location between the origin of the first sensor system 100 and the second sensor system 200 based on the marker of the origin matching jig 1000 through a pre-estimated internal variable. and estimation of an external variable that is rotation information.

As shown in FIG. 11, the second sensor system 200 captures an image in which the origin of the first sensor system 100 and the origin of the second sensor system 200 are matched. , origin matching between the first sensor system 100 and the second sensor system 200 can be confirmed.

Markers that can be confirmed in the origin matching jig 1000 are rectangles, and 2-dimensional coordinates of positions of four vertices can be estimated. Homography means a relationship between two planes in a two-dimensional projection geometry, and here means an external variable between the marker and the second sensor system 200.

Referring to FIG. 12 , it can be seen that a plane existing in the real world, that is, a marker 1234 that can be confirmed in the origin matching jig is projected through a camera and becomes a plane 1'2'3'4'. The relationship between the two planes can be explained through the camera projection matrix according to Equation 2.

Equation 2 is a pinhole camera model, and since the Z value is 0 at the coordinates of a point input in the real world, the value of the third column of the projection matrix becomes 0 by matrix multiplication. Therefore, by rearranging Equation 2, it can be expressed as a 3x3 matrix as in Equation 3 below, which is called homography.

At this time, homography becomes position and posture information between cameras from markers existing in the real world, and this can be referred to as a camera external variable.

3 and 4 again, after the annotation device 300 matches the coordinate system of the first sensor system 100 and the coordinate system of the second sensor system 200 (S400), the first sensor system 100 captures a 3D image including joint position information of the skeleton (S410), and the second sensor system 200 captures a 2D image (S420). FIG. 13 is an example in which the first sensor system 100 and the second sensor system 200 simultaneously capture an object, and FIG. 14 is an example of an image captured through the second sensor system 200 .

That is, as shown in step S708 of FIG. 7 and FIG. 13 , the first sensor system 100 may perform motion capture, and the second sensor system 200 may capture a real-life image. 13 and 14, when a person wearing the reflective marker included in the optical motion capture module 110 and the sensor included in the sensor type motion capture module 120 moves within the studio, the first sensor system 100 ) performs motion capture, and the second sensor system 200 may record a live action image.

At this time, the step of capturing a 3D image including the joint position information of the skeleton by the first sensor system 100 (S410) and the step of capturing a 2D image by the second sensor system 200 (S420) are performed simultaneously. It can be.

Next, the annotation device 300 projects the joint position information of the skeleton included in the 3D image captured by the first sensor system 100 on the virtual camera 2000 (S430). Here, the virtual camera 2000 may be a virtual camera to which internal and external variables of the second sensor system 200 are applied. That is, as shown in FIG. 7 , the annotation device 300 loads the joint position information of the skeleton included in the 3D image captured by the first sensor system 100 (S710), and the pre-estimated second The internal and external variables of the sensor system 200 are loaded (S712), and the coordinates of the virtual camera 2000 reflecting the internal and external variables of the second sensor system 200 estimated in advance are set (S714), The joint position information of the skeleton included in the 3D image captured by the first sensor system 100 is projected onto the virtual camera (S716). 15 to 17 are views for explaining a process of projecting joint position information of a skeleton included in a 3D image captured by a first sensor system to a virtual camera, and FIG. 18 is a skeleton motion-captured through a virtual camera. It is data. At this time, the virtual camera 2000 includes the degree of distortion and location information of the second sensor system 200 . 15 to 18, a skeleton located in a 3-dimensional space is projected on a virtual camera 2000 and shown as a 2-dimensional image. The joint position information of the skeleton stored in 3-dimensional coordinates is based on the pinhole camera formula. It is projected onto a dimensional image plane. The virtual camera 2000 projected with joint position information of the skeleton is a virtual camera expressed by simulating the second sensor system 200 of an actual captured image. That is, since the location of the image of the virtual camera 2000 and the location of the shooting camera in 3D space are the same, it can be seen that the location of the joints of the skeleton is projected at the same point as the location of the joints of the actor in the actual shooting image.

Accordingly, referring again to FIGS. 3 and 4 , the two-dimensional image captured by the second sensor system 200 is obtained by using the joint position information of the skeleton obtained by the first sensor system 100 and projected on the virtual camera. Joints in the image are automatically annotated (S440). That is, the joints in the 2D image captured through the second sensor system 200 are automatically annotated using the joint position information of the skeleton projected on the virtual camera and the 2D image captured through the second sensor system 200. do. As shown in FIG. 7 , the annotation device 300 loads the 2D image captured by the second sensor system 200 (S718), and transmits the joint position information of the skeleton projected as the 2D image to the virtual camera. The 2D image captured through the second sensor system 200 is overlapped with the 2D image captured through the second sensor system 200 (S720), and the joint position information of the skeleton projected as a 2D image on the virtual camera is captured through the second sensor system 200. Through the process of matching (S722) to, it is possible to perform human body pose annotation (S724). 19 is position and posture information of individual joints of a human skeleton, and FIG. 20 is an automatic annotation result according to an embodiment of the present invention. As shown in FIG. 19 , a human skeleton having a hierarchical structure stored through motion capture may have position and posture information of individual joints. As shown in FIG. 20, when the human skeleton structure is projected on the virtual camera, the 3D joint coordinates are converted into 2D image coordinates, which is a certain part in the 2D image captured through the second sensor system 200. The joints of can be matched at the same time.

According to this, human body joint annotation for AI learning can be automatically performed.

In particular, according to an embodiment of the present invention, since a video can be used for annotation instead of a general photo, human joint annotation for AI learning can be quickly performed.

In addition, according to an embodiment of the present invention, since 3D skeleton information of the hybrid motion capture module is used, it is possible to secure data on body parts covered by accessories or clothes in a 2D image, so effective annotation is possible.

In addition, according to an embodiment of the present invention, since the number of reflective markers attached to body parts is significantly reduced using the hybrid motion capture module, it is possible to secure data for learning AI wearing various clothes.

The term '~unit' used in this embodiment means software or a hardware component such as a field-programmable gate array (FPGA) or ASIC, and '~unit' performs certain roles. However, '~ part' is not limited to software or hardware. '~bu' may be configured to be in an addressable storage medium and may be configured to reproduce one or more processors. Therefore, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

Although the above has been described with reference to the embodiments, this is only an example and does not limit the present invention, and those skilled in the art to which the present invention belongs will not deviate from the essential characteristics of the present embodiment. It will be appreciated that various variations and applications are possible. For example, each component specifically shown in the embodiment can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present invention as defined in the appended claims.

Claims (10)

In the motion data annotation method for AI (Artificial Intelligence) learning,
Matching the coordinate system of the first sensor system and the coordinate system of the second sensor system;
Taking a three-dimensional image including joint position information of a skeleton using the first sensor system;
Taking a two-dimensional image through the second sensor system;
projecting the joint position information of the skeleton to a virtual camera to which internal and external variables of the second sensor system are applied; and
Automatically annotating joints in the two-dimensional image captured through the second sensor system using joint position information of the skeleton projected by the virtual camera and the two-dimensional image captured through the second sensor system. How to. According to claim 1,
In the projecting step,
The method of projecting the joint position information of the skeleton as a two-dimensional image on the virtual camera. According to claim 2,
In the annotating step,
Wherein the joint position information of the skeleton projected as a 2D image by the virtual camera overlaps the 2D image captured through the second sensor system. According to claim 2,
In the annotating step,
Wherein the joint position information of the skeleton projected on the virtual camera is converted into 2D image coordinates, and the 2D image coordinates are automatically matched with a 2D image captured through the second sensor system. According to claim 1,
estimating the internal parameters of the second sensor system prior to the matching step. According to claim 1,
The first sensor system,
An optical motion capture module that outputs infrared light and receives the infrared light reflected from the object to estimate the position of the object;
A sensor-type motion capture module for sensing the movement of the object, and
and an integration module integrating information estimated from the optical motion capture module and information sensed from the sensor motion capture module. According to claim 6,
In the matching step,
A method in which origins of the correction plate of the optical motion capture module and the marker of the second sensor system are matched. According to claim 7,
The external variable of the second sensor system includes position and attitude information between the marker and the second sensor system. According to claim 1,
The step of photographing the 3D image and the step of photographing the 2D image are performed simultaneously. A first sensor system for capturing a 3D image including joint position information of a skeleton;
A second sensor system for taking a two-dimensional image, and
The coordinate system of the first sensor system and the coordinate system of the second sensor system are matched, the joint position information of the skeleton is projected on a virtual camera to which the internal and external variables of the second sensor system are applied, and projected on the virtual camera. An annotation device that automatically annotates joints in a 2D image captured through the second sensor system using the joint position information of the skeleton and the 2D image captured through the second sensor system.
A motion data annotation system for AI (Artificial Intelligence) learning that includes.

Images