KR102439922B1 - Simultaneous real-time acquisition system of RGB image/joint position label data pairs using heterogeneous cameras - Google Patents

Abstract

The present invention relates to a system for simultaneous real-time acquisition of RGB image/joint position label data pairs using heterogeneous cameras,
IR marker; When acquiring RGB video using heterogeneous IR camera and RGB camera; joint position is also acquired in real time at the same time.
The present invention has a remarkable effect of extracting learning data for artificial neural networks of two-dimensional joint extraction based on RGB images with only one heterogeneous camera.

Description

Simultaneous real-time acquisition system of RGB image/joint position label data pairs using heterogeneous cameras

The present invention relates to a system for simultaneous real-time acquisition of RGB image/joint position label data pairs using heterogeneous cameras, and more specifically, with only one heterogeneous camera It relates to a system for simultaneous real-time acquisition of RGB image/joint position label data pairs using heterogeneous cameras that can be extracted.

In general, research to capture the motion and posture of a person using a general RGB camera rather than motion capture using an expensive motion capture device and IR camera is being actively conducted in recent years. In particular, many models have been proposed in which when a general RGB image is input to an artificial neural network using deep learning technology, an image showing human joint parts is output as output. However, when trying to learn such an artificial neural network, since human joints are not displayed in general RGB images that are intended to be used as training data, manual work is required to manually mark joint parts.

And as a prior patent technology, there is disclosed a method in Korean Patent Application Laid-Open No. 10-2016-0040230, comprising: capturing an RGB image of a person's hand;

A method is disclosed, comprising capturing a near infrared (NIR) image of the hand, and determining geometric characteristics of the hand based on the captured RGB and NIR images, the summary of which includes a human hand systems, including a method comprising capturing an RGB image of a hand, capturing a near infrared (NIR) image of a hand, and determining geometric characteristics of the hand based on the captured RGB and NIR images; Apparatus, devices, methods, computer program media and articles, and other implementations are disclosed. In some embodiments, determining the geometric features may include, for example, an angle between two fingers of a first pair in the extracted image data for the hand and/or a second pair in the extracted image data for the hand determining from the extracted image data for the hand one or more values indicative of spatial characteristics relative to the hand, such as one or more of a length or distance relative to the fingers of In some embodiments, it is stated that the method may further comprise determining the identity of the person based on the determined geometric characteristics of the hand.

In addition, Korean Patent Publication No. 10-1031848 discloses a digital camera system having an RGB correction function using the characteristics of a light source.

However, the conventional techniques still have the disadvantage that, when trying to learn such an artificial neural network, a manual operation of manually displaying the joint parts is required because the human joint parts are not displayed in a general RGB image that is intended to be used as learning data.

Therefore, the present invention has been devised to solve the above problems, and the present invention is an RGB image/joint using a heterogeneous camera that extracts learning data for artificial neural networks based on RGB image-based two-dimensional joint region extraction with only one heterogeneous camera. It is intended to provide a system for simultaneous real-time acquisition of location label data pairs.

The present invention relates to a system for simultaneous real-time acquisition of an RGB image/joint position label data pair using a heterogeneous camera, comprising: an IR marker; It is characterized in that the joint position is simultaneously acquired in real time when the RGB video is acquired using heterogeneous IR camera and RGB camera.

Therefore, the present invention has a remarkable effect of extracting learning data for artificial neural networks based on RGB image-based two-dimensional joint region extraction with only one heterogeneous camera.

1 is an IR, RGB image studio configuration diagram using a heterogeneous camera of the present invention;
2 is a block diagram of a method for simultaneous acquisition of RGB images/joint position labels for artificial neural network learning of the present invention

The present invention relates to a system for simultaneous real-time acquisition of RGB image/joint position label data pairs using heterogeneous cameras,

IR marker; It is characterized in that the joint position is simultaneously acquired in real time when the RGB video is acquired using heterogeneous IR camera and RGB camera.

In addition, it is characterized in that the heterogeneous camera box is configured so that the RGB camera and the IR camera, which are the heterogeneous cameras, can view the same scene.

In addition, it is characterized in that the RGB camera and the IR camera, which are the heterogeneous cameras, acquire the same frame in real time for the same scene.

The present invention will be described in detail with reference to the accompanying drawings. Fig. 1 is an IR, RGB image studio configuration diagram using a heterogeneous camera of the present invention, and Fig. 2 is a simultaneous RGB image/joint position label for artificial neural network learning of the present invention. This is the configuration diagram of the enemy acquisition method.

In order to avoid such manual work, the present invention proposes a method in which joint positions are simultaneously acquired in real time when acquiring RGB video using an IR marker, an IR camera, and an RGB camera.

As shown in FIG. 1, the system configuration of the present invention is as follows.

- Configure heterogeneous camera boxes so that RGB and IR cameras can see the same scene.

It composes a hardware system that synchronizes the acquisition process so that the RGB camera and IR camera, which are heterogeneous cameras, acquire the same frame in real time for the same scene and numbering is performed so that the same frame number is assigned to the same frame.

Build a database system that can store frames acquired from multiple types of heterogeneous cameras in a database in real time. Since it is acquired in real time, a mixed database of formal and non-formal forms is constructed.

From the IR frame images, the system is configured to recognize the position of the joint measured by the IR light as a true label. For this purpose, software dedicated to IR marker that recognizes only IR light is used.

If a studio based on multiple heterogeneous cameras using multiple heterogeneous cameras is configured, this can be used to acquire 3D information in the future.

The method of photographing the subject and extracting the RGB image and joint label is shown in FIG. 2 .

When shooting the subject, the subject has a marker that emits IR light attached to each joint. Therefore, both IR and RGB cameras are photographed with IR light (①).

Extract the IR marker from the acquired IR image and measure the true position of the joint (②). After extracting the IR marker based on color from the RGB camera (③,④), align it with the IR marker acquired through the IR camera (⑤,⑥) ,⑦) The joint positions of IR and RGB cameras are aligned by a multi-point based alignment algorithm.

If the position of the joint extracted from the RGB image is very different from the position of the joint extracted from the IR image when aligning by the multi-point-based alignment algorithm, it is assumed that the joint position extraction from the RGB image is wrong and the position of this joint is set to the IR Not used for alignment of images and RGB images. This is because there are many errors when extracting joints from RGB images with RGB color of light.

Since the IR marker acquired through the RGB camera can affect deep learning learning, the IR light is detected and the IR marker is removed from the RGB image through the Inpaining algorithm (⑧,⑨). At this time, using the information on the marker position of the IR image aligned with the RGB image (⑨), the area to be inpainted is made centered on the marker position.

The location of the extracted IR marker from the IR image aligned with the RGB image is saved as a true label (true label), and the inpainted RGB image is saved as image data. IR marker and RGB image are saved together for each frame, so that IR marker and RGB image become one data pair. A large-capacity data set is constructed by storing these data pairs in real time for each frame.

Therefore, the present invention has a remarkable effect of extracting learning data for artificial neural networks based on RGB image-based two-dimensional joint extraction with only one heterogeneous camera.

Claims (3)

IR marker; When acquiring RGB video using heterogeneous IR camera and RGB camera; joint position is also acquired in real time at the same time.
The heterogeneous RGB camera and the IR camera constitute a heterogeneous camera box to view the same scene, and the heterogeneous RGB camera and the IR camera acquire the same frame in real time for the same scene. In the simultaneous real-time acquisition system of RGB image/joint position label data pair using a camera,
The heterogeneous RGB camera and IR camera acquire the same frame in real time for the same scene, but configure a hardware system that synchronizes the acquisition process so that the same frame number is numbered so that the same frame number is assigned to the same frame. do,
We build a database system that can store frames acquired from multiple types of heterogeneous cameras in a database in real time.
From the IR frame images, the system is configured to recognize the position of the joint measured by the IR light as a true label, but using the IR marker software that recognizes only the IR light,
When shooting the target, since the target has a marker that emits IR light attached to each joint, the IR light is captured on both the IR and RGB cameras (①), and the IR marker is extracted from the obtained IR image to determine the true condition of the joint. After measuring the position (②) and extracting the IR marker based on the color from the RGB camera (③,④), align it with the IR marker acquired through the IR camera (⑤,⑥,⑦) to multiply the position of the joints of the IR and RGB camera. Sort by a point-based sorting algorithm,
If the position of the joint extracted from the RGB image is much different from the position of the joint extracted from the IR image when aligning by the multi-point-based alignment algorithm, it is assumed that the position of the joint in the RGB image is wrong and the position of this joint is It is not used to align IR and RGB images.
Since the IR marker acquired through the RGB camera can affect deep learning learning, it detects the IR light and removes the IR marker from the RGB image through the inpaining algorithm (⑧,⑨), at this time the RGB image. Using the information about the marker position of the IR image aligned with (⑨), the area to be inpainted is made centered on the marker position. defined as an area,
RGB image using a heterogeneous camera, characterized in that the position of the IR marker extracted from the IR image aligned with the RGB image is stored as a true label, and the inpained RGB image is saved as image data. /Simultaneous real-time acquisition system of joint position label data pairs delete delete

Images