KR20220116940A - Learning data collection system and method - Google Patents

Abstract

The present invention relates to a system and method for collecting learning data which is a subject of learning in artificial intelligence. The learning data collection system according to the present invention comprises the steps of: generating a 3D modeling object that performs modeling on an object and collecting a plurality of first images each including the 3D modeling object having a different posture in a first space having a first reference coordinate system; collecting a second image of the object disposed in a second space using a camera disposed in the second space having a second reference coordinate system different from the first reference coordinate system; and generating learning data for the object using the degree of freedom information of the camera and the 3D modeling object included in the first images. Accordingly, it is possible to collect learning data closer to a real environment.

Description

LEARNING DATA COLLECTION SYSTEM AND METHOD

The present invention relates to a learning data collection system that is a subject of learning in artificial intelligence and a learning data collection method using the same.

The dictionary meaning of artificial intelligence is a technology that realizes human learning ability, reasoning ability, perceptual ability, and natural language understanding ability through computer programs. Such artificial intelligence made a leap forward due to deep learning, which added a neural network that mimics the human brain to machine learning.

Deep learning is a technology that allows a computer to judge and learn like a human, and to cluster or classify objects or data through this. It is actively used in the industrial field.

For example, in various industrial fields such as robot field, autonomous driving field, medical field, through a deep learning-based learning network (hereinafter referred to as “deep learning network”), learning is performed based on the data to be learned, By deriving meaningful learning results, it is usefully used in each industry field.

As an example, in the field of robots, in order to understand the work performed by the robot, it should be possible to accurately determine the situation around the robot or the work object disposed around the robot, and for this, deep learning-based image recognition technology (For example, robot vision technology) is being actively utilized.

On the other hand, in the field of artificial intelligence such as machine learning as well as deep learning, as learning is performed on a larger amount of data, the accuracy increases and it is possible to derive better quality results. Therefore, in the field of artificial intelligence, it is essential to collect data to be learned.

In particular, a deep learning network or a machine learning network based on image data may estimate the position or posture of an object (or object) corresponding to image data, and for this estimation, the degree of freedom of the object along with image data Information (position information and posture information) must be secured as learning data.

Conventionally, in order to collect image data and corresponding degree of freedom information as learning data, labeling is performed on the image data (eg, an operation for identifying a specific image object corresponding to an object in the image data), Since the manual mapping of the image object and the degree of freedom information had to be done, an enormous amount of labor was required to secure the training data.

For example, Korean Patent Registration No. 10-2010085 discloses a method and a device for generating a labeling image of a microstructure using a superpixel, which is only for simplifying labeling of a specific image object corresponding to an object, Manual work is still required for mapping specific image objects and degrees of freedom information.

Accordingly, there is an urgent need to improve a method for collecting learning data including degree of freedom information in an automated manner.

The present invention relates to a learning data collection system and method for collecting learning data for learning of an artificial intelligence network.

More specifically, the present invention relates to a learning data collection system and method for collecting learning data including degree of freedom information.

Furthermore, the present invention relates to a learning data collection system and method capable of automatically collecting learning data including degree of freedom information.

Furthermore, the present invention relates to a learning data collection system and method for collecting learning data for objects having various postures.

Furthermore, the present invention relates to a learning data collection system and method capable of minimizing the time and labor required to collect the learning data.

In order to solve the above problems, the learning data collection method according to the present invention generates a three-dimensional modeling object that has been modeled on an object, and has a different posture in a first space having a first reference coordinate system. Collecting a plurality of first images each including a dimensional modeling object, the object disposed in the second space using a camera disposed in a second space having a second reference coordinate system different from the first reference coordinate system It may include collecting a second image obtained by photographing and generating learning data for the object by using the degree of freedom information of the camera and the three-dimensional modeling object included in the plurality of first images. have.

Furthermore, the learning data collection system according to the present invention includes a modeling unit that generates a three-dimensional modeling object that has performed modeling on an object in a first space, and a camera disposed in a second space different from the first space. It may include a communication unit for receiving an image of , and a control unit for collecting a plurality of images each including the 3D modeling object having different postures in the first space having a first reference coordinate system.

Furthermore, the controller may generate learning data for the object by using the degree of freedom information of the camera and the 3D modeling object included in the plurality of images.

The control unit may control the first image for each of the three-dimensional modeling objects having different postures based on the relationship between the graphic object corresponding to the object in the first image and the three-dimensional modeling object included in the second image. Information on the degree of freedom of the camera disposed in one space may be extracted.

Furthermore, the method for extracting a degree of freedom posture according to the present invention comprises the steps of receiving a photographed image of an object from a camera, and searching for a specific reference image corresponding to the photographed image from a plurality of reference images included in a preset data set. and extracting a degree of freedom posture of the object corresponding to the captured image by using the degree of freedom information matched to the specific reference image, wherein the plurality of reference images have different postures with respect to the object Each of the three-dimensional modeling objects having

As described above, the learning data collection system and method according to the present invention generates a three-dimensional modeling object for an object, and the relationship between the three-dimensional modeling object generated from the graphic object corresponding to the object included in the photographed image. can be used to extract the degree of freedom information of the 3D modeling object. Through this, the present invention can generate learning data in which lighting, shadows, and the like in the real environment are reflected by reflecting the 3D modeling object in the image captured in the real environment. Consequently, according to the present invention, it is possible to collect learning data closer to the real environment.

1 is a conceptual diagram for explaining an example in which learning data collected according to the present invention is utilized.
2 is a conceptual diagram for explaining a method of generating a 3D modeling object.
3 is a conceptual diagram for explaining a learning data collection system according to the present invention.
4 is a flowchart illustrating a learning data collection method according to the present invention.
5, 6, 7, 8, 9 and 10 are conceptual diagrams for explaining a method of collecting learning data.
11 and 12 are conceptual diagrams for explaining a method of using the collected learning data.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components will be given the same reference numerals regardless of reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The present invention relates to a learning data collection system and method for collecting learning data for learning of an artificial intelligence network, and in particular, learning data capable of automatically collecting learning data including degrees of freedom information (or degrees of freedom posture). It is about a collection method and system.

As mentioned above, thanks to the development of artificial intelligence, image recognition technology is being used in various industrial fields. In particular, in the field of robots, based on artificial intelligence-based image recognition technology (eg, deep learning-based image recognition technology), the robot analyzes and understands the working environment to which the robot belongs, and based on this, the robot's target task is performing

For example, as shown in FIG. 1 , when the robot R is given a specific task (eg, dish-washing), the robot R or a camera disposed around the robot R (not shown) may take an image corresponding to the working environment of the robot R. And, the control unit of the robot R, based on the captured image, in order for the robot R to perform a specific task, how It is possible to make a determination as to whether to operate, and control the robot R to operate according to the determination.

In this case, the control unit of the robot R recognizes the object (A, or object, for example, the bowl (a1, a2)) to be the target of the operation in the captured image, and the object (A) By analyzing the position and posture (or pose, pose), the robot R must be controlled so that the robot R can perform a target operation on the object.

To this end, the control unit of the robot (R) must collect various information from the captured image, for example, i) the type of the object to be worked on, ii) the size of the object to be worked on, iii) The shape of the object to be worked on, iv) the position of the object to be worked on (for example, where the bowl a1 is placed in the sink, as shown in FIG. 1, etc.), v) the posture of the object to be worked (for example, as shown in FIG. 1, the posture in which the bowl a1 is placed on the sink (ex: whether it is inclined at an angle, etc.)), vi) to photograph the object The robot R can be precisely controlled by using a plurality of pieces of information among the camera posture information.

Here, the position and posture of the object to be the target of the work or the camera for photographing the object may be expressed as “degree of freedom”, “position of freedom” or “degree of freedom information”, and in this specification, for convenience of description, , “degree of freedom information” should be unified and named.

Meanwhile, the degree of freedom information may be understood as a concept including location information and posture information. This, degree of freedom information, position information (or three-dimensional position information) corresponding to the three-dimensional position (x, y, z) and three-dimensional posture (r (roll), θ (pitch), Φ (yaw)) The corresponding posture information (or 3D posture information) may be included.

On the other hand, in order for the robot R to accurately perform an operation on an object to be operated, it is very important to grasp the degree of freedom information.

For example, the control unit of the robot R controls the robot arms R1 and R2 at what angle and in what posture to hold the objects a1 and a2 to be operated. , because it is determined based on at least one of the posture and the position of the object (or the camera that captures the object) to be the target of the operation.

At this time, if the degree of freedom information of the object (or the camera that photographed the object) can be recognized only by recognizing the object (for example, a1, a2) that is the object of the operation from the photographed image, the accuracy of the operation as well as Rather, it is possible to secure the efficiency of the work.

To this end, an image (or a mask) of an object having a specific shape (or a specific posture) obtained from a photographed image is matched with the posture information of the object, and may be used as learning data.

On the other hand, the posture information about the object includes: i) the degree of freedom information of the object reference for what position or posture it has based on the reference coordinate system of the object when the object has a specific shape, and ii) when the object has a specific shape, the object The photographed camera may include at least one of camera-referenced degree of freedom information regarding which position or which posture the photographed camera has based on the reference coordinate system of the camera.

The degree of freedom information described in the present invention may be understood as a concept in which the degree of freedom information based on the object and the degree of freedom information based on the camera are mixed.

That is, the information on the degree of freedom of the object may be understood as information on the degree of freedom of the camera that has photographed the object (or is looking at the object). Conversely, it goes without saying that the degree of freedom information of the camera photographing the object may be understood as the degree of freedom information of the object.

This is because the reference coordinate system of the object and the reference coordinate system of the camera have a relative positional relationship with each other.

For example, when inverse transformation is performed on the degree of freedom information of the object with respect to the reference coordinate system of the object, information on the degree of freedom of the camera with respect to the reference coordinate system of the object may be obtained.

Conversely, when inverse transformation is performed on the camera's degree of freedom information with respect to the camera's reference coordinate system, information on the degree of freedom of the object with respect to the camera's reference coordinate system can be obtained.

Furthermore, when the relative positional relationship between the reference coordinate system of the object and the reference coordinate system of the camera is defined, information on the degree of freedom of the camera with respect to the reference coordinate system of the camera may be obtained from information on the degree of freedom of the object with respect to the reference coordinate system of the object.

Conversely, it goes without saying that information on the degree of freedom of the object with respect to the reference coordinate system of the object may be obtained from the information on the degree of freedom of the camera with respect to the reference coordinate system of the camera.

For example, when the relative positional relationship between the reference coordinate system of the object and the reference coordinate system of the camera is reflected with respect to the degree of freedom information of the object, the degree of freedom information of the camera may be obtained. Conversely, when the relative positional relationship between the camera reference coordinate system and the object reference coordinate system is reflected with respect to the camera degree of freedom information, the degree of freedom information of the object may be obtained.

Here, the relative positional relationship may mean a degree of rotation and translation of one reference coordinate system with respect to the other reference coordinate system.

On the other hand, in order for the robot R to perform an accurate task, an artificial intelligence algorithm (eg, a deep learning algorithm or a deep learning network) learned based on massive learning data is required. Accordingly, in the present invention, a method of collecting learning data will be described in more detail with the accompanying drawings. 2 is a conceptual diagram for explaining a method of generating a 3D modeling object, and FIG. 3 is a conceptual diagram for explaining a learning data collection system according to the present invention. Furthermore, FIG. 4 is a flowchart for explaining a method of collecting learning data according to the present invention, and FIGS. 5, 6, 7, 8, 9 and 10 are conceptual diagrams for explaining a method of collecting learning data. . Furthermore, FIGS. 11 and 12 are conceptual diagrams for explaining a method of using the collected learning data.

Prior to the description of the present invention, the "object" referred to in the present specification is not limited in its type, and may be interpreted as a wide variety of objects. The object has a specific shape that can be visually or physically distinguished, and may be understood to include not only the object (or object), but also the concept of a person or an animal.

As described above, in order to achieve higher performance of a robot or an autonomous vehicle, learning is performed based on as much learning data as possible. To this end, it is very important to secure learning data, and the present invention proposes a method of securing learning data by using a 3D modeling object.

As shown in FIG. 2 , in the present invention, a three-dimensional modeling object 610 that has been modeled on an object (eg, a cup shown in FIG. 2 ) may be generated and generated. The three-dimensional modeling object 610 may be modeled to have the same shape and size (or size ratio) as much as possible to the real object. A method of modeling the 3D modeling object 610 is very diverse, and may be generated through, for example, 3D CAD. The 3D modeling object 610 may correspond to a mesh model to which a texture is applied. The texture applied to the 3D modeling object 610 may be the same as or similar to that of the real object.

Furthermore, in the present invention, a plurality of first images 611 to 616 each including the three-dimensional modeling objects 610 having different postures in a first space having a first reference coordinate system W1 may be collected. can

Here, the first reference coordinate system W1 may mean a reference coordinate system of a virtual environment (modeling environment) including the 3D modeling object 610 .

Meanwhile, the generation of the modeling object and the collection of images may be performed by the modeling unit 131 or the control unit 130 to be considered in FIG. 3 , and for convenience of explanation, the control unit 130 will be unified and described.

As shown in FIG. 2 , the plurality of first images 611 to 616 may include 3D modeling objects 610 having different postures.

The three-dimensional modeling object 610 included in the plurality of first images (refer to 611 to 616) is positioned at a predetermined position and in a predetermined posture within the first reference coordinate system with respect to the first reference coordinate system W1. can do. These predetermined positions and predetermined postures may be information on degrees of freedom of each 3D modeling object 610 .

That is, the controller 130 may generate a plurality of first images 611 to 616 such that the three-dimensional modeling object 610 having different degrees of freedom information is included in the plurality of first images 611 to 616 . have. Furthermore, the controller 130 may generate a 3D point cloud for the 3D modeling object 610 included in the plurality of first images 611 to 616, and based on this, a 3D modeling object from each image Depth information of 610 may be secured.

Meanwhile, the plurality of first images 611 to 616 are images generated on the premise that the virtual camera 620 having the first camera reference coordinate system C1 photographed the 3D modeling object 610 . can be

That is, the controller 130 may generate a plurality of first images 611 to 616 on the assumption that the virtual camera 620 has captured the 3D modeling object 610 .

In this case, the plurality of first images 611 to 616 is a premise that i) the 3D modeling object 610 is fixed and the 3D modeling object 610 is photographed by the virtual camera 620 in various postures. or ii) the virtual camera 620 may be an image captured on the premise that the 3D modeling object 610 has moved in various postures in a fixed state.

As described above, in the present invention, the controller 130 itself generates a plurality of first images 611 to 616 using the three-dimensional modeling object 610, so that the plurality of first images 611 to 616 are A variety of information can be obtained.

Various types of information include i) degrees of freedom information based on the first reference coordinate system W1 of the 3D modeling object 610 included in each of the plurality of first images 611 to 616 , ii) the plurality of first images Degree of freedom information for the first camera coordinate system C1 of the camera 620 photographing (611 to 616), iii) relative positional relationship information between the first reference coordinate system W1 and the first camera coordinate system C1, iv ) information on the degree of freedom for the first reference coordinate system W1 of the camera 620 photographing the plurality of first images 611 to 616, v) 3D included in each of the plurality of first images 611 to 616 At least among the degree of freedom information for the first camera coordinate system C1 of the modeling object 610, vii) depth (depth) information of the 3D modeling object 610 included in each of the plurality of first images 611 to 616 may contain one.

Meanwhile, the various information listed above may be stored in the storage unit 120 (refer to FIG. 3 ). Furthermore, at least one of the above-listed various pieces of information and a first image corresponding thereto (or an image corresponding to the modeling object 610 included in the first image) may be matched and stored in the storage unit 120 .

That is, each of the plurality of first images 611 to 616 may be stored while being matched with information related to the posture of the 3D modeling object 610 included in each of the first images.

Next, the learning data collection system 100 according to the present invention will be looked at in more detail with reference to FIG. 3 . The learning data collection system 100 according to the present invention may include at least one of a communication unit 110 , a storage unit 120 , and a control unit 130 .

The communication unit 110 is a means for receiving the image photographed from the camera 200, and there is no particular limitation on the communication method.

The communication unit 110 may be configured to perform at least one of wired or wireless communication. The communication unit 110 may be configured to perform communication with various communication-enabled targets.

Meanwhile, the communication unit 110 may be configured to communicate with at least one external server. Here, the external server may include at least one of a cloud server and a database corresponding to at least a part of the storage unit 120 . Meanwhile, the external server may be configured to perform at least a part of the control unit 130 . That is, it is possible to perform data processing or data operation in an external server, and the present invention does not place any particular limitation on this method.

Meanwhile, the communication unit 110 may support various communication methods according to a communication standard of a communication target.

For example, the communication unit 110 may include a Wireless LAN (WLAN), a Wireless-Fidelity (Wi-Fi), a Wireless Fidelity (Wi-Fi) Direct, a Digital Living Network Alliance (DLNA), a Wireless Broadband (WiBro), and a WiMAX (Wireless Broadband). World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5th Generation Mobile Telecommunication (5G) , Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal) Serial Bus) technology may be used to perform communication.

Meanwhile, the camera 200 is a means for capturing an image, and may be included in the system 100 according to the present invention or may be provided separately. In the present invention, the camera 200 may also be referred to as an “image sensor”.

The camera 200 may be configured to photograph at least one of a static image and a dynamic image, and may be provided in singular or plural number.

The camera 200 may include a 3D depth camera or RGB-depth camera capable of acquiring depth information of an object (or a subject, or object, refer to reference numeral 300). have. When the camera 200 is a 3D depth camera, the depth value of each pixel constituting the photographed image may be known, and depth information of the object may be obtained through this.

As shown in FIG. 3 , the camera 200 may be configured to photograph the object 300 located in the second space 400 having the second reference coordinate system 300 . The camera 200 may mean a camera 200 existing in a real environment (real space).

Furthermore, the camera 200 may be configured to have a second camera coordinate system C2.

That is, in the present invention, the first space including the 3D modeling object (see FIG. 2, 610) has the first reference coordinate system W1, and is defined as photographing the 3D modeling object (see FIG. 2, 610). The virtual camera 620 may have a first camera coordinate system C1.

Furthermore, the camera 200, existing in the real environment (real space, 400), has a second camera coordinate system (C2), and this camera 200 is the first in the second space (real environment or real space, 400). 2 It may be made to photograph the object 300 placed in the reference coordinate system W2.

Meanwhile, the storage unit 120 may be configured to store various information according to the present invention. The types of the storage unit 120 may be very diverse, and at least some of them may mean an external server (at least one of a cloud server and a database (DB)). That is, it may be understood that a space in which information related to the storage unit 120 is stored is sufficient, and there is no restriction on the physical space.

In the storage unit 120, i) various information related to the three-dimensional modeling object discussed with reference to FIG. 2, ii) learning data collected by the data collection system according to the present invention, iii) an image captured by the camera 200 , iv) degree of freedom information related to at least one of an object and a camera related to a captured image, v) a first reference coordinate system (W1), a first camera coordinate system (C1), a second reference coordinate system (W2), and a second camera coordinate system At least one of information on a relative positional relationship between at least two of (C2) may be stored.

Next, the controller 130 may be configured to control the overall operation of the learning data collection system 100 related to the present invention. The controller 130 may include a processor capable of processing an artificial intelligence algorithm (or artificial intelligence processor).

The controller 130 may further include a modeling unit 131 for generating the 3D modeling object 610 shown in FIG. 2 and generating various first images related thereto.

Furthermore, the controller 130 may further include a learning unit 132 that performs learning based on the collected images and information on degrees of freedom. The learning unit 132 may have a neural network structure.

Meanwhile, the controller 130 may recognize and track the object 300 photographed by the camera 200 in the image photographed by the camera 200 based on the deep learning algorithm. This operation may also be called tracking.

Furthermore, the controller 130 may collect (or generate) various learning data by using the image (hereinafter, the second image) captured by the camera 200 and the first image of the 3D modeling object.

In the present invention, an image of a 3D modeling object is called a “first image”, and an image captured by the camera 200 existing in a real environment is called a “second image”.

Meanwhile, the control unit 130 captures the object 300 based on a marker board 410 included in the second image or the texture of the background excluding the graphic object corresponding to the object 300 in the second image. Information on degrees of freedom of one camera 200 may be extracted.

The controller 130 may extract information on the degree of freedom of the camera 200 based on a visual feature included in the second image.

The visual characteristic may be defined based on a marker board 410 included in the second image or a texture of a background excluding a graphic object corresponding to the object 300 in the second image.

On the other hand, the degree of freedom information of the camera 200 is position information (or three-dimensional position information (degree of freedom of translational motion) corresponding to the three-dimensional position (x, y, z) of the camera 200 photographing the object 300 . (corresponding to )) and three-dimensional posture information (or three-dimensional posture information (corresponding to the degree of freedom of rotational motion)) may include.

The degree of freedom information of the camera 200 includes the degree of freedom information with respect to the second reference coordinate system W2 on the second space 400 in which the object 300 is located and the second camera coordinate system C2 of the camera 200 . It may include at least one of degrees of freedom information.

Furthermore, it goes without saying that the controller 130 may acquire information on the degree of freedom of the object 300 corresponding to the second image. Based on the degree of freedom information of the camera 200 , the controller 130 may acquire information on the degree of freedom of the object 300 or obtain information on the degree of freedom of the object 300 from the second image.

The control unit 130 calculates the degree of freedom information of the object 300 from the degree of freedom information of the camera 200 based on the relative positional relationship between the second reference coordinate system W2 and the camera second camera coordinate system C2, Conversely, the degree of freedom information of the camera 200 may be calculated from the degree of freedom information of the object 300 .

This is because the second reference coordinate system W2 of the object 300 and the reference coordinate system of the camera have a relative positional relationship with each other.

For example, when inverse transformation is performed on the degree of freedom information of the object 300 with respect to the second reference coordinate system W2 of the object 300 , the camera 200 for the second reference coordinate system W2 of the object 300 . ) degrees of freedom information can be obtained.

As such, the controller 130 calculates and calculates various information based on the relative positional relationship between the second camera coordinate system C2 and the second reference coordinate system W2 included in the camera 200 and the object 300, respectively. and store such information in the storage unit 120 .

Various types of information include: i) degree of freedom information based on the second reference coordinate system W2 of the object 300 , ii) the second camera coordinate system of the camera 200 photographing the second image including the object 300 . (C2) information on the degree of freedom, iii) the relative positional relationship information between the second reference coordinate system (W2) and the second camera coordinate system (C2), iv) the second reference coordinate system of the camera 200 that captured the second image ( Information on degrees of freedom for W2), 2 v) information on degrees of freedom for the second camera coordinate system C2 of the object 300 corresponding to the second image, vii) a graphic corresponding to the object 300 in the second image It may include at least one of information about pixel coordinates of an object.

Meanwhile, in the present invention, the controller 130 controls i) a plurality of first images of the three-dimensional modeling object and a second image obtained by photographing the object 200, ii) the object 200 in the first and second images. degree of freedom information, iii) degree of freedom information of the camera (or virtual camera) in the first and second images, iv) a first reference coordinate system (W1), a first camera coordinate system (C1), and a second reference coordinate system (W2) , by using at least a part of the relative positional relationship between at least two of the second camera coordinate system C2, it is possible to collect or generate a vast amount of learning data.

The training data may include a mask (MASK) for each of the 3D modeling objects included in the plurality of first images, and degree of freedom information related to the posture of the 3D modeling object matched to the mask, respectively.

Here, the mask may be an image in which a 3D modeling object is synthesized (or projected) on the second image captured in the real space 400 or an image of the 3D modeling object itself.

On the other hand, the degree of freedom information related to the posture of the 3D modeling object matched to the mask is information on the degree of freedom of the object 300 or the camera 200 with respect to the second reference coordinate system W2, or the second camera coordinate system C2. It may be information on the degree of freedom of the object 300 or the camera 200 for . Here, the object 300 may be a real object for the 3D modeling object 600 shown in FIG. 2 .

In this way, the control unit 130 may obtain information on the degree of freedom of the object 300 or the camera 200 that can be utilized in the real space 400 by using a plurality of images including the 3D modeling object. On the other hand, since the amount of data that can be generated using the 3D modeling object is very large, from tens of thousands to tens of millions or more, according to the present invention, it is possible to generate a sufficient amount of learning data necessary for learning.

Hereinafter, based on the configuration of the learning data collection system according to the present invention as described above, a method for collecting learning data will be described in more detail.

First, according to the learning data collection method according to the present invention, a process of generating a 3D modeling object in which modeling is performed on the object may be performed ( S410 ).

2 , the control unit 130 performs modeling on an object (for example, a cup shown in FIG. 2 , refer to an object of reference numeral 300 in FIG. 3 ) 3D modeling An object 610 may be created and created.

The three-dimensional modeling object 610 may be modeled to have the same shape and size (or size ratio) as much as possible to the actual object 300 .

A method of modeling the 3D modeling object 610 is very diverse, and may be generated through, for example, 3D CAD. The 3D modeling object 610 may correspond to a mesh model to which a texture is applied. The texture applied to the 3D modeling object 610 may be the same as or similar to the actual object 300 .

At this time, the three-dimensional modeling object 610 is a specific posture (three-dimensional posture (r () (corresponding to roll), θ (pitch), Φ (yaw)) and a specific position (corresponding to a three-dimensional position (x, y, z)).

Here, the first reference coordinate system W1 may mean a reference coordinate system of a virtual environment (modeling environment) including the 3D modeling object 610 .

Next, in the present invention, a process of collecting a plurality of first images each including three-dimensional modeling objects 610 having different postures in a first space having a first reference coordinate system W1 may be performed ( S420).

As shown in FIG. 2 above, the controller 130 controls a plurality of first images ( 611-616) can be collected.

Here, the first reference coordinate system W1 may mean a reference coordinate system of a virtual environment (modeling environment) including the 3D modeling object 610 .

As shown in FIG. 2 , the plurality of first images 611 to 616 may include 3D modeling objects 610 having different postures.

The three-dimensional modeling object 610 included in the plurality of first images (refer to 611 to 616) is positioned at a predetermined position and in a predetermined posture within the first reference coordinate system with respect to the first reference coordinate system W1. can do. These predetermined positions and predetermined postures may be information on degrees of freedom of each 3D modeling object 610 .

That is, the controller 130 may generate a plurality of first images 611 to 616 such that the three-dimensional modeling object 610 having different degrees of freedom information is included in the plurality of first images 611 to 616 . have. Furthermore, the controller 130 may generate a 3D point cloud for the 3D modeling object 610 included in the plurality of first images 611 to 616, and based on this, a 3D modeling object from each image Depth information of 610 may be secured.

On the other hand, the plurality of first images 611 to 616 is based on the assumption that the virtual camera 620 (refer to FIG. 2 ) having the first camera reference coordinate system C1 captures the 3D modeling object 610 . It may be an image generated by .

That is, the controller 130 may generate a plurality of first images 611 to 616 on the assumption that the virtual camera 620 has captured the 3D modeling object 610 .

At this time, the plurality of first images 611 to 616 is a premise that i) the 3D modeling object 610 is fixed and the 3D modeling object 610 is photographed by the virtual camera 620 in various postures. or ii) the virtual camera 620 may be an image captured on the premise that the 3D modeling object 610 is moved in various postures in a fixed state.

As described above, in the present invention, the controller 130 itself generates a plurality of first images 611 to 616 using the three-dimensional modeling object 610, so that the plurality of first images 611 to 616 are A variety of information can be obtained.

As described above, various types of information include i) degrees of freedom information based on the first reference coordinate system W1 of the 3D modeling object 610 included in each of the plurality of first images 611 to 616, ii) Degree of freedom information on the first camera coordinate system C1 of the camera 620 that has captured the plurality of first images 611 to 616, iii) the relative relation between the first reference coordinate system W1 and the first camera coordinate system C1 positional relationship information, iv) degree of freedom information for the first reference coordinate system W1 of the camera 620 photographing the plurality of first images 611 to 616, v) each of the plurality of first images 611 to 616 Degree of freedom information on the first camera coordinate system C1 of the 3D modeling object 610 included in vii) the depth of the 3D modeling object 610 included in each of the plurality of first images 611 to 616 ( depth) information.

Meanwhile, the various information listed above may be stored in the storage unit 120 (refer to FIG. 3 ). Furthermore, in the storage 120 , at least one of the various information listed above and a first image corresponding thereto (or an image corresponding to the modeling object 610 included in the first image) may be matched and stored.

That is, each of the plurality of first images 611 to 616 may be stored while being matched with information related to the posture of the 3D modeling object 610 included in each of the first images.

Next, in the present invention, a process of collecting a second image obtained by photographing an object (or object) disposed in the second space using a camera disposed in a second space having a second reference coordinate system may be performed (S430) ).

On the other hand, it goes without saying that the process S430 may be performed before or simultaneously with the processes S410 and S420 described above.

More specifically, the control unit 130, as shown in (a) to (c) of Figure 5, with respect to the object 300 existing in the real environment (real space, 400), through the camera 200 may be photographed, and a second image obtained as a photographing result may be collected.

In this case, the second space corresponding to the real environment may have a second reference coordinate system W2 , and the camera 200 may have a second camera coordinate system C2 .

The camera 200 may be configured to photograph the object 300 placed in the second reference coordinate system W2 in the second space (real environment or real space, 400 ).

The controller 130 may change the three-dimensional posture and three-dimensional position of the camera 200 with respect to the object 300 as shown in (a), (b) and (c) of FIG. 5 , Through this, the plurality of second images may include graphic objects having different postures with respect to the object 300 .

Meanwhile, the control unit 130 captures the object 300 based on a marker board 410 included in the second image or the texture of the background excluding the graphic object corresponding to the object 300 in the second image. Information on degrees of freedom of one camera 200 may be extracted.

The visual characteristic may be defined based on a marker board 410 included in the second image or a texture of a background excluding a graphic object corresponding to the object 300 in the second image.

For example, looking at a method of extracting the degree of freedom information of the camera 200 through the marker board 410 , the control unit 130 is shown in (a), (b) and (c) of FIG. 5 . The degree of freedom information of the camera 200 can be extracted from the second images 601 , 602 , and 603 shown in (a), (b) and (c) of FIG. 6 taken in the second space 400 . have.

The control unit 130 captures the object 300 based on the arrangement position and the degree of rotation of the graphic objects 410a, 410b, and 410c corresponding to the marker board 410 on the second images 601, 602, and 603. Information on degrees of freedom of one camera 200 may be extracted. Furthermore, the controller 130 considers the arrangement position and the degree of rotation of the graphic objects 300a, 300b, and 300c corresponding to the object 300 on the second images 601, 602, 603, the camera 200 The degree of freedom information can be extracted.

On the other hand, the degree of freedom information of the camera 200 is position information (or three-dimensional position information (degree of freedom of translational motion) corresponding to the three-dimensional position (x, y, z) of the camera 200 photographing the object 300 . (corresponding to )) and three-dimensional posture information (or three-dimensional posture information (corresponding to the degree of freedom of rotational motion)) may include.

The degree of freedom information at this time may be degree of freedom information of the camera 200 based on the second reference coordinate system W2 or information on the degree of freedom of the camera 200 based on the second camera coordinate system C2. .

Since the controller 130 knows the relative positional relationship between the second reference coordinate system W2 and the second camera coordinate system C2 with respect to the real space 400 , from the second images 601 , 602 , and 603 , the second reference Even when the degree of freedom information of the camera 200 based on the coordinate system is extracted, the degree of freedom information of the camera 200 based on the second camera coordinate system C2 can be extracted.

Furthermore, it goes without saying that the controller 130 may acquire information on the degree of freedom of the object 300 corresponding to the second images 601 , 602 , and 603 . Based on the degree of freedom information of the camera 200 , the controller 130 may acquire information on the degree of freedom of the object 300 or obtain information on the degree of freedom of the object 300 from the second image.

The control unit 130 calculates the degree of freedom information of the object 300 from the degree of freedom information of the camera 200 based on the relative positional relationship between the second reference coordinate system W2 and the camera second camera coordinate system C2, Conversely, the degree of freedom information of the camera 200 may be calculated from the degree of freedom information of the object 300 .

This is because the second reference coordinate system W2 of the object 300 and the reference coordinate system of the camera have a relative positional relationship with each other.

For example, when inverse transformation is performed on the degree of freedom information of the object 300 with respect to the second reference coordinate system W2 of the object 300 , the camera 200 for the second reference coordinate system W2 of the object 300 . ) of the degree of freedom information can be obtained.

As such, the controller 130 calculates and calculates various information based on the relative positional relationship between the second camera coordinate system C2 and the second reference coordinate system W2 included in the camera 200 and the object 300, respectively. and store such information in the storage unit 120 .

Various types of information include: i) degree of freedom information based on the second reference coordinate system W2 of the object 300 , ii) the second camera coordinate system of the camera 200 photographing the second image including the object 300 . (C2) information on the degree of freedom, iii) the relative positional relationship information between the second reference coordinate system (W2) and the second camera coordinate system (C2), iv) the second reference coordinate system of the camera 200 that captured the second image ( Information on degrees of freedom for W2), 2 v) information on degrees of freedom for the second camera coordinate system C2 of the object 300 corresponding to the second image, vii) a graphic corresponding to the object 300 in the second image It may include at least one of information about pixel coordinates of an object.

On the other hand, when a plurality of first images and second images are obtained through steps S410 to S430 and degree of freedom information related to the object 300 and the camera 200 is obtained, in the present invention, the object (or The process of generating learning data for the object) may proceed (S440).

More specifically, the controller 130 may generate learning data for the target (or object) object by using the degree of freedom information of the camera 200 and the 3D modeling object included in the plurality of first images.

The training data may include a mask (MASK) for each of the 3D modeling objects included in the plurality of first images, and degree of freedom information related to the posture of the 3D modeling object matched to the mask, respectively. In this case, the degree of freedom information related to the posture of the three-dimensional modeling object 600 (see FIG. 2 ) is the degree of freedom information for the second space 400 , and includes a second reference coordinate system W2 or a second camera coordinate system C2 ). may be information about

))를 이용할 수 있다. 도 8에 도시된 “T”는 homogeneous transformation matrix를 의미할 수 있다.In the present invention, the controller 130 obtains information on the degree of freedom of the 3D modeling object 600 with respect to the second space 400 , the first reference coordinate system W1 of the first space (or modeling space, virtual space). ), the relative positional relationship 800 between the camera coordinate systems C2 of the camera 200 disposed in the second space 400 (real environment or real space) 800, see FIG. 8 (

)) can be used. “T” shown in FIG. 8 may mean a homogeneous transformation matrix.

To this end, as shown in FIG. 7 , the control unit 130 includes the plurality of first images (images for the 3D modeling object) into the second images (images captured by the camera 200 , 720 ). A first image ( 710) can be specified.

The controller 130 may specify the first image 710 and the second image 720 having the most similar postures to the object from among the plurality of first and second images stored in the storage unit 120 , respectively. can In this case, as shown in FIG. 7 , the controller 130 may specify the first image 710 and the second image 720 by performing local feature matching.

))를 추출하기 위하여, 특정된 제1 및 제2 영상(710, 720)을 이용할 수 있다.As such, when the first and second images 710 and 720 are specified, the controller 130 controls the second space 400 with respect to the first reference coordinate system W1 of the first space (or modeling space, virtual space). , a relative positional relationship 800 between the second camera coordinate system C2 of the camera 200 disposed in the real environment or real space (see FIG. 8 (

)), the specified first and second images 710 and 720 may be used.

))를 추출하기 위하여, 특정된 제1 영상(710)에 포함된 상기 3차원 모델링 객체(711)와 특정된 제2 영상(820)에서 대상물에 대응되는 그래픽 객체(300a) 간의 관계성을 이용할 수 있다.The control unit 130 controls the relative positional relationship 800, (

)), the relationship between the 3D modeling object 711 included in the specified first image 710 and the graphic object 300a corresponding to the object in the specified second image 820 is used. can

))는, 제1 공간의 제1 기준 좌표계(W1)가 카메라(200)의 제2 카메라 좌표계(C2)에 대하여 회전(rotation) 및 변환(translation)된 정도를 의미할 수 있다. 즉, 상대적인 위치 관계(800, (

))는, 제2 카메라 좌표계(C2)에 대한 제1 공간의 제1 기준 좌표계(W1)의 상대적인 위치 관계를 의미할 수 있다.On the other hand, the relative positional relationship (800, (

)) may mean a degree of rotation and translation of the first reference coordinate system W1 of the first space with respect to the second camera coordinate system C2 of the camera 200 . That is, the relative positional relationship (800, (

)) may mean a relative positional relationship of the first reference coordinate system W1 in the first space with respect to the second camera coordinate system C2 .

More specifically, the degree to which the second camera coordinate system C2 of the camera 200 is rotated and translated with respect to the first reference coordinate system W1 of the first space is determined by the specified first image ( It may be specified based on the relationship between the 3D modeling object 721 included in the 710 and the graphic object 300a corresponding to the object (or object) in the second image 720 .

As shown in FIGS. 7 and 8 , the control unit 130 controls the pixel coordinates (u, v (refer to reference numeral 810 in FIG. 8 )) of the graphic object 300a in the specified second image 720 and specific Using the three-dimensional coordinates (x, y, z (refer to reference numeral 820 of FIG. 8 )) of the three-dimensional modeling object 7110 in the first image 710 ), the second camera coordinate system C2 becomes the first A degree of rotation and translation with respect to the first reference coordinate system W1 of the space may be extracted.

This relationship is based on the fact that the first image 710 and the second image 720 are composed of a 3D modeling object 711 and a graphic object 300a that are most similar to each other with respect to a specific posture of an object.

))에 대응되는 매트릭스(행렬, 830)을 적용할 수 있다. 그리고, 제어부(130)는 iii)카메라(200)에 의해 촬영된 대상물에 해당하는 그래픽 객체(300a)의 픽셀 좌표(810)가 결과값으로 도출됨을 이용하여, iv)제1 공간의 제1 기준 좌표계(W1)에 대하여 제2 카메라 좌표계(C2)가 회전 및 변환된 정도를 정의한 행렬(매트릭스, 830)를 추출할 수 있다.As shown in (a) of FIG. 8 , the controller 130 puts the three-dimensional coordinates 820 of the three-dimensional modeling object 711 into the PnP equation corresponding to i) the three-dimensional modeling object 711 as an input. , ii) a relative positional relationship 800, (

)) corresponding to the matrix (matrix, 830) may be applied. Then, the control unit 130 iii) using the pixel coordinates 810 of the graphic object 300a corresponding to the object photographed by the camera 200 are derived as a result value, iv) the first standard of the first space A matrix (matrix) 830 defining the degree of rotation and transformation of the second camera coordinate system C2 with respect to the coordinate system W1 may be extracted.

))에 해당하는 매트릭스(행렬, 830)를 추출할 수 있다.At this time, the pixel coordinates (u, v (refer to reference numeral 810 in FIG. 8)) of the graphic object 300a in the specified second image 720 and the 3D modeling object in the specified first image 710 ( 711), since the three-dimensional coordinates (x, y, z (refer to reference numeral 820 in FIG. 8 )) correspond to values already known by the control unit 130 , the control unit 130 determines that the second camera coordinate system C2 is the second The relative positional relationship 800, ((()

)) corresponding to the matrix (matrix, 830) can be extracted.

Meanwhile, in this PnP equation, an intrinsic parameter 840 is applied, which may correspond to a camera-specific parameter indicating the characteristics of the camera 200 .

On the other hand, as shown in (a) of FIG. 8 , when describing the parameters for the PnP equation, “S” is a scale constant (ex: 1), “r” is a skew parameter (corresponding to a kind of distortion correction constant) , “t1”, “t2”, and “t3” may refer to matrices and vectors for rotation and transformation.

))가 도출되면, 제어부(130)는 이를 이용하여, 도 8의 (b)에 도시된 것과 같이, 제1 기준 좌표계(W1)와 제2 기준 좌표계(W2)간의 상대적인 위치 관계(801, (

))를 추출할 수 있다.On the other hand, as described above, through the PnP equation, the relative positional relationship 800 of the first reference coordinate system W1 with respect to the second camera coordinate system C2 (see FIG. 8 (

)) is derived, the control unit 130 uses it, as shown in (b) of FIG. 8 , the relative positional relationship 801, (

)) can be extracted.

The control unit 130 determines the degree to which the second reference coordinate system W2 is rotated and transformed with respect to the first reference coordinate system W1, or information about the rotation and transformation of the first reference coordinate system W1 with respect to the second reference coordinate system W2. can be extracted.

))추출할 수 있다. 제1 기준 좌표계(W1)에 대하여 제2 기준 좌표계(W2)가 변환된 정도(801, (

))는, “제1 기준 좌표계(W1)에 대한 제2 기준 좌표계(W2)의 상대적인 위치관계”라고도 설명될 수 있다.In (b) of FIG. 8 , the controller 130 controls the degree of transformation 801, (

)) can be extracted. The degree to which the second reference coordinate system W2 is transformed with respect to the first reference coordinate system W1 (801, (

)) may also be described as “relative positional relationship of the second reference coordinate system W2 with respect to the first reference coordinate system W1”.

))는, i)제1 기준 좌표계(W1)에 대한 제2 카메라 좌표계(C2) 간의 상대적인 위치 관계(802, (

))와 ii)제2 카메라 좌표계(C2)와 제2 기준 좌표계 간의 상대적인 위치 관계(803, (

))의 곱으로 구해질 수 있다.On the other hand, as shown in (b) of FIG. 8, the relative positional relationship 801, (

)) is i) a relative positional relationship 802, (

)) and ii) the relative positional relationship 803, (

)) can be obtained by multiplying

))는 위에서 도 8의 (a)와 함께 살펴본. 제2 카메라 좌표계(C2)에 대한 제1 기준 좌표계(W1)의 상대적인 위치 관계(800, 도 8 참조(

))의 역행렬(또는 역변환, (

), 804)을 통하여 얻어질 수 있다. 나아가, 제2 카메라 좌표계(C2)와 제2 기준 좌표계(W2) 간의 상대적인 위치 관계(803, (

))는, 저장부(120)에 기 확보된 정보에 해당할 수 있다.A relative positional relationship (802, () between the second camera coordinate system C2 with respect to the first reference coordinate system W1

)) looked at together with (a) of FIG. 8 above. A relative positional relationship 800 of the first reference coordinate system W1 with respect to the second camera coordinate system C2 (see FIG. 8 (

)) of the inverse matrix (or inverse transform, (

), 804). Furthermore, the relative positional relationship 803, (

)) may correspond to information previously secured in the storage unit 120 .

))를 얻을 수 있다.Therefore, the control unit 130 uses the above relationship, as shown in FIG. 8(b), the relative positional relationship 801, (

)) can be obtained.

) 를 얻었으면, 이러한 위치관계(801, 806)와 ii) 제2 기준 좌표계(W2)에 대한 제2 카메라 좌표계(C2)의 상대적인 위치 관계(807, (

))의 곱을 이용하여, 제1 기준 좌표계(W1)에 포함된 3차원 모델링 객체에 대한 제2 카메라 좌표계(C2)의 자유도 정보를 추출할 수 있다.In this way, the control unit 130 through the estimation of the relative positional relationship described above, i) the relative positional relationships 801, 806, (

), these positional relationships 801, 806 and ii) the relative positional relationships 807, (

)), information on the degree of freedom of the second camera coordinate system C2 for the 3D modeling object included in the first reference coordinate system W1 may be extracted.

))를 도출하기 위한 과정으로서, 도면부호 805에 따른 관계성은, 특정된 제1 및 제2 영상뿐만 아니라, 임의의 자세를 갖는 3차원 모델링 객체에 대한 임의의 제1 영상에도 적용될 수 있다.(a) and (b) of FIG. 8 show the relationship (or relative positional relationship, 805, (

))), the relationship indicated by reference numeral 805 may be applied not only to the specified first and second images, but also to any first image of a 3D modeling object having an arbitrary posture.

))은, 앞서 도 8의 (a) 및 (b)의 관계식을 통하여 도출된, i) 제1 기준 좌표계(W1)에 대한 제2 기준 좌표계(W2)의 상대적인 위치관계(801, (

), 806)와 ii) 제2 기준 좌표계(W2)에 대한 제2 카메라 좌표계(C2)의 상대적인 위치 관계(807)의 곱을 통하여, 도출될 수 있다.8(C), the relationship 805, (

)) is, i) the relative positional relationship 801, (

), 806) and ii) the second reference coordinate system W2 and the second camera coordinate system C2 are multiplied by the relative positional relationship 807 .

Meanwhile, the relative positional relationship described with reference numerals 800, 801, 802, 803, 804, 805, 806, and 807 described in the present invention may be understood to mean a homogeneous transformation matrix. The values of the elements constituting the homogeneous transformation matrix are the relative positional relationship (degree of rotation and transformation) between coordinate systems, information on the degree of freedom of the 3D modeling object included in the first image, and the object included in the second image. At least one of the degree of freedom information of the graphic object and the degree of freedom information of the camera 200 photographing the second image may be utilized.

))을 이용하여, 서로 다른 자세를 갖는 3차원 모델링 객체를 포함한 복수의 제1 영상에 적용하여, 복수의 제1 영상에 각각 포함된 3차원 모델링 객체에 대한 카메라(200)의 제2 카메라 좌표계(C2)의 자유도 정보를 추출할 수 있다.Meanwhile, the controller 130 controls the relationship (or relative positional relationship, 805, (

)), applied to a plurality of first images including three-dimensional modeling objects having different postures, and the second camera coordinate system of the camera 200 for three-dimensional modeling objects included in the plurality of first images, respectively. (C2) degree of freedom information can be extracted.

That is, the controller 130 may extract information on the degree of freedom of the camera 200 when the 3D modeling object is photographed through the camera 200 disposed in the real space (the second space).

In this case, the reference coordinate system of the extracted degree of freedom information may be for the second camera coordinate system C2 or for the second reference coordinate system W2 .

On the other hand, since the control unit 130 has information on the relative positional relationship with respect to the second reference coordinate system W2 and the second camera C2 coordinate system in the storage unit 120 , the freedom of the coordinate system of the necessary form in some cases. information can also be extracted.

Through this, when the 3D modeling object is placed in an actual space, the controller 130 may extract information on the degree of freedom for the camera or the 3D modeling object.

The degree of freedom information extracted in this way may be included in the learning data. As shown in FIG. 9 , the training data includes a mask corresponding to each of the 3D modeling objects 911', 921', and 931' included in the plurality of first images 911, 921, and 931, and a mask and The degree of freedom information related to the postures of the 3D modeling objects 911', 921', and 931' respectively matched to the masks 910', 920', and 930' may be included.

Here, the mask is an image in which three-dimensional modeling objects (910', 920', 930') are synthesized (or projected) on the second images 910, 920, and 930 captured in real space, or a three-dimensional (3D) image. It may be an image of the modeling object itself.

Furthermore, as shown in FIG. 10 , on the training data 1010, information on the mask and information on the degree of freedom related to the posture of the 3D modeling object matched to the mask may be included. On the other hand, the degree of freedom information related to the posture of the 3D modeling object matched to the mask is information on the degree of freedom of the object 300 or the camera 200 with respect to the second reference coordinate system W2, or the second camera coordinate system C2. It may be information on the degree of freedom of the object 300 or the camera 200 for .

The controller 130 may obtain information on the degree of freedom of the object 300 or the camera 200 that can be utilized in the real space 400 by using a plurality of images including the 3D modeling object. On the other hand, since the amount of data that can be generated using the 3D modeling object is very large, from tens of thousands to tens of millions or more, according to the present invention, it is possible to generate a sufficient amount of learning data necessary for learning.

Meanwhile, as shown in FIG. 11 , the learning data using the 3D modeling object discussed above may form a reference image data set 1110 .

The method 600 of utilizing the learning data by utilizing the 3D modeling object is replaced with the description of FIG. 2 .

In this data set 1110, i) an image (or modeling image, rendering image) of a three-dimensional modeling object, ii) a depth map of a three-dimensional modeling object, iii) a camera that renders a three-dimensional modeling object at least two of the degree of freedom information (which may be the degree of freedom information of the camera in the second reference coordinate system W2 or the camera degree of freedom information in the second camera coordinate system C2), iv) the intrinsic parameters of the camera Matching may exist. The data set 1110 may be obtained through the method described above.

The controller 130 may extract degree of freedom information on an object sensed by a camera (not shown) in a real environment by using this data set 1110 .

As shown in FIG. 12 , the controller 130 receives (S1210) a photographed image (or input image) of an object (eg, a cup CUP) from a camera (not shown), the input image ( A comparison between images of the graphic object 1211 ′ corresponding to the object included in 1211 ) and the 3D modeling object included in the data set 1110 may be performed.

The control unit 130, based on the global feature, a specific image ( 1221) can be searched (S1220). The controller 130 may search for the specific image 1221 based on a difference in orientation of a camera looking at an object corresponding to the graphic object 1211 ′ in the input image 1211 .

Then, the controller 130 performs local feature matching between the input image 1211 and the specific image 1221 ( S1230 ) to extract a matching point (matching point) between the input image 1211 and the specific image 1221 . can Matching points corresponding to each of the input image 1211 and the specific image 1221 may form a pair.

The controller 130 may perform RGB-based local feature matching, and may extract a matching point by using a descriptor of key points between the input image 1211 and the specific image 1221 .

On the other hand, the global feature matching and local feature matching discussed in Fig. 1 can be performed in the Dual Feature Network. The dual feature network may be a deep neural network that simultaneously extracts a global feature used for image retrieval and a local feature used for pose estimation of an object.

When the number of the above matching points exceeds (or satisfies) the threshold (reference value), the controller 130 uses the matching points to correspond to the graphic object 1211 ′ included in the input image 1211 . The posture of the object or the camera photographing it may be estimated (S1240).

On the other hand, if the number of the above matching points does not exceed the threshold (reference value), the control unit 130, the object corresponding to the graphic object 1211' included in the input image 1211 or the posture of the camera photographing the same. estimation may not be made.

In detail with respect to the self-estimation process, the controller 130 extracts the posture of the degree of freedom of the object based on the relationship between the graphic object 1211 ′ and the 3D modeling object 1221 ′ included in the input image 1211 . can do.

More specifically, the control unit 130 is included in the data set 1110 in the three-dimensional coordinate information 820 as seen in (a) of FIG. 8, and coordinate information corresponding to the matching point of the specific image 1221. By inputting and inputting the coordinate information corresponding to the matching point of the input image 1211 in the two-dimensional coordinate information (810, or pixel coordinates), the PnP equation (algorithm) is solved, and the object itself or the freedom of the camera to photograph the object information can also be estimated. The process of solving the PnP equation (algorithm) will be replaced with the description of FIG. 8 .

On the other hand, since the data set 1110 includes the degree of freedom information of the 3D modeling object included in the specific image 1221 , the controller 130 determines the relative positional relationship between the matching points discussed above with respect to the corresponding degree of freedom information. By reflecting, information on the degree of freedom of the object corresponding to the graphic object 1211 ′ included in the input image 1211 may be extracted.

In this case, the degree of freedom information may be information on the degree of freedom of the object itself or a camera photographing the object.

As described above, the learning data collection system and method according to the present invention generates a three-dimensional modeling object for an object, and the relationship between the three-dimensional modeling object generated from the graphic object corresponding to the object included in the photographed image. can be used to extract the degree of freedom information of the 3D modeling object.

Through this, the present invention can generate learning data in which lighting, shadows, and the like in the real environment are reflected by reflecting the 3D modeling object in the image captured in the real environment. Consequently, according to the present invention, it is possible to collect learning data closer to the real environment.

Meanwhile, the present invention described above may be implemented as a program storable in a computer-readable medium (or recording medium), which is executed by one or more processes in a computer.

Furthermore, the present invention as seen above can be implemented as computer-readable codes or instructions on a medium in which a program is recorded. That is, the present invention may be provided in the form of a program.

Meanwhile, the computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is this.

Furthermore, the computer-readable medium may be a server or a cloud storage that includes a storage and that an electronic device can access through communication. In this case, the computer may download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit, Central Processing Unit), and there is no particular limitation on the type thereof.

On the other hand, the above detailed description should not be construed as limiting in all respects, but should be considered as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (15)

generating a 3D modeling object on which modeling is performed on an object, and collecting a plurality of first images each including the 3D modeling object having different postures in a first space having a first reference coordinate system;
collecting a second image obtained by photographing the object disposed in the second space by using a camera disposed in a second space having a second reference coordinate system different from the first reference coordinate system; and
and generating learning data for the object by using the degree of freedom information of the camera and the 3D modeling object included in the plurality of first images. According to claim 1,
The learning data is
A method for collecting learning data, comprising: a mask (MASK) for each of the three-dimensional modeling objects included in the plurality of first images; . 3. The method of claim 2,
The mask is an image obtained by combining the 3D modeling object included in the plurality of first images with the second image. 3. The method of claim 2,
The degree of freedom information related to the posture of the 3D modeling object matched to the mask,
Learning data collection method, characterized in that the degree of freedom information for the second space. 5. The method of claim 4,
The degree of freedom information related to the posture of the 3D modeling object matched to the mask,
degree of freedom information of a three-dimensional modeling object matched to the mask in the second reference coordinate system, and
Learning data collection method comprising at least one of information on the degree of freedom of the camera corresponding to the posture of the 3D modeling object matched to the mask, respectively. 5. The method of claim 4,
In the step of generating the learning data,
Using a relative positional relationship between the camera coordinate system of the camera disposed in the second space with respect to the first reference coordinate system in the first space,
Learning data collection method, characterized in that for extracting the degree of freedom information related to the posture of the 3D modeling object each matched to the mask in the second space. 7. The method of claim 6,
The relative positional relationship is
The method for collecting learning data, comprising specifying the camera coordinate system of the camera based on the degree of rotation and translation with respect to the reference coordinate system of the first space. 8. The method of claim 7,
The degree of rotation and translation is,
Learning data collection method, characterized in that specified based on a relationship between the 3D modeling object included in the first image and the graphic object corresponding to the object in the second image. 9. The method of claim 8,
In the step of generating the learning data,
specifying, from the plurality of first images, a first image including a specific three-dimensional modeling object having a posture similar to the posture of the object corresponding to the graphic object included in the second image,
The learning data collection method, characterized in that extracting the relationship using the second image and the specified first image. 10. The method of claim 9,
The degree to which the camera is rotated and translated is,
Learning data collection method, characterized in that extracted by using the pixel coordinates of the graphic object in the second image and the three-dimensional coordinates of the specific three-dimensional modeling object in the specified first image. a modeling unit for generating a three-dimensional modeling object on which modeling is performed on an object in a first space;
a communication unit configured to receive an image of the object from a camera disposed in a second space different from the first space; and
A control unit for collecting a plurality of images each including the 3D modeling objects having different postures in the first space having a first reference coordinate system,
The control unit is
Learning data collection method, characterized in that by using the degree of freedom information of the camera and the three-dimensional modeling object included in the plurality of images to generate the learning data for the object. A computer program recorded on a computer readable recording medium for executing the method according to any one of claims 1 to 10. receiving a photographed image of an object from a camera;
searching for a specific reference image corresponding to the captured image from a plurality of reference images included in a preset data set; and
extracting the degree of freedom posture of the object corresponding to the captured image by using the degree of freedom information matched to the specific reference image,
The plurality of reference images,
A method of extracting a degree of freedom posture comprising each of three-dimensional modeling objects having different postures with respect to the object. 14. The method of claim 13,
In the step of searching for the specific reference image,
Through comparison between the graphic object corresponding to the object included in the captured image and the 3D modeling object included in each of the plurality of reference images,
Among the three-dimensional modeling objects included in each of the plurality of reference images, the method for extracting a degree of freedom posture, characterized in that the specific reference image including a specific three-dimensional modeling object having the most similar posture to the graphic object is retrieved. 15. The method of claim 14,
In the step of extracting the degree of freedom posture of the object,
The degree of freedom posture extraction method, characterized in that extracting the degree of freedom posture of the object corresponding to the captured image based on a relationship between matching points corresponding to each other between the graphic object and the specific 3D modeling object.

Images