KR102657338B1 - Learning data collection system and method - Google Patents

Abstract

In order to solve the problems described above, the learning data collection method according to the present invention includes the steps of acquiring a reference image of a target object and a marker located around the target object through a camera, and selecting the marker from the image. Detecting and specifying at least one marker reference point of the marker included in the image, specifying a marker coordinate system of the marker with respect to the camera coordinate system based on the marker reference point, and based on the origin of the marker coordinate system Projecting a grid area aligned to the image, specifying the target object in the grid area based on movement of the grid area, and using the specific result for the target object in the grid area, It may include collecting learning data about the target object included in the image.

Description

Learning data collection system and method {LEARNING DATA COLLECTION SYSTEM AND METHOD}

The present invention relates to a learning data collection system that is an object of learning in artificial intelligence and a learning data collection method using the same.

The dictionary meaning of artificial intelligence is a technology that realizes human learning ability, reasoning ability, perception ability, and natural language understanding ability through computer programs. This artificial intelligence has made rapid progress due to deep learning, which adds a neural network that mimics the human brain to machine learning.

Deep learning is a technology that enables computers to judge and learn like humans, and to cluster or classify objects or data. Recently, it has become possible to analyze not only text data but also video data, enabling a wide variety of applications. It is actively used in industrial fields.

For example, in various industrial fields such as robotics, autonomous driving, and medical fields, learning is performed based on learning target data through a deep learning-based learning network (hereinafter referred to as “deep learning network”). By deriving meaningful learning results, it is usefully utilized in each industry field.

As an example, in the field of robotics, in order to understand the tasks performed by the robot, it must be possible to accurately judge the situation around the robot or the work objects placed around the robot. To this end, deep learning-based image recognition technology (For example, robot vision technology) is being actively used.

Meanwhile, in artificial intelligence fields such as machine learning as well as deep learning, as learning is performed on a larger amount of data, accuracy increases and it is possible to produce better quality results. Therefore, in the field of artificial intelligence, it is essential to collect data that is the subject of learning.

In particular, a deep learning network or machine learning network based on image data can estimate the position or posture of an object (or object) corresponding to the image data, and for this estimation, the degree of freedom of the object is used along with the image data. Information (position information and posture information) must be secured as learning data.

Conventionally, in order to collect image data and the corresponding degree-of-freedom information as learning data, manual work must be done to individually specify the areas containing the target object in the image data and extract the coordinates for the specified areas, so that the learning data A huge amount of labor was needed to secure it.

Accordingly, there is a great need for improvement in methods for collecting learning data including degree-of-freedom information.

The present invention relates to a learning data collection system and method for collecting learning data for learning of an artificial intelligence network.

More specifically, the present invention relates to a learning data collection system and method for collecting learning data including degree of freedom information.

Furthermore, the present invention relates to a learning data collection system and method for collecting learning data for target objects with various postures.

Furthermore, the present invention relates to a learning data collection system and method that can minimize the time and labor required to collect learning data.

In order to solve the problems described above, the learning data collection method according to the present invention includes the steps of acquiring a reference image of a target object and a marker located around the target object through a camera, and selecting the marker from the image. Detecting and specifying at least one marker reference point of the marker included in the image, specifying a marker coordinate system of the marker with respect to the camera coordinate system based on the marker reference point, and based on the origin of the marker coordinate system Projecting a grid area aligned to the image, specifying the target object in the grid area based on movement of the grid area, and using the specific result for the target object in the grid area, It may include collecting learning data about the target object included in the image.

Furthermore, the learning data collection system according to the present invention includes a camera for capturing a reference image of a target object and a marker located around the target object, a storage unit for storing the image, and detecting the marker from the image. , may include a control unit that specifies at least one marker reference point of the marker included in the image, and specifies a marker coordinate system of the marker with respect to the camera coordinate system based on the marker reference point.

Furthermore, the control unit projects a grid area aligned with the origin of the marker coordinate system onto the image, specifies the target object in the grid area based on movement of the grid area, and specifies the target object in the grid area. Using specific results about the target object, learning data about the target object included in the image can be collected.

Furthermore, the program, which is executed by one or more processes in the electronic device according to the present invention and stored in a computer-readable recording medium, uses a camera to measure the target object and the markers located around the target object. Acquiring an image, detecting the marker from the image, and specifying at least one marker reference point of the marker included in the image, based on the marker reference point, a marker coordinate system of the marker with respect to the camera coordinate system specifying, projecting a grid area aligned with the origin of the marker coordinate system onto the image, specifying the target object in the grid area based on movement of the grid area, and in the grid area It may include instructions for performing a step of collecting learning data for the target object included in the image using a specific result for the target object.

As seen above, the learning data collection system and method according to the present invention can secure learning data for the target object by using a grid area defined based on the marker in an image of a marker taken together with the target object. there is. At this time, in the present invention, a three-dimensional box area surrounding the target object can be defined and secured for the target object through horizontal rotation and vertical movement with respect to the grid area. Therefore, according to the present invention, in order to secure learning data for the target object, it is not necessary to individually select learning data extraction points of the target object from the image, thereby absolutely reducing labor and time required for securing learning data.

1 is a conceptual diagram illustrating an example in which learning data collected according to the present invention is utilized.
Figure 2 is a conceptual diagram for explaining the learning data collection system according to the present invention.
Figure 3 is a flow chart to explain the learning data collection method according to the present invention.
Figures 4, 5a, 5b, 6, 7, 8, 9, 10, and 11 are conceptual diagrams for explaining a method of collecting learning data.
Figure 12 is a conceptual diagram for explaining collected learning data.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings. However, identical or similar components will be assigned the same reference numbers regardless of drawing symbols, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

The present invention relates to a learning data collection system and method for collecting learning data for learning of an artificial intelligence network, and in particular, learning data that can automatically collect learning data including degree-of-freedom information (or degree-of-freedom posture). It is about collection methods and systems.

As seen above, thanks to the development of artificial intelligence, image recognition technology is being used in various industrial fields. In particular, in the robotics field, based on artificial intelligence-based image recognition technology (for example, deep learning-based image recognition technology), the work environment to which the robot belongs is analyzed and understood, and based on this, the robot's target task is determined. is carrying out.

For example, as shown in FIG. 1, when a specific task (or mission) (e.g., serving) is given to the robot R, the robot R or a camera placed around the robot R (not shown) shown) can capture images corresponding to the working environment of the robot (R). And, the control unit of the robot (R) makes a judgment on how the robot (R) should operate in order to perform a specific task, based on the captured image, and controls the robot (R) to operate according to the judgment. can do.

In this case, the control unit of the robot (R) recognizes the object (A, or object (object), for example, plate, cup (a1, a2)) that is the target of the work in the captured image, and recognizes the object (A) ), the position and posture (or pose) must be analyzed to control the robot (R) so that it can perform the target task on the object.

For this purpose, the control unit of the robot (R) must collect various information from the captured image, for example, i) the type of object that is the target of the work, ii) the size of the object that is the target of the work, iii) Shape of the object that is the target of the work, iv) Location of the object that is the target of the work (for example, as shown in Figure 1, the plate and cup (a1, a2) are placed somewhere on the table presence, etc.), v) the posture of the object that is the target of the work (e.g., as shown in Figure 1, the posture in which the plate and cup (a1, a2) are placed on the table (e.g., whether it is tilted at an angle, etc.) ), vi) The robot (R) can be accurately controlled using a plurality of pieces of information among the information about the posture of the camera that photographs the object.

Furthermore, the need to secure various information about an object can be useful not only for the robot work described above, but also for robot driving. In an environment where humans and robots coexist, in order for the robot to safely navigate the surrounding environment, it is necessary to accurately understand information about the environment located around the robot. At this time, various information about various objects located around the robot are needed. This is because, based on this, the robot's driving and movements can be appropriately controlled.

Furthermore, the need to secure various information about objects can also be useful in autonomous driving technology. For autonomous driving, it is necessary for the device performing autonomous driving (e.g., a car) to accurately recognize the surrounding environment. In this case, in order to respond appropriately to various objects located around the device, information on the various postures of the objects is required. need.

Meanwhile, the position and posture of the object or the camera that photographs the object may also be expressed as “degree of freedom,” “degree of freedom posture,” or “degree of freedom information,” and in this specification, for convenience of explanation, it is referred to as “degree of freedom information.” Let's name it in a unified way.

Degree of freedom information can be understood as a concept including position information and posture information. This degree of freedom information includes position information (or 3-dimensional position information) corresponding to the 3-dimensional position (x, y, z) and 3-dimensional posture (r (roll), θ (pitch), ? (yaw)). Corresponding posture information (or 3D posture information) may be included.

As seen above, it is very important to understand the degree of freedom information in order for the robot (R) or various devices to accurately perform work on the object that is the target of the work.

For example, the control unit of the robot (R) must decide at what angle to control the robot arms (R1, R2) and in what posture to grasp the objects (a1, a2) that are the targets of work. , This is because it is determined based on at least one of the posture and position of the object (or the camera that photographs the object) that is the target of the work.

At this time, if the degree of freedom information of the object (or the camera that photographed the object) can be recognized simply by recognizing the object (for example, a1, a2) that is the target of the work from the captured image, not only the accuracy of the work Rather, work efficiency can be secured.

To this end, the image (or video) of an object with a specific shape (or specific posture) obtained from a captured image and the posture information about the object can be matched with each other and used as learning data.

On the other hand, the posture information about the object includes i) when the object has a specific shape, the degree of freedom information based on the object about what position or posture it has based on the reference coordinate system of the object, and ii) when the object has a specific shape, the object Camera-based degree of freedom information about what position or posture the photographed camera has based on the camera's reference coordinate system, iii) pixels for at least one point in the area corresponding to the object in the image in which the object was photographed It may contain at least one of the coordinates.

As above, the degree of freedom information described in the present invention may include at least one of object-based degree of freedom information and camera-based degree of freedom information.

At this time, the degree of freedom information of the object may be expressed as at least one of the degree of freedom information of the object with respect to the object's reference coordinate system based on the object or the degree of freedom information of the camera that photographed the object (or viewed the object). At this time, the camera degree-of-freedom information may be degree-of-freedom information about the camera's reference coordinate system based on the camera.

This is because the reference coordinate system based on the object and the reference coordinate system based on the camera have a relative positional relationship with each other.

For example, when performing inverse transformation on the object's degree of freedom information with respect to the object-based reference coordinate system, the camera's degree-of-freedom information with respect to the object-based reference coordinate system may be obtained.

Conversely, when inverse transformation is performed on the camera's degree of freedom information with respect to the camera's reference coordinate system, information on the object's degree of freedom with respect to the camera's reference coordinate system can be obtained.

Furthermore, when the relative positional relationship between the reference coordinate system of the object and the reference coordinate system of the camera is defined, the degree of freedom information of the camera with respect to the reference coordinate system of the camera can be obtained from the degree of freedom information of the object with respect to the reference coordinate system of the object.

Conversely, of course, the degree of freedom information of the object with respect to the reference coordinate system of the object can be obtained from the camera's degree of freedom information with respect to the camera's reference coordinate system.

For example, with respect to the degree of freedom information of the object, if the relative positional relationship between the reference coordinate system of the object and the reference coordinate system of the camera is reflected, the degree of freedom information of the camera may be obtained. Conversely, when the camera's degree of freedom information reflects the relative positional relationship between the camera's reference coordinate system and the object's reference coordinate system, the object's degree of freedom information can be obtained.

Here, the relative positional relationship may mean the degree to which one reference coordinate system is rotated or translated with respect to another reference coordinate system.

Furthermore, in the present invention, as shown in FIG. 4, the object 420 can be photographed together with the marker 430 in order to collect degree of freedom information about the object. In this case, the marker 430 may have a reference coordinate system of the marker 430, and the degree of freedom information of the marker 430 is the relative relationship between the reference coordinate system of the camera and the reference coordinate system of the marker 430, as in the relationship discussed above. It can be obtained by reflecting the positional relationship.

Furthermore, of course, the degree of freedom information of the object can be obtained by reflecting the relative positional relationship between the reference coordinate system of the marker 430 and the reference coordinate system of the object.

That is, in the present invention, it is possible to collect degree-of-freedom information about at least one of the camera, the marker, and the object based on the relative positional relationship between the reference coordinate system of the camera, the reference coordinate system of the marker, and the reference coordinate system of the object.

Meanwhile, in order for a robot (R) or self-driving device to perform accurate tasks, an artificial intelligence algorithm (for example, a deep learning algorithm or deep learning network) learned based on massive learning data is required. Therefore, in the present invention, the method of collecting learning data will be examined in more detail with the attached drawings.

FIG. 2 is a conceptual diagram for explaining the learning data collection system according to the present invention, and FIG. 3 is a flowchart for explaining the learning data collection method according to the present invention. Furthermore, Figures 4, 5a, 5b, 6, 7, 8, 9, 10, and 11 are conceptual diagrams for explaining a method of collecting learning data, and Figure 12 shows the collected learning data. This is a concept diagram for explanation.

Prior to the description of the present invention, the “object” mentioned in this specification is not limited in its type and can be interpreted as a wide variety of objects. As shown in Figures 2 (a) and (b), the object has a specific form that can be visually or physically distinguished, and is not only an object (or object, 211, 213, 215, 217, 221), It can be understood to include the concept of people or animals.

As seen above, for higher performance of robots or autonomous vehicles, learning is performed based on as much learning data as possible. For this purpose, securing learning data is very important, and in the present invention, learning data for the object is secured based on a 3D box (or box area, or bounding box) of the object included in the image. Suggest a method. At this time, the 3D box may be in the shape of a cube or rectangular parallelepiped, and the size and shape of the 3D box may vary depending on the size and shape of the object included in the 3D box.

As shown in Figures 2 (a) and (b), in the present invention, in the image taken of the object, three-dimensional boxes (212, 214) surrounding each object (211, 213, 215, 217, 221) , 216, 218, 222), degree-of-freedom information related to at least one point of the three-dimensional box (212, 214, 216, 218, 222) can be collected as learning data. At this time, information about at least one point of the 3D box may be information about an apex, a vertex, angular point, a corner point where lines meet in the 3D box. That is, the degree of freedom information about the object may be the degree of freedom information about the vertices of a three-dimensional box surrounding the object.

Ewha Womans University, in the present invention, information about each vertex of a three-dimensional box (the vertices of a cube or rectangular parallelepiped (eight vertices)) can be collected as learning data for the object.

Accordingly, in the learning data collection system according to the present invention, the three-dimensional box surrounding the object included in the image is specified, and the degree of freedom information for a plurality of points included in the three-dimensional box is acquired, thereby providing learning data for the object. We propose a method to collect.

The learning data collection system 100 according to the present invention may include at least one of a communication unit 110, a storage unit 120, and a control unit 130.

The communication unit 110 is a means for receiving images captured by the camera 400, and there are no particular restrictions on the communication method.

The communication unit 110 may be configured to perform at least one of wired or wireless communication. The communication unit 110 may be configured to communicate with various objects capable of communication.

Meanwhile, the communication unit 110 may be configured to communicate with at least one external server. Here, the external server may include at least one of a cloud server or a database corresponding to at least a portion of the storage unit 120. Meanwhile, the external server may be configured to perform at least part of the role of the control unit 130. In other words, data processing or data computation can be performed on an external server, and the present invention does not place any special restrictions on this method.

Meanwhile, the communication unit 110 can support various communication methods depending on the communication standard of the communication target.

For example, the communication unit 110 supports wireless LAN (WLAN), wireless-fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, digital living network alliance (DLNA), wireless broadband (WiBro), and WiMAX ( World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5th Generation Mobile Telecommunication (5G) , Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal) Communication may be performed using at least one of the Serial Bus (Serial Bus) technologies.

Meanwhile, the camera 400 is a means for capturing images and may be included in the system 100 according to the present invention or may be provided separately. In the present invention, the camera 400 may also be referred to as an “image sensor.”

The camera 400 may be configured to capture at least one of static images and dynamic images, and may be provided in singular or plural forms.

The camera 400 may be configured as a 3D depth camera or an RGB-depth camera capable of acquiring depth information of an object (or subject, or object). If the camera 400 is configured as a 3D depth camera, the depth value of each pixel constituting the captured image can be known, and depth information of the object can be obtained through this.

This camera 400 may be configured to have a camera coordinate system {C}, as shown in FIG. 4.

Meanwhile, in the present invention, at least three reference coordinate systems may be defined, and the first reference coordinate system may be a camera coordinate system {C} set based on the camera 400.

The camera coordinate system {C} may be a three-dimensional coordinate system with a specific point of the camera 400 as the origin.

Furthermore, the second reference coordinate system may be a marker coordinate system {W} set based on the marker 430 included in the image 410 captured through the camera 400, as shown in FIG. 4.

The marker coordinate system {W} may be a three-dimensional coordinate system whose origin is a specific point of the marker 430 included in the image.

Furthermore, the third reference coordinate system may be an object (or object) coordinate system {O} set based on the object 420 included in the image 410 captured through the camera 400, as shown in FIG. there is.

The object coordinate system {O} may be a three-dimensional coordinate system whose origin is a specific point of the object 420 included in the image.

Meanwhile, in the present invention, the marker 430 and the object 420 included in the image captured by the camera may be the result of photography using markers and objects in the real world as subjects. That is, the marker 430 and the object 420 included in the image are graphic objects that respectively correspond to the marker and object included in the real world. In this specification, for convenience of explanation, the term for the graphic object is not separately defined, Graphic objects included in images obtained through filming of a marker are identically named “markers,” and graphic objects included in images obtained through filming of objects are identically named “objects.”

Meanwhile, the storage unit 120 can be configured to store various information according to the present invention. The types of storage unit 120 may be very diverse, and at least some of them may refer to external servers (at least one of a cloud server and a database (DB)). In other words, it can be understood that any space where information related to the storage unit 120 is stored is sufficient, and there are no restrictions on physical space.

The storage unit 120 includes at least one of i) learning data collected by the data collection system according to the present invention, ii) images captured through the camera 400, and iii) markers, objects, and cameras related to the captured images. Related degree of freedom information, iv) information about each of the first reference coordinate system (or camera coordinate system) {C}, the second reference coordinate system (or marker coordinate system) {W}, and the third reference coordinate system (or object coordinate system) {O}, v) At least one of information about the relative positional relationship between at least two first to third reference coordinate systems may be stored.

Next, the control unit 130 may be configured to control the overall operation of the learning data collection system 100 related to the present invention. The control unit 130 may include a processor (or artificial intelligence processor) capable of processing artificial intelligence algorithms.

The control unit 130 may use the captured image to create (or specify) a three-dimensional box for the object viewed in FIG. 2.

Furthermore, the control unit 130 may further include a learning unit that performs learning based on the collected images and degree-of-freedom information. This learning unit may have a neural network network structure.

Meanwhile, the control unit 130 can recognize and track the object 300 captured by the camera 400 in the image captured by the camera 400, based on a deep learning algorithm. This task may also be called tracking.

Furthermore, the learning data collection system according to the present invention can provide an interface that interacts with the user to specify (create) a 3D box included in the image.

This interface may be provided in the form of a program or application for collecting learning data.

In this case, in the present invention, collection of learning data about an object can be accomplished through a program that provides a process for setting a 3D box for a captured image.

Meanwhile, in order to collect learning data of the object, the control unit 130 includes camera parameters, a first reference coordinate system (or camera coordinate system) {C}, a second reference coordinate system (or marker coordinate system) {W}, and a third reference coordinate system ( Alternatively, an operation based on at least one of the object coordinate system) {O} may be performed. This operation is performed on at least one of camera parameters, a first reference coordinate system (or camera coordinate system) {C}, a second reference coordinate system (or marker coordinate system) {W}, and a third reference coordinate system (or object coordinate system) {O}. It may be calculating a PnP (Perspective-n-Point) equation using information (values).

Meanwhile, in the present invention, by placing a marker around the object that is the target of learning data collection and photographing the marker together with the object, the degree of freedom information about the object can be collected based on the degree of freedom information of the marker. In this section, we will look in more detail at the method of collecting learning data based on the configuration of the learning data collection system according to the present invention discussed above.

First, according to the learning data collection method according to the present invention, a process of acquiring images of the object and markers located around the object through a camera can be performed (S310, see FIG. 3).

As shown in FIG. 4, in the present invention, a marker 430 can be positioned around the object 420 for which learning data is to be collected. Here, the marker 430 may include at least one April tag, as shown. The April tag is a mark that serves as a visual standard and may be composed of two-dimensional information.

In the present invention, the relative position (distance apart, height) between the marker 430 and the object 420 is not particularly limited, and the object 420 and the object 420 are located in the field of view (FoV) of the camera 400. It is sufficient as long as the marker 430 is included. That is, in the present invention, as long as the object 420 and the marker 430 are included together in the image captured by the camera 400, there is no special limitation on the position where the marker 430 is placed.

Furthermore, the marker 430 may be located on the same plane as the plane on which the object 420 is placed, as shown in FIG. 4 . Meanwhile, of course, the marker 430 may be located on a different plane from the plane on which the object 420 is placed. In this case, the plane on which the marker 430 is placed and the plane on which the object 420 is placed may be planes of different heights.

In the present invention, images captured to include the object 420 and the marker 430 may be named by various terms.

For example, if there is no pre-secured learning data for the object 420, the image captured by the camera may be called a reference image (or reference image). This reference image may be an image from which the first learning data for the object 420 is collected.

Furthermore, in the present invention, a plurality of images can be captured after the reference image through the camera 400, and these images may be referred to as follow-up images. Follow-up images may refer to images captured to collect learning data about various postures of the object 420. The subsequent image may be captured to include the marker 430, or may be captured without the marker 430.

When the marker 430 is not included in the subsequent image, learning data for the object 420 in the subsequent image may be collected based on learning data for the object 420 collected in the reference image.

Next, in the present invention, a marker can be detected from the captured image, a process of specifying at least one marker reference point of the marker included in the image, and a process of specifying the marker coordinate system of the marker with respect to the camera coordinate system based on the marker reference point may be performed. There is (S320, S330).

The control unit 130 can detect the marker 430 included in the image 410, as shown in FIG. 4. In this case, the control unit 130 may specify at least one marker reference point included in the marker 430 based on a preset marker detection algorithm (or program). The marker reference point may mean at least one corner point corresponding to each corner included in the marker 430.

As in the enlarged portion 440 of FIG. 4, the control unit 130 recognizes the marker 430 from the captured image, and sets at least one marker reference point (or corner) corresponding to each corner of the recognized marker 430. Points (0, 1, 2, 3, 4, 5, 6, 7) can be specified (extracted).

And the control unit 130 may specify the marker coordinate system {W} of the marker 430, as shown in FIG. 5A, based on the specified marker reference point. The control unit 130 may measure marker coordinate system {W} degree-of-freedom information (or 6-degree-of-freedom posture (POSE)) with respect to the camera coordinate system {C}.

The marker coordinate system {W} may include a three-dimensional axis with any one of the specified marker reference points (e.g., 0, 1, 2, 3, 4, 5, 6, 7) as the origin, and is shown in FIGS. 4 and 4 As shown in 5a, the marker reference point corresponding to “0” can be used as the origin.

Meanwhile, the control unit 130 may extract degree-of-freedom information (six degrees-of-freedom posture) of the marker coordinate system {W} based on the camera coordinate system {C} of the camera 400. Through this, the control unit 130 extracts the degree to which the marker 430 or the marker coordinate system {W} included in the image 410 is rotated and translated with respect to the camera coordinate system {C}. can do.

The control unit 130 specifies the pixel coordinates corresponding to the marker reference point in the captured image 410, and uses the pixel coordinates corresponding to the marker reference point and the camera coordinate system {C} of the camera 400. Using this, the degree-of-freedom attitude of the marker coordinate system {W} with respect to the camera coordinate system {C} can be specified.

More specifically, the control unit 130 may calculate PnP, as shown in FIG. 5B, in order to extract degree of freedom information (or relative position relationship) of the marker coordinate system {W} with respect to the camera coordinate system {W}.

Meanwhile, the relative positional relationship of the marker coordinate system {W} with respect to the camera coordinate system {W} may mean the degree to which the marker coordinate system {W} is rotated and translated with respect to the camera coordinate system {W}. . Information according to this degree of rotation and transformation may correspond to degree-of-freedom information (degree-of-freedom posture) of the marker coordinate system {W} with respect to the camera coordinate system {C}.

In order to extract degree-of-freedom information (degree-of-freedom posture) of the marker coordinate system {W} with respect to the camera coordinate system {C}, the control unit 130 selects a marker ( Pixel coordinates (u, v (reference numeral 510 in FIG. 5B)) corresponding to at least one marker reference point 430) and three-dimensional coordinates (x, y, z (reference numeral 520 in FIG. 5B) of the camera 400) )) can be substituted into the PnP equation to extract degree-of-freedom information (degree-of-freedom posture) of the marker coordinate system {W} with respect to the camera coordinate system {C} (see reference numeral 530 in FIG. 5B).

Meanwhile, an intrinsic parameter 540 is applied to this PnP equation, which may correspond to a camera-specific parameter representing the characteristics of the camera 400. Camera-specific parameters may be stored in the storage unit 120.

Meanwhile, as shown in Figure 5b, when describing the parameters for the PnP equation, “S” is a scale constant (ex: 1), “r” is a skew parameter (corresponding to a type of distortion correction constant), and “t1” , “t2”, and “t3” may refer to matrices and vectors for rotation and transformation.

Since the control unit 130 knows in advance information about the camera coordinate system {C} of the camera 400 and information about the camera's unique parameters, it uses this to obtain information on the degree of freedom of the marker coordinate system {W} in the image 410. It can be extracted.

Next, in the present invention, a process of projecting the grid area aligned based on the origin of the marker coordinate system onto the image may be performed (S340).

As shown in FIG. 6, the grid area 610 can be understood as a plane including a grid. The control unit 130 may position the grid area 610 on the image 410. The process of projecting or positioning the grid area 610 of the image 410 can be accomplished through a program (software, application, etc.) operated by the learning data collection system according to the present invention.

As shown in FIG. 6 , the grid area 610 may be projected to be located on the floor corresponding to the bottom portion of the object 620 . The control unit 130 may set the initial position of the grid area 610 to be on the same plane as the bottom surface (floor plane) of the object 620.

As shown in FIG. 6 , the grid area 610 may include a plurality of square areas formed by vertically intersecting a plurality of horizontal lines 621 and a plurality of vertical lines 631.

The plurality of horizontal lines 621 of the grid area 610 are parallel to the first axis (e.g., X-axis) of the marker coordinate system {W}, and the plurality of vertical lines 631 of the grid area 610 are marker It may be parallel to the second axis (eg, Y axis) of the coordinate system {W}.

When projecting or positioning the grid area 610 on the image 410, the control unit 130 controls the arrangement position of the grid area 610 so that the grid area 610 is aligned with the marker coordinate system {W}. can be performed.

The grid area 610 may include a reference horizontal line 620, which is one of the plurality of horizontal lines 621, and a reference vertical line 630, which is one of the plurality of vertical lines 631. The reference horizontal line 620 and the reference vertical line 630 may perpendicularly intersect each other.

The alignment of the marker coordinate system {W} and the grid area 610 is such that the reference horizontal line 620 among the plurality of horizontal lines 621 and the reference vertical line 630 among the plurality of vertical lines 631 are aligned with the marker coordinate system {W} It can mean being sorted with respect to .

More specifically, as shown in FIG. 6, the marker coordinate system {W} includes a first axis (X axis) and a second axis (Y axis) orthogonal to each other, and a grid area for the marker coordinate system {W} ( The alignment of 610) matches the first axis (X-axis) of the marker coordinate system {W} with the reference horizontal line 620 of the grid area 610, and aligns the second axis (Y-axis) of the marker coordinate system {W} with the grid. This can be achieved by matching the reference vertical line 630 of the area 610.

In this case, the intersection point of the reference horizontal line 620 and the reference vertical line 630 may coincide with the origin (0, 0, 0) of the marker coordinate system {W}.

In this case, the angle formed between the first axis (X-axis) of the marker coordinate system {W} and the reference horizontal line 620 may be 0 degrees (see 640). Likewise, the angle formed between the second axis (Y-axis) and the reference straight line 630 may be 0 degrees.

In this way, when alignment is made between the marker coordinate system {W} and the grid area 610, even if the grid area 610 is moved, the specific point included in the grid area 610 is rotated or rotated relative to the marker coordinate system {W}. It is possible to determine whether the conversion has occurred, and ultimately, it is possible to determine the degree of freedom information (six degrees of freedom posture) of a specific point in the grid area 610 with respect to the camera coordinate system {C} or marker coordinate system {W}.

As seen above, when alignment is achieved between the marker coordinate system {W} and the grid area 610, the control unit 130 can then move the grid area 610 to specify the object 620. .

More specifically, in the present invention, the process of specifying a target object in the grid area based on the movement of the grid area and collecting learning data about the object included in the image using the specific results for the object in the grid area will be carried out. (S350, S360).

The control unit 130 horizontally rotates the grid area 610 based on the origin (0, 0, 0, see FIG. 6) of the marker coordinate system {W}, so that the object included in the grid area 610 and the image 410 Alignment between (420) can be performed.

As shown in FIG. 7, the control unit 130 horizontally rotates the grid area 610 by a predetermined angle θ based on the origin (0, 0, 0, see FIG. 6) of the marker coordinate system {W}, Alignment of the grid area 610 and the object 420 can be performed.

The control unit 130 may rotate the grid area 610 based on the origin of the marker coordinate system {W} until the grid area 610 and the object 420 are aligned. At this time, the predetermined angle θ may mean an angle at which the grid area 610 and the object 620 are aligned.

Meanwhile, the alignment between the grid area 610 and the object 420 is such that, as shown in FIG. 8, the reference horizontal line 620 or the reference vertical line 630 of the grid area 610 is horizontal or horizontal with the object 420. It can mean being vertical.

The reference horizontal line 620 or the reference vertical line 630 of the grid area 610 being horizontal or perpendicular to the object 420 means an area (or one side, the first side) in the image where the object 420 is placed. This may mean that at least one edge of and the reference horizontal line 620 or the reference vertical line 630 of the grid area 610 are horizontal or vertical.

At this time, the first surface on which the object 420 is placed can be defined in various ways, and as shown in FIG. 9, among the outermost points of the first surface (or one area, floor surface) on which the object 910 is placed It may be a plane 920 or 930 of a specific shape formed including at least one point. At this time, the specific shape may be square. At this time, the planes 920 and 930 of a specific shape include all of the area 910 occupied by the object, but may be specified so that the area is minimal. For example, the area of the plane 920 corresponding to (a) of FIG. 9 is smaller than the area of the plane 930 corresponding to (b) of FIG. 9, but is made to include all of the area 910 occupied by the object. It can be done.

Meanwhile, as shown in FIG. 8, the control unit 130 performs alignment of the first surface 810 (bottom surface or one surface) of the object 420 and the grid area 610, thereby dividing the area occupied by the object 420. At least one of the edges surrounding a square-shaped plane (one side, first side, or area 810) may cause the reference horizontal line 620 or the reference vertical line 630 of the grid area 610 to be horizontal or vertical. there is.

Furthermore, the specification of the first face 810 of the object is determined by rotating the grid area 610 so that it is horizontal (or perpendicular) to the object 420, the first face 810 on which the object 420 is located. This can be achieved by specifying at least one point (811, 812, 813, 814) in the border portion.

The at least one point (811, 812, 813, 814) may be named a keypoint. The control unit 130 may define the first side 810 of the object as a square shape and extract four key points 811, 812, 813, and 814 of the square shape corresponding to the first side 810. You can. At this time, the key points 811, 812, 813, and 814 may be specified by the control unit 130 or the user. The control unit 130 is also capable of automatically extracting at least three remaining key points when one key point is specified by the user. In this case, the control unit 130 uses the key points 811, 812, 813, and 814 to specify the first surface of the rectangular shape including the entire bottom surface of the object 420. ) can be specified.

Meanwhile, as shown in FIG. 8, in the present invention, an object coordinate system {O} with one key point 811 among the key points 811, 812, 813, and 814 as the origin can be specified.

The control unit 130 may extract coordinate information corresponding to a plurality of points (key points 811, 812, 813, and 814) corresponding to the first surface 810 of the object 420.

At this time, the coordinate information may correspond to three-dimensional coordinates. If necessary, the control unit 130 provides coordinate information in the object coordinate system {O}, coordinate information in the marker coordinate system {W}, and camera coordinate system {C) for a plurality of points (key points 811, 812, 813, 814). At least one of coordinate information in } and pixel coordinates in the image 410 can be extracted. This is because the object coordinate system {O}, marker coordinate system {W}, camera coordinate system {C}, and pixel coordinates all have a relative positional relationship, so if one coordinate information is known, coordinate information for the others can be extracted (or estimated). ) is possible.

The object coordinate system {O} is a reference coordinate system based on the object, and may be a coordinate system that is horizontally or vertically moved and rotated by a predetermined angle from the marker coordinate system {W}.

Meanwhile, the first surface 810 defined by the key points 811, 812, 813, and 814 may be the bottom surface of the three-dimensional box area including the object 420.

Next, the control unit 130 can specify the ceiling surface of the three-dimensional box area through vertical movement with respect to the grid area 610.

More specifically, as discussed above, the control unit 130 horizontally rotates the grid area 610 based on the marker coordinate system {W}, so that the grid area 610 and the object 420 are aligned. (610) can be moved vertically. As shown in (a), (b), and (c) of FIG. 10, the control unit 130 vertically moves the grid area 610 to create a first surface facing the first side 810 of the object 420. Two sides (830) can be specified.

As shown in (c) of FIG. 10, the control unit 130 controls the grid area from the first surface (bottom surface, 810) corresponding to the lowest part of the object 420 to the highest part of the object 420. By vertically moving 610, the second surface (ceiling surface, 820) can be specified.

As shown in Figures 10 (a) and (c) and Figure 11, the first surface 810 and the second surface 820 are a three-dimensional box (three-dimensional box area, 1110), the sides may be facing each other.

At this time, the areas of the first surface 810 and the second surface 820 may be the same. Furthermore, of course, in some cases, the areas of the first surface 810 and the second surface 820 may be different from each other.

In this way, when the second surface 820 is specified, the control unit 130 selects a plurality of points (key points, 815, 816) corresponding to the second surface 820 of the object 420, as shown in FIG. , 817, and 818), respectively, can be extracted.

At this time, the coordinate information may correspond to three-dimensional coordinates. If necessary, the control unit 130 provides coordinate information in the object coordinate system {O}, coordinate information in the marker coordinate system {W}, and camera coordinate system {C) for a plurality of points (key points 815, 816, 817, 818). At least one of coordinate information in } and pixel coordinates in the image 410 can be extracted. This is because the object coordinate system {O}, marker coordinate system {W}, camera coordinate system {C}, and pixel coordinates all have a relative positional relationship, so if one coordinate information is known, coordinate information for the others can be extracted (or estimated). ) is possible.

As such, in the present invention, on the grid area 610, key points 811, 812, 813, 814 of the first side 810 of the object 420 and key points 815, 816 of the second side 820 , 817, 818), learning data for the object 420 can be collected.

In this way, the control unit 130 specifies the key points 811, 812, 813, 814, 815, 816, 817, and 818 through the grid area 610 to control the first side 810 and the second side 820. ), and further by specifying the three-dimensional box 1110 defined by the first side 810 and the second side 820, learning data for the object 420 can be collected.

The learning data for the object 420 may be coordinate information of key points 811, 812, 813, 814, 815, 816, 817, and 818 of the three-dimensional box 1110.

The learning data is coordinate information (or three-dimensional coordinates) for the object coordinate system {O} corresponding to a plurality of points corresponding to the key points 811, 812, 813, and 814 of the first surface 810 of the object 420. information), coordinate information (or three-dimensional coordinate information) may be included.

Furthermore, the learning data includes pixel coordinate information of the image 410 corresponding to a plurality of points corresponding to key points 811, 812, 813, and 814 of the first side 810 and the second side 820. It may further include pixel coordinate information of the image 410 corresponding to a plurality of points corresponding to the key points 815, 816, 817, and 818.

Pixel coordinate information may be coordinate information based on the camera coordinate system {C}.

Meanwhile, the learning data may further include degree-of-freedom information (degree-of-freedom posture) of the object coordinate system {O} with respect to the camera coordinate system {C}.

Furthermore, as shown in FIG. 12 , learning data may be stored in the storage unit 120 by matching images of various postures of the object and information on the degree of freedom of the object for each of the various postures of the object.

At this time, the degree of freedom information may be information based on any one of the camera coordinate system {C}, marker coordinate system {W}, and object coordinate system {O}.

The control unit 130 is also capable of calculating degree of freedom information according to various postures of the object for each camera coordinate system {C}, marker coordinate system {W}, and object coordinate system {O}. In this case, for each various coordinate system, the data set This can be built. This is because the object coordinate system {O}, marker coordinate system {W}, camera coordinate system {C}, and pixel coordinates all have a relative positional relationship, so if the control unit 130 knows the degree of freedom information for any one coordinate system, the remaining It is possible to extract (or estimate) degree of freedom information about the coordinate system for .

Meanwhile, when the control unit 130 secures learning data for an object from an image in which a marker and an object are captured together (referred to as a “reference image”) using the method described above, the image captured after the learning data is obtained ( In (termed “follow-up video”), it is possible to collect learning data for objects included in the follow-up video, even if no marker is included.

That is, in the present invention, a follow-up image of an object can be captured through the camera 400, and learning data about the object included in the follow-up image can be obtained using learning data about the object obtained through the reference image. You can.

This is because the relative positional relationship between at least two of the camera coordinate system {C}, the object coordinate system {O}, and the marker coordinate system {W} is known, and through this, the control unit 130 operates in the grid area projected in the subsequent image using the method described above. It is possible to calculate coordinate information of key points corresponding to the first surface (floor surface) and the second surface (ceiling surface) of the object.

Furthermore, according to the present invention, it is also possible to secure learning data for objects with various postures through repetition of processes S310 to S360.

Meanwhile, after learning data for an object is secured in the manner described above, the control unit 13 can perform learning on a deep neural network using the learning data. A deep neural network can learn degree-of-freedom information about various postures for objects that are not included in the learning data, based on already secured learning data.

Once learning of this deep neural network is completed, robots or autonomous devices acquire images of the object through a camera and obtain degree-of-freedom information (six degrees-of-freedom posture) information about the object through the learned deep neural network. can do.

At this time, the degree-of-freedom information acquired by the robot or self-driving devices may be 6-degree-of-freedom posture information of the object coordinate system with respect to the camera coordinate system provided in the robot or self-driving devices.

Furthermore, robots or self-driving devices can obtain pixel coordinates of key points of a 3D box for an object included in an image captured by a robot or self-driving device through a learned deep neural network.

As seen above, the learning data collection system and method according to the present invention can secure learning data for the target object by using a grid area defined based on the marker in an image of a marker taken together with the target object. there is. At this time, in the present invention, a three-dimensional box area surrounding the target object can be defined and secured for the target object through horizontal rotation and vertical movement with respect to the grid area. Therefore, according to the present invention, in order to secure learning data for the target object, it is not necessary to individually select learning data extraction points of the target object from the image, thereby absolutely reducing labor and time required for securing learning data.

Meanwhile, the present invention discussed above can be implemented as a program that is executed by one or more processes on a computer and can be stored in a medium (or recording medium) that can be read by such a computer.

Furthermore, the present invention discussed above can be implemented as computer-readable codes or instructions on a program-recorded medium. That is, the present invention may be provided in the form of a program.

Meanwhile, computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is.

Furthermore, the computer-readable medium may be a server or cloud storage that includes storage and can be accessed by electronic devices through communication. In this case, the computer can download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit), and there is no particular limitation on its type.

Meanwhile, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (17)

In the learning data collection method using a learning data collection system,
Obtaining images of an object and a marker located around the object through a camera;
Detecting the marker from the image and specifying at least one marker reference point of the marker included in the image;
Specifying a marker coordinate system of the marker with respect to a camera coordinate system of the camera based on the marker reference point;
Projecting a grid area aligned based on the origin of the marker coordinate system onto the image;
specifying the object in the grid area based on movement of the grid area; and
Comprising: collecting learning data for the object included in the image using a specific result for the object in the grid area,
The specific result for the object includes information about a first side of the object and a second side facing the first side, which are specified based on horizontal rotation and vertical movement with respect to the grid area. How to collect training data. According to claim 1,
In the step of specifying the marker coordinate system,
In the image, specify pixel coordinates corresponding to the marker reference point,
A learning data collection method characterized by specifying the degree-of-freedom posture of the marker coordinate system with respect to the camera coordinate system using pixel coordinates corresponding to the marker reference point and the camera coordinate system of the camera. According to paragraph 2,
The grid area includes a plurality of square areas formed by vertically intersecting a plurality of horizontal lines and a plurality of vertical lines,
Alignment of the grid area with respect to the marker coordinate system is,
A learning data collection method, characterized in that the marker coordinate system, a reference horizontal line among the plurality of horizontal lines, and a reference vertical line among the plurality of vertical lines are aligned. According to paragraph 3,
The marker coordinate system includes a first axis and a second axis orthogonal to each other,
Alignment of the grid area with respect to the marker coordinate system is:
A learning data collection method characterized by matching the first axis of the marker coordinate system with the reference horizontal line, and matching the second axis of the marker coordinate system with the reference vertical line. According to paragraph 4,
The step of specifying the object in the grid area includes:
A learning data collection method comprising the step of horizontally rotating the grid area based on the origin of the marker coordinate system and performing alignment between the grid area and the object included in the image. According to clause 5,
Alignment between the grid area and the object is,
A learning data collection method, wherein the reference horizontal line or the reference vertical line in the grid area is horizontal or vertical to the object. According to clause 6,
A learning data collection method, characterized in that an object coordinate system for the object rotated by a specific angle according to the horizontal rotation with respect to the marker coordinate system is defined through alignment between the grid area and the object. In clause 7,
The step of specifying the object in the grid area includes:
A learning data collection method further comprising specifying the first side of the object while the grid area is aligned with the object through the horizontal rotation. According to clause 8,
In the step of specifying the first side of the object,
A learning data collection method characterized by extracting coordinate information about the object coordinate system corresponding to a plurality of points respectively corresponding to the first face of the object. According to clause 9,
The step of specifying the object in the grid area includes:
A learning data collection method further comprising specifying the second side of the object by vertically moving the grid region while the grid region is aligned with the object through the horizontal rotation. According to clause 10,
In the step of specifying the second side of the object,
A learning data collection method characterized by extracting coordinate information about the object coordinate system corresponding to a plurality of points respectively corresponding to the second surface of the object. According to clause 11,
In the step of specifying the object,
Learning data, wherein a three-dimensional box area surrounding the object is specified based on coordinate information extracted corresponding to the first side of the object and coordinate information extracted corresponding to the second side of the object. Collection method. According to clause 12,
The learning data is,
Coordinate information on the object coordinate system corresponding to a plurality of points on the first surface of the object, coordinate information on the object coordinate system corresponding to a plurality of points on the second surface of the object, and plural points on the first surface A learning data collection method comprising at least one of pixel coordinate information of the image corresponding to each and pixel coordinate information of the image corresponding to each of a plurality of points on the second surface. According to clause 13,
The coordinate information on the object coordinate system corresponding to a plurality of points on the first surface of the object and the coordinate information on the object coordinate system respectively corresponding to a plurality of points on the second surface of the object are three-dimensional coordinate information. How to collect learning data. According to clause 14,
Taking a follow-up image of the object through the camera; and
A learning data collection method further comprising acquiring learning data for the object included in the subsequent image using learning data for the object obtained through the image. A camera that captures a reference image of an object and a marker located around the object;
a storage unit that stores the image; and
Detecting the marker from the image and specifying at least one marker reference point of the marker included in the image,
A control unit that specifies a marker coordinate system of the marker relative to the camera coordinate system of the camera, based on the marker reference point,
The control unit,
Projecting a grid area aligned based on the origin of the marker coordinate system onto the image,
Based on the movement of the grid area, the object is specified in the grid area,
Collecting learning data for the object included in the image using specific results for the object in the grid area,
The specific result for the object includes information about a first side of the object and a second side facing the first side, which are specified based on horizontal rotation and vertical movement with respect to the grid area. Learning data collection system. A program that is executed by one or more processes in an electronic device and stored on a computer-readable recording medium,
The above program is,
Obtaining a reference image of an object and a marker located around the object through a camera;
Detecting the marker from the image and specifying at least one marker reference point of the marker included in the image;
Specifying a marker coordinate system of the marker with respect to a camera coordinate system of the camera based on the marker reference point;
Projecting a grid area aligned based on the origin of the marker coordinate system onto the image;
specifying the object in the grid area based on movement of the grid area; and
Includes instructions to perform a step of collecting learning data for the object included in the image using a specific result for the object in the grid area,
The specific result for the object includes information about a first face of the object and a second face facing the first face, which are specified based on horizontal rotation and vertical movement with respect to the grid area. A program stored on a recording medium that can be read.

Images