KR102555667B1 - Learning data collection system and method - Google Patents

Abstract

The present invention relates to a learning data collection system and method for learning in artificial intelligence. The learning data collection system according to the present invention includes a sensing unit provided in each of a plurality of objects and including sensors for sensing degree-of-freedom information for each of the plurality of objects, a camera unit configured to photograph the plurality of objects, the Based on the image object corresponding to each of the plurality of objects included in the initial image received from the camera, a different label is set for each of the plurality of objects, and identification information of a sensor provided in each of the plurality of objects and a controller for one-to-one mapping the labels set for each of the plurality of objects.

Description

Learning data collection system and method {LEARNING DATA COLLECTION SYSTEM AND METHOD}

The present invention relates to a learning data collection system that is an object of learning in artificial intelligence and a learning data collection method using the same.

The dictionary definition of artificial intelligence is a technology that realizes human learning, reasoning, perception, and understanding of natural language through computer programs. Such artificial intelligence has made rapid progress due to deep learning, which adds a neural network that mimics the human brain to machine learning.

Deep learning is a technology that enables computers to judge and learn like humans, and clusters or classifies objects or data through this. It is actively used in industry.

For example, in various industrial fields such as robot field, autonomous driving field, and medical field, learning is performed based on learning target data through a deep learning-based learning network (hereinafter referred to as “deep learning network”), By deriving meaningful learning results, it is being used usefully in each industrial field.

As an example, in the field of robots, in order to understand the work performed by the robot, it is necessary to accurately determine the situation around the robot or the work object placed around the robot. For this purpose, deep learning-based image recognition technology (For example, robot vision technology) is being actively utilized.

On the other hand, in the field of artificial intelligence such as deep learning as well as machine learning, as learning is performed on a larger amount of data, accuracy increases and it is possible to derive higher quality results. Therefore, in the field of artificial intelligence, it is essential to collect data that is the subject of learning.

In particular, a deep learning network or machine learning network based on image data can estimate the position or posture of an object corresponding to the image data. and attitude information) must be secured as learning data.

Conventionally, in order to collect image data and degree-of-freedom information corresponding thereto as learning data, labeling is performed on the image data (for example, a task for identifying a specific image object corresponding to an object in the image data), and Since the manual work of mapping image objects and degree-of-freedom information must be done, a tremendous amount of labor was required to secure training data.

For example, Korean Patent Registration No. 10-2010085 discloses a method and apparatus for generating a labeling image of a microstructure using superpixels, which is only for simplifying labeling of a specific image object corresponding to an object, Manual work is still required for mapping specific image objects and degree-of-freedom information.

Accordingly, there is a great need for improvement of a method for automatically collecting learning data including information on the degree of freedom of an object.

The present invention relates to a learning data collection system and method for collecting learning data for learning of an artificial intelligence network.

More specifically, the present invention relates to a learning data collection system and method for collecting learning data including degree-of-freedom information.

Furthermore, the present invention relates to a learning data collection system and method capable of automatically collecting learning data including degree-of-freedom information.

Furthermore, the present invention relates to a learning data collecting system and method for collecting learning data for objects having various postures.

Furthermore, the present invention relates to a learning data collection system and method capable of minimizing the time and labor required to collect learning data.

In order to solve the above problems, the learning data collection method according to the present invention includes the steps of receiving, through a camera, initial images of a plurality of objects each equipped with a degree of freedom sensor, and the plurality of images included in the initial images. Setting a different label for each of the plurality of objects based on an image object corresponding to each of the objects, identification information of a degree of freedom sensor provided in each of the plurality of objects and a label set for each of the plurality of objects one-to-one mapping, tracking the plurality of objects from subsequent images received through the camera, and extracting segmentation masks corresponding to labels set for each of the plurality of objects, and for each of the plurality of objects The method may include collecting learning data including the segmentation mask and degree of freedom information sensed by the degree of freedom sensor based on the set label.

Furthermore, the learning data collection system according to the present invention includes a sensing unit provided in each of a plurality of objects and including sensors for sensing degree-of-freedom information for each of the plurality of objects, and a camera unit configured to photograph the plurality of objects. , Based on the image object corresponding to each of the plurality of objects included in the initial image received from the camera, different labels are set for each of the plurality of objects, and the sensor provided in each of the plurality of objects It may include a control unit for one-to-one mapping identification information and labels set on each of the plurality of objects.

The control unit tracks the plurality of objects from subsequent images received through the camera, extracts segmentation masks corresponding to labels set for each of the plurality of objects, and uses the labels set for each of the plurality of objects. As a reference, the segmentation mask and degree-of-freedom information collected from the sensors may be collected as training data.

Furthermore, a program that is executed by one or more processes in an electronic device according to the present invention and can be stored in a computer-readable recording medium is, through a camera, an initial image of a plurality of objects each equipped with a degree of freedom sensor. Receiving a label, setting a different label for each of the plurality of objects based on an image object corresponding to each of the plurality of objects included in the initial image, freedom provided for each of the plurality of objects One-to-one mapping of identification information of a degree sensor and labels set for each of the plurality of objects, tracking the plurality of objects from subsequent images received through the camera, and segmentation masks respectively corresponding to the labels set for each of the plurality of objects It may include instructions for performing the step of extracting a segmentation mask and the step of storing the segmentation mask and degree of freedom information collected from the degree of freedom sensor as training data based on the label set for each of the plurality of objects. there is.

As described above, in the learning data collection system and method according to the present invention, an object to be studied is provided with a sensor for collecting degree-of-freedom information, and image data about the object and degree-of-freedom information of the object are collected together. can do. In this case, in the present invention, segmentation may be performed on image objects corresponding to each object based on each object in the image data. Furthermore, in the present invention, by mapping mask labels corresponding to segmented masks and degree-of-freedom information for each object, it is possible to collect image data and degree-of-freedom information for an object at once just by capturing an image of the object. can Through this, according to the present invention, labor and time consumed by manually mapping image data and degree-of-freedom information corresponding to conventional objects can be absolutely reduced.

Furthermore, the learning data collection system and method according to the present invention can secure learning data for each of a plurality of objects for each image frame by placing a plurality of objects within the field of view of the camera. In this way, according to the present invention, it is possible to absolutely reduce the time required to collect learning data for various objects.

1 is a conceptual diagram for explaining an example in which learning data collected according to the present invention is utilized.
2 is a conceptual diagram for explaining a learning data collection system according to the present invention.
3 is a flowchart illustrating a learning data collection method according to the present invention.
4, 5a, 5b, 6, 7, 8 and 9 are conceptual diagrams for explaining a method of collecting learning data.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same reference numerals will be assigned to the same or similar components regardless of reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

The present invention relates to a learning data collection system and method for collecting learning data for learning of an artificial intelligence network, and particularly to a learning data collection method and system capable of automatically collecting learning data including degree-of-freedom information. will be.

As reviewed above, thanks to the development of artificial intelligence, image recognition technology is being used in various industries. In particular, in the field of robots, based on artificial intelligence-based image recognition technology (eg, deep learning-based image recognition technology), analyze and understand the work environment to which the robot belongs, and based on this, the robot's target task is performing

For example, as shown in FIG. 1 , when the robot R is given a specific task (eg, dish-washing), the robot R or a camera disposed around the robot R shown) may capture an image corresponding to the working environment of the robot R. And, based on the captured image, the control unit of the robot R, in order for the robot R to perform a specific task, how It is possible to make a judgment as to whether or not to operate, and control the robot R to operate according to the judgment.

In this case, the control unit of the robot R recognizes an object (A or object, for example, bowls a1 and a2), which is the target of the work, from the captured image, and By analyzing the position and posture (or pose), the robot (R) must be controlled so that the robot (R) can perform a target task with respect to the object.

To this end, the control unit of the robot R must collect various information from the captured image, for example, i) the type of object to be worked on, ii) the size of the object to be worked on, iii) The shape of the object to be worked on, iv) the location of the object to be worked on (for example, as shown in FIG. 1, where the bowl (a1) is placed in the sink), v) A plurality of pieces of information among information about the posture of the object to be worked on (for example, as shown in FIG. 1, the posture in which the bowl a1 is placed in the sink (ex: whether it is tilted at an angle, etc.)) By using, it is possible to accurately control the robot (R).

(yaw))에 해당하는 자세 정보(또는 3차원 자세 정보)를 포함할 수 있다.Here, the position and posture of the object to be worked may be expressed as “degree of freedom” or “degree of freedom information”, and degree of freedom information may be understood as a concept including position information and posture information. This degree-of-freedom information includes position information (or 3-dimensional position information) corresponding to a 3-dimensional position (x, y, z) and 3-dimensional posture (r (roll), θ (pitch),

(yaw)) may include attitude information (or 3D attitude information).

On the other hand, in order for the robot R to accurately perform work on an object to be worked on, it is very important to grasp information on degrees of freedom.

For example, the control unit of the robot R must determine at what angle to control the robot arms R1 and R2 and in what posture to grip them in order to grab the objects a1 and a2 that are the target of work. , This is because it is determined based on at least one of the posture and position of the object to be worked on.

At this time, if the degree of freedom information of the object can be recognized only by recognizing the object (eg, a1, a2) that is the target of the work from the captured image, it is possible to secure not only the accuracy of the work but also the efficiency of the work. can

To this end, learning data in which i) information on a specific shape of an object included in a photographed image and ii) information on a position or posture of an object in a specific shape are mutually matched may be used.

On the other hand, in order for the robot R to perform an accurate task, an artificial intelligence algorithm (eg, a deep learning algorithm or a deep learning network) learned based on vast amounts of learning data is required. Therefore, in the present invention, a method of collecting learning data will be looked at in more detail together with the accompanying drawings. 2 is a conceptual diagram for explaining a learning data collection system according to the present invention, and FIG. 3 is a flowchart for explaining a learning data collection method according to the present invention. Furthermore, FIGS. 4, 5a, 5b, 6, 7, 8, and 9 are conceptual diagrams for explaining a method of collecting learning data.

Prior to the description of the present invention, the “object” referred to in this specification is not limited in its type and can be interpreted as a very diverse object. An object has a specific shape that can be visually or physically distinguished, and can be understood to include not only objects but also concepts of people or animals.

As shown in FIG. 2, the learning data collection system 100 according to the present invention uses at least one of a camera 110, a sensing unit 120, a storage unit 130, a communication unit 140, and a control unit 150. can include

The camera 110 is a means for capturing images, and may be included in the system 100 according to the present invention or provided separately. In the present invention, the camera 110 may also be referred to as an “image sensor”.

The camera 110 may be configured to capture at least one of a static image and a dynamic image, and may be provided in singular or plural numbers.

The camera 110 may include a 3D depth camera or an RGB-depth camera capable of obtaining depth information of an object (or subject). When the camera 110 is formed of a 3D depth camera, a depth value of each pixel constituting a photographed image may be known, and through this, depth information of an object may be obtained.

Next, the sensing unit 120 may include a plurality of sensors 120a, 120b, and 120c. As shown in FIG. 2 , each of the plurality of sensors 120a, 120b, and 120c included in the sensing unit 120 may be provided in the objects 200, 201, 202, and 203, respectively.

In FIG. 2, for convenience of description, reference numerals are assigned only to the first, second, and third objects 201, 202, and 203, and the first, second, and third sensors 120a, 120b, 120c) is given reference numerals only. However, as shown in FIG. 2, the description according to the present invention relates to the objects shown in FIG. 2 (eg, the first, second, and third objects 201, 202, and 203 shown in FIG. 2) It will be apparent to those skilled in the art that the same applies to (including nine objects including).

On the other hand, the sensing unit 120 is provided in each object 200, it is possible to sense the degree of freedom information of the object (200).

As illustrated, the first object 201 is provided with a first sensor 120a, the second object 202 is provided with a second sensor 120b, and the third object 203 is provided with a third sensor 120c. ) may be provided. Furthermore, at least one sensor may be provided in the same manner for other objects to which reference numerals are not assigned.

On the other hand, in the present specification, it is described that one sensor (or degree of freedom sensor) is provided for each object, but the present invention is not limited thereto. That is, it is also possible that a plurality of sensors are provided in the object according to circumstances.

(yaw,요우))에 해당하는 자세 정보(또는 3차원 자세 정보(회전 운동의 자유도에 해당함))를 센싱할 수 있다.According to the present invention, through the sensing unit 120, the positional information corresponding to each of the three-dimensional positions (x, y, z) of the object 200 (or three-dimensional positional information (corresponding to the degree of freedom of translational motion)) and 3 Dimensional posture (r (roll), θ (pitch),

(yaw, yaw)) can sense posture information (or 3-dimensional posture information (corresponding to the degree of freedom of rotational motion)) corresponding to.

Here, the sensing unit 120 may also be referred to as a degree of freedom sensor or a degree of freedom (DoF) sensor. The sensing unit 120 may include an inertial measurement unit (IMU).

Next, the storage unit 130 may be configured to store various information according to the present invention. The types of the storage unit 130 may be very diverse, and at least some of them may mean an external server (at least one of a cloud server and a database (DB)). That is, it can be understood that a space in which information related to the storage unit 130 is stored is sufficient, and there is no restriction on physical space.

The storage unit 130 includes i) learning data collected by the data collection system according to the present invention, ii) an image captured through the camera 110, iii) a segmentation mask of an image object corresponding to an object included in the image, At least one of iv) a label (or label information) of the segmentation mask, and v) sensing information (eg, degree of freedom information) collected through the sensing unit 130 may be stored.

Next, the communication unit 140 may be configured to perform at least one of wired or wireless communication. The communication unit 140 may be configured to perform communication with various targets capable of communication. For example, the communication unit 140 may communicate with at least one of the camera 110 and the sensing unit 120 .

The communication unit 140 may receive an image photographed (or sensed) through the camera 110 through communication with the camera 110 .

Furthermore, the communication unit 140 may receive sensing information (eg, degree of freedom information) received through the sensing unit 120 through communication with the sensing unit 120 .

Furthermore, the communication unit 140 may communicate with at least one external server. Here, the external server, as described above, may include at least one of a cloud server or a database corresponding to at least a part of the configuration of the storage unit 130 . Meanwhile, an external server may be configured to perform at least a part of the role of the control unit 150. That is, data processing or data calculation can be performed in an external server, and the present invention does not place any particular restrictions on this method.

On the other hand, the communication unit 140 may support various communication methods according to the communication standard of the target to be communicated with.

For example, the communication unit 140 may include Wireless LAN (WLAN), Wireless-Fidelity (Wi-Fi), Wireless Fidelity (Wi-Fi) Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), WiMAX ( World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5th Generation Mobile Telecommunication (5G) , Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal) Serial Bus) technology may be used to perform communication.

Next, the control unit 150 may be configured to control the overall operation of the learning data collection system 100 related to the present invention. The control unit 150 may include a processor (or artificial intelligence processor) capable of processing an artificial intelligence algorithm. The controller 150 may recognize and track the object 200 photographed by the camera 110 in an image photographed by the camera 110 based on a deep learning algorithm. This task may also be referred to as tracking. When a plurality of objects 201, 202, and 203 are photographed through the camera 110, the controller 150 recognizes each of the plurality of objects from the image captured through the camera 110, and tracks each object. that is also possible

Therefore, while the controller 150 is photographing the plurality of objects 201, 202, and 203 through the camera 110, even if the position and posture of at least one of the plurality of objects 201, 202, and 203 is changed, , Each of a plurality of objects may be tracked in real time or at preset time intervals.

On the other hand, deep learning techniques for recognizing and tracking objects from images are very diverse, and the present invention does not place any particular limitations on them.

Hereinafter, based on the configuration of the learning data collection system according to the present invention described above, a method of collecting learning data will be described in more detail.

First, according to the learning data collection method according to the present invention, a process of receiving an initial image of an object each equipped with a degree of freedom sensor through the camera 110 may proceed (S310).

More specifically, in step S310 of obtaining an initial image, as shown in FIG. In the present invention, an object is an object to be photographed by the camera 110 and may also be referred to as a “subject”.

Meanwhile, in this specification, for description of the present invention, images captured by a camera are divided into “initial images” and “subsequent images”.

In the present invention, an “initial image” is an image used for labeling an object to be described later, and does not necessarily mean a first received image. That is, the initial image may be a target for setting (setting) for collecting learning data according to the present invention based on an image object included in the image (meaning an image corresponding to an object to be photographed). can

Furthermore, a “subsequent image” is an image to which learning data is directly collected, and may refer to an image received after an initial image is received. The controller 150 may collect learning data for an object by using an image object included in a subsequent image and a sensor (or a degree of freedom sensor) included in the object.

In the present invention, the initial image and the subsequent image may be at least one of a static image and a dynamic image. Furthermore, it is also possible that the initial image and the subsequent image are a plurality of images (or frames).

As shown in (b) of FIG. 4, when the initial image 400 of the object 200 is received through the camera 110, in the present invention, the initial image 400 of the plurality of objects 200 A process of performing labeling may proceed (S320).

In the process of labeling a plurality of objects 200 in the present invention, in subsequent images, i) settings for tracking (or tracking) each of the plurality of objects 200, ii) each of the plurality of objects 200 Settings for one-to-one mapping of the sensors provided in and unit areas for each of the plurality of objects 200, iii) settings for extracting a segmentation mask for each of the plurality of objects 200, iv) a plurality of A setting for one-to-one mapping of a sensor provided in each object 200 and a segmentation mask for each of the plurality of objects 200 may be made.

First, in the present invention, the setting for tracking each of the plurality of objects 200 may mean the setting for recognizing each of the plurality of objects 200 from a subsequent image even if the posture and position of the plurality of objects are changed. .

To this end, in the present invention, as shown in (b) of FIG. 4, based on a deep learning algorithm, in the initial image 400, the plurality of objects 200 can be tracked from the subsequent image. A unit area 200a for specifying an initial position of the image object 400 corresponding to each of the plurality of objects 200 may be designated.

In the present invention, the unit area 200a may also be referred to as a “box” or a “bounding box”.

As shown in (b) of FIG. 4 , the controller 150 controls the image object 400 so that the image object 400 corresponding to each of the plurality of objects 200 is included in the unit area 200a. The initial location can be specified.

That is, the unit area 200a may be specified for each image object 400 corresponding to each of the plurality of objects 200 in the initial image 400 .

The controller 150 may set an area where the image object 400 is located as the unit area 200a based on a user's or operator's selection. Furthermore, the controller 150 recognizes the image object 400 included in the initial image 400 based on the image recognition algorithm, and unit area 201a, 202a, 203a) can be set.

Therefore, each unit area 200a may include one image object corresponding to each unit area 200a among the image objects 400 corresponding to each of the plurality of objects 200 .

For example, as shown in (b) of FIG. 4 , the first unit area 201a includes the first image object 411 corresponding to the first object 201, and the second unit area 202a ) may include a second image object 412 corresponding to the second object 202 . Also, a third image object 413 corresponding to the third object 203 may be included in the third unit area 203a. In this case, each image object 410 does not have to be completely included in the unit area 200a, and it is also possible that each image object 410 is included at least partially.

Meanwhile, in the present invention, the shape of the unit area 200a is depicted as a rectangle, but there is no particular limitation thereto. For example, the unit area 200a may be set at an edge of the image object 410 along the shape of the image object 410 .

In this way, when the unit area 200a is set for each image object 410, different labels may be set for each unit area 200a.

Setting a label for each unit area 200a may be understood as having the same or similar meaning as setting a label for each of a plurality of objects 200 . This is because each unit area 200a is set based on each of the plurality of objects 200 .

As shown in FIG. 5A, different labels may be set in each unit area 200a based on a plurality of objects 200 (FIGS. 5A, 5B, 6, and 8 are for convenience of description). Labels (e.g., BOX_01, BOX_02, ID_01, ID_02) are shown. These illustrations are only for convenience of explanation, and it is known to those skilled in the art that the image displaying the above labels may not exist in the actually obtained data. self-explanatory).

As shown in FIG. 5A , the controller 150 may set different labels for the unit areas 200a each including the image objects 410 corresponding to the plurality of objects 200 . Here, the label can also be understood as “identifying information”. That is, the label is identification information for distinguishing each unit area, and different unit areas have different labels, that is, identification information.

The controller 150 may track the object 200 corresponding to the image object 410 included in each unit area 200a based on each unit area 200a. Therefore, in order to track each object 200, it is essential to distinguish each object 200, and the controller 150 recognizes the object to be tracked based on the unit area 200a and tracks it. can

Accordingly, as shown in FIG. 5A , a first label 501 may be set or specified in the unit area 201a including the first image object 411 corresponding to the first object. Similarly, different labels may be set or specified for different unit areas, respectively.

Meanwhile, as in the process of S320, when labeling is performed for each unit area 200a, in the present invention, the identification information of the degree of freedom sensor provided in each of a plurality of objects and the label set for each of the plurality of objects are mapped one-to-one. A process may proceed (S330).

That is, in the present invention, in the present invention, a process of one-to-one mapping between identification information of a degree of freedom sensor provided in each of a plurality of objects and a label set in a unit area may be performed.

This one-to-one mapping is performed based on a plurality of objects. That is, whether identification information of a certain degree of freedom sensor (or sensor) is to be mapped one-to-one to a certain unit area may be determined based on each of a plurality of objects.

For example, as shown in FIGS. 2, 4, and 5B, when the first object 201 includes the first degree of freedom sensor 120a, the first image corresponding to the first object 201 Identification information 502 (eg, “ID_01”) of the first DOF sensor 120a may be mapped to the first unit area 201a including the object 200a. Accordingly, the identification information 502 (eg, “ID_01”) of the first degree of freedom sensor 120a is included in the first label 501 (eg, “BOX_01”) corresponding to the first unit area 201a. can be mapped.

Similarly, identification information (eg, “ID_02”) of the second DOF sensor 120b may be mapped to a second label (eg, “BOX_02”) corresponding to the second unit area 202a. .

In this way, identification information of degree of freedom sensors respectively provided in each other object and identification information of a unit area corresponding to each other object may also be mapped to other objects to be photographed.

Therefore, as shown in FIG. 7 , the control unit 150 maps (or matched) information can be collected. This mapping information may be stored in the storage unit 130 . Through this, even if the position or attitude of the object is changed, the object may be tracked based on the unit area, and degree of freedom information for this may be collected.

Meanwhile, mapping of the label of the unit area and identification information of the degree of freedom sensor may be performed under the control of the controller 150 . The controller 150 may sequentially map the ID information of the degree of freedom sensor according to the order in which the unit area is designated. For example, the controller 150 may determine identification information of a first degree of freedom sensor having first identification information (ID_01) among a plurality of degree of freedom sensors in a label (eg, ID_01) of a first designated unit area. (ID_01) can be mapped. Then, the controller 150 maps identification information (ID_02) of a second degree of freedom sensor having second identification information among a plurality of degree of freedom sensors to a label (eg, ID_02) of a second designated unit area. can do. The order of identification information of the degree of freedom sensor may be based on an ascending order or a descending order.

Meanwhile, in the present invention, as shown in (a) and (b) of FIG. 6 , the controller 150 performs segmentation on the image objects included in each unit area 200a, and respectively A segmentation mask (or segmented mask, 300) for each image object included in the unit area 200a of may be extracted.

In the present invention, segmentation is a task of recognizing and extracting image objects based on objects in an image, and the controller 150 recognizes different image objects corresponding to different objects as separate types or classes. Instance segmentation can be performed.

For example, as shown in FIG. 6 , when segmentation is performed on the initial image 400, the controller 150 recognizes each image object as a separate meaningful subject, and based on each image object A segmentation mask 300 may be generated.

That is, the controller 150 recognizes and tracks each object based on the image object included in the unit area, generates and extracts a segmentation mask for the image object included in the unit area, and shapes information about each object. can be obtained.

Accordingly, the controller 150 extracts the first segmentation mask 301 for the first image object 411 corresponding to the first object 201 included in the first unit area 201a from the initial image 400. can do. Also, the controller 150 may extract the second segmentation mask 302 for the second image object 412 corresponding to the second object 202 included in the second unit area 202a. In this way, the controller 150 can extract segmentation masks corresponding to all objects.

Meanwhile, each extracted segmentation mask may have a different label (or mask label). In this case, a label of the segmentation mask may be determined based on a label set for each unit region. The controller 150 may set the label of the unit area and the label of the segmentation mask to be the same or to correspond to each other.

For example, as shown in FIG. 7 , when the label of the first unit area is BOX_01, the label of the first segmentation mask corresponding to the image object included in the first unit area may also be BOX_01.

As shown in FIG. 7 , the controller 150 may map a unit area, a degree of freedom sensor, and a segmentation mask to each other based on each object. Therefore, as shown in FIG. 7, i) identification information (ID_01) of the first degree of freedom sensor 120a provided in the first object 201, ii) a first image corresponding to the first object 201 A label (BOX_01) of the first unit area 201a including the object 411, iii) a first segmentation mask 301 of the first image object 411 corresponding to the first object 201, iv) Labels of the first segmentation mask 301 of the first image object 411 corresponding to one object 201 (which may be identical to or similar to labels of unit regions) are mutually mapped, and mapping information is stored in the storage unit 130. ) can be stored in

The mapping described above may be performed identically for all objects. Accordingly, the controller 150 may track the object even if the position or posture of the object is changed in a subsequent image, and may collect the segmentation mask and degree of freedom information of the object as learning data.

That is, extraction of segmentation may be continuously performed in subsequent images. The controller 150 may track each object based on the unit area in the subsequent image and collect image objects included in each unit area as a segmentation mask for the tracked object. Meanwhile, the control unit 150 extracts an object in real time or at predetermined intervals based on a unit area, and has information about the position of the object before the current point in time and the segmentation mask to be tracked. Accordingly, the control unit 150 controls the case where an image object for a specific object appears again in a subsequent image after the occlusion is removed, even if an image object for a specific object does not exist in a subsequent image at any point in time because the specific object is occluded by another object. , it is possible to continuously track a specific object.

Meanwhile, as shown in FIG. 7 , the controller 150 may collect degree of freedom information from a degree of freedom sensor provided in an object. The degree of freedom information may include at least one of 3D position information and 3D posture information of an object.

Furthermore, since the collected DOF information is stored after being mapped to a label of a unit area mapped to identification information of each DOF sensor, the controller 150 can acquire the DOO information on an object together with a segmentation mask. . For example, the degree of freedom information (or sensing information) collected from the first degree of freedom sensor may include the identification information (ID_01) of the first degree of freedom sensor, the label (BOX_01) of the first unit area mapped, and the first segmentation mask. (301).

On the other hand, as shown in FIG. 7, when sensing information (or DOF information) received from the DOF sensor is received from a communication port (PORT) mapped to identification information (Sensor_ID) of the DOF sensor, such Identification information (PORT ID) for a communication port may also be mapped with identification information (Sensor_ID) of a DOF sensor. Accordingly, the control unit 150 may store the DOF information received through the communication port to be mapped to identification information of the DOF sensor mapped to each communication port.

As shown in FIG. 7, using an initial image, one-to-one between at least two of a label of a unit area, identification information of a degree of freedom sensor, a communication port, initial position information and posture information of an object, a segmentation mask, and a label of the segmentation mask. When mapping is completed, settings for learning data collection may be completed.

When these settings are completed, the controller 150 tracks objects based on each object from subsequent images, extracts a segmentation mask for the tracked object, and collects information on degrees of freedom for the tracked object. can In addition, the segmentation mask extracted from the subsequent image and DOF information of the object may be mutually mapped. Accordingly, even when the position and posture of the object are changed, the controller 150 may extract a segmentation mask for the changed posture of the object based on the object, and collect degree of freedom information of the object at this time. Accordingly, the controller 150 may collect learning data for objects having various postures.

More specifically, in the present invention, segmentation is performed on a plurality of objects in a subsequent image (S340), and degree of freedom information (or sensing information) collected (or received) from a degree of freedom sensor (or sensor) and segmentation performance A process of storing the resulting extracted segmentation mask as learning data may be performed (S350).

As shown in (a) of FIG. 8, the controller 150 tracks the plurality of objects 410 from the subsequent image received through the camera 110, and as shown in (b) of FIG. , Segmentation masks 300a corresponding to labels set for each of the plurality of objects 410 (eg, unit zero labels or segmentation mask labels) may be extracted.

The label of the segmentation mask extracted from the subsequent image is the same as the label of the segmentation mask set in the initial image. In this case, the label of the segmentation mask may be the same as or similar to the label of the unit area.

As shown in FIG. 9, based on labels set for each of a plurality of objects, the control unit 150 controls the degree of freedom information sensed from the degree of freedom sensor and the segmentation mask 300a corresponding to each object (“freedom” in FIG. 9). (Refer to “Do” item) can be collected.

The controller 150 may match and store degree of freedom information sensed from the degree of freedom sensor at a time point corresponding to a time point at which a frame from which each segmentation mask is extracted is captured and stored.

If the subsequent video is a video, the sensing frequency (Hz) of the DOF sensor may correspond to the frame per second (FPS) of the video. That is, in the case of a video having 30 FPS, the sensing frequency of the DOF sensor may also be configured to be 30 Hz, and in this case, the time point at which each frame is captured coincides with the time point at which the DOF information of the object is sensed by the DOF sensor. can

Meanwhile, in the present invention, a follow-up image may include a video including a preset number of frames per second or a plurality of images captured at preset time intervals. Such a plurality of images may be referred to as a frame.

The control unit 150 generates learning data by extracting a segmentation mask for all frames included in the subsequent image, or the plurality of objects at predetermined intervals (or time intervals) among a plurality of frames constituting the subsequent image. It is also possible to generate training data by extracting segmentation masks corresponding to each.

Therefore, the control unit 150 i) sets the first segmentation mask 301a corresponding to the first object 411 among the plurality of objects 410 and the second object 412 at predetermined intervals or every frame. A second segmentation mask 302a is extracted, and ii) the first degree of freedom information collected from the first degree of freedom sensor ID_01 provided in the first object is matched with the first segmentation mask 301a to obtain the first degree of freedom. 1 stored as learning data for the object 411, and iii) the second degree of freedom information collected from the second degree of freedom sensor (ID_02) provided in the second object 412 and the second segmentation mask 302a may be matched and stored as learning data for the second object 412 .

This is possible because a unit area is set for each of a plurality of objects so that simultaneous tracking and segmentation mask extraction of a plurality of objects are possible in the initial image.

In this way, as shown in FIG. 9 in the subsequent image, the controller 150 extracts a segmentation mask for each object in the subsequent image, and stores degree-of-freedom information of the object when the corresponding segmentation mask is obtained by matching with each other. By doing so, it is possible to efficiently collect a vast amount of learning data for a plurality of objects.

Furthermore, in the present invention, by applying an external force to at least one of a plurality of objects (eg, an external force for intentionally changing at least one of the posture and position of a plurality of objects (eg, mixing a plurality of objects) operation, etc.)), a segmentation mask having a different shape for each of a plurality of objects and information on the degree of freedom corresponding thereto may be intentionally collected as learning data.

On the other hand, the control unit 150 according to the present invention not only mutually matches and stores only the segmentation mask for each of a plurality of objects and information on the degree of freedom thereof, but also other objects (eg, adjacent objects) disposed around each object. Alternatively, information on mutual positional relationships between a plurality of objects may be secured as learning data by storing segmentation masks and degree-of-freedom information for overlapping objects, etc.) together.

For example, as shown in (a) of FIG. 8 , the controller 150 is disposed around the second object 412 together with the segmentation mask 302a and the degree of freedom information of the second object 412. The segmentation masks 301a and 303a and degree-of-freedom information corresponding to the first and third objects 411 and 413 may be included as training data for the second object 412 .

Furthermore, when a specific object is covered by another object, the controller 150 may also store information on this as learning data.

Through this, in the present invention, it is possible to secure learning data for an object that can consider the surrounding situation.

As described above, in the learning data collection system and method according to the present invention, an object to be studied is provided with a sensor for collecting degree-of-freedom information, and image data about the object and degree-of-freedom information of the object are collected together. can do. In this case, in the present invention, segmentation may be performed on image objects corresponding to each object based on each object in the image data. Furthermore, in the present invention, by mapping mask labels corresponding to segmented masks and degree-of-freedom information for each object, it is possible to collect image data and degree-of-freedom information for an object at once just by capturing an image of the object. can Through this, according to the present invention, labor and time consumed by manually mapping image data and degree-of-freedom information corresponding to conventional objects can be absolutely reduced.

Furthermore, the learning data collection system and method according to the present invention can secure learning data for each of a plurality of objects for each image frame by placing a plurality of objects within the field of view of the camera. In this way, according to the present invention, it is possible to absolutely reduce the time required to collect learning data for various objects.

Meanwhile, the present invention described above may be implemented as a program that is executed by one or more processes in a computer and can be stored in a computer-readable medium (or a recording medium).

Furthermore, the present invention described above can be implemented as computer readable codes or instructions in a medium on which a program is recorded. That is, the present invention may be provided in the form of a program.

On the other hand, the computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is

Furthermore, the computer-readable medium may be a server or cloud storage that includes storage and can be accessed by electronic devices through communication. In this case, the computer may download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the above-described computer is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit), and there is no particular limitation on its type.

On the other hand, the above detailed description should not be construed as limiting in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (15)

In the learning data collection method performed by the computing system,
Receiving an initial image of a plurality of objects each equipped with a degree of freedom sensor through a camera;
setting, by a controller, different labels for each of the plurality of objects, based on an image object corresponding to each of the plurality of objects included in the initial image;
mapping, by the control unit, identification information of a degree of freedom sensor provided to each of the plurality of objects and a label set to each of the plurality of objects on a one-to-one basis;
extracting, by the control unit, a segmentation mask respectively corresponding to a label set for each of the plurality of objects by tracking the plurality of objects from subsequent images received through the camera; and
and collecting, by the control unit, learning data including the degree of freedom information sensed by the degree of freedom sensor and the segmentation mask based on a label set for each of the plurality of objects. Learning data collection method. According to claim 1,
The subsequent video is a video including a preset number of frames per second,
In the step of extracting the segmentation mask,
The control unit extracts a segmentation mask corresponding to each of the plurality of objects at predetermined intervals among successive frames included in the subsequent image,
In the step of collecting the learning data,
Wherein the control unit stores the degree of freedom information sensed from the degree of freedom sensor at a time point corresponding to a time point at which the frame from which the segmentation mask is extracted is captured by matching with the segmentation mask. According to claim 2,
The plurality of objects include a first object and a second object,
In the step of extracting the segmentation mask,
The control unit extracts a first segmentation mask corresponding to the first object and a second segmentation mask corresponding to the second object for each frame of the preset interval,
In the step of collecting the learning data,
In the control unit, first degree of freedom information collected from a first degree of freedom sensor provided in the first object is matched with the first segmentation mask and stored in a storage unit as learning data for the first object,
In the control unit, the second degree of freedom information collected from the second degree of freedom sensor provided in the second object is matched with the second segmentation mask and stored in the storage unit as learning data for the second object. How to collect learning data. According to claim 3,
Each of the first and second degree of freedom information,
Learning data collection method characterized in that it comprises at least one of 3-dimensional position information and 3-dimensional attitude information for each of the first and second objects. According to claim 4,
The first object,
At least one of the position and posture is changed by an external force while the subsequent image is received;
The learning data for the first object,
Learning data collection method characterized by comprising three-dimensional position information and three-dimensional posture information corresponding to the at least one changed position and posture. According to claim 2,
The learning data collection method, characterized in that the sensing frequency of the degree of freedom sensor corresponds to FPS (Frame Per Second) of the video. According to claim 1,
In the step of setting the label,
Based on a deep learning algorithm, to enable tracking of the plurality of objects from the subsequent image,
The learning data collection method, characterized in that a unit area for specifying an initial position of an image object corresponding to each of the plurality of objects in the initial image is designated by the control unit. According to claim 7,
The unit area is
In the initial image, each image object corresponding to each of the plurality of objects is specified,
Learning data collection method, characterized in that each unit area includes any one image object corresponding to each unit area among image objects corresponding to each of the plurality of objects. According to claim 8,
The learning data collection method, characterized in that the different labels set for each of the plurality of objects are labels set based on the plurality of objects in each unit area. According to claim 9,
In the step of setting the label,
In the control unit, performing segmentation on image objects included in each unit area and extracting a segmentation mask for each image object included in each unit area,
The method of collecting learning data, characterized in that the label of the segmentation mask extracted for each image object is specified based on the label set for each unit area. According to claim 10,
The plurality of objects include a first object and a second object,
Identification information of a first degree of freedom sensor provided in the first object is mapped to a first label set in a first unit area including a first image object corresponding to the first object among the respective unit areas,
The identification information of the second degree of freedom sensor provided in the second object is mapped to a second label set in a second unit area including a second image object corresponding to the second object among the respective unit areas. Characterized learning data collection method. According to claim 11,
Among the segmentation masks extracted from the initial image, a first mask label of a first segmentation mask corresponding to the first image object corresponds to the first label,
The learning data collection method, characterized in that the second mask label of the second segmentation mask corresponding to the second image object corresponds to the second label. According to claim 12,
A mask label of a segmentation mask corresponding to the first object extracted from the subsequent image corresponds to the first mask label,
A mask label of a segmentation mask corresponding to the second object extracted from the subsequent image corresponds to the second mask label. a sensing unit provided on each of a plurality of objects and including sensors configured to sense degree of freedom information for each of the plurality of objects;
a camera unit configured to photograph the plurality of objects;
Based on an image object corresponding to each of the plurality of objects included in the initial image received from the camera, different labels are set for each of the plurality of objects;
And a control unit for one-to-one mapping identification information of a sensor provided on each of the plurality of objects and a label set on each of the plurality of objects,
The control unit,
Tracking the plurality of objects from subsequent images received through the camera and extracting segmentation masks respectively corresponding to labels set for each of the plurality of objects;
The learning data collection system, characterized in that for collecting the segmentation mask and degree-of-freedom information collected from the sensors as learning data based on labels set for each of the plurality of objects. A program that is executed by one or more processes in an electronic device and can be stored in a computer-readable recording medium,
said program,
Receiving an initial image of a plurality of objects each equipped with a degree of freedom sensor through a camera;
setting different labels for each of the plurality of objects based on image objects corresponding to each of the plurality of objects included in the initial image;
one-to-one mapping the identification information of the degree of freedom sensor provided to each of the plurality of objects and the label set to each of the plurality of objects;
tracking the plurality of objects from subsequent images received through the camera and extracting segmentation masks corresponding to labels set for each of the plurality of objects; and
Based on the label set for each of the plurality of objects, the step of storing the degree of freedom information collected from the segmentation mask and the degree of freedom sensor as learning data A computer-readable record comprising instructions for performing the step. A program that can be stored on media.

Images