KR20220115742A - Method and system of generating data for machine learning training - Google Patents

Abstract

The present invention relates to a method performed by a device for generating data for learning of object posture recognition. A data generating method includes the steps of: generating shape information about an object to be recognized as a posture of an object; generating a virtual environment in which the object is located using shape information; and photographing the object in a virtual environment. According to the present invention, data suitable for an objective function of an artificial intelligence learning model can be generated.

Description

Data generation method and system for machine learning training

The present invention relates to a data generating method and system for machine learning training, and more particularly, to a data generating method and system for learning of an artificial neural network for posture recognition of an object.

Manufacturing process automation is more efficient than manual processes, and has been steadily developing since the Industrial Revolution due to the increase in the need for replacement of automated equipment due to the reduction of manpower and the development of vision and robot technology. Manufacturing process automation has been focused on the development of robots that can replace simple repetitive tasks, and has been mainly researched in hardware-related fields such as improving robot repeatability.

However, automation of parts packaging and assembly process work in the electronic field still relies on manual work due to the limitations of technology development. With the recent rise of the smart factory field due to the 4th industrial revolution, artificial intelligence technology for automation is being applied to fields such as real-time monitoring, parts assembly, logistics and work history tracking management, work status identification, and defect management.

As the demand for electronic component packaging and assembly process automation increases, the development of visual servoing technology that recognizes and tracks moving objects based on artificial intelligence technology so that the robot can grip the object is actively progressing in the parts assembly field. Object tracking technology using visual servoing can be classified into Position Based Visual Servoing (PBVS) and Image Based Visual Servoing (IBVS).

PBVS is a control technique using the relationship between an object in a 3D space and an image in a 2D space, and has a disadvantage in that it is necessary to extract internal and external parameters of the camera and to compensate for errors due to image distortion.

Although IBVS does not require the extraction of internal and external parameters of the camera, it is difficult to control due to the influence of external environment changes such as lighting because it depends on the camera image to control the movement of the feature points acquired from the camera image to the target position on the image. There is this. Therefore, in order to compensate for these shortcomings, research and development, such as a technology that combines the two methods, is in progress.

In order to replace the assembly field where workers at the manufacturing site are working with robots, high object detection accuracy and real-time processing technology are required. In the current research stage, the visual servoing technology for low-speed dynamic objects has been developed until the robot stably tracks the object.

In the case of high-speed dynamic object assembly, research has been carried out to increase the accuracy of object detection and posture estimation, but in the case of dynamic objects, the higher the speed, the more image blurring occurs. Occurs.

For visual servoing, the robot must first determine the target object and recognize the position and direction of the object. The next robot recognizes an object through a known DB and obtains three-dimensional coordinates for the object through a stereo camera. Also, for visual servoing, the posture of the object must be estimated. Visual servoing may be performed by estimating the posture of the object.

For object recognition, SIFT (Scale Invariant Feature Transform) is used, which uses a descriptor that is invariant to the size, rotation, and movement of an object. Objects are recognized through SIFT.

Stereo vision can be used to obtain three-dimensional information. Matching between two images is performed using SIFT, and then 3D information is obtained. Posture estimation of a desired object is performed using the obtained 3D information. For accurate posture estimation, matching verification is performed through position error using 3D information.

In order to be practically applied to the industrial field manufacturing process, first, stable object detection in real time must be performed. However, when the conventional method is applied to dynamic object assembly, there may be a problem in that the accuracy of accurate object detection and posture estimation is deteriorated.

Recently, artificial intelligence algorithms, for example, object recognition technology based on artificial neural networks, have been used for visual servoing of robots. However, in order to use the artificial neural network for visual servoing, learning about the artificial neural network must be preceded.

Learning of an artificial neural network requires data for training and testing, and since the learning efficiency of an artificial neural network depends on the quantity and quality of data, a large amount of high-quality data is required. However, it is difficult to obtain training data and test data suitable for the objective function because public data that has already been created is used rather than when the data required for learning is directly created and used.

The invention of Publication No. 10-2021-0002410 as a related invention relates to a method, apparatus and program for constructing a dataset for learning an appearance recognition model, and includes a step of acquiring image data for learning, a step of providing image data for learning, a step of acquiring detailed individual appearance characteristics, and Although a method including a training dataset construction step is disclosed, there is no specific mention on how to acquire the training image data.

The invention of Registration No. 10-2207489 as a related invention relates to a data generation method, and there is no mention of how to generate artificial data based on actual data in data collection related to the operation of a refrigeration device.

The invention of Publication No. 10-2021-0004162 as a related invention relates to an apparatus for automatically generating a dataset for artificial intelligence machine learning learning, a SW, and a control method thereof, including a camera for acquiring an image of an article located in space Although an artificial intelligence device for automatically generating datasets is disclosed, there is a problem in automation because the camera can only be manually operated by a human.

Although the related inventions introduced above may have in common with the present invention in terms of the purpose of the invention of generating data necessary for learning an artificial intelligence model, it is distinguished from the configuration of the present invention that directly generates an image through a virtual environment, and thus There is also a big difference in effectiveness.

Korean Patent Publication No. 10-2021-0002410 (published on Aug. 8, 2021) Korean Registered Patent No. 10-2207489 (Announced on Jan. 26, 2021) Korean Patent Laid-Open Patent No. 10-2021-00004162 (published on January 13, 2021)

One problem to be solved by the present invention is to solve the problem of learning and training of an artificial intelligence learning model according to the prior art that performed learning depending on public data.

One problem to be solved by the present invention is to propose a method and system capable of directly generating data for learning and testing suitable for an objective function of an artificial intelligence learning model.

One problem to be solved by the present invention is to propose a method and system capable of directly generating data required for learning of posture recognition of an object by using three-dimensional design information of the object.

A data generating method according to an embodiment of the present invention is a method performed by an apparatus for generating data for learning the posture recognition of an object, comprising: generating shape information about an object as a posture recognition target of the object; generating a virtual environment in which the object is located by using shape information; and photographing an object in a virtual environment.

A data generating method according to an embodiment of the present invention is a method performed by an apparatus for generating data for learning the posture recognition of an object, comprising: generating shape information about an object as a posture recognition target of the object; generating a virtual environment in which the object is located by using shape information; and photographing the object in a virtual environment, wherein the generating of the shape information about the object may include generating a CAD model of the object.

A data generating method according to an embodiment of the present invention is a method performed by an apparatus for generating data for learning the posture recognition of an object, comprising: generating shape information about an object as a posture recognition target of the object; generating a virtual environment in which the object is located by using shape information; and photographing an object in a virtual environment, wherein the generating of the virtual environment includes: receiving information about a reference axis to define a three-dimensional space to which the object belongs; receiving information about a reference posture of the object with respect to a reference axis; and receiving information regarding the background of the object.

A data generating method according to an embodiment of the present invention is a method performed by an apparatus for generating data for learning the posture recognition of an object, comprising: generating shape information about an object as a posture recognition target of the object; generating a virtual environment in which the object is located by using shape information; and photographing the object in a virtual environment, wherein the photographing of the object may be configured to include collecting RGB information and depth information about the object.

A data generation system according to an embodiment of the present invention includes: a virtual environment setting unit for generating an object and a three-dimensional space to which the object belongs by using shape information about an object that is a target of posture recognition of the object; a virtual photographing unit for photographing the object in a virtual environment; and a data generating unit for learning that generates data including a photographed image, information about an object, and camera information used for photographing.

In addition, the data generating method may be configured to further include the step of setting a reference axis of the posture information of the object output as an Euler angle using a runtime handler.

Also, the data generating method may be configured to further include generating a mesh file by using a CAD file related to the object.

Also, the data generating method may be configured to further include receiving information about the camera.

Also, the data generating method may be configured to further include receiving coordinate value information and rotation information of the object.

In addition, the data generating method may be configured to further include generating data including information about the size of the image, the position of the object, the posture of the object, and the occupancy range of the object.

Specific details of other embodiments are included in "Details for carrying out the invention" and attached "drawings".

Advantages and/or features of the present invention, and methods for achieving them, will become apparent with reference to various embodiments described below in detail in conjunction with the accompanying drawings.

However, the present invention is not limited to the configuration of each embodiment disclosed below, but may also be implemented in a variety of different forms, and each embodiment disclosed herein only makes the disclosure of the present invention complete, It is provided to fully inform those of ordinary skill in the art to which the scope of the present invention belongs, and it should be understood that the present invention is only defined by the scope of each of the claims.

According to the present invention, it is possible to generate data suitable for the objective function of the artificial intelligence learning model.

In addition, data including position and rotation information of an object can be used for learning an artificial intelligence model of object posture recognition.

1 is a block diagram of a data generation system according to an embodiment of the present invention.
2 is a network relationship diagram of a data generating system according to an embodiment of the present invention.
3 is a flowchart of a data generation method according to an embodiment of the present invention.
4 is an exemplary diagram of a data generation method according to an embodiment of the present invention.
5 is an exemplary diagram of a data generation system user interface according to an embodiment of the present invention.
6 is an exemplary diagram of a data generation system user interface according to an embodiment of the present invention.
7 is an exemplary diagram of data generated by a data generating method according to an embodiment of the present invention.

Before describing the present invention in detail, the terms or words used herein should not be construed as being unconditionally limited to their ordinary or dictionary meanings, and in order for the inventor of the present invention to describe his invention in the best way It should be understood that the concepts of various terms can be appropriately defined and used, and further, these terms or words should be interpreted as meanings and concepts consistent with the technical idea of the present invention.

That is, the terms used in this specification are only used to describe preferred embodiments of the present invention, and are not used for the purpose of specifically limiting the content of the present invention, and these terms represent various possibilities of the present invention. It should be noted that the term has been defined with consideration in mind.

Also, in this specification, it should be understood that, unless the context clearly indicates otherwise, the expression in the singular may include a plurality of expressions, and even if it is similarly expressed in plural, it should be understood that the meaning of the singular may be included. .

In the case where it is stated throughout this specification that a component "includes" another component, it does not exclude any other component, but further includes any other component unless otherwise indicated. It could mean that you can.

Furthermore, when it is described that a component is "exists in or is connected to" of another component, this component may be directly connected or installed in contact with another component, and a certain It may be installed spaced apart by a distance, and in the case of being installed spaced apart by a certain distance, a third component or means for fixing or connecting the corresponding component to another component may exist, and now It should be noted that the description of the components or means of 3 may be omitted.

On the other hand, when it is described that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the third element or means does not exist.

Likewise, other expressions describing the relationship between components, such as "between" and "immediately between", or "adjacent to" and "directly adjacent to", have the same meaning. should be interpreted as

In addition, if terms such as "one side", "other side", "one side", "other side", "first", "second" are used in this specification, with respect to one component, this one component is It is used to be clearly distinguished from other components, and it should be understood that the meaning of the component is not limitedly used by such terms.

In addition, in this specification, terms related to positions such as "upper", "lower", "left", "right", etc., if used, should be understood as indicating a relative position in the drawing with respect to the corresponding component, Unless an absolute position is specified for their position, these position-related terms should not be construed as referring to an absolute position.

In addition, in this specification, in specifying the reference numerals for each component of each drawing, the same component has the same reference number even if the component is indicated in different drawings, that is, the same reference is made throughout the specification. The symbols indicate the same components.

In the drawings attached to this specification, the size, position, coupling relationship, etc. of each component constituting the present invention are partially exaggerated, reduced, or omitted for convenience of explanation or in order to sufficiently clearly convey the spirit of the present invention. may be described, and thus the proportion or scale may not be exact.

In addition, in the following, in describing the present invention, a detailed description of a configuration determined that may unnecessarily obscure the subject matter of the present invention, for example, a detailed description of a known technology including the prior art may be omitted.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the related drawings.

1 is a block diagram of a data generation system according to an embodiment of the present invention.

Referring to FIG. 1 , the data generation system 100 includes a controller 110 , an input device 120 , an output device 130 , a storage device 140 , a communication device 150 , and a memory 160 . can be configured. In addition, the memory 160 may be configured to include a runtime handler 161 , an import/export 162 , a virtual environment setting unit 163 , a scene controller 164 , and a camera controller 165 .

The control unit 110 may be implemented in the form of a central processing unit corresponding to a processor, and performs a function of executing an instruction set constituting a data generation method.

The controller 110 may basically control their operations through data transmission to, and data reception from, a user input device, a camera, an output device, a storage device, and a communication device.

The user input device 121 performs a function of receiving various user inputs necessary for executing the data generating method S100, for example, characters and numbers.

The camera 122 functions to provide an actual photographed image. For example, the camera 122 is a background image to be used in a virtual environment, and an image of an assembly line in an actual factory, for example, an image including RGB information and depth information, is provided to the data generating device 100 as a background image. can do.

The output device 130 performs a function of displaying various data and information to the user, for example, a runtime handler 161, a user interface, or the like, or outputting a sound when the data generating method S100 is executed.

The storage device 140 performs a function of storing various data and information, for example, 3D shape information of an object, camera information, and the like in executing the data generating method.

The communication device 150 performs a function of transmitting and receiving data through various protocols including a camera and various external devices and external devices, for example, a server in executing the data generating method. Accordingly, the communication device may be configured to include a communication module corresponding to various protocols internally, for example, a parallel communication module, a serial communication module, a short-range communication module, a cellular communication module, a wireless LAN communication module, and an Internet communication module. .

The memory 160 may be implemented through a non-volatile or volatile semiconductor device in which stored data is maintained or disappeared according to ON/OFF of power. The memory 170 loads and stores command execution modules implemented in software form.

The runtime handler 161 sets a reference axis in the three-dimensional virtual space and performs a function of arranging an object along the reference axis.

The import/export 162 reads three-dimensional shape information of an object, for example, a 3D CAD file, and outputs a converted file in another format.

The virtual environment setting unit 163 functions to receive information about an object and information about a background, set a virtual environment using the information, and photograph an object according to the settings of a camera disposed in the virtual environment.

The virtual environment setting unit 163 may control the position, direction, and illuminance of the lighting in the virtual environment so that the real object is placed or the object moves close to the working environment of the production line.

The scene controller 164 functions to control a scene including an object and a background in the virtual environment.

The camera controller 165 functions to control the arranged cameras based on camera information in the virtual environment. A camera disposed in the virtual environment corresponds to a virtual camera. The user may select a camera to be used for shooting among various types of cameras in the virtual environment by using the virtual environment setting unit 163 .

2 is a network relationship diagram of a data generating system according to an embodiment of the present invention.

Referring to FIG. 2 , the data generating system 100 may be connected to the server 200 to communicate through the network 300 .

The server 300 may provide three-dimensional shape information about an object, a three-dimensional CAD file, and the like to the data generating system 100 . In addition, the server 300 may provide the data generating system 100 with information about a camera required for the virtual environment.

The network 300 is a wired and wireless network, for example, a local area network (LAN), a wide area network (WAN), the Internet of TCP/IP, an intranet and an extranet, and a mobile network, eg, any suitable communication network, including cellular, 3G, LTE, WiFi networks, ad hoc networks, and combinations thereof.

Network 300 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 300 may include one or more connected networks, eg, multiple network environments, including public networks such as the Internet and private networks such as secure enterprise private networks. Access to network 300 may be provided via one or more wired or wireless access networks.

3 is a flowchart of a data generation method according to an embodiment of the present invention.

4 is an exemplary diagram of a data generation method according to an embodiment of the present invention.

Referring to FIG. 4 , referring to FIG. 3 , the data generation method ( S100 ) includes generating shape information about an object ( S110 ), creating a virtual environment using shape information about the object ( S120 ), and photographing an object in the virtual environment It may be configured to include (S130) and data generation (S140) necessary for learning using the photographing information.

The data generating system 100 of FIG. 1 corresponds to a subject performing the data generating method S100 of FIG. 3 .

In step S110 , the data generating system 100 may generate shape information about the object. Referring to FIG. 4 , shape information about an object may be a 3D design file generated by computer aided design (CAD). The data generation system 100 reads a 3D design file produced through computer-aided design by using the import/export 162, converts it into a file in another format so that other program modules can share it, and outputs it. .

In step S120 , the data generating system 100 may generate a virtual environment using shape information about the object. Referring to FIG. 4 , the virtual environment may be configured to include a reference axis setting, a reference posture setting of an object, and a scene setting. In the scene setting, the surrounding environment and the material of the object may be expressed.

In step S130 , the data generating system 100 may photograph the object in the virtual environment. Referring to FIG. 4 , the data generating system 100 may arrange a virtual camera in a virtual environment and photograph a random surface of an object. In order to implement an arbitrary posture and position, the camera may rotate or the object may freely fall.

In step S140 , the data generation system 100 may generate data necessary for learning by using the photographing information. Referring to FIG. 4 , data for training and data for testing may be generated, and these data may include RGB images, depth images, and information about objects and backgrounds.

A method and system for generating data according to an embodiment of the present invention learns a Convolutional Neural Network (CNN), which is a kind of artificial neural network, using a large amount of image data to autonomously generate a large amount of data for an algorithm that provides various information. can

The data generating system 100 may generate data similar to an actual photographic image by using a rendering technology capable of generating a high-resolution image, and may generate a depth image using a depth value in a virtual environment.

With respect to the image generated by the data generation method, the coordinate values that can specify the position of the object and the Euler angle, which is the degree of rotation about each axis, are output so that the degree of rotation of the object on the image can be known. This rotation information is an angle generated by considering both the position and angle of the camera, and has information on the posture taken by the object on the image.

According to an embodiment of the present invention, the data generating method S100 may be implemented in the form of a program including an instruction set for executing it. The program for executing the data generation method may be configured to include a runtime handler corresponding to the reference axis setting program, a shared folder, and a virtual environment setting unit corresponding to the scene shooting program.

The program according to an embodiment of the present invention has a structure in which a program for receiving information required by a user and an actual operation program are separated. You can choose the background, etc. This information is generated by outputting it to a shared folder that both the reference axis setting program and the scene shooting program can access. The scene shooting program uses this information to change the configuration of the virtual environment and the properties of the camera so that the user can create desired data.

The runtime handler 161 corresponding to the reference axis setting program may adjust details so that the user can output desired data.

The import/export 162 imports 3D model files (.obj, .3ds) from which you want to generate data and a graphical user interface that provides an environment in which information can be input and import/export to output files to be stored in a shared folder. function is provided.

The runtime handler 161 provides a function of aligning the reference axis by directly rotating the object so that the user can set the reference axis of the posture information of the object output as the Euler angle. Information about this is stored in a shared folder.

The files saved in the shared folder are a mesh file converted so that the 3D model file can be re-implemented in the Unity program, and a file containing information about camera settings and objects. The shared folder is a folder set to be accessible to both the runtime handler 161 corresponding to the reference axis setting program and the virtual environment setting unit 163 corresponding to the scene shooting program.

The virtual environment setting unit 163 sets the details of the camera used to generate an image through the file information stored in the shared folder, sets properties such as the material of the target object, and sets the files in the shared folder. An image of an object can be created in a virtual environment in which all information about the object is reflected.

The virtual environment setting unit 163 may make an object to be recognized take a random posture and may generate an image of the object using a virtual camera. Alternatively, an image of an object may be captured while the camera itself rotates at a certain angle. The camera may be configured to first take an RGB image of the object, and then generate a depth image of the object after matching the position and posture of the object and the camera to the same as when generating the RGB image.

In order to naturally generate various faces of the object to be recognized, the object is allowed to fall freely in a virtual environment, and the object whose posture changes as it falls may be photographed. Here, according to the shape of the object, a random surface may be photographed through a physical law, for example, rotation by a moment during free fall.

The import/export 162 creates a binary file with the same name as each image for the images generated by the virtual environment setting unit 163, which includes the size of the image, the position of the object, the posture of the object, and the image. Contains information about the range the object occupies.

5 is an exemplary diagram of a data generation system user interface according to an embodiment of the present invention.

Referring to FIG. 5 , after a reference axis is set, a virtual environment in which an object is disposed is depicted.

6 is an exemplary diagram of a data generation system user interface according to an embodiment of the present invention.

Referring to FIG. 6 , after a reference axis is set, a background screen of a virtual environment in which an object is disposed is depicted.

7 is an exemplary diagram of data generated by a data generating method according to an embodiment of the present invention.

Referring to FIG. 7 , the structure of data generated by the data generating system 100 according to an embodiment of the present invention is depicted. The generated data may include an RGB image, depth image, and posture information. Here, the posture information may include information about the center coordinates (x, y, z) of the object, and rotation information compared to the reference posture from the camera point of view.

As described above, according to an embodiment of the present invention, it is possible to generate data suitable for the objective function of the artificial intelligence learning model.

In addition, data including position and rotation information of an object can be used for learning an artificial intelligence model of object posture recognition.

In the above, although several preferred embodiments of the present invention have been described with some examples, the descriptions of various various embodiments described in the "Specific Contents for Carrying Out the Invention" item are merely exemplary, and the present invention Those of ordinary skill in the art will understand well that the present invention can be practiced with various modifications or equivalents to the present invention from the above description.

In addition, since the present invention can be implemented in various other forms, the present invention is not limited by the above description, and the above description is intended to complete the disclosure of the present invention, and is generally It should be understood that this is only provided to fully inform those with knowledge of the scope of the present invention, and that the present invention is only defined by each of the claims.

100: data generation system
110: control unit
120: input device
121: user input device
122: camera
130: output device
140: storage device
150: communication device
160: memory
161: runtime handler
162: import/export
163: virtual environment setting unit
164: scene controller
165: camera controller

Claims (5)

A method performed by an apparatus for generating data for learning of posture recognition of an object, comprising:
generating shape information about an object that is a target for posture recognition of the object;
generating a virtual environment in which the object is located by using the shape information; and
configured to include photographing the object in the virtual environment,
How to create data. A method performed by an apparatus for generating data for learning of posture recognition of an object, comprising:
generating shape information about an object that is a target for posture recognition of the object;
generating a virtual environment in which the object is located by using the shape information; and
Comprising the step of photographing the object in the virtual environment,
The step of generating shape information about the object comprises:
configured to include generating a CAD model of the object;
How to create data. A method performed by a device for generating data for learning of posture recognition of an object, comprising:
generating shape information about an object that is a target for posture recognition of the object;
generating a virtual environment in which the object is located by using the shape information; and
Comprising the step of photographing the object in the virtual environment,
Creating the virtual environment includes:
receiving information about a reference axis to define a three-dimensional space to which the object belongs;
receiving information about a reference posture of the object with respect to the reference axis; and
configured to include receiving information regarding the background of the object;
How to create data. A method performed by an apparatus for generating data for learning of posture recognition of an object, comprising:
generating shape information about an object that is a target for posture recognition of the object;
generating a virtual environment in which the object is located by using the shape information; and
Comprising the step of photographing the object in the virtual environment,
The step of photographing the object is
configured to include collecting RGB information and depth information for the object,
How to create data. a virtual environment setting unit for generating the object and a three-dimensional space to which the object belongs by using shape information about an object that is a target of posture recognition of the object;
a virtual photographing unit for photographing the object in the virtual environment; and
It is configured to include a data generating unit for learning that generates data including the captured image, information about the object, and camera information used for photographing,
data generation system.

Images