JP2022024189A - Learning data creation method, learning data creation device, and program - Google Patents

Abstract

To easily create learning data.SOLUTION: One embodiment is characterized in that a computer executes: a first creation procedure in which, a virtual space is created by a three-dimensional simulator, where one or more objects are positioned; a second creation procedure in which, in response to a first image being created where an inside of a prescribed range in a real space corresponding to the virtual space is captured, a second image is created in which the inside of the range in the virtual space is drawn by the three-dimensional simulator; and a third creation procedure in which, prescribed information obtainable from the second image is added to the first image as teacher information, and learning data to be used for a prescribed machine learning model to practice learning is created.SELECTED DRAWING: Figure 2

Description

The present invention relates to a learning data creation method, a learning data creation device, and a program.

In recent years, various tasks have been executed by machine learning techniques. As one of such tasks, for example, semantic segmentation and the like are known. Semantic segmentation is a task of classifying each pixel in an image taken by a camera device or the like into a class (for example, the object name of an object represented by the pixel) according to the meaning indicated by the pixel. ..

Here, in many tasks such as semantic segmentation, machine learning models are often learned by supervised learning.

Japanese Unexamined Patent Publication No. 2017-182129 Japanese Unexamined Patent Publication No. 2016-71597

However, learning data used for supervised learning is often created manually. For example, in semantic segmentation, teacher information (classification of each pixel) is given to the image by filling each pixel in the image with the color of the class according to the meaning indicated by this pixel. Training data is created.

In addition, depending on the task to be executed by the machine learning method, it is necessary to create learning data to which a plurality of teacher information is added. For example, in addition to the above classification, learning data in which the posture (direction, rotation, etc.) of an object in an image, the state of the object, and the like are added as teacher information may be required.

Furthermore, in general, learning a machine learning model often requires a large amount of learning data. Therefore, it may take a lot of labor and a huge amount of time to create learning data used for supervised learning.

The embodiment of the present invention has been made in view of the above points, and an object thereof is to easily create learning data.

In order to achieve the above object, the embodiment of the present invention includes a first creation procedure for creating a virtual space in which one or more objects are arranged by a three-dimensional simulator, and a predetermined real space corresponding to the virtual space. A second creation procedure for creating a second image drawn by the three-dimensional simulator in the range of the virtual space according to the creation of the first image captured in the range, and the first creation procedure. A third creation procedure for creating learning data used for learning a predetermined machine learning model by adding predetermined information obtained from the second image to the image of the above as teacher information, and a computer. Is characterized by executing.

Training data can be easily created.

It is a figure which shows an example of the whole structure of the learning data creation system in embodiment of this invention. It is a figure for schematically explaining an example of learning data creation. It is a figure for demonstrating an example of the flow of the advance preparation procedure. It is a figure for demonstrating an example of the flow of the learning data creation procedure. It is a figure which shows an example of a teacher information list. It is a figure which shows an example of the hardware composition of the learning data creation apparatus in embodiment of this invention.

Hereinafter, embodiments of the present invention will be described. Hereinafter, a learning data creation system 1 capable of easily creating learning data of a machine learning model that executes a predetermined task will be described. Predetermined tasks include, for example, recognition and classification of an object in an image taken by a camera device or the like, grasping the state of the object, and some action related to the object (for example, gripping action of the object or avoidance action of the object). ) And other tasks.

In the embodiment of the present invention, an image taken by a camera device (that is, a virtual camera device installed in the virtual space) of a virtual space created by a three-dimensional simulator (hereinafter, also referred to as a "virtual shot image"). ) And an image of the real space corresponding to the virtual space taken by an actual camera device (hereinafter, also referred to as "actually shot image"), obtained from the virtual shot image with respect to the actual shot image. By adding the teacher information to be used, learning data is created. The teacher information includes, for example, contour line information of an object in a virtual photographed image, a class in which the object is classified, an object name of the object, state information of the object, a depth to the object, an attitude of the object, and the subject. Information for performing a predetermined action regarding an object and the like can be mentioned.

Here, the real space corresponding to the virtual space is, for example, a real space in which the same object is arranged at the same position as the virtual space created by the three-dimensional simulator. The fact that the positions are the same in the virtual space and the real space means that, for example, when the same coordinate system is set in the virtual space and the real space, the position coordinates are the same. However, for example, a coordinate system that can be converted to each other may be set in the virtual space and the real space. In the following, "position" and "posture" shall represent positions and postures in the same coordinate system set in the virtual space and the real space.

In addition, the same object in the virtual space and the real space means that the object represented by the three-dimensional model arranged in the virtual space and the actual object arranged in the real space are the same. There is. An object arranged in the virtual space is also referred to as an "object" in order to distinguish it from an actual object arranged in the real space.

<Overall configuration of learning data creation system 1>
First, the overall configuration of the learning data creation system 1 according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram showing an example of the overall configuration of the learning data creation system 1 according to the embodiment of the present invention.

As shown in FIG. 1, the learning data creating system 1 according to the embodiment of the present invention includes a learning data creating device 10, one or more camera devices 20, and one or more tracking devices 30. Further, the learning data creation device 10, the camera device 20, and the tracking device 30 are communicably connected via a communication network such as a wireless LAN (Local Area Network). In addition, this communication network may be, for example, a wired LAN or the like in whole or in part.

The learning data creation device 10 is a computer or a computer system that creates learning data. The learning data creating device 10 includes a three-dimensional simulator 100, a learning data creating unit 200, and a storage unit 300.

The three-dimensional simulator 100 is a simulator capable of simulating a three-dimensional virtual space. In the three-dimensional simulator 100, objects can be arranged in a virtual space, and physical operations (for example, collision determination between objects) that simulate the physical rules of objects can be performed.

Further, in the three-dimensional simulator 100, it is possible to draw a virtual photographed image taken by a virtual camera device in the virtual space. At this time, in the three-dimensional simulator 100, for example, the contour line information and the object name of the object (object) in the virtual photographed image, the class to which the object is classified, the state information of the object, the depth to the object, and the object. The posture of the object, the result of performing a predetermined physical calculation on the object, and the like can be given to the virtual photographed image. This information is generated by an operation or the like in the three-dimensional simulator 100.

Such a three-dimensional simulator 100 is realized by, for example, a game engine such as Unity, Unreal Engine 4 (UE4), or Blender. However, the three-dimensional simulator 100 is not limited to these game engines, and may be realized by any three-dimensional simulation software.

The learning data creation unit 200 creates learning data by adding teacher information obtained from a virtual captured image to an actual captured image. The teacher information is information given to the virtual captured image when the three-dimensional simulator 100 draws the virtual captured image (for example, contour line information and object name of an object in the virtual captured image, and a class in which the object is classified. , State information of the object, depth to the object, posture of the object, result of performing a predetermined physical calculation on the object, etc.).

In this way, the learning data creation unit 200 creates learning data by adding the information obtained from the virtual captured image drawn by the three-dimensional simulator 100 to the actual captured image as teacher information.

The storage unit 300 stores various information. The information stored in the storage unit 300 includes, for example, an actual photographed image taken by the camera device 20 in the real space, a virtual photographed image drawn by the three-dimensional simulator 100, tracking information acquired from the tracking device 30, and in the virtual space. A three-dimensional model of an object to be arranged or the like can be mentioned. Here, the tracking information is information that tracks the position and posture of the camera device 20 in the real space. That is, the tracking information is information indicating both the position and the posture of the camera device 20 at each time. However, the tracking information may be, for example, information indicating only the position of the camera device 20 at each time.

The camera device 20 is an image pickup device that captures a real space and creates an actual photographed image. The camera device 20 is fixed to, for example, a portable camera stand or the like on which the tracking device 30 is mounted. The actual photographed image created by the camera device 20 is transmitted to the learning data creating device 10 and stored in the storage unit 300. The camera device 20 may be, for example, a depth camera capable of creating an actual photographed image to which depth information is added.

The tracking device 30 is a device (for example, a sensing device equipped with a position sensor and a posture sensor) that tracks the position and posture of the camera device 20 and creates tracking information. The tracking device 30 is attached to, for example, a portable camera stand or the like. In this way, one tracking device 30 is installed in association with one camera device 20. The tracking information created by the tracking device 30 is transmitted to the learning data creation device 10 and stored in the storage unit 300. The tracking device 30 may be directly mounted on the camera device 20, or may be built in the camera device 20, for example.

The configuration of the learning data creation system 1 shown in FIG. 1 is an example, and may be another configuration. For example, the learning data creation system 1 may have an arbitrary number of camera devices 20 and a tracking device 30 corresponding to these camera devices 20.

Further, the camera device 20 may not have a tracking device 30 corresponding to the camera device 20 as long as the position and posture in the real space are known. For example, when the camera device 20 is fixedly installed at a predetermined position and in a predetermined posture, the tracking device 30 corresponding to the camera device 20 may not be provided.

<How to create learning data>
Here, an outline of a case where learning data is created by the learning data creating device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 2 is a diagram for schematically explaining an example of creating learning data.

As shown in FIG. 2, an actual photographed image taken by a camera device 20 in a certain posture at a certain position in a real space is referred to as an “actually photographed image G110”. Further, a virtual photographed image taken by a virtual camera device having the same posture at the same position in the virtual space corresponding to the real space is referred to as a “virtual photographed image G210”.

At this time, information generated by the calculation of the three-dimensional simulator 100 or the like (in FIG. 2, "contour line" and "object name" as an example) is added to the virtual photographed image G210. That is, in the example shown in FIG. 2, the contour line of each object in the virtual photographed image G210 and the object name of each object are given. Of the information that can be generated by the calculation of the three-dimensional simulator, what kind of information is given to the virtual photographed image G210 depends on the task to be executed by the machine learning model.

The learning data creation device 10 is for learning by imparting information (that is, "contour line" and "object name") given to the virtual captured image G210 to the actual captured image G110 as teacher information. Create data G120. As a result, the learning data G120 represented by a set of the actual photographed image G110 and the teacher information (that is, the “contour line” and the “object name”) is created.

As described above, the learning data creation device 10 according to the embodiment of the present invention is a virtual image G110 actually captured in a certain range of the real space and a virtual image captured in the same range of the virtual space. The learning data G120 is created by using the captured image G210 and applying the information obtained from the virtual captured image G210 (that is, the information generated by the calculation of the three-dimensional simulator or the like) to the actual captured image G110. .. Therefore, in the learning data creating device 10 according to the embodiment of the present invention, the learning data G120 can be easily created.

Moreover, in the learning data creating device 10 according to the embodiment of the present invention, the position and posture of the camera device 20 are specified by the tracking information acquired from the tracking device 30, so that the position of the virtual camera device in the virtual space is specified. And the posture can be synchronized with the camera device 20. Therefore, for example, the user can easily obtain an actual photographed image and a virtual photographed image corresponding to the actual photographed image only by photographing the inside of the real space with the camera device 20.

<Flow of preparation procedure>
In the embodiment of the present invention, as described above, the virtual space and the real space need to correspond to each other. Therefore, it is necessary to associate the virtual space with the real space as a preliminary preparation for creating the learning data. Therefore, in the following, by arranging an object (object) in the virtual space by the three-dimensional simulator 100 and then arranging the actual object in the real space so as to correspond to this virtual space, the virtual space and the real space can be described. The procedure for associating the above with reference to FIG. 3 will be described with reference to FIG. FIG. 3 is a diagram for explaining an example of the flow of the preparatory procedure.

Step S101: The three-dimensional simulator 100 acquires a three-dimensional model of an object (object) arranged in the virtual space from the storage unit 300. This means, for example, importing the data of the three-dimensional model stored in the storage unit 300 into the three-dimensional simulator 100. In the three-dimensional model, not only the shape of the object but also information such as an object ID, an object name, and a category to which the object belongs is given.

The three-dimensional model may be created in advance by any method and then stored in the storage unit 300. As a method of creating a three-dimensional model, for example, the three-dimensional shape of an actual object may be created by scanning it with a three-dimensional scanner or the like, or it may be manually created by a three-dimensional model creation software or the like. good.

Step S102: The three-dimensional simulator 100 arranges an object represented by the three-dimensional model in the virtual space. For example, the user selects a desired 3D model from the plurality of 3D models imported in step S101 above, and then drags and drops the selected 3D model into the virtual space. , Objects can be placed in the virtual space. In addition to this, the user may be able to arrange the object represented by the three-dimensional model in the virtual space by specifying the position coordinates in the virtual space.

Here, when arranging the object represented by the three-dimensional model in the virtual space, the user may arbitrarily tilt or rotate the object and then arrange the object. In addition to this, the user may arrange the object after enlarging or reducing the object, for example.

When arranging a plurality of objects in the virtual space, the above step S102 may be repeated.

By the above steps S101 to S102, a virtual space in which one or more objects (objects) are arranged at desired positions is created by the three-dimensional simulator 100.

Step S103: The user arranges an actual object in the real space so as to correspond to the virtual space created by the above steps S101 to S102.

Here, for example, the user can superimpose and display an object arranged in the virtual space on the real space, and use a display device equipped with a position sensor and a posture sensor to display this display. The actual object may be placed at the same position as the object displayed on the device. Examples of such a display device include a head mount display equipped with a position sensor, an attitude sensor, and a camera, a head mount display equipped with a position sensor and an attitude sensor that can transparently visually recognize the real space, and projection mapping. Examples thereof include a device, a tablet terminal equipped with a position sensor, a posture sensor and a camera, and a smartphone equipped with a position sensor, a posture sensor and a camera.

With these display devices, it is possible to display an image in which an object is superimposed in the real space after synchronizing the position in the virtual space and the position in the real space. Therefore, for example, the user moves in the real space after carrying or wearing the display device, and arranges the same object in the real space at the same position and in the same posture as the object in the video. Can be done.

Thereby, the virtual space created by the above steps S101 to S102 can be made to correspond to the real space. In addition to the above, for example, by creating a mixed reality that fuses real space and virtual space with a technology such as MR (Mixed Reality), the same position as the object placed in the virtual space is the same. The same object may be placed in real space in a posture.

<Flow of learning data creation procedure>
Next, regarding the procedure for creating learning data by creating an actual photographed image and a virtual photographed image corresponding to the actual photographed image, and then using these the actual photographed image and the virtual photographed image. This will be described with reference to FIG. FIG. 4 is a diagram for explaining an example of the flow of the learning data creation procedure.

Step S201: The user uses the camera device 20 to photograph a desired range in the real space. As a result, the actual photographed image is created by the camera device 20 and transmitted to the learning data creating device 10. In the learning data creating device 10, the actually captured image is stored in the storage unit 300.

At this time, the tracking device 30 corresponding to the camera device 20 transmits the tracking information to the learning data creation device 10. As a result, in the learning data creating device 10, the tracking information is stored in the storage unit 300. As described above, the tracking information is information indicating the position and posture of the camera device 20.

In step S201 described above, the tracking device 30 stores the tracking information created by tracking the position and posture of the camera device 20 in the storage unit 300, but the present invention is not limited to this. After tracking the position and posture of the camera device 20 by any method, the storage unit 300 may store the tracking information indicating the tracking result. For example, even if a two-dimensional code such as a QR code (registered trademark) is pasted on the camera device 20 in advance and the two-dimensional code is read by a camera or the like to track the position and posture of the camera device 20. good.

Step S202: The three-dimensional simulator 100 takes a picture of the inside of the virtual space with the virtual camera device at the same position and posture as the camera device 20 taken in the above step S201. That is, the three-dimensional simulator 100 draws (renders) within the shooting range of the virtual camera device having the same position and orientation as the camera device 20 shot in step S201 above in the virtual space.

Here, the three-dimensional simulator 100 can specify the position and the posture of the camera device 20 from the tracking information created in the above step S201. Therefore, the three-dimensional simulator 100 can install a virtual camera device in the virtual space at the same position and posture as the camera device 20 in the real space. As a result, a virtual captured image corresponding to the actual captured image created in step S201 is created.

At this time, the three-dimensional simulator 100 adds predetermined information generated by acquisition or calculation in the virtual space to the virtual captured image. Then, the three-dimensional simulator 100 stores the virtual photographed image to which the predetermined information is added in the storage unit 300.

Here, as the predetermined information, as described above, for example, the contour line information and the object name of the object (object) in the virtual photographed image, the class to which the object is classified, the state information of the object, and the object. Depth, the posture of the object, the result of performing a predetermined physical calculation on the object, and the like. Further, as a result of performing a predetermined physical calculation on an object, for example, information on an operation in which a robot arm capable of a preset operation can grip the object at the position can be mentioned. Alternatively, for example, information on an operation in which a mobile robot capable of a preset operation can avoid the object at the position can be mentioned. It should be noted that these robot arms and mobile robots are examples of operation main bodies capable of preset operations.

The above step S202 may be automatically executed after the above step S201, for example, or may be executed in response to a user operation (for example, a rendering start operation in the virtual space). ..

Step S203: The learning data creation unit 200 assigns predetermined information given to the virtual captured image created in step S202 to the actual captured image created in step S201 as teacher information. do. As a result, learning data represented by a set of the actual photographed image and the teacher information is created.

Here, the teacher information included in the learning data is represented in a list format, for example. As an example, FIG. 5 shows a plurality of teacher information represented in a list format (this is also referred to as a “teacher information list”). FIG. 5 is an example of a teacher information list assigned to a certain actual photographed image (image ID: image101).

Each teacher information included in the teacher information list shown in FIG. 5 is information in which an object ID, position information, contour line information, contact information, and gripping motion information are associated with each other.

The object ID is an ID that identifies an object (object). The object ID is, for example, information given to a three-dimensional model of an object arranged in a virtual space.

The position information is the position coordinates where the object (object) is arranged. The position information is, for example, information given to the object represented by the three-dimensional model when the object represented by the three-dimensional model is arranged in step S102.

The contour line information is information indicating the contour line of an object (object). The contour line information can be obtained, for example, from the rendering result when the virtual captured image is drawn (rendered) in step S202.

The contact information is information indicating the object ID of the other object, the contact position with the other object, and the like when the object of the object ID is in contact with another object (object). The contact information can be obtained, for example, from the calculation result of the physical calculation of the three-dimensional simulator 100.

The gripping motion information is, for example, information related to a motion in which a robot arm capable of a preset motion can grip an object having the object ID at a shooting position of a virtual captured image. The gripping motion information can be obtained, for example, from the calculation result of the physical calculation of the three-dimensional simulator 100.

As described above, the teacher information list shown in FIG. 5 is a list of teacher information in which position information, object names, contour line information, contact information, and gripping motion information are associated with each object. Is. In addition to this, any information that can be acquired or calculated by the three-dimensional simulator 100 may be associated with the teacher information. For example, the virtual photographed image itself or a part of the area of the virtual photographed image may be associated with the teacher information. Specifically, for example, an image area portion representing an object of the object ID may be associated with the object ID in the image area of the virtual photographed image.

Further, each teacher information may be information to which only a part of the above information (position information, object name, contour line information, contact information, gripping motion information, etc.) is associated.

It should be noted that the teacher information included in the learning data is represented in a list format as an example, and the teacher information included in the learning data may be represented in any other format.

<Hardware configuration of learning data creation device 10>
Next, the hardware configuration of the learning data creation device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 6 is a diagram showing an example of the hardware configuration of the learning data creation device 10 according to the embodiment of the present invention.

As shown in FIG. 6, the learning data creating device 10 according to the embodiment of the present invention includes an input device 401, a display device 402, an external I / F403, a communication I / F404, and a RAM (Random Access Memory). It has a 405, a ROM (Read Only Memory) 406, a processor 407, and an auxiliary storage device 408. Each of these hardware is connected to each other by a bus 409.

The input device 401 is, for example, a keyboard, a mouse, a touch panel, or the like, and is used for a user to input various operations. The display device 402 is, for example, a display or the like, and displays various processing results of the learning data creation device 10. The learning data creation device 10 does not have to have at least one of the input device 401 and the display device 402.

The external I / F 403 is an interface with an external device. The external device includes a recording medium 403a and the like. The recording medium 403a and the like can be read and written via the learning data creation device 10 and the external I / F 403. The recording medium 403a may record one or more programs that realize the three-dimensional simulator 100 and the learning data creation unit 200.

The recording medium 403a includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), a USB (Universal Serial Bus) memory card, and the like.

The communication I / F 404 is an interface for connecting the learning data creation device 10 to the communication network. One or more programs that realize the three-dimensional simulator 100 and the learning data creation unit 200 may be acquired (downloaded) from a predetermined server device or the like via the communication I / F 404.

The RAM 405 is a volatile semiconductor memory that temporarily holds programs and data. The ROM 406 is a non-volatile semiconductor memory that can hold programs and data even when the power is turned off. The ROM 406 stores, for example, settings related to an OS (Operating System), settings related to a communication network, and the like.

The processor 407 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like, and is an arithmetic unit that reads programs and data from a ROM 406, an auxiliary storage device 408, or the like onto a RAM 405 and executes processing. The three-dimensional simulator 100 and the learning data creation unit 200 are realized, for example, by processing one or more programs stored in the auxiliary storage device 408 to be executed by the processor 407. The learning data creation device 10 may have both a CPU and a GPU as the processor 407, or may have only one of the CPU and the GPU.

The auxiliary storage device 408 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and is a non-volatile storage device that stores programs and data. The auxiliary storage device 408 stores, for example, an OS, various application software, one or more programs that realize a three-dimensional simulator 100 and a learning data creation unit 200, and the like. The storage unit 300 is realized by using, for example, an auxiliary storage device 408. However, the storage unit 300 may be realized by using, for example, a storage device that is communicably connected to the learning data creation device 10 via a communication network, instead of the auxiliary storage device 408.

The learning data creation device 10 according to the embodiment of the present invention can realize the above-mentioned various processes by having the hardware configuration shown in FIG. In the example shown in FIG. 6, the case where the learning data creating device 10 according to the embodiment of the present invention is realized by one device (computer) has been described, but the present invention is not limited to this. The learning data creation device 10 in the embodiment of the present invention may be realized by a plurality of devices (computers).

<Summary>
As described above, the learning data creation system 1 according to the embodiment of the present invention uses the information obtained from the virtual captured image (that is, the information that can be acquired or calculated by the three-dimensional simulator 100) as the teacher information in the actual captured image. By adding it, learning data is created. Therefore, in the learning data creation system 1 according to the embodiment of the present invention, learning data can be easily created without performing work such as manually adding teacher information to an actual photographed image. You will be able to do it. In particular, for example, even when the number of teacher information is large (for example, when the number of types of objects is large or the number of categories is large), learning data can be easily created.

Further, for example, in the case of performing semantic segmentation, in the learning data creation system 1 according to the embodiment of the present invention, the three-dimensional simulator 100 acquires the boundary line of the object (object), so that the segmentation of the object can be performed with high accuracy. You will be able to do it.

Further, for example, in the learning data creation system 1 according to the embodiment of the present invention, even if the teacher information is difficult to be manually given, such as the depth and the posture of the object, the teacher information is included. Training data can be easily created.

Moreover, in the learning data creation system 1 according to the embodiment of the present invention, for example, the user simply captures an arbitrary range with the camera device 20 while moving in the real space, and the actual captured image and the actual captured image. Since a virtual captured image corresponding to the above is created, a large amount of learning data can be easily created. Therefore, for example, a large amount of learning data can be obtained at low cost as compared with the case where teacher information is added to an actual photographed image by using crowdsourcing or the like.

Therefore, by using the learning data creation system 1 in the embodiment of the present invention, for example, recognition of a robot that cleans a room in a real space or cleans up an object in the room. A large amount of learning data used for learning an engine (ie, a machine learning model that performs tasks such as cleaning and tidying up a room) can be easily obtained.

In the embodiment of the present invention, as a preliminary preparation procedure, a virtual space is created and an object is arranged in the real space so as to correspond to the virtual space, but the present invention is not limited to this. For example, a virtual space may be created so as to correspond to the real space in which the object is actually arranged.

Further, in the embodiment of the present invention, the description has been made assuming that the actual photographed image and the virtual photographed image are still images, but the present invention is not limited to this. The actual shot image and the virtual shot image may be moving images.

Further, in the embodiment of the present invention, the set of the actual photographed image and the teacher information acquired from the three-dimensional simulator 100 is used as learning data, but the present invention is not limited to this. For example, the actual photographed image may be used as teacher information, and the set of the virtual photographed image and the teacher information (actually photographed image) may be used as learning data. In this case, it is possible to create learning data for a task of predicting an actual photographed image from a virtual photographed image created by the three-dimensional simulator 100.

The present invention is not limited to the above-described embodiment specifically disclosed, and various modifications and modifications can be made without departing from the scope of claims.

1 Learning data creation system 10 Learning data creation device 20 Camera device 30 Tracking device 100 Three-dimensional simulator 200 Learning data creation unit 300 Storage unit

Claims (8)

When the first object in the real space and the second object in the virtual space created by the three-dimensional simulator have the same coordinate system as the other space and the same origin in the coordinate system in each space. The first step of arranging so that they have the same coordinates,
A second procedure of photographing a predetermined range including the first object in the real space and creating a first image, and
A third procedure of drawing a predetermined range in the virtual space corresponding to the first image by the three-dimensional simulator and creating a second image, and
At least one of the information obtained from the second image and the information generated by the three-dimensional simulator is added to the first image as teacher information to create learning data for a machine learning model. A method for creating learning data, characterized in that a computer executes the fourth procedure. The first creation procedure for creating a virtual space in which one or more objects are arranged by a three-dimensional simulator, and
In response to the creation of the first image taken within a predetermined range of the real space corresponding to the virtual space, a second image drawn by the three-dimensional simulator in the range of the virtual space is created. The second creation procedure to be done and
A third creation procedure for creating learning data used for learning a predetermined machine learning model by adding predetermined information obtained from the second image to the first image as teacher information. ,
A learning data creation method characterized by a computer performing. The second creation procedure is
The first image is obtained by photographing the inside of the virtual space with the virtual camera device of the posture at the position in the virtual space by using the information indicating the position and the posture of the camera device that created the first image. The method for creating learning data according to claim 2, wherein the image of 2 is created. The third creation procedure is
A claim characterized in that the learning data represented by either a set of the first image and the predetermined information or a set of the second image and the first image is created. The learning data creation method according to 2 or 3. The learning data creation method according to any one of claims 2 to 4, wherein the predetermined information is information acquired or calculated by the three-dimensional simulator. The above-mentioned predetermined information includes
Information indicating the contour line of an object represented by the three-dimensional model arranged in the range of the virtual space, information indicating the object name of the object, information indicating the state of the object, and the depth to the object. Information to be shown, information to show the posture of the object, information for an operating subject capable of performing a preset operation to perform a predetermined action with respect to the object, and an image area of all or a part of the second image. The method for creating learning data according to claim 5, wherein at least one piece of information is included. The first creation unit that creates a virtual space in which one or more objects are arranged by a three-dimensional simulator,
In response to the creation of the first image taken within a predetermined range of the real space corresponding to the virtual space, a second image drawn by the three-dimensional simulator in the range of the virtual space is created. The second creation part to do,
A third creation unit that creates learning data used for learning a predetermined machine learning model by adding predetermined information obtained from the second image to the first image as teacher information. ,
A learning data creation device characterized by having. The first creation procedure for creating a virtual space in which one or more objects are arranged by a three-dimensional simulator, and
In response to the creation of the first image taken within a predetermined range of the real space corresponding to the virtual space, a second image drawn by the three-dimensional simulator in the range of the virtual space is created. The second creation procedure to be done and
A third creation procedure for creating learning data used for learning a predetermined machine learning model by adding predetermined information obtained from the second image to the first image as teacher information. ,
A program characterized by having a computer execute.

Images