JP2020166473A - Trained model generation device and program and trained model generation system - Google Patents

Abstract

To provide a trained model generation device and the like that can generate appropriate trained models by using the data to be trained based on the 3D model.SOLUTION: A device comprises an image data generation part 112 that generates pseudo-image data based on the 3D data, an annotation data generation part 114 that generates annotation data based on the 3D data, a learning data generation part 110 that generates data for learning including the pseudo-image data and the annotation data, and a trained model generation part 102 that uses the learning data to generate a learned model for inferring data for output from the input data.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to a trained model generator and the like.

Conventionally, in machine learning, in order to generate a trained model, learning data (teacher data, training data) including data to be trained and correct answer data (annotation data) for the data to be trained is used. It had to be put into the learning model.

For example, an invention that generates a trained model by adding correct answer data to training image data (see, for example, Patent Document 1), or a trained model is generated by giving a sample image and correct answer data. The invention (see, for example, Patent Document 2) is disclosed.

JP-A-2018-5640 Japanese Unexamined Patent Publication No. 2010-21146

Conventionally, in order to generate a trained model, it has been necessary for the user to prepare training data. In order to improve the accuracy of the trained model, the user had to prepare a large amount of data to be trained.

Moreover, even if the data to be learned is prepared, it is necessary for the user to annotate the data to be learned. There is a problem that if the amount of data to be learned becomes enormous, the amount of work to add annotations also increases.

An object of the present disclosure is to provide a trained model generator or the like capable of generating an appropriate trained model by using data of a learning target based on a 3D model.

The trained model generator according to one aspect of the present disclosure includes an image data generation unit that generates pseudo image data based on 3D data, an annotation data generation unit that generates annotation data based on 3D data, and the pseudo image. A training data generation unit that generates training data including data and the annotation data, and a trained model that generates a trained model for inferring output data from input data using the training data. It has a generator.

The program according to one aspect of the present disclosure includes an image data generation function that generates pseudo image data based on 3D data, an annotation data generation function that generates annotation data based on 3D data, and the pseudo image data. A training data generation function that generates training data including the annotation data, and a trained model generation that generates a trained model for inferring output data from input data using the training data. Realize the function.

The trained model generation system according to one aspect of the present disclosure includes an image data generation means that generates pseudo image data based on 3D data, an annotation data generation means that generates annotation data based on 3D data, and the pseudo image. A training data generation means for generating training data including data and the annotation data, and a trained model for generating a trained model for inferring output data from input data using the training data. It is equipped with a generation means.

According to the present disclosure, by using the data to be learned based on the 3D model, it is possible to generate an appropriate trained model in a short time without much manpower.

It is a figure for demonstrating the whole system in this Embodiment. It is a figure for demonstrating the outline with 3D data, pseudo image data, and annotation data in this embodiment. It is a figure for demonstrating the functional structure of the trained model generator in this embodiment. It is operation flow diagram for demonstrating the process of generating the trained model in this embodiment. It is a figure for demonstrating the operation example in this embodiment. It is a figure for demonstrating the outline of the object inference apparatus in this embodiment. It is a figure for demonstrating the modification of this embodiment. It is a figure for demonstrating another embodiment.

Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. The following embodiments are examples for explaining the present invention, and the technical scope of the invention described in the claims is not limited to the following description.

[1. System configuration]
FIG. 1 is a diagram for explaining a system A including a trained model generator. The outline of the system A will be described below. The system A includes a trained model generation device 1 and an object inference device 3.

First, an outline of the operation of the trained model generator 1 will be described. The user or system determines the structure of the learning data (10). As the trained model (20), what kind of model is generated is determined by the user or the system.

The material data (12) is data that serves as a material for generating image data (pseudo-image data) to be learned. In this embodiment, 3D data is used as a material. The 3D data includes, for example, at least 3D model data, and includes one or more image data of parameters and materials.

The 3D model data is data for generating a 3D model of an object. For example, the coordinates on a certain coordinate space are composed of the coordinates of the vertices of the object and the coordinates of the feature points. The parameters are also used when creating an object from a 3D model. For example, as parameters, the angle, inclination, magnification, color, material, reference coordinate space data, and the like of the object can be used. The image data of the material includes a texture image of the object (for example, a wood grain image, an image showing the metal material, an image according to the type of crystal, etc.) and a background image (for example, when the object is underwater). And various images such as (images that are the background of them, such as when they are in a landscape) are available.

That is, the 3D data may include at least 3D model data and may further include parameters. Further, as described above, the image data of the material may be included in the 3D data, or may be prepared separately from the 3D data. Further, the information required for 3D data is not limited to these data, and may further include information that can be easily conceived by those skilled in the art.

By executing the learning data generation process (14), the trained model generation device 1 executes the learning data (14), and the learning data (16) corresponding to the structure of the learning data determined by the user or the system from the material data (12). To generate. The learning data includes image data (pseudo-image data) and annotation data which is correct answer data.

The image data (pseudo image data) in the present embodiment is a concept that includes not only still image data but also animation data and moving image data.

Further, the annotation data in the present embodiment is the coordinates indicating the position of the object included in the image data (real image data or pseudo image data), the type of the object (number for classification, and the number for classification). Name etc.).

For example, the relative coordinates of the object generated based on the 3D data are converted into the spatial coordinates required for learning and become annotation data.

Further, the trained model generation device 1 executes an image data generation process for generating image data and an annotation data generation process for generating annotation data as training data generation processing.

The trained model generation device 1 executes an image data generation process, and generates pseudo image data by using, for example, the 3D model data included in the material data (12) or the image data. Further, the trained model generation device 1 executes an annotation data generation process and uses the 3D model data to generate annotation data for the pseudo image data.

FIG. 2 is a diagram for explaining pseudo image data generated in the learning data generation process and annotation data. As an example, a case where pseudo image data including a crystal image is generated will be described with reference to this figure.

FIG. 2A is a diagram schematically showing pseudo image data for one crystal. The trained model generation device 1 executes an image data generation process and generates pseudo image data based on the 3D data (3D model data). Further, the trained model generation device 1 may further add texture data based on the image data included in the material data (12) to the pseudo image data in the image data generation process. Further, the trained model generation device 1 may determine the size, rotation, direction, etc. at any time according to parameters when generating pseudo image data in the image data generation process.

The trained model generation device 1 executes the annotation generation process and generates annotation data for the pseudo image data. The annotation data can be the position of the crystal image included in the pseudo image data when the crystal generated from the 3D model data and the parameters is the object. For example, it is possible to calculate that the crystal of FIG. 2A exists at the position (x1, y1) − (x2, y2) in the pseudo image data.

That is, the 3D model data includes the coordinates of the vertices of the object shown in FIG. 2 (a) (in the case of a rectangular parallelepiped, at least three three-dimensional coordinates, and in the case of other three-dimensional shapes, the three-dimensional coordinates of each vertex). Shown. Annotation data is obtained by converting this into two-dimensional coordinates including pseudo-image data or three-dimensional coordinates.

As a result, in the annotation generation process, for example, annotation data indicating the position of the crystal included in the pseudo image data of FIG. 2B can be generated at any time. For example, in the trained model generation device 1, the annotation generation process generates annotation data such as pseudo image data P10, P12, P14 from 3D model data and parameters.

The learning AI program (18) generates a trained model (20) by using the learning data (16). Here, as a method of generating a trained model from the training data, any known machine learning method is used. For example, as a machine learning method, a method such as classification, segmentation, object protection, or key point protection utilizing deep learning can be used.

The trained model (20) generated by the trained model generation device 1 can be used in any device. The trained model of the present embodiment may be used as a program module which is a part of artificial intelligence software.

The object inference device 3 is a device that outputs output data inferred from input data by using a trained model from input data. For example, the object inference device 3 uses the trained model (20) generated by the trained model generation device 1.

Specifically, the AI program (30) uses the trained model (20) when recognizing or inferring the input input data (image data, metadata, etc.). As a result, the AI program (30) can output the recognized result data (34) (output as output data).

According to this embodiment, a more appropriate trained model (20) can be generated by efficiently generating the training data (16). Hereinafter, the processing in this embodiment will be described.

[2. Functional configuration]
The functional configuration of the trained model generator 1 will be described with reference to FIG.

The control unit 100 controls the entire trained model generation device 1. The control unit 100 realizes various functions by reading and executing various programs stored in the storage unit 120, and is composed of one or a plurality of arithmetic units (for example, a CPU (Central Processing Unit)). There is.

Further, the control unit 100 operates as the following functional unit by executing the program stored in the storage unit 120.

Further, the control unit 100 realizes the functions of the learning data generation unit 110 that generates the learning data and the trained model generation unit 102 that generates the trained model.

The learning data generation unit 110 generates learning data based on the pseudo image data and the annotation data.

The image data generation unit 112 executes the image data generation process. The image data generation unit 112 generates pseudo image data to be learned based on the 3D model data stored in the 3D model data storage area 122 by referring to parameters as necessary.

Further, the image data generation unit 112 may further use the parameters stored in the parameter storage area 124 when generating the pseudo image data.

Further, the image data generation unit 112 may further use the texture image data or the background image data stored in the image data storage area 126 when generating the pseudo image data.

Specifically, the image data generation unit 112 uses the texture image data when giving pseudo image data attributes such as texture and material.

Further, when the 3D model is a human being or the like, the image data generation unit 112 may generate pseudo image data by using real image data such as a mountain or a lake behind the human as background image data.

The annotation data generation unit 114 executes the annotation data generation process. The annotation data generation unit 114 generates annotation data based on 3D model data of pseudo image data and parameters.

The annotation data generation unit 114 can obtain the position of the object included in the pseudo image data based on the 3D model data, and the position becomes the annotation data. In addition, the annotation data generation unit 114 can also generate annotation data based on the pseudo image data generated by referring to the parameters from the 3D model data.

The parameter determination unit 116 determines the parameters used when the image data generation unit 112 generates pseudo image data. For example, based on the 3D model data, the parameters are determined so that pseudo image data subjected to image processing such as rotation, enlargement, reduction, and inversion is generated. Further, the parameter determination unit 116 determines different parameters by executing the same 3D model data a plurality of times. As a result, the image data generation unit 112 can generate different pseudo image data. Further, the annotation data generation unit 114 can also generate different annotation data in the same manner.

The parameter determination unit 116 may arbitrarily determine the parameter by the user, or may determine the parameter based on a predetermined process. For example, the parameter may be changed for each constant value, or a random number may be used.

The trained model generation unit 102 generates a trained model from the training data. The trained model generation unit 102 stores the generated learning model in the trained model storage area 130.

Here, the process of generating the trained model in the present embodiment is not only the process of generating a new trained model based on the training data, but also the trained model stored in the trained model storage area 130. It includes the process of updating.

The storage unit 120 is a functional unit in which various programs and various data necessary for the operation of the trained model generation device 1 are stored. The storage unit 120 is composed of, for example, a semiconductor memory SSD (Solid State Drive), an HDD (Hard Disk Drive), or the like.

The 3D model data storage area 122 stores 3D model data (3D model data) for generating pseudo image data including an object. Further, the parameter storage area 124 stores the parameters corresponding to the 3D model data. The parameters stored in the parameter storage area 124 are used by the parameter determination unit 116, and newly determined parameters are stored. Further, the parameters stored in the parameter storage area 124 may be stored in advance corresponding to the 3D model data, or may be stored when the 3D model data is read.

The image data storage area 126 stores image data. For example, the image data storage area 126 can store various image data such as pseudo image data, texture image data, and background image data.

Annotation data storage area 128 stores annotation data corresponding to pseudo image data. Specifically, the annotation data output from the annotation data generation unit 114 is stored.

The information including the pseudo image data stored in the image data storage area 126 and the annotation data stored in the annotation data storage area 128 becomes the learning data.

The trained model storage area 130 stores the trained model output from the trained model generation unit 102. The trained model storage area 130 stores one or more trained models.

The data and information stored in the storage unit 120 described above may be stored in an external storage device or storage medium connected to the learned model generation device 1, or are connected via the communication unit 160. It may be stored in NAS (Network Attached Storage) or a storage area on a cloud server.

The input unit 140 receives an input of an operation instruction from the user, and the output unit 150 outputs an instruction to be written to the user. For example, the input unit 140 is a keyboard or a mouse. Further, the output unit 150 is a display device (for example, a liquid crystal display or the like).

Further, the input unit 140 and the output unit 150 may be integrated into a touch panel. Further, the input unit 140 and the output unit 150 may be implemented as normal interfaces, and can be realized by connecting an external device to, for example, USB (Universal Serial Bus).

The communication unit 160 communicates with an external device, a server, or the like. For example, it connects to a LAN (Local Area Network) by a network interface and communicates with other devices and the like. In addition to that, it may be realized by wireless LAN or LTE (Long Term Evolution) / 5G communication.

In addition, the communication unit 160 may also be an interface for receiving image data from the camera device. In this case, it is realized by an interface such as USB or HDMI (registered trademark) (High-Definition Multimedia Interface).

Further, the input unit 140, the output unit 150, and the communication unit 160 may be mounted one or more as necessary, and may not be mounted.

Further, by storing each data and program stored in the storage unit 120 in the storage area provided on the cloud, the processing of the present embodiment may be executed via the network. .. In this case, data (for example, 3D model data, image data, parameters, etc.) may be received from the terminal device as needed, and the learning data generation unit 110 and the trained model generation unit 120 may be executed on the cloud. ..

In addition, some functions may be realized by hardware or software. The trained model generator 1 may further have necessary functions in addition to the above-described configuration.

[3. Process flow]
The process executed by the control unit 100 (learned model generation unit 102) in the present embodiment will be described with reference to FIG.

The image data generation unit 112 inputs 3D model data from the 3D model data storage area 122 (step S102). Here, when there is texture image data, background image data, or the like, the image data generation unit 112 also inputs the image data from the image data storage area 126.

Subsequently, the parameter determination unit 116 determines the parameter (step S104). The parameter determination unit 116 determines the parameters corresponding to the 3D model data, but predetermined parameters may be used. When generating pseudo image data only from 3D model data, it is not necessary to use the parameters.

In addition to the above, the image data generation unit 112 may consider other methods for handling (using) the parameters. Hereinafter, how to use the parameters in the image data generation unit 112 will be described.

(1) Using the stored parameters The image data generation unit 112 inputs the parameters together with the 3D model data in step S102. In this case, the parameter corresponding to the 3D model data is stored in the parameter storage area 124. The image data generation unit 112 reads and uses the parameters stored in the parameter storage area 124 at any time in step S104.

(2) Using the initial parameters The image data generation unit 112 inputs the parameters together with the 3D model data in step S102. In this case, the parameter corresponding to the 3D model data is stored in the parameter storage area 124 (initial state). Further, the parameter determination unit 116 determines a new parameter in step S104 based on the parameter in the initial state. The image data generation unit 112 uses the parameters determined by the parameter determination unit 116 from the second time onward. The parameter determination unit 116 may use the parameters stored in advance on the way.

Subsequently, the image data generation unit 112 generates image data (pseudo image data) based on the 3D model data and the determined parameters (step S106). At this time, if there is image data of the material such as texture image data or background image data, the image data generation unit 112 also uses the image data of the material. The image data generation unit 112 stores the generated pseudo image data in the image data storage area 126. When the parameters are not used as described above, pseudo image data is generated from the 3D model data or the 3D model data and the image data of the material.

Subsequently, the annotation data generation unit 114 generates annotation data based on the 3D data by executing the annotation data generation process (step S108). The annotation data generation unit 114 stores the generated annotation data in the annotation data storage area 128.

Subsequently, the control unit 100 (learning data generation unit 110) generates learning data based on the pseudo image data and the annotation data (step S110).

The trained model generation unit 102 generates a trained model based on the training data (step S112). The trained model generation unit 102 stores the trained model in the trained model storage area 130. Further, the trained model generation unit 102 may perform a process of updating the trained model when the trained model already exists in the trained model storage area 130.

Here, when a trained model is generated using further different parameters, the trained model generation unit 102 repeatedly executes the process from step S104 (step S114; Yes → step S104). At this time, the parameter determination unit 116 may determine the parameters at random, or may determine the parameters by shifting them by a fixed amount. For example, the parameter determination unit 116 may determine the parameter as one of the parameters so as to shift the rotation angle by 90 degrees.

Here, the trained model generation unit 102 generates (updates) the trained model only with the training data based on the pseudo image data, but the trained model is based on the image data (real image data) of the real image. May be updated.

When generating learning data based on real image data (step S116; Yes), for example, the control unit 100 receives the real image data from the communication unit 160 and stores it in the image data storage area 126. The actual image data stored in the image data storage area 126 is used for processing.

Subsequently, the annotation data generation unit 114 creates annotation data corresponding to the actual image data (step S120). As the annotation data here, for example, the user may confirm the actual image data and generate the annotation data. Further, annotation data may be generated by using another image recognition function.

Then, the control unit 100 (learning data generation unit 110) generates learning data based on the actual image data input in step S118 and the annotation data created in step S120 (step S122).

The trained model generation unit 102 executes the trained model generation process using the learning data generated in step S122 (step S130). By this process, the trained model already generated in step S112 will be updated.

By executing the process of FIG. 4 in this way, a trained model based on the pseudo image data is generated. Further, by generating the trained data based on the actual image data, a trained model with higher accuracy is generated.

FIG. 5 is a diagram showing an image of learning data. For example, in the past, it was necessary for an operator to manually add annotations from the image of FIG. 5A.

However, according to the present embodiment, the annotation of the individual pseudo image data is automatically calculated. Therefore, as shown in FIG. 5B, annotation data is automatically generated without the operator manually adding annotations.

It should be noted that the object inference processing using the generated trained model can be executed. FIG. 6 is a diagram showing an example of object inference processing. The object inference process is, for example, a process included in the AI program 30 of the object inference device 3.

The input unit 1000 inputs input data. As the input data, for example, image data (input image data) is input. Subsequently, the inference unit 1100 uses the trained model 1200 and the AI program 30 generated by the trained model generation unit 102 to determine the position of the object included in the input image data from the input image data. Make inferences. Subsequently, the output unit 1300 sets the data indicating the area including the object and the type of the object as the output data as a result of the inference. The inference unit 1100 uses the corresponding trained model 1200 and the AI program 30 depending on the object for the image data.

For example, if the object is a person, the inference unit 1100 infers the input image data input to the input unit 1000 by using the learned model learned about the person reflected in the image data. Further, if the object is a crystal, it is possible to infer the number and size of crystals included in the input image data. In this way, by using the trained model to which the present embodiment is applied, the object can be appropriately determined.

[4. Other embodiments]
Apart from the above-described embodiment, a trained model generation device, an object inference device, and the like may be realized by a server. FIG. 8 is a diagram illustrating a system B using a server-client model.

For example, the server 50 is composed of one or more servers. The server 50 can execute the trained model generation process described above (for example, FIG. 4).

The terminal device 60 connects to the server device 50 via the network NW. For example, the trained model generation process may be provided such as ASP (Application Service Provider) / SaaS (Software as a Service). For example, the terminal device 60 connects to the server device 50 using a Web browser or a dedicated application, and executes the trained model generation process.

At this time, the server device 50 may receive the material data 12 from the terminal device 60, generate the learned model 20, and return it to the terminal device 60. Further, the server device 50 may generate the learning data 16 used in the trained model generation process and return it to the terminal device 60. In this case, the terminal device 60 uses the received learning data 16 to generate a trained model. That is, the trained generation model generation process is executed by the server device 50 and the terminal device 60.

Further, the mobile terminal device 70 connects to the server device 50 via the network NW. For example, when the server device 50 executes the object inference process, the mobile terminal device 70 can infer the object included in the image captured by the camera.

As described above, the processes and data described in the present specification can be distributed and stored or distributed and processed within a technically consistent range. That is, for convenience of explanation, it is described as a trained model generation device and an object inference device, but it is not necessarily limited to being composed of one device.

Further, each process including the process executed by the learning data generation unit 110 and the trained model generation unit 102 may be provided by the application. For example, by installing and executing not only a server but also a commercially available computer (information processing device) or a portable terminal device (tablet, etc.), it will function as a trained model generator and an object inference device. ..

[5. Modification example]
Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and the design and the like within a range not deviating from the gist of the present invention are also within the scope of claims. include.

In the above-described embodiment, the crystal image data has been described as an example as the pseudo image data, but other image data may be used.

For example, FIG. 7 shows animal data as 3D data. In this case, the control unit 100 (learning data generation unit 110) generates pseudo image data from the 3D model data and parameters as 3D data of the animal.

In addition, the control unit 100 calculates the coordinates corresponding to the 3D model data and the parameters as the 3D data of the animal. Then, the control unit 100 (annotation data generation unit 114) generates annotation data so that the calculated coordinates are included. As a result, the control unit 100 (learning data generation unit 110) can generate learning data from the pseudo image data and the annotation data. As described above, various data such as people, automobiles, and buildings can be used as the pseudo image data.

Further, in the above-described embodiment, the pseudo image data is stored in the image data storage area 126, but the trained model generation unit 102 may use it as it is. That is, the pseudo image data may be temporarily stored in the storage unit 120, or may be generated each time the process is executed.

Further, in the above-described embodiment, the trained model generation device 1 is described as a single device such as a server device, but if necessary, it functions as a server device connected to a network or a device connected to each other. May be dispersed. It may be realized as a trained model generation system in which a 3D model is input and a trained model is output.

Further, the program that operates in each device in the embodiment is a program that controls an arithmetic unit such as a CPU (a program that causes a computer to function) so as to realize the functions of the above-described embodiment. Then, the information handled by these devices is temporarily stored in a temporary storage device (for example, RAM) at the time of processing, and then various ROMs regardless of the connection via a cable or the connection via a network. It is stored in the storage device of the HDD or SSD, and is read, modified / written by the CPU as needed.

Further, in the case of distribution to the market, the program can be stored and distributed in a portable recording medium, or transferred to a server computer connected via a network such as the Internet. In this case, it goes without saying that the storage device of the server computer is also included in the present invention.

100 Control unit 102 Learned model generation unit 110 Learning data generation unit 112 Image data generation unit 114 Annotation data generation unit 116 Parameter determination unit 120 Storage unit 122 3D model data storage area 124 Parameter storage area 126 Image data storage area 128 Annotation data Storage area 130 Trained model storage area

Claims (6)

An image data generator that generates pseudo image data based on 3D data,
Annotation data generator that generates annotation data based on 3D data,
A learning data generation unit that generates learning data including the pseudo image data and the annotation data.
A trained model generation unit that generates a trained model for inferring output data from input data using the training data, and a trained model generation unit.
A trained model generator characterized by being equipped with. The trained model generator according to claim 1, wherein the 3D data includes 3D model data and parameters. The trained model generation device according to claim 2, further comprising a parameter determination unit that determines the parameters when generating the pseudo image data and when generating the annotation data. It also has an acquisition unit that acquires actual image data.
The annotation data generation unit generates annotation data based on the actual image data, and then generates annotation data.
The learning data generation unit further generates learning data including the actual image data and the annotation data.
The trained model generator according to any one of claims 1 to 3, wherein the trained model generator is characterized in that. On the computer
An image data generation function that generates pseudo image data based on 3D data,
Annotation data generation function that generates annotation data based on 3D data,
A learning data generation function that generates learning data including the pseudo image data and the annotation data, and
A trained model generation function that generates a trained model for inferring output data from input data using the training data, and a trained model generation function.
A program to realize. Image data generation means that generates pseudo image data based on 3D data,
Annotation data generation means that generates annotation data based on 3D data,
A learning data generation means for generating learning data including the pseudo image data and the annotation data,
A trained model generation means for generating a trained model for inferring output data from input data using the training data, and a trained model generation means.
A trained model generation system characterized by being equipped with.