JP6887199B1 - Computer system, dataset creation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently improve the accuracy of machine learning by generating a large number of sphere images and their annotation data as a data set for machine learning from a small number of sphere images and their annotation data. A computer system for creating a data set for machine learning acquires a first sphere image obtained by photographing a sphere, designates a predetermined area of the sphere in the acquired first sphere image as annotation data, and acquires the image. Inverse UV mapping is performed on the first sphere image, a texture for making the first sphere image three-dimensional is generated, and the generated texture is UV-mapped to the shape of the sphere, and the sphere is used. The third-dimensional data of the above is generated, the generated three-dimensional data is rotated at a predetermined angle, and a plurality of second sphere images obtained by capturing the three-dimensional data at each angle are acquired, and the acquired second sphere image is acquired. The designated annotation data is set for each of the sphere images. [Selection diagram] Fig. 2

Description

The present invention relates to a computer system for creating a dataset for machine learning, a dataset creation method, and a program.

In recent years, artificial intelligence has been used in various fields. In particular, artificial intelligence is being utilized in the medical field. In such artificial intelligence, data desired by the user is collected, an annotation is specified for this data, and the data for which this annotation is specified is used as teacher data for machine learning to improve accuracy. Such machine learning requires a large data set.

In the case of machine learning using images, an image and an annotation specified for this image are required as a data set. As a method for increasing the number of images, for example, it is conceivable to generate a three-dimensional object from an image and use this three-dimensional object as data for a data set. As a technique for generating a three-dimensional object from an image, a technique for reproducing a person appearing in this RGB image in three dimensions from one RGB image is known (see Non-Patent Document 1).

shizuma, "Tips Development Diary", [online], July 3, 2019, [Search June 22, 2020], Internet <https://blog.seishin55.com/entry/2019/07/03/005723 >

However, in the data set in the medical field, the collection of medical images and their annotations require a high degree of specialization, and a large amount of time is required for the collection and annotation of medical images. Therefore, creating a data set for machine learning is costly.
The technique of Non-Patent Document 1 merely generates an image into a three-dimensional object, and is not for creating a data set for machine learning.

An object of the present invention is to generate a large number of sphere images and their annotation data as a data set for machine learning from a small number of sphere images and their annotation data, and to efficiently improve the accuracy of machine learning. And.

The present invention provides the following solutions.

The present invention is a computer system that creates a data set for machine learning.
A first sphere image acquisition means for acquiring a first sphere image obtained by photographing a sphere, and
Annotation data designation means for designating a predetermined area of the sphere in the acquired first sphere image as annotation data, and
A texture generation means that performs inverse UV mapping on the acquired first sphere image and generates a texture for making the first sphere image three-dimensional.
A three-dimensional data generation means that UV-maps the generated texture to the shape of the sphere and generates three-dimensional data of the sphere.
A second sphere image acquisition means for rotating the generated three-dimensional data at predetermined angles and acquiring a plurality of second sphere images obtained by capturing the three-dimensional data at each angle.
Annotation data setting means for setting the designated annotation data in each of the acquired second sphere images, and
Provide a computer system characterized by the provision of.

According to the present invention, a computer system that creates a data set for machine learning acquires a first sphere image obtained by photographing a sphere, and designates a predetermined region of the sphere in the acquired first sphere image as annotation data. Inverse UV mapping is performed on the acquired first sphere image, a texture for making the first sphere image three-dimensional is generated, and the generated texture is UV-mapped to the shape of the sphere. The three-dimensional data of the sphere is generated, the generated three-dimensional data is rotated at predetermined angles, and a plurality of second sphere images obtained by capturing the three-dimensional data at each angle are acquired and acquired. The designated annotation data is set in each of the second sphere images.

The present invention is in the category of systems, but other categories such as computers, methods, and programs also exhibit similar actions and effects according to the categories.

According to the present invention, a large number of sphere images and their annotation data are generated as a data set for machine learning from a small number of sphere images and their annotation data, and the accuracy of machine learning is efficiently improved. Is possible.

FIG. 1 is an overall configuration diagram of the data set creation system 1. FIG. 2 is a diagram showing a flowchart of a data set creation process executed by the computer 10. FIG. 3 is a diagram schematically showing annotation data designated for the first fundus image. FIG. 4 is a diagram schematically showing annotation data set in the second fundus image.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. It should be noted that this is just an example, and the technical scope of the present invention is not limited to this.

[Configuration of data set creation system 1]
A data set creation system 1 which is a preferred embodiment of the present invention will be described. The data set creation system 1 may be configured by a computer 10 (see FIG. 1), and creates a data set for machine learning (consisting of a spherical image (for example, a fundus image) and annotation data specified for the spherical image). Is a computer system. In the present specification, the sphere is described as a fundus. Further, the first sphere image and the second sphere image will be described as a first fundus image and a second fundus image, respectively.
The fundus image is an image of the fundus of a patient or the like. Annotation data is added to annotations (object detection, area extraction, attribute classification by image classification, etc.) specified in a predetermined area (area showing abnormality, area showing normality, other areas, etc.) in a spherical image. It may be data indicating a tag or the like).

The data set creation system 1 may include a fundus camera for photographing the fundus of a patient or the like, other terminals, and other computers and devices for recording a database in which a fundus image is registered. In this case, the data set creation system 1 is composed of the computer 10, the terminal, the computer, the device, or a combination of a plurality of the processes described later.

The computer 10 may be a computer having a server function, a mainframe, a personal computer, or the like, and is connected to the above-mentioned fundus camera, other terminals, computers, devices, etc. by data communication via a public network or the like. It is used to send and receive necessary data and perform various processes.

The computer 10 may be realized by, for example, one computer, or may be realized by a plurality of computers such as a cloud computer. The cloud computer in the present specification includes a computer that uses an arbitrary computer in a scalable manner when performing a specific function, and a plurality of functional modules for realizing a certain system, and uses the functions in any combination. It may be any of the above.

The computer 10 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as a control unit, and has other terminals and devices as a communication unit. A device for enabling communication, for example, a Wi-Fi (Wi-Fi (Wi-Fi) compatible device compliant with IEEE802.11) and the like are provided. Further, the computer 10 includes a data storage unit such as a hard disk, a semiconductor memory, a recording medium, and a memory card as a recording unit. Further, the computer 10 includes various devices and the like that execute various processes as a processing unit.

In the computer 10, the control unit reads a predetermined program to realize the first fundus image acquisition module 20 shown in FIG. 1 in cooperation with the communication unit. Further, in the computer 10, the control unit reads a predetermined program to realize the recording module 30 shown in FIG. 1 in cooperation with the recording unit. Further, in the computer 10, the control unit reads a predetermined program, and in cooperation with the processing unit, the annotation data designation module 40, the position information identification module 41, the texture generation module 42, and the three-dimensional data shown in FIG. The generation module 43, the second fundus image acquisition module 44, the estimation module 45, and the annotation data setting module 46 are realized.

In each module, the first fundus image acquisition module 20 realizes the processing of the first spherical image acquisition means, the recording module 30 realizes the processing of the recording means, and the annotation data designation module 40 realizes the annotation data designation means. The texture generation module 42 realizes the processing of the texture generation means, the three-dimensional data generation module 43 realizes the processing of the three-dimensional data generation means, and the second fundus image acquisition module 44 realizes the processing of the third. The processing of the two-sphere image acquisition means is realized, the guessing module 45 realizes the processing of the guessing means, and the annotation data setting module 46 realizes the processing of the annotation data setting means.

[Summary]
The outline of each process executed by the data set creation system will be described.

The computer 10 acquires a first sphere image in which a sphere (for example, the fundus of a patient or the like) is photographed.
For example, the computer 10 acquires this first sphere image from a fundus camera or other computer. The fundus camera transmits the fundus image taken by itself to the computer 10, and the other computers transmit the fundus image registered in the database recorded by the fundus in advance to the computer 10. The computer 10 acquires the first sphere image by receiving the fundus image as the first sphere image. Further, the computer 10 acquires the fundus image as the first sphere image from the database in which the fundus image recorded by the computer 10 is registered.

The computer 10 designates a predetermined area of the sphere in the acquired first sphere image as annotation data. The predetermined region is a region showing an abnormality, a region showing normality, another region, or the like in the first sphere image as described above. Further, the annotation data is data indicating the attribute classification of this predetermined area by object detection, area extraction, image classification, etc. as described above, the tags attached to the attribute classification, and the like.
The computer 10 accepts the designation of annotation data for this predetermined area from the annotator. The annotator inputs the annotation data to the predetermined area via the input device of the computer 10. The computer 10 will specify the input annotation data for the first sphere image.

The computer 10 performs inverse UV mapping on the acquired first sphere image to generate a texture for making the first sphere image three-dimensional. UV mapping expands an image (in this specification, a first sphere image) into a UV coordinate system and attaches a two-dimensional texture to a three-dimensional object. Inverse UV mapping indicates that in a rendered image of a 3D object, each pixel of the 3D object transforms which position in the texture it corresponds to. The texture is data for expressing the texture, pattern, unevenness, etc. necessary for making the sphere appearing in the sphere image three-dimensional.
The computer 10 may estimate and complement the non-photographed region of the sphere based on the shape of the sphere to generate this texture. For example, the computer 10 estimates a non-photographed area, which is a non-photographed area of the sphere, based on the shape of the sphere, and complements the estimated non-photographed area to the sphere shown in the first sphere image. You may generate a texture.

The computer 10 UV-maps the generated texture to the shape of the sphere and generates three-dimensional data of the sphere. The computer 10 performs UV mapping on the shape of the sphere based on the generated texture on the first sphere image, and generates a three-dimensional object of the sphere as three-dimensional data.

The computer 10 rotates the generated three-dimensional data at predetermined angles, and acquires a plurality of second sphere images obtained by capturing the three-dimensional data at each angle. The computer 10 rotates the generated three-dimensional object at a predetermined angle (every 1 degree, every 2 degrees, every 5 degrees, etc.). The direction of rotation may be centered on any of the X-axis, Y-axis, and Z-axis. The computer 10 acquires a second sphere image of the rotated three-dimensional object. The computer 10 will acquire a plurality of second sphere images taken for each angle.

The computer 10 sets the designated annotation data for each of the acquired second spheres. The computer 10 estimates the position information of the annotation data in each of the second sphere images based on the angle at which the three-dimensional object is rotated and the position information of the annotation data in the acquired first sphere image. The computer 10 sets annotation data in each of the second sphere images based on the position information of the annotation data in each of the estimated second sphere images.

The above is the outline of each process executed by the data set creation system 1.

[Data set creation process]
The data set creation process executed by the computer 10 will be described with reference to FIG. FIG. 2 is a diagram showing a flowchart of a data set creation process executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process.

First, the first fundus image acquisition module 20 (first spherical image acquisition means) acquires a first fundus image obtained by photographing the fundus (step S10). The first fundus image acquisition module 20 acquires the first fundus image from a fundus camera that photographs the fundus of a patient or the like or another computer that records a database in which the fundus images of the patient or the like are photographed. The fundus camera and other computers transmit the fundus images taken or recorded by themselves to the computer 10. The first fundus image acquisition module 20 acquires the first fundus image by receiving the fundus image as the first fundus image.
The first fundus image acquisition module 20 may be configured to acquire the fundus image recorded by the recording module 30 as the first fundus image.

Next, the annotation data designation module 40 (annotation data designation means) designates a predetermined area of the fundus in the acquired first fundus image as annotation data (step S11). The predetermined region is a region showing an abnormality as described above, a region showing normality, another region, or the like. Here, the region showing an abnormality is a region in which stenosis, bleeding, vitiligo, edema, etc. of capillaries are observed. Further, the annotation data is data indicating attribute classification by object detection, area extraction, image classification, etc. as described above, tags attached to the attribute classification, and the like.
The annotation data designation module 40 accepts the designation of annotation data for a predetermined area from the annotator. The annotator inputs via an input device or the like to specify the annotation data as shown in FIG. 3 described later. At this time, the annotator performs the attribute category of the predetermined area by surrounding the predetermined area or the like. The annotator further adds a tag to the attribute classification by inputting a tag for this predetermined area. The annotation data designation module 40 designates the input attribute classification and tag as annotation data.

Next, the position information specifying module 41 specifies the position information of the annotation data in the first fundus image (step S12). The position information specifying module 41 specifies the position information of the predetermined area designated by the process of step S11 described above as the position information of the annotation data. This position information may be indicated by an XY coordinate system such as (X1, Y1). When the predetermined area is a circle or the like, the position information specifies one or more points on the center and the circumference of the circle as the position information of the predetermined area. Further, in this position information, when the predetermined area is a rectangle or the like, each vertex is specified as the position information of the predetermined area.
Note that this position information is not limited to the above-mentioned example, and can be changed as appropriate.

FIG. 3 is a diagram schematically showing an example of annotation data designated by the annotation data designation module 40. As a result of the process of step S11 described above, in the first fundus image 100, a region 110 showing an abnormality and a tag 120 for this region 110 are designated as predetermined regions. The annotation data designation module 40 designates the area 110 and the tag 120 as annotation data.
As a result of the process of step S12 described above, the position information specifying module 41 specifies the position information 130 of this region 110 placed in the first fundus image 100 in the XY coordinate systems such as (X1, Y1) and (X2, Y2). ..

Returning to FIG. 2, the process following the process in step S12 will be described.
The recording module 30 (recording means) records the position information of the annotation data specified by the process of step S11 described above and the annotation data specified by the process of step S12 described above (step S13). The computer 10 uses the annotation data recorded by this process and the position information of the annotation data in the process of step S17 (estimation process of the position information of the annotation data) described later.

Next, the texture generation module 42 (texture generation means) performs inverse UV mapping on the acquired first fundus image to generate a texture for making the first fundus image three-dimensional (step S14). The texture generation module 42 performs inverse UV mapping on the first fundus image acquired by the process of step S10 described above. Inverse UV mapping converts the position of each pixel of the three-dimensional object in the texture in the rendered image of the three-dimensional object as described above. The texture is data for reproducing the texture, pattern, unevenness, etc. necessary for three-dimensionalizing the fundus reflected in the first fundus image as described above.
The texture generation module 42 generates a texture for making the first fundus image three-dimensional based on the result of the inverse UV mapping. At this time, the texture generation module 42 estimates the non-photographed area of the fundus that is not shown in the first fundus image based on the shape of the fundus, and complements the fundus. The texture generation module 42 generates a texture for three-dimensionalizing the fundus based on the fundus and the fundus that complements the area to be photographed in the acquired first fundus image. The texture generation method may be one in which data necessary for three-dimensionalization is pasted on each pixel of the fundus in the first fundus image and the fundus in the non-photographed region.

Next, the three-dimensional data generation module 43 (three-dimensional data generation means) UV-maps the generated texture to the shape of the fundus to generate three-dimensional data of the fundus (step S15). In UV mapping, as described above, an image (in this specification, a first fundus image) is developed in a UV coordinate system, and a two-dimensional texture is attached to a three-dimensional object.
The three-dimensional data generation module 43 UV-maps the texture generated by the process of step S14 to the shape of the fundus. The three-dimensional data generation module 43 provides three-dimensional data of the fundus, such as the patient's fundus, based on the results of UV mapping and the fundus that complements the fundus and the area to be imaged in the first fundus image. Create a three-dimensional object. That is, this three-dimensional object is a sphere that complements not only the photographed fundus portion but also the non-photographed region portion of the fundus.

Next, the second fundus image acquisition module 44 (second sphere image acquisition means) rotates the generated three-dimensional data at predetermined angles, and captures the three-dimensional data at each angle. Acquire an image (step S16). The computer 10 rotates the generated three-dimensional object at a predetermined angle. The predetermined angle is a predetermined numerical step such as every 1 degree, every 2 degrees, every 5 degrees as described above. The direction of rotation may be centered on any of the X-axis, Y-axis, and Z-axis as described above. The computer 10 photographs a three-dimensional object rotated by a predetermined angle. The second fundus image acquisition module 44 acquires the captured image of the three-dimensional object as the second fundus image. The second fundus image acquisition module 44 acquires the angle and direction of rotation in accordance with the second fundus image. The second fundus image acquisition module 44 acquires a plurality of second fundus images by acquiring the second fundus image for each angle.
When the computer 10 is rotated to a predetermined angle, it is desirable to suppress this angle and rotate it to suppress an error between the complemented non-photographed area and the actual state.

Next, the guessing module 45 (guessing means) guesses the position information of the annotation data in each of the second fundus images based on the rotated angle and the recorded position information (step S17). The estimation module 45 takes each of the second fundus images based on the angle and direction in which the three-dimensional object is rotated by the process of step S16 described above and the position information of the annotation data recorded by the process of step S13 described above. Guess the position information of the annotation data that can be placed in. The estimation module 45 calculates the position information of the annotation data in the rotated three-dimensional object based on the rotation angle and direction of the recorded annotation data, and the estimation module 45 calculates the calculated annotation data. Based on the position information of, the position information of the annotation data in the second fundus image corresponding to the rotated angle and direction is estimated. The guessing module 45 executes this process for each of the second fundus images.

Next, the annotation data setting module 46 (annotation data setting means) sets the designated annotation data in each of the acquired second fundus images (step S18). The annotation data setting module 46 sets the annotation data specified by the process of step S11 described above in each of the second fundus images acquired by the process of step S16 described above. Specifically, the annotation data setting module 46 sets the annotation data for the second fundus image based on the position information of the annotation data in each of the second images estimated by the process of step S17 described above. The annotation data setting module 46 executes this process for each of the second fundus images.

FIG. 4 is a diagram schematically showing an example of annotation data set in the second fundus image by the annotation data setting module 46. FIG. 4 shows two types of second fundus images 200 in which a three-dimensional object is rotated in the positive direction about the Z axis in the right-handed coordinate system. The annotation data setting module 46 sets a region 210 indicating an abnormality and a tag 220 for this region 210 in each of the second fundus images 200, as in FIG. 3 described above. The annotation data setting module 46 sets the area 210 and the tag 220 as annotation data.

The computer 10 creates the second fundus image acquired in this way and the annotation data set in the second fundus image as a data set for machine learning. As a result, the computer 10 creates a plurality of data sets for machine learning from one fundus image. The computer 10 performs machine learning using the created data set as correct answer data, so that a large number of fundus images and their annotation data are obtained from a small number of fundus images and their annotation data as a data set for machine learning. It is possible to efficiently improve the accuracy of machine learning.

The above is the data set creation process executed by the data set creation system 1.

Although the present invention has been described based on the fundus as an example of a sphere in the above description, it can also be applied to a sphere other than the fundus.

The above-mentioned means and functions are realized by a computer (including a CPU, an information processing device, and various terminals) reading and executing a predetermined program. The program may be provided, for example, in the form of software as a service (Software as a Service) provided from a computer via a network, a WEB service, or a cloud service. Further, the program may be provided, for example, in a form recorded on a computer-readable recording medium.

Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments described above. In addition, the effects described in the embodiments of the present invention merely list the most preferable effects arising from the present invention, and the effects according to the present invention are limited to those described in the embodiments of the present invention. is not it.

1 Data set creation system,
10 Computer 20 1st fundus image acquisition module 30 Recording module 40 Annotation data specification module 41 Position information identification module 42 Texture generation module 43 3D data generation module 44 2nd fundus image acquisition module 45 2nd fundus image acquisition module 46 Annotation data setting module

Claims (6)

A computer system that creates datasets for machine learning
A first sphere image acquisition means for acquiring a first sphere image obtained by photographing a sphere, and
Annotation data designation means for designating a predetermined area of the sphere in the acquired first sphere image as annotation data, and
A texture generation means that performs inverse UV mapping on the acquired first sphere image and generates a texture for making the first sphere image three-dimensional.
A three-dimensional data generation means that UV-maps the generated texture to the shape of the sphere and generates three-dimensional data of the sphere.
A second sphere image acquisition means for rotating the generated three-dimensional data at predetermined angles and acquiring a plurality of second sphere images obtained by capturing the three-dimensional data at each angle.
Annotation data setting means for setting the designated annotation data in each of the acquired second sphere images, and
A computer system characterized by being equipped with. The texture generation means estimates and complements the non-photographed region of the sphere based on the shape of the sphere, and generates the texture.
The computer system according to claim 1. A recording means for recording the position information of the annotation data in the first sphere image, and
A guessing means for estimating the position information of the annotation data in each of the second sphere images based on the rotated angle and the recorded position information.
Further prepare
The annotation data setting means sets annotation data based on the estimated position information of the annotation data in each of the second sphere images.
The computer system according to claim 1. The sphere is the fundus.
The computer system according to claim 1. Creating a dataset for machine learning A dataset creation method performed by a computer system.
Steps to acquire the first sphere image of the sphere,
A step of designating a predetermined area of the sphere in the acquired first sphere image as annotation data, and
A step of performing inverse UV mapping on the acquired first sphere image to generate a texture for making the first sphere image three-dimensional, and
A step of UV mapping the generated texture to the shape of the sphere to generate three-dimensional data of the sphere, and
A step of rotating the generated three-dimensional data at a predetermined angle and acquiring a plurality of second sphere images obtained by capturing the three-dimensional data at each angle.
A step of setting the designated annotation data in each of the acquired second sphere images, and
A method for creating a data set, which comprises. On a computer that creates a dataset for machine learning,
Step to acquire the first sphere image of the sphere,
A step of designating a predetermined area of the sphere in the acquired first sphere image as annotation data,
A step of performing inverse UV mapping on the acquired first sphere image to generate a texture for making the first sphere image three-dimensional.
A step of UV mapping the generated texture to the shape of the sphere and generating three-dimensional data of the sphere.
A step of rotating the generated three-dimensional data at predetermined angles and acquiring a plurality of second sphere images obtained by capturing the three-dimensional data at each angle.
A step of setting the designated annotation data in each of the acquired second sphere images,
A computer-readable program for running.

Images