JP7300828B2 - Learning data generation system, learning data generation method, learning method for machine learning model - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law "Fiscal 2018 Empirical Research Project for Advancement and Efficiency of Prior Design Trademark Searches Utilizing Artificial Intelligence Technology, etc." Application date: July 27, 2018

The present invention relates to a learning data generation system, a learning data generation method, and a machine learning model learning method for generating learning data used for learning a machine learning model.

Conventionally, when similar image retrieval is performed for an illustration image (input image) composed of a plurality of types of parts images, points of interest, which are local features in figures of each parts image of the illustration image and the comparative image to be compared, are respectively Detecting.
Then, the degree of matching between the points of interest detected in each of the illustration image and the comparison image is obtained, and the degree of similarity is determined based on the degree of matching.
For example, it is disclosed that the outline shape, skeletal shape, constituent elements, and arrangement features are extracted from the illustration image, and the elements that characterize the entire illustration image are determined by individually weighting these features. (For example, Patent Document 1).

By the way, a machine learning model for discrimination (machine learning model for discrimination) may be used for this similarity judgment, and a large amount of learning data is required to train this machine learning model for discrimination. .
However, in the case of the determination method that compares the points of interest described above, there is a risk that points of interest other than the part images that are to be truly compared, i. It is difficult to determine the similarity of parts images).

JP-A-4-268969

However, in the case of searching for similar images using machine learning models, there are also illustration images formed by overlapping other general polygonal or circular figures or characters on the symbolic parts image. It is difficult to separate completely from the image.
In other words, when trying to separate or remove general polygonal or circular graphics or characters using a machine learning model, it is very troublesome to prepare a large amount of learning data necessary for learning the model. In addition, separation or removal from the symbol parts image may be imperfect for portions where other graphics are overlapped.

The present invention has been made in view of such circumstances, and in machine learning using deep learning, etc., it is possible to easily generate a large number of learning data used for learning of each machine learning model. An object of the present invention is to provide a learning data generation system, a learning data generation method, and a learning method for a machine learning model.

The present invention has been made to solve the above-described problems. A learning data generation system according to the present invention is a machine learning model that generates a modified image by modifying a predetermined input image according to a predetermined rule. a learning data generation system for generating learning data consisting of a learning input image and a learning changed image used for learning of a plurality of image databases storing part images serving as constituent elements of an image; a selection unit that selects the parts image from each; a composite image generation unit that generates a composite image combining the parts images selected by the selection unit and is used as the learning input image; and the parts image selected by the selection unit. and a modified image generation unit that generates the modified learning image composed of the specific parts image in the above.

The learning data generation system of the present invention comprises a symbolic graphic database in which symbolic part images are accumulated, which are symbolic part images given when the visual impression of the input image for learning is observed, in the image database; and a general graphic database in which general parts images, which are general parts images having no symbolism themselves, are stored to supplement the symbolicity of the image.

The learning data generation system of the present invention is characterized in that the image database includes a character/graphic database in which character part images, which are part images representing characters, are accumulated.

In the learning data generation system of the present invention, the input image for learning is composed of the symbol parts image and either or both of the general parts image and the character parts image, and the modified learning image is , wherein only the symbol parts image is arranged in the input image for learning.

The learning data generation system of the present invention is characterized by further comprising a shape expansion unit that deforms the shape of each part image.

In the learning data generation system of the present invention, the learning data generation system includes a generator and a discriminator, and the generator is a machine that extracts predetermined elements from the input image for learning and generates a generated image for learning. a learning model, wherein the classifier evaluates learning data consisting of the learning input image and the learning modified image, or learning data consisting of the learning input image and the learning generated image. It consists of

In the learning data generation system of the present invention, the machine learning model is semantic segmentation, the modified image for learning is selected for each type of the parts image in the input image for learning, and a label indicating the type is added. Characterized by changes being made.

The learning data generation method of the present invention comprises a learning input image and a learning modified image used for learning a machine learning model that generates a modified image by modifying a predetermined input image according to a predetermined rule. A learning data generation method in which a computer system generates learning data, comprising: a selection process of selecting the parts image from each of a plurality of image databases in which parts images constituting an image are stored; a composite image generation process of generating a composite image by combining the parts images, and using it as the input image for learning; and an image generation process.

The machine learning model learning method of the present invention is a learning method in which a computer system learns a machine learning model that generates a modified image by modifying a predetermined input image according to a predetermined rule. a selection process of selecting the parts image used for generating the learning input image, which is learning data for the predetermined input image, from each of a plurality of image databases storing element parts images; A composite image generation process of generating a composite image by combining the selected part images and using it as the learning input image, and generating a learning modified image composed of a specific part image in the part image selected in the selection process. and a learning process of inputting the learning input image and training the machine learning model outputting the learning modified image.

According to the present invention, in machine learning using deep learning or the like, a learning data generation system capable of easily generating a large number of learning data used for learning each machine learning model, and a learning data generation system. A method, a learning method for a machine learning model can be provided.

FIG. 1 is a diagram showing a configuration example of a learning data generation system according to a first embodiment of the present invention. FIG. 3 is a conceptual diagram illustrating the flow of learning data generation in the learning data generation system of the present embodiment; 3 is a diagram showing an example of learning data stored in a learning image data storage unit 20; FIG. FIG. 10 is a flowchart showing an operation example of generating learning data used for GAN learning. FIG. FIG. 4 is a conceptual diagram illustrating initial learning of a discriminator, which is a machine learning model; FIG. 4 is a conceptual diagram illustrating learning of a GAN generator and a discriminator using learning data; It is a figure which shows the structural example of the learning data generation system by the 2nd Embodiment of this invention. FIG. 3 is a conceptual diagram illustrating the flow of learning data generation in the learning data generation system of the present embodiment; FIG. 4 is a conceptual diagram illustrating a learning example of a machine learning model that determines the types of parts images in an input image to be input and assigns a label to each pixel in each parts image.

<First embodiment>
A learning data generation system according to a first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing a configuration example of a learning data generation system according to a first embodiment of the present invention. In this embodiment, generation of learning data for a GAN (Generative Adversarial Network) will be described as an example.
In FIG. 1, the learning data generation system 1 in this embodiment includes a control unit 11, a parts image selection unit 12, a character string generation unit 13, a shape extension unit 14, a composite image generation unit 15, an image display unit 16, a symbolic graphic database 17, a general graphic database 18, a character graphic database 19, and a learning image data storage unit 20, respectively. Also, the learning data generation system 1 may be configured to include a discriminator 501 and a generator 502 (see FIGS. 5 and 6), which will be described later.

The control unit 11 makes a control signal input from an input means (not shown) (screen selection by a keyboard or mouse) correspond to the control contents indicated by the control signal, and selects a parts image selection unit 12, a character string generation unit 13, a shape It outputs to each of the expansion unit 14, the composite image generation unit 15, and the image display unit 16. FIG. The control unit 11 also writes and stores the symbolic parts images, general parts images, and character parts images supplied from the outside into the symbolic graphic database 17, the general graphic database 18, and the character graphic database 19, respectively.

The parts image selection unit 12 displays a parts image selection screen on the display screen of the image display unit 16 . Here, in the present embodiment, the parts image is an image that is combined into one composite image and used to generate a learning input image in the learning data. In this embodiment, the data for learning is configured as a set of image data of each of the input image for learning and the modified image for learning.

FIG. 2 is a conceptual diagram illustrating the flow of learning data generation in the learning data generation system of this embodiment. 2A shows an image area 16S of learning data on the display screen of the image display unit 16. FIG. Further, in FIG. 2A, no parts image is displayed, and only the image area 16S is displayed. This image area 16S is sized by specifying the height and width of the area by a predetermined number of pixels in each of the vertical and horizontal directions.
FIG. 2(b) shows a state in which the symbolic parts image 101 is selected by the operator and the selected symbolic parts image 101 is placed at a predetermined position in the image area 16S on the display screen of the image display unit 16. there is Here, the symbolic part image is image data of a symbolic figure that gives an observer a visual impression of a feature when observing an arbitrary image, and has a relatively complicated shape compared to the general part image. This is an image with In this embodiment, a plurality of symbolic part images are written and stored in the symbolic figure database 17 in advance.

Next, in FIG. 2C, a general parts image 102 is selected by the operator, and the selected general parts image 102 is displayed at a predetermined position in the image area 16S on the display screen of the image display unit 16 as the symbol parts image 101. It shows a state where it is arranged with Here, the general parts image is a figure, such as a straight line, a curved line, or a polygonal shape, which is combined with the above-described symbolic parts image to give a general visual impression to the observer when generating a learning input image as a composite image. It is image data of a shape or a circular figure, and is an image having a relatively simple shape compared to the symbol parts image. That is, the general parts image is a general image that supplements the symbolism of the symbolic parts image and does not have symbolism itself. In this embodiment, a plurality of general parts images are written in advance and stored in the general graphic database 18 .

FIG. 2D shows that the character part image 103 is selected by the operator, and the selected character part image 103 is symbolized as a character string image at a predetermined position in the image area 16S on the display screen of the image display unit 16. It shows a state where it is arranged together with a parts image 101 and a general parts image 102 . Here, the character part image is an image of a single character such as kanji, hiragana, katakana, alphabets, numbers, etc., and is combined with the above-described symbol part image when generating a learning input image as a composite image. Also, the character part image 103 is used as a character string indicating a meaningful word, singularly or plurally, but is not limited to this and may be a meaningless character string. In this embodiment, character parts images generated by designating character decorations such as fonts and colors based on predetermined character codes (text format) are written in advance and stored in the character graphic database.

In this embodiment, FIG. 2(b) (that is, FIG. 2(e)) is used as the modified image for learning, and FIG. 2(d) is used as the input image for learning. That is, the learning data 100 is composed of a set of the learning input image shown in FIG. 2(d) and the learning modified image shown in FIG. 2(e).

Returning to FIG. 1, the character string generation unit 13 generates a character string image in the order in which the operator arranges each character code. For example, the character string image of "Nigonigo House" in FIG. The images are arranged and formed in order. When arranging as a character string, the character codes to be arranged in this character string may be specified at random, in which case a meaningless character string image is formed.
In addition, the character part images of "ni", "go", "ni", "go", "ha", "u", and "su" are selected by the operator's setting by the parts image selection unit 12. be. In the image area 16S, the arrangement of character decorations may be specified at random. In this case, a plurality of character string images are formed as character decorations from the designated number of character strings.

The shape extension unit 14 performs processing for changing the shape of each of the placed symbol parts images, general parts images, and character parts images in accordance with conditions input by the operator. Note that the shape extension unit 14 may randomly perform the process of changing the shape of the figure described above.
For example, in the case of a symbol parts image, enlargement, reduction, aspect ratio, thickness of line segments forming a figure, etc. are changed. In general parts images as well, enlargement, reduction, aspect ratio, thickness of line segments forming figures, etc. are changed. In the character part image, enlargement, reduction, aspect ratio, thickness of line segment forming the figure, font, color, etc. are changed. Note that the shape extension unit 14 may randomly perform the process of changing the shape of the character part image described above for each character part image in the character string of the character decoration.

The shape extension unit 14 changes the position of each figure of the symbol parts image, the general parts image, and the character parts image arranged in the image area 16S in accordance with the conditions input by the operator. The process of changing the position of each figure of the symbol parts image, the general parts image, and the character parts image may be performed randomly. Further, when the processing for changing the position of the figure is performed randomly, the image area 16S is divided into a plurality of sections, and one section in which one of the symbol parts image, the general parts image, and the character parts image is arranged. may be arranged so as to avoid overlap of each of the symbol parts image, the general parts image, and the character parts image by arranging another image different from one image in different sections.

The composite image generation unit 15 generates a composite image by compositing each of the symbol parts image, the general parts image, and the character parts image selected by the parts image selection unit 12 and arranged in the image area 16S. is an input image for learning.
Then, the composite image generation unit 15 sets an image in which only the symbol image is arranged in the image region 16S as a modified image for learning, configures learning data together with the input image for learning, and stores the image data for learning 20 as a modified image for learning. be written to and stored.
The image display unit 16 is an image display device, such as a color liquid crystal panel.

The symbolic figure database 17 is a database in which images of figures classified into the types of symbolic part images are accumulated.
The general figure database 18 is a database in which images of figures classified into types of general parts images are accumulated.
The character/figure database 19 is a database in which images of figures classified into types of character part images are accumulated. These symbol parts images, general parts images, and character parts images are used as constituent elements when constructing an input image for learning.

The learning image data storage unit 20 stores learning data written by the composite image generation unit 15 .
FIG. 3 is a diagram showing an example of learning data stored in the learning image data storage unit 20. As shown in FIG. In FIG. 3A, learning data 201 is formed by a learning input image 201a and a learning modified image 201b. In FIG. 3B, learning data 202 is formed by a learning input image 202a and a learning changed image 202b. In FIG. 3C, learning data 203 is formed by a learning input image 203a and a learning changed image 203b. In FIG. 3D, the learning data 204 is formed by the learning input image 204a and the learning changed image 204b.

Each learning input image in FIGS. 3A and 3B is formed using symbol parts images, general parts images, and character parts images. However, FIG. 3A is formed from character part images in which character strings are arranged in the horizontal direction. On the other hand, FIG. 3B is formed from character part images in which character strings are arranged in the vertical direction.
The input image for learning in FIG. 3(c) is composed of symbol parts images and deformed general parts images, and does not contain character parts images.
The input image for learning in FIG. 3(d) is composed of a symbol part image and a character string formed by arranging character part images (a character string of "Yamagoya Hotel"), and includes a general part image. not present.

As described above, the learning input images are stored in the general parts image stored in the general graphic database 18 and the character graphic database 19, with the symbolic parts image stored in the symbolic graphic database 17 as the core image. A composite image is formed by arranging one or both of the character part images.
On the other hand, the modified image for learning is formed as an image in which only symbol parts images are arranged.

In GAN, there are two types of machine learning models: a generator (generator 502 in FIG. 6) and a classifier (classifier 501 in FIGS. 5 and 6). It has a function (predetermined rule) of outputting a generated learning image obtained by extracting only the symbolic portion of the learning input image. Then, the control unit 11 configures the learning generated image obtained from the machine learning model of the generator (generator 502) as learning data together with the learning input image, and stores it in the learning image data storage unit 20. Write it down and store it.

In addition, the machine learning model of the classifier (classifier 501) receives learning data consisting of a learning input image and a learning modified image, or learning data consisting of a learning input image and a learning generated image. do. When the learning data is input to the discriminator 501, if the input learning data includes a modified learning image (that is, an image consisting only of the symbolic part images stored in the symbolic graphic database 17), has a function of judging that it is "genuine", and judging that it is "fake" if it contains a generated image for learning (that is, an image generated by the generator 502).

Then, as for the entire machine learning model (classifier 501 and generator 502), in the learning process, the classification (discrimination) of known learning images as “genuine” or “fake” is determined by the classifier 501. Compare the degree of "genuine" or "fake". In order to cause the generator 502 to generate a learning generated image whose determination result is "genuine" by deceiving the classifier 501 according to the degree of "fake"-likeness determined by the classifier 501, the determination result of the classifier 501 is changed. An output value (evaluation value described later) is fed back to the generator 502 .
In other words, when the discriminator 501 determines that a “fake” is likely to be a “fake”, the accuracy of the generated learning image generated by the generator 502 (an image obtained by extracting a portion that seems to be a characteristic part from the input image for learning) is low and the classifier 501 can easily identify that it is different from the modified image for learning (the image of the feature part to be finally extracted). feedback.

Furthermore, as the learning of the generator 502 progresses (it becomes possible to generate a learning generated image that deceives the classifier 501 and the determination result is “genuine”), the learning generated image generated by the generator 502 becomes a learning image. The modified image is gradually approximated. At this stage, the discriminator 501 cannot correctly determine whether the input learning data is “genuine” or “fake,” resulting in an erroneous determination.
In other words, when the classifier 501 erroneously determines the “fake” to be the “genuine”, the classifier 501 believes that the learning generated image generated by the generator 502 has high accuracy and is different from the learning modified image. If you are not sure, you can evaluate. In this way, the fact that the generator 502 generates the generated learning image in which the discriminator 501 erroneously determines the “fake” to be the “genuine” signifies the completion of the learning of the generator 502 .

Then, if a system for judging image similarity is configured using the generator 502, which is a learned model obtained by subjecting the machine learning model to machine learning, as described above, an unknown general image can be generated with high accuracy. (for example, an abstract image portion similar to an abstract parts image) can be extracted.

FIG. 4 is a flowchart showing an operation example of generating learning data used for GAN learning.
Step S101: The parts image selection section 12 reads out the symbolic parts images stored in the symbolic graphic database 17, and displays each of the read out symbolic parts images in an area near the image area 16S on the display screen of the image display section 16. indicate.

Step S102: The operator observes each symbolic part image displayed on the display screen of the image display unit 16, and selects a symbolic part image to be used for generating learning data from among these symbolic part images. .
Then, the operator moves the selected symbol parts image to a predetermined position in the image area 16S and arranges it.

Step S103: Next, the parts image selection unit 12 displays a notification on the display screen of the image display unit 16 to select whether or not to select a general parts image.
That is, the parts image selection unit 12 prompts the operator to determine whether or not to include the general parts image together with the symbolic parts image in the learning input image.
At this time, if the operator determines that the general parts image should not be included in the input image for learning, the operator performs an input indicating that the general parts image is not to be selected. On the other hand, when the operator determines that the general parts image should be included in the learning input image, the operator performs an input indicating selection of the general parts image.

Then, when the operator selects to arrange the general parts image in the learning input image, the parts image selection unit 12 advances the process to step S104.
On the other hand, when the operator selects not to arrange the general parts image in the learning input image, the parts image selection unit 12 advances the process to step S106.

Step S104: The parts image selection unit 12 reads the general parts images stored in the general figure database 18, and displays each of the read general parts images in an area near the image area 16S on the display screen of the image display unit 16. indicate.

Step S105: The operator observes each of the general parts images displayed on the display screen of the image display unit 16, and selects a general parts image to be used for generating learning data from among these general parts images. .
Then, the operator moves the selected general parts image to a predetermined position in the image area 16S and arranges it in correspondence with the position of the symbol parts image already arranged in the image area 16S.

Step S106: The parts image selection unit 12 displays a notification on the display screen of the image display unit 16 to select whether or not to select a character parts image.
That is, the parts image selection unit 12 prompts the operator to determine whether or not to include the character parts image together with the symbol parts image in the learning input image.
At this time, when the operator determines that the input image for learning does not include the character part image, the operator performs an input indicating that no character part image is to be selected. On the other hand, when the operator determines to include the character part image in the input image for learning, the operator performs an input indicating selection of the character part image.

Then, when the operator selects to arrange the character parts image in the input image for learning, the parts image selection unit 12 advances the process to step S107.
On the other hand, when the operator selects not to arrange the character parts image in the input image for learning, the parts image selection unit 12 advances the process to step S109.

Step S107: The part image selection unit 12 reads the character part images stored in the character/graphic database 19, and displays each of the read character part images in an area near the image area 16S on the display screen of the image display unit 16. indicate.

Step S108: The operator observes each of the character part images displayed on the display screen of the image display unit 16, and selects one or more character part images to be used for generating learning data from among these character part images. Select multiple.
Then, the character string generator 13 causes the operator to move each of the selected character part images to a predetermined position in the image area 16S in correspondence with the position of the symbol parts image already arranged in the image area 16S. A character string corresponding to the process (when a single character is selected, it becomes a character string of one character) is generated as one character string image.

Step S109: The composite image generation unit 15 generates a composite image by synthesizing the images in the image area 16S on the display screen of the image display unit 16, and uses this composite image as an input image for learning.
In addition, the composite image generating unit 15 generates an image of only the symbolic parts image as a modified image for learning from the image area 16S in which the symbolic parts image is arranged in step S102.
Then, the composite image generation unit 15 combines the generated learning input image and learning modified image, and writes and stores them in the learning image data storage unit 20 as learning data.

Further, here, the composite image generating unit 15 determines the top, bottom, left, and right layout positions in the image area 16S, the degree of overlap of each part image, etc., in the layout in which the symbol parts image, the general parts image, and the character string image are combined. may be displayed, and the symbol parts image, the general parts image, and the character string image may be arranged according to the layout pattern selected by the user. Note that the same processing may be performed on the symbol parts image.

Step S110: Next, the shape extension unit 14 displays a notification image on the image display unit 16 to prompt a determination as to whether or not to perform shape extension processing to transform the image in the image area 16S on the display screen of the image display unit 16. Display on the display screen.
That is, the shape extension unit 14 performs, for example, deformation of the general parts image, deformation of the character string image, movement of the arrangement position of the general parts image within the image region 16S, and image The operator is prompted to determine whether or not to execute shape expansion processing such as movement of the arrangement position within the area 16S and movement of the arrangement position of the symbol parts image within the image area 16S. It should be noted that the above-described shape expansion processing may be similarly performed on the symbol parts image.

Then, when the operator selects to execute the shape expansion process on the image within the image area 16S, the shape expansion unit 14 advances the process to step S111.
On the other hand, when the operator selects not to perform the shape expansion process on the image within the image area 16S, the parts image selection unit 12 advances the process to step S112.

Step S111: Next, the shape extension unit 14 performs data extension that transforms the figure with either or both of the general parts image and the character parts image arranged in the image area 16S in response to the operator's operation. process.
Here, as described above, the data extension processing includes deformation of the general parts image, deformation of the character string image, movement of the arrangement position of the general parts image within the image area 16S, This includes movement of the arrangement position, movement of the arrangement position within the image area 16S of the symbol parts image, and the like.

General part images can be transformed, for example, by changing the thickness and length of straight lines (including enlargement and reduction), changing line types such as dashed lines, wavy lines, and dashed-dotted lines, changing colors, and changing polygons. If so, processing such as changing the shape, changing the position relative to the symbol parts image, reversing or rotating, or a combination of these processing is included.
Transformation of the character string image includes, for example, changing the font of each character part image, bending the character string image, deleting the character part image in the character string, and scaling each character part image in the character string. , changing the relative position with respect to the symbol part image, reversing or rotating the character part image, adjusting the interval between the character part images, or a combination of these processes.

Further, deformation of the symbol parts image includes processing such as enlargement, reduction, inversion or rotation, movement of the arrangement position within the image area 16S, change of line type for drawing figures, or a combination of these processing. .
By the above-described data augmentation processing, a large number of transformed input images for learning are generated from the images in the image region 16S that have been arranged once. data can be generated.

Step S112: The composite image generation unit 15 displays on the display screen of the image display unit 16 a notification image prompting a determination as to whether or not to end the generation of learning data.
Then, when the operator selects to terminate the generation of learning data, the shape extension unit 14 terminates the processing.
On the other hand, when the operator selects not to finish generating the learning data, the parts image selection unit 12 advances the process to step S101.

Next, as the learning of the machine learning model in the GAN, learning of the classifier is performed using the learning data generated by the above-described processing.
FIG. 5 is a conceptual diagram illustrating initial learning of a classifier, which is a machine learning model. In FIG. 5, a learning input image 201a and a learning modified image 201b (genuine) of the learning data 201 are input to a classifier 501, and since they are genuine, learning is performed with the identification information (evaluation value) set to "1". Let it be done. Similarly, a learning input image 202a and a learning modified image 202b (genuine) of the learning data 202 are input to the classifier 501, and since they are genuine, learning is performed with the identification information (evaluation value) set to "1". Let Also, a learning input image 205a and a previously prepared learning generated image 205b (fake) are input to the classifier 501 as learning data 205, and the identification information (evaluation value) is set to "0" because it is a fake. Let them learn.
The learning process for the discriminator 501 is performed using a large amount of learning data accumulated in the learning image data storage unit 20, and initial learning is performed.

FIG. 6 is a conceptual diagram illustrating learning of a GAN generator and classifier using learning data. In FIG. 6, the generator 502 is a machine learning model similar to the discriminator 501, but it is a machine learning model that generates an image instead of performing authenticity determination with the discriminator 501. FIG. That is, the present embodiment is a machine learning model that extracts the symbol parts image from the input image and changes it to a generated image for learning, in other words, removes the general parts image and the character parts image from the input image. Here, it is assumed that the classifier 501 has already completed the initial learning.

For GAN learning, a learning input image 201 a in learning data 201 is input to the generator 502 . As a result, the generator 502 makes some changes to the learning input image 201a and outputs a learning generated image 201a' as a modified image.
Then, the classifier 501 is supplied with learning data consisting of a set of the input image for learning 201a and the changed image for learning 201b, or learning data consisting of a set of the input image for learning 201a and the generated image for learning 201a′. do.
The discriminator 501 determines that the input (supplied) learning data is “genuine” when it includes a modified learning image (that is, an image consisting only of the symbolic part images stored in the symbolic figure database 17). A value close to "1" is output as the evaluation value, and if the generated image for learning (that is, the image generated by the generator 502) is included, it is determined as "fake" and a value close to "0" is set as the evaluation value. Output. The discriminator 501 outputs the evaluation value of the determination result to the generator 502 .

Since the purpose of the learning of the generator 502 is to generate a training generated image to deceive the classifier 501, the evaluation value output from the training generated image by the classifier 501 is “1”. It is desirable to be
Therefore, in the learning of the generator 501, the parameter of the generator 502 (for example, the machine learning model function parameters) are updated.
In this learning, learning data (a set of a learning input image and a learning generated image) stored in the learning image data storage unit 20 is used as the learning data.

As a result, the generator 502 is gradually learned by the GAN learning described above so as to generate a generated learning image by extracting only the symbol parts image from the inputted learning input image.
On the other hand, the classifier 501 compares the learning input image and the learning generated image that are gradually input, and the probability that the learning generated image can be determined as a fake decreases.
Therefore, theoretically, in a trained GAN, the classifier 501 determines the authenticity of the learning input image and the learning generated image generated by the generator 502 at a rate of 50%. Each of the discriminator 501 and the generator 502 is trained until it determines that the is genuine or fake. In practice, the difference between the output value of the system and the expected value is called a loss. , the learning is repeated until both widths of decrease converge. Further, the learning of the discriminator 501 and the generator 502 may end when the generation of the generated learning image and the feedback of the evaluation value of the generated learning generated image are repeated a predetermined number of times.

As described above, according to this embodiment, the symbol parts image, the general parts image, and the character parts image are selected from the symbolic figure database 17, the general figure database 18, and the character figure database 19, respectively, and the image display unit 16 By the processing of arranging in the image area 16S on the display screen, it is possible to generate learning data in which the learning input image and the learning modified image are combined. A large amount of learning data used for model learning can be easily generated.

Further, according to the present embodiment, after each of the symbol parts image, the general parts image, and the character parts image is arranged in the image area 16S on the display screen of the image display unit 16 to create one piece of learning data, this arrangement By changing the layout of or by performing data extension processing for each of the symbol part images, general part images, and character part images, a modified version of the created learning data is generated, making it easy to create variations in the learning data. can be increased, and a large amount of training data can be obtained easily.

In this embodiment, the selection of each part image (steps S101, S104, S107) is performed by the operator, but systematization (selection of the part image for the learning data generation system 1 It may be performed mechanically by adding an application of a function performed based on preset selection rules. In this case, as the selection rule, for example, the images stored in the symbolic graphic database 17, the general graphic database 18, and the character graphic database 18 are selected in order or randomly. good.

In addition, in the present embodiment, the placement of each part image in the image area 16S (steps S102, S105, and S108) was performed by the operator, but systematization (parts The arrangement of images may be performed mechanically by adding an application having a function of arranging images based on preset arrangement rules. In this case, as the arrangement rule, for example, the image area 16S is virtually divided into a plurality of sections, and each part image is arranged in each section in order or randomly. good.

In the present embodiment, the operator determines whether to use the general parts image or the character parts image (steps S103 and S106). The determination of use may be performed mechanically by adding an application having a function of performing the determination of use based on preset usage determination rules. In this case, as the usage determination rule, for example, whether or not to use each part image may be randomly determined.

In this embodiment, the operator determines whether to use general parts images, to use character parts images, and to perform data extension processing (steps S103, S106, and S110). It may be performed mechanically by adding an application having a function of performing the above-described execution judgment based on a preset execution judgment rule to the data generation system 1 . In this case, as the implementation determination rule, for example, the use of general parts images, the use of character parts images, and the execution of data extension processing may be determined at random.

In the present embodiment, the size of the image area 16S is specified by specifying the height and width using a predetermined number of pixels. The size of 16S may be mechanically performed by adding the application of a function based on preset size specification rules that specify height and width by a predetermined number of pixels. In this case, as the size designation rule, for example, the number of pixels in each of the height and width may be set randomly.

<Second embodiment>
A learning data generation system according to a second embodiment of the present invention will be described. FIG. 7 is a diagram showing a configuration example of a learning data generation system according to the second embodiment of the present invention. In this embodiment, generation of training data for semantic segmentation will be described as an example.
Also, as one of machine learning models, there is semantic segmentation for detecting an image of a predetermined type of object from an input image. Semantic segmentation does not detect the entire input image or a part of the input image, but rather labels each pixel (or pixel) in the input image with the meaning that the pixel (or pixel) indicates. Then, parts images, which are images of each object in the input image, are detected.
For this reason, in the case of semantic segmentation, training data is prepared by labeling the part images to be detected, that is, by labeling each pixel (or pixel) in the part image to indicate the type of object. There is a need.
7, the learning data generation system 10 in this embodiment includes a control unit 111, a parts image selection unit 112, a labeling unit 113, a shape extension unit 114, a composite image generation unit 115, an image display unit 116, and an animal figure database 117. , a building graphic database 118, a road graphic database 119, and a learning image data storage unit 120, respectively.

The control unit 111 makes a control signal input from input means (not shown) (screen selection by a keyboard or mouse) correspond to the control contents indicated by the control signal, and controls the parts image selection unit 112, the labeling unit 113, the shape extension unit 113, and the It outputs to the unit 114, the composite image generation unit 115, and the image display unit 116, respectively. The control unit 11 also writes the animal part images, the building part images, and the road part images supplied from the outside to the animal graphic database 117, the building graphic database 118, and the road graphic database 119, respectively, for storage. In the present embodiment, an explanation has been given of generating learning data by combining animal part images, building part images, and road part images. It is an example of a figure, and any kind of figure may be used to combine when generating learning data.

The parts image selection unit 112 displays a parts image selection screen on the display screen of the image display unit 116 . Here, in the present embodiment, the parts images are, for example, animal parts images, building parts images, road parts images, etc., and are combined to form one composite image. image used. In the present embodiment, as in the first embodiment, the learning data is configured as a set of image data of each of an input image for learning and a modified image for learning.

FIG. 8 is a conceptual diagram illustrating the flow of learning data generation in the learning data generation system of this embodiment. 8A shows an image area 116S of learning data on the display screen of the image display unit 116. FIG. Further, in FIG. 8A, no parts image is displayed, and only the image area 116S is displayed.
FIG. 8B shows a state in which an animal part image 401 is selected by the operator and the selected animal part image 401 is placed at a predetermined position in the image area 116S on the display screen of the image display unit 116. there is Here, the animal parts image is a kind of parts image for generating a learning input image generated as a composite image, and is image data of figures of animals such as humans, dogs, and cats.

Next, in FIG. 8C, a building parts image 402 is selected by the operator, and the selected building parts image 402 is displayed at a predetermined position in the image area 116S on the display screen of the image display unit 116, and the animal parts image 401 is displayed. It shows a state where it is arranged with Here, the building parts image is a kind of parts image for generating learning input images as composite images, and is image data of figures such as houses, factories, supermarkets, buildings, etc. combined with the animal parts images described above. .

8(d), the road parts image 403 is selected by the operator, and the selected road parts image 403 is displayed at a predetermined position in the image area 116S on the display screen of the image display unit 116. It shows a state where it is arranged together with the building parts image 402 . Here, the road part image is graphic image data such as highways, farm roads, sidewalks, pedestrian crossings, general roads (with multiple types of lanes), and the like. In this embodiment, FIG. 8D is used as the learning input image. , and data obtained by adding a label to each part of the input image for learning is used as a modified image for learning.

FIG. 8(e) is an image in which labels are assigned to the pixels constituting each parts image in FIG. 8(d). In FIG. 8E, the color of each pixel is given as a label, and the pixel of the animal part image is red, the pixel of the building part image is blue, and the pixel of the road part image is yellow.
In this way, each part of the input image for learning, that is, each pixel of the animal parts image has a label indicating that it is an animal, each pixel of the building parts image has a label that indicates that it is a building, and each pixel of the building parts image has a label indicating that it is a building. Each pixel of is given a label indicating that it is a road.
In the present embodiment, FIG. 8E is a modified image for learning, which is image data in which each pixel of each part of this input image for learning is labeled.

The label assigning unit 113 assigns a color set by the worker as a label indicating the type of the part image to each pixel of the part image (animal part image, building part image, road part image) selected by the operator. do.

The composite image generator 115 synthesizes each of the animal part images, the building part images, and the road part images placed in the image area 116S on the display screen of the image display unit 116 by the operator to generate a learning input image.
In addition, the composite image generation unit 115 synthesizes each of the labeled animal part images, the building part images, and the road part images, which are placed in the image area 116S on the display screen of the image display unit 116 by the operator, to perform learning. Generate a modified image for
Then, as in the first embodiment, the composite image generation unit 115 combines the generated learning input image and learning modified image, and writes them as learning data to the learning image data storage unit 120. Memorize.

As in the first embodiment, the shape extension unit 114 performs data extension processing for each of the animal part images, building part images, and road part images arranged in the image area 116S according to the operator's input.
Here, the composite image generation unit 115 synthesizes each of the animal part images, the building part images, and the road part images placed in the image area 116S subjected to the data extension processing by the shape extension unit 114 to form learning data. Then, it is written and stored in the learning image data storage unit 120 as learning data.

The animal figure database 117 is a database in which images of figures classified into types of animal part images are accumulated.
The building figure database 118 is a database in which images of figures classified into the types of building part images are accumulated.
The road figure database 119 is a database in which images of figures classified into types of road part images are accumulated.
In the learning image data storage unit 120, learning data, which is a set of a learning input image and a learning modified image, is written and stored.

FIG. 9 is a conceptual diagram illustrating a learning example of a machine learning model that determines the types of parts images in an input image to be input and assigns a label to each pixel in each parts image. The generator 551 is a semantic segmentation machine learning model that determines the type of the parts image in the input image to be input, and assigns a label to each pixel in the determined type of the parts image (predetermined rule).
The learning data is sequentially read from the learning image data storage unit 120, the learning input image 421 is input to the generator 551, and learning is performed so that the learning modified image 422 is generated as an output.

That is, each pixel of the animal part image 401 in the learning input image 421 is output as red, each pixel of the building part image 402 is output as blue, and each pixel of the road part image 403 is output as yellow. Thus, the generator 551 is trained using the created learning data.
Thus, it is possible to train the generator 551 that determines the type of image in the input image and assigns a label corresponding to the type of the image to each pixel that constitutes the image.

As described above, according to this embodiment, an animal part image, a building part image, and a road part image are selected from each of the animal graphic database 117, the building graphic database 118, and the road graphic database 119, and the image display unit 116 By the processing of arranging in the image area 116S on the display screen, it is possible to generate learning data in which the input image for learning and the modified image for learning are combined. It is possible to easily generate a large amount of learning data used for

Further, according to this embodiment, the types of each part image (animal part image, building part image, road part image) are known, and a label such as a color is assigned to the pixels of the part image collectively. As a result, it is not necessary to perform the labor-intensive task of labeling each pixel as in the conventional art, and it is possible to easily assign a label indicating the type to each pixel of the parts image.

Further, according to the present embodiment, after each of the animal part image, the building part image, and the road part image is arranged in the image area 116S on the display screen of the image display unit 116 to create one piece of learning data, this arrangement is performed. By changing the layout of the image, or by performing data extension processing for each of the animal part images, building part images, and road part images, a modified version of the created learning data is generated, making it easy to vary the learning data. can be increased, and a large amount of training data can be obtained easily.

1 and the learning data generation system 10 shown in FIG. 7. A program for realizing a function of generating learning data composed of learning input images and learning changed images of each of the learning data generation system 1 and FIG. is recorded on a computer-readable recording medium, and the computer system loads and executes the program recorded on this recording medium to calculate the predicted value of the occurrence of defects in infrastructure equipment and to perform inspections based on each index value. A priority calculation process may be performed. It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices.

The "computer system" also includes the home page providing environment (or display environment) if the WWW system is used.
The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. Furthermore, "computer-readable recording medium" refers to a program that dynamically retains programs for a short period of time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It also includes those that hold programs for a certain period of time, such as volatile memories inside computer systems that serve as servers and clients in that case. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and designs and the like are included within the scope of the gist of the present invention.

Reference Signs List 1, 10 Learning data generation system 11, 111 Control unit 12, 112 Part image selection unit 13 Character string generation unit 14, 114 Shape extension unit 15, 115 Composite image generation unit 16, 116 Image display unit 16S, 116S... Image area 17... Symbolic figure database 18... General figure database 19... Character figure database 20, 120... Learning image data storage unit 117... Animal figure database 118... Building figure database 119... Road figure database

Claims (9)

A learning data generation system that generates learning data consisting of a training input image and a learning modified image used for learning a machine learning model that generates a modified image by modifying a predetermined input image according to a predetermined rule. and
a plurality of image databases in which part images that are components of images are stored;
a selection unit that selects the parts image from each of the image databases;
a composite image generation unit that generates a composite image obtained by combining the part images selected by the selection unit and uses it as the learning input image;
A learning data generation system, comprising: a modified image generation unit that generates the modified learning image composed of a specific parts image in the parts image selected by the selection unit. The image database is
a symbolic figure database in which symbolic part images, which are symbolic part images given when a visual impression of the input image for learning is observed, are accumulated;
2. The general graphic database in which general parts images, which are general parts images having no symbolism per se, are accumulated to supplement the symbolicity of the symbolic parts images, according to claim 1. Learning data generation system. The image database is
3. The learning data generation system according to claim 2, further comprising a character/graphic database in which character part images, which are part images representing characters, are accumulated. wherein the input image for learning is composed of the symbol parts image and either or both of the general parts image and the character parts image,
4. The learning data generation system according to claim 3, wherein only the symbol parts image in the input image for learning is arranged in the modified image for learning. 5. The learning data generation system according to any one of claims 1 to 4, further comprising a shape expansion unit that deforms the shape of each part image. The learning data generation system comprises a generator and a discriminator,
The generator comprises a machine learning model that extracts a predetermined element from the input image for learning and generates a generated image for learning,
The classifier comprises a machine learning model that evaluates learning data composed of the input image for learning and the modified image for learning, or learning data composed of the input image for learning and the generated image for learning.
The learning data generation system according to any one of claims 1 to 5, characterized by: the machine learning model is semantic segmentation;
2. The learning data according to claim 1, wherein the modified image for learning is obtained by selecting the type of each of the part images in the input image for learning and adding a label indicating the type. generation system. Learning in which a computer system generates learning data consisting of a training input image and a learning modified image used for learning a machine learning model that generates a modified image by modifying a predetermined input image according to a predetermined rule. A data generation method,
a selection process of selecting a part image from each of a plurality of image databases storing part images that are constituent elements of an image;
A composite image generation step of generating a composite image by combining the part images selected in the selection step and using it as the learning input image;
and a learning data generation method, comprising: a modified image generating step of generating the modified learning image composed of a specific part image in the part images selected in the selecting step. A learning method in which a computer system learns a machine learning model that generates a modified image by modifying a predetermined input image according to a predetermined rule,
a selection step of selecting the parts image to be used for generating the learning input image, which is the learning data for the predetermined input image, from each of a plurality of image databases in which the parts images constituting the image are stored;
A composite image generating step of generating a composite image by combining the part images selected in the selection step and using it as the learning input image;
a modified image generation process for generating a learning modified image composed of a specific part image in the part images selected in the selection process;
and a learning step of learning the machine learning model outputting the modified learning image by inputting the learning input image.

Images