WO2021106961A1 - Image generation device - Google Patents

Image generation device Download PDF

Info

Publication number
WO2021106961A1
WO2021106961A1 PCT/JP2020/043904 JP2020043904W WO2021106961A1 WO 2021106961 A1 WO2021106961 A1 WO 2021106961A1 JP 2020043904 W JP2020043904 W JP 2020043904W WO 2021106961 A1 WO2021106961 A1 WO 2021106961A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
unit
trained model
learning
image
Prior art date
Application number
PCT/JP2020/043904
Other languages
French (fr)
Japanese (ja)
Inventor
謙祐 横田
杉浦　直樹
Original Assignee
株式会社小糸製作所
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社小糸製作所 filed Critical 株式会社小糸製作所
Priority to JP2021561466A priority Critical patent/JPWO2021106961A1/ja
Publication of WO2021106961A1 publication Critical patent/WO2021106961A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Definitions

  • the present invention relates to an image generator.
  • Patent Document 1 discloses an image generator that generates intermediate image data by morphing processing.
  • the difficulty of predicting the new image data generated from the first image data and the second image data is not set in the morphing process. Therefore, there is a concern that the new image data can be easily predicted from the first image data and the second image data. Further, since the intermediate image data is an intermediate image data between the first image data and the second image data, the new image data is apparently the data recalled from the first image data or the second image data. There is a concern. Therefore, after setting the difficulty of prediction, it is required to generate image data having a new design property that is not easy to predict from the first image data and the second image data.
  • an object of the present invention is to provide an image generation device capable of generating image data having a new design property that is not easy to predict from a plurality of image data after setting the difficulty of prediction.
  • the image generator of the present invention includes a recording unit that records a plurality of first image data and a plurality of second image data, a first specific gravity of the first domain of the first image data, and the above.
  • a setting unit for setting the second specific weight of the second domain of the second image data, and learned corresponding to the first specific weight and the second specific weight from the plurality of first image data and the plurality of second image data.
  • a learning unit that generates a model for each value of the first specific gravity and the second specific gravity, a trained model storage unit that stores a plurality of the trained models, and a plurality of the trained model storage units that are stored in the trained model storage unit.
  • a trained model selection unit that selects one trained model from the trained models, a test image data input unit that inputs test image data, and the trained model selected by the trained model selection unit. It is characterized by including an image data generation unit that generates new image data from the test image data input from the test image data input unit.
  • the values of the first specific density and the second specific gravity indicating the difficulty of prediction can be set, and a trained model corresponding to each of the values of the first specific density and the second specific gravity is generated, and a plurality of trained models are generated.
  • New image data is generated from the test image data using one of the trained models of.
  • the degree of conversion of the test image data changes according to the trained model corresponding to the values of the first specific density and the second specific gravity.
  • the new image data may become image data that is difficult to predict from the first image data and the second image data, and may have a new design property. Therefore, the image generation device of the present embodiment can generate new image data having a new design property that is not easy to predict from the plurality of first and second image data after setting the difficulty of prediction.
  • the learning unit may generate a trained model according to the Cycle GAN method.
  • the learning unit may perform the calculation used in the Cycle GAN method by the number of learning times set in each of the learned models to generate each of the learned models.
  • the image generator of the present invention may further include an output unit that outputs new image data.
  • an image generation device capable of generating image data having a new design property that is not easy to predict from a plurality of image data after setting the difficulty of prediction. can do.
  • FIG. 1 is a block diagram of an image generator according to an embodiment of the present invention.
  • FIG. 2 is a diagram for explaining the third term in the formula of the loss function of the learning unit.
  • FIG. 3 is a flowchart showing the trained model generation steps.
  • FIG. 4 is a flowchart showing the learning process in the trained model generation step.
  • FIG. 5 is a flowchart showing an image generation step.
  • FIG. 1 is a block diagram of the image generation device 10 according to the present embodiment.
  • the image generation device 10 generates a trained model from a plurality of image data according to the Cycle GAN method in the hostile generation network (GAN (Generative Adversarial Network)) method, and generates new image data using the generated trained model. To do.
  • GAN Geneative Adversarial Network
  • FIG. 1 an example of generating a trained model from each domain of two image data is shown.
  • a domain indicates a feature in image data.
  • the image generation device 10 includes a recording unit 21, a first image data input unit 23, a second image data input unit 25, a specific gravity input unit 27, a learning frequency input unit 29, a learning unit 41, and an image data generation unit. It includes a control unit 40 including 47, a trained model storage unit 51, a test image data input unit 53, a trained model selection unit 55, and an image output unit 57.
  • each block of the image generation device 10 may be configured by hardware, may be configured by software, or may be configured by a combination of hardware and software.
  • the recording unit 21 records a plurality of first image data and a plurality of second image data.
  • Each first image data is data in which the appearance of each first image data is similar when each first image data is output as an image (for example, a still image), and the appearance of each first image data.
  • the first image data has been described, but the same applies to each of the second image data.
  • each first image data is image data indicating cat's eyes
  • each first image data is data classified into the same category such as cat's eyes.
  • each second image data is image data indicating a vehicle headlight
  • each second image data is data classified into the same category such as a vehicle headlight, and is the first data.
  • the data is classified into a category different from the image data.
  • the plurality of first image data has a first domain
  • the plurality of second image data has a second domain.
  • the first image data is image data indicating the eyes of a cat
  • the first domain indicates, for example, the size and shape of the eyes.
  • the second image data is image data indicating a vehicle headlight
  • the second domain indicates, for example, the size and shape of the vehicle headlight.
  • the recording unit 21 records training image data and test image data, which will be described later.
  • the recording unit 21 is, for example, a memory.
  • the first image data input unit 23 inputs to the learning unit 41 an instruction to cause the learning unit 41 of the control unit 40 to read a plurality of first image data recorded in the recording unit 21.
  • the second image data input unit 25 inputs to the learning unit 41 an instruction to cause the learning unit 41 to read a plurality of second image data recorded in the recording unit 21.
  • the specific gravity input unit 27 learns the specific density ⁇ A of the first domain of the first image data used at the time of learning of the learning unit 41 and the specific gravity ⁇ B of the second domain of the second image data used at the time of learning of the learning unit 41. It is input to the setting unit 43 described later of 41.
  • the specific densities ⁇ A and ⁇ B indicate the difficulty of prediction described later.
  • the values of the specific densities ⁇ A and ⁇ B can be appropriately set by the user.
  • the learning number input unit 29 inputs the learning number of the learning unit 41, which will be described later, to the setting unit 43.
  • the first image data input unit 23, the second image data input unit 25, the specific gravity input unit 27, and the learning frequency input unit 29 are devices for input such as a keyboard and a mouse.
  • the control unit 40 includes a CPU (Central Processing Unit) and a memory.
  • the control unit 40 comprehensively controls the operation of the image generation device 10 by reading and executing the control program recorded in the memory by the CPU.
  • the learning unit 41 has a setting unit 43.
  • the setting unit 43 sets the specific densities ⁇ A and ⁇ B input from the specific densities input unit 27, and inputs the set specific densities ⁇ A and ⁇ B to the generation unit 45 and the identification unit 46 described later of the learning unit 41. Further, the setting unit 43 sets the learning number input from the learning number input unit 29, and inputs the set learning number to the generation unit 45 and the identification unit 46.
  • the learning unit 41 has a generation unit 45 and an identification unit 46, and the generation unit 45 and the identification unit 46 form a neural network in machine learning.
  • the fake image data and the training image data used in the generation unit 45 and the identification unit 46 will be described.
  • the learning unit 41 is mainly both the generation unit 45 and the identification unit 46.
  • the fake image data is fake data obtained by converting a certain image data so as to approximate it to the training image data.
  • the training image data is real data that is a basis for improving the accuracy of the fake image data in order to approximate the fake image data to the training image data.
  • the approximation here indicates the appearance when the image data is output as an image (for example, a still image).
  • the training image data becomes the second image data
  • the fake image data approximates the first image data to the second image data which is the training image data. It becomes the converted fake second image data.
  • the training image data becomes the first image data
  • the fake image data approximates the second image data to the first image data which is the training image data. It becomes the fake first image data converted into.
  • the generation unit 45 reads the above-mentioned image data from the recording unit 21, converts the image data, and generates fake image data from the image data.
  • the fake image data is input to the identification unit 46.
  • the identification unit 46 discriminates between the fake image data input from the generation unit 45 and the training image data read from the recording unit 21.
  • the identification unit 46 calculates information regarding the deviation between the fake image data and the training image data, and outputs the information to the generation unit 45.
  • the generation unit 45 reads image data different from the image data read from the recording unit 21 from the recording unit 21, and converts the other image data read from the other image data based on the information from the identification unit 46. To generate fake image data different from the above. Another fake image data is input to the identification unit 46, and the identification unit 46 discriminates between the other fake image data and the training image data.
  • the generation unit 45 and the identification unit 46 compete with each other alternately, and as a result, the generation unit 45 and the identification unit 46 deepen the learning.
  • the generation unit 45 can generate fake image data that is close to the training image data.
  • the identification unit 46 does not output the information to the generation unit 45, and the generation unit 45 does not generate the fake image data.
  • the generation unit 45 and the identification unit 46 have image data. Both the first image data and the second image data are generated and identified, and this point will be described below.
  • the Cycle GAN method is represented by the following loss function equation (1).
  • a special loss function called "Cycle loss”, which is the third term, is added to the loss function which is the first term and the second term.
  • the first term of the equation (1) is a loss function that converts the first image data into fake image data approximated to the second image data.
  • X is shows the first image data
  • Y represents a second image data
  • G represents the generator 45 for generating a false image data from the first image data
  • D Y is the training image data
  • the identification unit 46 for distinguishing from the fake image data is shown.
  • the second term of the equation (1) is a loss function that converts the second image data into fake image data approximated to the first image data.
  • Y represents a second image data
  • X is shows the first image data
  • F is shows a generator 45 which generates false image data from the second image data
  • D X is the training image data
  • the identification unit 46 for distinguishing from the fake image data is shown.
  • the specific densities ⁇ A and ⁇ B set by the setting unit 43 are integrated as coefficients.
  • the third term of the formula (1) is represented by the following formula (2).
  • the equation (2) will be described below with reference to FIG.
  • the generation unit 45 reduces the difference between the first image data real_X and the restored first image data rec_X by the specific gravity ⁇ A, thereby reducing the difference between the first image data real_X and the fake image data fake_Y. Suppress excessive conversion to.
  • the specific density ⁇ A is integrated, and the smaller the specific density ⁇ A, the weaker the above-mentioned suppression, the greater the degree of conversion of the first image data real_X, and the more dynamically the first image data real_X becomes. Will be converted. Therefore, the fake image data fake_Y generated by the conversion is not more similar to the second image data which is the training image data, and is a new image having a new design that is difficult to predict from the first image data real_X. It becomes data.
  • the fake image data fake_Y generated by the conversion is not similar to the second image data which is the training image data, and is a new image having a new design that is not easy to predict from the first image data real_X. It becomes data.
  • the smaller the specific density ⁇ A the more difficult it is to predict, and the fake image data fake_Y becomes image data that is more difficult to predict from the first image data real_X.
  • the identification unit 46 denoted D Y in Figure 2 identifies the second image data is a training image data false image data fake_Y generated by the generator 45.
  • the generation unit 45 reduces the difference between the second image data real_Y and the restored second image data rec_Y by the specific gravity ⁇ B, so that the fake image data fake_X from the second image data real_Y Suppress excessive conversion to.
  • the specific density ⁇ B is integrated, and the smaller the specific density ⁇ B, the weaker the above-mentioned suppression, the greater the degree of conversion of the second image data real_Y, and the more dynamically the second image data real_Y. Will be converted. Therefore, the fake image data fake_X generated by the conversion is not more similar to the first image data which is the training image data, and is a new image having a new design that is difficult to predict from the second image data real_Y. It becomes data.
  • the fake image data fake_X generated by the conversion is not similar to the first image data which is the training image data, and is a new image having a new design that is not easy to predict from the second image data real_Y. It becomes data.
  • the smaller the specific density ⁇ B the more difficult it is to predict, and the fake image data fake_X becomes image data that is more difficult to predict from the second image data real_Y.
  • the identification unit 46 denoted as D x in Figure 2 identifies the first image data is a training image data false image data fake_X generated by the generator 45.
  • the learning process which is a calculation using the loss function represented by the above equation (1), is performed by the learning unit 41 for the number of learning times set by the setting unit 43, so that the specific gravity ⁇ A
  • One trained model in ⁇ B is constructed.
  • the trained model is constructed for each value of the specific densities ⁇ A and ⁇ B.
  • the first, second, and third trained models are constructed as trained models.
  • the first trained model is constructed with a specific density ⁇ A1 and a specific density ⁇ B1 smaller than the specific density ⁇ A1
  • the second trained model is constructed with a specific density ⁇ A2 and a specific density ⁇ B2 having the same specific density ⁇ A2
  • the third trained model is constructed with a specific density ⁇ A3. It is constructed with a specific density ⁇ B3 larger than the specific density ⁇ A3.
  • the trained model storage unit 51 stores each trained model constructed as described above as independent data.
  • the trained model is input to the trained model storage unit 51 each time the trained model is constructed as one model by the learning unit 41.
  • the trained model storage unit 51 is, for example, a memory.
  • the test image data input unit 53 inputs to the image data generation unit 47 an instruction to cause the image data generation unit 47 to read the test image data recorded in the recording unit 21.
  • the test image data is an image used when the image data generation unit 47 generates the image data.
  • the test image data is, for example, image data showing a cat's eyes such as the first image data, or image data showing a vehicle headlight such as the second image data.
  • the trained model selection unit 55 selects a trained model from the trained model storage unit 51, and inputs an instruction to the image data generation unit 47 to read the selected trained model into the image data generation unit 47.
  • test image data input unit 53 and the trained model selection unit 55 are devices for input such as a keyboard and a mouse.
  • the image data generation unit 47 accesses the trained model storage unit 51 according to the instruction from the trained model selection unit 55, and reads the trained model selected by the trained model selection unit 55 from the trained model storage unit 51. Next, the image data generation unit 47 generates new image data from the test image data using the read learned model. The generated new image data is input to the image output unit 57.
  • the image output unit 57 is, for example, a monitor.
  • the image output unit 57 outputs new image data generated by the image data generation unit 47 as an image.
  • the operation of the image generation device 10 includes a trained model generation step and an image generation step as main steps.
  • FIG. 3 is a flowchart showing the trained model generation steps.
  • Step S1 the first image data input unit 23 inputs an instruction to cause the learning unit 41 to read a plurality of first image data, and the learning unit 41 inputs a plurality of first image data from the recording unit 21. read out. Further, the second image data input unit 25 inputs an instruction to cause the learning unit 41 to read the second image data, and the learning unit 41 reads the second image data from the recording unit 21.
  • the process proceeds to step S2.
  • Step S2 the specific gravity input unit 27 inputs the specific densities ⁇ A1 and ⁇ B1 to the setting unit 43, and the setting unit 43 sets the specific densities ⁇ A1 and ⁇ B1 as the specific densities ⁇ A and ⁇ B.
  • the set specific densities ⁇ A1 and ⁇ B1 are input to the generation unit 45 and the identification unit 46, and the process proceeds to step S3.
  • Step S3 the learning number input unit 29 inputs the learning number of the learning unit 41 to the setting unit 43, and the setting unit 43 sets the input learning number.
  • the set number of learnings is input to the generation unit 45 and the identification unit 46, and the process proceeds to step S4.
  • the number of learnings is set to, for example, 100 times.
  • Step S4 the learning unit 41 checks the current number of learning times. If the number of learnings is less than 100, the process proceeds to step S5, and if the number of learnings is not less than 100, the process proceeds to step S7.
  • the trained model generation step is started and the process shifts to the first step S4, the number of trainings is set to 0.
  • Step S5 In this step, the learning unit 41 shifts to the learning process described later. When the learning process is completed, the process proceeds to step S6.
  • Step S6 In this step, the learning unit 41 adds one to the current number of learnings, and the process returns to step S4.
  • Step S7 the first trained model corresponding to the specific densities ⁇ A1 and ⁇ B1 set in step S2 is completed by the learning process 100 times, and the completed first trained model is stored in the trained model storage unit 51. It is stored.
  • FIG. 4 is a flowchart showing the learning process of the learning unit 41.
  • Step S11 the learning unit 41 allocates the order i to each of the first image data and each second image data read from the recording unit 21 in step S1. As described above, since the number of the first image data and the number of the second image data are 14,000 each, the order i is 1 to 14000. When the order is assigned, the process proceeds to step S12.
  • Step S12 the learning unit 41 checks the order i of the first image data and the second image data to be learned. If the order i is less than 14,000 described above, the process proceeds to step S13. If the order i is not less than 14,000, it is assumed that the learning process is performed on all the first image data and all the second image data with the specific densities ⁇ A1 and ⁇ B1 set in step S2, and the process proceeds to step S6.
  • Step S13 the learning unit 41 acquires the i-th first image data and the second image data, and the process proceeds to step S14.
  • i is set to 1.
  • Step S14 the learning unit 41 performs a calculation using the loss function represented by the equation (1) for the i-th first image data and the i-th second image data, and the process proceeds to step S15.
  • Step S15 learning is performed in the learning unit 41, and the process proceeds to step S16.
  • Step S16 the learning unit 41 adds one of the current order i, and the process returns to step S12.
  • the first image data and the second image data in the 1st to 14000th positions are expressed by the equation (1).
  • the learning process which is a calculation using the shown loss function, is performed 100 times.
  • the first trained model corresponding to the specific densities ⁇ A1 and ⁇ B1 is completed.
  • the second trained corresponding to the specific densities ⁇ A2 and ⁇ B2 is similar to the generation of the first trained model.
  • the model is generated by 100 learning processes.
  • the third trained model corresponding to the specific gravities ⁇ A3 and ⁇ B3 is generated by 100 times of learning processing in the same manner as the generation of the first trained model. ..
  • the trained model is generated for each value of the specific densities ⁇ A and ⁇ B set in step S2, and each generated trained model is stored in the trained model storage unit 51.
  • each trained model is stored in the trained model storage unit 51, the process in the trained model generation step ends.
  • FIG. 5 is a flowchart showing an image generation step.
  • the image generation step is performed after a plurality of trained models have been constructed by the trained model generation step.
  • Step S21 the test image data is input from the recording unit 21 to the image data generation unit 47 by the test image data input unit 53. Further, the trained model selected by the trained model selection unit 55 is input from the trained model storage unit 51 to the image data generation unit 47.
  • the test image data is image data indicating the cat's eyes such as the first image data, and is data classified into the same category as the first image data.
  • the training image data is the second image data.
  • Step S22 the image data generation unit 47 checks the trained model input to the image data generation unit 47. If the input trained model is the first trained model, the process proceeds to step S23. If the input trained model is the second trained model, the process proceeds to step S24. If the input trained model is the third trained model, the process proceeds to step S25.
  • Step S23 the image data generation unit 47 generates new image data from the test image data using the first trained model.
  • the specific gravity ⁇ A1 is larger than the specific density ⁇ B1. Therefore, the degree of conversion of the test image data becomes large, and the new image data is closer to the first domain than the second domain, and the image data has a new design that is not easy to predict from the test image data.
  • the new image data generated in this step is the image data indicating the vehicle headlight that most closely resembles the cat's eyes. Then, the new image data is input to the image output unit 57, and the process proceeds to step S26.
  • Step S24 the image data generation unit 47 generates new image data from the test image data using the second trained model.
  • the second trained model since the specific gravity ⁇ A2 is the same as the specific density ⁇ B2, the new image data is an intermediate image data between the cat's eyes and the vehicle headlight. Then, the new image data is input to the image output unit 57, and the process proceeds to step S26.
  • Step S25 the image data generation unit 47 generates new image data from the test image data using the third trained model.
  • the specific density ⁇ A3 is smaller than the specific density ⁇ B3. Therefore, the degree of conversion of the test image data becomes larger, and the new image data is closer to the second domain than the first domain, and the image data has a new design that is difficult to predict from the test image data.
  • the new image data generated in this step is image data showing a vehicle headlight that approximates the eyes of a cat. Then, the new image data is input to the image output unit 57, and the process proceeds to step S26.
  • Step S26 the image output unit 57 outputs new image data as an image, and the process in the image generation step ends.
  • the image generation device 10 of the present embodiment includes a recording unit 21 that records a plurality of first image data and a plurality of second image data, and a first specific gravity and a first domain of the first image data. 2
  • the setting unit 43 for setting the second specific weight of the second domain of the image data is provided.
  • the image generation device 10 generates a trained model corresponding to the first specific gravity and the second specific gravity from the plurality of first image data and the plurality of second image data for each value of the first specific gravity and the second specific gravity.
  • a trained model storage unit 51 that stores a plurality of trained models, and a trained model selection unit 55 that selects one trained model from a plurality of trained models stored in the trained model storage unit 51. Further prepare.
  • the image generation device 10 is input from the test image data input unit 53 using the test image data input unit 53 for inputting the test image data and the trained model selected by the trained model selection unit 55. It further includes an image data generation unit 47 that generates new image data from the test image data.
  • the values of the specific gravity ⁇ A and the specific gravity ⁇ B indicating the difficulty of prediction can be set, and a trained model corresponding to each value of the specific gravity ⁇ A and the specific gravity ⁇ B is generated, and a plurality of trainings are performed.
  • New image data is generated from the test image data using one trained model from the completed models.
  • the degree of conversion of the test image data changes according to the trained model corresponding to the values of the specific gravity ⁇ A and the specific gravity ⁇ B.
  • the new image data can become image data that is difficult to predict from the first image data and the second image data, which are training image data, and can be provided with new designability. Therefore, the image generation device 10 of the present embodiment can generate new image data having a new design property that is not easy to predict from the plurality of first and second image data after setting the difficulty of prediction. ..
  • the setting unit 43 sets a plurality of specific densities ⁇ A and a plurality of specific densities ⁇ B
  • the learning unit 41 sets a plurality of learned models corresponding to the plurality of specific radii ⁇ A and the plurality of specific densities ⁇ B.
  • the image data generation unit 47 generates new image data using one trained model from among the plurality of trained models. By generating a plurality of trained models, various kinds of new image data can be generated as compared with the case where only one trained model is generated.
  • the image data generation unit 47 may be the generation unit 45 learned in the learning unit 41.
  • the learning unit 41 may provide the generation unit 45 learned in the learning process to the image data generation unit 47.
  • the number of learnings set in step S3 may be set for each learned model to be constructed. Therefore, for example, the number of trainings in the construction of the first trained model may be the same as the number of trainings in the construction of other trained models, and may be more or less than the number of trainings in the construction of other trained models. Good. As the number of learnings increases, when a trained model with a large number of learnings is used, new image data having a new design that is not easy to predict can be easily generated from the plurality of first and second image data. Also, the smaller the number of trainings, the faster the trained model can be generated. Further, although the learning unit 41 generates three trained models, it is not necessary to limit the learning unit 41 to this, and at least one trained model may be generated.
  • the learning unit 41 generates a trained model according to the Cycle GAN method, but the learning unit 41 does not have to be limited to this.
  • the setting unit 43 sets the specific densities input from the specific densities input unit 27 as the specific densities ⁇ A and ⁇ B, but it is not necessary to be limited to these.
  • the setting unit 43 may set the specific densities preset in the memory of the control unit 40 as the specific densities ⁇ A and ⁇ B.
  • the setting unit 43 sets the value input from the learning number input unit 29 as the learning number, but the setting unit 43 does not have to be limited to this.
  • the setting unit 43 may set a value preset in the memory of the control unit 40 as the number of learnings.
  • Each first image data is described as image data indicating the eyes of a cat, but it is not necessary to be limited to this, and image data indicating the eyes of other animals may be used.
  • an image generation device capable of generating image data having a new design that is not easy to predict from a plurality of image data after setting the difficulty of prediction is provided, and the image generation is performed.
  • the device can be used in the field of image generation and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)

Abstract

An image generation unit (10) comprises: a storage unit (21); a setting unit (43) that sets, respectively, a first specific gravity of a first domain of first image data and a second specific gravity of a second domain of second image data; a learning unit (41) that, for each first specific gravity and second specific gravity value, generates, from a plurality of first image data pieces and a plurality of second image data pieces, a learned model corresponding to the first specific gravity and the second specific gravity; a learned model storage unit 51; and a learned model selection unit 55. In addition, the image generation device (10) comprises a test image data input unit 53, and an image data generation unit (47) that uses a selected learned model to generate new image data from test image data.

Description

画像生成装置Image generator
　本発明は、画像生成装置に関する。 The present invention relates to an image generator.
　近年、画像処理技術が進展しており、注目される画像処理技術の１つとしてモーフィング処理が挙げられる。モーフィング処理とは、原画像となる第１画像データと目的画像となる第２画像データとの間に、第１画像データと第２画像データとをつなぐ中間画像データを生成し、第１画像データを第２画像データに滑らかに変化させる処理である。特許文献１には、モーフィング処理によって中間画像データを生成する画像生成装置が開示されている。 In recent years, image processing technology has been advanced, and morphing processing can be mentioned as one of the image processing technologies that attracts attention. The morphing process generates intermediate image data that connects the first image data and the second image data between the first image data that is the original image and the second image data that is the target image, and the first image data. Is a process of smoothly changing to the second image data. Patent Document 1 discloses an image generator that generates intermediate image data by morphing processing.
特開２００１－０７６１７７号公報Japanese Unexamined Patent Publication No. 2001-0767177
　モーフィング処理において生成される中間画像データを新たな画像データとして利用する場合、モーフィング処理では、第１画像データ及び第２画像データから生成される新たな画像データの予想の困難性は設定されない。従って、新たな画像データは、第１画像データ及び第２画像データから予想が容易となる懸念がある。また、中間画像データは、第１画像データ及び第２画像データの中間的な画像データであるため、新たな画像データは、見た目上、第１画像データまたは第２画像データから想起されるデータとなる懸念がある。従って、予想の困難性を設定したうえで、第１画像データ及び第２画像データから予想が容易ではない新たなデザイン性を備える画像データの生成が求められる。 When the intermediate image data generated in the morphing process is used as new image data, the difficulty of predicting the new image data generated from the first image data and the second image data is not set in the morphing process. Therefore, there is a concern that the new image data can be easily predicted from the first image data and the second image data. Further, since the intermediate image data is an intermediate image data between the first image data and the second image data, the new image data is apparently the data recalled from the first image data or the second image data. There is a concern. Therefore, after setting the difficulty of prediction, it is required to generate image data having a new design property that is not easy to predict from the first image data and the second image data.
　そこで、本発明は、予想の困難性を設定したうえで、複数の画像データから予想が容易ではない新たなデザイン性を備える画像データを生成し得る画像生成装置を提供することを目的とする。 Therefore, an object of the present invention is to provide an image generation device capable of generating image data having a new design property that is not easy to predict from a plurality of image data after setting the difficulty of prediction.
　上記課題を解決するため、本発明の画像生成装置は、複数の第１画像データ及び複数の第２画像データを記録する記録部と、前記第１画像データの第１ドメインの第１比重及び前記第２画像データの第２ドメインの第２比重をそれぞれ設定する設定部と、前記複数の第１画像データ及び前記複数の第２画像データから前記第１比重及び前記第２比重に対応する学習済モデルを前記第１比重及び前記第２比重の値毎に生成する学習部と、複数の前記学習済モデルを格納する学習済モデル格納部と、前記学習済モデル格納部に格納される前記複数の学習済モデルから１つの学習済モデルを選択する学習済モデル選択部と、テスト用画像データを入力するテスト用画像データ入力部と、前記学習済モデル選択部によって選択される前記学習済モデルを用いて前記テスト用画像データ入力部から入力される前記テスト用画像データから新たな画像データを生成する画像データ生成部と、備えることを特徴とする。 In order to solve the above problems, the image generator of the present invention includes a recording unit that records a plurality of first image data and a plurality of second image data, a first specific gravity of the first domain of the first image data, and the above. A setting unit for setting the second specific weight of the second domain of the second image data, and learned corresponding to the first specific weight and the second specific weight from the plurality of first image data and the plurality of second image data. A learning unit that generates a model for each value of the first specific gravity and the second specific gravity, a trained model storage unit that stores a plurality of the trained models, and a plurality of the trained model storage units that are stored in the trained model storage unit. Using a trained model selection unit that selects one trained model from the trained models, a test image data input unit that inputs test image data, and the trained model selected by the trained model selection unit. It is characterized by including an image data generation unit that generates new image data from the test image data input from the test image data input unit.
　本発明の画像生成装置では、予想の困難性を示す第１比重及び第２比重の値は設定可能であり、第１比重及び第２比重の値毎に対応する学習済モデルが生成され、複数の学習済モデルのなかから１つの学習済モデルを用いてテスト用画像データから新たな画像データを生成する。新たな画像データの生成時において、第１比重及び第２比重の値に対応する学習済モデルに応じてテスト用画像データの変換度合いが変化する。変換度合いが変化すると、新たな画像データは、第１画像データ及び第２画像データから予想が困難な画像データとなり得、新たなデザイン性を備えることになり得る。従って、本実施形態の画像生成装置は、予想の困難性を設定したうえで、複数の第１，２画像データから予想が容易ではない新たなデザイン性を備える新たな画像データを生成し得る。 In the image generator of the present invention, the values of the first specific density and the second specific gravity indicating the difficulty of prediction can be set, and a trained model corresponding to each of the values of the first specific density and the second specific gravity is generated, and a plurality of trained models are generated. New image data is generated from the test image data using one of the trained models of. When new image data is generated, the degree of conversion of the test image data changes according to the trained model corresponding to the values of the first specific density and the second specific gravity. When the degree of conversion changes, the new image data may become image data that is difficult to predict from the first image data and the second image data, and may have a new design property. Therefore, the image generation device of the present embodiment can generate new image data having a new design property that is not easy to predict from the plurality of first and second image data after setting the difficulty of prediction.
　また、本発明の画像生成装置では、前記学習部は、Ｃｙｃｌｅ　ＧＡＮ方式に従って学習済モデルを生成してもよい。 Further, in the image generation device of the present invention, the learning unit may generate a trained model according to the Cycle GAN method.
　また、本発明の画像生成装置では、前記学習部は、前記Ｃｙｃｌｅ　ＧＡＮ方式において用いられる計算をそれぞれの前記学習済モデルに設定される学習回数行いそれぞれの前記学習済モデルを生成してもよい。この構成によって、学習回数が多いほど、学習回数が多い学習済モデルが用いられる場合に、複数の第１，２画像データから予想が容易ではない新たなデザイン性を備える新たな画像データが生成され易くなり得る。また、学習回数が少ないほど、学習済モデルは早く生成され得る。 Further, in the image generation device of the present invention, the learning unit may perform the calculation used in the Cycle GAN method by the number of learning times set in each of the learned models to generate each of the learned models. With this configuration, when a trained model with a large number of learnings is used as the number of learnings increases, new image data having a new design that is not easy to predict is generated from the plurality of first and second image data. It can be easier. Also, the smaller the number of trainings, the faster the trained model can be generated.
　また、本発明の画像生成装置は、新たな画像データを出力する出力部をさらに備えてもよい。 Further, the image generator of the present invention may further include an output unit that outputs new image data.
　以上のように、本発明によれば、予想の困難性を設定したうえで、複数の画像データから予想が容易ではない新たなデザイン性を備える画像データを生成することができる画像生成装置を提供することができる。 As described above, according to the present invention, there is provided an image generation device capable of generating image data having a new design property that is not easy to predict from a plurality of image data after setting the difficulty of prediction. can do.
図１は、本発明の実施形態における画像生成装置のブロック図である。FIG. 1 is a block diagram of an image generator according to an embodiment of the present invention. 図２は、学習部の損失関数の式における第３項を説明する図である。FIG. 2 is a diagram for explaining the third term in the formula of the loss function of the learning unit. 図３は、学習済モデル生成ステップを示すフローチャートである。FIG. 3 is a flowchart showing the trained model generation steps. 図４は、学習済モデル生成ステップにおける学習処理を示すフローチャートである。FIG. 4 is a flowchart showing the learning process in the trained model generation step. 図５は、画像生成ステップを示すフローチャートである。FIG. 5 is a flowchart showing an image generation step.
　以下、本発明に係る画像生成装置の好適な実施形態について図面を参照しながら詳細に説明する。以下に例示する実施形態は、本発明の理解を容易にするためのものであり、本発明を限定して解釈するためのものではない。本発明は、その趣旨を逸脱することなく、変更、改良することができる。また、本発明は、以下に例示する各実施形態における構成要素を適宜組み合わせてもよい。なお、理解の容易のため、それぞれの図において一部が誇張して記載される場合等がある。 Hereinafter, a preferred embodiment of the image generator according to the present invention will be described in detail with reference to the drawings. The embodiments illustrated below are for facilitating the understanding of the present invention, and are not for limiting the interpretation of the present invention. The present invention can be modified and improved without departing from the spirit of the present invention. In addition, the present invention may appropriately combine the components in each of the embodiments exemplified below. For ease of understanding, some parts may be exaggerated in each figure.
　図１は、本実施形態における画像生成装置１０のブロック図である。画像生成装置１０は、敵対的生成ネットワーク（ＧＡＮ（Generative Adversarial Network））方式におけるＣｙｃｌｅ　ＧＡＮ方式に従って複数の画像データから学習済モデルを生成し、生成した学習済モデルを用いて新たな画像データを生成する。本実施形態の画像生成装置１０では、２つの画像データのそれぞれのドメインから学習済モデルを生成する例が示されている。ドメインとは、画像データにおける特徴を示す。 FIG. 1 is a block diagram of the image generation device 10 according to the present embodiment. The image generation device 10 generates a trained model from a plurality of image data according to the Cycle GAN method in the hostile generation network (GAN (Generative Adversarial Network)) method, and generates new image data using the generated trained model. To do. In the image generation device 10 of the present embodiment, an example of generating a trained model from each domain of two image data is shown. A domain indicates a feature in image data.
　画像生成装置１０は、記録部２１と、第１画像データ入力部２３と、第２画像データ入力部２５と、比重入力部２７と、学習回数入力部２９と、学習部４１及び画像データ生成部４７を含む制御部４０と、学習済モデル格納部５１と、テスト用画像データ入力部５３と、学習済モデル選択部５５と、画像出力部５７とを備える。ここで、画像生成装置１０の各ブロックは、ハードウェアによって構成されていてもよいし、ソフトウェアによって構成されていてもよいし、ハードウェアとソフトウェアの組み合わせによって構成されていてもよい。 The image generation device 10 includes a recording unit 21, a first image data input unit 23, a second image data input unit 25, a specific gravity input unit 27, a learning frequency input unit 29, a learning unit 41, and an image data generation unit. It includes a control unit 40 including 47, a trained model storage unit 51, a test image data input unit 53, a trained model selection unit 55, and an image output unit 57. Here, each block of the image generation device 10 may be configured by hardware, may be configured by software, or may be configured by a combination of hardware and software.
　記録部２１は、複数の第１画像データ及び複数の第２画像データを記録する。それぞれの第１画像データは、それぞれの第１画像データを画像（例えば、静止画）として出力した場合においてそれぞれの第１画像データの見た目が近似したデータであり、それぞれの第１画像データの見た目が同じとは限らない。ここでは、第１画像データについて説明したが、それぞれの第２画像データについても同様である。例えば、それぞれの第１画像データが猫の目を示す画像データである場合、それぞれの第１画像データは、猫の目といった同じカテゴリーに区分されるデータである。また、例えば、それぞれの第２画像データが車両用前照灯を示す画像データである場合、それぞれの第２画像データは、車両用前照灯といった同じカテゴリーに区分されるデータであり、第１画像データとは異なるカテゴリーに区分されるデータである。複数の第１画像データは第１ドメインを有し、複数の第２画像データは第２ドメインを有する。第１画像データが猫の目を示す画像データである場合、第１ドメインは、例えば目の大きさや形状等を示す。第２画像データが車両用前照灯を示す画像データである場合、第２ドメインは、例えば車両用前照灯の大きさや形状等を示す。本実施形態の画像生成装置１０では、例えば、第１画像データの数及び第２画像データの数は、それぞれ１４０００個とされる。また、記録部２１は、後述する訓練用画像データ及びテスト用画像データを記録する。記録部２１は、例えば、メモリである。 The recording unit 21 records a plurality of first image data and a plurality of second image data. Each first image data is data in which the appearance of each first image data is similar when each first image data is output as an image (for example, a still image), and the appearance of each first image data. Are not always the same. Here, the first image data has been described, but the same applies to each of the second image data. For example, when each first image data is image data indicating cat's eyes, each first image data is data classified into the same category such as cat's eyes. Further, for example, when each second image data is image data indicating a vehicle headlight, each second image data is data classified into the same category such as a vehicle headlight, and is the first data. The data is classified into a category different from the image data. The plurality of first image data has a first domain, and the plurality of second image data has a second domain. When the first image data is image data indicating the eyes of a cat, the first domain indicates, for example, the size and shape of the eyes. When the second image data is image data indicating a vehicle headlight, the second domain indicates, for example, the size and shape of the vehicle headlight. In the image generation device 10 of the present embodiment, for example, the number of the first image data and the number of the second image data are 14,000, respectively. In addition, the recording unit 21 records training image data and test image data, which will be described later. The recording unit 21 is, for example, a memory.
　第１画像データ入力部２３は、記録部２１に記録される複数の第１画像データを制御部４０の学習部４１に読み出させる指示を学習部４１に入力する。 The first image data input unit 23 inputs to the learning unit 41 an instruction to cause the learning unit 41 of the control unit 40 to read a plurality of first image data recorded in the recording unit 21.
　第２画像データ入力部２５は、記録部２１に記録される複数の第２画像データを学習部４１に読み出させる指示を学習部４１に入力する。 The second image data input unit 25 inputs to the learning unit 41 an instruction to cause the learning unit 41 to read a plurality of second image data recorded in the recording unit 21.
　比重入力部２７は、学習部４１の学習時に用いられる第１画像データの第１ドメインの比重λＡと、学習部４１の学習時に用いられる第２画像データの第２ドメインの比重λＢとを学習部４１の後述する設定部４３に入力する。比重λＡと比重λＢとは、後述する予想の困難性を示す。比重λＡ，λＢの値は、使用者によって適宜設定可能となっている。 The specific gravity input unit 27 learns the specific density λA of the first domain of the first image data used at the time of learning of the learning unit 41 and the specific gravity λB of the second domain of the second image data used at the time of learning of the learning unit 41. It is input to the setting unit 43 described later of 41. The specific densities λA and λB indicate the difficulty of prediction described later. The values of the specific densities λA and λB can be appropriately set by the user.
　学習回数入力部２９は、学習部４１の後述する学習回数を設定部４３に入力する。 The learning number input unit 29 inputs the learning number of the learning unit 41, which will be described later, to the setting unit 43.
　第１画像データ入力部２３、第２画像データ入力部２５、比重入力部２７、及び学習回数入力部２９は、キーボード、マウスといった入力するためのデバイスである。 The first image data input unit 23, the second image data input unit 25, the specific gravity input unit 27, and the learning frequency input unit 29 are devices for input such as a keyboard and a mouse.
　制御部４０は、ＣＰＵ（Central Processing Unit）とメモリとを備える。制御部４０は、ＣＰＵがメモリに記録されている制御プログラムを読み出して実行することによって、画像生成装置１０の動作を統括的に制御する。 The control unit 40 includes a CPU (Central Processing Unit) and a memory. The control unit 40 comprehensively controls the operation of the image generation device 10 by reading and executing the control program recorded in the memory by the CPU.
　次に、制御部４０における学習部４１について説明する。 Next, the learning unit 41 in the control unit 40 will be described.
　学習部４１は、設定部４３を有する。 The learning unit 41 has a setting unit 43.
　設定部４３は、比重入力部２７から入力された比重λＡ，λＢを設定し、設定した比重λＡ，λＢを学習部４１の後述する生成部４５及び識別部４６に入力する。また、設定部４３は、学習回数入力部２９から入力された学習回数を設定し、設定した学習回数を生成部４５及び識別部４６に入力する。 The setting unit 43 sets the specific densities λA and λB input from the specific densities input unit 27, and inputs the set specific densities λA and λB to the generation unit 45 and the identification unit 46 described later of the learning unit 41. Further, the setting unit 43 sets the learning number input from the learning number input unit 29, and inputs the set learning number to the generation unit 45 and the identification unit 46.
　また、学習部４１は生成部４５及び識別部４６を有し、生成部４５及び識別部４６は機械学習におけるニューラルネットワークを構成する。 Further, the learning unit 41 has a generation unit 45 and an identification unit 46, and the generation unit 45 and the identification unit 46 form a neural network in machine learning.
　次に、生成部４５及び識別部４６において用いられる偽画像データ及び訓練用画像データについて説明する。なお、以下において、学習部４１とは、主に生成部４５及び識別部４６の両方であることを示す。 Next, the fake image data and the training image data used in the generation unit 45 and the identification unit 46 will be described. In the following, it is shown that the learning unit 41 is mainly both the generation unit 45 and the identification unit 46.
　偽画像データとは、ある画像データを訓練用画像データに近似させるように変換したｆａｋｅデータである。訓練用画像データとは、偽画像データを訓練用画像データに近似させるために偽画像データの精度を向上させるための基となるｒｅａｌデータである。ここでいう近似とは、画像データを画像（例えば、静止画）として出力した場合の見た目を示す。例えば、上記した画像データが第１画像データである場合、訓練用画像データは第２画像データとなり、偽画像データは第１画像データを訓練用画像データである第２画像データに近似させるように変換した偽の第２画像データとなる。逆に、上記した画像データが第２画像データである場合、訓練用画像データは第１画像データとなり、偽画像データは第２画像データを訓練用画像データである第１画像データに近似させるように変換した偽の第１画像データとなる。 The fake image data is fake data obtained by converting a certain image data so as to approximate it to the training image data. The training image data is real data that is a basis for improving the accuracy of the fake image data in order to approximate the fake image data to the training image data. The approximation here indicates the appearance when the image data is output as an image (for example, a still image). For example, when the above image data is the first image data, the training image data becomes the second image data, and the fake image data approximates the first image data to the second image data which is the training image data. It becomes the converted fake second image data. On the contrary, when the above-mentioned image data is the second image data, the training image data becomes the first image data, and the fake image data approximates the second image data to the first image data which is the training image data. It becomes the fake first image data converted into.
　生成部４５は、上記のある画像データを記録部２１から読み出し、当該画像データを変換して当該画像データから偽画像データを生成する。偽画像データは、識別部４６に入力される。 The generation unit 45 reads the above-mentioned image data from the recording unit 21, converts the image data, and generates fake image data from the image data. The fake image data is input to the identification unit 46.
　識別部４６は、生成部４５から入力された偽画像データと記録部２１から読み出した訓練用画像データとを識別する。 The identification unit 46 discriminates between the fake image data input from the generation unit 45 and the training image data read from the recording unit 21.
　また、識別部４６は、偽画像データが訓練用画像データに近似していない場合、偽画像データ及び訓練用画像データのずれに関する情報を算出して、当該情報を生成部４５に出力する。生成部４５は、記録部２１から読み出した上記画像データとは別の画像データを記録部２１から読み出すと共に、読み出した別の画像データから識別部４６からの情報を基に上記とは変換の程度を変えて上記とは別の偽画像データを生成する。別の偽画像データは識別部４６に入力され、識別部４６は別の偽画像データと訓練用画像データとを識別する。生成部４５の生成と識別部４６の識別とが交互に繰り返されることで、生成部４５及び識別部４６は交互に競合し、結果として、生成部４５及び識別部４６は学習を深める。学習が深まることで、生成部４５は、訓練用画像データに近似した偽画像データを生成可能となる。偽画像データが訓練用画像データに近似している場合、識別部４６は情報を生成部４５に出力せず、生成部４５は偽画像データを生成しない。本実施形態の学習部４１では、上記のように生成部４５及び識別部４６が交互に競合する過程において、Ｃｙｃｌｅ　ＧＡＮ方式が用いられているため、生成部４５及び識別部４６は、画像データが第１画像データ及び第２画像データである場合の両方において、生成及び識別を行っており、この点について以下に説明する。 Further, when the fake image data does not match the training image data, the identification unit 46 calculates information regarding the deviation between the fake image data and the training image data, and outputs the information to the generation unit 45. The generation unit 45 reads image data different from the image data read from the recording unit 21 from the recording unit 21, and converts the other image data read from the other image data based on the information from the identification unit 46. To generate fake image data different from the above. Another fake image data is input to the identification unit 46, and the identification unit 46 discriminates between the other fake image data and the training image data. By alternately repeating the generation of the generation unit 45 and the identification of the identification unit 46, the generation unit 45 and the identification unit 46 compete with each other alternately, and as a result, the generation unit 45 and the identification unit 46 deepen the learning. By deepening the learning, the generation unit 45 can generate fake image data that is close to the training image data. When the fake image data is close to the training image data, the identification unit 46 does not output the information to the generation unit 45, and the generation unit 45 does not generate the fake image data. In the learning unit 41 of the present embodiment, since the Cycle GAN method is used in the process in which the generation unit 45 and the identification unit 46 alternately compete with each other as described above, the generation unit 45 and the identification unit 46 have image data. Both the first image data and the second image data are generated and identified, and this point will be described below.
　Ｃｙｃｌｅ　ＧＡＮ方式は、下記損失関数の式（１）によって表される。この損失関数の式（１）では、第１項及び第２項である損失関数に、第３項である「Ｃｙｃｌｅ　ｌｏｓｓ」と呼ばれる特別な損失関数が付け加えられている。
Figure JPOXMLDOC01-appb-I000001
The Cycle GAN method is represented by the following loss function equation (1). In the formula (1) of this loss function, a special loss function called "Cycle loss", which is the third term, is added to the loss function which is the first term and the second term.
Figure JPOXMLDOC01-appb-I000001
　式（１）の第１項は、第１画像データを第２画像データに近似させた偽画像データへ変換する損失関数である。第１項において、Ｘは第１画像データを示し、Ｙは第２画像データを示し、Ｇは第１画像データから偽画像データを生成する生成部４５を示し、ＤＹは訓練用画像データと偽画像データとを識別する識別部４６を示す。 The first term of the equation (1) is a loss function that converts the first image data into fake image data approximated to the second image data. In the first paragraph, X is shows the first image data, Y represents a second image data, G represents the generator 45 for generating a false image data from the first image data, D Y is the training image data The identification unit 46 for distinguishing from the fake image data is shown.
　式（１）の第２項は、第２画像データを第１画像データに近似させた偽画像データへ変換する損失関数である。第２項において、Ｙは第２画像データを示し、Ｘは第１画像データを示し、Ｆは第２画像データから偽画像データを生成する生成部４５を示し、ＤＸは訓練用画像データと偽画像データとを識別する識別部４６を示す。 The second term of the equation (1) is a loss function that converts the second image data into fake image data approximated to the first image data. In the second term, Y represents a second image data, X is shows the first image data, F is shows a generator 45 which generates false image data from the second image data, D X is the training image data The identification unit 46 for distinguishing from the fake image data is shown.
　式（１）の第３項には、設定部４３によって設定される比重λＡ，λＢが係数として積算される。式（１）の第３項は、以下の式（２）によって表される。
Figure JPOXMLDOC01-appb-I000002
In the third term of the equation (1), the specific densities λA and λB set by the setting unit 43 are integrated as coefficients. The third term of the formula (1) is represented by the following formula (2).
Figure JPOXMLDOC01-appb-I000002
　ここで、式（２）について、図２を用いて以下に説明する。
　式（２）の第１項では、図２においてＧと示される生成部４５が第１本体部として第１画像データｒｅａｌ＿Ｘから訓練用画像データである第２画像データに近似させた偽画像データｆａｋｅ＿Ｙを生成し、図２においてＦと示される生成部４５が第１復元部として当該偽画像データｆａｋｅ＿Ｙを第１画像データｒｅｃ＿Ｘに復元する処理を行っている。式（２）の第１項では、生成部４５は、比重λＡによって第１画像データｒｅａｌ＿Ｘと復元した第１画像データｒｅｃ＿Ｘとの差を少なくすることで、第１画像データｒｅａｌ＿Ｘから偽画像データｆａｋｅ＿Ｙへの過度な変換を抑制する。式（２）の第１項では比重λＡが積算されており、比重λＡが小さいほど、上記した抑制が弱まり、第１画像データｒｅａｌ＿Ｘの変換度合いがより大きくなり、第１画像データｒｅａｌ＿Ｘはダイナミックに変換される。従って、変換によって生成される偽画像データｆａｋｅ＿Ｙは、訓練用画像データである第２画像データとはより似ておらず、第１画像データｒｅａｌ＿Ｘから予想が困難な新たなデザイン性を備える新たな画像データとなる。逆に、比重λＡが大きいほど、上記した抑制が強まり、第１画像データｒｅａｌ＿Ｘの変換度合いが大きくなり、第１画像データｒｅａｌ＿Ｘは比重λＡが小さい場合に比べてダイナミックに変換されない。従って、変換によって生成される偽画像データｆａｋｅ＿Ｙは、訓練用画像データである第２画像データとは似ておらず、第１画像データｒｅａｌ＿Ｘから予想が容易ではない新たなデザイン性を備える新たな画像データとなる。上記したように、比重λＡが小さいほど、予想の困難性がより高まり、偽画像データｆａｋｅ＿Ｙは第１画像データｒｅａｌ＿Ｘから予想がより困難な画像データとなる。また、比重λＡが大きいほど、予想の困難性が高まり、偽画像データｆａｋｅ＿Ｙは第１画像データｒｅａｌ＿Ｘから予想が困難な画像データとなる。第１画像データｒｅａｌ＿Ｘと第１画像データｒｅｃ＿Ｘとの差は、ｃｙｃｌｅ－ｃｏｎｓｉｓｔｅｎｃｙ　ｌｏｓｓと呼ばれる再構築誤差である。なお、図２においてＤＹと示される識別部４６は、生成部４５によって生成される偽画像データｆａｋｅ＿Ｙと訓練用画像データである第２画像データとを識別している。 Here, the equation (2) will be described below with reference to FIG.
In the first term of the formula (2), the fake image data fake_Y in which the generation unit 45 represented by G in FIG. 2 is approximated from the first image data real_X to the second image data which is the training image data as the first main body unit. Is generated, and the generation unit 45 shown as F in FIG. 2 performs a process of restoring the fake image data fake_Y to the first image data rec_X as the first restoration unit. In the first term of the equation (2), the generation unit 45 reduces the difference between the first image data real_X and the restored first image data rec_X by the specific gravity λA, thereby reducing the difference between the first image data real_X and the fake image data fake_Y. Suppress excessive conversion to. In the first term of the equation (2), the specific density λA is integrated, and the smaller the specific density λA, the weaker the above-mentioned suppression, the greater the degree of conversion of the first image data real_X, and the more dynamically the first image data real_X becomes. Will be converted. Therefore, the fake image data fake_Y generated by the conversion is not more similar to the second image data which is the training image data, and is a new image having a new design that is difficult to predict from the first image data real_X. It becomes data. On the contrary, as the specific gravity λA is larger, the above-mentioned suppression is strengthened, the degree of conversion of the first image data real_X is larger, and the first image data real_X is not dynamically converted as compared with the case where the specific gravity λA is small. Therefore, the fake image data fake_Y generated by the conversion is not similar to the second image data which is the training image data, and is a new image having a new design that is not easy to predict from the first image data real_X. It becomes data. As described above, the smaller the specific density λA, the more difficult it is to predict, and the fake image data fake_Y becomes image data that is more difficult to predict from the first image data real_X. Further, the larger the specific density λA, the more difficult it is to predict, and the false image data fake_Y becomes image data that is difficult to predict from the first image data real_X. The difference between the first image data real_X and the first image data rec_X is a reconstruction error called cycle-consistency loss. The identification unit 46, denoted D Y in Figure 2 identifies the second image data is a training image data false image data fake_Y generated by the generator 45.
　式（２）の第２項では、図２においてＦと示される生成部４５が第２本体部として第２画像データｒｅａｌ＿Ｙから訓練用画像データである第１画像データに近似させた偽画像データｆａｋｅ＿Ｘを生成し、図２においてＧと示される生成部４５が第２復元部として当該偽画像データｆａｋｅ＿Ｘを第２画像データｒｅｃ＿Ｙに復元する処理を行っている。式（２）の第２項では、生成部４５は、比重λＢによって第２画像データｒｅａｌ＿Ｙと復元した第２画像データｒｅｃ＿Ｙとの差を少なくすることで、第２画像データｒｅａｌ＿Ｙから偽画像データｆａｋｅ＿Ｘへの過度な変換を抑制する。式（２）の第２項では比重λＢが積算されており、比重λＢが小さいほど、上記した抑制が弱まり、第２画像データｒｅａｌ＿Ｙの変換度合いがより大きくなり、第２画像データｒｅａｌ＿Ｙはダイナミックに変換される。従って、変換によって生成される偽画像データｆａｋｅ＿Ｘは、訓練用画像データである第１画像データとはより似ておらず、第２画像データｒｅａｌ＿Ｙから予想が困難な新たなデザイン性を備える新たな画像データとなる。逆に、比重λＢが大きいほど、上記した抑制が強まり、第２画像データｒｅａｌ＿Ｙの変換度合いが大きくなり、第２画像データｒｅａｌ＿Ｙは比重λＢが小さい場合に比べてダイナミックに変換されない。従って、変換によって生成される偽画像データｆａｋｅ＿Ｘは、訓練用画像データである第１画像データとは似ておらず、第２画像データｒｅａｌ＿Ｙから予想が容易ではない新たなデザイン性を備える新たな画像データとなる。上記したように、比重λＢが小さいほど、予想の困難性がより高まり、偽画像データｆａｋｅ＿Ｘは第２画像データｒｅａｌ＿Ｙから予想がより困難な画像データとなる。また、比重λＡが大きいほど、予想の困難性が高まり、偽画像データｆａｋｅ＿Ｘは第２画像データｒｅａｌ＿Ｙから予想が困難な画像データとなる。第２画像データｒｅａｌ＿Ｙと第２画像データｒｅｃ＿Ｙとの差は、ｃｙｃｌｅ－ｃｏｎｓｉｓｔｅｎｃｙ　ｌｏｓｓと呼ばれる再構築誤差である。なお、図２においてＤｘと示される識別部４６は、生成部４５によって生成される偽画像データｆａｋｅ＿Ｘと訓練用画像データである第１画像データとを識別している。 In the second term of the equation (2), the fake image data fake_X in which the generation unit 45 shown as F in FIG. 2 is approximated from the second image data real_Y to the first image data which is the training image data as the second main body unit. Is generated, and the generation unit 45 shown as G in FIG. 2 performs a process of restoring the fake image data fake_X to the second image data rec_Y as the second restoration unit. In the second term of the equation (2), the generation unit 45 reduces the difference between the second image data real_Y and the restored second image data rec_Y by the specific gravity λB, so that the fake image data fake_X from the second image data real_Y Suppress excessive conversion to. In the second term of the equation (2), the specific density λB is integrated, and the smaller the specific density λB, the weaker the above-mentioned suppression, the greater the degree of conversion of the second image data real_Y, and the more dynamically the second image data real_Y. Will be converted. Therefore, the fake image data fake_X generated by the conversion is not more similar to the first image data which is the training image data, and is a new image having a new design that is difficult to predict from the second image data real_Y. It becomes data. On the contrary, as the specific gravity λB is larger, the above-mentioned suppression is strengthened, the degree of conversion of the second image data real_Y is larger, and the second image data real_Y is not dynamically converted as compared with the case where the specific gravity λB is small. Therefore, the fake image data fake_X generated by the conversion is not similar to the first image data which is the training image data, and is a new image having a new design that is not easy to predict from the second image data real_Y. It becomes data. As described above, the smaller the specific density λB, the more difficult it is to predict, and the fake image data fake_X becomes image data that is more difficult to predict from the second image data real_Y. Further, the larger the specific density λA, the more difficult it is to predict, and the false image data fake_X becomes image data that is difficult to predict from the second image data real_Y. The difference between the second image data real_Y and the second image data rec_Y is a reconstruction error called cycle-consistency loss. The identification unit 46, denoted as D x in Figure 2 identifies the first image data is a training image data false image data fake_X generated by the generator 45.
　本実施形態の画像生成装置１０では、上記式（１）で示される損失関数を用いる計算である学習処理を、学習部４１が設定部４３によって設定される学習回数を行うことで、比重λＡ，λＢにおける１つの学習済モデルが構築される。当該学習済モデルは、比重λＡ，λＢの値毎に構築される。 In the image generation device 10 of the present embodiment, the learning process, which is a calculation using the loss function represented by the above equation (1), is performed by the learning unit 41 for the number of learning times set by the setting unit 43, so that the specific gravity λA One trained model in λB is constructed. The trained model is constructed for each value of the specific densities λA and λB.
　本実施形態の画像生成装置１０では、学習済モデルとして、第１，２，３学習済モデルが構築される例が示されている。第１学習済モデルは比重λＡ１と比重λＡ１よりも小さい比重λＢ１とによって構築され、第２学習済モデルは比重λＡ２と比重λＡ２と同じ比重λＢ２とによって構築され、第３学習済モデルは比重λＡ３と比重λＡ３よりも大きい比重λＢ３とによって構築される。 In the image generation device 10 of the present embodiment, an example in which the first, second, and third trained models are constructed as trained models is shown. The first trained model is constructed with a specific density λA1 and a specific density λB1 smaller than the specific density λA1, the second trained model is constructed with a specific density λA2 and a specific density λB2 having the same specific density λA2, and the third trained model is constructed with a specific density λA3. It is constructed with a specific density λB3 larger than the specific density λA3.
　ここで、図１に戻り、画像生成装置１０の各ブロックの説明を続ける。 Here, returning to FIG. 1, the description of each block of the image generator 10 is continued.
　学習済モデル格納部５１は、上記のように構築されたそれぞれの学習済モデルを独立したデータとして格納する。学習済モデルは、学習部４１によって１つのモデルとして構築される度に学習済モデル格納部５１に入力される。学習済モデル格納部５１は、例えば、メモリである。 The trained model storage unit 51 stores each trained model constructed as described above as independent data. The trained model is input to the trained model storage unit 51 each time the trained model is constructed as one model by the learning unit 41. The trained model storage unit 51 is, for example, a memory.
　テスト用画像データ入力部５３は、記録部２１に記録されるテスト用画像データを画像データ生成部４７に読み出させる指示を画像データ生成部４７に入力する。テスト用画像データは、画像データ生成部４７の画像データ生成時に用いられる画像である。テスト用画像データは、例えば、第１画像データのような猫の目を示す画像データや、第２画像データのような車両用前照灯を示す画像データである。 The test image data input unit 53 inputs to the image data generation unit 47 an instruction to cause the image data generation unit 47 to read the test image data recorded in the recording unit 21. The test image data is an image used when the image data generation unit 47 generates the image data. The test image data is, for example, image data showing a cat's eyes such as the first image data, or image data showing a vehicle headlight such as the second image data.
　学習済モデル選択部５５は、学習済モデル格納部５１から学習済モデルを選択し、選択した学習済モデルを画像データ生成部４７に読み出させる指示を画像データ生成部４７に入力する。 The trained model selection unit 55 selects a trained model from the trained model storage unit 51, and inputs an instruction to the image data generation unit 47 to read the selected trained model into the image data generation unit 47.
　テスト用画像データ入力部５３及び学習済モデル選択部５５は、キーボード、マウスといった入力するためのデバイスである。 The test image data input unit 53 and the trained model selection unit 55 are devices for input such as a keyboard and a mouse.
　画像データ生成部４７は、学習済モデル選択部５５からの指示によって学習済モデル格納部５１にアクセスし、学習済モデル選択部５５によって選択された学習済モデルを学習済モデル格納部５１から読み出す。次に、画像データ生成部４７は、読み出した学習済モデルを用いてテスト用画像データから新たな画像データを生成する。生成された新たな画像データは、画像出力部５７に入力される。 The image data generation unit 47 accesses the trained model storage unit 51 according to the instruction from the trained model selection unit 55, and reads the trained model selected by the trained model selection unit 55 from the trained model storage unit 51. Next, the image data generation unit 47 generates new image data from the test image data using the read learned model. The generated new image data is input to the image output unit 57.
　画像出力部５７は、例えば、モニタである。画像出力部５７は、画像データ生成部４７によって生成される新たな画像データを画像として出力する。 The image output unit 57 is, for example, a monitor. The image output unit 57 outputs new image data generated by the image data generation unit 47 as an image.
　次に、画像生成装置１０の動作について説明する。画像生成装置１０の動作は、学習済モデル生成ステップと、画像生成ステップとを主なステップとして備える。 Next, the operation of the image generator 10 will be described. The operation of the image generation device 10 includes a trained model generation step and an image generation step as main steps.
　図３は、学習済モデル生成ステップを示すフローチャートである。 FIG. 3 is a flowchart showing the trained model generation steps.
　（ステップＳ１）
　本ステップでは、第１画像データ入力部２３は複数の第１画像データを学習部４１に読み出させる指示を学習部４１に入力し、学習部４１は複数の第１画像データを記録部２１から読み出す。また、第２画像データ入力部２５は第２画像データを学習部４１に読み出させる指示を学習部４１に入力し、学習部４１は第２画像データを記録部２１から読み出す。学習部４１が複数の第１画像データ及び複数の第２画像データを読み出すと、処理はステップＳ２に移行する。 (Step S1)
In this step, the first image data input unit 23 inputs an instruction to cause the learning unit 41 to read a plurality of first image data, and the learning unit 41 inputs a plurality of first image data from the recording unit 21. read out. Further, the second image data input unit 25 inputs an instruction to cause the learning unit 41 to read the second image data, and the learning unit 41 reads the second image data from the recording unit 21. When the learning unit 41 reads out the plurality of first image data and the plurality of second image data, the process proceeds to step S2.
　（ステップＳ２）
　本ステップでは、比重入力部２７は比重λＡ１，λＢ１を設定部４３に入力し、設定部４３は比重λＡ，λＢとして比重λＡ１，λＢ１を設定する。設定された比重λＡ１，λＢ１は生成部４５及び識別部４６に入力され、処理はステップＳ３に移行する。 (Step S2)
In this step, the specific gravity input unit 27 inputs the specific densities λA1 and λB1 to the setting unit 43, and the setting unit 43 sets the specific densities λA1 and λB1 as the specific densities λA and λB. The set specific densities λA1 and λB1 are input to the generation unit 45 and the identification unit 46, and the process proceeds to step S3.
　（ステップＳ３）
　本ステップでは、学習回数入力部２９は学習部４１の学習回数を設定部４３に入力し、設定部４３は入力学習回数を設定する。設定された学習回数は生成部４５及び識別部４６に入力され、処理はステップＳ４に移行する。ここでは、学習回数を例えば１００回とする。 (Step S3)
In this step, the learning number input unit 29 inputs the learning number of the learning unit 41 to the setting unit 43, and the setting unit 43 sets the input learning number. The set number of learnings is input to the generation unit 45 and the identification unit 46, and the process proceeds to step S4. Here, the number of learnings is set to, for example, 100 times.
　（ステップＳ４）
　本ステップでは、学習部４１は、現在の学習回数をチェックする。学習回数が１００回未満である場合には処理はステップＳ５に移行し、学習回数が１００回未満でない場合には処理はステップＳ７に移行する。なお、学習済モデル生成ステップが開始されて処理が一度目のステップＳ４に移行した場合、学習回数は０とされる。 (Step S4)
In this step, the learning unit 41 checks the current number of learning times. If the number of learnings is less than 100, the process proceeds to step S5, and if the number of learnings is not less than 100, the process proceeds to step S7. When the trained model generation step is started and the process shifts to the first step S4, the number of trainings is set to 0.
　（ステップＳ５）
　本ステップでは、学習部４１は、後述する学習処理に移行する。学習処理が終了すると、処理はステップＳ６に移行する。 (Step S5)
In this step, the learning unit 41 shifts to the learning process described later. When the learning process is completed, the process proceeds to step S6.
　（ステップＳ６）
　本ステップでは、学習部４１は現在の学習回数を１つ加算し、処理はステップＳ４に戻る。 (Step S6)
In this step, the learning unit 41 adds one to the current number of learnings, and the process returns to step S4.
　（ステップＳ７）
　本ステップでは、ステップＳ２において設定された比重λＡ１，λＢ１に対応する第１学習済モデルが１００回の学習処理により完成したことになり、完成した第１学習済モデルは学習済モデル格納部５１に格納される。 (Step S7)
In this step, the first trained model corresponding to the specific densities λA1 and λB1 set in step S2 is completed by the learning process 100 times, and the completed first trained model is stored in the trained model storage unit 51. It is stored.
　次に、ステップＳ５における学習部４１の学習処理について説明する。図４は、学習部４１の学習処理を示すフローチャートである。 Next, the learning process of the learning unit 41 in step S5 will be described. FIG. 4 is a flowchart showing the learning process of the learning unit 41.
　（ステップＳ１１）
　本ステップでは、学習部４１は、ステップＳ１において記録部２１から読み出したそれぞれの第１画像データ及びそれぞれの第２画像データに順番ｉを割り振る。上記したように、第１画像データの数及び第２画像データの数がそれぞれ１４０００個であるため、順番ｉは１から１４０００となる。順番が割り振られると、処理はステップＳ１２に進む。 (Step S11)
In this step, the learning unit 41 allocates the order i to each of the first image data and each second image data read from the recording unit 21 in step S1. As described above, since the number of the first image data and the number of the second image data are 14,000 each, the order i is 1 to 14000. When the order is assigned, the process proceeds to step S12.
　（ステップＳ１２）
　本ステップでは、学習部４１は、これから学習処理を行う第１画像データ及び第２画像データの順番ｉをチェックする。順番ｉが上記した１４０００未満であれば、処理はステップＳ１３に進む。順番ｉが１４０００未満でなければ、全ての第１画像データ及び全ての第２画像データにステップＳ２において設定された比重λＡ１，λＢ１で学習処理が行われたとして、処理はステップＳ６に進む。 (Step S12)
In this step, the learning unit 41 checks the order i of the first image data and the second image data to be learned. If the order i is less than 14,000 described above, the process proceeds to step S13. If the order i is not less than 14,000, it is assumed that the learning process is performed on all the first image data and all the second image data with the specific densities λA1 and λB1 set in step S2, and the process proceeds to step S6.
　（ステップＳ１３）
　本ステップでは、学習部４１はｉ番目の第１画像データ及び第２画像データを取得し、処理はステップＳ１４に進む。なお、学習処理が開始されて処理が一度目のステップＳ１３に移行した場合、ｉは１とされる。 (Step S13)
In this step, the learning unit 41 acquires the i-th first image data and the second image data, and the process proceeds to step S14. When the learning process is started and the process shifts to the first step S13, i is set to 1.
　（ステップＳ１４）
　本ステップでは、学習部４１はｉ番目の第１画像データ及びｉ番目の第２画像データに対して式（１）で示される損失関数を用いる計算を行い、処理はステップＳ１５に進む。 (Step S14)
In this step, the learning unit 41 performs a calculation using the loss function represented by the equation (1) for the i-th first image data and the i-th second image data, and the process proceeds to step S15.
　（ステップＳ１５）
　本ステップでは、学習部４１において学習を行い、処理はステップＳ１６に進む。 (Step S15)
In this step, learning is performed in the learning unit 41, and the process proceeds to step S16.
　（ステップＳ１６）
　本ステップでは、学習部４１は現在の順番ｉを１つ加算し、処理はステップＳ１２に戻る。 (Step S16)
In this step, the learning unit 41 adds one of the current order i, and the process returns to step S12.
　図３及び図４に示す処理では、比重λＡ，λＢが比重λＡ１，λＢ１と設定された状態で、１番目から１４０００番目それぞれにおける第１画像データ及び第２画像データに対して式（１）で示される損失関数を用いる計算である学習処理が１００回行われる。学習処理が１００回行われると、比重λＡ１，λＢ１に対応する第１学習済モデルが完成する。 In the processes shown in FIGS. 3 and 4, with the specific densities λA and λB set to the specific densities λA1 and λB1, the first image data and the second image data in the 1st to 14000th positions are expressed by the equation (1). The learning process, which is a calculation using the shown loss function, is performed 100 times. When the learning process is performed 100 times, the first trained model corresponding to the specific densities λA1 and λB1 is completed.
　第１学習済モデルが完成した後に、比重λＡ，λＢがステップＳ２において比重λＡ２，λＢ２に設定されると、第１学習済モデルの生成と同様に、比重λＡ２，λＢ２に対応する第２学習済モデルが１００回の学習処理により生成される。また、比重λＡ，λＢが比重λＡ３，λＢ３に設定されると、第１学習済モデルの生成と同様に、比重λＡ３，λＢ３に対応する第３学習済モデルが１００回の学習処理により生成される。従って、学習済モデルはステップＳ２において設定された比重λＡ，λＢの値毎に生成され、生成されたそれぞれの学習済モデルは学習済モデル格納部５１に格納される。それぞれの学習済モデルが学習済モデル格納部５１に格納されると、学習済モデル生成ステップにおける処理は終了する。 When the specific densities λA and λB are set to the specific densities λA2 and λB2 in step S2 after the first trained model is completed, the second trained corresponding to the specific densities λA2 and λB2 is similar to the generation of the first trained model. The model is generated by 100 learning processes. Further, when the specific densities λA and λB are set to the specific densities λA3 and λB3, the third trained model corresponding to the specific gravities λA3 and λB3 is generated by 100 times of learning processing in the same manner as the generation of the first trained model. .. Therefore, the trained model is generated for each value of the specific densities λA and λB set in step S2, and each generated trained model is stored in the trained model storage unit 51. When each trained model is stored in the trained model storage unit 51, the process in the trained model generation step ends.
　次に、図５を参照して、画像生成ステップについて説明する。図５は、画像生成ステップを示すフローチャートである。画像生成ステップは、学習済モデル生成ステップによって、複数の学習済モデルが構築された後に行われる。 Next, the image generation step will be described with reference to FIG. FIG. 5 is a flowchart showing an image generation step. The image generation step is performed after a plurality of trained models have been constructed by the trained model generation step.
　（ステップＳ２１）
　本ステップでは、テスト用画像データがテスト用画像データ入力部５３によって記録部２１から画像データ生成部４７に入力される。また、学習済モデル選択部５５によって選択された学習済モデルが学習済モデル格納部５１から画像データ生成部４７に入力される。ここでは、テスト用画像データは、第１画像データのような猫の目を示す画像データとしており、第１画像データと同じカテゴリーに区分されるデータである。また、訓練用画像データは、第２画像データとしている。 (Step S21)
In this step, the test image data is input from the recording unit 21 to the image data generation unit 47 by the test image data input unit 53. Further, the trained model selected by the trained model selection unit 55 is input from the trained model storage unit 51 to the image data generation unit 47. Here, the test image data is image data indicating the cat's eyes such as the first image data, and is data classified into the same category as the first image data. The training image data is the second image data.
　（ステップＳ２２）
　本ステップでは、画像データ生成部４７は、画像データ生成部４７に入力された学習済モデルをチェックする。入力された学習済モデルが第１学習済モデルである場合、処理はステップＳ２３に移行する。入力された学習済モデルが第２学習済モデルである場合、処理はステップＳ２４に移行する。入力された学習済モデルが第３学習済モデルである場合、処理はステップＳ２５に移行する。 (Step S22)
In this step, the image data generation unit 47 checks the trained model input to the image data generation unit 47. If the input trained model is the first trained model, the process proceeds to step S23. If the input trained model is the second trained model, the process proceeds to step S24. If the input trained model is the third trained model, the process proceeds to step S25.
　（ステップＳ２３）
　本ステップでは、画像データ生成部４７は、第１学習済モデルを用いてテスト用画像データから新たな画像データを生成する。ここで、第１学習済モデルでは、比重λＡ１が比重λＢ１よりも大きくされている。従って、テスト用画像データの変換度合いが大きくなり、新たな画像データは、第２ドメインよりも第１ドメインに近似し、テスト用画像データから予想が容易ではない新たなデザイン性を備える画像データとなる。本ステップにおいて生成される新たな画像データは、猫の目に最も近似した車両用前照灯を示す画像データとなる。そして、新たな画像データは画像出力部５７に入力され、処理はステップＳ２６に移行する。 (Step S23)
In this step, the image data generation unit 47 generates new image data from the test image data using the first trained model. Here, in the first trained model, the specific gravity λA1 is larger than the specific density λB1. Therefore, the degree of conversion of the test image data becomes large, and the new image data is closer to the first domain than the second domain, and the image data has a new design that is not easy to predict from the test image data. Become. The new image data generated in this step is the image data indicating the vehicle headlight that most closely resembles the cat's eyes. Then, the new image data is input to the image output unit 57, and the process proceeds to step S26.
　（ステップＳ２４）
　本ステップでは、画像データ生成部４７は、第２学習済モデルを用いてテスト用画像データから新たな画像データを生成する。第２学習済モデルでは、比重λＡ２が比重λＢ２と同じであるため、新たな画像データは、猫の目と車両用前照灯との中間の画像データとなる。そして、新たな画像データは画像出力部５７に入力され、処理はステップＳ２６に移行する。 (Step S24)
In this step, the image data generation unit 47 generates new image data from the test image data using the second trained model. In the second trained model, since the specific gravity λA2 is the same as the specific density λB2, the new image data is an intermediate image data between the cat's eyes and the vehicle headlight. Then, the new image data is input to the image output unit 57, and the process proceeds to step S26.
　（ステップＳ２５）
　本ステップでは、画像データ生成部４７は、第３学習済モデルを用いてテスト用画像データから新たな画像データを生成する。ここで、第３学習済モデルでは、比重λＡ３が比重λＢ３よりも小さくされている。従って、テスト用画像データの変換度合いがより大きくなり、新たな画像データは、第１ドメインよりも第２ドメインに近似し、テスト用画像データから予想が困難な新たなデザイン性を備える画像データとなる。本ステップにおいて生成される新たな画像データは、猫の目に近似した車両用前照灯を示す画像データとなる。そして、新たな画像データは画像出力部５７に入力され、処理はステップＳ２６に移行する。 (Step S25)
In this step, the image data generation unit 47 generates new image data from the test image data using the third trained model. Here, in the third trained model, the specific density λA3 is smaller than the specific density λB3. Therefore, the degree of conversion of the test image data becomes larger, and the new image data is closer to the second domain than the first domain, and the image data has a new design that is difficult to predict from the test image data. Become. The new image data generated in this step is image data showing a vehicle headlight that approximates the eyes of a cat. Then, the new image data is input to the image output unit 57, and the process proceeds to step S26.
　（ステップＳ２６）
　本ステップでは、画像出力部５７は新たな画像データを画像として出力し、画像生成ステップにおける処理は終了する。 (Step S26)
In this step, the image output unit 57 outputs new image data as an image, and the process in the image generation step ends.
　以上のように、本実施形態の画像生成装置１０は、複数の第１画像データ及び複数の第２画像データを記録する記録部２１と、第１画像データの第１ドメインの第１比重及び第２画像データの第２ドメインの第２比重をそれぞれ設定する設定部４３とを備える。画像生成装置１０は、複数の第１画像データ及び複数の第２画像データから第１比重及び第２比重に対応する学習済モデルを第１比重及び第２比重の値毎に生成する学習部４１と、複数の学習済モデルを格納する学習済モデル格納部５１と、学習済モデル格納部５１に格納される複数の学習済モデルから１つの学習済モデルを選択する学習済モデル選択部５５とをさらに備える。また、画像生成装置１０は、テスト用画像データを入力するテスト用画像データ入力部５３と、学習済モデル選択部５５によって選択される学習済モデルを用いてテスト用画像データ入力部５３から入力されるテスト用画像データから新たな画像データを生成する画像データ生成部４７とをさらに備える。 As described above, the image generation device 10 of the present embodiment includes a recording unit 21 that records a plurality of first image data and a plurality of second image data, and a first specific gravity and a first domain of the first image data. 2 The setting unit 43 for setting the second specific weight of the second domain of the image data is provided. The image generation device 10 generates a trained model corresponding to the first specific gravity and the second specific gravity from the plurality of first image data and the plurality of second image data for each value of the first specific gravity and the second specific gravity. A trained model storage unit 51 that stores a plurality of trained models, and a trained model selection unit 55 that selects one trained model from a plurality of trained models stored in the trained model storage unit 51. Further prepare. Further, the image generation device 10 is input from the test image data input unit 53 using the test image data input unit 53 for inputting the test image data and the trained model selected by the trained model selection unit 55. It further includes an image data generation unit 47 that generates new image data from the test image data.
　本実施形態の画像生成装置１０では、予想の困難性を示す比重λＡ及び比重λＢの値は設定可能であり、比重λＡ及び比重λＢの値毎に対応する学習済モデルが生成され、複数の学習済モデルのなかから１つの学習済モデルを用いてテスト用画像データから新たな画像データを生成する。新たな画像データの生成時において、比重λＡ及び比重λＢの値に対応する学習済モデルに応じてテスト用画像データの変換度合いが変化する。変換度合いが変化すると、新たな画像データは、訓練用画像データである第１画像データ及び第２画像データから予想が困難な画像データとなり得、新たなデザイン性を備えることになり得る。従って、本実施形態の画像生成装置１０は、予想の困難性を設定したうえで、複数の第１，２画像データから予想が容易ではない新たなデザイン性を備える新たな画像データを生成し得る。 In the image generation device 10 of the present embodiment, the values of the specific gravity λA and the specific gravity λB indicating the difficulty of prediction can be set, and a trained model corresponding to each value of the specific gravity λA and the specific gravity λB is generated, and a plurality of trainings are performed. New image data is generated from the test image data using one trained model from the completed models. At the time of generating new image data, the degree of conversion of the test image data changes according to the trained model corresponding to the values of the specific gravity λA and the specific gravity λB. When the degree of conversion changes, the new image data can become image data that is difficult to predict from the first image data and the second image data, which are training image data, and can be provided with new designability. Therefore, the image generation device 10 of the present embodiment can generate new image data having a new design property that is not easy to predict from the plurality of first and second image data after setting the difficulty of prediction. ..
　また、本実施形態の画像生成装置１０では、設定部４３は複数の比重λＡ及び複数の比重λＢを設定し、学習部４１は複数の比重λＡ及び複数の比重λＢに対応する複数の学習済モデルを生成する。また、本実施形態の画像生成装置１０では、画像データ生成部４７は、複数の学習済モデルのなかから１つの学習済モデルを用いて新たな画像データを生成する。複数の学習済モデルが生成されることによって、１つの学習済モデルのみが生成される場合に比べて、様々な種類の新たな画像データが生成され得る。 Further, in the image generation device 10 of the present embodiment, the setting unit 43 sets a plurality of specific densities λA and a plurality of specific densities λB, and the learning unit 41 sets a plurality of learned models corresponding to the plurality of specific radii λA and the plurality of specific densities λB. To generate. Further, in the image generation device 10 of the present embodiment, the image data generation unit 47 generates new image data using one trained model from among the plurality of trained models. By generating a plurality of trained models, various kinds of new image data can be generated as compared with the case where only one trained model is generated.
　以上、本発明について、上記実施形態を例に説明したが、本発明はこれらに限定されるものではない。 Although the present invention has been described above by taking the above-described embodiment as an example, the present invention is not limited thereto.
　画像データ生成部４７は、学習部４１において学習した生成部４５であってもよい。または、学習部４１は、学習処理において学習した生成部４５を画像データ生成部４７に提供してもよい。 The image data generation unit 47 may be the generation unit 45 learned in the learning unit 41. Alternatively, the learning unit 41 may provide the generation unit 45 learned in the learning process to the image data generation unit 47.
　ステップＳ３において設定される学習回数は、構築されるそれぞれの学習済モデル毎に設定されてもよい。従って、例えば、第１学習済モデルの構築における学習回数は、他の学習済モデルの構築における学習回数と同じであってもよいし、他の学習済モデルの構築よりも多くても少なくてもよい。学習回数が多いほど、学習回数が多い学習済モデルが用いられる場合に、複数の第１，２画像データから予想が容易ではない新たなデザイン性を備える新たな画像データが生成され易くなり得る。また、学習回数が少ないほど、学習済モデルは早く生成され得る。また、学習部４１は、３つの学習済モデルを生成しているが、これに限定する必要はなく、少なくとも１つの学習済モデルを生成すればよい。 The number of learnings set in step S3 may be set for each learned model to be constructed. Therefore, for example, the number of trainings in the construction of the first trained model may be the same as the number of trainings in the construction of other trained models, and may be more or less than the number of trainings in the construction of other trained models. Good. As the number of learnings increases, when a trained model with a large number of learnings is used, new image data having a new design that is not easy to predict can be easily generated from the plurality of first and second image data. Also, the smaller the number of trainings, the faster the trained model can be generated. Further, although the learning unit 41 generates three trained models, it is not necessary to limit the learning unit 41 to this, and at least one trained model may be generated.
　学習部４１は、Ｃｙｃｌｅ　ＧＡＮ方式に従って学習済モデルを生成するが、これに限定される必要はない。 The learning unit 41 generates a trained model according to the Cycle GAN method, but the learning unit 41 does not have to be limited to this.
　設定部４３は、比重入力部２７から入力された比重を比重λＡ，λＢとして設定しているが、これに限定される必要はない。例えば、設定部４３は、制御部４０のメモリに予め設定される比重を比重λＡ，λＢとして設定してもよい。また、設定部４３は、学習回数入力部２９から入力された値を学習回数として設定しているが、これに限定される必要はない。例えば、設定部４３は、制御部４０のメモリに予め設定される値を学習回数として設定してもよい。 The setting unit 43 sets the specific densities input from the specific densities input unit 27 as the specific densities λA and λB, but it is not necessary to be limited to these. For example, the setting unit 43 may set the specific densities preset in the memory of the control unit 40 as the specific densities λA and λB. Further, the setting unit 43 sets the value input from the learning number input unit 29 as the learning number, but the setting unit 43 does not have to be limited to this. For example, the setting unit 43 may set a value preset in the memory of the control unit 40 as the number of learnings.
　それぞれの第１画像データは、猫の目を示す画像データとして説明しているが、これに限定される必要はなく、他の動物の目などを示す画像データであってもよい。 Each first image data is described as image data indicating the eyes of a cat, but it is not necessary to be limited to this, and image data indicating the eyes of other animals may be used.
　本発明によれば、予想の困難性を設定したうえで、複数の画像データから予想が容易ではない新たなデザイン性を備える画像データを生成することができる画像生成装置が提供され、当該画像生成装置は画像生成の分野等において利用可能である。
  According to the present invention, an image generation device capable of generating image data having a new design that is not easy to predict from a plurality of image data after setting the difficulty of prediction is provided, and the image generation is performed. The device can be used in the field of image generation and the like.

Claims (4)

  1. 　複数の第１画像データ及び複数の第２画像データを記録する記録部と、
    　前記第１画像データの第１ドメインの第１比重及び前記第２画像データの第２ドメインの第２比重をそれぞれ設定する設定部と、
    　前記複数の第１画像データ及び前記複数の第２画像データから前記第１比重及び前記第２比重に対応する学習済モデルを前記第１比重及び前記第２比重の値毎に生成する学習部と、
    　複数の前記学習済モデルを格納する学習済モデル格納部と、
    　前記学習済モデル格納部に格納される前記複数の学習済モデルから１つの学習済モデルを選択する学習済モデル選択部と、
    　テスト用画像データを入力するテスト用画像データ入力部と、
    　前記学習済モデル選択部によって選択される前記学習済モデルを用いて前記テスト用画像データ入力部から入力される前記テスト用画像データから新たな画像データを生成する画像データ生成部と、
    を備えることを特徴とする画像生成装置。     A recording unit that records a plurality of first image data and a plurality of second image data,
    A setting unit for setting the first specific density of the first domain of the first image data and the second specific density of the second domain of the second image data, respectively.
    A learning unit that generates a trained model corresponding to the first specific density and the second specific density from the plurality of first image data and the plurality of second image data for each value of the first specific density and the second specific gravity. ,
    A trained model storage unit that stores a plurality of the trained models,
    A trained model selection unit that selects one trained model from the plurality of trained models stored in the trained model storage unit, and a trained model selection unit.
    A test image data input unit for inputting test image data,
    An image data generation unit that generates new image data from the test image data input from the test image data input unit using the trained model selected by the trained model selection unit.
    An image generator characterized by comprising.
  2. 　前記学習部は、Ｃｙｃｌｅ　ＧＡＮ方式に従って前記学習済モデルを生成する
    ことを特徴とする請求項１に記載の画像生成装置。     The image generation device according to claim 1, wherein the learning unit generates the trained model according to a Cycle GAN method.
  3. 　前記学習部は、前記Ｃｙｃｌｅ　ＧＡＮ方式において用いられる計算をそれぞれの前記学習済モデルに設定される学習回数行いそれぞれの前記学習済モデルを生成する
    ことを特徴とする請求項２に記載の画像生成装置。     The image generation device according to claim 2, wherein the learning unit performs a calculation used in the Cycle GAN method by a number of learning times set in each of the learned models to generate each of the learned models. ..
  4. 　前記新たな画像データを出力する出力部をさらに備える
    ことを特徴とする請求項１から３のいずれか１項に記載の画像生成装置。
          The image generation device according to any one of claims 1 to 3, further comprising an output unit for outputting the new image data.
PCT/JP2020/043904 2019-11-27 2020-11-25 Image generation device WO2021106961A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2021561466A JPWO2021106961A1 (en) 2019-11-27 2020-11-25

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-214201 2019-11-27
JP2019214201 2019-11-27

Publications (1)

Publication Number Publication Date
WO2021106961A1 true WO2021106961A1 (en) 2021-06-03

Family

ID=76129498

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/043904 WO2021106961A1 (en) 2019-11-27 2020-11-25 Image generation device

Country Status (2)

Country Link
JP (1) JPWO2021106961A1 (en)
WO (1) WO2021106961A1 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019093126A (en) * 2017-11-24 2019-06-20 キヤノンメディカルシステムズ株式会社 Medical data processor, magnetic resonance imaging device and learned model creation method

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019093126A (en) * 2017-11-24 2019-06-20 キヤノンメディカルシステムズ株式会社 Medical data processor, magnetic resonance imaging device and learned model creation method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SUGIURA, AKIHIKO,: "A simple test of pediatric depression using facial expressions cognition", vol. 20, no. 6, June 2009 (2009-06-01), pages 8 - 31 *

Also Published As

Publication number Publication date
JPWO2021106961A1 (en) 2021-06-03

Similar Documents

Publication Publication Date Title
KR102042168B1 (en) Methods and apparatuses for generating text to video based on time series adversarial neural network
US20220215267A1 (en) Processes and methods for enabling artificial general intelligence capable of flexible calculation, prediction, planning and problem solving with arbitrary and unstructured data inputs and outputs
KR102115534B1 (en) Method for detecting anomaly using generative adversarial networks, apparatus and system thereof
US20050119982A1 (en) Information processing apparatus and method
JP2007265345A (en) Information processor and method, learning device and method, and program
JP7192895B2 (en) MODEL BUILDING DEVICE, MODEL BUILDING METHOD, COMPUTER PROGRAM AND RECORDING MEDIUM
WO2021106961A1 (en) Image generation device
CN116912629B (en) General image text description generation method and related device based on multi-task learning
US11568303B2 (en) Electronic apparatus and control method thereof
JP6622369B1 (en) Method, computer and program for generating training data
WO2020071213A1 (en) Acoustic model learning device, voice synthesis device, and program
WO2016203757A1 (en) Control device, information processing device in which same is used, control method, and computer-readable memory medium in which computer program is stored
KR20200058297A (en) Method and device for explainable few-shot image classification
WO2020202244A1 (en) Model generating device, model adjusting device, model generating method, model adjusting method, and recording medium
KR102105951B1 (en) Constructing method of classification restricted boltzmann machine and computer apparatus for classification restricted boltzmann machine
JP4773680B2 (en) Information processing apparatus and method, program recording medium, and program
US7324980B2 (en) Information processing apparatus and method
WO2021220343A1 (en) Data generation device, data generation method, learning device, and recording medium
JP7487556B2 (en) MODEL GENERATION PROGRAM, MODEL GENERATION DEVICE, AND MODEL GENERATION METHOD
WO2020054402A1 (en) Neural network processing device, computer program, neural network manufacturing method, neural network data manufacturing method, neural network use device, and neural network downscaling method
CN113077383A (en) Model training method and model training device
JP6902071B2 (en) Motion synthesizer and motion synthesizer
JPWO2018066083A1 (en) Learning program, information processing apparatus and learning method
JP2022075329A (en) Recognition system, recognition method, program, learning method, learned model, distillation model, and method for generating dataset for learning
JP2021135683A (en) Learning device, deduction device, method for learning, and method for deduction

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20892675

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021561466

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20892675

Country of ref document: EP

Kind code of ref document: A1