JP6719497B2 - Image generation method, image generation device, and image generation system - Google Patents

Description

The present invention relates to an image generation method, an image generation device, and an image generation system.

In recent years, in image processing technology, by using a machine learning method (a neural network such as deep learning), detection accuracy of recognizing a specific object from a captured image has been improved. One of the methods for optimizing these machine learning methods is to input a large number of images to the system as training samples and train the machine learning.

For example, as a system for performing high-accuracy image recognition by automatically collecting a recognition target and an image pattern not including the recognition target and using the collected image pattern in machine learning There is a technique described in 088787 (Patent Document 1). In this publication, "the object tracking section extracts a region in which a recognition target is reflected from the image of each frame forming a moving image. The image conversion section performs geometric transformation on the image in this area. A sample to be recognized is generated based on the image, the region cutting unit sets a region for an image of a frame forming a moving image, and the image synthesis unit 35 sets a plurality of images in the image in each set region. The non-recognition target sample image is generated based on the image obtained by combining the regions of 1. The learning unit learns the recognition target by using the recognition target sample and the non-recognition target sample.

JP 2012-088787 A

If training sample images are easily available, machine learning performance is likely to improve, but if training sample images are difficult or unavailable, improve the accuracy of image object detection by machine learning. Is difficult. As a result, the user will have to obtain sample images to train machine learning at a cost. However, in Patent Document 1, how to deal with the case where the training sample image is difficult to obtain has not been deeply examined, and regarding the cost burden for the user to acquire the training sample image, , Still unsolved.

Therefore, in the present invention, an image for machine learning training is generated from data such as a vector model or a 3D model by using a neural network, and the generated image is used for machine learning training, whereby the efficiency of machine learning training is improved. The purpose is to improve the detection accuracy of images and images.

In order to solve the above-mentioned problems, one of the representative image generation methods of the present invention is a background image acquisition step of acquiring a background image by an image selection unit, and a detection provided with metadata by the image selection unit. A detection target image specifying step of specifying the target image from the source image, a model generation step of generating a detection target image model corresponding to the detection target image by the model creating section,
An image generation method including a detection target image establishing step of establishing a final image by combining the background image and the detection target image model by the model creating unit.

An image for machine learning training is generated from a data such as a vector model or a 3D model by a neural network, and the generated image is used for machine learning to improve the efficiency of machine learning training and the accuracy of image detection. You can

It is a figure which shows the whole system configuration of the hardware which concerns on embodiment of this invention. 3 is a flowchart of an image generation method according to the first embodiment of the present invention. It is a figure for demonstrating the calculation method of the camera parameter which concerns on 1st Embodiment of this invention. 9 is a flowchart of a modified example of the image generation method according to the first embodiment of the present invention. It is a figure which shows an example of the vector model and image which concern on 1st Embodiment of this invention. It is a figure which shows an example of the correspondence of the vector model and image which concern on 1st Embodiment of this invention. It is a figure which shows an example of the process for establishing the detection target image which concerns on 1st Embodiment of this invention. 6 is a flowchart of an image generation method according to the embodiment of the present invention. It is a figure which shows an example of the process for establishing the detection target image which concerns on 2nd Embodiment of this invention. It is a figure which shows an example of the process for establishing the image for detection which concerns on 3rd Embodiment of this invention. It is a figure which shows an example of the process of improving the ability to create the machine learning image which concerns on 4th Embodiment of this invention. It is a figure which shows an example of the method of producing|generating the image which concerns on 5th Embodiment of this invention. It is a figure which shows an example of the process of improving the detection accuracy of the machine learning which concerns on 6th Embodiment of this invention. It is a figure which shows an example of the system architecture which concerns on embodiment of this invention. It is a conceptual diagram explaining an example of embodiment of this invention.

Hereinafter, a conventional example and a first embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to this embodiment. In the description of the drawings, the same parts are designated by the same reference numerals.

First, an example of the concept of the embodiment of the present invention will be described with reference to FIG.

In machine learning for detecting an object, a large amount of images of an object to be detected are required to train a machine learning system. For example, when trying to detect a person who has a white cane (hereinafter, also referred to as “white cane”) using machine learning, conventionally, it was not possible to learn without a large number of images of the white cane. Therefore, in the present invention, even when the image of the detection target (for example, a white cane person) is small, the image of a general pedestrian (a large amount is available) and the image of a white cane person (a small amount) are used. By doing so, a large amount of white cane images are generated, and machine learning is efficiently enhanced.

[Image generation system configuration]
FIG. 1 is a diagram showing an overall system configuration of hardware according to an embodiment of the present invention. As shown in FIG. 1, this system includes a central server 100, a client terminal 130, a client terminal 140, and a network (Internet LAN or the like) 150. The central server 100, the client terminal 130, and the client terminal 140 may be communicatively connected to each other via the network 150.

The central server 100 is a device that performs image generation requested by the client terminals 130 and 140 via the network 150. Specifically, the central server 100 can include a functional unit that performs functions such as image selection, model creation, image processing, and machine learning in the image generation process. Further, the central server 100 may include a unit (for example, the storage unit 120) that stores image data such as a background image and a detection target image described later, and model data such as a vector model and a 3D model.

The client terminal 130 and the client terminal 140 are devices for transmitting an image generation request to the central server 100 via the network 150. Specifically, the user can input image generation conditions to the client terminal 130 and the client terminal 140. For example, the user may use the client terminal 130 or the client terminal 140 to specify a detection target object to be described later or a background image used for image generation. Instructions such as conditions input at the client terminals 130 and 140 are transmitted to the central server 100 via the network 150.

[Configuration of central server 100]
As described above, the central server 100 is a device that performs the image generation requested by the client terminals 130 and 140 via the network 150. As shown in FIG. 1, the central server 100 includes a processing unit 110 that performs each function of image generation, and a storage unit 120 that stores information used for the image generation.

The processing unit 110 includes a function unit for performing each function according to the embodiment of the present invention. Specifically, the processing unit 110 acquires a background image, generates an image selection unit 112 that identifies a detection target image including metadata from a source image, and generates a detection target image model corresponding to the detection target image, A model creating unit 114 that establishes a final image by combining a background image and a detection target image model, an image processing unit 116 that performs image processing on the model, a process for improving machine learning detection accuracy, and a machine learning image. It is composed of a machine learning unit 118 that executes each step of the processing for improving the creation ability.

The processing unit 110 functions as each of the functional units described above when an arithmetic processing unit such as a CPU (Central Processing Unit) inside the device executes the control program stored in the memory.

The storage unit 120 includes an image database 122 and an image/model database 124. The image database 122 is a database (apparatus or logical storage area) that stores a background image used for image generation and data of a detection target image described later. The image database 122 may store, for example, image data showing the appearance of a platform at a station as shown in FIG. 7 and metadata included in the image. In one embodiment, the storage unit 120 receives an image (source image, background image, desired detection target image) designated by the user from the client terminal 130 or the client terminal 140, and uses the received image data for image generation. The image data may be stored in the image database 122 in the obtained format. The image/model database 124 is a database (apparatus or logical storage area) that stores a specific image and a model associated with the image in a form associated with each other. For example, as shown in FIGS. 5 to 6 described later, a model (vector model, point cloud, etc.) used for image generation and a realistic image associated with the model are stored in the image/model database 124. May be saved. The image database 122 and the image/model database 124 of the storage unit 120 are realized by a semiconductor memory device such as a RAM (Random Access Memory) and a flash memory (Flash Memory), or a storage unit such as a hard disk and an optical disk. May be.

[Configuration of Client Terminal 130]
As described above, the client terminal 130 is a device for transmitting an image generation request to the central server 100 via the network 150. The client terminal 130 includes a processing unit 132 that executes a command sent from another functional unit in the terminal, an instruction receiving unit 134 that receives an instruction (image generation condition, etc.) from a user, and an image (source image, background image, An image selection unit 136 that selects a detection target image), a communication unit 138 that manages the exchange with the central server 100 and another network terminal (for example, the client terminal 140), and information (image data, a command from the user, etc.). And a storage unit 139 for storing. As described above, in an embodiment, the user can use the client terminal 130 to input image generation conditions and specify a source image, a background image, or a detection target image used for image generation. .. The client terminal 130 may send conditions and instructions input by the user to the central server 100.

[Configuration of Client Terminal 140]
Like the client terminal 130, the client terminal 140 is a device for transmitting an image generation request to the central server 100 via the network 150. Further, the client terminal 140, like the client terminal 130, a processing unit 142 that executes an instruction sent from another functional unit in the terminal, and an instruction receiving unit 144 that receives an instruction (image generation condition or the like) from a user. A communication unit 146 that manages exchanges with the central server 100 and another network terminal (for example, the client terminal 130), and a storage unit 148 that stores information (image data, commands from the user, etc.). The client terminal 140 differs from the client terminal 130 in that it does not have an image selection unit such as the image selection unit 136. Therefore, when the image generation request is sent from a terminal such as the client terminal 140 that does not have the image selection unit, the image selection unit 112 of the central server 100 selects the image used for the image generation according to the user's instruction. Alternatively, the image selection unit 112 of the central server 100 may be automatically (for example, randomly) selected.

Next, with reference to FIG. 2, background image acquisition, detection target identification, detection target image model creation, and detection target image establishment in the first embodiment will be described. FIG. 2 is a flowchart showing the flow of the image generation method according to the first embodiment of the present invention.

First, in step S200, a background image is acquired. As used herein, the term obtaining includes obtaining, receiving, securing, procuring, selecting, and designating. The background image is an image that becomes a final image when a detection target image described later is arranged. The background image may be, for example, an image showing various environments such as a platform of a station, a boarding gate of an airport, a venue of a concert or a sports game, a shopping mall, and the like. This background image may be designated by the user (for example, the image selection unit 136 of the client terminal 130), and is stored in the image database 122 of the storage unit 120 according to an instruction input by the user (for example, via the client terminal 140). The selected image may be selected by the image selection unit 112 of the central server 100.

Next, in step S220, the detection target image is specified. As used herein, the term identifying includes selecting, selecting, setting, designating, identifying, or detecting. The detection target image is an image showing an object that the user wants to arrange in the background image. For example, the detection target image may be an image showing a target object for which the machine learning unit is trained so that it can be detected from the image. The detection target image may include, for example, a person holding a white cane, a person in a specific outfit, a baggage exceeding a predetermined size, a certain type of animal, and the like. Further, the detection target image may include metadata. The metadata here indicates information indicating the position of the detection target image such as two-dimensional coordinates, the shape (rectangle, round) and size (length/height in pixels, etc.) of the detection target image. Information and information indicating the property of the detection target image (label of human, animal, luggage, automobile, etc.) may be included. This metadata may be used for machine learning training described below.

The detection target image may be specified by the image selection unit 112 of the central server 100 from the source image input by the user, or may be specified directly by the user's instruction. As an example, when the background image is designated as the image of the construction site, the instruction receiving unit 134 of the client terminal 130 or the instruction receiving unit 144 of the client terminal 140 uses the “helmet” included in the source image as the detection target image. It is assumed that the request is designated as "non-wearing person" and the request is received from the user. In this case, the instruction receiving unit 134 that has received this request transmits the user's instruction to the central server 100, and the image selecting unit 112 of the central server 100 selects the person who does not wear a helmet from the source images in accordance with the user's request. You may specify as a detection target image.

Next, in step S240, a detection target image model is created. As used herein, the term creating includes including creating, creating, forming, preparing, and creating. The detection target image model is a model that embodies the shape and structure of the object shown in the detection target image. As the detection target image model, for example, a vector model, a point cloud, a 3D model, or the like may be used. The detection target image model may be automatically performed by a well-known model creation tool, for example. As will be described later, even if the detection target image model created here is processed in the image processing unit 116 of the central server 100 and a hostile generation network to be described later, it may be finished as an image closer to a real image. Good.

Next, in step S260, the final image is established. As used herein, the expression establishing includes establishing, setting, establishing, creating, constructing, providing, and creating. The final image is an image generated by combining the background image acquired in step S200 and the detection target image model created in step S240. Specifically, the final image may be generated by the model creating unit 114 of the central server 100 combining the background image and the detection target image model. The details of the process of combining the background image and the detection target image model to generate the final image will be described with reference to FIG. 7, and thus the description thereof is omitted here.

In this way, a background image is acquired, a detection target image having metadata is specified from the source image, a detection target image model corresponding to the detection target image is generated, and the background image and the detection target image model are combined. By establishing the final image, it is possible to generate an image for machine learning even when the actual detection target image is few and it is difficult to obtain.

Next, with reference to FIG. 3, a camera parameter calculation method according to the first embodiment of the present invention will be described. As shown in FIG. 3, the background image 320, the ticket gate 321, the horizontal line 323, and the detection target image model 327 are used to calculate the camera parameters.

In the process of establishing the detection target image described above, in order to arrange the detection target image model 327 in the background image 320 with an appropriate size or the like, it is necessary to calculate the camera parameter of the background image 320. By using the camera parameters calculated here, the model creation unit 114 can arrange the detection target image model 327 on the background image 320 at an appropriate position, size, and orientation.

To calculate the camera parameters, the reference object is first identified from the background image 320. The reference object here is an object that serves as a guide for the size of the detection target image model arranged in the background image 320. For example, the reference object may be one whose size is generally known or which is easy to be estimated and inferred. As an example, here, the ticket gate 321 may be identified as the reference object. Camera parameters are then calculated based on the identified reference object dimensional elements (height, length, etc.). Specifically, by setting a reference such as a horizontal line 323 that matches the position of the ticket gate 321 identified as the reference object in the background image 320, the length/height of the ticket gate 321 in terms of the number of pixels is measured. It Then, by using the ratio obtained by dividing the actual length/height of the ticket gate 321 by the length/height in terms of the number of pixels in the background image 320, the size of the detection target image model should be Can be easily calculated. Therefore, the final image can be generated by combining the detection target image model 327 with the background image 320 based on the camera parameters calculated here.

As described above, by calculating the camera parameter based on the dimension element of the reference object and combining the detection target image model with the background image based on the calculated camera parameter, the detection target image model is appropriately selected. It can be arranged in size, position and orientation.

Next, a flow of a modified example of the image generation method according to the first embodiment of the present invention will be described with reference to FIG.

First, in step S400, a background image is acquired. Since the background image acquisition here is substantially the same as the background image acquisition of S200 in FIG. 3, description thereof will be omitted here.

Next, in step S420, the vector model representing the detection target image is arranged on the background image acquired in step S400. The vector model is a model in which the shape and structure of the object shown in the detection target image is represented by a space vector. For example, an example of the vector model is shown in FIG. FIG. 5 is a diagram showing an example of a vector model and an image according to the first embodiment of the present invention. The vector model 531 shown in FIG. 5 is a vector model of the human body. The vector model 531 may be arranged in the background image at an appropriate position, size, and orientation based on the camera parameters described with reference to FIG. 3, for example.

Next, in step S440, the vector model is adjusted. The adjustment of the vector model may be performed by the image processing unit 116 of the central server 100. Here, the vector model adjustment refers to an image in which the vector model arranged in step S420 is closer to an actual image by using a generally known image processing technique (hereinafter, also referred to as “realistic image”). Means to convert to. Specifically, the adjustment of the vector model here may be performed by, for example, an adversarial generation network (sometimes referred to as a “Generic Adversary Network” or “GAN”). As an example, the image processing unit 116 compares the vector model arranged in the background image with a model stored in the image/model database of the storage unit 120 of the central server 100, and finds that the vector model has a similarity to the vector model. The highest image may be selected. For example, as shown in FIG. 5, the vector model 531 may be converted into a realistic image 532 such as a person holding a white cane. The final image may be generated by superimposing the image selected here with the vector model.

Next, in step S460, machine learning training is performed using the final image generated in step S440. The machine learning here may be performed by the machine learning unit 118 of the central server 100. As described above, the final image in which the background image and the detection target image are combined may be performed using a method of training a neural network such as an adversarial generation network. Since the details of the machine learning training process will be described later, the description thereof is omitted here.

Then, in step S480, the system of machine learning trained in S460 is actually applied. For example, a system for machine learning trained with images generated by the method of the present embodiment is used for actual training such as accident detection of a fully autonomous vehicle, crack detection of a structure, and simulation of a natural disaster. It is considered to be useful when image data is difficult to obtain.

By arranging and adjusting the vector model in this way, a high-quality image for machine learning can be obtained.

Next, with reference to FIG. 6, an example of the correspondence between the vector model and the image according to the first embodiment of the present invention will be described.

As described above, a vector model and a realistic image corresponding to the vector model may be used for image generation according to the present invention. Here, the correspondence between a realistic image and a vector model will be described. First, the detection target that is the source of the detection target image 641 can be obtained by being photographed in a laboratory environment. Next, a generally known well-known edge/orientation detection algorithm such as Openpose is applied to the detection target image 641 to generate a vector model superimposed on the detection target image 641. .. Next, the vector model 643 and the detection target image 641 are stored in the image/model database 124 of the storage unit 120 as a model/image pair 645 associated with each other, so that each of the processing units 110 of the central server 100 can be stored. The functional units can be made accessible.

In this way, by storing the vector model and the realistic image in a form associated with each other, there is an advantage that the adversarial generation network can be easily trained.

Next, an example of processing for establishing a detection target image according to the first embodiment of the present invention will be described with reference to FIG. 7.

As shown in FIG. 7, an image showing the appearance of the platform at the station is acquired as the background image 301. As described with reference to FIG. 3, the camera parameters of the background image 301 are calculated based on the reference object such as the track 302 and the ticket gate 303. Next, the model creation unit 114 arranges the vector model 304 corresponding to the detection target image requested by the user on the background image 301 according to the position, size, and orientation specified by the calculated camera parameters. Next, the image processing unit 116 generates a realistic image 305 corresponding to the vector model 304 and inserts it into the background image 301 at the same position, size, and orientation as the vector model 304. At this stage, image processing such as light adjustment and edge harmony for blending this realistic image with the background image 301 may be performed. In this way, the final image 309 is obtained by combining the realistic image 305 corresponding to the detection target image and the background image 301. Also, as described above, this final image 309 may be used to train machine learning techniques such as neural networks and support vector machines. As described above, even when the number of detection targets is small and it is difficult to obtain, the invention of this embodiment can be used to generate an image for machine learning.

It should be noted that although an example in which one detection target image is combined with a background image has been described in the present embodiment, the present invention is not limited thereto, and a plurality of detection target images are arranged on one background image by repeating the above steps. It is also possible to do so.

Next, the flow of the image generating method according to the embodiment of the present invention will be described with reference to FIG.

First, in step S400, a background image is acquired. In step S420, the vector model is placed on the background image. In S440, the vector model is adjusted. Since these steps are substantially the same as the image generation method described with reference to FIG. 4, description thereof will be omitted here.

Although the example of using the generated final image for machine learning training has been described above, the present invention is not limited thereto, and the final image may be used for another purpose. Therefore, in step S800, the final image generated is provided after adjusting the vector model in step S440. This final image may be provided to the person who requested the generation of the image, or may be transmitted to a third party. In addition to machine learning training, the final image may be applied to advertising, face recognition, object detection, image processing, etc. As described above, the image generation method according to the present invention may be applied to various fields as well as to train machine learning.

Next, with reference to FIG. 9, an example of a process for establishing a detection target image according to the second embodiment of the present invention will be described.

Depending on the condition of the image generation request, it may be difficult or unnecessary to generate a model (vector model or the like) representing the detection target image. For example, as an example, if it is not necessary to depict small elements (for example, color, shape, structure, etc.) of the detection target image, in order to suppress the file size of the final image, a vector model or a realistic image is used. There are cases in which a poor image may be used instead. Therefore, the process for establishing the detection target image according to the second embodiment of the present invention is not a vector model or a realistic image, but a partial image is simply inserted as a detection target image into the background image. The above problems are solved.

In FIG. 9, an image showing the appearance of the platform at the station is acquired as the background image 301. Then, in FIG. 9, as in FIG. 7, the background image 301 includes reference objects such as the track 302 and the ticket gate 303. Next, in the second embodiment, the position/size specified in the camera parameter (or the image generation condition input by the user) calculated based on the reference object such as the railroad track 302 and the ticket gate 303 is set. Accordingly, the partial image 314 is defined as the detection target image. The partial image is, for example, an image having an arbitrary size and shape and serving as a fixed area in the background image. It should be noted that although the partial image 314 shown in FIG. 9 is shown as a rectangular area, the shape of the area of the partial image 314 according to the present invention is not limited to a rectangle, and may be any shape. .. In this way, by setting the background image 301 in which the partial image 314 is inserted as the final image 315, a final image having a smaller file size than the final image obtained by the image generation method described in the first embodiment can be obtained. Further, as shown in FIG. 9, an image (for example, an image stored in the image database 122 or an image selected by the user) adjusted to the size of the partial image 314 is inserted in the area of the partial image 314. Good.

As described above, according to the second embodiment, an effect that the file size of the image can be suppressed can be obtained.

Next, with reference to FIG. 10, an example of the detection target image establishing process according to the third embodiment of the present invention will be described.

Depending on the selected background image, the object as the detection target image may be unclear or incomplete, and thus it may be difficult to generate a model (vector model or the like) representing the detection target image. For example, as an example, when a part of an object to be detected is blurred, cut, or a plurality of images are captured, it is difficult to create an accurate vector model, which is used for machine learning training. Image cannot be generated. Therefore, the processing for establishing the detection target image according to the third embodiment of the present invention solves the above problem by inserting or replacing a clear partial image into the background image as the detection target image.

As shown in FIG. 10, an image showing a state of a platform at a station in which a candidate object (for example, a person) 324 to be a detection target image is captured is acquired as a background image 301. However, this candidate object 324 may be difficult to generate a model that accurately represents this candidate object 324, for example, because it is unclear or is partially missing. In such a case, in the process for establishing the detection target image of the present embodiment, the partial image 325 is drawn so as to surround all or part of the candidate object 324. Then, the partial image 325 may be designated by the user via a GUI or the like, or may be automatically generated by the machine learning unit 118. Next, the image processing unit 116 of the central server 100 performs image processing on the area in which the partial image 325 is designated, thereby finishing the candidate object 324 as the detection target image 326, and the detection target image 326 is obtained. By setting the reflected background image 301 as the final image 329, a final image that can be used for machine learning can be obtained. The partial image 325 here is substantially the same as the partial image 314 described in the third embodiment.

As described above, according to the third embodiment, it is possible to generate a high-quality image even when the detection target image is unclear and incomplete, and it is possible to obtain the effect of pushing the file size of the image.

Next, with reference to FIG. 11, an example of a machine learning image creation capability improving step according to the fourth embodiment of the present invention will be described.

As described above, the aspect of the present invention relates to the use of the image generated by the image generation method described above for machine learning training. Hereinafter, an example of improving the image creation capability of machine learning will be described with respect to a hostile generation network, but the present invention is not limited thereto and may be applied to any machine learning method such as a support vector machine.

The adversarial generation network is a network composed of two networks, a generation network (generator) and an identification network (discriminator), and learns by competing two data sets. Specifically, when a pair of a basic image as a base and a target image desired to be generated in the network is input, the generation side generates and outputs a generated image as a result. The more similar this created image is to the target image, the better. The discriminator compares the created image with the target image to determine the accuracy of the created image. In this way, the generator learns to deceive the discriminator and the discriminator learns to discriminate more accurately.

In the present embodiment, first, in step 1110, the detection target that is the source of the detection target image is photographed in a laboratory environment to acquire the detection target image. For example, as shown in FIG. 11, by photographing a person having a white appearance, a person having a white appearance can be obtained as a detection target image. Next, in step 1120, a vector corresponding to the detection target image is applied by applying a generally known well-known edge/direction detection algorithm such as Openpose to the detection target image acquired in step 1110. A model is generated. For example, when the detection target image is a person, as shown in FIG. 11, a vector model representing the head, shoulders, arms, torso, legs, etc. of the person may be generated. In addition, when the detection target has an edge such as a suitcase or an automobile, the edge extraction technique may be applied. These edges may be represented by splines or the like.

Next, in step 1130, the detection target image captured in step 1110 and the vector model generated in step 1120 are stored in the image/model database 124 of the storage unit 120 as a model/image pair associated with each other. May be saved. Next, in step 1140, the background image on which this vector model is arranged is input as a basic image to the adversarial generation network (sometimes called a second neural network). Then, in step 1150, the generation network of the adversarial generation network converts the vector model into a realistic image based on the model/image pair associated in step 1130, so that the realistic network corresponding to the vector model is obtained. An image in which a different image appears in the background image is created as a created image.

Next, the adversarial generation network compares the detection target (target image) imaged in step 1110 with the created image created in step 1150. Specifically, the identification network of the adversarial generation network compares the metadata of each of the target image and the created image (information defining the position, shape, size, property, etc. of the object in the image). Good. Further, the identification network may compare the target image and the created image using a predetermined similarity measure. This similarity criterion may be, for example, a threshold value of the degree to which two or more images are similar to each other. If the target image and the generated image meet a predetermined similarity criterion (that is, if it is determined that the target image and the generated image are sufficiently similar to each other), the parameters of the adversarial generation network are adjusted. It This parameter adjustment includes, for example, setting the conditions used for creating this created image so that they are also applied to other image generation.

In this way, the basic image is input to the hostile generation network, the created image is created based on the basic image, the created image and the target image are compared, and the created image and the target image achieve the predetermined similarity criterion. In such a case, by adjusting the parameters of the adversarial generation network, the adversarial generation network capable of generating a good final image can be obtained.

Next, with reference to FIG. 12, an example of an image generation method according to the fifth embodiment of the present invention will be described.

According to the image generation method of the present invention, not only one model can be generated for one detection target, but also a plurality of detection target models can be generated for one detection target. As shown in FIG. 12A, one detection target 1203 may be imaged by a plurality of cameras 1207, 1208, 1209. In this way, the detection target images captured by the cameras 1207, 1208, and 1209 and the background image captured by the cameras 1207, 1208, and 1209 are used in the above-described image generation method to perform the same detection. A final image can be generated that shows the target 1203 from different perspectives.

When the detection target moves, the verification target model needs to be represented as an image series in order to express the motion of the detection target. For example, as shown in FIG. 12B, it is assumed that the detection target 1213 advances in the direction indicated by the arrow 1215. The movement of the detection target 1213 is captured by the camera 1217. Therefore, the detection target model (vector model or the like) can be generated for each frame of the movement of the detection target 1213 by using the image captured by the camera 1217 in the image generation method described above. Image processing by a neural network is performed on each of these detection target models to obtain an image sequence that smoothly represents the movement of the detection target 1213. Note that not only when the detection target moves, it is also possible to generate an image showing the same detection target in different illumination environments (for example, morning and night, or natural light and artificial light).

Here, an example of generating a detection target image in which a single detection target is viewed from different viewpoints has been described, but the present invention is not limited to this, and the detection target image model representing a plurality of different objects has the same background image. It is also possible to combine with. Specifically, the model creation unit 114 may generate a first detection target model corresponding to the first detection target image and a second detection target image model corresponding to the second detection target image. Next, as described above, the model creation unit 114 acquires the first background image. Finally, the model creation unit 114 may insert the first detection target image model and the second detection target image model into the first background image.

In this way, an image with a high training effect can be obtained by generating a plurality of detection target models for one detection target or generating an image series as a verification target model.

Next, with reference to FIG. 13, an example of a machine learning detection accuracy improving step according to the sixth embodiment of the present invention will be described.

As described above, the aspect of the present invention relates to the use of the image generated by the above-described image generation method for machine learning training. Hereinafter, an example of improving the object detection accuracy of machine learning will be described with respect to a neural network such as Faster-RCNN or SVM, but the present invention is not limited thereto and may be applied to any object detection algorithm or machine learning method. ..

First, in order to optimize the first neural network (also called object detection neural network), metadata associated with the image model to be detected is provided to the object detection neural network. As described above, this metadata may be information defining characteristics such as the position, shape, size, and nature of the detection target model 402 in the image 401. The image 401 may be a final image generated by any of the image generation methods described above (for example, the final image 309 in FIG. 3, the final image 315 in FIG. 9, the final image 329 in FIG. 10, etc.). Alternatively, not only the metadata of the detection target model 402 but also the entire image 401 including the detection target model may be provided to the object detection neural network.

The target image 404 is then provided to the object detection network. The target image 404 is, for example, an image including a target object 405 that is the same as or similar to the detection target model 402 shown in the image 401, and is an image that is a target of object detection. Next, the object detection network performs object detection optimization 403 on the target image 404, and tries to identify the target object 405 from the target image 404 based on the metadata of the detection target model 402. Specifically, the object detection network compares the metadata of the detection target model 402 with the object shown in the target image 404, and identifies the object having the highest match with this metadata. As shown in FIG. 10, the object detection network may be represented by a rectangular area 406 or the like surrounding the specified target object 405.

Next, the identification accuracy of the object detection network is calculated based on the result of the object detection. This identification accuracy evaluates factors such as how well the object identified by the object detection network matches the detection target model 402, whether all target objects have been identified, and whether objects other than the target object have been incorrectly identified. And the result is expressed in a quantitative form. This specificity may be expressed as a percentage, such as 75% or 91%. As an example, when 9 out of 10 target objects are correctly identified, the calculated identification accuracy may be 90%. Next, the calculated specific accuracy may be compared with a predetermined specific accuracy criterion (predetermined accuracy threshold). If the calculated specific accuracy does not reach the predetermined specific accuracy criterion, it may be determined that the above-described object detection optimization is repeated (that is, a better specific accuracy can be obtained by repeating the object detection. Ask).

In this way, the metadata associated with the detection target image model is provided to the object detection network, and the detection target image is specified from the target images to the object detection network based on the metadata, and the specific result is determined. Thus, the effect of improving the detection accuracy of the object detection network can be obtained by calculating the identification accuracy and performing the object detection optimization.

Next, an example of the system architecture according to the embodiment of the present invention will be described with reference to FIG.

As explained above, the present invention may be configured as a client/server architecture. Specifically, as shown in FIG. 14, the user 1401 may specify a desired background image and a desired detection target via a terminal 1402 such as a computer, a tablet PC, or a smartphone. Next, the server on the cloud 1403 may generate the final image using the detection target 1409 designated by the user 1401 and the background image 1408 and/or the data stored in the storage unit 1404.

As another system architecture, a configuration not including the terminal 1402 is also possible. In this case, the camera 1405 may be directly connected to the cloud 1403, and the image or video captured by the camera 1405 may be transmitted to the image generation service provider without the user's terminal. In this case, the user may contact the desired detection target using another means such as an electronic mail, a telephone, or a smartphone.

100 central server 110 processing unit 112 image selection unit 114 model creation unit 116 image processing unit 118 machine learning unit 120 storage unit 122 image database 124 image/model database 130 client terminal 140 client terminal

Claims (15)

An image generation method,
A background image acquisition step of acquiring a background image by the image selection unit,
By the image selection unit, a detection target image specifying step of specifying a detection target image including metadata from a source image,
A model generation step of generating a detection target image model corresponding to the detection target image by the model generation unit,
And a detection target image establishing step of establishing a final image by combining the background image and the detection target image model by the model creating unit. The image generation method according to claim 1, wherein the metadata includes information indicating a position of the detection target image, information indicating a shape and size of the detection target image, and information indicating a property of the detection target image. .. The image generation method according to claim 1, wherein the detection target image model is selected from a vector model, a 3D model, and a point cloud model. The model generation step,
A vector model generation step of generating the vector model corresponding to the detection target image by the model generation unit;
A conversion step of converting the vector model into a realistic image by using a hostile generation network based on an image/model database in which a model and an image are associated with each other. The described image generation method. The detection target image establishing step,
From the background image, an object identification step of identifying a reference object,
A camera parameter calculating step of calculating a camera parameter based on the dimension element of the reference object;
The image generation method according to claim 1, wherein the final image is established by a combining step of combining the detection target image model with the background image based on the calculated camera parameter. An image generation method including a machine learning detection accuracy improving step,
The machine learning detection accuracy improving step,
In order to optimize the first neural network, the metadata associated with the detection target image model is provided to the first neural network, and the detection target image is selected from among the target images based on the metadata. A detection target image specifying training step for specifying the first neural network to
Based on the result of the detection target image identification training step, a specific accuracy calculation step of calculating the specific accuracy,
By comparing the specific accuracy with a predetermined specific accuracy standard, if the specific accuracy does not reach the predetermined specific accuracy standard, a specific accuracy determination step of deciding to repeat the detection target image specific training step is included. The image generation method according to claim 1. An image creating method including a machine learning image creating ability improving step,
The machine learning image creation capability improving step inputs a basic image to a second neural network, and creates a created image based on the basic image.
A comparison step of comparing the created image and the target image,
A parameter adjusting step of adjusting a parameter of the second neural network when the created image and the target image achieve a predetermined similarity criterion;
The image generation method according to claim 1, further comprising: The image generation method according to claim 7, wherein the second neural network is an adversarial generation network. The detection target image establishing step,
A first target model generation step of generating a first detection target image model corresponding to the first detection target image by the model creation unit;
A second target model generation step of generating a second detection target image model corresponding to the second detection target image by the model creation unit;
A first background image acquisition step of acquiring a first background image by the model creation unit;
Inserting the first detection target image and the second detection target image into the first background image;
The image generation method according to claim 1, further comprising: The image generation method according to claim 1, wherein the detection target image model includes an image series corresponding to the detection target image. The detection target image establishing step,
By the model creation unit,
If there is a blurry part in the part of the source image,
The image generation method according to claim 1, wherein the final image is generated by replacing or inserting the portion with another clear image. An image generation device,
An image selection unit that acquires a background image and specifies a detection target image including metadata from a source image,
A model creation unit that creates a detection target image model corresponding to the detection target image, and combines the background image and the detection target image model to establish a final image,
An image generation device having a. The image generation apparatus according to claim 12, wherein the detection target image model is selected from a vector model, a 3D model, and a point cloud model. The model creation unit,
Generating the vector model corresponding to the detection target image,
The image generating device,
The image generation device according to claim 13, further comprising a machine learning unit that performs image processing on the vector model to generate the detection target image model. An image generation system in which a central server and client terminals are connected via a network,
The client terminal has an image selection unit,
The central server has a model creation unit,
The image selection unit,
The background image is acquired by user input,
Identify the image to be detected with metadata from the source image,
Sending the background image and the detection target image to a central server,
The central server is
Receiving the background image and the detection target image from the client terminal,
The model creation unit,
Model generation for generating a detection target image model corresponding to the detection target image,
Establishing a final image by combining the background image and the detection target image model,
An image generation system characterized by the above.

Images