JP2020086869A - Image generation device, image generation method, and computer program - Google Patents

Abstract

To improve the recognition accuracy by generating a new image for a technique that uses an image to recognize a living thing or a robot.SOLUTION: An image generation device includes an image generation unit configured to newly generate an image of a specific subject based on parameters of an image generator obtained by performing learning processing using, as teacher data, a combination of each image of a sub-learning image group, which is a group of a plurality of images in which the specific subject that is a common subject satisfying a specific criterion is captured, and posture information of the specific subject in each image of the sub-learning image group. The image generation unit generates an image in which the specific subject has the posture indicated by the posture information by using the given posture information.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for generating a new image similar to an existing image group.

In order to accurately realize person recognition using images, a large amount of person images are generally required as teacher data. If it is attempted to realize person recognition for a specific person, a large amount of teacher data is required for the specific person. As described above, in order to realize person recognition with high accuracy, enormous effort was required to acquire a large amount of teacher data.
In response to such a problem, an image data augmentation technique has recently been proposed (see, for example, Non-Patent Document 1 and Non-Patent Document 2). In data augmentation of images, a new image associated with the teacher data is generated based on the teacher data that has already been acquired. By generating a new image in this way, it is possible to increase the teacher data.

A. G Howard, Some Improvements on Deep Convolutional Neural Network Based Image Classification, in arXiv preprint, 2013. L. Ma et al., Disentangled Person Image Generation, in Proc. CVPR, 2018.

However, simply increasing the amount of teacher data has limited the improvement in the accuracy of person recognition. Such a problem is not limited to the person recognition based on an image, but is a problem common to all techniques for recognizing a living thing or a robot using an image.
In view of the above circumstances, an object of the present invention is to provide a technique capable of improving the recognition accuracy by generating a new image, as opposed to the technique of recognizing a living thing or a robot using an image.

One aspect of the present invention is that each image of a sub-learning image group, which is a group of a plurality of images of a specific subject that is a common subject satisfying a specific criterion, and An image generation unit that newly generates an image of the specific subject based on parameters of an image generator obtained by performing a learning process using a combination of posture information of the specific subject as teacher data. The image generation unit is an image generation device that uses the given posture information to generate an image in which the specific subject has the posture indicated by the posture information.

One embodiment of the present invention is the above-described image generation device, wherein a storage unit that stores a plurality of pieces of posture information that are candidates for selection, the posture information that is read from the storage unit, and a candidate for the posture information that has been read out The image generation unit may further include a posture information selection unit that selects posture information indicating a posture that is dissimilar to the posture of the specific subject in the sub-learning image group from a predetermined reference. The image is generated by using the posture information selected by the information selection unit as the given posture information.

One aspect of the present invention is the image generation device described above, further comprising a posture information acquisition unit that acquires, for each image, posture information that is information indicating the posture of the specific subject in the sub-learning image group, The posture information selection unit uses the posture information acquired by the posture information acquisition unit as a selection candidate.

One aspect of the present invention is that each image of a sub-learning image group, which is a group of a plurality of images of a specific subject that is a common subject satisfying a specific criterion, and Based on the parameters of the image generator obtained by performing a learning process using a combination of posture information of the specific subject as teacher data, an image generating step of newly generating the image of the specific subject, In the image generating step, an image generating method is used, in which the given subject information is used to generate an image in which the specific subject has the posture indicated by the posture information.

One aspect of the present invention is a computer program for causing a computer to function as the above-described image generation device.

According to the present invention, it is possible to improve the recognition accuracy by generating a new image as compared with the technique of recognizing a living thing or a robot using an image.

It is a schematic block diagram which shows the structural example of the image generation apparatus of this invention. It is a figure which shows the specific example of posture information. It is a figure which shows the specific example of a posture image. It is a flow chart which shows a concrete example of a flow of processing at the time of image generator parameter generation of an image generation device. It is a flow chart which shows a concrete example of a flow of processing at the time of generating a posture information generator parameter of an image generating device. 9 is a flowchart showing a specific example of the flow of processing when an image is generated by the image generating apparatus.

Hereinafter, a specific configuration example of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic block diagram showing a configuration example of an image generating apparatus 10 of the present invention. The image generation device 10 is configured using an information processing device such as a personal computer, a server or a workstation. The image generation device 10 includes an image input unit 11, an image output unit 12, an instruction input unit 13, a posture information storage unit 14, a posture information generator storage unit 15, an image generator storage unit 16, and a control unit 17. The image generation device 10 will be described below.

The image input unit 11 receives the data of the learning image group input to the image generation device 10. The learning image group is a group of a plurality of already acquired images. The learning image group may include one or more sub-learning image groups. The sub-learning image group is a group of a plurality of images in which a common subject (hereinafter, referred to as “specific subject”) satisfying a specific criterion is captured. The specific criterion may be, for example, a specific person, a player of a specific event, or a person of a specific attribute (sex, age, race, etc.), It may be a specific type of organism or a specific type of robot. In addition, the specific criterion may be a specific person and wearing specific clothes. In addition, the specific reference may be that a specific person is also performing a specific action (for example, exercise of a specific type, behavior of a specific type). For example, a group of a plurality of images showing a figure of a certain player playing basketball as a subject may be formed as a sub-learning image group.

The image input unit 11 may receive the data of the learning image group from another device by performing data communication via wired communication or wireless communication. In this case, the image input unit 11 may be configured using a communication interface. The image input unit 11 may read the data of the learning image group recorded in a recording medium such as a CD-ROM or a USB memory (Universal Serial Bus Memory) from the recording medium. In this case, the image input unit 11 may be configured using a device such as a CD-ROM drive or a USB interface. The image input unit 11 may receive a learning image group captured by a still camera or a video camera from the camera. In this case, the image input unit 11 may be configured using an interface of a communication protocol capable of data communication with the camera. When the image generating apparatus 10 is built in a still camera, a video camera, or an information processing apparatus (smartphone or the like) including a camera, the image input unit 11 receives a captured image or an image before capturing from the bus. You may. The image input unit 11 may be configured in any manner as long as it can receive the data of the learning image group. Further, it is not necessary that each image has already formed the learning image group at the time of being input to the image input unit 11, and the learning image group is input to the image generating device 10 as a result of inputting a plurality of images. May be done.

The image output unit 12 outputs the image data generated by the control unit 17. The image output unit 12 may transmit an image to another device (for example, another information processing device or another storage device) by performing data communication via wired communication or wireless communication. In this case, the image output unit 12 may be configured using a communication interface. The image output unit 12 may record an image on a recording medium such as a DVD-ROM or a USB memory. In this case, the image output unit 12 may be configured using a device such as a DVD-R drive or a USB interface. The image output unit 12 may record the image in a storage device included in the image generation device 10. The image output unit 12 may be configured in any manner as long as it can output image data.

The instruction input unit 13 may be configured using an existing input device such as a keyboard, a pointing device (mouse, tablet, etc.), a button, a touch panel, or the like. In this case, the instruction input unit 13 is operated by the user when the user's instruction is input to the image generating apparatus 10. The input device described above may be configured using a microphone and a voice recognition device for receiving voice input. The instruction input unit 13 may be an interface for connecting an input device to the image generation device 10. In this case, the instruction input unit 13 inputs the input signal generated by the input device according to the user's input to the image generation device 10. The instruction input unit 13 may receive a user instruction from another device by performing data communication via wired communication or wireless communication. In this case, the instruction input unit 13 may be configured using a communication interface.

The attitude information storage unit 14 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The posture information storage unit 14 stores the posture information obtained from the image data included in the learning image group in association with the image identification information indicating the image from which the posture information is obtained. The posture information storage unit 14 may store the posture information in association with the sub-learning image group identification information indicating the sub-learning image group including the image for which the posture information is obtained.

The posture information is information indicating the posture of the subject of each image included in the learning image group. The posture information may be, for example, information indicating the positions of a plurality of predetermined characteristic parts with respect to the subject. Such a characteristic part may be a predetermined part of the human body. As specific examples of such characteristic parts, a head, a body, a right shoulder, a left shoulder, a right foot, and a left foot may be defined. Other specific examples of such characteristic parts include right eye, left eye, nose, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, right hand, left hand, neck, waist, right knee, left knee. , Right ankle, left ankle, right ankle, left ankle may be defined.

The posture information may be defined as information having coordinates (for example, x coordinate and y coordinate) indicating a position on the image for each of these characteristic parts. FIG. 2 is a diagram showing a specific example of the posture information. In the example of FIG. 2, the image coordinates of each characteristic part such as the head, body, right shoulder, left shoulder, and left foot are defined as a combination of x and y values.

The posture information may be defined as an image showing the position of each characteristic part (hereinafter referred to as “posture image”). For example, the posture image may be defined as an image using a node indicating each characteristic part and a link connecting the nodes. FIG. 3 is a diagram showing a specific example of the posture image. In FIG. 3, a posture image indicating the posture information of a person is formed by combining the image of the node 21 representing each characteristic part and the image of the link 22 connecting the nodes 21. In the posture image, each link may be represented by a different color, or each node may be represented by a different color.

Returning to the explanation of FIG. The attitude information generator storage unit 15 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The posture information generator storage unit 15 stores information necessary for the posture information generator that generates posture information to operate (hereinafter referred to as “posture information generator parameter”). Such attitude information generator parameters are obtained by the processing of the control unit 17.

The image generator storage unit 16 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The image generator storage unit 16 stores information necessary for the operation of the image generator that generates an image (hereinafter referred to as “image generator parameter”). Such image generator parameters are obtained by the processing of the control unit 17.

The control unit 17 includes a processor such as a CPU (Central Processing Unit) connected by a bus and a memory. When the control unit 17 executes the image generation program, the control unit 17 causes the posture information acquisition unit 171, the image generator learning unit 172, the posture information generator learning unit 173, the posture information generation unit 174, the posture information selection unit 175, and It operates as the image generation unit 176. Note that all or part of each configuration may be realized using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). Further, all or part of each configuration may be realized by a dedicated processor such as a GPU (Graphics Processing Unit) executing a program. The image generation program may be recorded in a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a semiconductor memory device (for example, SSD: Solid State Drive), a hard disk or a semiconductor memory built in a computer system. A storage device such as a device. The image generation program may be transmitted via a telecommunication line.

The posture information acquisition unit 171 estimates the posture of the subject of each image included in the learning image group input by the image input unit 11. The posture information acquisition unit 171 generates posture information as the posture estimation result. The technique described in the following references may be applied to the posture information acquisition unit 171, for example. The posture information acquisition unit 171 records the generated posture information in the posture information storage unit 14.

Reference 1: Z. Cao et al., Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, in Proc. CVPR, 2017.
Reference 2: S. -E. Wei et al., Convolutional Pose Machines, in Proc. CVPR, 2016.

The image generator learning unit 172 acquires an image generator parameter for each sub-learning image group. The image generator learning unit 172 performs machine learning using, for example, a combination of a plurality of images included in the sub-learning image group to be processed and the posture information estimated in each image as a teacher image. , Image generator parameters may be obtained. The image generator represented by the image generator parameter receives the posture image as an input and generates an image of a living thing or a robot having the posture indicated by the posture image. What kind of image of a living thing or a robot is generated depends on the sub-learning image group. That is, an image is generated in which the subject that satisfies the specific criterion defined in the sub-learning image group that is the processing target has the posture indicated by the input posture image. The image generator learning unit 172 may be configured to acquire the image generator parameters by adversarial learning such as GAN. The techniques described in the following references may be applied to the image generator learning unit 172, for example. The image generator learning unit 172 records the generated image generator parameters in the image generator storage unit 16.

Reference 3: X. Mao et al., Least Squares Generative Adversarial Networks, in Proc. ICCV, 2017.
Reference 4: I. Gulrajani, Improved Training of Wasserstein GANs, in Proc. ICLR, 2018.

The posture information generator learning unit 173 acquires the posture information generator parameters. The posture information generator learning unit 173 performs the machine learning using the posture information obtained by the posture information acquisition unit 171 from a plurality of images included in the learning image group to be processed as the teacher data, for example. Information generator parameters may be obtained. All the posture information obtained from the images included in the learning image group may be used as the teacher data used for learning the posture information generator parameters, or may be obtained from the images of the specific sub-learning image group. Only the posture information may be used.

The posture information generator represented by the posture information generator parameter is configured to generate a posture image of a predetermined size and a predetermined number of channels, for example, by inputting a random numerical value sequence of a predetermined dimension number. For example, a posture image having 192 vertical pixels, 192 horizontal pixels, and 3 channels may be generated. The posture information generator learning unit 173 acquires the posture information generator parameters by using, for example, a variational auto encoder (VAE: see reference 5 below) or a hostile generation network (GAN: see reference 6 below). It may be configured as follows. The technique described in the following references may be applied to the posture information generator learning unit 173, for example. The posture information generator learning unit 173 records the generated posture information generator parameters in the posture information generator storage unit 15.

Reference 5: DP Kingma et al., Auto-Encoding Variational Bayes, in Proc. ICLR, 2014.
Reference 6: I. Goodfellow et al., Generative Adversarial Networks, in NIPS, 2014.

The posture information generator 174 operates as a posture information generator based on the posture information generator parameters stored in the posture information generator storage unit 15. The posture information generation unit 174 generates posture information by operating as a posture information generator. The posture information generation unit 174 is provided with predetermined input parameters that were assumed to be provided to the posture information generator in the posture information generator learning unit 173. The predetermined input parameter may be, for example, a numerical sequence having a predetermined number of dimensions. Any method of generating the predetermined input parameter may be realized. For example, when the number of dimensions is set to “10”, a random numerical value sequence may be generated by acquiring the value of each dimension based on the normal distribution. The posture information generation unit 174 outputs the generated posture information to the posture information selection unit 175.

The posture information selection unit 175 selects the posture information to be processed from the posture information stored in the posture information storage unit 14 and the posture information generated by the posture information generation unit 174. In the following description, the posture information stored in the posture information storage unit 14 and the posture information generated by the posture information generation unit 174 are collectively referred to as “candidate posture information”.

The posture information selection unit 175 sets, in the candidate posture information, posture information that has not yet been obtained as the posture information (hereinafter referred to as “specific posture information”) of the specific subject that is the target of image generation in the image generation unit 176. Select from. More specifically, the posture information selection unit 175 calculates the degree of similarity for each piece of specific posture information that has already been obtained for the posture information that is the determination target that is selected from among the candidate posture information, and calculates all the calculated postures. When the similarity indicates that the similarity is not equal to or more than a predetermined reference, the posture information to be determined is selected. The posture information selection unit 175 may select one piece of posture information, may select a predetermined number of pieces of posture information, and may be indicated by an instruction input via the instruction input unit 13. You may select several posture information. The posture information selecting unit 175 determines the degree of similarity when selecting a predetermined number of posture information or when selecting the number of posture information indicated by the instruction input via the instruction input unit 13. The posture information may be selected in order from a value showing that the values are not the most similar. The posture information selection unit 175 outputs the selected posture information to the image generation unit 176.

The image generation unit 176 operates as an image generator by using the image generator parameter of the specific subject to be processed among the image generator parameters stored in the image generator storage unit 16. The image generation unit 176 uses the posture information selected by the posture information selection unit 175 when generating the image. The image generation unit 176 operates as an image generator using the posture information to generate an image in which the specific subject has the posture indicated by the selected posture information. The image generation unit 176 outputs the generated image data to the image output unit 12.

FIG. 4 is a flowchart showing a specific example of the flow of processing when the image generator parameters are generated by the image generating apparatus 10. First, the image input unit 11 inputs image data of a sub-learning image group to be processed (step S101). The posture information acquisition unit 171 estimates the posture information of the subject in each image data of the sub-learning image group to be processed (step S102). The image generator learning unit 172 acquires an image generator parameter by performing a learning process using a plurality of combinations of image data of a sub-learning image group to be processed and posture information as teacher data (step S103). .. The image generator learning unit 172 acquires the image generator parameter for each sub-learning image group by repeatedly executing the processing of steps S101 to S103 for each sub-learning image group. The image generator learning unit 172 records the acquired image generator parameters in the image generator storage unit 16 in association with the sub-learning image group.

FIG. 5 is a flowchart showing a specific example of the flow of processing when the orientation information generator parameters of the image generating apparatus 10 are generated. First, the image input unit 11 inputs image data of a learning image group to be processed (step S201). The posture information acquisition unit 171 estimates the posture information of the subject in each image data of the learning image group to be processed (step S202). The posture information generator learning unit 173 acquires a posture information generator parameter by performing a learning process using a plurality of posture information obtained from the learning image group of the processing target as teacher data (step S203). The posture information generator learning unit 173 records the acquired posture information generator parameters in the posture information generator storage unit 15 in association with the learning image group.

FIG. 6 is a flowchart showing a specific example of the flow of processing when the image generating apparatus 10 generates an image. First, the posture information selection unit 175 acquires a plurality of posture information (step S301). For example, the posture information selection unit 175 may acquire the posture information stored in the posture information storage unit 14 and the posture information generated by the posture information generation unit 174. The posture information selection unit 175 selects posture information from the acquired plurality of posture information (candidate posture information) (step S302). The image generation unit 176 generates an image based on the posture information selected by the posture information selection unit 175 and the image generator parameter corresponding to the sub-learning image group to be processed (step S303).

According to the image generating apparatus 10 configured as described above, it is possible to improve the recognition accuracy by generating a new image as compared with the technique of recognizing a living thing or a robot using an image. Details are as follows.

A technique for recognizing a living thing or a robot by using an image requires a huge amount of teacher data, but even if a large number of similar images are included in the teacher data, improvement in recognition accuracy is limited. In response to such a problem, the image generating apparatus 10 described above generates an image of a new posture that has not existed as an image of the subject until then. By using the image of such a new posture as the teacher data, the recognition accuracy can be improved.

Further, in the image generation device 10, the posture of each specific subject included in the learning image group is estimated only by inputting the learning image group, and the posture information indicating a new posture that does not exist in each specific subject is selected, An image of the specific subject having the posture indicated by the selected posture information is generated. Therefore, the user does not need to bother to determine the posture and input the posture, and the labor of the user can be reduced.

Further, in the image generating apparatus 10, the image generator parameter is generated for each specific subject satisfying the specific standard common to the sub-learning image group. Therefore, it is possible to generate an image having more distinctive features of each specific subject, as compared with the case where an image generator parameter corresponding to a general creature or race is generated. By performing the learning process using the image generated in this way, the recognition accuracy of the specific subject can be improved.

(Modification)
The posture acquisition unit 171 may be configured to acquire posture information previously determined by another device or a person from the outside, instead of estimating the posture information based on the image. In this case, the posture information acquisition unit 171 externally acquires the posture information determined in advance for each image, and records the posture information in the posture information storage unit 14.

The image generation device 10 may be configured not to include the image generator learning unit 172. In this case, the image generation unit 176 executes the process by recording in the image generator storage unit 16 the image generator parameters obtained by the image generator learning unit 172 mounted in another device in advance. Is possible.

The image generation device 10 may be configured not to include the posture information generator learning unit 173. In this case, by storing the posture information generator parameters obtained by the posture information generator learning unit 173 installed in another device in the posture information generator storage unit 15 in advance, the posture information generator 174 performs processing. Can be executed.

The image generation device 10 may be configured without the posture information generation unit 174. In this case, the posture information selection unit 175 selects posture information from the posture information stored in the posture information storage unit 14, that is, the posture information obtained from the images of the learning image group. Since the posture information also includes the posture information of other specific subjects, the posture information selection unit 175 can select the posture information.

Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes a design and the like within a range not departing from the gist of the present invention.

10... Image generating device, 11... Image input unit, 12... Image output unit, 13... Instruction input unit, 14... Attitude information storage unit, 15... Attitude information generator storage unit, 16... Image generator storage unit, 17... Control unit, 171... Attitude information acquisition unit, 172... Image generator learning unit, 173... Attitude information generator learning unit, 174... Attitude information generation unit, 175... Attitude information selection unit, 176... Image generation unit

Claims (5)

Each image of a sub-learning image group, which is a group of a plurality of images in which a specific subject that is a common subject satisfying a specific criterion is captured, and posture information of the specific subject in each image of the sub-learning image group, An image generation unit that newly generates an image of the specific subject based on the parameters of the image generator obtained by performing the learning process using the combination of
The image generation apparatus, wherein the image generation unit generates an image in which the specific subject has the posture indicated by the posture information by using the given posture information. A storage unit that stores a plurality of posture information items that are candidates for selection,
The posture information is read from the storage unit, and posture information indicating a posture that is dissimilar to the posture of the specific subject in the sub-learning image group based on a predetermined reference from the read posture information candidates. Further comprising a posture information selection unit for selecting
The image generation device according to claim 1, wherein the image generation unit generates the image by using the posture information selected by the posture information selection unit as given posture information. The sub-learning image group further includes a posture information acquisition unit that acquires posture information, which is information indicating the posture of the specific subject, for each image,
The image generation apparatus according to claim 2, wherein the posture information selection unit uses the posture information acquired by the posture information acquisition unit as a selection candidate. Each image of a sub-learning image group, which is a group of a plurality of images in which a specific subject that is a common subject satisfying a specific criterion is captured, and posture information of the specific subject in each image of the sub-learning image group, Based on the parameters of the image generator obtained by performing the learning process using the combination of as a teacher data, an image generating step of newly generating the image of the specific subject,
An image generating method, wherein in the image generating step, the given subject information is used to generate an image in which the specific subject has the posture indicated by the posture information. A computer program for causing a computer to function as the image generation device according to claim 1.

Images