JP2019045984A - Data composing apparatus and method - Google Patents

Abstract

To provide a technique which enables verbal data composition.SOLUTION: A data composing apparatus includes: a database in which element objects being compositable data relating to objects describable in a natural language are preliminarily stored; an extraction unit which extracts at least one of a concept or a context from a sentence described in the natural language; a conversion unit which converts the concept or the context to a feature vector represented as a vector in a prescribed feature space; and a composing unit which preliminarily holds a neural network model for composing element objects in accordance with the input feature vector, selects an element object from the database on the basis of the neural network model and the feature vector, and generates composite data by using the element object.SELECTED DRAWING: Figure 2

Description

  The present invention relates to techniques that allow verbal data synthesis.

  By associating images with natural language words, various applications of images become possible. For example, associating an image with a word makes it possible to search an image by words. As a technique for associating an image with a word, for example, there is an image annotation technique. The image annotation technology extracts a feature amount from an image area of a target image, selects an image having a feature amount closest to the target image from images in which features have been learned in advance and metadata have been added, and It is a technology that gives metadata to a target image. Further, in Patent Document 1, a plurality of feature quantities are extracted from a learning image, a plurality of feature quantities are classified using a binary discriminator, and a learning model for correlating identification information with the feature quantity is identified as identification information and Discriminant information by creating a formula for each type of feature and approximating the equation for finding the conditional probability of identification information with a sigmoid function and optimizing the parameters of the sigmoid function so that the conditional probability of identification information is maximized. A technique for optimizing a learning model is disclosed for each. Thereby, reliable identification information can be given to the image.

  There is also a technology for combining images by arranging words such as "mountain" and "sea" in a square canvas as an image combining technology.

JP 2012-038244 A

In recent years, new utilization of images is required. For example, it is also required to create a desired image expressed in words. However, the conventional image annotation technology described above selects a desired image from a database and can not create a new image. Although the technique of arranging words and combining images can represent simple positional relationships, it can not represent temporal and spatial relationships such as "run and get on a train" and operating conditions. Can not be synthesized. Also, in the conventional image synthesis technology, a model for generating an image from text is constructed by learning a large number of images and their descriptive sentences, but it is suitable for unknown events not included in learning data. It is difficult to generate an image.
In addition, if various other data such as voice and sensor data can be associated with the language of the natural language as in the case of the image described above, it is conceivable that the utilization of the data will be greatly expanded.

  An object of the present invention is to provide a technology that enables verbal data synthesis.

  A data synthesizing apparatus according to one aspect of the present invention is a database in which element objects which are synthesizable data relating to objects that can be described in natural language are stored in advance, and an extraction that extracts at least one of concept or context from natural language sentences. A neural network model for combining element objects according to an input feature vector, and a conversion unit for converting the concept or the context into a feature vector represented by a vector in a predetermined feature space; And a composition unit that selects an element object from the database and generates composition data using the element object based on the network model and the feature vector.

  According to the present invention, a neural network model that combines element objects in accordance with feature vectors is held in advance, at least one of concept and context is extracted from natural language sentences, and converted into feature vectors. Since element objects are synthesized to generate synthesized data, user-desired synthesized data expressed in natural language sentences can be generated from the element objects stored in the database. Also, an image of an unknown scene not used for learning can be generated.

FIG. 1 is a block diagram showing a physical configuration of an image combining system according to a first embodiment. FIG. 1 is a block diagram of an image combining device according to a first embodiment. 5 is a flowchart of an image combining process according to the first embodiment. FIG. 6 is a sequence diagram of an image combining process according to the first embodiment. FIG. 7 is a sequence diagram showing an operation example of the image combining device according to the first embodiment. FIG. 7 is a block diagram of an image combining device according to a second embodiment. 7 is a flowchart of a machine learning process according to a second embodiment. FIG. 13 is a block diagram showing a physical configuration of an image combining system according to a third embodiment. FIG. 10 is a block diagram of a terminal according to a third embodiment. FIG. 18 is a diagram showing an example of data accumulated by the database according to the fourth embodiment. FIG. 16 is a block diagram of an image combining device according to a fifth embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the physical configuration of the image combining system according to the first embodiment. Referring to FIG. 1, the image combining system includes a terminal 100, an image combining device 200, and a database 300. The terminal 100 and the image combining device 200 are connected by a communication network 101, and the image combining device 200 is connected by a communication network 201. The communication network 101 is, for example, the Internet. The communication network 201 is, for example, a LAN (Local Area Network).

  The terminal 100 is an information terminal that the user 10 directly uses, and is, for example, a personal computer, a tablet terminal, a smartphone or the like. In accordance with the instruction of the user 10, information of a sentence in which a synthesized image desired by the user is expressed in natural language is sent to the image synthesizing device 200, and the synthesis of the image matching the sentence is requested. The terminal 100 also receives, from the image combining device 200, the data of the combined image created by the image combining device 200, records the data in an internal storage device (not shown), and displays the combined image on the screen.

  The database 300 is a database that accumulates various image data used for image composition so as to be obtainable from the image composition device 200. The database 300 stores data of images (object images) for displaying various objects. Information (labels) indicating objects displayed in the image is added to the object image. For example, in the object image, information of the name of the object displayed in the image is added as a label.

  The image synthesizing device 200 is a computer that synthesizes an image (synthesized image) desired by the user 10 using data stored in the database 300 according to a request from the terminal 100 operated by the user 10.

  FIG. 2 is a block diagram of the image combining device according to the first embodiment. Referring to FIG. 2, the image combining device 200 includes a sentence processing unit 210, a concept extraction unit 220, a context extraction unit 230, an embedding conversion unit 240, an image combining unit 250, and an image output unit 260.

  The sentence processing unit 210 analyzes the sentence received from the terminal 100, converts the smallest unit entity (hereinafter referred to as "minimum entity") capable of interpreting the meaning, and determines the part of speech of each entity. Thus, the sentence is in a form that can be processed by the concept extraction unit 220 and the context extraction unit 230.

  The concept extraction unit 220 extracts a concept from the sentence analyzed by the sentence processing unit 210. Specifically, for example, among the entities obtained by the analysis by the sentence processing unit 210, the concept extraction unit 220 may extract an entity whose part of speech is a noun as an input.

  The concept describes, for example, the relationship between attributes and features to nouns. For example, related words such as "animal" and "hair" are described for "dog".

  The context extraction unit 230 extracts a context from the sentence analyzed by the sentence processing unit 210. Specifically, for example, the context extraction unit 230 uses the concept extracted by the concept extraction unit 220 as it is as a context, and the smallest entity of a part of speech other than a noun such as a verb among the entities obtained by analysis by the sentence processing unit 210 What connected the concept and the concept, and what connected the concept and the concept with the verb may be extracted as the context.

  The context describes the relationship of the sentence expression of the phrase. For example, the relationship between "dog" and "run" is described.

  The embedding transform unit 240 transforms the concept and the context into a feature vector which is a vector in feature space. Feature vectors represent the meaning of a concept or context. This makes it possible to process the meaning of concepts and contexts with feature vectors. In this embodiment, feature vectors of concept and context become inputs of the neural network model.

  The image synthesis unit 250 holds in advance a neural network (NN) model obtained by machine learning, acquires an object image according to the feature vector of the concept generated by the embedding conversion unit 240 from the database 300, and A composite image is generated by arranging an object image based on the feature vector of the context and the NN model. For example, the NN model takes as input a feature vector of context including a concept, and outputs a composite image or a method of generating the composite image. In the NN model, a high dimensional plane is generated for each layer from the input feature vector, and a final image is output as a composite image in which the concept in the context is interpreted. The synthetic image generation method includes, for example, which object image is to be used, and the positional relationship between the object images and the like. For example, the image combining unit 250 acquires an object image corresponding to the concept extracted by the concept extraction unit 220 from the database 300 according to the output of the NN model, and sets the object image to the context extracted by the context extraction unit 230. The composite image may be generated by arranging based on this.

  The present method is characterized in that the NN model is configured such that features of both concept and context can be linked to an output image. As a result, images can be synthesized even for objects for which relationships have not been learned in advance, which can not be synthesized by the NN model learned only by the concept. For example, it is possible to synthesize an example-free image such as “person swimming in the cloud”.

  The image output unit 260 transmits the data of the combined image generated by the image combining unit 250 to the terminal 100.

  FIG. 3 is a flowchart of the image combining process according to the first embodiment. Referring to FIG. 3, first, in step S <b> 210, the sentence processing unit 210 analyzes a sentence received from the terminal 100. Next, in step S220, the concept extraction unit 220 extracts a concept from the sentence analyzed by the sentence processing unit 210. Subsequently, in step S230, the context extraction unit 230 extracts a context from the sentence analyzed by the sentence processing unit 210.

  Depending on the complexity of the sentence, it is possible to combine images by extracting only the concept. Moreover, although the example which extracts context after extraction of a concept is shown here, it is not limited to this order. As another example, concepts and contexts may be extracted simultaneously.

  Next, in step S240, the embedding transform unit 240 transforms the concept and the context into a feature vector which is a vector in the feature space. For example, Word2Vec can map individual concepts and contexts to high dimensional vector represented features. The meaning of a concept or context is represented by a feature vector. This makes it possible to process the meaning of the concept and context.

  Furthermore, in step S250, the image combining unit 250 acquires an object image corresponding to the feature vector of the concept from the database 300, and arranges the object image based on the feature vector of the concept and context using the NN model. To generate a composite image. Then, in step S 260, image output unit 260 transmits the data of the composite image generated by image combining unit 250 to terminal 100.

  FIG. 4 is a sequence diagram of the image combining process according to the first embodiment. Referring to FIG. 4, in step S501, a sentence is transmitted from the terminal 100 to the image combining device 200. In step S502, the concept is extracted by the image combining device 200. In step S503, the image combining device 200 requests the database 300 to match the extracted concept. At step S504, the requested image matching the concept is returned from the database 300 to the image combining device 200. In step S505, the context is extracted by the image combining device 200. In step S506, a composite image is generated by combining the images matching the concept according to the context. In step S507, the composite image is returned from the image combining device 200 to the terminal 100.

  FIG. 5 is a sequence diagram showing an operation example of the image combining device according to the first embodiment.

  In step S610, the terminal 110 transmits the sentence “Man running to train at station.” Input by the user 10 to the image combining device 200.

The image synthesizing device 200 divides a sentence into minimum entities. Then, the image synthesizing device 200 extracts the concept from the sentence converted to the example of the minimum entity. The minimum entities are, for example, "Man: A", "Running: P", "Train: B", "at station: C" and the like. Furthermore, the image synthesizing device 200 extracts the context in which those concepts exist from the sentence. In the example of FIG. 5, the following contexts can be extracted. These concepts and contexts are input to the NN model. Here, alphabets such as A, B, C and P are identification codes given to a concept or context.
Man: A
Train: B
At Station: C
Man: A running: P
Man: A running: P Train: B
Train: B at station: C
Man: A running: P at Station: C

  The image combining device 200 acquires the concept at step S602, and requests the database 300 for an image corresponding to the concept at step S603. In this example, images of man and train are required. In step S604, an image corresponding to the concept is sent back from the database 300 to the image combining device 200. In this example, an image of a man and an image of a train stopping at a station are sent back. The image combining device 200 acquires the context in step S605, and combines the image in step S606. Here, a two-dimensional composite image is generated by superimposing and arranging the image of man and the image of train. The image combining device 200 transmits the data of the created combined image to the terminal 100 in step S 607. Image composition can be composed from concept and context, context only, or concept alone.

  In the present embodiment, an example of combining images is shown, but if various other data such as audio data and sensor data can be associated with the language of natural language as in the case of images, the utilization of the data is also significant. It is thought that it spreads. For various data, composite data can be generated using element objects that can be composited data.

  As described above, the image combining apparatus 200 according to the present embodiment includes the database 300 in which element objects (object images), which are data (image data) that can be combined with objects that can be described in natural language, are stored in advance An extraction unit (concept extraction unit 220, context extraction unit 230) for extracting at least one of a concept or context from a language sentence, and a conversion unit (embedding for converting the concept or context into a feature vector represented by a vector in a predetermined feature space A transformation unit (240) and a neural network model for synthesizing element objects according to the input feature vector are held in advance, and based on the neural network model and the feature vector, the element object is selected from the database. Synthesis unit that generates synthesized data (composite image) with the element object of perilla (image combining unit 250), a has. In this way, a neural network model that synthesizes an element object according to a feature vector is held in advance, at least one of a concept and a context is extracted from natural language sentences, and converted into a feature vector. Can be generated to generate synthetic data, so that user-desired synthetic data expressed in natural language sentences can be generated from element objects stored in the database. As for images, object images are held in the database as element objects in advance, at least one of concepts and contexts is extracted from natural language sentences, they are converted to feature vectors, and object images are synthesized based on neural network models. Since the composite image is generated, it is possible to generate a user-desired composite image represented by natural language sentences.

  Also, the extraction unit generates a context by connecting the concept with the smallest entity which is a noun among the smallest entities which are the smallest units capable of semantic interpretation of a sentence, and connecting the concept with other minimal entities. A noun is a concept, and a concept is connected to other minimal entities to generate a context, so that objects represented by sentences can be interpreted and reflected in synthetic data.

  Further, the image combining unit generates the combined image by arranging a plurality of object images so as to overlap or align. A composite image can be generated with a relatively small amount of processing.

  Further, in the present embodiment, any image combining method may be used. For example, a neural network model is a neural network model according to General Adversalial Networks which performs learning with two opposing models of a generator and a discriminator, and a generator generates a composite image. A composite image can be generated that is relatively close to what the user desires. As a simpler method, a composite image may be generated by collage in which a plurality of images are combined and arranged.

  The first embodiment shows an example in which the image synthesis system is provided in advance with a trained neural network model, and images are synthesized using the NN model. The second embodiment shows an example in which the image combining system learns an NN model and combines images using the learned NN model.

  The image synthesizing system according to the second embodiment has the same physical configuration as that of the first embodiment shown in FIG.

  FIG. 6 is a block diagram of an image combining device according to a second embodiment. The image combining apparatus according to the second embodiment is different from that of the first embodiment shown in FIG. 2 in that it has a concept selection unit 310, a context generation unit 320, an image determination unit 350, and an end condition determination unit 360. .

  The image combining process according to the second embodiment is the same process as that of the first embodiment shown in FIGS. 3 and 4.

  FIG. 7 is a flowchart of machine learning processing according to the second embodiment.

  First, in step S310, the concept selection unit 310 selects a concept from among various known concepts, for example, randomly or according to a predetermined selection method. Next, in step S320, the context generation unit 320 generates a natural context including the concept. For example, various sentences of natural language may be stored, and a sentence including the selected concept may be extracted as a context.

  Next, in step S330, the embedding transform unit 240 transforms the concept and the context into a feature vector. At this time, the embedding conversion unit 240 may convert the concept and the context into a feature vector by the same method as the image combining described in the first embodiment. The feature vector generated by the embedding conversion unit 240 is provided to the image combining unit 250 as in the image combining described in the first embodiment.

  Next, in step S340, the image combining unit 250 generates a combined image by inputting feature vectors of concept and context into the NN model. At this time, the image combining unit 250 performs image combining in the same manner as the image combining described in the first embodiment.

  Next, in step S350, the image determination unit 350 determines whether the created composite image correctly represents the concept and the context. If the composite image is not one that correctly expresses the concept and context, the process returns to step S340 to redo the image composition. If the composite image correctly represents the concept and the context, then in step S360, the termination condition determination unit 360 determines whether a predetermined termination condition is satisfied. If the end condition is satisfied, the machine learning process ends. If the termination condition is not satisfied, the process returns to step S310. The termination condition may be terminated when, for example, the machine learning process loops a predetermined number of times. Alternatively, the process may be ended when the machine learning process is performed for a predetermined time.

  As in the first embodiment, the present embodiment is also applicable to data other than image data such as audio data and sensor data.

  As described above, in the present embodiment, the image combining device 200 includes the concept selection unit 310 that selects the concept used for learning the neural network model, and the context generation unit 320 that generates the context used for learning the neural network model. And a determination unit (image determination unit 350) that evaluates a composite image generated by the neural network model. Then, the conversion unit (embedding conversion unit 240) converts the concept selected by the concept selection unit 310 and the context generated by the context generation unit 320 into a feature vector. The synthesizing unit (image synthesizing unit 250) selects an element object (object image) from the database 300 based on the neural network model and the feature vector, and generates synthesized data using the element object. The determination unit evaluates the composite data, and if the composite data can not obtain a predetermined evaluation, the processing of the combination unit and the determination unit is repeated. Since the data synthesis device can perform machine learning of data synthesis, it becomes possible to learn the neural network model by itself and use it by itself.

  The first embodiment shows an example in which the image synthesizing device 200 synthesizes an image based on a natural language document input by the user 10 from the terminal 100 and transmits the image to the terminal 100. The third embodiment shows an example in which the terminal alone performs image synthesis.

  FIG. 8 is a block diagram showing the physical configuration of the image combining system according to the third embodiment. Referring to FIG. 8, the image combining system is configured of only the terminal 110. The terminal 110 is, for example, an information device such as a personal computer, a tablet terminal, or a smartphone. FIG. 9 is a block diagram of a terminal according to the third embodiment. 9, the terminal 110 according to the third embodiment includes an input unit 410, a display unit 420, a database storage unit 430, a sentence processing unit 210, a concept extraction unit 220, a context extraction unit 230, an embedding conversion unit 240, and an image synthesis. It has a part 250.

  The sentence processing unit 210, the concept extraction unit 220, the context extraction unit 230, the embedding conversion unit 240, and the image combining unit 250 are the same as those included in the image combining apparatus 200 in the first embodiment shown in FIG.

  The input unit 410 is an input operation unit that inputs a sentence related to an image that the user 10 wants to generate.

  The display unit 420 is a display unit that displays the composite image generated by the image combining unit 250.

  The database storage unit 430 is a storage unit for accumulating data of object images corresponding to the database 300 of FIG.

  In the first embodiment, a large number of object images are accumulated in the database 300, and the image combining unit 250 acquires specific object images from the database 300, and creates a combined image by combining the object images. The object image is an image of a predetermined object such as a human, a train, or a rabbit. However, the present invention is not limited to the configuration and process of the first embodiment. What is necessary is just to produce | generate a synthetic image by combining a visual expression object. Visual representation objects include not only images of objects, but also patches that indicate the appearance features of the objects. The patch makes it possible to correct the appearance features of the object on the image. The characteristic is, in the rabbit example, "white fur", "short leg", "long ear", etc. and combining it with the white fur patch to the rabbit object image to generate the "white rabbit" image can do.

  In the fourth embodiment, as a visual representation object, in addition to the object image, a patch indicating a detailed feature of the appearance of the object is accumulated in the database 300, and a desired object image is stored using the object image and the patch. An example of creating is shown. The basic configuration of the image combining device 200 of the fourth embodiment is the same as that of the first embodiment shown in FIG. In the fourth embodiment, the image combining unit 250 creates a desired object image using an object image and a patch, and creates a combined image using the created object image.

  FIG. 10 is a diagram of an example of data accumulated by the database according to the fourth embodiment. Referring to FIG. 10, in the database 300, an object image 510 and a patch 520 associated therewith are accumulated. The image combining unit 250 corrects the object image with a patch using the NN model based on the concept and context extracted from the sentence input by the user 10, and combines the corrected images to generate a combined image.

  The present embodiment is also applicable to not only image data but also other data such as audio data and sensor data.

  As described above, according to the present embodiment, the element object (object image) is a visual expression object indicating a visual expression that is an element of the image, and the combining unit (image combining unit 250) To generate a composite image by combining the visual representation objects on the basis of. A user-desired image represented by natural language sentences can be synthesized by combining visual presentation objects.

  In addition, the visual representation object includes a patch representing the feature of the object to be displayed in the composite image, and the combining unit corrects the object image with the patch, and creates a composite image from the corrected object image. The patch corrects the object image so as to be suitable for the composite image desired by the user, and thus enables generation of a composite image closer to the user's desired.

  The example shown in the first embodiment is an example in which image synthesis is performed using an NN model acquired by learning in advance. However, the present invention is not limited to this. As another example, Example 5 shows an example of performing zero-shot learning using information obtained by image combination using an NN model, and updating the NN model. In the process of generating the composite image, information (hereinafter referred to as “object image information”) is obtained regarding what the object image used to generate the composite image indicates. In this embodiment, zero-shot learning is performed using object image information obtained in the process of generating composite data, and the NN model is updated. Since the NN model is updated by zero-shot learning using information obtained in the generation of synthetic data, the performance of verbal data synthesis can be continuously improved.

  The physical configuration of the image combining system according to the fifth embodiment is the same as that of the first embodiment shown in FIG. The image combining device 200 of the fifth embodiment is partially different from that of the first embodiment. FIG. 11 is a block diagram of an image combining device according to a fifth embodiment.

  Referring to FIG. 11, the image combining device 200 includes a sentence processing unit 210, a concept extraction unit 220, a context extraction unit 230, an embedding conversion unit 240, an image combining unit 250, an image output unit 260, and a model updating unit 610. ing. The image combining apparatus 200 of the fifth embodiment can perform image combining in the same manner as that of the first embodiment, and the sentence processing unit 210, the concept extracting unit 220, the context extracting unit 230, the embedding conversion unit 240, and the image combining unit 250. The image output unit 260 is the same as that of the first embodiment shown in FIG.

  Information on the object displayed in the object image is added as metadata (label) to the object image stored in the database 300. The model updating unit 610 does not give the object image information about the object image used for generating the composite image obtained in the process of generating the composite image by the image combining unit 250 to the object image stored in the database 300. It recognizes that it is information (hereinafter referred to as "sample image information without sample"). For example, information on objects not stored in the database 300 or information not added to an object image stored in the database 300 can be considered. When recognizing the non-sampled object image information, the model updating unit 610 determines to perform zero shot learning of the NN model using the non-sampled object image information. The model update unit 610 performs zero shot learning using object image information on an object image used for generating a composite image, which is obtained in the process of generating a composite image by the image composition unit 250. Update the NN model accordingly. Thereafter, the image combining unit 250 performs image combining using the updated NN model. When information not given a sample is recognized, that information is used for zero shot learning, so information without a sample can be efficiently zero shot learned.

  The model updating unit 610 may perform zero shot learning at any timing, but may perform it at fixed intervals, for example, or at a timing instructed by the administrator.

  The fifth embodiment shows an example in which the image synthesizing device 200 performs zero-shot learning using information obtained in the process of generating a synthesized image requested by the user, but the present invention is not limited to this. . As another example, in the sixth embodiment, a sentence (hereinafter referred to as "sample sentence") to be used for zero shot learning of the NN model is given to the image synthesizer, and the image synthesizer uses the sample sentences to extract a concept. , Context extraction, and embedding transformation, and object image information on what the object image indicates in the process is used for zero-shot learning. However, the image combining unit 250 does not perform image combining in the processing on a sample sentence given for use in zero-shot learning of the NN model. The process of generating a composite image is a high-load process, but in the present embodiment, object image information can be obtained without completing the process until the composite image is actually generated. It is something to use.

  Information extracted from concepts and contexts as feature vectors (hereinafter referred to as "feature information") is obtained by performing concept extraction, context extraction, and embedding transformation. In the sixth embodiment, feature information obtained without generating a composite image is used for zero-shot learning. Therefore, in the sixth embodiment, in order to obtain feature information used for zero shot learning of the NN model, processes of concept extraction, context extraction, and embedding transformation are performed.

  The physical configuration of the image combining system according to the sixth embodiment is the same as that of the first embodiment (or the fifth embodiment) shown in FIG. The configuration of the image combining device 200 of the sixth embodiment is the same as that of the image combining device according to the fifth embodiment shown in FIG.

  Sample sentences are given to the image synthesizing device 200 for zero shot learning. The sample sentence is a sentence related to a virtual composite image, and is a sentence for learning of the NN model. The concept extraction unit 220 extracts a concept from the sample sentence as in the first embodiment. The context extraction unit 230 extracts a context from the sample sentence as in the first embodiment. Furthermore, the embedding transform unit 240 transforms the concept and the context into feature vectors. Object image information (feature information) is obtained by these. However, the image combining unit 250 does not perform image combining in the process on the sample sentence. The model update unit 610 updates the NN model by performing zero-shot learning using object image information (feature information). Since the NN model can be updated by zero-shot learning without generating synthetic data and the amount of processing is small, high-speed learning is possible, and the performance of verbal data synthesis can be rapidly improved.

  In the fifth embodiment, an example of updating the NN model using a general neural network zero-shot learning method has been described. In the seventh embodiment, zero-shot learning by hostile training using generic adaptive networks (GAN) is performed. Shows an example of updating the NN model. In Example 7, not all features of the concept and context are known, but zero shot learning is performed. Given a limited subset of concepts and context features, the image synthesizer can select features of the two concepts and combine new features between the features of the two concepts. The resulting feature is not a concept, but a mixture of both. An example of a possible approach to computing such features and training an NN model is given below.

  The physical configuration of the image combining system according to the seventh embodiment is the same as that of the first embodiment (or the fifth embodiment) shown in FIG. The configuration of the image combining apparatus according to the seventh embodiment is the same as that of the fifth embodiment shown in FIG. The model update unit 610 of this embodiment generates various learning samples using the condition generation model, and updates the NN model by zero shot learning by hostile training using generative adaptive networks (GAN). Since the sample for learning is generated by the condition generation model and the neural network model is updated by zero shot learning by GAN, data synthesis by the NN model can be continuously improved and the NN model can be made robust.

Zero shot learning according to this embodiment will be described. Here, the following notation is used. The learning data D is expressed as equation (1).

Here, the image x _n is an element of the image set as shown in equation (2).

y _i is the label of the image x _i , which labels are obtained from the label space of the known class shown in equation (3).

The label space of the unknown class is represented by equation (4).

Each known class shown in the equation (5) can be expressed by one-shot expression or word embedding (feature vector of a word).

The goal of zero-shot learning in this example is to find an optimal semantic embedding (feature vector representing meaning) that can map unknown images close to the mapping of the correct class label (label indicating class). It is. In other words, the purpose is to project an image and a class label in a semantic space that the embedding of an image is close to the embedding of the same class label as the image and the embedding of an image is different from the embedding of a label of a different class from the image. Do. The loss targeted for the minimization can be expressed as equation (6).

Here, d (x _i , y _i ) is an index indicating the similarity between the semantic embedding of an image and the semantic embedding of a label of the same class. The dot product widely used in semantic matching is used as an index of similarity.

Here, the condition generation model is represented as Gen (y _i , z). The label y _i is the word embedding (feature vector of a word) of the image x _i . Generating model Gen from the same distribution as the image x _i, a NN model for outputting the visual characteristics is used as a condition in the image synthesis. Hereinafter, this visual feature is shown by the symbol shown in equation (7).

Also, the interpolation of the word embedding y _i can be expressed as equation (8).

The interpolation of equation (8) is considered as an input of the generation model Gen, and an interpolation u _i in the visual feature space can be obtained. Two loss functions (Eqs. (9) and (10)) can be obtained for the generated visual features.

Equations (9) and (10) mean that the mapping of visual features interpolated with interpolation ui in visual feature space is between the mapping of word embedding yi and word embedding yj. The loss functions of the following Equations (11) and (12) can be used for the visual features (Equation (7)) generated by the NN model.

  Minimizing L in Equations (11) and (12) means that the mapping of the visual features generated in the NN model is to be closer to that of the image xi. Equations (9) and (10) try to determine the interpolation ui so as not to break the relationship identifying xi, yi, and yj. By using Equations (11) and (12), it is possible to train Equation (6) using more diverse sample visual features.

  Each example mentioned above is an illustration for explanation of the present invention, and is not the meaning which limits the range of the present invention to them. Those skilled in the art can implement the present invention in various other aspects without departing from the scope of the present invention.

Reference Signs List 10 user 100 terminal 101 communication network 110 terminal 200 image combining device 201 communication network 210 statement processing unit 220 concept extraction unit 230 context extraction unit 240 embedding conversion The unit 250, an image combining unit 260, an image output unit 300, a database 310, a concept selection unit 320, a context generation unit 350, an image determination unit 360, an end condition determination unit 410, an input unit 420, and so on. Display unit, 430 ... database storage unit, 510 ... object image, 520 ... patch, 610 ... model update unit

Claims (13)

A database in which element objects, which are synthesizable data related to objects that can be described in natural language, are stored in advance;
An extraction unit that extracts at least one of a concept or context from natural language sentences;
A transformation unit for transforming the concept or the context into a feature vector represented by a vector in a predetermined feature space;
A neural network model for combining element objects according to the input feature vector is held in advance, and based on the neural network model and the feature vector, an element object is selected from the database and combined data is generated using the element object The synthesis unit to
A data synthesizer comprising: The element object includes an object image which is data of an image representing a predetermined object,
The composite data includes a composite image to be composited by the object image,
The neural network model receives a feature vector of at least one of the concept and the context, acquires an object image from the database, and generates a composite image using the acquired object image.
The data synthesizing apparatus according to claim 1.   The extraction unit extracts a concept related to a minimal entity that is a noun among minimal entities that are semantically interpretable units of the sentence, and connects the minimal entity that is a noun to another minimal entity. The data synthesizer according to claim 1, wherein the context is generated. The image combining unit generates the combined image by overlapping or arranging a plurality of object images.
The data synthesizer according to claim 2. The neural network model is a neural network model based on generative adaptive networks which performs learning with two mutually opposing models of a generator and a discriminator, and the generator generates the composite image by the generator.
The data synthesizer according to claim 2. A concept selection unit for selecting a concept to be used for learning the neural network model;
A context generation unit that generates a context used to learn the neural network model;
A determination unit that evaluates a composite image generated by the neural network model;
And have
The conversion unit converts the concept selected by the concept selection unit and the context generated by the context generation unit into a feature vector;
The synthesizing unit selects an element object from the database based on the neural network model and the feature vector, and generates synthesized data using the element object.
The determination unit evaluates the combined data;
If the synthesized data can not obtain a predetermined evaluation, the processing of the synthesizing unit and the judging unit is repeated.
The data synthesizing apparatus according to claim 1. The element object is a visual expression object indicating a visual expression that is an element of an image,
The combining unit generates a combined image by combining the visual representation objects based on the feature vector.
The data synthesizing apparatus according to claim 1. The visual representation object includes a patch representing a feature of an object to be displayed in the composite image.
The combining unit corrects the object image with the patch, and creates a combined image from the corrected object image.
The data synthesizer according to claim 7.   The information processing method according to claim 1, further comprising: an update unit that performs zero-shot learning by using information on element objects used to generate the synthetic data obtained in the process of generating the synthetic data, and updates the neural network model. Data synthesizer as described in. The update unit recognizes that the information on the element object used for generating the composite data, which is obtained in the process of generating the composite data, is information that is not given to the object image stored in the database. Then, it is decided to perform the zero shot learning.
The data synthesizer according to claim 9. It further comprises an updating unit that performs zero-shot learning using information on element objects and updates the neural network model,
The extraction unit extracts at least one of a concept or a context from sample sentences given for learning of the neural network model;
The conversion unit converts the concept or the context into a feature vector represented by a vector in a predetermined feature space;
The updating unit performs zero-shot learning using information on the feature vector, and updates the neural network model.
The data synthesizing apparatus according to claim 1.   The data synthesizing device according to claim 9 or 11, wherein the updating section updates the neural network model by zero shot learning by hostile training using generative adversary networks. Pre-hold a neural network model that synthesizes element objects according to the input feature vector,
It has a database in which element objects that are synthesizable data related to objects that can be described in natural language are stored in advance
The extraction means extracts at least one of a concept or context from natural language sentences,
Transformation means transforms the concept or the context into a feature vector represented by a vector in a predetermined feature space;
And combining means, based on the neural network model and the feature vector, selects an element object from the database and generates synthesized data using the element object.
Data synthesis method.

Images