KR102445932B1 - Image generation technique using multi-modal mapping information on knowledge distillation - Google Patents

Abstract

An image generation technique using multimodal mapping information based on knowledge distillation is disclosed. An image generating method according to an embodiment includes: generating text summary information from text data using a pre-trained text summary model; and generating a new image for the generated text summary information using the pre-learned image model as the generated text summary information is input to the pre-trained image model.

Description

Image generation technique using multimodal mapping information based on knowledge distillation {IMAGE GENERATION TECHNIQUE USING MULTI-MODAL MAPPING INFORMATION ON KNOWLEDGE DISTILLATION}

The description below relates to techniques for generating images from text data.

In order to solve the visual quality problem, the GAWWN methodology has the limitations of image creation and partial key point location with a clear global structure for each object. The generated image has the correct object shape and color, but is not natural. To solve the visual quality problem, the Stack-GAN consists of two steps, the first step is an image of the coarse structure during use in the second step. Using these images as input, more local images are further generated.

In the non-patent literature (Zhang et al. "Photographic text-to-image synthesis with a hierarchically-nested adversarial network,") hierarchically-nested adversarial network, the hierarchically-nested adversarial network was introduced to create photographic images as a network capable of modeling high-resolution image statistics. has been disclosed.

However, the prior art has various problems such as the following semantic consistency problem, visual quality problem, difficulty and speed problem of learning.

The semantic consistency problem refers to the problem of inconsistency between text description and image content that is inconsistent in both distribution and presentation. Semantic consistency issues also exist for other cross-modal tasks such as image captions. For the semantic consistency problem, we learn the mapping from the text feature space to the image pixel space and show the visual information of the text description in the composite image. However, their performance is limited by heterogeneous correlation frameworks that cannot fully utilize visual information in text descriptions.

The visual quality problem refers to the problem of inconsistency between the data distribution of the real image and the synthetic image in a high-dimensional space. It is difficult to approximate the image data distribution directly in high-dimensional space. Visual quality issues are common in image compositing tasks.

The previously widely used adversarial training paradigm has had difficulties in optimization and parameter adjustment within the model due to the feed-forward structure. there was.

A method and system for generating an image from text data may be provided. Specifically, a method and system for generating a new image by combining the result of text summarization through knowledge distillation from text data and the result of knowledge distillation from a pre-trained image recognition model can provide

The image generating method includes: generating text summary information from text data using a pre-trained text summary model; and generating a new image for the generated text summary information using the pre-learned image model as the generated text summary information is input to the pre-trained image model.

The generating of the text summary information includes inputting the text data into a pre-trained text summary model, and extracting text summary information from the text data through a knowledge distillation technique using the pre-trained text summary model. generating and outputting a text embedding as a result of the text summary.

The generating of the image may include: searching for data distribution in a pixel feature space of an image from the generated text summary information by performing knowledge extraction through a knowledge distillation technique using a pre-learned image model may include.

The step of generating the image includes converting the generated text summary information into image information through a text to image process, and searching for the most similar object by searching for object information from the converted image information. may include steps.

The generating of the image may include inputting an image including the found most similar object into a pre-trained image model, and obtaining a first similar image related to the found most similar object using the pre-trained image model may include the step of

The generating of the image may include inputting the obtained first similarity image to a GAN-based image generator, and using the GAN-based image generator to generate a second similar image based on pixel information on the obtained first similar image. It may include generating a second similar image.

The generating of the image may include evaluating the accuracy of the generated second similar image with the embedding vector through an image to text process, and the generated second similar image according to the evaluated accuracy. providing a car-like image as a result of generating the image.

The image generating system includes: a text summary unit generating text summary information from text data using a pre-trained text summary model; and an image generator configured to generate a new image for the generated text summary information using the pre-learned image model as the generated text summary information is input to the pre-trained image model.

The text summary unit inputs the text data into a pre-trained text summary model, and generates text summary information from the text data through a knowledge distillation technique using the pre-learned text summary model, and summarizes the text. You can output text embeddings as a result of

The image generator may search for data distribution in a pixel feature space of an image from the generated text summary information by performing knowledge extraction through a knowledge distillation technique using a pre-learned image model.

The image generator may convert the generated text summary information into image information through a text-to-image process, and retrieve object information from the converted image information to search for the most similar object.

The image generator may input an image including the found most similar object into a pre-trained image model, and obtain a first similar image related to the found most similar object using the pre-trained image model. .

The image generating unit inputs the obtained first similarity image to a GAN-based image generator, and based on pixel information on the obtained first similarity image through the GAN-based image generator, second-order similarity You can create an image.

The image generator is configured to evaluate the accuracy of the generated second similarity image with the embedding vector through an image-to-text process, and the generated second similarity image according to the evaluated accuracy. can be provided as a result of image creation.

By directly creating an imaginary image that could only be expressed in words, image creation technology can be developed.

In addition, a realistic image can be generated from text data through a knowledge distillation technique.

In addition, by shortening the time of text summarization and image generation and reducing the computational cost, it is possible to efficiently generate an image from text data.

1 is a block diagram illustrating the configuration of an image generating system according to an embodiment.
2 is a flowchart illustrating an image generating method in an image generating system according to an exemplary embodiment.
3 is a diagram for explaining an overview operation of an image generating system according to an exemplary embodiment.
4 is a diagram for explaining a knowledge distillation operation in an image generating system according to an exemplary embodiment.
5 is a diagram for describing an image generating operation in an image generating system according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a configuration of an image generating system according to an embodiment, and FIG. 2 is a flowchart illustrating an image generating method in the image generating system according to an embodiment.

The processor of the image generating system 100 may include a text summary unit 110 and an image generating unit 120 . These processor components may be representations of different functions performed by the processor according to control instructions provided by program code stored in the image generating system. The processor and components of the processor may control the image generating system to perform steps 210 to 220 included in the image generating method of FIG. 2 . In this case, the processor and the components of the processor may be implemented to execute instructions according to the code of the operating system included in the memory and the code of at least one program.

The processor may load the program code stored in the file of the program for the image generating method into the memory. For example, when a program is executed in the image generating system, the processor may control the image generating system to load a program code from a file of the program into a memory according to the control of the operating system. At this time, each of the text summary unit 110 and the image generation unit 120 executes a command of a corresponding part of the program code loaded in the memory to execute the subsequent steps 210 to 220 with different functionalities of the processor. can be expressions.

In operation 210 , the text summary unit 110 may generate text summary information from text data using a pre-learned text summary model. The text summary unit 110 inputs text data into a pre-learned text summary model, generates text summary information from the text data through a knowledge distillation technique using the pre-learned text summary model, and summarizes the text. You can output text embeddings as a result of

In operation 220 , the image generator 120 may generate a new image for the generated text summary information using the pre-learned image model as the generated text summary information is input to the pre-trained image model. The image generator 120 may search for data distribution in a pixel feature space of an image from text summary information generated by performing knowledge extraction through a knowledge distillation technique using a pre-trained image model. The image generator 120 may convert the generated text summary information into image information through a text-to-image process and search for the most similar object by searching for object information from the converted image information. The image generator 120 may input an image including the found most similar object to the pre-trained image model, and obtain a first similar image related to the found most similar object using the pre-trained image model. The image generator 120 inputs the obtained first similar image to a GAN-based image generator, and based on pixel information on the first similar image obtained through the GAN-based image generator, a second similar image can create The image generator 120 evaluates the accuracy of the generated second similar image with an embedding vector through an image to text process, and uses the second similar image generated according to the evaluated accuracy as an image. It can be provided as a result of creation.

3 is a diagram for explaining an overview operation of an image generating system according to an exemplary embodiment.

The image generating system 100 may obtain text summary information by using a text summary model that has already been trained on the text book 301 . In this case, the text book may mean a book made of text data. In the embodiment, it is assumed that it is an encyclopedia and will be described. In addition, it can be applied not only to books made of text data but also to all contents (documents, files, articles, etc.) made of text data. Also, even a book containing text data and image data, only a text data portion can be used.

The image generating system 100 may obtain text summary information by using a pre-learned text summary model. Specifically, the image generating system 100 may input text data configured in the text book 301 into a pre-trained text summary model. The image generating system 100 may obtain a text description of the text data by using a pre-learned text summary model. In this case, as the pre-trained text summary model, Kogpt2, kobert, koeletra, etc. may be used. The image generating system 100 may use a knowledge distillation technique and a fine tuning technique using a pre-learned text summary model. A text summarization 302 may be generated from the text book 301 . Text summarization techniques can be largely divided into abstract summarization and abstract summarization. Extractive summary technology is a method of extracting several important key sentences or phrases from a document and creating a summary composed of them. Therefore, the sentences or phrases in the summary sentences resulting from the summary are the sentences in the original text. A typical algorithm is the TextRank algorithm. Abstract summary technique is more difficult than abstract summary by creating and extracting sentences that were not in the original text. Artificial neural networks are mainly used, and a representative model is seq2seq. For abstract summarization, there must be a set of original texts and actual summaries because it uses supervised learning methods. That is, learning is possible only when there is label data. Since a large data set is required, preparation of the data set and the learning time are difficult to use the actual text summarization technique.

A text description 303 may be extracted through the text summary 302 of the text book 301 . In other words, text descriptions related to text summaries may be extracted. The image generation system 100 may obtain text summary information as a result of the text summary 302 . At this time, as a result of the text summary 302 , a text embedding 304 may be returned as an output. The image generating system 100 may perform Text to Image Synthesis 305 for generating an image by using the text summary information. In other words, the text embedding 304 may be input as input data to generate an image. The image generating system 100 may perform a knowledge distillation technique using a pre-learned image model. At this time, other algorithms are trained based on the image and label information to directly generate the image, but in the embodiment, the data distribution in the pixel feature space of the image is obtained through a pre-trained image model without inputting image data. can be found The image generating system 100 may input text embeddings into a pre-trained image model. The image generating system 100 may obtain an artificial image 306 for text embedding from a pre-trained image model.

In detail, text embedding (text summary information) may be transmitted to a multi-modal mapping module and converted into text-to-image information. The multi-modal mapping module can also be converted into rough image information by using a pre-trained model (model for text-to-image conversion), and the converted image information can find the most similar image in the pre-trained image model. have. For example, if we say "a small white bird with a brown beak", we find the most similar image among several images such as "swan" and "crane", and the image corresponding to "brown beak" in the image pixel information found. It is possible to generate an image (new image) that is more expressive by using a GAN-based image generator.

The generated image information is converted back to image-to-text in the multi-modal mapping module, and accuracy with the original embedding vector information is checked to evaluate how accurate the image is generated. After the evaluation, if the target value is reached, image generation may be terminated, and if the target value is not reached, the image generation process may be repeated.

As an example, the image generation system learns the text book to summarize the image of the bird, delivers the summarized image of the bird, and extracts the general bird appearance from the previously learned model, and the image of the bird according to the content of the summarized sentence can be changed to create image data. As another example, the image generating system may extract a color matching an object through text book learning, and automatically generate an object with a color matching the image. The image generation system can automatically create an image that combines two colors if white and red go well together.

Instead of the previously widely used adversarial training paradigm, the approach proposed in Examples uses feed-forward structure and knowledge distillation learning, and with the help of multi-modal modules, optimization and parameter adjustment, etc. more stable than

In the embodiment, the text-to-image and image-to-text semantic consistency problems can be solved through the cross evaluation method in multi-modal.

In the embodiment, since the visual quality problem has already distilled the most similar object, the blur problem of the image itself can be solved, and the problem can be overcome by improving the performance of the GAN algorithm.

Since the methodology presented in the embodiment uses a method of distilling knowledge from a pretrained teacher model, it is superior to a low level learning method in terms of time and speed in terms of difficulty in learning.

4 is a diagram for explaining a knowledge distillation operation in an image generating system according to an exemplary embodiment.

The image generation system may perform the text summarization 410 and image generation 420 operations using a knowledge distillation technique. For the operation of the text summary 410 , a pre-trained text summary model (a teacher model) and a learning model may be configured.

Knowledge distillation is to transfer the knowledge of a previously learned network (Teacher network) to a small network (Student network) that is actually used. A deep learning model has a large number of parameters, and if the amount of computation is large, feature extraction is performed well, so the performance in classification or regression problems is improved. If deep learning can achieve the same performance as a larger model with a small model beyond the technical level where a model with good performance is simply a good model, computing resource, energy, and memory more efficient in The knowledge distillation method aims to improve the performance of a student network by transferring the knowledge of a large network to a small network during the learning process so that even a small network can achieve similar performance to a large network.

The image generating system may acquire text summary information based on a knowledge distillation technique from text data using a pre-learned text summary model. Specifically, the text summary model may be a model trained using a data set. In this case, the data set may mean data for text summary. This pre-learned text summary model corresponds to a teacher model, and the student model may generate text summary information from the extracted knowledge distillation technique.

The image generating system may generate a new image from the generated text summary information using the pre-trained image model. Specifically, the image model may be a model trained using a data set. In this case, the data set may include data for image generation. This pre-learned image model corresponds to a teacher model. By performing a knowledge distillation technique from a pre-trained image model (for example, a model learned from open data such as CoCo/Imagenet), the Student Model operates, and a new image is generated from the text summary information and the distilled image. can create For example, the new image may be an image extracted in response to text summary information stored in a database, or may be an image on which correction, processing (specific processing/conversion), and editing are performed in relation to text summary information.

5 is a diagram for describing an image generating operation in an image generating system according to an exemplary embodiment.

The image generating system may receive text summary information. In this case, the text summary information may be in the form of text embedding (embedding vector). The image generating system may receive the embedding vector transmitted to the multi-modal mapping module as an input and pass the text to image model from the multi-modal mapping module 520 . The text-to-image model may be a model in which the multi-modal mapping module 520 converts text data into image data.

The image generating system may obtain the most similar object by searching for the most similar object 530 in order to obtain a basic image from the pre-trained image model. The basic image may mean an image converted from text data through a text-to-image process. For example, the image generating system may retrieve an object similar to text summary information from an image converted through text-to-image. Alternatively, the image generating system may acquire an image including the most similar object found through text-to-image.

The image generating system inputs an image including the searched most similar object to the pre-trained image model 540, and uses the pre-trained image model 540 to obtain a first similar image related to the most similar object found. can In this case, the searched most similar object may be input to the pre-trained image model, or knowledge distillation may be performed by inputting an image including the searched most similar object to the pre-trained model.

The image generating system may acquire the target image by passing the obtained image including the most similar object to the GAN-based image generator 510 . In this case, the target image may mean an image to be finally generated in relation to the text summary information. GAN (Generative Adversarial Network) trains the classification model first in Adversarial Network. After that, the process of learning the generative network is repeated by giving and receiving each other. The training of the classification model consists of two steps: one is the process of inputting real data and training the artificial neural network to classify the data as real, and the second is, contrary to the first, by inputting fake data generated by the generative model. This is the process of learning to classify the data as fake. Through this process, the classification model can classify real data as real and fake data as fake. After training the classification model, the generative model is trained in a way that deceives the learned classification model. The fake data created by the generative model is input into the discriminant model, and the generative model is trained to generate data similar to real data enough to classify the fake data as real. A large amount of data is required for GAN learning, and an image is generated using the results learned by classification. The image generating system inputs the obtained first-order similar image to the GAN-based image generator 510, and based on the pixel information for the first-order similar image obtained through the GAN-based image generator 510, the second A car-like image can be created.

The image generation system may pass the target image back through the image-to-text model in the multi-modal mapping module 520 . The image-to-text model may be a model in which the multi-modal mapping module 520 converts image data into text data. In other words, the target image can be converted from an image to a text form. The image generating system may evaluate the accuracy between the target image converted into text data and the embedding vector. The image generating system may evaluate the accuracy of the generated second-order similar image with the embedding vector through an image-to-text process, and provide the generated second-order similar image according to the evaluated accuracy as a result of image generation. .

The image generating system may evaluate ( 550 ) accuracy between the text data converted from the image to the text form and the embedding vector. For example, when the accuracy is greater than or equal to a preset reference value (reference score), the image generating system may terminate the process, and if the accuracy is less than or equal to the preset reference value, repeat the image creation process of FIG. 5 to generate an image. Alternatively, the image generating system may calculate the difference between the text data converted from the image to the text form and the embedding vector, and when the calculated difference value is greater than or equal to a preset reference value, terminate the process, and the calculated difference value is a preset reference value In the following cases, the image may be generated by repeating the image generation process of FIG. 5 .

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

In the image generation method,
generating text summary information from text data through a knowledge distillation technique using a pre-learned text summary model; and
Generating a new image for the generated text summary information through a knowledge distillation technique using the pre-learned image model as the generated text summary information is input into the pre-trained image model
including,
Creating the image includes:
Searching for object information from the generated image information to search for the most similar object, inputting an image including the searched most similar object into a pre-trained image model, and using the pre-trained image model to search for the most similar object Acquire a first similar image related to an object, input the obtained first similar image to a GAN-based image generator, and pixel information on the obtained first similar image through the GAN-based image generator A second similar image is generated based on providing a second-order similar image as a result of image creation;
An image creation method comprising According to claim 1,
The step of generating the text summary information includes:
Input the text data to a pre-trained text summary model, generate text summary information from the text data through a knowledge distillation technique using the pre-trained text summary model, and embed text as a result of text summary step to output
An image creation method comprising According to claim 1,
Creating the image includes:
Exploring data distribution in a pixel feature space of an image from the generated text summary information by performing knowledge extraction through a knowledge distillation technique using a pre-learned image model
An image creation method comprising 4. The method of claim 3,
Creating the image includes:
converting the generated text summary information into image information through a text-to-image process
An image creation method comprising delete delete delete An image generating system comprising:
a text summary unit generating text summary information from text data through a knowledge distillation technique using a pre-learned text summary model; and
An image generating unit that generates a new image for the generated text summary information through a knowledge distillation technique using the pre-learned image model as the generated text summary information is input to the pre-trained image model
including,
The image generating unit,
Searching for object information from the generated image information to search for the most similar object, inputting an image including the searched most similar object into a pre-trained image model, and using the pre-trained image model to search for the most similar object Acquire a first similar image related to an object, input the obtained first similar image to a GAN-based image generator, and pixel information on the obtained first similar image through the GAN-based image generator A second similar image is generated based on to provide a second-order similar image as a result of image creation.
image creation system. 9. The method of claim 8,
The text summary section,
The text data is input to a pre-trained text summary model, and the text summary information is generated from the text data through a knowledge distillation technique using the pre-learned text summary model, and as a result of text summary, the text print the embedding
An image generating system, characterized in that. 9. The method of claim 8,
The image generating unit,
It is a method of searching for data distribution in the pixel feature space of an image from the generated text summary information by performing knowledge extraction through a knowledge distillation technique using a pre-learned image model.
An image generating system, characterized in that. 11. The method of claim 10,
The image generating unit,
Converting the generated text summary information into image information through a text-to-image process
An image generating system, characterized in that. delete delete delete

Images