CN112257775B

CN112257775B - Poetry method by graph based on convolutional neural network and unsupervised language model

Info

Publication number: CN112257775B
Application number: CN202011130476.9A
Authority: CN
Inventors: 李浩天; 汪鹏; 朱佳涛; 曹思辰; 李翔宇; 曾家俊
Original assignee: Southeast University
Current assignee: Southeast University
Priority date: 2020-10-21
Filing date: 2020-10-21
Publication date: 2022-11-15
Anticipated expiration: 2040-10-21
Also published as: CN112257775A

Abstract

The invention provides a poetry mapping method based on a convolutional neural network and an unsupervised language model. According to the method, the user does not need to manually input the text for poetry, only the target image needs to be input when the user uses the method, the entity words and the emotional words are automatically extracted from the input image by using the convolutional neural network, and the poetry elements are enriched by performing similarity expansion on the extracted entity words according to the emotional words, so that a keyword set is formed. The method adopts an unsupervised language model with a self-attention mechanism, automatically generates Chinese poems of which the contents and emotions conform to the image mood according to keywords and emotion labels by using a bidirectional generation algorithm, and designs a two-level content inspection method to further ensure the quality of the generated poems and achieve a good poem generation effect.

Description

Poetry method by graph based on convolutional neural network and unsupervised language model

Technical Field

The invention belongs to the technical field of artificial intelligence, relates to computer vision, natural language processing and Chinese ancient poetry generating technology, and particularly relates to a poetry from picture method based on a convolutional neural network and an unsupervised language model.

Background

Ancient poetry generation is an important entry point for research on automated computer analysis, understanding and use of human language. The ancient poetry generating system aims at generating corresponding ancient poetry according to multi-mode information input by a user, so that the system is required to extract and summarize key information or characteristics from input information, and the ancient poetry is generated by guiding ancient poetry in an input ancient poetry generating model. With the development of deep learning, poetry generating models based on a recurrent neural network and variants thereof are widely used. In recent years, the ancient poetry generating model based on the language model basically realizes the function of generating high-quality poetry on the premise that a user gives a text.

The ancient poetry generating method utilizing the text information requires a user to manually input the text, key information is extracted, screened and expanded from the text by the model to be used as poetry elements, and then corresponding ancient poetry generation is completed according to the key information. This approach has significant limitations. On one hand, the poetry is not in line with the poetry habit of 'feeling scene and living conditions' of ancient poetry in China, and on the other hand, when the input text relates to a modern theme, the content of generating the ancient poetry is incoherent and even logically disordered. In addition, the method is difficult to control the emotion generating the ancient poems and lacks aesthetic value.

Therefore, a method for generating an ancient poem based on image content is thought to be applied, the application number is CN201710610311.3, and the method comprises the following steps: 1) Performing target detection on image content based on a single multi-frame target detection framework to obtain the name of an object; 2) Dividing words of a set number of first Tang poems by adopting a Chinese lexical analysis tool to obtain a vocabulary table, performing feature learning on each word in the vocabulary table by utilizing a word2vec tool, and mapping each word to a vector space; 3) Inputting the obtained object name into a word2vec tool to obtain an object name mapping vector, calculating the cosine similarity between the object name mapping vector and the ancient poetry vector, and selecting a part as a subject term corresponding to the object after setting a threshold; 4) And expanding the keywords by using the theme words, and inputting the keywords into an RNN model obtained after learning Tang poetry to generate ancient poetry. Although the entity extracted from the image is used, the emotion expressed by the image is not considered, and the emotion has great influence on the style of the generated ancient poem; in addition, the ancient poems are generated by replacing entity words with subject words, and the relevance of the ancient poems and the images is weakened to a certain extent. According to the ancient poetry, different technical routes are adopted, the emotion expressed in the image is emphasized, the model can automatically extract emotion words from the image to determine the emotion base of the ancient poetry, and the emotion words and the entity words can be used as poetry contents to appear in the generated ancient poetry, so that the relevance between the image and the ancient poetry is further enhanced.

Disclosure of Invention

In order to solve the problems, the invention provides a poem formation method based on a convolutional neural network and an unsupervised language model, which guides the generation of ancient poems by using visual information and characteristics extracted from images input by a user. The method identifies entities and emotion key tones in an image through two parallel convolutional neural networks, the identified key information is used as an initial key word and is input into a key word expansion module, the module outputs a key word set containing a plurality of entity nouns and an emotion word, finally the key words are used as seed words and are input into an ancient poem generation model, and the ancient poems are generated under the guidance of emotion labels. The ancient poems generated by the method have strong diversity and obtain higher scores on grammar and coherence, and the patent provides a poem forming method by figures based on a convolutional neural network and an unsupervised language model, which comprises the following modules and is characterized in that:

1) Image entity identification and emotion identification: forming a physical word set and an emotion word set by taking common images and emotions in Chinese ancient poems as guidance, classifying input target images by utilizing two parallel convolutional neural networks, and extracting corresponding physical words and emotion words from the images to form an initial keyword set;

2) And (3) keyword expansion: performing word segmentation and frequency statistics on ancient poetry linguistic data with emotion marks, and selecting high-frequency words by taking the frequency as a reference to form a keyword dictionary with different emotion basic tones; randomly selecting a plurality of entity words in an initial keyword set in the module 1), projecting the words to a vector space and performing cosine distance measurement with each word in a corresponding keyword dictionary, and randomly selecting keywords with the distance within a threshold range as expansion keywords to form a keyword set comprising a plurality of entity nouns and an emotional word;

3) Generating ancient poems and checking multi-level contents: pre-training an ancient poetry generating model by utilizing a large number of ancient poetry and ancient language materials, and finely adjusting the ancient poetry language materials with emotion marks; inputting the keyword set obtained in the module 2) into a trained ancient poetry generating model, wherein each keyword is used as a seed word of each sentence of ancient poetry, and ancient poetry is generated under the guidance of an emotional tag; controlling the format of the ancient poetry according to the physique of a user requirement, and ensuring that the generated ancient poetry meets a rule of narrow rhyme according to a rhyme table constructed manually; and grading the generated ancient poems by using the automatic evaluation indexes in two levels of grammar and continuity, regenerating the ancient poems when the scores are lower than a threshold value, and outputting the generated ancient poems to the user and receiving the feedback of the user when the scores are higher than the threshold value.

As a further improvement of the method, in the step 1), the image classification technology is used for carrying out image entity and emotion recognition by utilizing two convolutional neural networks with different parameters, so that higher accuracy is achieved; determining the emotional tone of the ancient poems according to the color characteristics of the images, and meeting the subjective feeling of the user; in order to overcome the problem of extracting multi-entity nouns, the input picture is subjected to random area selection, and the randomly selected area and the original picture are input into the network model together so as to achieve the purpose of extracting a plurality of entity words from the image at the same time.

As a further improvement of the invention, the keywords are projected to a high-dimensional word vector space by using a word vector model in the step 2) to expand and randomly select the keywords, so that the accuracy and diversity of the expanded keywords are ensured; a plurality of keyword dictionaries are constructed by using the ancient poetry corpus labeled with emotion, and keywords which accord with a specific emotion tone can be expanded.

As a further improvement of the invention, the keywords extracted from the image in the step 3) directly appear in the generated ancient poetry, so that the high association between the ancient poetry content and the emotion key and the picture is ensured; an unsupervised language model with a self-attention mechanism and a mask is adopted to complete the ancient poetry generating task, so that the continuity and readability of generated contents are ensured; the emotion label is used for assisting the generation of the ancient poems, so that the emotional fluency is ensured, and the infectivity of the ancient poems is enhanced; when poetry sentences are generated, characters in an alternative range are randomly selected by using a multi-sampling strategy, and meanwhile, keywords can appear at any position of the ancient poetry by using a bidirectional generation algorithm, so that the repeatability of poetry sentence generation is reduced, and the fluency of the whole ancient poetry is enhanced; in the generating process, the quality of the generated ancient poems is automatically judged by a multi-level content inspection method, and the quality of the output ancient poems is indirectly improved.

Compared with the prior art, the invention has the following advantages and beneficial effects:

the method can effectively extract visual information and characteristics of the image, is beneficial to generating the ancient poems which are highly related to the image content and emotion, and has good ancient poems generating effect and performance. Compared with the traditional ancient poem generating method, the ancient poem generating method based on the image information does not need a user to input texts or specific key words, but automatically extracts entity words and emotion words from the input image to serve as initial poem making elements, and then further expands and screens the key words to enrich the poem making elements, so that the ancient poem generating capability of a language model based on deep learning is fully exerted. The method adopts various measures to ensure the emotional relevance of the image and the generated ancient poems, and comprises the steps of constructing keyword dictionaries with different emotion moods and guiding the generation of the ancient poems by using emotion labels. The ancient poetry generating network model used by the invention ensures the ancient poetry quality by utilizing a self-attention mechanism and a bidirectional generating algorithm, and additionally adds a network accessory for checking generated contents from two aspects of grammar and continuity, thereby further improving the readability of generating the ancient poetry. The structure of the three-layer module is easier to modify and optimize the model, so the method has stronger robustness and universality and wider application prospect.

Drawings

FIG. 1 is a logic flow diagram of graphed poetry based on deep learning provided by the present invention;

FIG. 2 is an example of multi-entity extraction using a region selection method;

FIG. 3 is a flow chart of the image keyword extraction and expansion logic based on the convolutional neural network and the word vector model;

FIG. 4 is a logic flow diagram of the application of a bidirectional poetry generating algorithm and a multi-level inspection mechanism;

FIG. 5 is a diagram of an embodiment;

fig. 6 is all test pictures used herein.

Detailed Description

The invention is described in further detail below with reference to the following detailed description and accompanying drawings:

the invention provides a graph poetry method focusing on emotion based on a convolutional neural network and an unsupervised language model. The method identifies entities and emotion key tones in an image through two parallel convolutional neural networks, the identified key information is used as an initial key word and is input into a key word expansion module, the module outputs a key word set containing a plurality of entity nouns and an emotion word, finally the key words are used as seed words and are input into an ancient poem generation model, and the ancient poems are generated under the guidance of emotion labels. The ancient poems generated by the method have strong diversity and obtain higher scores on grammar and coherence.

The poetry from graph method based on the convolutional neural network and the unsupervised language model, as shown in figure 1, comprises the following steps:

1) Image entity and emotion recognition. Before ancient poems are generated, related elements for poem making, namely keywords are required to be obtained from images, the keywords are used as a part of ancient poem content to participate in ancient poem generation, and the ancient poem content is mainly divided into two types: as entity nouns of the image of the ancient poetry and affective words for determining the emotional tone of the ancient poetry, the module needs to perform two tasks, namely mapping of image features to entity keywords and affective keyword sets:

a) The mapping of image features to entity keywords is a one-to-many mapping, and features of an input image are extracted by using a convolutional neural network, and the mapping is completed by means of image classification. The basic architecture of the model is based on the DenseNet network structure. DenseNet proposes a more aggressive dense connection mechanism than ResNet, i.e., each layer accepts all its previous layers as additional input. Another big feature of DenseNet is the feature multiplexing by the connection of features on the channel. These features allow DenseNet to achieve better performance than ResNet with less parametric and computational cost.

Before the model is formally trained, an entity keyword set, namely an output range of the model, is firstly constructed. Considering the content and logistical problems associated with creating classical poems using modern words, alternative physical words are defined as common images in classical poems in china, such as "fallen leaves", "peony", "running water", which fundamentally eliminates the possibility of modern words appearing in classical poems.

In order to realize the one-to-many mapping relation, a threshold value is set for the output of the convolutional neural network model, and all the prediction results larger than the threshold value can be output as entity keywords. In addition, in order to further obtain more poetry elements, a plurality of areas (for example, an upper left area and a lower right area) can be additionally selected from the input image, and the selected areas are input into the model together with the original image, so that as many entities in the image as possible are extracted. As shown in fig. 2, if only the original picture is used, only two physical words, namely "white cloud" and "country" can be extracted, and after the application area is selected, the "running water" can be extracted more. The above-described multi-entity extraction method has many advantages over traditional object detection or multi-label classification. On one hand, the method can obtain more entity categories, enrich poem elements and improve the robustness of the model. On the other hand, in the subsequent application process, new entities can be continuously added, so that the method has stronger expandability, and when a new entity category is added, the multi-label classification model needs to consider whether the entity appears in the existing pictures of the data set (if the entity appears, the corresponding label needs to be changed), so that more labor and time are consumed.

b) In the invention, the mapping from the image characteristics to the emotion keywords is in one-to-one relationship, because the emotion keyword set only comprises three types of emotions: sorrow, happy and feeble tendency. The reason for dividing the emotion in ancient poems into three categories instead of defining an emotion adjective set (such as busy and cool) is mainly two. First, the sorrow (negative) happiness (positive) and the non-emotional tendency (neutral) between the two are the emotional tone of most classical poems in China, which can perfectly summarize the emotional tone of the classical poems. Second, as mentioned above, the emotion words will also be used as seed words for generating ancient poems, which means that the emotion words will appear in specific ancient poems (e.g., who is sad and unbounded in autumn, "sad" emotion keywords appear in ancient poems), and if the choice is made to use the set of adjectives as keywords, the problem of generating ancient poems with modern words may occur. Similar to entity recognition, emotion classification is also implemented using a DenseNet-based convolutional neural network, except that the training data set is exchanged for a library of emotion pictures. Although this method is limited to recognizing emotions by low-level features such as image color features, classification based on features such as color has a better effect when there are fewer emotion categories and conforms to human aesthetic characteristics. The logic flow of the image keyword extraction based on the convolutional neural network is shown in fig. 3, and the entity recognition CNN and the emotion recognition CNN together complete the image entity and emotion recognition and extraction work.

2) And (5) expanding the keywords. Before generating ancient poems by using keywords, further processing, including screening, expanding, and random selection, needs to be performed on the entity and emotion keywords preliminarily extracted from the images. The specific method comprises the following steps:

a) And constructing a keyword dictionary. When the initial keyword needs to be expanded, the expanded keyword is selected from a keyword dictionary constructed in advance. The expansion here is mainly to expand the extracted entity keywords. Similar to the entity keyword set, the alternative keywords in the keyword dictionary also satisfy two requirements: first, it must be an image common to ancient poems; second, no modern words are to be found. In addition, thanks to the ancient poetry corpus of the existing emotion labeling, the keyword dictionary can be further subdivided into an active keyword dictionary containing active alternative words (spring, bamboo, clear sky), a passive keyword dictionary containing passive alternative words (autumn, swan, and frigid cicada), and a neutral keyword dictionary without obvious emotional tendency. And performing word segmentation and frequency statistics on the ancient poem corpus, and selecting high-frequency words to form each keyword dictionary by taking the frequency as a reference. Meanwhile, the corpus is used for training a word vector model word2vec, the model expresses initial entity keywords in a vector form, and therefore the keywords are expanded by finding the closest keywords in a word vector space through cosine distance calculation and dictionary vocabulary range constraint. It is noted that the entity words expanded by the method are not near-synonyms of the original entity words, but are entities most likely to appear in the same poem in the same emotional mood as the original entity words.

b) And generating a final keyword set. For the situation of generating a quadruple ancient poem, logically three entity words and an emotion word are needed as seed words to respectively guide the generation of each poem sentence. Let the initial set of keywords extracted from the image be [ k ] ₁ ，k ₂ ...k _n-1 ，k _n ]Wherein the first n-1 keywords are entity words, k _n Is an emotional word.

When n-1<And 3, extracting less than three entity words, wherein keyword expansion is required. From k ₁ To k _n-1 Randomly selecting a key word k _i And inputting the words into a word vector model, outputting a certain number of alternative keywords by the model according to the probability, selecting a plurality of words from the alternative keywords, and adding the words into the initial set until the number of the entity words is three.

When n-1=3, no operation is required on the initial keyword set.

When n-1 >.

After the entity words are determined, the emotional words also need to be judged. When the emotional words are 'sad' or 'happy', no operation is performed, and the generation of the final keyword set is completed; when the emotional words are neutral, because the words can not appear in the ancient poems, the invention provides two solutions, one is to randomly select one from other two emotional words to replace the words, and the other is to replace the emotional words by additionally selected entity words, namely, the final keyword set contains four entity words. FIG. 3 shows the logic flow of keyword expansion (excluding emotion word determination). The final set of keywords is denoted as [ K ] ₁ ，K ₂ ，K ₃ ，K ₄ ]。

3) Ancient poems are generated and multi-level content inspection is carried out. The ancient poetry generating model receives a keyword set, takes each keyword in the keyword set as a seed word, and completes generation of each group of the ancient poetry under the guidance of an emotional tag. For the same picture, the ancient poems generated every time need to be different, and need satisfy basic rhythm requirements, the content needs to be coherent, and readability needs to be strong, and the ancient poems that the ancient poems generated the model output promptly need to have very strong variety, normalization, continuity. To achieve this effect, both the natural language processing model itself is required to have strong expressive power and some extra constraints and checking mechanisms need to be artificially established. The specific process of the invention for solving the problems is as follows:

a) And (4) selecting a generation model. The ancient poem generation model selects a GPT-2 language model based on deep learning. The GPT model is mainly formed by stacking Transformer decoders, GPT-2 is an improved version of GPT, a fine-tuning layer is removed, more Transformer decoders are stacked, more parameters are possessed, and the text generation effect is more excellent. The input of the GPT-2 model is word embedding vector and position coding. The word embedding vector is a number list which can represent a certain word and capture the meaning of the word; the position code indicates order information of the transform decoder words. The output of the model is the output probability of each word in the vocabulary under the preceding condition. The powerful expressive power of GPT-2 comes from its own mechanism of autoregression and masked attention. The former means that each time a word is output by the model, the word is added behind a previously generated word sequence, which becomes a new input for the next step of the model; the latter means that the model incorporates the understanding of the foregoing when interpreting a word, placing an emphasis on words that are more relevant. Aiming at the generation of the ancient poems, the input of the model is a keyword set (K) processed by a keyword expansion module ₁ ，K ₂ ，K ₃ ，K ₄ ](quadruple ancient poems). Firstly, randomly selecting a physical word and inputting the physical word into a model, and generating a first allied poetry sentence word by the model under the control of an emotion label, a format requirement and a rhyme table; the model then selects other keysAnd repeating the above operations until the whole ancient poem is generated. It is worth noting that the position of the verses generated by selecting emotional words (if any) is controlled in the third and fourth couplets, because most of the Chinese classical poetry singers directly express emotions in the second couplet.

b) And controlling the emotion key. The expression method of the ancient poetry emotion can be generally divided into two types: indirect expression through specific imagery and direct expression through emotional words. In the invention, the former is realized by expanding the key words, and the latter is realized by extracting the emotional words from the image, so that the emotion accuracy of the generated ancient poetry at the poetry sentence level is ensured. In addition, in order to ensure the emotional fluency of the whole ancient poetry, an emotional tag representing the emotional mood should also be used as the input of the ancient poetry model. Therefore, after pre-training is finished by using a large number of ancient poems and ancient texts, the ancient poetry generating model also finely tunes the ancient poetry corpus marked by emotion, and inputs the ancient poetry corpus in a form of < CLS > emotion label < BODY > ancient poetry content < EOS >. When generating ancient poems, the emotion label is determined by emotion words, after inputting the existing content into the model, the emotion label needs to be additionally input, and the emotion label with the format of < SOS > generated content < CLS > emotion label. Under the guidance of the emotion label, the model tends to output contents more conforming to the emotion label, and the emotion mood of the whole ancient poem is further ensured.

c) A bi-directional generation algorithm. For the task of generating ancient poems guided by multiple keywords, the generation method in a) may cause a problem that the keywords always appear at the starting positions of a poem and the consistency between poems is very poor (because each poem is generated separately), which greatly deteriorates the quality of generating the ancient poems. The simplest method for solving the problems is to lead the generation of the whole poem by using one keyword, and randomly insert other keywords in the generation process. This method, while ensuring randomness of the keyword location, may produce a large meaningful bias at the randomly inserted location. The present invention proposes a bi-directional generation algorithm to solve this problem. In addition to the forward generation of the GPT-2 of the ancient poetry, the reverse GPT-2 of the linguistic data of the ancient poetry in the reverse order also participates in the generation of the ancient poetry. Logic flow diagram for a bidirectional generation algorithmAs shown in fig. 4, the numbered arrows in the figure mark the logical sequence of the first and second couples of the ancient poetry, wherein the solid arrow is the first couple and the dotted arrow is the second couple. Specifically, let i-th and j-th words be x _i，j When generating the ith link, firstly, the keyword K is used _i (assuming one word) at random position x _i，j Then, a word (x) of the free position before the keyword is generated by using a reverse GPT-2 model _i，1 ，x _i，2 ，...，x _i，j ) Finally, all the generated contents (including contents of the 1 st to i-1 st links) are input (x) ₁ ，x ₂ ，...，x _i，1 ，x _i，2 ，...，x _i，j ) To complete the union of the remaining content (x) _i，j+1 ，x _i，j+2 ，...，x _i，end ) And (4) generating. In addition, a top-k sampling strategy is used for randomly selecting words with high probability in the generating process, so that the variety of ancient poems is further improved.

d) Format and prosody control. The ancient poetry style mainly comprises regular poems and absolute sentences of five languages and seven languages, the number of the ancient poetry couplets (sentences) is four, but the number of the ancient poetry couplets (sentences) is different, and the ancient poetry couplets (sentences) have strict rhythm requirements. The number of words of each sentence of poetry and the rhythm of each word are respectively controlled by a preset length control parameter and a rhythm table which is manually set up in the process of generating the ancient poetry by the model so as to meet the requirement of controlling the ancient poetry temperament.

e) A multi-level content inspection method. The stability of the quality of the output ancient poems cannot be guaranteed only by depending on the expression capability of the ancient poem generation model, and the robustness of the model is poor. And although the top-k sampling strategy is adopted to randomly select the generated words, the variety of the ancient poems is improved, the grammar specification and the fluency of the ancient poems are damaged to a certain extent. Therefore, in order to further improve the quality of generating the ancient poems, inspectors are arranged at two levels of the grammar of the single sentence and the continuity of the whole poem, and the contents which do not meet the inspection requirements are removed and new contents are regenerated. The logic flow of the multilevel content inspection method is shown in fig. 4. On a grammatical level, the scores of the poetry sentences on the grammar are automatically calculated by using ancient poetry linguistic data with part of speech tagging and adopting an n-gram model, and the poetry sentences with lower scores are abandoned and regenerated again; on a consistency level, n-gram is also used for checking the content of the whole poem, the similarity between the generated ancient poems and excellent ancient poems with a corpus concentrated is evaluated through BLEU, and the ancient poems with a score larger than a certain number are output as a final result.

The invention provides a poetry from graph method based on a convolutional neural network and an unsupervised language model, and the implementation process of the invention is described by taking a test picture as an implementation case. The corresponding illustration is seen in fig. 5.

User input image (FIG. 6 (b))

1) Image entity and emotion recognition:

inputting: image for poetry (FIG. 6 (b))

And (3) outputting: [ { "entity": dog "}, {" entity ":" fallen leaves "}, {" infection ": sad" } { ]

2) And (3) keyword expansion:

inputting: [ { "entity": dog "}, {" entity ":" fallen leaves "}, {" infection ": sad" } { ]

And (3) outputting: [ { "entity": dog "}, {" entity ":" fallen leaves "}, {" entity ": autumn" }, { "observation": "sad" } the method of the present invention is applicable to all people in the same country and has no influence on the life of the people

In this case, the number of input keywords is less than four, and the keyword expansion module needs to further expand the entities in the keyword set. The fallen leaves are randomly selected to expand the keywords, and the expanded results include autumn, residual yang, chrysanthemum and the like. Finally "autumn" is selected as the extended keyword and added to the original set.

3) Generating ancient poems:

inputting:

[

"entity": dog "},

{ "entity": "fallen leaves" },

{ "entity": autumn "},

"emotion": sad "},

{ "gene": seven-language absolute sentence' },

]

and (3) outputting:

[

{ "line1": white jade hook unknown at autumn night, "},

{ "line2": frost of Laoderma heterophyllus is filled in the tree head. "},

{ "line3": who is sad and sad, and "},

{ "line4": canine sheep Shanli Canoe. "},

]

in addition to the set of keywords, the input should also include a selection of a genre. The output in the above example is a seven-word absolute. If the body style is limited to the five-language rhythm poems, the output is:

[

{ "line1": A "autumn teenager tour, not a Chang' an building. "},

{ "line2": tsunday empty city head, eastern mountain dog horse fallow. "},

"line3" means "Jinchun Huazhou, mountain spring fallen leaves stream. "},

{ "line4": this is too sory, and the meeting is too worrying. "},

]

the above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, but any modifications or equivalent variations made according to the technical spirit of the present invention are within the scope of the present invention as claimed.

Claims

1. A poetry method by picture based on a convolutional neural network and an unsupervised language model comprises the following modules, and is characterized in that:

2) And (3) keyword expansion: performing word segmentation and frequency statistics on ancient poetry linguistic data with emotion marks, and selecting high-frequency words by taking the frequency as a reference to form a keyword dictionary with different emotion basic tones; randomly selecting a plurality of entity words in an initial keyword set in the module 1), projecting the words to a vector space and performing cosine distance measurement on the words and each word in a corresponding keyword dictionary, and randomly selecting keywords with the distance within a threshold range as expansion keywords to form a keyword set comprising a plurality of entity nouns and an emotional word;

2. The poetry method by figure based on convolutional neural network and unsupervised language model as claimed in claim 1, characterized in that, in step 1), image entity and emotion recognition is carried out by using convolutional neural networks with two different parameters by using an image classification technology, so as to achieve higher accuracy; determining the emotional tone of the ancient poems according to the color characteristics of the images, and meeting the subjective feeling of the user; in order to overcome the problem of extracting multi-entity nouns, the random area selection is carried out on the input picture, and the randomly selected area and the original picture are input into the network model together so as to achieve the purpose of extracting a plurality of entity words from the image at the same time.

3. The poetry method by figure based on convolutional neural network and unsupervised language model as claimed in claim 1, characterized in that, in step 2), the word vector model is used to project the keywords to the high-dimensional word vector space for expansion and random selection of the keywords, thus ensuring the accuracy and diversity of the expanded keywords; a plurality of keyword dictionaries are constructed by using the ancient poetry corpus labeled with emotion, and keywords which accord with a specific emotion tone can be expanded.

4. The poetry method based on the convolutional neural network and the unsupervised language model as claimed in claim 1, characterized in that the keywords extracted from the images in step 3) directly appear in the generated poetry, thereby ensuring the high association between the content and emotional mood of the poetry and the images; an unsupervised language model with a self-attention mechanism and a mask is adopted to complete the ancient poetry generating task, so that the consistency and readability of generated contents are ensured; the emotion labels are used for assisting the generation of the ancient poems, so that the emotion fluency of the whole ancient poems is ensured, and the infectivity of the ancient poems is enhanced; when poetry sentences are generated, characters in an alternative range are randomly selected by using a multi-sampling strategy, and meanwhile, keywords can appear at any position of the ancient poetry by using a bidirectional generation algorithm, so that the repeatability of poetry sentence generation is reduced, and the fluency of the whole ancient poetry is enhanced; the quality of the generated ancient poems is automatically judged by a multi-level content inspection method in the generation process, and the quality of the output ancient poems is indirectly improved.