CN112257775A - Poetry method by graph based on convolutional neural network and unsupervised language model - Google Patents

Poetry method by graph based on convolutional neural network and unsupervised language model Download PDF

Info

Publication number
CN112257775A
CN112257775A CN202011130476.9A CN202011130476A CN112257775A CN 112257775 A CN112257775 A CN 112257775A CN 202011130476 A CN202011130476 A CN 202011130476A CN 112257775 A CN112257775 A CN 112257775A
Authority
CN
China
Prior art keywords
ancient
poetry
emotion
poems
words
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011130476.9A
Other languages
Chinese (zh)
Other versions
CN112257775B (en
Inventor
李浩天
汪鹏
朱佳涛
曹思辰
李翔宇
曾家俊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN202011130476.9A priority Critical patent/CN112257775B/en
Publication of CN112257775A publication Critical patent/CN112257775A/en
Application granted granted Critical
Publication of CN112257775B publication Critical patent/CN112257775B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/166—Editing, e.g. inserting or deleting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/237—Lexical tools
    • G06F40/242—Dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation
    • G06N5/022—Knowledge engineering; Knowledge acquisition
    • G06N5/025—Extracting rules from data

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Machine Translation (AREA)

Abstract

The invention provides a poem mapping method based on a convolutional neural network and an unsupervised language model. According to the method, the user does not need to manually input the text for poetry, only the target image needs to be input when the user uses the method, the entity words and the emotional words are automatically extracted from the input image by using the convolutional neural network, and the poetry elements are enriched by performing similarity expansion on the extracted entity words according to the emotional words, so that a keyword set is formed. The method adopts an unsupervised language model with a self-attention mechanism, automatically generates the Chinese ancient poems of which the contents and the emotions conform to the image mood according to the keywords and the emotion labels by utilizing a bidirectional generation algorithm, and designs a two-level content inspection method to further ensure the quality of the generated ancient poems and achieve good ancient poems generating effect.

Description

Poetry method by graph based on convolutional neural network and unsupervised language model
Technical Field
The invention belongs to the technical field of artificial intelligence, relates to computer vision, natural language processing and Chinese ancient poetry generating technology, and particularly relates to a poetry from picture method based on a convolutional neural network and an unsupervised language model.
Background
Ancient poetry generation is an important entry point for research on automated computer analysis, understanding and use of human language. The ancient poetry generating system aims at generating corresponding ancient poetry according to multi-mode information input by a user, so that the system is required to extract and summarize key information or characteristics from input information, and the ancient poetry is generated by guiding ancient poetry in an input ancient poetry generating model. With the development of deep learning, poetry generation models based on recurrent neural networks and variants thereof are widely used. In recent years, the ancient poetry generating model based on the language model basically realizes the function of generating high-quality poetry on the premise that a user gives a text.
The ancient poetry generating method utilizing the text information requires a user to manually input the text, key information is extracted, screened and expanded from the text by the model to be used as poetry elements, and then corresponding ancient poetry generation is completed according to the key information. This approach has significant limitations. On one hand, the poetry is not in line with the poetry habit of 'feeling scene and living conditions' of ancient poetry in China, and on the other hand, when the input text relates to a modern theme, the content of generating the ancient poetry is incoherent and even logically disordered. In addition, the method is difficult to control the emotion generating the ancient poems and lacks aesthetic value.
Therefore, it is thought that an ancient poetry generating method based on image content is applied, the application number is CN201710610311.3, and the method comprises the following steps: 1) performing target detection on the image content based on a single multi-frame target detection frame to obtain the name of an object; 2) dividing words of a set number of first Tang poems by adopting a Chinese lexical analysis tool to obtain a vocabulary table, performing feature learning on each word in the vocabulary table by utilizing a word2vec tool, and mapping each word to a vector space; 3) inputting the obtained object name into a word2vec tool to obtain an object name mapping vector, calculating the cosine similarity between the object name mapping vector and the ancient poetry vector, and selecting a part as a subject term corresponding to the object after setting a threshold; 4) and expanding the keywords by using the theme words, and inputting the keywords into an RNN model obtained after learning Tang poetry to generate ancient poetry. Although the entity extracted from the image is used, the emotion expressed by the image is not considered, and the emotion has great influence on the style of the generated ancient poem; in addition, the ancient poems are generated by replacing entity words with subject words, and the relevance of the ancient poems and the images is weakened to a certain extent. According to the ancient poetry, different technical routes are adopted, the emotion expressed in the image is emphasized, the model can automatically extract emotion words from the image to determine the emotion base of the ancient poetry, and the emotion words and the entity words can be used as poetry contents to appear in the generated ancient poetry, so that the relevance between the image and the ancient poetry is further enhanced.
Disclosure of Invention
In order to solve the problems, the invention provides a poem formation method based on a convolutional neural network and an unsupervised language model, which guides the generation of ancient poems by using visual information and characteristics extracted from images input by a user. The method identifies entities and emotion key tones in an image through two parallel convolutional neural networks, the identified key information is used as an initial key word and is input into a key word expansion module, the module outputs a key word set containing a plurality of entity nouns and an emotion word, finally the key words are used as seed words and are input into an ancient poem generation model, and the ancient poems are generated under the guidance of emotion labels. The ancient poems generated by the method have strong diversity and obtain higher scores on grammar and coherence, and the patent provides a poem forming method by figures based on a convolutional neural network and an unsupervised language model, which comprises the following modules and is characterized in that:
1) image entity identification and emotion identification: forming a physical word set and an emotion word set by taking common images and emotions in Chinese ancient poems as guidance, classifying input target images by utilizing two parallel convolutional neural networks, and extracting corresponding physical words and emotion words from the images to form an initial keyword set;
2) and (3) keyword expansion: performing word segmentation and frequency statistics on ancient poetry linguistic data with emotion marks, and selecting high-frequency words by taking the frequency as a reference to form a keyword dictionary with different emotion basic tones; randomly selecting a plurality of entity words in an initial keyword set in the module 1), projecting the words to a vector space and performing cosine distance measurement with each word in a corresponding keyword dictionary, and randomly selecting keywords with the distance within a threshold range as expansion keywords to form a keyword set comprising a plurality of entity nouns and an emotional word;
3) generating ancient poems and checking multi-level contents: pre-training an ancient poetry generating model by utilizing a large number of ancient poetry and ancient language materials, and finely adjusting the ancient poetry language materials with emotion marks; inputting the keyword set obtained in the module 2) into a trained ancient poetry generating model, wherein each keyword is used as a seed word of each sentence of ancient poetry, and ancient poetry is generated under the guidance of an emotional tag; controlling the format of the ancient poem according to the genre required by a user, and ensuring that the generated ancient poem meets the rule of zeptose rhyme according to a rhyme table constructed manually; and grading the generated ancient poems by using the automatic evaluation indexes in two levels of grammar and continuity, regenerating the ancient poems when the scores are lower than a threshold value, and outputting the generated ancient poems to the user and receiving the feedback of the user when the scores are higher than the threshold value.
As a further improvement of the method, in the step 1), the image classification technology is used for carrying out image entity and emotion recognition by utilizing two convolutional neural networks with different parameters, so that higher accuracy is achieved; determining the emotional tone of the ancient poems according to the color characteristics of the images, and meeting the subjective feeling of the user; in order to overcome the problem of extracting multi-entity nouns, the random area selection is carried out on the input picture, and the randomly selected area and the original picture are input into the network model together so as to achieve the purpose of extracting a plurality of entity words from the image at the same time.
As a further improvement of the invention, the keywords are projected to a high-dimensional word vector space by using a word vector model in the step 2) to expand and randomly select the keywords, so that the accuracy and diversity of the expanded keywords are ensured; a plurality of keyword dictionaries are constructed by using the ancient poetry corpus labeled with emotion, and keywords which accord with a specific emotion tone can be expanded.
As a further improvement of the invention, the keywords extracted from the image in the step 3) directly appear in the generated ancient poetry, so that the high association between the ancient poetry content and the emotion key and the picture is ensured; an unsupervised language model with a self-attention mechanism and a mask is adopted to complete the ancient poetry generating task, so that the consistency and readability of generated contents are ensured; the emotion labels are used for assisting in the generation of the ancient poems, so that the emotion fluency is ensured, and the infectivity of the ancient poems is enhanced; when poetry sentences are generated, characters in an alternative range are randomly selected by using a multi-sampling strategy, and meanwhile, keywords can appear at any position of the ancient poetry by using a bidirectional generation algorithm, so that the repeatability of poetry sentence generation is reduced, and the fluency of the whole ancient poetry is enhanced; in the generating process, the quality of the generated ancient poems is automatically judged by a multi-level content inspection method, and the quality of the output ancient poems is indirectly improved.
Compared with the prior art, the invention has the following advantages and beneficial effects:
the method can effectively extract visual information and characteristics of the image, is beneficial to generating the ancient poems which are highly related to the image content and emotion, and has good ancient poems generating effect and performance. Compared with the traditional ancient poem generating method, the ancient poem generating method based on the image information does not need a user to input texts or specific key words, but automatically extracts entity words and emotion words from the input image to serve as initial poem making elements, and then further expands and screens the key words to enrich the poem making elements, so that the ancient poem generating capability of a language model based on deep learning is fully exerted. The method adopts various measures to ensure the emotional relevance of the image and the generated ancient poems, and comprises the steps of constructing keyword dictionaries of different emotion tones and guiding the generation of the ancient poems by using emotion labels. The ancient poetry generating network model used by the invention ensures the ancient poetry quality by utilizing a self-attention mechanism and a bidirectional generating algorithm, and additionally adds a network accessory for checking generated contents from two aspects of grammar and continuity, thereby further improving the readability of generating the ancient poetry. The structure of the three-layer module is easier to modify and optimize the model, so the method has stronger robustness and universality and wider application prospect.
Drawings
FIG. 1 is a logic flow diagram of graphed poetry based on deep learning provided by the present invention;
FIG. 2 is an example of multi-entity extraction using a region selection method;
FIG. 3 is a flowchart of the image keyword extraction and expansion logic based on a convolutional neural network and a word vector model;
FIG. 4 is a logic flow diagram of the application of a bidirectional poetry generating algorithm and a multi-level inspection mechanism;
FIG. 5 is a view of an embodiment;
fig. 6 is all test pictures used herein.
Detailed Description
The invention is described in further detail below with reference to the following detailed description and accompanying drawings:
the invention provides a poetry from graph method focusing on emotion based on a convolutional neural network and an unsupervised language model. The method identifies entities and emotion key tones in an image through two parallel convolutional neural networks, the identified key information is used as an initial key word and is input into a key word expansion module, the module outputs a key word set containing a plurality of entity nouns and an emotion word, finally the key words are used as seed words and are input into an ancient poem generation model, and the ancient poems are generated under the guidance of emotion labels. The ancient poems generated by the method have strong diversity and obtain higher scores on grammar and consistency.
The poetry from graph method based on the convolutional neural network and the unsupervised language model, as shown in figure 1, comprises the following steps:
1) image entity and emotion recognition. Before ancient poems are generated, related elements for poem making, namely keywords are required to be obtained from images, the keywords are used as a part of ancient poem content to participate in ancient poem generation, and the ancient poem content is mainly divided into two types: as entity nouns of the image of the ancient poetry and affective words for determining the emotional tone of the ancient poetry, the module needs to perform two tasks, namely mapping of image features to entity keywords and affective keyword sets:
a) the mapping of image features to entity keywords is a one-to-many mapping, and features of an input image are extracted by using a convolutional neural network, and the mapping is completed by means of image classification. The basic architecture of the model is based on the DenseNet network structure. Compared to ResNet, densnet proposes a more aggressive mechanism of dense connections, i.e., each layer accepts all its previous layers as additional input. Another big feature of DenseNet is the feature multiplexing by the connection of features on the channel. These features allow DenseNet to achieve better performance than ResNet with less parametric and computational cost.
Before the model is formally trained, an entity keyword set, namely an output range of the model, is firstly constructed. Considering the content and logistical problems associated with creating classical poems using modern words, alternative physical words are defined as common images in classical poems in china, such as "fallen leaves", "peony", "running water", which fundamentally eliminates the possibility of modern words appearing in classical poems.
In order to realize the one-to-many mapping relation, a threshold value is set for the output of the convolutional neural network model, and the prediction results larger than the threshold value can be output as entity keywords. In addition, in order to further obtain more poetry elements, a plurality of areas (for example, an upper left area and a lower right area) can be additionally selected from the input image, and the selected areas are input into the model together with the original image, so that as many entities in the image as possible are extracted. As shown in fig. 2, if only the original picture is used, only two entity words, namely "white cloud" and "country" can be extracted, and after the application area is selected, the image of "running water" can be extracted. The above-described multi-entity extraction method has many advantages over traditional object detection or multi-label classification. On one hand, the method can obtain more entity categories, enrich poem elements and improve the robustness of the model. On the other hand, in the subsequent application process, new entities can be continuously added, so that the method has stronger expandability, and when a new entity category is added, the multi-label classification model needs to consider whether the entity appears in the existing pictures of the data set (if the entity appears, the corresponding label needs to be changed), so that more labor and time are consumed.
b) In the invention, the mapping from the image features to the emotion keywords is in one-to-one relationship, because the emotion keyword set only contains three types of emotions: sorrow, happy and feeble tendency. The reason for dividing the emotion in ancient poems into three categories instead of defining an emotion adjective set (such as busy and cool) is mainly two. First, the sorrow (negative) happiness (positive) and the non-emotional tendency (neutral) between the two are the emotional tone of most classical poems in China, which can perfectly summarize the emotional tone of the classical poems. Second, as mentioned above, the emotion words will also be used as seed words for generating ancient poems, which means that the emotion words will appear in specific ancient poems (e.g., who is sad and unbounded in autumn, "sad" emotion keywords appear in ancient poems), and if the choice is made to use the set of adjectives as keywords, the problem of generating ancient poems with modern words may occur. Similar to entity recognition, emotion classification is also implemented using a DenseNet-based convolutional neural network, except that the training data set is exchanged for a library of emotion pictures. Although this method is limited to recognizing emotions by low-level features such as image color features, classification based on features such as color has a better effect when there are fewer emotion categories and conforms to human aesthetic characteristics. The logic flow of the image keyword extraction based on the convolutional neural network is shown in fig. 3, and the entity recognition CNN and the emotion recognition CNN together complete the image entity and emotion recognition and extraction work.
2) And (5) expanding the keywords. Before generating ancient poems by using keywords, further processing, including screening, expanding, and random selection, needs to be performed on the entity and emotion keywords preliminarily extracted from the images. The specific method comprises the following steps:
a) and constructing a keyword dictionary. When the initial keyword needs to be expanded, the expanded keyword is selected from a keyword dictionary constructed in advance. The expansion here is mainly to expand the extracted entity keywords. Similar to the entity keyword set, the alternative keywords in the keyword dictionary also satisfy two requirements: first, it must be a common image in ancient poems; second, no modern words can appear. In addition, thanks to the ancient poetry corpus of the existing emotion labeling, the keyword dictionary can be further subdivided into an active keyword dictionary containing active alternative words (spring, bamboo, clear sky), a passive keyword dictionary containing passive alternative words (autumn, swan, and frigid cicada), and a neutral keyword dictionary without obvious emotional tendency. And performing word segmentation and frequency statistics on the ancient poem corpus, and selecting high-frequency words to form each keyword dictionary by taking the frequency as a reference. Meanwhile, the corpus is utilized to train a word vector model word2vec, the model expresses initial entity keywords in a vector form, and therefore the keywords are expanded by finding the closest keywords in a word vector space through cosine distance calculation and dictionary vocabulary range constraint. It is noted that the entity words expanded by the method are not near-synonyms of the original entity words, but are entities most likely to appear in the same poem in the same emotional mood as the original entity words.
b) And generating a final keyword set. For the situation of generating a quadruple ancient poem, logically three entity words and an emotion word are needed as seed words to respectively guide the generation of each poem sentence. Let the initial set of keywords extracted from the image be k1，k2...kn-1，kn]Wherein the first n-1 keywords are entity words, knIs an emotional word.
When n-1<And 3, extracting less than three entity words, and performing keyword expansion. From k1To kn-1In randomly selecting a keyword kiInput into a word vector model with high probabilityAnd outputting a certain number of alternative keywords at a low level, and selecting a plurality of words from the alternative keywords and adding the words into the initial set until the number of the entity words is three.
When n-1 ═ 3, no operation is required on the initial keyword set.
When n-1>3, screening is carried out on n-1 keywords, namely three entity words are randomly selected from the keywords.
After the entity words are determined, the emotion words need to be judged. When the emotional words are 'sad' or 'happy', no operation is performed, and the generation of the final keyword set is completed; when the emotion words are 'neutral', because the words can not appear in the ancient poems, the invention provides two solutions, one is to randomly select one from other two emotion words to replace the words, and the other is to replace the emotion words by additionally selected entity words, namely, the final keyword set comprises four entity words. FIG. 3 shows the logic flow of keyword expansion (excluding emotion word judgment). The final set of keywords is denoted as [ K ]1，K2，K3，K4]。
3) Ancient poems are generated and multi-level content inspection is carried out. The ancient poetry generation model receives the keyword set, takes each keyword in the keyword set as a seed word, and completes generation of each unit of the ancient poetry under the guidance of the emotional tag. For the same picture, the ancient poems generated every time need to be different, basic rhythm requirements need to be met, the content needs to be coherent, and the readability needs to be strong, namely the ancient poems output by the ancient poem generation model need to have strong diversity, normalization and coherence. To achieve this effect, both the natural language processing model itself is required to have strong expressive power and some extra constraints and checking mechanisms need to be artificially established. The specific process of the invention for solving the problems is as follows:
a) and (4) selecting a generation model. The ancient poem generation model selects a GPT-2 language model based on deep learning. The GPT model is mainly formed by stacking the Transformer decoders, GPT-2 is an improved version of GPT, a fine-tuning layer is removed, more Transformer decoders are stacked, more parameters are possessed, and the text generation effect is more excellent. Input of GPT-2 model is word embeddingVector and position coding. The word embedding vector is a number list which can represent a certain word and capture the meaning of the word; the position code indicates order information of the transform decoder words. The output of the model is the output probability of each word in the vocabulary under the preceding condition. The powerful expressive power of GPT-2 comes from its own mechanism of autoregression and masked attention. The former means that every time a word is output by the model, the word is added behind a word sequence generated before, and the word sequence becomes a new input of the next step of the model; the latter means that the model incorporates the understanding of the foregoing when interpreting a word, placing an emphasis on words with higher relevance. Aiming at the generation of the ancient poems, the input of the model is a keyword set (K) processed by a keyword expansion module1，K2，K3，K4](quadruple ancient poems). Firstly, randomly selecting a physical word and inputting the physical word into a model, and generating a first allied poetry sentence word by the model under the control of an emotion label, a format requirement and a rhyme table; and then the model selects other keywords and repeats the operations until the whole ancient poem is generated. It is worth noting that the position of the verse generated by selecting the emotional words (if any) is controlled in the third and fourth couplets, because most of the Chinese classical poetry singers directly express emotions in the two couplets.
b) And controlling the emotion key. The expression method of the ancient poetry emotion can be generally divided into two types: indirect expression through specific imagery and direct expression through emotional words. In the invention, the former is realized by expanding the key words, and the latter is realized by extracting the emotional words from the image, so that the emotion accuracy of the generated ancient poetry at the poetry sentence level is ensured. In addition, in order to ensure the emotional fluency of the whole ancient poetry, an emotional tag representing the emotional mood should also be used as the input of the ancient poetry model. Therefore, after pre-training is finished by using a large number of ancient poems and ancient texts, the ancient poem generating model of the invention also finely adjusts the ancient poem corpus marked by emotion, and the input format is < CLS > emotion label < BODY > ancient poem content < EOS >. When generating ancient poems, the emotion label is determined by emotion words, after inputting the existing content into the model, the emotion label needs to be additionally input, and the emotion label with the format of < SOS > generated content < CLS > emotion label. Under the guidance of the emotion label, the model tends to output contents more conforming to the emotion label, and the emotion tone of the whole ancient poem is further guaranteed.
c) A two-way generation algorithm. For the task of generating ancient poems guided by multiple keywords, the generation method in a) may cause a problem that the keywords always appear at the starting positions of a poem and the consistency between poems is very poor (because each poem is generated separately), which greatly deteriorates the quality of generating the ancient poems. The simplest method for solving the problems is to lead the generation of the whole poem by using one keyword, and randomly insert other keywords in the generation process. This method, while ensuring randomness of the keyword location, may produce a large meaningful bias at the randomly inserted location. The present invention proposes a bi-directional generation algorithm to solve this problem. In addition to the forward generation of the GPT-2 of the ancient poetry, the reverse GPT-2 of the ancient poetry corpus utilizing the reverse order also participates in the generation of the ancient poetry. The logic flow diagram of the bidirectional generating algorithm is shown in fig. 4, wherein the arrows with numbers in the diagram mark the logic sequence when the first and second couples of ancient poems are generated, wherein the solid line arrow is the first couple, and the dotted line arrow is the second couple. Specifically, let i-th word and j-th word be xi，jWhen generating the ith link, firstly, the keyword K is usedi(assuming one word) at random position xi，jThen, a word (x) of the free position before the keyword is generated by using a reverse GPT-2 modeli，1，xi，2，...，xi，j) Finally, all the generated contents (including contents of the 1 st to i-1 st links) are input (x)1，x2，...，xi，1，xi，2，...，xi，j) To complete the union of the remaining content (x)i，j+1，xi，j+2，...，xi，end) And (4) generating. In addition, a top-k sampling strategy is used for randomly selecting words with high probability in the generating process, so that the variety of ancient poems is further improved.
d) Format and prosody control. The style of the ancient poems generated by the invention is mainly the rhythm poems and the absolute sentences of five languages and seven languages, and the number of the ancient poems is four, but the number of the ancient poems is different and the ancient poems have strict rhythm requirements. In the process of generating the ancient poetry by the model, the word number of each sentence poetry and the rhythm of each word are respectively controlled by a preset length control parameter and a rhythm table which is manually set up so as to meet the requirement of controlling the ancient poetry law.
e) A multi-level content inspection method. The stability of the quality of the output ancient poems cannot be guaranteed only by depending on the expression capability of the ancient poem generation model, and the robustness of the model is poor. And although the top-k sampling strategy is adopted to randomly select the generated words, the variety of the ancient poems is improved, the grammar specification and the fluency of the ancient poems are damaged to a certain extent. Therefore, in order to further improve the quality of generating the ancient poems, inspectors are arranged at two levels of the grammar of the single sentence and the continuity of the whole poem, and the contents which do not meet the inspection requirements are removed and new contents are regenerated. The logic flow of the multilevel content inspection method is shown in fig. 4. On a grammatical level, the scores of the poetry sentences on the grammar are automatically calculated by using ancient poetry linguistic data with part of speech tagging and adopting an n-gram model, and the poetry sentences with lower scores are abandoned and regenerated again; on a consistency level, the content of the whole poem is checked by using an n-gram, the similarity between the generated ancient poems and excellent ancient poems with concentrated linguistic data is evaluated through BLEU, and the ancient poems with a certain score are output as a final result.
The invention provides a poetry from graph method based on a convolutional neural network and an unsupervised language model, and the implementation process of the invention is described by taking a test picture as an implementation case. The corresponding illustration is seen in fig. 5.
User input image (FIG. 6(b))
1) Image entity and emotion recognition:
inputting: image for poetry (FIG. 6(b))
And (3) outputting: [ { "entity": dog "}, {" entity ":" fallen leaves "}, {" infection ": sad" } { ]
2) And (3) keyword expansion:
inputting: [ { "entity": dog "}, {" entity ":" fallen leaves "}, {" infection ": sad" } { ]
And (3) outputting: [ { "entity": dog "}, {" entity ":" fallen leaves "}, {" entity ": autumn" }, { "observation": "sad" } the method of the present invention is applicable to all people in the same country and has no influence on the life of the people
In this case, the number of input keywords is less than four, and the keyword expansion module needs to further expand the entities in the keyword set. The fallen leaves are randomly selected to expand the keywords, and the expanded results include autumn, residual yang, chrysanthemum and the like. Finally "autumn" is selected as the expanded keyword and added to the original set.
3) Generating ancient poems:
inputting:
[
{ "entity": dog "},
{ "entity": "fallen leaves" },
{ "entity": autumn "},
{ "implantation": sadness "},
{ "gene": seven-language absolute sentence' },
]
and (3) outputting:
[
{ "line1": white jade hook unknown at autumn night, "},
{ "line2": the root of Laoderma heterophyllus is full of tree. "},
{ "line3": who is sad and endless, and "},
{ "line4": Canine sheep Shanli Canoe. "},
]
in addition to the set of keywords, the input should also include a selection of a genre. The output in the above example is a seven-word absolute. If the body style is limited to the five-language rhythm poems, the output is:
[
{ "line1": the autumn juvenile swimming is not a Changan building. "},
{ "line2": Tsunday night sky city head, eastern mountain dog horse break. "},
{ "line3": "Jinchun Huazhou, mountain spring deciduous leaf flow. "},
{ "line4": this is too sad, and the meeting is worried about. "},
]
the above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, but any modifications or equivalent variations made according to the technical spirit of the present invention are within the scope of the present invention as claimed.

Claims (4)

1. A poetry method by graph based on a convolutional neural network and an unsupervised language model comprises the following modules, and is characterized in that:
1) image entity identification and emotion identification: forming a physical word set and an emotion word set by taking common images and emotions in Chinese ancient poems as guidance, classifying input target images by utilizing two parallel convolutional neural networks, and extracting corresponding physical words and emotion words from the images to form an initial keyword set;
2) and (3) keyword expansion: performing word segmentation and frequency statistics on ancient poetry linguistic data with emotion marks, and selecting high-frequency words by taking the frequency as a reference to form a keyword dictionary with different emotion basic tones; randomly selecting a plurality of entity words in an initial keyword set in the module 1), projecting the words to a vector space and performing cosine distance measurement with each word in a corresponding keyword dictionary, and randomly selecting keywords with the distance within a threshold range as expansion keywords to form a keyword set comprising a plurality of entity nouns and an emotional word;
3) generating ancient poems and checking multi-level contents: pre-training an ancient poetry generating model by utilizing a large number of ancient poetry and ancient language materials, and finely adjusting the ancient poetry language materials with emotion marks; inputting the keyword set obtained in the module 2) into a trained ancient poetry generating model, wherein each keyword is used as a seed word of each sentence of ancient poetry, and ancient poetry is generated under the guidance of an emotional tag; controlling the format of the ancient poem according to the genre required by a user, and ensuring that the generated ancient poem meets the rule of zeptose rhyme according to a rhyme table constructed manually; and grading the generated ancient poems by using the automatic evaluation indexes in two levels of grammar and continuity, regenerating the ancient poems when the scores are lower than a threshold value, and outputting the generated ancient poems to the user and receiving the feedback of the user when the scores are higher than the threshold value.
2. The poetry method by figure based on convolutional neural network and unsupervised language model as claimed in claim 1, characterized in that, in step 1), image entity and emotion recognition is carried out by using convolutional neural networks with two different parameters by using an image classification technology, so as to achieve higher accuracy; determining the emotional tone of the ancient poems according to the color characteristics of the images, and meeting the subjective feeling of the user; in order to overcome the problem of extracting multi-entity nouns, the random area selection is carried out on the input picture, and the randomly selected area and the original picture are input into the network model together so as to achieve the purpose of extracting a plurality of entity words from the image at the same time.
3. The poetry method by figure based on convolutional neural network and unsupervised language model as claimed in claim 1, characterized in that, in step 2), the word vector model is used to project the keywords to the high-dimensional word vector space for expansion and random selection of the keywords, thus ensuring the accuracy and diversity of the expanded keywords; a plurality of keyword dictionaries are constructed by using the ancient poetry corpus labeled with emotion, and keywords which accord with a specific emotion tone can be expanded.
4. The poetry method by picture based on convolutional neural network and unsupervised language model as claimed in claim 1, characterized in that the keywords extracted from the image in step 3) will directly appear in the generated poetry, ensuring the high association of the content and emotion key of the poetry with the picture; an unsupervised language model with a self-attention mechanism and a mask is adopted to complete the ancient poetry generating task, so that the consistency and readability of generated contents are ensured; the emotion labels are used for assisting the generation of the ancient poems, so that the emotion fluency of the whole ancient poems is ensured, and the infectivity of the ancient poems is enhanced; when poetry sentences are generated, characters in an alternative range are randomly selected by using a multi-sampling strategy, and meanwhile, keywords can appear at any position of the ancient poetry by using a bidirectional generation algorithm, so that the repeatability of poetry sentence generation is reduced, and the fluency of the whole ancient poetry is enhanced; in the generating process, the quality of the generated ancient poems is automatically judged by a multi-level content inspection method, and the quality of the output ancient poems is indirectly improved.
CN202011130476.9A 2020-10-21 2020-10-21 Poetry method by graph based on convolutional neural network and unsupervised language model Active CN112257775B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011130476.9A CN112257775B (en) 2020-10-21 2020-10-21 Poetry method by graph based on convolutional neural network and unsupervised language model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011130476.9A CN112257775B (en) 2020-10-21 2020-10-21 Poetry method by graph based on convolutional neural network and unsupervised language model

Publications (2)

Publication Number Publication Date
CN112257775A true CN112257775A (en) 2021-01-22
CN112257775B CN112257775B (en) 2022-11-15

Family

ID=74264437

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011130476.9A Active CN112257775B (en) 2020-10-21 2020-10-21 Poetry method by graph based on convolutional neural network and unsupervised language model

Country Status (1)

Country Link
CN (1) CN112257775B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112800750A (en) * 2021-01-26 2021-05-14 浙江香侬慧语科技有限责任公司 Unsupervised non-autoregressive ancient poetry generation method and device and storage medium
CN113010717A (en) * 2021-04-26 2021-06-22 中国人民解放军国防科技大学 Image verse description generation method, device and equipment
CN113553822A (en) * 2021-07-30 2021-10-26 网易（杭州）网络有限公司 Ancient poetry generation model training method, ancient poetry generation equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107480132A (en) * 2017-07-25 2017-12-15 浙江工业大学 A kind of classic poetry generation method of image content-based
CN107832292A (en) * 2017-11-02 2018-03-23 合肥工业大学 A kind of conversion method based on the image of neural network model to Chinese ancient poetry
CN109086270A (en) * 2018-07-24 2018-12-25 重庆大学 System and method of composing poem automatically based on classic poetry corpus vectorization

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107480132A (en) * 2017-07-25 2017-12-15 浙江工业大学 A kind of classic poetry generation method of image content-based
CN107832292A (en) * 2017-11-02 2018-03-23 合肥工业大学 A kind of conversion method based on the image of neural network model to Chinese ancient poetry
CN109086270A (en) * 2018-07-24 2018-12-25 重庆大学 System and method of composing poem automatically based on classic poetry corpus vectorization

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
吴斌等: "基于迁移学习的唐诗宋词情感分析", 《电子学报》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112800750A (en) * 2021-01-26 2021-05-14 浙江香侬慧语科技有限责任公司 Unsupervised non-autoregressive ancient poetry generation method and device and storage medium
CN113010717A (en) * 2021-04-26 2021-06-22 中国人民解放军国防科技大学 Image verse description generation method, device and equipment
CN113553822A (en) * 2021-07-30 2021-10-26 网易（杭州）网络有限公司 Ancient poetry generation model training method, ancient poetry generation equipment and storage medium
CN113553822B (en) * 2021-07-30 2023-06-30 网易（杭州）网络有限公司 Ancient poetry generating model training, ancient poetry generating method, equipment and storage medium

Also Published As

Publication number Publication date
CN112257775B (en) 2022-11-15

Similar Documents

Publication Publication Date Title
CN112257775B (en) Poetry method by graph based on convolutional neural network and unsupervised language model
CN107741928B (en) Method for correcting error of text after voice recognition based on domain recognition
CN112184858B (en) Virtual object animation generation method and device based on text, storage medium and terminal
CN111159368B (en) Reply generation method of personalized dialogue
CN109086270B (en) Automatic poetry making system and method based on ancient poetry corpus vectorization
CN108416065A (en) Image based on level neural network-sentence description generates system and method
CN101187990A (en) A session robotic system
CN113435211B (en) Text implicit emotion analysis method combined with external knowledge
CN112966525B (en) Law field event extraction method based on pre-training model and convolutional neural network algorithm
CN112183109B (en) MASS-based poetry sentence generation information steganography method
CN115599901A (en) Machine question-answering method, device, equipment and storage medium based on semantic prompt
CN114153973A (en) Mongolian multi-mode emotion analysis method based on T-M BERT pre-training model
CN112183106A (en) Semantic understanding method and device based on phoneme association and deep learning
CN115293168A (en) Multi-language abbreviation disambiguation algorithm based on pre-training model semantic understanding
CN111221964A (en) Text generation method guided by evolution trends of different facet viewpoints
CN114298031A (en) Text processing method, computer device and storage medium
CN116092478A (en) Voice emotion conversion method, device, equipment and storage medium
CN115858736A (en) Emotion text generation method based on emotion prompt fine adjustment
CN114492382A (en) Character extraction method, text reading method, dialog text generation method, device, equipment and storage medium
CN114446324A (en) Multi-mode emotion recognition method based on acoustic and text features
CN112163414A (en) Chinese lyric generating method based on Word2Vec, LSTM and attention mechanism
CN112766101B (en) Method for constructing Chinese lip language identification modeling unit set
Ma et al. Learning phone embeddings for word segmentation of child-directed speech
CN117236338B (en) Named entity recognition model of dense entity text and training method thereof
Rahman et al. CapNet: An Encoder-Decoder based Neural Network Model for Automatic Bangla Image Caption Generation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant