WO2021042517A1

WO2021042517A1 - Artificial intelligence-based article gist extraction method and device, and storage medium

Info

Publication number: WO2021042517A1
Application number: PCT/CN2019/116936
Authority: WO
Inventors: 陈一峰; 周骏红; 汪伟
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-09-02
Filing date: 2019-11-10
Publication date: 2021-03-11
Also published as: CN110705268A

Abstract

An artificial intelligence-based article gist extraction method, comprising: receiving a text data set, and performing word segmentation and merging operations on the text data set to obtain a word text set; performing an encoding operation on the word text set and then converting same into a word matrix set, and inputting the word matrix set into a word vector transformation model for training to obtain a word vector set; performing a dimensionality reduction operation on the word vector set, and then inputting same into a convolutional neural network model for training; and converting text data inputted by the user into word vectors, and then inputting same into the trained convolutional neural network model so as to obtain an article gist and outputting same. Also provided are an artificial intelligence-based article gist extraction device, and a computer-readable storage medium. The method can achieve a precise and efficient article gist extraction function based on artificial intelligence.

Description

Article subject extraction method, device and storage medium based on artificial intelligence

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on September 2, 2019. The application number is 201910826795.4 and the invention title is "Artificial intelligence-based article subject extraction method, device and computer-readable storage medium". The entire content is incorporated into this application by reference.

Technical field

This application relates to the field of artificial intelligence technology, and in particular to a method, device and computer-readable storage medium for extracting the subject matter of articles based on artificial intelligence.

Background technique

At present, the main points of most articles rely on professional industry professionals for analysis, such as manual reading and research of enterprise development reports, and then summing up the main points for senior leaders to make decisions, academic reports are summarized by relevant persons and then simplifying the main points for others to learn, etc. The mode is particularly time-consuming and labor-intensive. In addition, there is an article subject extraction based on the traditional Naive Bayes algorithm. However, due to the large computational resources of the Naive Bayes algorithm and the high error rate of the extracted subject, it cannot meet the actual requirements.

Summary of the invention

This application provides an artificial intelligence-based article subject extraction method, device, and computer-readable storage medium, the main purpose of which is to perform intelligent subject extraction based on the article input by the user.

In order to achieve the above-mentioned purpose, an artificial intelligence-based article subject extraction method provided in this application includes: receiving a text data set, performing word segmentation and merging operations on the text data set to obtain a word text set; The text set is converted into a word matrix set after the encoding operation, and the word matrix set is input into the word vector conversion model to train to obtain the word vector set; the word vector set is subjected to dimensionality reduction operation and then input into the convolutional neural network model The training value is obtained by training, and the size of the training value and the preset threshold is judged, if the training value is greater than the preset threshold, the convolutional neural network model continues training, and if the training value is less than the preset threshold , The convolutional neural network model completes training; receives text data input by the user, converts the text data input by the user into a word vector, and then inputs it into the trained convolutional neural network model to obtain the main idea of the article and output it .

In addition, in order to achieve the above purpose, this application also provides an artificial intelligence-based article subject extraction device, which includes a memory and a processor, and the memory stores artificial intelligence-based articles that can run on the processor. The subject extraction program, when the artificial intelligence-based article subject extraction program is executed by the processor, the following steps are implemented: receiving a text data set, and performing operations including word segmentation and merging on the text data set to obtain a word text set; The word text set is converted into a word matrix set after an encoding operation, and the word matrix set is input into a word vector conversion model to train to obtain a word vector set; the word vector set is subjected to a dimensionality reduction operation and then input to the convolutional nerve The training value is obtained by training in the network model, and the size of the training value and the preset threshold is judged. If the training value is greater than the preset threshold, the convolutional neural network model continues training, and if the training value is less than the With a preset threshold, the convolutional neural network model completes training; receives text data input by the user, converts the text data input by the user into a word vector, and enters it into the trained convolutional neural network model to obtain an article Subject and output.

In addition, in order to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium on which is stored an artificial intelligence-based article subject extraction program, which can be One or more processors are executed to implement the steps of the above-mentioned artificial intelligence-based article subject extraction method.

This application first performs word segmentation and merging operations on the text data set to obtain a word text set, which can avoid the influence of wrong words on the subject of the entire article. At the same time, the word text set is encoded and word vector transformed to obtain a word vector set. The encoding operation and the word vector transformation reduce the dimension of the word while amplifying the feature attributes. Further, the convolutional neural network model has excellent feature extraction capabilities, can efficiently identify word features, and improve the subject of the article. Output accuracy rate. Therefore, the artificial intelligence-based article subject extraction method, device, and computer-readable storage medium proposed in this application can achieve accurate article subject output results.

Description of the drawings

FIG. 1 is a schematic flowchart of an artificial intelligence-based article subject extraction method provided by an embodiment of the application;

2 is a schematic diagram of the internal structure of an artificial intelligence-based article subject extraction device provided by an embodiment of the application;

3 is a schematic diagram of modules of an artificial intelligence-based article subject extraction program in an artificial intelligence-based article subject extraction device provided by an embodiment of the application.

The realization, functional characteristics, and advantages of the purpose of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

detailed description

It should be understood that the specific embodiments described here are only used to explain the application, and not used to limit the application.

This application provides an article subject extraction method based on artificial intelligence. Referring to FIG. 1, it is a schematic flowchart of an artificial intelligence-based article subject extraction method provided by an embodiment of this application. The method can be executed by a device, and the device can be implemented by software and/or hardware.

In this embodiment, the method for extracting the subject matter of an article based on artificial intelligence includes:

S1. Receive a text data set, and perform operations including word segmentation and merging on the text data set to obtain a word text set.

Preferably, the text data set includes multiple types of texts, such as news, social, academic, government development planning, and corporate investment.

The cleaning is to remove stop words, Arabic letters and other heteromorphic words in the text data set, because heteromorphic words that have no actual meaning will reduce the text classification effect. The stop words have no practical meaning and have no effect on text analysis, but are frequently used words, such as commonly used pronouns and prepositions. Specifically, the cleaning is to construct a table of heteromorphic words in advance, sequentially traverse the words in the text data set, and if the words are the same as those in the table of heteromorphic words, remove them until the traversal is completed.

The word segmentation is to segment each sentence in the text data set to obtain a single word. Because there is no clear separation mark between words in Chinese representation, word segmentation is indispensable. Preferably, the word segmentation described in this application can be processed using a stuttering word database based on programming languages such as Python, JAVA, etc. The stuttering word database is developed for research and development based on Chinese part-of-speech features, and is a collection of the text data The number of occurrences of each word is converted to frequency, and the path with the maximum probability is found based on dynamic programming, and the maximum segmentation combination based on word frequency is found. For example, there are text fragments in the text data set: When people know how to exchange with the system, they can tell their true self and discernment, because in their eyes, before making an equivalent exchange with the system, what is true to them? Nor is it. After the stuttering word database is processed, it becomes: when people know how to exchange with the system, they can tell the truth about themselves, because in their eyes, before making equivalent exchanges with the system, the truth does nothing to them. It's not. Wherein, the space part represents the processing result of the stuttering word database.

Further, since the subject of multiple sentences may be the same, the merging is to merge multiple sentences with the same subject to achieve the purpose of greatly reducing the words in the text data set. Preferably, the merging includes: traversing each text in the text data set, dividing the text according to paragraphs to obtain several paragraphs, and presetting words that appear more than twice in each paragraph as hypothetical subjects, and constructing Constructing a log-likelihood function based on the conditional probability model of each sentence in each paragraph and the hypothetical subject, and optimizing the conditional probability model based on the log-likelihood function to obtain the subject of each sentence, Combine several sentences with the same subject into one sentence to complete the combination operation.

Specifically, the conditional probability model is:

Among them, y ₁ ,..., y _N , y _i are the hypothetical subjects, N is the number of the hypothetical subjects, D is the paragraph, j is the number of the paragraph, such as D ₁ is the number of the text In the first paragraph, s is the sentence in the paragraph, P(y _i |s) is the probability of assuming that the subject y _i is the subject of the sentence s, and s(i, y _i ) represents the hypothetical subject of the sentence i is y _i .

Preferably, the log likelihood function is:

Where argmax is the hypothetical subject corresponding to the maximum partial derivative of the conditional probability model to all the hypothetical subjects.

S2. The word text set is converted into a word matrix set after an encoding operation, and the word matrix set is input into a word vector conversion model for training to obtain a word vector set.

Preferably, the encoding adopts a one-hot encoding form, and the one-hot encoding is to first number each word in the word text set to obtain the largest number, and then create the largest number. Encoding matrices with the same numerical numbering dimension, traverse each sentence in the word text set in turn, map each sentence to the encoding matrix, and according to the numerical number of each word in the word text set Complete the encoding operation to obtain the word matrix set. For example, the collection of words and texts is: when people know how to exchange with the system, they can tell their true self and the truth. This is reality. After the text is numbered, it is: when ¹ person ² understands the exchange of ³ and ⁴ system ⁵ ^{, 6} hours ⁷ , they ⁸ can ⁹ make ^{10 the} real ¹¹ ^{themselves 12} and tell them ¹³ , this is ¹⁴ reality ¹⁵ , and get the biggest The number is 15, and then a 15-dimensional encoding matrix is created. Further, if the traversal sentence is: This is reality, then the encoded is [0,0,0,0,0,0,0,0,0,0 ,0,0,0,1,1].

Preferably, the word vector conversion model includes assuming a weight relationship between a word matrix in the word matrix set and a word word vector in the word vector set, and calculating the weight based on the weight relationship to complete the The conversion process from the word matrix set to the word vector set.

Specifically, the weight relationship is:

d={(t ₁ ,w ₁ ),(t ₂ ,w ₂ ),……,(t _i ,w _i ),……,(t _n ,w _n )}

Where, d is the word matrix set, t ₁ , t ₂ ,..., t _n are word matrices in the word matrix set, as in the above [0,0,0,0,0,0,0,0 ,0,0,0,0,0,1,1] etc., w ₁ , w ₂ , ..., w _n are the weights of the corresponding word matrix.

Further, the calculation method of the weight is:

Wherein, f _i represents the number of occurrences of the word matrix in the word matrix set, N is the total number of texts in the text data set, N _j represents the total number of _{words in the text data set, and N i} represents the word i in the text data set The number of occurrences of, F _m is the weighting factor, and the value is generally less than 1.

S3. After performing a dimensionality reduction operation on the word vector set, input it into a convolutional neural network model for training to obtain a training value, and determine the size of the training value and a preset threshold, and if the training value is greater than the preset threshold, The convolutional neural network model continues to be trained, and if the training value is less than the preset threshold, the convolutional neural network model completes the training.

Preferably, the dimensionality reduction operation includes calculating the covariance of each word vector in the word vector set, and removing word vectors in the covariance whose absolute value is greater than a preset covariance threshold to obtain a dimensionality-reduced word vector set.

Further, the covariance is:

Wherein, x _i , x _j represent each word vector of the word vector set, n is the number of the word vector set, and cov(x _i , x _j ) represents the calculation of the covariance between _{x i} and x _j. If the calculated covariance cov(x _i ,x _j ) is not 0, if it is greater than 0, it means a positive correlation, and if it is less than 0, it means a negative correlation.

In a preferred embodiment of the present application, the convolutional neural network model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input layer receives the word vector set, and the convolutional layer , Pooling layer, fully connected layer combined with activation function training to obtain training values and output through the output layer.

The activation function in the preferred embodiment of the present application may include a Softmax function, and the loss function is a least square function. The Softmax function is:

Among them, O _j represents the output value of the jth neuron in the fully connected layer, I _j represents the input value of the jth neuron in the output layer, t represents the total number of neurons in the output layer, and e is infinite Do not cycle decimals;

The least square method L(s) is:

Where, s is the training value, k is the number of word vector sets after dimensionality reduction, y _i is the word vector set, and y′ _i is the predicted value of the convolutional neural network model.

S4. Receive the text data input by the user, convert the text data input by the user into a word vector, and input it into the trained convolutional neural network model to obtain and output the subject matter of the article.

For example, receiving a user input describing an article describing ancient literary prisons, the convolutional neural network model after the completion of the training outputs the article subject: the article describing ancient literary prisons exposed the feudal rule against literati The cruel tyranny of the author shows the author's deep sympathy for intellectuals and strong resentment of the brutal rule.

The invention also provides an article subject extraction device based on artificial intelligence. Referring to FIG. 2, it is a schematic diagram of the internal structure of an artificial intelligence-based article subject extraction device provided by an embodiment of the present application.

In this embodiment, the artificial intelligence-based article subject extraction device 1 may be a PC (Personal Computer, personal computer), or a terminal device such as a smart phone, a tablet computer, or a portable computer, or a server. The artificial intelligence-based article subject extraction device 1 at least includes a memory 11, a processor 12, a communication bus 13, and a network interface 14.

The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, and the like. In some embodiments, the memory 11 may be an internal storage unit of the article subject extraction device 1 based on artificial intelligence, such as the hard disk of the artificial intelligence-based article subject extraction device 1. In other embodiments, the memory 11 may also be an external storage device of the article subject extraction device 1 based on artificial intelligence, such as a plug-in hard disk equipped on the article subject extraction device 1 based on artificial intelligence, and a smart media card (Smart Media Card). , SMC), Secure Digital (SD) card, Flash Card, etc. Further, the memory 11 may also include both an internal storage unit of the article subject extraction device 1 based on artificial intelligence and an external storage device. The memory 11 can be used not only to store application software and various data installed in the artificial intelligence-based article subject extraction device 1, such as the code of the artificial intelligence-based article subject extraction program 01, etc., but also to temporarily store the output or The data to be output.

In some embodiments, the processor 12 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip, for running program codes or processing stored in the memory 11 Data, such as the implementation of the article subject extraction program 01 based on artificial intelligence.

The communication bus 13 is used to realize the connection and communication between these components.

The network interface 14 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface), and is usually used to establish a communication connection between the apparatus 1 and other electronic devices.

Optionally, the device 1 may also include a user interface. The user interface may include a display (Display) and an input unit such as a keyboard (Keyboard). The optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display the information processed in the artificial intelligence-based article subject extraction device 1 and to display a visualized user interface.

Figure 2 only shows an artificial intelligence-based article subject extraction device 1 with components 11-14 and an artificial intelligence-based article subject extraction program 01. Those skilled in the art can understand that the structure shown in Figure 1 does not constitute The limitation on the article subject extraction device 1 based on artificial intelligence may include fewer or more components than shown, or a combination of certain components, or a different component arrangement.

In the embodiment of the device 1 shown in FIG. 2, the memory 11 stores an artificial intelligence-based article subject extraction program 01; when the processor 12 executes the artificial intelligence-based article subject extraction program 01 stored in the memory 11, the following steps are implemented:

Step 1: Receive a text data set, and perform operations including word segmentation and merging on the text data set to obtain a word text set.

Further, since the subject of multiple sentences may be the same, the merging is to merge multiple sentences with the same subject to achieve the purpose of greatly reducing the words in the text data set. Preferably, the merging includes: traversing each text in the text data set, dividing the text according to paragraphs to obtain several paragraphs, and presetting words that appear more than twice in each paragraph as hypothetical subjects, and constructing Constructing a log-likelihood function based on the conditional probability model of each sentence in each paragraph and the hypothetical subject, and optimizing the conditional probability model based on the log-likelihood function to obtain the subject of each sentence, Several sentences with the same subject are merged into one sentence, and the merge operation is completed.

Specifically, the conditional probability model is:

Preferably, the log likelihood function is:

Step 2: Perform an encoding operation on the word text set and turn it into a word matrix set, and input the word matrix set into a word vector conversion model for training to obtain a word vector set.

Preferably, the encoding adopts a one-hot encoding form, and the one-hot encoding is to first number each word in the word text set to obtain the largest number, and then create the largest number. Encoding matrices with the same numerical numbering dimension, traverse each sentence in the word text set in turn, map each sentence to the encoding matrix, and according to the numerical number of each word in the word text set Complete the encoding operation to obtain the word matrix set. For example, the collection of words and texts is: when people know how to exchange with the system, they can tell their true self and the truth. This is reality. After the text is numbered, it is: when ¹ person ² understands the exchange of ³ and ⁴ system ⁵ ^{, 6} hours ⁷ , they ⁸ can ⁹ make ^{10 the} real ¹¹ ^{themselves 12} and tell them ¹³ , this is ¹⁴ reality ¹⁵ , and get the biggest The number is 15, and then a 15-dimensional coding matrix is created. Further, if the traversal sentence is: This is reality, then the coded is [0,0,0,0,0,0,0,0,0,0 ,0,0,0,1,1].

Specifically, the weight relationship is:

d={(t ₁ ,w ₁ ),(t ₂ ,w ₂ ),……,(t _i ,w _i ),……,(t _n ,w _n )}

Further, the calculation method of the weight is:

Step 3: After performing the dimensionality reduction operation on the word vector set, input the training value to the convolutional neural network model for training, and determine the size of the training value and the preset threshold, if the training value is greater than the preset threshold , The convolutional neural network model continues to be trained, and if the training value is less than the preset threshold, the convolutional neural network model completes the training.

Further, the covariance is:

The least square method L(s) is:

Step 4: Receive text data input by the user, convert the text data input by the user into a word vector, and input it into the trained convolutional neural network model to obtain and output the subject of the article.

Optionally, in other embodiments, the artificial intelligence-based article subject extraction program can also be divided into one or more modules, and the one or more modules are stored in the memory 11 and run by one or more processors ( This embodiment is executed by the processor 12) to complete the application. The module referred to in this application refers to a series of computer program instruction segments that can complete specific functions, and is used to describe the article subject extraction program based on artificial intelligence. The execution process of the article subject extraction device.

For example, referring to FIG. 3, a schematic diagram of program modules of an artificial intelligence-based article subject extraction program in an embodiment of an artificial intelligence-based article subject extraction device of this application. In this embodiment, the artificial intelligence-based article subject The extraction program can be divided into a data receiving module 10, a word vector solving module 20, a model training module 30, and an article subject output module 40. Illustratively:

The data receiving module 10 is used for receiving a text data set, and performing operations including word segmentation and merging on the text data set to obtain a word text set.

The word vector solving module 20 is configured to: perform an encoding operation on the word text set and convert it into a word matrix set, and input the word matrix set into a word vector conversion model for training to obtain a word vector set.

The model training module 30 is configured to: perform a dimensionality reduction operation on the word vector set and input it into a convolutional neural network model for training to obtain a training value, and determine the size of the training value and a preset threshold. If the training value is If the training value is greater than the preset threshold, the convolutional neural network model continues training, and if the training value is less than the preset threshold, the convolutional neural network model completes training.

The article subject output module 40 is configured to receive text data input by a user, convert the text data input by the user into a word vector and input it into the trained convolutional neural network model to obtain and output the article subject.

The functions or operation steps implemented by the program modules such as the data receiving module 10, the word vector solving module 20, the model training module 30, and the article subject output module 40 when executed are substantially the same as those in the foregoing embodiment, and will not be repeated here.

In addition, the embodiment of the present application also proposes a computer-readable storage medium, the computer-readable storage medium stores an artificial intelligence-based article subject extraction program, and the artificial intelligence-based article subject extraction program can be used by one or more Each processor executes to achieve the following operations:

A text data set is received, and operations including word segmentation and merging are performed on the text data set to obtain a word text set.

The word text set is converted into a word matrix set after an encoding operation, and the word matrix set is input into a word vector conversion model for training to obtain a word vector set.

After performing the dimensionality reduction operation on the word vector set, input it into a convolutional neural network model to obtain training values, and determine the size of the training value and a preset threshold. If the training value is greater than the preset threshold, the The convolutional neural network model continues to be trained, and if the training value is less than the preset threshold, the convolutional neural network model completes the training.

The text data input by the user is received, and the text data input by the user is converted into a word vector and then input into the trained convolutional neural network model to obtain and output the subject matter of the article.

It should be noted that the serial numbers of the foregoing embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. And the terms "include", "include" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements, but also includes those elements that are not explicitly included. The other elements listed may also include elements inherent to the process, device, article, or method. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, device, article, or method that includes the element.

Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disk, optical disk), including a number of instructions to make a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the method described in each embodiment of the present application.

The above are only the preferred embodiments of the application, and do not limit the scope of the patent for this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of the application, or directly or indirectly applied to other related technical fields , The same reason is included in the scope of patent protection of this application.

Claims

An article subject extraction method based on artificial intelligence, characterized in that the method includes:

Receiving a text data set, and performing operations including word segmentation and merging on the text data set to obtain a word text set;

After performing an encoding operation on the word text set, it is converted into a word matrix set, and the word matrix set is input into a word vector conversion model for training to obtain a word vector set;

After performing the dimensionality reduction operation on the word vector set, input it into a convolutional neural network model to obtain training values, and determine the size of the training value and a preset threshold. If the training value is greater than the preset threshold, the The convolutional neural network model continues to be trained, and if the training value is less than the preset threshold, the convolutional neural network model completes the training;

The text data input by the user is received, and the text data input by the user is converted into a word vector and then input into the trained convolutional neural network model to obtain and output the subject matter of the article.
The method for extracting the subject of an article based on artificial intelligence according to claim 1, wherein the merging operation comprises:

Traverse each text data in the text data set, and divide the text data according to paragraphs to obtain several paragraphs;

Presupposing words that appear more than twice in the plurality of paragraphs as hypothetical subjects, and constructing a conditional probability model of each sentence in the plurality of paragraphs and the hypothetical subject;

Construct a log-likelihood function, optimize the conditional probability model based on the log-likelihood function to obtain the subject of each sentence, merge several sentences with the same subject into one sentence, and complete the merge operation.
3. The method for extracting the subject of an article based on artificial intelligence according to claim 2, wherein the conditional probability model is:

Among them, y 1 ,..., y N , y i are the hypothetical subjects, N is the number of the hypothetical subjects, D is the paragraph, j is the number of the paragraph, and s is the sentence in the paragraph , P(y i |s) is the probability of assuming that the subject y i is the subject of the sentence s, and s(i, y i ) indicates that the hypothetical subject of the sentence i is y i .
The method for extracting the subject matter of an article based on artificial intelligence according to claim 1, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating an encoding matrix with the same dimension as the largest number number, traversing the sentences in the word text set in turn, and mapping all the sentences to the encoding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
5. The method for extracting the subject matter of an article based on artificial intelligence according to claim 2, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating a coding matrix with the same dimension as the largest number number, sequentially traversing the sentences in the word text set, and mapping all the sentences to the coding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
The method for extracting the subject of an article based on artificial intelligence according to claim 3, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating an encoding matrix with the same dimension as the largest number number, traversing the sentences in the word text set in turn, and mapping all the sentences to the encoding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
The artificial intelligence-based article subject extraction method according to any one of claims 4-6, wherein the dimensionality reduction operation comprises:

Calculating the covariance of each word vector in the word vector set;

The word vectors whose absolute value is greater than the preset covariance threshold in the covariance are removed to obtain the word vector set after dimensionality reduction.
An artificial intelligence-based article subject extraction device, characterized in that the device includes a memory and a processor, and an artificial intelligence-based article subject extraction program that can be run on the processor is stored on the memory. When the artificial intelligence-based article subject extraction program is executed by the processor, the following steps are implemented:

Receiving a text data set, and performing operations including word segmentation and merging on the text data set to obtain a word text set;

After performing an encoding operation on the word text set, it is converted into a word matrix set, and the word matrix set is input into a word vector conversion model for training to obtain a word vector set;

After performing the dimensionality reduction operation on the word vector set, input it into a convolutional neural network model to obtain training values, and determine the size of the training value and a preset threshold. If the training value is greater than the preset threshold, the The convolutional neural network model continues to be trained, and if the training value is less than the preset threshold, the convolutional neural network model completes the training;

The text data input by the user is received, and the text data input by the user is converted into a word vector and then input into the trained convolutional neural network model to obtain and output the main idea of the article.
8. The artificial intelligence-based article subject extraction device according to claim 8, wherein the merging operation comprises:

Traverse each text data in the text data set, and divide the text data according to paragraphs to obtain several paragraphs;

Presupposing words that appear more than twice in the plurality of paragraphs as hypothetical subjects, and constructing a conditional probability model of each sentence in the plurality of paragraphs and the hypothetical subject;

Construct a log-likelihood function, optimize the conditional probability model based on the log-likelihood function to obtain the subject of each sentence, merge several sentences with the same subject into one sentence, and complete the merge operation.
The article subject extraction device based on artificial intelligence according to claim 9, characterized in that the conditional probability model is:

Among them, y 1 ,..., y N , y i are the hypothetical subjects, N is the number of the hypothetical subjects, D is the paragraph, j is the number of the paragraph, and s is the sentence in the paragraph , P(y i |s) is the probability of assuming that the subject y i is the subject of the sentence s, and s(i, y i ) indicates that the hypothetical subject of the sentence i is y i .
8. The artificial intelligence-based article subject extraction device according to claim 8, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating an encoding matrix with the same dimension as the largest number number, traversing the sentences in the word text set in turn, and mapping all the sentences to the encoding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
9. The artificial intelligence-based article subject extraction device according to claim 9, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating an encoding matrix with the same dimension as the largest number number, traversing the sentences in the word text set in turn, and mapping all the sentences to the encoding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
10. The artificial intelligence-based article subject extraction device according to claim 10, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating a coding matrix with the same dimension as the largest number number, sequentially traversing the sentences in the word text set, and mapping all the sentences to the coding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
The artificial intelligence-based article subject extraction device according to any one of claims 11-13, wherein the dimensionality reduction operation comprises:

Calculating the covariance of each word vector in the word vector set;

The word vectors whose absolute value is greater than the preset covariance threshold in the covariance are removed to obtain the word vector set after dimensionality reduction.
A computer-readable storage medium, characterized in that an artificial intelligence-based article subject extraction program is stored on the computer-readable storage medium, and the artificial intelligence-based article subject extraction program can be executed by one or more processors To achieve the following steps:

Receiving a text data set, and performing operations including word segmentation and merging on the text data set to obtain a word text set;

After performing an encoding operation on the word text set, it is converted into a word matrix set, and the word matrix set is input into a word vector conversion model for training to obtain a word vector set;

After performing the dimensionality reduction operation on the word vector set, input it into a convolutional neural network model to obtain training values, and determine the size of the training value and a preset threshold. If the training value is greater than the preset threshold, the The convolutional neural network model continues to be trained, and if the training value is less than the preset threshold, the convolutional neural network model completes the training;

The text data input by the user is received, and the text data input by the user is converted into a word vector and then input into the trained convolutional neural network model to obtain and output the main idea of the article.
15. The computer-readable storage medium of claim 15, wherein the merging operation comprises:

Traverse each text data in the text data set, and divide the text data according to paragraphs to obtain several paragraphs;

Presupposing words that appear more than twice in the plurality of paragraphs as hypothetical subjects, and constructing a conditional probability model of each sentence in the plurality of paragraphs and the hypothetical subject;

Construct a log-likelihood function, optimize the conditional probability model based on the log-likelihood function to obtain the subject of each sentence, merge several sentences with the same subject into one sentence, and complete the merge operation.
15. The computer-readable storage medium of claim 16, wherein the conditional probability model is:

Among them, y 1 ,..., y N , y i are the hypothetical subjects, N is the number of the hypothetical subjects, D is the paragraph, j is the number of the paragraph, and s is the sentence in the paragraph , P(y i |s) is the probability of assuming that the subject y i is the subject of the sentence s, and s(i, y i ) indicates that the hypothetical subject of the sentence i is y i .
15. The computer-readable storage medium of claim 15, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating a coding matrix with the same dimension as the largest number number, sequentially traversing the sentences in the word text set, and mapping all the sentences to the coding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
The computer-readable storage medium according to claim 16 or 17, wherein the encoding operation comprises:

Digitally number each word in the word text set and obtain the largest digital number;

Creating a coding matrix with the same dimension as the largest number number, sequentially traversing the sentences in the word text set, and mapping all the sentences to the coding matrix;

The encoding matrix is processed according to the digital number of each word in the word text set to obtain a word matrix set.
The computer-readable storage medium of claim 19, wherein the dimensionality reduction operation comprises:

Calculating the covariance of each word vector in the word vector set;

The word vectors whose absolute value is greater than the preset covariance threshold in the covariance are removed to obtain the word vector set after dimensionality reduction.