CN113792119A - Article originality evaluation system, method, device and medium - Google Patents

Abstract

The present disclosure relates to an article originality evaluation system, method, device and medium, wherein the system includes: a text data preprocessing module, an inventory document management subsystem, including a document storage module, and a similar document candidate submodule ES, The semantically similar document candidate submodule Milvus, the feature storage submodule Mongo, and the article originality calculation subsystem, wherein the article originality calculation subsystem is specifically composed of a candidate similar document retrieval module and an originality calculation module; the text data pre- The processing module is between the article originality calculation subsystem and the inventory document management subsystem; the semantically similar document candidate submodule ES and the semantically similar document candidate submodule Milvus are between the article originality calculation The subsystem and the inventory document management subsystem are respectively connected with the document storage module and the candidate similar document retrieval module.

Description

An article originality evaluation system, method, equipment and medium

technical field

The present disclosure relates to the technical field of data processing, and more particularly, to an article originality evaluation system, method, device and medium.

Background technique

The rapid expansion of the scale of data held by humans is accompanied by the existence of a large number of similar data. In some scenarios, we need to measure the originality of a document to decide how to process the document. For example, academic journals will only consider whether to accept submissions after initially confirming the originality of a manuscript through "duplication checking"; there are a large number of plagiarism, reprinting and other phenomena on the Internet, which need to be efficiently discovered based on originality calculation tools .

The current originality calculation tools are mainly based on string matching to measure the similarity between two documents, and the processing ability of "manufacturing" is low.

SUMMARY OF THE INVENTION

In order to solve the technical problem that the article originality evaluation method in the prior art cannot meet the needs of users.

In order to achieve the above-mentioned technical purpose, the present disclosure provides a method for evaluating the originality of an article, which is characterized in that it includes:

Preprocess the documents to be stored and store them in the database;

Perform candidate processing of similar documents in literal meaning, candidate processing of documents in similar literal meaning, and/or feature extraction and storage on the newly entered documents;

There may be similar documents in the inventory document and the document to be evaluated in the recalled document library;

Based on the degree of similarity between the document to be assessed and the inventory document, the originality of the document to be assessed is calculated, wherein the inventory document is a document whose content is highly original in a business scenario and is determined to require intellectual property protection. Inventory documents are stored in document repositories.

Further, it is characterized in that the preprocessing of the documents to be stored specifically includes:

Perform document cleaning and feature extraction on documents;

Calculate the word features and distributed representation of the document to be evaluated, that is, the document to be evaluated is divided into N paragraphs, and the obtained paragraph set paras,

paras=(p ₁ , p ₂ ,..., p _n ,..., p _N ), where p _n , n=1, 2... N represents the segmented document paragraph, and N is greater than or equal to 2 the integer.

Further, it is characterized in that the recalling of the inventory documents in the document library and the documents to be evaluated may have similar documents specifically includes:

_{Cand_list wordbag} = ( _{c 1} , c ₂ , . . . c _i , _. , i=1, 2,...I, I is an integer greater than or equal to 2;

The set of semantic candidate similar paragraphs of paragraph _pn obtained by tag retrieval is cand_list _distvec =(d ₁ , d ₂ , ... d _j , ... d _J ); wherein, d _j represents the semantic similarity retrieved by paragraph _pn Paragraph, j=1, 2,...J, J is an integer greater than or equal to 2;

Use a threshold to determine whether the semantic candidate similar paragraph set cand_list _wordbag and the semantic candidate similar paragraph set cand_list _distvec are recalled, and record all the recalled candidate similar paragraph sets:

cand _list =cand_list _wordbag ∪cand_list _distvec =(s ₁ , s ₂ ,...,s _k ,...,s _K );

Wherein, k=1, 2, ..., K, K represents an integer greater than or equal to 2.

Further, the calculation of the originality of the document to be evaluated specifically includes:

Use the following formula to calculate the originality of the article;

式中score_n为待评估文档的第n段文本的原创度；Article originality

In the formula, score _n is the originality of the n-th text of the document to be evaluated;

in,

score _n = min(score_wordbagn, score_distvec _n ),

Among them, score_wordbag _n is the originality score of the nth article paragraph under the word bag model, and score_distvec _n is the originality score of the nth article paragraph under the distributed representation.

Further, the originality score under the bag-of-words model is calculated by the following formula:

in,

是段落pn内词语的集合，

表示段落p_n和s_k包含的相同词语的个数；分母中

表示两个段落长度的差距绝对值；系数β表示在计算文本相似度时，文档长度差异因素的权重，默认为0.5。In the formula,

is the set of words in paragraph pn,

Indicates the number of identical words contained in paragraphs p _n and _sk ; in the denominator

Represents the absolute value of the difference between the lengths of two paragraphs; the coefficient β represents the weight of the document length difference factor when calculating the text similarity, and the default is 0.5.

Further, the originality score under the distributed representation is calculated by the following formula:

in,

式中

为第n段文本的分布式表示。cosine distance

in the formula

is the distributed representation of the nth paragraph of text.

To achieve the above technical purpose, the present disclosure can also provide a method for evaluating the originality of an article, including:

Utilize the text data preprocessing module to perform document cleaning and feature extraction on the document to be evaluated, and calculate word features and distributed representations of the document to be evaluated;

Using the candidate similar document retrieval module to recall the stock documents in the document library and the documents to be evaluated may have similar documents;

Using the originality calculation module to calculate the originality of the document to be assessed based on the similarity between the document to be assessed and the inventory documents in the document library.

Further, it also includes:

Utilize the text data preprocessing module to perform document cleaning and feature extraction on the documents to be stored;

The document storage module is used to store the preprocessed document data to be stored in the corresponding databases: the semantically similar document candidate submodule ES, the semantically similar document candidate submodule Milvus, and the feature storage submodule Mongo.

In order to achieve the above technical purpose, the present disclosure can also provide a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, is used to implement the steps of the above-mentioned article originality evaluation method.

In order to achieve the above technical purpose, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the above-mentioned article originality evaluation is realized when the processor executes the computer program. steps of the method.

The beneficial effects of the present disclosure are:

The main calculation of the article originality evaluation system of the present disclosure is completed offline, and the reasoning speed is very fast, and the article originality evaluation system of the present disclosure can support real-time calculation.

The article originality evaluation system of the present disclosure takes into account both literal and semantic dimensions, and can effectively deal with the situation of "washing manuscripts".

Description of drawings

FIG. 1 shows a schematic structural diagram of the system of Embodiment 1 of the present disclosure;

2 shows a schematic structural diagram of a text data preprocessing module of the system according to Embodiment 1 of the present disclosure;

3 shows a schematic flowchart of the method of Embodiment 2 of the present disclosure;

FIG. 4 shows a schematic structural diagram of Embodiment 4 of the present disclosure.

Detailed ways

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that these descriptions are exemplary only, and are not intended to limit the scope of the present disclosure. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concepts of the present disclosure.

Various structural schematic diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. The figures are not to scale, some details have been exaggerated for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the figures, as well as the relative sizes and positional relationships between them are only exemplary, and may vary in practice due to manufacturing tolerances or technical limitations, and those skilled in the art will It needs to be possible to additionally design regions/layers with different shapes, sizes, relative positions.

Example 1:

As shown in Figure 1:

The present disclosure provides an article originality evaluation system, including:

Text data preprocessing module for preprocessing documents;

An inventory document management subsystem, including a document storage module, is used to maintain a document library, where inventory documents are stored in the document library, wherein the inventory documents are those whose content is highly original in business scenarios and are deemed to be required Documentation for intellectual property protection;

Literally similar document candidate sub-module ES, which is used to provide literally similar candidate similar documents;

The semantically similar document candidate submodule Milvus is used to provide semantically similar candidate similar documents;

The feature storage sub-module Mongo is used to store all feature data of the document;

The document warehousing module is used to store document data in the semantically similar document candidate submodule ES, the semantically similar document candidate submodule Milvus, and the feature storage submodule Mongo;

The article originality calculation subsystem is used to calculate the originality of the evaluation article;

Wherein, the article originality calculation subsystem is specifically composed of a candidate similar document retrieval module and an originality calculation module;

The candidate similar document retrieval module is used to recall the stock documents in the document library and the documents to be evaluated that may have similar documents;

The originality calculation module is used to calculate the originality of the document to be assessed based on the similarity between the document to be assessed and the inventory documents in the document library;

The text data preprocessing module is located between the article originality calculation subsystem and the inventory document management subsystem;

The semantically similar document candidate submodule ES and the semantically similar document candidate submodule Milvus are located between the article originality calculation subsystem and the inventory document management subsystem, and are respectively connected with the document storage module and the document storage module. The candidate similar document retrieval module is connected;

The feature storage sub-module Mongo is located between the article originality calculation subsystem and the inventory document management subsystem, and is respectively connected with the document storage module and the originality calculation module.

as shown in picture 2,

Further, the text data preprocessing module is specifically used for:

Perform document cleaning and feature extraction on documents;

That is, the document to be evaluated is divided into N paragraphs, and the obtained paragraph set paras,

paras=(p ₁ , p ₂ ,..., p _n ,..., p _N ), where p _n , n=1, 2... N represents the segmented document paragraph, and N is greater than or equal to 2 the integer.

Calculate the word features and distributed representation of the document to be evaluated:

Perform word segmentation and stop word removal processing on the segmented documents to obtain word features, that is, word bag model;

Sentence vector calculation is performed on the segmented document to obtain a distributed representation.

Further, the candidate similar document retrieval module is specifically used for:

Mark the word-sense candidate similar paragraph set of paragraph _pn retrieved from the word-sense similar document candidate sub-module ES as _{cand_list wordbag} =(c ₁ , c ₂ ,...ci ,...c _I ); wherein, c _i represents a paragraph with similar meanings retrieved by paragraph p _n , i=1, 2, ...... I, I is an integer greater than or equal to 2;

Mark the semantic candidate similar paragraph set of paragraph _pn retrieved from the semantically similar document candidate sub-module Milvus as cand_list _distvec =(d ₁ , d ₂ ,...d _j ,...d _J ); wherein, d _j represents the semantically similar paragraph retrieved by paragraph _pn , j=1, 2,...J, J is an integer greater than or equal to 2;

Use a threshold to determine whether the semantic candidate similar paragraph set cand_list _wordbag and the semantic candidate similar paragraph set cand_list _distvec are recalled, and record all the recalled candidate similar paragraph sets:

cand _list =cand_list _wordbag ∪cand_list _distvec =(s ₁ , s ₂ ,...,s _k ,...,s _K );

Wherein, k=1, 2, ..., K, K represents an integer greater than or equal to 2.

Further, the originality calculation module is specifically used for:

Use the following formula to calculate the originality of the article;

式中score_n为待评估文档的第n段文本的原创度；Article originality

In the formula, score _n is the originality of the n-th text of the document to be evaluated;

in,

score _n = min(score_wordbagn, score_distvec _n ),

Among them, score_wordbag _n is the originality score of the nth article paragraph under the word bag model, and score_distvec _n is the originality score of the nth article paragraph under the distributed representation.

Further, the originality score under the bag-of-words model is calculated by the following formula:

in,

是段落p_n内词语的集合，

表示段落p_n和s_k包含的相同词语的个数；分母中

表示两个段落长度的差距绝对值；系数β表示在计算文本相似度时，文档长度差异因素的权重，默认为0.5。In the formula,

is the set of words in paragraph p _n ,

Indicates the number of identical words contained in paragraphs p _n and _sk ; in the denominator

Represents the absolute value of the difference between the lengths of two paragraphs; the coefficient β represents the weight of the document length difference factor when calculating the text similarity, and the default is 0.5.

Further, the originality score under the distributed representation is calculated by the following formula:

in,

式中

为第n段文本的分布式表示。cosine distance

in the formula

is the distributed representation of the nth paragraph of text.

The main calculation of the article originality evaluation system of the present disclosure is completed offline, and the reasoning speed is very fast, and the article originality evaluation system of the present disclosure can support real-time calculation.

The article originality evaluation system of the present disclosure takes into account both literal and semantic dimensions, and can effectively deal with the situation of "washing manuscripts".

Embodiment 2:

As shown in Figure 3,

The present disclosure can also provide an article originality evaluation method, including:

S201: Use the text data preprocessing module to perform document cleaning and feature extraction on the document to be evaluated, and calculate word features and distributed representations of the document to be evaluated;

specifically,

Mark the word-sense candidate similar paragraph set of paragraph _pn retrieved from the word-sense similar document candidate sub-module ES as _{cand_list wordbag} =(c ₁ , c ₂ ,...ci ,...c _I ); wherein, c _i represents a paragraph with similar meanings retrieved by paragraph p _n , i=1, 2, ...... I, I is an integer greater than or equal to 2;

Mark the semantic candidate similar paragraph set of paragraph _pn retrieved from the semantically similar document candidate sub-module MilVus as cand_list _distvec =(d ₁ , d ₂ ,...d _j ,...d _J ); wherein, d _j represents the semantically similar paragraph retrieved by paragraph _pn , j=1, 2,...J, J is an integer greater than or equal to 2;

S202: Use the candidate similar document retrieval module to recall that the inventory document in the document library and the document to be evaluated may have similar documents;

specifically,

Use a threshold to determine whether the semantic candidate similar paragraph set cand_list _wordbag and the semantic candidate similar paragraph set cand_list _distvec are recalled, and record all the recalled candidate similar paragraph sets:

cand _list =cand_list _wordbag ∪cand_list _distvec =(s ₁ , s ₂ ,...,s _k ,...,s _K );

Among them, k=1, 2,..., K, K represents an integer greater than or equal to 2;

S203: Calculate the originality of the document to be assessed based on the similarity between the document to be assessed and the inventory document in the document library by using the originality calculation module.

specifically,

Use the following formula to calculate the originality of the article;

式中score_n为待评估文档的第n段文本的原创度；Article originality

In the formula, score _n is the originality of the n-th text of the document to be evaluated;

in,

score _n = min(score_wordbag _n , score_distvec _n ),

Among them, score_wordbag _n is the originality score of the nth article paragraph under the word bag model, and score_distvec _n is the originality score of the nth article paragraph under the distributed representation.

Further, the originality score under the bag-of-words model is calculated by the following formula:

in,

是段落p_n内词语的集合，

表示段落p_n和s_k包含的相同词语的个数；分母中

表示两个段落长度的差距绝对值；系数β表示在计算文本相似度时，文档长度差异因素的权重，默认为0.5。In the formula,

is the set of words in paragraph p _n ,

Indicates the number of identical words contained in paragraphs p _n and _sk ; in the denominator

Represents the absolute value of the difference between the lengths of two paragraphs; the coefficient β represents the weight of the document length difference factor when calculating the text similarity, and the default is 0.5.

Further, the originality score under the distributed representation is calculated by the following formula:

in,

式中

为第n段文本的分布式表示。cosine distance

in the formula

is the distributed representation of the nth paragraph of text.

Further, it also includes:

Utilize the text data preprocessing module to perform document cleaning and feature extraction on the documents to be stored;

The document storage module is used to store the preprocessed document data to be stored in the corresponding databases: the semantically similar document candidate submodule ES, the semantically similar document candidate submodule Milvus, and the feature storage submodule Mongo.

Embodiment three:

The present disclosure can also provide a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, is used to implement the steps of the above-mentioned article originality evaluation system.

The computer storage medium of the present disclosure may be implemented using semiconductor memory, magnetic core memory, magnetic drum memory, or magnetic disk memory.

Semiconductor memory, the semiconductor memory elements mainly used in computers are mainly Mos and bipolar. Mos components have high integration, simple process but slow speed. Bipolar components have complex process, high power consumption, low integration but fast speed. After the advent of NMos and CMos, Mos memory began to occupy a dominant position in semiconductor memory. NMos is fast, such as Intel's 1K-bit SRAM access time of 45ns. The CMos consumes less power, and the 4K-bit CMos static memory access time is 300ns. The above semiconductor memories are all random access memories (RAM), that is, new contents can be read and written randomly during the working process. The semiconductor read-only memory (ROM) can be randomly read but not written in the working process, it is used to store the solidified program and data. ROM is divided into two types of non-rewritable fuse-type read-only memory ─ ─ PROM and rewritable read-only memory EPROM.

Magnetic core memory has the characteristics of low cost and high reliability, and has more than 20 years of practical experience. Before the mid-1970s, magnetic core memory was widely used as main memory. Its storage capacity can reach more than 10 bits, and the fastest access time is 300ns. The international typical magnetic core memory capacity is 4MS ~ 8MB, and the access cycle is 1.0 ~ 1.5μs. After the rapid development of semiconductor storage has replaced the magnetic core memory as the main memory, the magnetic core memory can still be applied as a large-capacity expansion memory.

Drum memory, an external memory for magnetic recording. Because of its fast information access speed and stable and reliable work, although its capacity is small, it is gradually being replaced by disk memory, but it is still used as external memory for real-time process control computers and medium and large computers. In order to meet the needs of small and microcomputers, ultra-small magnetic drums have appeared, which are small in size, light in weight, high in reliability and easy to use.

Disk storage, a type of external storage for magnetic recording. It has the advantages of both the magnetic drum and the magnetic tape memory, that is, its storage capacity is larger than that of the magnetic drum, and the access speed is faster than that of the magnetic tape memory, and it can be stored offline, so the magnetic disk is widely used in various computer systems. capacity of external memory. Disks are generally divided into two categories: hard disks and floppy disks.

There are many varieties of hard disk storage. Structurally, there are two types: replaceable and fixed. Replaceable disk platters are replaceable, while fixed disk platters are fixed. Both replaceable and fixed disks have multi-chip and single-chip structures, and can be divided into fixed-head type and movable-head type. The fixed-head type magnetic disk has a small capacity, low recording density and high access speed, but high cost. The movable head type disk has a high recording density (up to 1000-6250 bits/inch), so the capacity is large, but the access speed is lower than that of the fixed head disk. The storage capacity of magnetic disk products can reach hundreds of megabytes, the bit density is 6250 bits per inch, and the track density is 475 tracks per inch. Among them, the multi-chip replaceable disk storage has a large off-body capacity because the disk group can be replaced, and has a large capacity and high speed, and can store large-capacity information materials. It is widely used in online information retrieval systems and database management systems.

Embodiment 4:

The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the above steps of the article originality evaluation system are implemented.

FIG. 4 is a schematic diagram of the internal structure of an electronic device in one embodiment. As shown in FIG. 4 , the electronic device includes a processor, a storage medium, a memory, and a network interface connected through a system bus. Wherein, the storage medium of the computer device stores an operating system, a database and computer-readable instructions, and the database can store a sequence of control information. When the computer-readable instructions are executed by the processor, the processor can achieve a degree of originality of the article. evaluation system. The processor of the electrical equipment is used to provide computing and control capabilities and support the operation of the entire computer equipment. Computer-readable instructions may be stored in the memory of the computer device, and when executed by the processor, the computer-readable instructions may cause the processor to execute an article originality evaluation system. The network interface of the computer equipment is used for communication with the terminal connection. Those skilled in the art can understand that the structure shown in FIG. 4 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

The electronic devices include, but are not limited to, smart phones, computers, tablet computers, wearable smart devices, artificial intelligence devices, power banks, and the like.

In some embodiments, the processor may be composed of integrated circuits, such as a single packaged integrated circuit, or a plurality of integrated circuits packaged with the same function or different functions, including one or more central A processor (Central Processing Unit, CPU), a microprocessor, a digital processing chip, a graphics processor and a combination of various control chips, etc. The processor is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, by running or executing the program or module stored in the memory (for example, executing a remote control unit). data reading and writing programs, etc.), and call data stored in the memory to perform various functions of the electronic device and process data.

The bus may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (extended industry standard architecture, EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to enable connection communication between the memory and at least one processor or the like.

FIG. 4 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 4 does not constitute a limitation on the electronic device, and may include fewer or more components than those shown in the drawings. , or a combination of certain components, or a different arrangement of components.

For example, although not shown, the electronic device may also include a power source (such as a battery) for powering the various components, preferably, the power source may be logically connected to the at least one processor through a power management device, so as to be implemented by the power management device Charge management, discharge management, and power management functions. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device may also include a network interface, optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.) Establish a communication connection between other electronic devices.

Optionally, the electronic device may further include a user interface, and the user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display may also be appropriately called a display screen or a display unit, which is used for displaying information processed in the electronic device and for displaying a visual user interface.

Further, the computer usable storage medium may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, and the like; using the created data, etc.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

Embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only, and are not intended to limit the scope of the present disclosure. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims (10)

1. An article originality evaluation method is characterized by comprising the following steps:

preprocessing the document to be put in storage and storing the preprocessed document in storage;

performing word meaning similar document candidate processing, word meaning similar document candidate processing and/or feature extraction and storage on the newly-put documents;

recalling similar documents possibly existing in the stock documents and the documents to be evaluated in the document library;

and calculating the originality of the document to be evaluated based on the similarity of the document to be evaluated and the stock document, wherein the stock document is a document which has higher originality in the service scene and is determined to need intellectual property protection, and the stock document is stored in a document library.

2. The method of claim 1, wherein the text data preprocessing module is specifically configured to:

carrying out document cleaning and feature extraction on the document;

calculating to obtain word characteristics and distributed representation of the document to be evaluated, namely segmenting the document to be evaluated into N paragraphs to obtain a paragraph set paras,

paras＝(p₁，p₂，...，p_n，...，p_N) Wherein p is_nN is 1, 2 … … N represents a document paragraph after segmentation, and N is an integer of 2 or more.

3. The method according to claim 2, wherein the recalling that similar documents may exist in the inventory document and the document to be evaluated in the document library specifically comprises:

paragraph p obtained by label search_nThe word sense candidate similar segment set is cand _ list_wordbag＝(c₁，c₂，...c_i，...c_I) (ii) a Wherein, c_iRepresents paragraph p_nThe searched word sense similar paragraph, I is 1, 2, … … I, I is an integer greater than or equal to 2;

paragraph p obtained by label search_nThe semantic candidate similar segment set is cand _ list_distvec＝(d₁，d₂，...d_j，...d_J) (ii) a Wherein d is_jRepresents paragraph p_nThe searched semantically similar paragraphs, J is 1, 2, … … J, and J is an integer greater than or equal to 2;

a threshold is used to determine the word sense candidate similar segment set cand _ list_wordbagSimilar to semantic candidate paragraph set cand _ list_distvecAnd (4) whether to recall, recording all the recalled candidate similar section sets:

cand_list＝cand_list_wordbag∪cand_list_distvec＝(s₁，s₂，...，s_k，...，s_K)；

wherein K is 1, 2, … …, and K is an integer of 2 or more.

4. The method of claim 3, wherein the calculating the originality of the document to be evaluated comprises:

calculating the originality of the article by using the following formula;

degree of originality of article

In the formula score_nThe originality of the nth text of the document to be evaluated;

wherein,

score_n＝min(score-wordbag_n，score_distvec_n)，

wherein, score _ Wordbag_nIs the original creativity score of the nth article segment falling under the bag-of-words model, score _ distvec_nIs the originality score of the nth article paragraph under the distributed representation.

5. The method of claim 4, wherein the originality score under the bag of words model is calculated by the following formula:

wherein,

in the formula,

is a collection of words within a paragraph pn,

represents paragraph p_nAnd s_kThe number of the same words contained; in the denominator

Representing the absolute difference of the lengths of the two paragraphs; the coefficient β represents the weight of the document length difference factor in calculating the text similarity, and is 0.5 by default.

6. The method of claim 4, wherein the originality score under the distributed representation is calculated by:

wherein,

cosine distance

In the formula

Is a distributed representation of the nth text segment.

7. An article originality evaluation system is characterized by comprising:

the text data preprocessing module is used for preprocessing the document;

the system comprises an inventory document management subsystem and a document management module, wherein the inventory document management subsystem comprises a document storage module used for maintaining a document library, and inventory documents are stored in the document library, wherein the inventory documents are documents which have higher originality in service scenes and are identified to need intellectual property protection;

a word sense similar document candidate submodule ES for providing candidate similar documents which are similar in word;

the semantically similar document candidate submodule Milvus is used for providing semantically similar candidate similar documents;

the characteristic storage submodule Mongo is used for storing all characteristic data of the document;

the document storage module is used for storing document data into the semantic similar document candidate submodule ES, the semantic similar document candidate submodule Milvus and the feature storage submodule Mongo;

the article originality degree calculation operator system is used for calculating the originality degree of the evaluation article;

the article originality degree operator system is composed of a candidate similar document retrieval module and an originality degree calculation module;

the candidate similar document retrieval module is used for recalling the stock documents in the document library and the documents to be evaluated which may have similarity;

the originality degree calculation module is used for calculating the originality degree of the document to be evaluated based on the similarity degree of the document to be evaluated and the document stored in the document library;

the text data preprocessing module is arranged between the article originality calculating subsystem and the stock document management subsystem;

the word meaning similar document candidate submodule ES and the semantic similar document candidate submodule Milvus are arranged between the article originality measuring subsystem and the stock document management subsystem and are respectively connected with the document storage module and the candidate similar document retrieval module;

the characteristic storage submodule Mongo is arranged between the article originality degree computing subsystem and the stock document management subsystem and is respectively connected with the document storage module and the originality degree computing module.

8. The system of claim 7, wherein the text data preprocessing module is specifically configured to:

carrying out document cleaning and feature extraction on the document;

calculating to obtain word characteristics and distributed representation of the document to be evaluated, namely segmenting the document to be evaluated into N paragraphs to obtain a paragraph set paras,

paras＝(p₁，p₂，...，p_n，...，p_N) Wherein p is_nN is 1, 2 … … N represents a document paragraph after segmentation, and N is an integer of 2 or more.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and operable on the processor, wherein the processor implements the steps corresponding to the method for evaluating the originality of an article as claimed in any one of claims 1 to 6 when executing the computer program.

10. A computer storage medium having computer program instructions stored thereon, wherein the program instructions, when executed by a processor, are for implementing the steps corresponding to the article originality assessment method of any one of claims 1 to 6.

Images