Detailed Description
To further illustrate the technical means and effects of the present invention adopted to achieve the predetermined objects, the following detailed description of the embodiments of the system and method, the features and effects thereof according to the present invention will be made with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "an embodiment" does not necessarily mean the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
As shown in fig. 1, the paper plagiarism detection system (hereinafter referred to as system) of the present invention includes a material subsystem; a user subsystem; a suspected material extraction subsystem; the system comprises a comparison subsystem, a data processing subsystem and a data processing subsystem, wherein the material subsystem is used for preparing materials for plagiarism detection and comparison; the user subsystem is used for managing user login information by a user and determining the writing style of the user; the suspected material extracting subsystem is used for extracting suspected materials of the document to be identified from the comparison library; and the comparison subsystem is used for comparing the suspected material with the document to be identified to generate a comparison report.
According to an embodiment of the present invention, the material subsystem may further include: a comparison library; the word segmentation library comprises a synonym near-sense word library and a Chinese and foreign synonym word library; a word segmentation module; a word segmentation module; chinese and foreign language word group module; a word segmentation part-of-speech classification module; a phrase part-of-speech classification module; chinese and foreign language phrase part-of-speech classification module; a word segmentation characteristic value generation module; a word group characteristic value generation module; a Chinese and foreign language word segmentation characteristic value generation module; a word segmentation compact coefficient generation module; a word segmentation group compact coefficient generation module; a Chinese and foreign language word segmentation group compact coefficient generation module; a word segmentation compact coefficient feature vector generation module; a word group compact coefficient feature vector generation module; a Chinese and foreign language word segmentation group compact coefficient feature vector generation module; a word segmentation free vector dimension determining module; a word group free vector dimension determining module; a Chinese and foreign language word segmentation free vector dimension determining module; a word segmentation simplification vector dimension generation module; a word group simplifying vector dimension generating module; the Chinese and foreign language word group-dividing simplification vector dimension generation module; a word segmentation feature vector generation module; a word group feature vector generation module; and one or more Chinese and foreign language word segmentation feature vector generation modules.
According to an embodiment of the present invention, the user subsystem may further comprise: a user access mode detection module; a user detection mode determination module; a user writing style test module; a test picture text description characteristic value generation module; a test article text description characteristic value generation module; a test picture text description feature vector generation module; a test article text description feature vector generation module; a test picture reference feature vector generation module; a test article reference feature vector generation module; a user test picture text description characteristic value generation module; a user test picture text description feature vector generation module; a user picture writing style feature vector generation module; a text description characteristic value generation module of a user test article; a user test article text description feature vector generation module; the user article writing style feature vector generation module; a user writing style feature vector generation module; a feature value generating module of the document to be audited; a feature vector generation module for feature values of the document to be audited; a user writing style similarity calculation module; a user writing style judging module; one or more of the user writing style structure assistant word judging modules.
According to an embodiment of the present invention, the suspected material extracting subsystem may further include: a document word segmentation module to be identified; a document to be identified word group module; a Chinese and foreign language word group module of the document to be identified; the document word segmentation part-of-speech classification module to be identified; the document to be identified is divided into words and grouped into words; the system comprises a word classification module for Chinese and foreign language phrases in a document to be identified; a document word segmentation characteristic value generation module to be identified; a document to be identified word group characteristic value generation module; a module for generating characteristic values of Chinese and foreign language word groups of the document to be identified; a word segmentation compact coefficient generation module for the document to be identified; a document to be identified word group compactness coefficient generation module; a Chinese and foreign language word segmentation group compactness coefficient generation module in the document to be identified; a document word segmentation compact coefficient feature vector generation module to be identified; a document to be identified word group compact coefficient feature vector generation module; a module for generating the compact coefficient feature vector of Chinese and foreign language word segmentation groups in the document to be identified; the word segmentation free vector dimension determining module for the document to be identified; the document to be identified is divided into word groups and is subjected to free vector dimension determination; a module for determining the dimension of the free vector of the Chinese and foreign language word segmentation group in the document to be identified; the word segmentation and simplification vector dimension generation module for the document to be identified; the document to be identified is divided into word groups, and a vector dimension generating module is simplified; the Chinese and foreign language word group simplifying vector dimension generating module in the document to be identified; a document word segmentation feature vector generation module to be identified; a document to be identified word group feature vector generation module; a module for generating Chinese and foreign language word group feature vectors of the document to be identified; a document feature vector adjusting module to be identified; a material feature vector adjusting module; a common plagiarism identification similarity calculation module and an expanded plagiarism identification similarity calculation module; a similarity calculation module for multi-language plagiarism identification; a document to be identified compactness coefficient statistic module; a material compactness coefficient statistic module; a formula extraction module; a formula decomposition module; one or more of the compact coefficient suspect material extraction modules.
According to an embodiment of the present invention, the contrast subsystem may further comprise: a sliding window setting module; a sliding window comparison module and a comparison report generation module.
In one embodiment of the invention, the system includes a comparison library for containing material for use as comparison objects. The comparison library further comprises sub-libraries such as a book library, a theory library, a patent library, a formula library, a proverb custom language library, a proverb language library, a celebrity and speech library, a poem library and the like. The book library is used for recording and recording publicly published books; the paper library is used for collecting journal papers, meeting papers, academic papers and the like; patent libraries are used to contain patent publications and the like. When recording the material, it is necessary to further save the source of the material, such as the publication date, publication company, author, book number, etc. of the book; publication date of journal paper, corresponding to journal name, journal number, author, etc.; meeting name, meeting holding location, meeting holding date, author, etc. of the meeting paper; schools, graduation years, grade of academic, authors, etc. of academic papers; the material can be uniquely obtained by those skilled in the art based on the source information of the included material. Preferably, the materials included in the corpus are not limited to chinese materials, but further include foreign language materials. After the contrast library is established, maintenance is required to be performed regularly or irregularly, and newly added books, journal papers, conference papers, academic papers, patent publication documents and the like are supplemented. The proverb colloquial library is used for recording materials such as sentences, phrases and the like widely spread on the network or among the masses. The celebrity and famous language library is used for receiving and recording celebrity and famous language materials, and the poetry library is used for receiving and recording poems, words, songs, endowments and other materials. The purpose of further establishing a proverb custom language library, a celebrity language library, a poem library and the like in the comparison library is to further expand the material range serving as a comparison object from traditional books, thesis, patent documents and the like, and improve the comprehensiveness of plagiarism detection. Those skilled in the art will appreciate that the comparison library may further contain other types of material, which are not described in detail herein.
Preferably, the comparison library classifies the materials according to the fields to which the materials belong when the materials are recorded. According to an embodiment of the present invention, the domain identification may be a category in a chinese library taxonomy of 5 basic unit classes, 22 major classes, a mixed number combining chinese pinyin letters and arabic numerals, one letter representing one major class, the order of the major classes reflected in letter order, and a numeral behind the letter as a mark. For example, A1 represents marx, enges works, K6 represents history of oceania, and TN represents electronic technology or communication technology. For the development of the industrial technology, the double letters are adopted for the second-level category of the industrial technology. Those skilled in the art will recognize that other classification schemes may be used to perform domain identification on material.
Preferably, when the comparison library records the material, the recorded material is respectively indexed according to the manner of title, author, abstract and text. And establishing an association relation among the title, the author, the abstract and each part of the text of each material, namely obtaining the rest parts of the same material through any one part of the title, the author, the abstract and the text.
Preferably, when the comparison library receives and records the materials, the formula existing in the received and recorded materials is extracted and copied, and the formula library is established for independent storage. Each formula in the formula library establishes a relevant relationship with the extracted material, and the corresponding material full text can be obtained through the formulas in the formula library. According to an embodiment of the present invention, when a formula is included, the variable parameter, the dependent variable parameter, and the operation sign of the formula are extracted and stored. According to a specific embodiment of the invention, after respective variable parameters and dependent variable parameters of the formula are extracted, the specific meaning, dimension and value range of each parameter are further extracted and stored respectively. According to an embodiment of the present invention, after the operation symbol of the formula is extracted, the operator is further annotated with Chinese and foreign characters. In the formula library, each included formula stores the symbolic representation of the independent variable parameter and the dependent variable parameter corresponding to each included formula, the Chinese and foreign language expression of the specific meaning of each variable and dependent variable, the dimension and the value range, and Chinese and foreign language text annotations of the operator and the operator. The purpose of further establishing a formula library in the comparison library is to further expand the material range as a comparison object to formula comparison and improve the comprehensiveness of plagiarism detection. Those skilled in the art will appreciate that the comparison library may further extract other contents in the material, such as chemical formula, gene sequence, etc., which are not described herein.
According to an embodiment of the present invention, the comparison library is stored at different site locations in a distributed manner; when accessing the comparison library, a specific site can be selected for accessing according to the load conditions of different sites. Each site counts the number of the materials extracted from the comparison library in the current unit time period, wherein the number of the materials can be the number of the materials or the byte number of the materials; obtaining the average load capacity of the site; each site reports the average load of the site to the suspected material extraction subsystem periodically; when the suspected material extraction subsystem needs to extract materials from a comparison library for selecting suspected materials, selecting a site with the lowest average load according to the average load of the sites reported recently for access; wherein the unit time period is set by the system; can be selected to be 5 minutes, 10 minutes, 30 minutes or 60 minutes according to actual needs. According to an embodiment of the present invention, different sub-libraries in the contrast library may be stored at different site locations in a distributed manner; and respectively accessing according to the site positions stored in different sub-libraries when accessing the comparison library. And when the suspected material extraction subsystem needs to extract materials from the comparison library for selecting the suspected materials, different comparison sub-libraries are selected for access according to the belonged field or the belonged type of the materials to be extracted.
According to an embodiment of the present invention, the system comprises a thesaurus for receiving and recording the participles and the corresponding parts of speech. The word segmentation library is set by the system in advance, maintained regularly, added with new words and the like. Preferably, each participle in the participle library is uniquely numbered, and the unique number of a participle in the participle library can be represented by using the W _ ID. The participle library holds parts of speech of participles such as nouns, verbs, adjectives, numerologies, quantifiers, pronouns, adverbs, prepositions, conjunctions, helpwords, sighs, and vocabularies. According to a specific embodiment of the present invention, the word segmentation result is divided into real words and imaginary words according to the part of speech, wherein the real words include nouns, verbs, adjectives, numerators, quantifiers and pronouns; the fictional words include adverbs, prepositions, conjunctions, adjectives, sighs, and pseudonyms. Preferably, the thesaurus further includes a synonym thesaurus, wherein the same or similar meaning of the thesaurus is grouped into a group, and the group is numbered as a unit. A plurality of similar or identical meaning participles correspond to a participle group number, and a WG _ ID can be used to indicate a unique number of a participle in the participle library. Preferably, the word segmentation library further comprises a Chinese and foreign language synonym thesaurus, wherein Chinese and foreign language segmentations with the same or similar meanings form a group, and are numbered by taking the group as a unit. A plurality of Chinese and foreign language participles with the same or similar meanings correspond to a Chinese and foreign language participle group number, and a WFG _ ID can be used for representing the unique number of a certain Chinese and foreign language participle group in a participle library.
According to a specific embodiment of the present invention, the system includes a word segmentation module for segmenting words of each material and storing the segmentation result in the comparison library. Preferably, the word segmentation module compares the word segmentation result with the part of speech stored in the word segmentation library to determine the part of speech of the word segmentation result. Preferably, the word segmentation part-of-speech classification module classifies the word segmentation result according to the part-of-speech corresponding to the word segmentation result.
According to a specific embodiment of the present invention, the system includes a word segmentation module for segmenting words of each material and storing the word segmentation result in the comparison library. Preferably, the word segmentation module compares the word segmentation result with the part of speech stored in the word segmentation library to determine the part of speech of the word segmentation result. Preferably, the word segmentation and word grouping classification module classifies the word segmentation result according to the part of speech corresponding to the word segmentation result.
According to a specific embodiment of the present invention, the system comprises a chinese and foreign language word segmentation module for segmenting words of each material and storing the chinese and foreign language word segmentation result in a comparison library. Preferably, the Chinese and foreign language word segmentation module compares the Chinese and foreign language word segmentation result with the part of speech stored in the word segmentation library to determine the part of speech of the Chinese and foreign language word segmentation result. Preferably, the Chinese and foreign language word segmentation and classification module classifies the Chinese and foreign language word segmentation result according to the part of speech corresponding to the Chinese and foreign language word segmentation result.
According to a specific embodiment of the present invention, the word segmentation part-of-speech classification module, the word segmentation group part-of-speech classification module and the Chinese-foreign word segmentation part-of-speech classification module divide the word segmentation result, the word segmentation group result and the Chinese-foreign word segmentation group into a type a real word, a type B real word, a type C real word, a type D real word and a type V imaginary word according to the part-of-speech, wherein the type a real word includes a noun; the B-type real words comprise verbs and adjectives; the C-type real words comprise numeric words and quantifier words; the D-type real words comprise pronouns; the V-type fictional words include adverbs, prepositions, conjunctions, auxiliary words, sigh words and pseudonyms. Preferably, the term library further divides the term into a professional term and a general term. According to a specific embodiment of the present invention, the word segmentation result is divided into a type A1 real word, a type A2 real word, a type B real word, a type C real word, a type D real word, and a type V imaginary word according to the part of speech, wherein the type A1 real word includes a term noun; the A2 real words comprise common nouns; the B-type real words comprise verbs and adjectives; the C-type real words comprise numerals and quantifications; the D-type real words comprise pronouns; the V-type fictional words include adverbs, prepositions, conjunctions, auxiliary words, exclamation words and pseudonyms. The skilled person can choose different classification schemes according to actual needs.
According to an embodiment of the present invention, the segmentation feature value generation module counts the number of occurrences of each segmentation in the corresponding material, and generates a segmentation feature value WCV = [ W _ ID, W _ N ] corresponding to each segmentation, where W _ ID represents a unique number of the segmentation in the segmentation library, and W _ N represents the total number of occurrences of the segmentation in the material. Preferably, the participle part-of-speech feature value generation module generates a participle part-of-speech feature value WCCV = [ W _ ID, W _ N, W _ CHAR ], where W _ CHAR represents the part-of-speech of the participle, in consideration of the part-of-speech of each participle.
According to an embodiment of the present invention, the word segmentation characteristic value generation module counts the number of occurrences of each word segmentation group in the corresponding material, and generates a word segmentation group characteristic value WGCV = [ WG _ ID, WG _ N ] corresponding to each word segmentation group, where WG _ ID represents a unique number of the word segmentation group in the word segmentation library, and WG _ N represents a total number of occurrences of the word segmentation group in the material. Preferably, the word-segmentation-group feature-value generating module generates a word-segmentation-group-word-feature-value WGCCV = [ WG _ ID, WG _ N, WG _ CHAR ] in consideration of the part of speech of each word segmentation group, where WG _ CHAR represents the part of speech of the word segmentation group.
According to a specific embodiment of the present invention, the chinese-foreign language phrase feature value generation module counts the number of occurrences of each chinese-foreign language phrase in the corresponding material, and generates a phrase feature value WFGCV = [ WFG _ ID, WFG _ N ] corresponding to each chinese-foreign language phrase, where WFG _ ID represents the unique number of the chinese-foreign language phrase in the phrase library, and WFG _ N represents the total number of occurrences of the chinese-foreign language phrase in the material. Preferably, the participle group characteristic value generating module generates a Chinese and foreign language participle group characteristic value WFGCCV = [ WFG _ ID, WFG _ N, WFG _ CHAR ] in consideration of the part of speech of each Chinese and foreign language participle group, wherein WFG _ CHAR represents the part of speech of the Chinese and foreign language participle group.
According to an embodiment of the present invention, the word segmentation closeness coefficient generation module is configured to generate a word segmentation closeness coefficient. The word segmentation compactness coefficient refers to the number of word segmentations spaced by two adjacent occurrences of the same word segmentation in the whole material. According to an embodiment of the present invention, the participle compactness coefficient corresponding to each participle is represented as WGC = [ G _ W _ ID _1, G _w _ ID _2, …, G _ W _ ID _ (W _ N-1) ], where G _ W _ ID _1 represents the number of participles spaced between the first occurrence and the second occurrence of the participle in the material, G _ W _ ID _2 represents the number of participles spaced between the second occurrence and the third occurrence of the participle in the material, and G _ W _ ID _ (W _ N-1) represents the number of participles spaced between the W _ N-1 occurrence and the W _ N occurrence of the participle in the material; g _ W _ ID _1, G _W _ID _2, …, G _ W _ ID _ (W _ N-1) are the word segmentation compact coefficients corresponding to the word segmentation. According to an embodiment of the present invention, the participle compactness coefficient feature vector generation module generates a participle compactness coefficient feature vector WGCVE = [ W _ ID, W _ N, W _ CHAR, G _ W _ ID _1, G _w _id _, 2, …, G _ W _ ID _ (W _ N-1) ], where W _ ID represents a unique number of the participle in a participle library, W _ N represents a total number of participles of the specific participle in the story, and W _ CHAR represents a part of speech of the participle. Through the word segmentation compact coefficient, the overall distribution condition of the specific word segmentation in the corresponding material can be obtained.
According to an embodiment of the present invention, the segmentation group compactness coefficient generation module is configured to generate a segmentation group compactness coefficient. The phrase compactness coefficient refers to the number of the phrases of the same phrase which are adjacent and spaced twice in the whole material. According to an embodiment of the present invention, the phrase group compactness coefficient corresponding to each phrase group is represented as WGGC = [ G _ WG _ ID _1, G _wg _ ID _2, …, G _ WG _ ID _ (WG _ N-1) ], where G _ WG _ ID _1 represents the number of phrases spaced between the first occurrence and the second occurrence of the phrase group in the material, G _ WG _ ID _2 represents the number of phrases spaced between the second occurrence and the third occurrence of the phrase group in the material, and G _ WG _ ID _ (WG _ N-1) represents the number of phrases spaced between the WG _ N-1 occurrence and the WG _ N occurrence of the phrase group in the material; g _ WG _ ID _1, G _WG _ID _2, …, G _ WG _ ID _ (WG _ N-1) are the compact coefficients of the word-segmentation group corresponding to the word-segmentation group. According to an embodiment of the present invention, the phrase group compact coefficient eigenvector generation module generates phrase group compact coefficient eigenvectors WGGCVE = [ WG _ ID, WG _ N, WG _ CHAR, G _ WG _ ID _1, G _wg _id _2, …, G _ WG _ ID _ (WG _ N-1) ], wherein WG _ ID represents the unique number of the word segmentation group in the word segmentation library, WG _ N represents the total word segmentation times of the specific word segmentation group in the material, and WG _ CHAR represents the part of speech of the word segmentation group. Through the compact coefficient of the word segmentation group, the overall distribution condition of the specific word segmentation group in the corresponding material can be obtained.
According to a specific embodiment of the present invention, the Chinese and foreign language phrase generating module is configured to generate a Chinese and foreign language phrase group compactness coefficient. The Chinese and foreign language phrase compactness coefficient refers to the number of the phrases which are spaced by two adjacent occurrences of the same Chinese and foreign language phrase in the whole material. According to an embodiment of the present invention, the Chinese-foreign language phrase compactness factor corresponding to each Chinese-foreign language phrase is represented as WFGGC = [ G _ WFG _ ID _1, G _wfg _ ID _, …, G _ WFG _ ID _ (WFG _ N-1) ], where G _ WFG _ ID _1 represents the number of phrases spaced between the first occurrence and the second occurrence of the Chinese-foreign language phrase in the material, G _ WFG _ ID _2 represents the number of phrases spaced between the second occurrence and the third occurrence of the Chinese-foreign language phrase in the material, and G _ WFG _ ID _ (WFG _ N-1) represents the number of phrases spaced between the WFG _ N-1 occurrence and WFG _ N occurrence in the material; g _ WFG _ ID _1, G _WFG _ID _2, …, G _ WFG _ ID _ (WFG _ N-1) are the compact coefficients of the phrase groups corresponding to the Chinese and foreign phrase groups. According to an embodiment of the present invention, the chinese and foreign language segmentation group compact coefficient feature vector generation module generates a chinese and foreign language segmentation group compact coefficient feature vector WFGGCVE = [ WFG _ ID, WFG _ N, WFG _ CHAR, G _ WFG _ ID _1, G _WFG _ID _2, …, G _ WFG _ ID (WFG _ N-1) ], wherein WFG _ ID represents the unique number of the Chinese and foreign language phrase in the phrase library, WFG _ N represents the total number of times of the Chinese and foreign language phrase in the material, and WFG _ CHAR represents the part of speech of the Chinese and foreign language phrase. Through the Chinese and foreign language word group compact coefficient, the overall distribution condition of the specific Chinese and foreign language word groups in the corresponding material can be known.
According to a specific embodiment of the invention, the word segmentation free vector dimension determining module determines a word segmentation free vector dimension WFV according to a word segmentation result of a material; the word segmentation free vector dimension WFV is equal to the number of different word segments obtained after the word segmentation is carried out on the specific material. When the space of the material is shorter or the word segmentation result is less, the obtained word segmentation free vector dimension WFV is less; when the space of the material is longer or the word segmentation result is more, the obtained word segmentation free vector dimension WFV is more.
According to a specific embodiment of the invention, the word segmentation group free vector dimension determining module determines a word segmentation group free vector dimension WGFV according to a word segmentation result of a material; the word segmentation group free vector dimension WGFV is equal to the number of different word segmentation groups obtained after the word segmentation is carried out on the specific material. When the space of the material is shorter or the word segmentation group result is less, the obtained word segmentation group free vector dimension WGFV is less; when the space of the material is longer or the result of the word segmentation group is more, the obtained word segmentation group free vector dimension WGFV is more.
According to a specific embodiment of the invention, the Chinese and foreign language word division free vector dimension determining module determines the Chinese and foreign language word division free vector dimension WFGFV according to word division results of materials; the Chinese and foreign language word segmentation free vector dimension WFGFV is equal to the number of different Chinese and foreign language word segmentation groups obtained after word segmentation is carried out on specific materials. When the space of the material is shorter or the Chinese and foreign language phrase results are less, the obtained free vector dimension WFGFV of the Chinese and foreign language phrase is less; when the space of the material is longer or the result of the word segmentation group in the material is more, the obtained Chinese and foreign language word segmentation group has more free vector dimensions WFGFV.
According to a specific embodiment of the invention, the word segmentation reduced vector dimension generation module is used for reducing the word segmentation free vector dimension WFV of each material to generate a word segmentation reduced vector dimension RWV. The reduced word-segmentation vector dimension RWV is specified by the system. Preferably, the system specifies a participle reduced vector dimension RWV of 500. Preferably, the system specifies a tokenized reduced vector dimension RWV of 800. Preferably, the system specifies a tokenized reduced vector dimension RWV of 1000.
According to a specific embodiment of the invention, the word segmentation and simplification vector dimension generation module adopts an equal interval extraction method to simplify the word segmentation free vector dimension WFV. The simplification process is as follows: judging whether the word segmentation free vector dimension WFV is larger than the word segmentation simplified vector dimension RWV, if so, dividing the word segmentation free vector dimension WFV by the word segmentation simplified vector dimension RWV specified by the system, and performing rounding operation on the obtained quotient to further obtain a simplified coefficient REDU; extracting a characteristic value from characteristic values corresponding to the word segmentation free vector dimension WFV at intervals of REDU-1; after all the characteristic values are extracted, judging whether the number of the extracted characteristic values is equal to a word segmentation reduced vector dimension RWV; when the number of the extracted characteristic values is equal to the word segmentation simplified vector dimension RWV, word segmentation free vector dimension WFV simplification is completed; when the number of the extracted characteristic values is less than the simplified word segmentation vector dimension RWV, calculating the difference value between the simplified word segmentation vector dimension RWV and the number of the characteristic values; and randomly extracting feature values with the same number as the difference value of the word segmentation simplified vector dimension RWV and the feature values from the feature values which are not extracted, and completing the simplification of the word segmentation free vector dimension WFV.
According to a specific embodiment of the invention, the word segmentation and simplification vector dimension generation module adopts a part-of-speech screening method to simplify the word segmentation free vector dimension WFV. The simplification process is as follows: classifying the characteristic values of the word segmentation results according to the corresponding word segmentation parts of speech; according to an embodiment of the present invention, the feature values are divided into a class A1 real word feature value, a class A2 real word feature value, a class B real word feature value, a class C real word feature value, a class D real word feature value, and a class V particle feature value. It is generally considered that the similarity of the feature values corresponding to the real words plays a greater role in comparison, wherein the term noun in the technical field is more effective in representing the effective content of the material than the general term noun. The number of feature values in each category, i.e., estimate _ A1 (the number of real-word feature values in a category 1), estimate _ A2 (the number of real-word feature values in a category 2), estimate _ B (the number of real-word feature values in a category B), estimate _ C (the number of real-word feature values in a category C), estimate _ D (the number of real-word feature values in a category D), and estimate _ V (the number of real-word feature values in a category V) are counted. Calculating RWV _ S _ V of the participle reduced vector dimension RWV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V); if the sum is more than 0, the simplification is quitted; if the sum is equal to 0, the simplification is finished; if less than 0, further calculate the value RWV _ S _ D of the participle reduced vector dimension RWV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D); if the difference value is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWV _ S _ D from the feature values corresponding to AMOUNT _ V to finish the simplification; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating a value RWV _ S _ C of the participle reduced vector dimension RWV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C); if the difference value is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWV _ S _ C from the feature values corresponding to the AMOUNT _ D, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating the value RWV _ S _ B of the participle reduced vector dimension RWV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B); if the difference value is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWV _ S _ B from the feature values corresponding to AMOUNT _ C to finish the simplification; if the sum is equal to 0, the simplification is finished; if the dimension is smaller than 0, further calculating the value RWV _ S _ A2 of the participle reduced vector dimension RWV- (AMOUNT _ A1+ AMOUNT _ A2); if the difference value is larger than 0, randomly extracting the feature values with the quantity equal to the difference value RWV _ S _ A2 from the feature values corresponding to AMOUNT _ B, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension is smaller than 0, further calculating the value RWV _ S _ A1 of the word segmentation reduced vector dimension RWV-AMOUNT _ A1; if the difference value is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWV _ S _ A1 from the feature values corresponding to AMOUNT _ A2 to finish the simplification; if the sum is equal to 0, the simplification is finished; if the number of the feature values is smaller than 0, the feature values with the number equal to that of the reduced vector dimension RWV are randomly extracted from the feature values corresponding to the AMOUNT _ A1, and the reduction is completed.
For the case where the value RWV _ S _ V of the reduced word-segmentation vector dimension RWV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) is greater than 0, it means that the material space is small or the information AMOUNT is small, and therefore it is not suitable for comparison using the feature value.
When the word segmentation free vector dimension WFV is smaller than the word segmentation reduced vector dimension RWV, the dimension of the word segmentation free vector dimension WFV is small, and the magnitudes under other dimensions are equal to 0. This situation requires direct labeling in the system, with separate listing processing. Such as folk colloquial, celebrity, and the like, for index lookup. Full-text sliding windows can be used for full-text comparison subsequently.
According to a specific embodiment of the invention, the reduced word group vector dimension generation module is used for reducing the word group free vector dimension WGFV of each material to generate a reduced word group vector dimension RWGV. The reduced word-group vector dimension RWGV is specified by the system. Preferably, the system specifies a reduced word-group vector dimension RWGV of 500. Preferably, the system specifies a reduced word-group vector dimension RWGV of 800. Preferably, the system specifies a reduced-word-group vector dimension RWGV of 1000.
According to a specific embodiment of the invention, the word segmentation group reduction vector dimension generation module adopts an equal interval extraction method to reduce the word segmentation free vector dimension WGFV. The simplification process is as follows: judging whether the word segmentation group free vector dimension WGFV is larger than the simplified word segmentation group vector dimension RWGV, if so, dividing the word segmentation group free vector dimension WGFV by the system-specified simplified word segmentation group vector dimension RWGV, and performing rounding operation on the obtained quotient to further obtain a simplified coefficient REDU; extracting a characteristic value from characteristic values corresponding to the word segmentation group free vector dimension WGFV at intervals of REDU-1; after all the characteristic values are extracted, judging whether the number of the extracted characteristic values is equal to the reduced vector dimension RWGV of the word segmentation group or not; when the number of the extracted characteristic values is equal to the reduced vector dimension RWGV of the word segmentation group, finishing the reduction of the free vector dimension WGFV of the word segmentation group; when the number of the extracted characteristic values is smaller than the reduced word group vector dimension RWGV, calculating the difference value between the reduced word group vector dimension RWGV and the number of the characteristic values; and randomly extracting characteristic values with the same quantity as the difference value between the simplified vector dimension RWGV of the word segmentation group and the characteristic values from the unextracted characteristic values, and finishing the simplification of the free vector dimension WGFV of the word segmentation group.
According to a specific embodiment of the present invention, the word segmentation group reduction vector dimension generation module reduces the word segmentation free vector dimension WGFV by using a part-of-speech filtering method. The simplification process is as follows: classifying the characteristic values according to the corresponding word segmentation parts of speech; according to an embodiment of the present invention, the feature values are divided into a class A1 real word feature value, a class A2 real word feature value, a class B real word feature value, a class C real word feature value, a class D real word feature value, and a class V particle feature value. It is generally considered that the similarity of the feature values corresponding to the real words plays a greater role in comparison, wherein the term noun in the technical field is more effective in representing the effective content of the material than the general term noun. The number of feature values in each category, i.e., estimate _ A1 (the number of real-word feature values in a category 1), estimate _ A2 (the number of real-word feature values in a category 2), estimate _ B (the number of real-word feature values in a category B), estimate _ C (the number of real-word feature values in a category C), estimate _ D (the number of real-word feature values in a category D), and estimate _ V (the number of real-word feature values in a category V) are counted. Calculating a value RWGV _ S _ V of a reduced word-group vector dimension RWGV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V); if the sum is larger than 0, the simplification is quitted; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating a value RWGV _ S _ D of the reduced vector dimension RWGV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D); if the difference value RWGV is larger than 0, randomly extracting the characteristic values with the quantity equal to that of the difference value RWGV _ S _ D from the characteristic values corresponding to the AMOUNT _ V, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating a value RWGV _ S _ C of a participle reduced vector dimension RWGV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C); if the difference value RWGV _ S _ C is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWGV _ S _ C from the feature values corresponding to the AMOUNT _ D, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating a value RWGV _ S _ B of the reduced vector dimension RWGV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B); if the difference value RWGV _ S _ B is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWGV _ S _ B from the feature values corresponding to the AMOUNT _ C to finish the simplification; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating a value RWGV _ S _ A2 of a reduced vector dimension RWGV- (AMOUNT _ A1+ AMOUNT _ A2); if the difference value RWGV _ S _ A2 is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWGV _ S _ A2 from the feature values corresponding to the AMOUNT _ B, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension is smaller than 0, further calculating the value RWGV _ S _ A1 of the reduced vector dimension RWGV-AMOUNT _ A1 of the word group; if the difference value is larger than 0, randomly extracting feature values with the same number as the difference value RWGV _ S _ A1 from the feature values corresponding to the AMOUNT _ A2, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the number of the feature values is smaller than 0, the feature values with the number equal to the number of the reduced vector dimension RWGV are randomly extracted from the feature values corresponding to the AMOUNT _ A1, and the reduction is completed.
For the case that the value RWGV _ S _ V of the reduced vector dimension RWGV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) of the participle group is greater than 0, it means that the material space is small or the information AMOUNT is small, and therefore it is not suitable for comparison with the feature value.
When the word segmentation group free vector dimension WGFV is smaller than the word segmentation group reduced vector dimension RWGV, the dimension of the word segmentation group free vector dimension WGFV is small, and the magnitude values under other dimensions are equal to 0. This situation requires direct labeling in the system and separate inclusion processing. Such as folk colloquial, celebrity, and the like, for index lookup. Full-text sliding windows can be used for full-text comparison subsequently.
According to a specific embodiment of the invention, the Chinese-foreign language phrase-divided reduced vector dimension generation module is used for reducing the Chinese-foreign language phrase-divided free vector dimension WFGFV of each material to generate the Chinese-foreign language phrase-divided reduced vector dimension RWFFGV. The Chinese and foreign language word segmentation group reduced vector dimension RWFFGV is specified by the system. Preferably, the system specifies a chinese participle-group reduced vector dimension RWFGV of 500. Preferably, the system specifies a Chinese-foreign word-group reduced vector dimension RWFFGV of 800. Preferably, the system specifies a Chinese-foreign participle-group reduced vector dimension RWFFGV of 1000.
According to a specific embodiment of the invention, the Chinese-foreign word segmentation group reduction vector dimension generation module adopts an equal interval extraction method to reduce the free vector dimension WFGFV of the Chinese-foreign word segmentation group. The simplification process is as follows: judging whether the Chinese and foreign language word segmentation group free vector dimension WFGFV is larger than the Chinese and foreign language word segmentation group reduced vector dimension RWFFGV, if so, dividing the Chinese and foreign language word segmentation group free vector dimension WFGFV by the Chinese and foreign language word segmentation group reduced vector dimension RWFFGV specified by the system, and performing rounding operation on the obtained quotient to further obtain a reduced coefficient REDU; extracting a characteristic value from characteristic values corresponding to the Chinese and foreign word segmentation group free vector dimension WFGFV at intervals of REDU-1; after all the characteristic values are extracted, judging whether the number of the extracted characteristic values is equal to the Chinese and foreign language word grouping reduced vector dimension RWFFGV or not; when the number of the extracted characteristic values is equal to the Chinese and foreign language phrase reduced vector dimension RWFFGV, the Chinese and foreign language phrase free vector dimension WFGFV reduction is completed; when the number of the extracted characteristic values is smaller than the Chinese and foreign language phrase reduced vector dimension RWFFGV, calculating the difference value between the Chinese and foreign language phrase reduced vector dimension RWFFGV and the number of the characteristic values; and randomly extracting characteristic values with the same quantity as the difference value between the Chinese and foreign language word segmentation simplified vector dimension RWFFGV and the characteristic values from the unextracted characteristic values to finish the simplification of the Chinese and foreign language word segmentation free vector dimension WFGFV.
According to a specific embodiment of the invention, the Chinese-foreign word segmentation group reduction vector dimension generation module adopts a part-of-speech screening method to reduce the Chinese-foreign word segmentation group free vector dimension WFGFV. The simplification process is as follows: classifying the characteristic values according to the corresponding word segmentation parts of speech; according to an embodiment of the present invention, the feature values are divided into a class A1 real word feature value, a class A2 real word feature value, a class B real word feature value, a class C real word feature value, a class D real word feature value, and a class V particle feature value. It is generally considered that the similarity of the feature values corresponding to the real words plays a greater role in comparison, wherein the term noun in the technical field is more effective in representing the effective content of the material than the general term noun. The number of feature values in each category, i.e., estimate _ A1 (the number of real-word feature values in a category 1), estimate _ A2 (the number of real-word feature values in a category 2), estimate _ B (the number of real-word feature values in a category B), estimate _ C (the number of real-word feature values in a category C), estimate _ D (the number of real-word feature values in a category D), and estimate _ V (the number of real-word feature values in a category V) are counted. Calculating a value RWFFGV _ S _ V of the Chinese and foreign phrase reduced vector dimension RWFFVFVFVSV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V); if the sum is more than 0, the simplification is quitted; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating a value RWFVG _ S _ D of the Chinese and foreign language word-group reduced vector dimension RWFFV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D); if the difference value RWFDV _ S _ D is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWFDV _ S _ D from the feature values corresponding to AMUNT _ V, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension RWFDV is less than 0, further calculating the value RWFDV _ S _ C of the Chinese and foreign word segmentation group reduced vector dimension RWFDV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C); if the difference value RWFDV _ S _ C is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWFDV _ S _ C from the feature values corresponding to AMUNT _ D, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension RWFCV is less than 0, a value RWFCV _ S _ B of the Chinese and foreign language word-group reduced vector dimension RWFCV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B) is further calculated; if the difference value RWFDV _ S _ B is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWFDV _ S _ B from the feature values corresponding to AMUNT _ C to finish the simplification; if the sum is equal to 0, the simplification is finished; if less than 0, further calculating a value RWFFGV _ S _ A2 of the reduced vector dimension RWFFV- (AMOUNT _ A1+ AMOUNT _ A2) of the word group; if the difference value RWFFV _ S _ A2 is larger than 0, randomly extracting the feature values with the same number as the difference value RWFFV _ S _ A2 from the feature values corresponding to AMUNT _ B, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension RWFDV-AMOUNT _ A1 is smaller than 0, the value RWFDV _ S _ A1 of the Chinese and foreign language word segmentation group reduced vector dimension RWFDV-AMOUNT _ A1 is further calculated; if the difference value RWFFV _ S _ A1 is larger than 0, randomly extracting the feature values with the same number as the difference value RWFFV _ S _ A1 from the feature values corresponding to AMUNT _ A2, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the number of the feature values is smaller than 0, the feature values with the number equal to that of the reduced vector dimension RWFFGV are randomly extracted from the feature values corresponding to AMOUNT _ A1, and the reduction is completed.
For the case that the value RWFGV _ S _ V of the reduced vector dimension RWFGV- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) of the chinese and foreign word group is greater than 0, it means that the material space is small or the information AMOUNT is small, and therefore, it is not suitable for comparison with the feature value.
When the word grouping free vector dimension WFGFV is smaller than the word grouping reduced vector dimension RWFFGV, the dimension of the word grouping free vector dimension WFGFV is small, and the quantity values under other dimensions are equal to 0. This situation requires direct labeling in the system, with separate listing processing. Such as folk colloquial, celebrity, and the like, for index lookup. Full-text sliding windows can be used for full-text comparison subsequently.
According to a specific embodiment of the invention, the participle feature vector generation module extracts a feature value corresponding to the participle reduced vector dimension RWV in each story according to the participle reduced vector dimension RWV to generate a participle feature vector WVE _ RWV;
WVE_RWV=[W_ID1,W_N1,...,W_IDi,W_Ni,...,W_IDRWV,W_NRWV]
wherein W _ IDi represents the unique number of the participle in the participle library, W _ Ni represents the total times of the participle appearing in the material, and the times are taken as the characteristic value of the participle.
According to a specific embodiment of the invention, the word segmentation group feature vector generation module extracts a feature value corresponding to the word segmentation group reduced vector dimension RWGV in each material according to the word segmentation group reduced vector dimension RWGV to generate a word segmentation group feature vector WVE _ RWGV;
WVE_RWGV=[WG_ID1,WG_N1,...,WG_IDi,WG_Ni,...,WG_IDRWGV,WG_NRWGV]
wherein WG _ IDi represents the unique number of the segmentation group in the segmentation library, WG _ Ni represents the total times of the occurrence of the segmentation group in the material, and the times are taken as the characteristic value of the segmentation group.
According to a specific embodiment of the invention, a Chinese and foreign language word segmentation group feature vector generation module extracts feature values corresponding to the Chinese and foreign language word segmentation group reduced vector dimension RWFFGV in each material according to the Chinese and foreign language word segmentation group reduced vector dimension RWFFGV to generate a Chinese and foreign language word segmentation group feature vector WVE _ RWFFV;
WVE_RWFGV=[WFG_ID1,WFG_N1,...,WFG_IDi,WFG_Ni,...,WFG_IDRWFGV,WFG_NRWFGV]
WFG _ IDi represents the unique number of the Chinese and foreign language phrase in the phrase library, WFG _ Ni represents the total times of the Chinese and foreign language phrase in the material, and the times are used as the characteristic value of the Chinese and foreign language phrase.
According to one embodiment of the invention, the system provides multiple access modes for the user. And the user access mode detection module is used for detecting the access mode of the current user.
In one embodiment of the invention, the user may access the system on a trial basis, hereinafter referred to as the trial user. When the user access mode detection module detects that the user accesses in a trial mode, a prompt is sent to the trial user to inform that the current access mode is the trial mode and to inform the use permission of the trial user. According to one embodiment of the invention, the system provides only trial tests of a predetermined number of characters for the trial user, the predetermined number of words being set in advance by the system, for the user to be accessed in a trial manner. According to another embodiment of the invention, for users who are accessed on a trial basis, the system provides the trial user with a partial or full range of databases for testing the trial. According to another embodiment of the invention, for users accessed in a trial mode, the system only provides plagiarism rate prompts for plagiarism detection results provided by the trial users, and does not provide specific plagiarism positions and plagiarism comparison with plagiarism documents. According to another specific embodiment of the invention, for the users who visit in a trial mode, the system provides specific plagiarism positions for plagiarism detection results provided by the trial users, but fuzzification processing is carried out on the plagiarism comparison with the plagiarism document, so that the trial users can only know the specific plagiarism positions of the documents provided by the trial users, but cannot identify the specific information of the plagiarism documents.
According to an embodiment of the invention, the user accesses the system in a counting mode, hereinafter referred to as counting user. When the user access mode detection module detects that the user accesses in a counting mode, a prompt is sent to the counting user, the current access mode is informed of the counting mode, and the counting user is prompted to upload documents needing copy comparison. According to an embodiment of the invention, the system counts the number of characters of the document uploaded by the user, and calculates the cost of the text plagiarism detection according to the counted number of characters. According to another embodiment of the invention, the system provides a database with a partial or full range for the counting user to select, and calculates the cost of the text plagiarism detection according to the different database ranges selected by the counting user.
According to an embodiment of the invention, the user accesses the system in a time-based manner, and the user accessing in the time-based manner is referred to as the time-based user hereinafter. When the user access mode detection module detects that the user accesses in a timing mode, a prompt is sent to the timing user to inform that the current access mode is the timing mode, and the timing user is prompted to keep the current residual use duration. According to another embodiment of the invention, for the timing user, the system provides the remaining usage duration countdown prompt in real time for the timing user in the display interface during the use process. According to another embodiment of the invention, the system provides a partial or full range of databases for timing users to choose from. According to a specific embodiment of the invention, the system estimates the detection duration required by the document according to the selection of different database ranges by the timing user and the number of characters of the detection document uploaded by the timing user, and prompts the timing user whether the residual use duration can complete the current plagiarism detection.
According to a specific embodiment of the invention, after the timing user logs in the system, the user detection mode determining module determines the plagiarism detection mode. According to one embodiment of the invention, the system provides a self-auditing mode, a common plagiarism identification mode, an extended plagiarism identification mode, a multi-language plagiarism identification mode and a formula plagiarism identification mode for selection.
According to a specific embodiment of the present invention, when the user detection mode determination module determines that the current user detection mode is the self-auditing mode, the user writing style test module provides one or more test pictures for the user, and the user performs online description of not less than a specified number of words for the test pictures within a specified time. Preferably, the user writing style test module further provides the user with one or more test articles, and the user makes online text comments of not less than a specified number of words within a specified time. The test pictures or test articles are randomly selected from the test gallery and the test library by the user writing style test module. Whether a test picture or a test article is adopted, the user needs to perform online text description or comment, the specified time cannot be set too long, and is usually selected to be 30 minutes or 60 minutes, and the specified word number of the corresponding text description or text comment is usually selected to be 400 words/30 minutes or 800 words/60 minutes. Those skilled in the art can further set other prescribed time or prescribed word number as necessary. From experimental data, the set time is not suitable to be set too long, so that the situation that a user does not have enough time or a network is unstable and cannot complete corresponding tests is avoided; in addition, the ratio of the number of the specified words to the specified time is not too low, so as to avoid the inability to faithfully reflect the writing habit of the user. The text description or text comment has a limited length, and the text description or text comment has a limited space, and the feature value and feature vector of the text description or text comment extracted by only on-line testing may not truly reflect the writing habit of the user, so that it is necessary to further extract a test picture description reference feature vector and a test article description reference feature vector for correcting a feature vector deviation value caused by insufficient text description or text comment text.
According to an embodiment of the present invention, each test picture in the test gallery has a test picture reference feature vector. The test picture description reference feature vector is obtained by randomly selecting a preset number of reference testers from different background crowds, describing no less than a specified number of words on a specific test picture, collecting all the word descriptions, counting the test picture word description feature values of the same test picture, calculating the feature vector according to the test picture word description feature values, and performing weighted operation on the feature vector. And setting the weight value in the weighting operation by a system. Each test article in the test library has a test article benchmark feature vector. The test article reference feature vector is obtained by randomly selecting a predetermined number of reference testers from different background crowds, describing at least a specified word number on a specific test article, collecting all character descriptions, counting character description feature values of the test article aiming at the same test article, calculating the feature vector according to the character description feature values of the test article, and performing weighted operation on the feature vector. And the weight value in the weighted operation is set by a system.
According to one embodiment of the present invention, the predetermined number of benchmark panelists randomly selected from different background populations may be selected according to different age groups, and preferably may be divided into groups under 20 years old, groups 20-29 years old, groups 30-39 years old, groups 40-49 years old, and groups over 50 years old. Therefore, the description conditions of the crowds of different age groups on the same test picture or the same test article which is not less than the specified word number are collected.
According to an embodiment of the present invention, when randomly selecting a predetermined number of benchmark testers from different background groups, the benchmark testers may be selected according to different levels of academic records, and preferably, the benchmark testers may be classified into university department groups, master research groups, and doctor research groups. Therefore, the description conditions of the crowd of different academic calendar groups on the same test picture or the same test article which is not less than the specified word number are collected.
According to a specific embodiment of the present invention, when a predetermined number of benchmark testers are randomly selected from different background people, the benchmark testers may be selected according to different professional fields (the professional fields may be divided according to different test precision requirements, and are not described herein again), so as to collect descriptions of people in different professional field groups on the same test picture or the same test article that is not less than a specified number of words.
According to a specific embodiment of the invention, a test picture character description characteristic value generation module acquires a test picture description text of a benchmark tester acquired by the benchmark tester, and generates a user test picture character description characteristic value; the test picture text description feature values include but are not limited to: chinese character number, foreign character number, total word number, real word number, virtual word number, paragraph length distribution condition, sentence number, sentence length distribution condition, synonym expansion condition, virtual word use condition, punctuation mark use condition, part of speech use condition. According to an embodiment of the present invention, the number of chinese characters refers to the number of chinese characters included in the text description of each test picture except for the punctuation marks, and each chinese character is marked as a character; the foreign language number refers to the number of foreign language characters contained in the text description of each test picture except punctuations, and each word of the foreign language is marked as a character; the total word number refers to the total number of words obtained after word segmentation is carried out on each test picture text description, wherein Chinese word segmentation can be carried out by using a word segmentation library carried by the system, and foreign languages can be used for carrying out word segmentation by directly utilizing spaces between words according to writing habits of the foreign languages; the real word number refers to the number of real words in the text description of each test picture obtained by comparing the word segmentation result with the part of speech in the word segmentation library after word segmentation, and the real word number can be further divided into the number of Chinese real words and the number of foreign language real words, wherein the sum of the number of Chinese real words and the number of foreign language real words is equal to the number of real words; the number of virtual words refers to the number of virtual words in the text description of each test picture obtained by comparing the word segmentation result with the part of speech in the word segmentation library after word segmentation, and the number of virtual words can be further divided into the number of Chinese virtual words and the number of foreign language virtual words, wherein the sum of the number of Chinese virtual words and the number of foreign language virtual words is equal to the number of virtual words; the paragraph number refers to the number of paragraphs in the text description of each test picture; the paragraph length distribution condition refers to the number of words and the number of sentences contained in each paragraph in the text description of each test picture; the sentence number refers to the number of sentences in each text description of the test picture; the sentence length distribution condition refers to the number of words contained in each sentence in each test picture text description; the synonym and near-synonym expansion condition refers to that the word segmentation result in each test picture text description is compared with a synonym library, the words with the same or similar meanings form a set, the number of words in each set is calculated, and therefore the writing habits of synonyms and near-synonyms of an author described by the test picture text description are reflected, wherein if the number of words contained in the synonym or near-synonym set is more, the writing style of the author tends to be expanded by the synonym or near-synonym, and if the number of words contained in the synonym or near-synonym set is less, the writing style of the author tends not to be expanded by the synonym or near-synonym; the virtual word use condition refers to a statistical condition of virtual word use in each test picture text description, and includes but is not limited to a statistical ranking of virtual word use in each test picture text description, the number of words spaced between each different virtual word, and the number of words spaced between each same virtual word; for example, the usage of the three structural auxiliary words "in", "with" and "get" can be further counted, so as to reflect whether the author of the text description of the test picture uses the three structural auxiliary words "in", "with" and "get" separately or not; the punctuation mark use condition refers to the statistical condition of punctuation mark use in each test picture textual description, including but not limited to the statistical ranking of punctuation mark use in each test picture textual description, the number of words spaced between each different punctuation mark, and the number of words spaced between each same punctuation mark; the part-of-speech use condition refers to a statistical condition of each part-of-speech word segmentation in each test picture text description obtained by comparing a word segmentation result with the part-of-speech in a word segmentation library after word segmentation, for example, the quantity of a noun, a verb, an adjective, a number word, an quantifier, a pronoun, an adverb, a preposition, an assistant word, an exclamation word and a pseudonym, and the ratio of the quantity of each part-of-speech to the total quantity of the part-of-speech of the test picture text description are respectively obtained.
According to a specific embodiment of the present invention, the test picture text description feature value generation module generates a test picture text description feature vector according to the test picture text description feature value. According to an embodiment of the present invention, the system specifies the dimension of the text description feature vector of the test picture, and the specific content and arrangement order of the items in the feature vector. When the dimension of the feature vector of the test picture text description is n, TPCVE = [ TPC _1, …, TPC _ m, …, TPC _ n ], where TPC _1 is the first term value in the feature vector of the test picture text description, TPC _ m is the m-th term value in the feature vector of the test picture text description, and TPC _ n is the n-th term value in the feature vector of the test picture text description, may be expressed.
Preferably, the test picture textual description feature vector includes one or more of: the ratio of Chinese word number to total word number, the ratio of foreign word number to total word number, the ratio of real word number to total word number, the ratio of dummy word number to total word number, the ratio of total word number to paragraph number, the ratio of longest paragraph word number, the ratio of synonym, near-synonym, and total word number, the ratio of punctuation number to total word number, the ratio of noun number to total word number, the ratio of verb number to total word number, the ratio of adjective number to total word number, the ratio of digital number to total word number, the ratio of quantifier number to total word number, the ratio of pronoun number to total word number, the ratio of adverb number to total word number, the ratio of preposition number to total word number, the ratio of conjunctive number to total word number, the ratio of interword number to total word number, the ratio of helpword number to total word number, the ratio of exclamation number to total word number, and the ratio of quasi-phonatory number to total word number.
According to a specific embodiment of the invention, a test picture reference feature vector generation module counts test picture text description feature vectors for the same test; and performing weighting operation on the text description feature vector of the test picture to obtain a reference feature vector of the specific test picture, wherein a weight used in the weighting operation is set by a system. Preferably, the test picture reference feature vector generation module may count a predetermined number of test picture text description feature vectors for different age groups, school calendar groups, and professional field groups, and perform weighting operation to obtain specific test picture reference feature vectors for each age group, each school calendar group, and each professional field group.
The specific test picture reference feature vector may be expressed as:
wherein TPCVE _ ID represents a test picture reference feature vector numbered ID; k is the number of benchmark testers; TPC _1 i A first term value representing a feature vector of an ith benchmark tester; TPC _ m i An mth term value representing a feature vector of an ith benchmark tester; TPC _ n i An nth term value representing a feature vector of an ith benchmark tester; w 1,i Is TPC _1 i The weighting coefficient of (2); w m,i Is TPC _ m i The weighting coefficients of (a); w n,,i Is TPC _ n i The weighting coefficient of (2).
According to a specific embodiment of the invention, a test article text description characteristic value generation module acquires a test article description text of a benchmark tester acquired by the benchmark tester, and generates a user test article text description characteristic value; the text description feature values of the test article include but are not limited to: chinese character number, foreign character number, total word number, real word number, virtual word number, paragraph length distribution condition, sentence number, sentence length distribution condition, synonym expansion condition, virtual word use condition, punctuation mark use condition, part of speech use condition. According to one embodiment of the present invention, the number of chinese words refers to the number of chinese characters contained in the textual description of each test article, except for punctuation marks, where each chinese word is marked as a character; the foreign language word number refers to the number of foreign language characters contained in the text description of each test article except the punctuation marks, and each word of the foreign language is marked as a character; the word number refers to the total number of words obtained after word segmentation is carried out on the text description of each test article, wherein Chinese word segmentation can be carried out by using a word segmentation library carried by the system, and foreign languages can be used for carrying out word segmentation by directly utilizing spaces between every two words according to writing habits of the foreign languages; the number of real words refers to the number of real words in the text description of each test article obtained by comparing the word segmentation result with the part of speech in the word segmentation library after word segmentation, and the number of real words can be further divided into the number of Chinese real words and the number of foreign language real words, wherein the sum of the number of Chinese real words and the number of foreign language real words is equal to the number of real words; the number of virtual words refers to the number of virtual words in the text description of each test article obtained by comparing the word segmentation result with the part of speech in the word segmentation library after word segmentation, and the number of virtual words can be further divided into the number of Chinese virtual words and the number of foreign language virtual words, wherein the sum of the number of Chinese virtual words and the number of foreign language virtual words is equal to the number of virtual words; the number of paragraphs refers to the number of paragraphs in the text description of each test article; the paragraph length distribution condition refers to the number of words and the number of sentences contained in each paragraph in the text description of each test article; the sentence number refers to the number of sentences in the text description of each test article; the sentence length distribution condition refers to the number of words contained in each sentence in the text description of each test article; synonym and near-synonym expansion condition means that the word segmentation result in the text description of each test article is compared with a synonym near-synonym library, the words with the same or similar meanings form a set, the number of words in each set is calculated, and therefore the writing habits of synonyms and near-synonyms of the author of the text description of the test article are reflected, wherein if the number of words contained in the set of synonyms or near-synonyms is more, the writing style of the author is prone to adopt synonym or near-synonym expansion, and if the number of words contained in the set of synonyms or near-synonyms is less, the writing style of the author is prone to not adopt synonym or near-synonym expansion; the term usage refers to the statistical conditions of term usage in the text description of each test article, including but not limited to the statistical ranking of term usage in the text description of each test article, the number of words spaced between each different term and the number of words spaced between each identical term; for example, the usage of the three structural auxiliary words "in", "with" and "get" can be further counted, so as to reflect whether the author of the text description of the test article uses the three structural auxiliary words "in", "with" and "get" separately; the punctuation mark use condition refers to the statistical condition of punctuation mark use in each test article textual description, including but not limited to the statistical ranking of punctuation mark use in each test article textual description, the word number of the interval between each different punctuation mark, the word number of the interval between each same punctuation mark; the part-of-speech use condition refers to a statistical condition of each part-of-speech word in each test article text description obtained by comparing a word segmentation result with the part-of-speech in a word segmentation library after word segmentation, for example, the quantity of nouns, verbs, adjectives, numerators, quantifiers, pronouns, adverbs, prepositions, conjunctions, auxiliary words, sighs and pseudonyms and the ratio of the quantity of each part-of-speech to the total quantity of the part-of-speech of the test article text description are respectively obtained.
According to a specific embodiment of the invention, the test article text description feature value generation module generates a test picture text description feature vector according to the test article text description feature value. According to an embodiment of the invention, the system specifies the dimension of the text description feature vector of the test article, and the specific content and arrangement sequence of each item in the feature vector. When the dimension of the feature vector of the text description of the test article is n, TTCVE = [ TTC _1, …, TTC _ m, …, TTC _ n ], where TTC _1 is the first entry value in the feature vector of the text description of the test picture, TTC _ m is the m-th entry value in the feature vector of the text description of the test picture, and TTC _ n is the n-th entry value in the feature vector of the text description of the test picture.
Preferably, the test article textual description feature vector includes one or more of: the ratio of Chinese word number to total word number, the ratio of foreign word number to total word number, the ratio of real word number to total word number, the ratio of dummy word number to total word number, the ratio of total word number to paragraph number, the ratio of longest paragraph word number, the ratio of synonym, near-synonym, and total word number, the ratio of punctuation number to total word number, the ratio of noun number to total word number, the ratio of verb number to total word number, the ratio of adjective number to total word number, the ratio of digital number to total word number, the ratio of quantifier number to total word number, the ratio of pronoun number to total word number, the ratio of adverb number to total word number, the ratio of preposition number to total word number, the ratio of conjunctive number to total word number, the ratio of interword number to total word number, the ratio of helpword number to total word number, the ratio of exclamation number to total word number, and the ratio of quasi-phonatory number to total word number.
According to a specific embodiment of the invention, a test article reference feature vector generation module counts test article text description feature vectors for the same test; and performing weighted operation on the text description feature vectors of the test articles to obtain the reference feature vectors of the specific test articles, wherein the weight used in the weighted operation is set by a system. Preferably, the test article reference feature vector generation module may count a predetermined number of test article text description feature vectors for different age groups, academic calendar groups, and professional field groups, and perform weighting operation to obtain specific test article reference feature vectors for each age group, each academic calendar group, and each professional field group.
The particular article reference feature vector may be expressed as:
wherein TTCVE _ ID represents a reference feature vector of the test article with the number ID; k is the number of benchmark testers; TTC _1 i A first term value representing a feature vector of an ith benchmark tester; TTC _ m i An mth term value representing a feature vector of an ith benchmark tester; TTC _ n i An nth term value representing a feature vector of an ith benchmark tester; w 1,i Is TPC _1 i The weighting coefficient of (2); w m,i Is TPC _ m i The weighting coefficient of (2); w n,,i Is TPC _ n i The weighting coefficient of (2).
According to an embodiment of the invention, the dimension of the text description feature vector of the test picture and the dimension of the text description feature vector of the test article are consistent, and the meaning and the arrangement sequence of each feature value are consistent. For example, it may be set that first term feature values in the test picture text description feature vector and the test article text description feature vector are ratios of the number of Chinese words to the total number of words, second term feature values are ratios of the number of foreign words to the total number of words, third term feature values are ratios of the number of real words to the number of total words, fourth term feature values are ratios of the number of dummy words to the number of total words, fifth term feature values are ratios of the number of total words to the number of paragraphs, sixth term feature values are ratios of the number of longest paragraphs, seventh term feature values are ratios of the number of synonyms, the number of expansion of near-meaning words to the number of total words, eighth term feature values are ratios of the number of punctuation use to the number of total words, ninth term feature values are ratios of the number of exclamation words to the total number of words, tenth term feature values are ratios of the number of mobilization words to the total number of words, eleventh term feature values are ratios of the number of adjectives to the number of total words, twelfth term feature values are ratios of the number of lexes to the number of common words, thirteenth term feature values to the number of common words, fourteenth terms and total terms feature values are ratios of common terms, and common terms feature values are ratios of common terms, and common terms.
According to an embodiment of the present invention, the feature values in the text description feature vector of the test picture and the text description feature vector of the test article may be further added or deleted, but the dimensions of the text description feature vector of the test picture and the text description feature vector of the test article after the feature values are added or deleted and the meanings and the sequences of the feature values therein are still required to be consistent.
According to a specific embodiment of the invention, a user test picture character description characteristic value generation module acquires a user test picture description text and generates a user test picture character description characteristic value; the content of the user test picture text description characteristic value is consistent with that of the test picture text description characteristic value, and is not repeated here. The user test picture character description feature vector generation module calculates a user test picture character description feature vector according to the user test picture character description feature value; when the dimension of the test picture text description feature vector is n, the feature vector of the test picture text description of the picture with the number ID of the current USER can be represented as TPCVE _ ID _ USER = [ TPC _1_user, …, TPC _ m _ USER, …, TPC _ n _ USER ], where TPC _1 USER is the first entry value in the USER test picture text description feature vector of the current USER, TPC _ m _ USER is the m-th entry value in the USER test picture text description feature vector of the current USER, and TPC _ n _ USER is the n-th entry value in the USER test picture text description feature vector of the current USER.
The USER picture writing style feature vector generation module calculates a difference value between the USER test picture text description feature vector TPCVE _ ID _ USER and a test picture reference feature vector TPCVE _ ID corresponding to the test picture, and uses the difference value (TPCVE _ ID _ USER-TPCVE _ ID) as the USER picture writing style feature vector TPCVE _ USER.
According to a specific embodiment of the invention, a user test article word description characteristic value generation module acquires a user test article description text and generates a user test article word description characteristic value; the content of the text description characteristic value of the user test article is consistent with that of the text description characteristic value of the test article, and is not described again here. The user test article word description feature vector generating module calculates a user test article word description feature vector according to the user test article word description feature value; when the dimension of the test article textual description feature vector is n, the feature vector of the test article textual description for the article numbered ID of the current USER may be represented as: TTCVE _ ID _ USER = [ TTC _1_user, …, TTC _ m _ USER, …, TTC _ n _ USER ], where TTC _1 _useris the first item value in the USER test article textual description feature vector of the current USER, TTC _ m _ USER is the m-th item value in the USER test article textual description feature vector of the current USER, and TTC _ n _ USER is the n-th item value in the USER test article textual description feature vector of the current USER.
The USER article writing style feature vector generation module calculates a difference value between the text description feature vector TTCVE _ ID _ USER of the USER test article and a test article reference feature vector TPCVE _ ID corresponding to the test article, and uses the difference value (TTCVE _ ID _ USER-TTCVE _ ID) as the USER article writing style feature vector TTCVE _ USER.
According to a specific embodiment of the present invention, when a plurality of test pictures or a plurality of test articles are used, or one or a plurality of test pictures and one or a plurality of test articles are used simultaneously, a user test picture character description feature value generation module and a user test article character description feature value generation module generate a user test picture and/or an article character description feature value according to each test picture description text and test article description text of a user, respectively, and a user test picture character description feature vector generation module and a user test article character description feature vector generation module generate a user test picture and/or an article character description feature vector according to the user test picture and/or the article character description feature value, respectively; the user picture writing style characteristic vector generation module and the user article writing style characteristic vector generation module respectively calculate the difference value between each user test picture and/or article text description characteristic vector and the corresponding test picture and/or article reference characteristic vector; performing weighting operation on the difference values to respectively obtain a picture writing style feature vector TPCVE _ USER and an article writing style feature vector TTCVE _ USER of the USER; the USER writing style feature vector generation module performs weighted operation on a picture writing style feature vector TPCVE _ USER and an article writing style feature vector TTCVE _ USER of a USER to obtain a USER writing style feature vector TVE _ USER; the weight value of the weighted operation can be selected according to actual needs.
TVE_USER=TPCVE_USER*W P +TTCVE_USER*W T
Wherein, W P Writing a style feature vector TPCVE _ USER weighting coefficient for the USER picture; w T The USER article is written with a style feature vector TTCVE _ USER weighting factor. When the user performs only a picture composition test or an article composition test, the weighting coefficient of the participating item may be set to 1 and the weighting coefficient of the non-participating item may be set to 0. Preferably, the weights may be chosen to be equal.
The user writing style feature vector may be expressed as: TVE _ USER = [ TVE _1, …, TVE _ m, …, TVE _ n ], where TVE _1 is the first term value in the USER writing style feature vector, TVE _ m is the mth term value in the USER writing style feature vector, and TVE _ n is the nth term value in the USER writing style feature vector.
According to a specific embodiment of the invention, the user detection mode determination module is used for further prompting a user to upload a document to be checked; and the characteristic value generating module of the document to be audited is used for generating the characteristic value of the document to be audited. The characteristic values of the document to be audited include but are not limited to: chinese character number, foreign character number, total word number, real word number, virtual word number, paragraph length distribution condition, sentence number, sentence length distribution condition, synonym expansion condition, virtual word use condition, punctuation mark use condition, part of speech use condition. According to a specific embodiment of the invention, the number of Chinese words refers to the number of Chinese characters contained in each document to be audited except punctuations, and each Chinese word is marked as a character; the foreign language word number refers to the number of foreign language characters contained in each document to be audited except the punctuation marks, and each word of the foreign language is marked as a character; the word number refers to the total number of words obtained after word segmentation is carried out on each document to be examined, wherein Chinese word segmentation can be carried out by using a word segmentation library carried by the system, and foreign languages can be used for carrying out word segmentation by directly utilizing spaces between words according to writing habits of the foreign languages; the number of real words refers to the number of real words in each document to be checked, which is obtained by comparing the word segmentation result with the part of speech in the word segmentation library, and the number of real words can be further divided into the number of Chinese real words and the number of foreign language real words, wherein the sum of the number of Chinese real words and the number of foreign language real words is equal to the number of real words; the number of the virtual words refers to the number of the virtual words in each document to be checked, which is obtained by comparing the word segmentation result with the part of speech in the word segmentation library, and the number of the virtual words can be further divided into the number of Chinese virtual words and the number of foreign language virtual words, wherein the sum of the number of the Chinese virtual words and the number of the foreign language virtual words is equal to the number of the virtual words; the number of paragraphs refers to the number of paragraphs in each document to be audited; the paragraph length distribution condition refers to the number of words and the number of sentences contained in each paragraph in each document to be audited; the sentence number refers to the number of sentences in each document to be checked; the sentence length distribution condition refers to the number of words contained in each sentence in each document to be checked; synonym and synonym expansion conditions refer to the fact that the segmentation result in each document to be checked is compared with a synonym thesaurus, the same or similar meaning segmentation is formed into a set, the word number in each set is calculated, and therefore the writing habits of synonyms and synonyms of an author of the document to be checked are reflected, wherein if the number of words contained in the synonym or synonym set is more, the writing style of the author tends to be expanded by the synonym or the synonym, and if the number of words contained in the synonym or the synonym set is less, the writing style of the author tends not to be expanded by the synonym or the synonym; the usage condition of the dummy words refers to the statistical condition of the usage of the dummy words in each document to be checked, and includes but is not limited to the statistical ranking of the usage of the dummy words in each document to be checked, the number of words spaced between each different dummy words, and the number of words spaced between each same dummy word; for example, the usage of the three structural auxiliary words "in", "with" and "get" can be further counted, so as to reflect whether the author of the document to be reviewed uses the three structural auxiliary words "in", "with" and "get" separately or not; the punctuation mark use condition refers to the statistical condition of punctuation mark use in each document to be audited, including but not limited to the statistical ranking of punctuation mark use in each document to be audited, the word number of the interval between each different punctuation mark, and the word number of the interval between each same punctuation mark; the part-of-speech use condition refers to a statistical condition of each part-of-speech word in each document to be audited obtained by comparing the word segmentation result with the part-of-speech in the word segmentation library after word segmentation, for example, the quantity of nouns, verbs, adjectives, numerators, quantifiers, pronouns, adverbs, prepositions, conjunctions, auxiliary words, sighs and pseudonyms and the ratio of the quantity of each part-of-speech to the total quantity of the part-of-speech of the document to be audited are respectively obtained.
According to a specific embodiment of the invention, the feature vector generation module of the feature value of the document to be audited generates the feature vector of the document to be audited according to the feature value of the document to be audited. According to a specific embodiment of the invention, the system specifies the dimension of the feature vector of the document to be audited, and the specific content and arrangement sequence of each item in the feature vector; the dimension of the feature vector of the document to be checked and the specific content and arrangement sequence of each item in the feature vector should be consistent with the dimension of the reference feature vector of the test picture and the reference feature vector of the test article and the meaning and sequence of each feature value. When the dimension of the feature vector of the document to be audited is n, the dimension may be represented as TDCVE _ USER = [ TDC _1, …, TDC _ m, …, TDC _ n ], where TDC _1 is a first entry value in the feature vector of the document to be audited, TDC _ m is an mth entry value in the feature vector of the document to be audited, and TDC _ n is an nth entry value in the feature vector of the document to be audited.
Preferably, the feature vector of the document to be audited includes a ratio of Chinese word number to total word number, a ratio of foreign word number to total word number, a ratio of real word number to total word number, a ratio of imaginary word number to total word number, a ratio of total word number to paragraph number, a ratio of longest paragraph word number, a ratio of synonym number, synonym number to total word number, a ratio of punctuation number to total word number, a ratio of noun number to total word number, a ratio of verb number to total word number, a ratio of adjective number to total word number, a ratio of digital word number to total word number, a ratio of quantifier number to total word number, a ratio of pronoun number to total word number, a ratio of adverb number to total word number, a ratio of conjunct number to total word number, a ratio of auxiliary word number to total word number, a ratio of exclamation number to total word number, and a ratio of dummy word number to total word number.
The user writing style similarity calculation module is used for calculating the current user writing style similarity, and can calculate through the following formula:
the user writing style similarity judging module judges the current user writing style similarity Sim T (USER) comparing with a self-auditing threshold preset by the system; when the user writes style similarity Sim T When the USER is higher than the self-auditing threshold, the document to be audited submitted by the current USER is considered to be inconsistent with the writing style of the USER; when the user writes style similarity Sim T When the (USER) is lower than the self-auditing threshold, the document to be audited submitted by the current USER is considered to be consistent with the writing style of the USER.
The self-auditing threshold is set in advance for the system. If the self-auditing threshold value is set to be too high, misjudgment that the document to be audited submitted by the current user is inconsistent with the writing style of the user is easily caused; if the self-audit threshold value is set too low, it is easy to misjudge that the document to be audited submitted by the current user is consistent with the writing style of the user. Usually, the self-checking threshold value is selected and verified by the system through experiments in advance, and the system can be adjusted at any time according to the operation condition.
According to an embodiment of the present invention, a first self-auditing threshold and a second self-auditing threshold may be set respectively; the first self-audit threshold is higher than a second self-audit threshold; when the user writes style similarity Sim T When the (USER) is higher than the first self-auditing threshold, the document to be audited submitted by the current USER is considered to be inconsistent with the writing style of the USER; when the user writes style similarity Sim T (USER) when the self-auditing threshold is lower than the second self-auditing threshold, the document to be audited submitted by the current USER is considered to be consistent with the writing style of the USER; when the user writes style similarity Sim T (USER) is higher than or equal to the second self-audit threshold and lower than or equal to the first self-audit threshold; the user writing style is further verified.
The first self-auditing threshold and the second self-auditing threshold are set in advance for the system. If the first self-auditing threshold value is set to be too high, misjudgment that the document to be audited submitted by the current user is inconsistent with the writing style of the user is easily caused; if the self-auditing threshold value is set too low, the misjudgment that the document to be audited submitted by the current user is consistent with the writing style of the user is easily caused; if the interval between the first self-auditing threshold and the second self-auditing threshold is set to be too large, the user writing style is easily verified too much again. Usually, the first self-auditing threshold value and the second self-auditing threshold value are selected and verified by the system through experiments in advance, and the system can be adjusted at any time according to the operating condition.
According to a specific embodiment of the present invention, said further verifying the user writing style refers to a user writing style structure assistant word judging module; and judging the use conditions of the three structural auxiliary words of 'ground' and 'get' in the description text of the document to be audited and the description text of the user test picture and/or the description text of the user test article, thereby reflecting the distinguishing degrees of the author of the document to be audited and the current user for the three structural auxiliary words of 'ground' and 'get'. The user writing style structure auxiliary word judging module judges the use conditions of three structure auxiliary words of 'ground' and 'obtained' of the document to be checked, namely, the use times of 'ground' and 'obtained' in the whole text of the document to be checked are counted and respectively marked as T 1 、T 2 And T 3 (ii) a Further counting the times of nouns which are the parts of speech of the participles followed by the word in the whole text of the document to be checked, and marking the times as D 1 (ii) a Counting the number of times that the part of speech of the participle following the 'ground' in the document to be audited is a verb, and marking as D 2 (ii) a Counting the times of adjectives of the part of speech of the participle following the 'ground' in the whole text of the document to be audited, and recording the times as D 3 (ii) a Calculating the ratio D of the number of times that the part of speech of the word is noun to the total number of times that the word is used 1 /T 1 (ii) a Calculating the ratio D of the number of times of verbs to the total number of times of using the ' ground ' in the whole text according to the part of speech of the participle following the ' ground 2 /T 2 (ii) a Calculating the number of times that the part of speech of the participle following the 'get' is the verb and the total number of times of using the 'get' in the whole textRatio D of 3 /T 3 (ii) a Calculating the "ground" and "de" distinguishing coefficients DC _ TD. The value of the discrimination coefficient DC _ TD is greater than or equal to 0 and less than or equal to 3.
The use of the three structural auxiliary words "ground" and "de" in the user test picture description text and/or the user test article description text means that the use times of "ground" and "de" in the user test picture description text and/or the user test article description text full text (if the user tests multiple pictures and/or multiple articles, all description texts are combined as the full text) are counted and respectively marked as T 1 ’、T 2 ' and T 3 '; further counting the times that the part of speech of the participle followed by the part of speech of the whole text of the document to be checked is a noun and recording the number of times as D 1 '; counting the number of times that the part of speech of the participle following the 'ground' in the document to be audited is a verb, and marking as D 2 '; counting the times of adjectives of the part of speech of the participle following the 'ground' in the whole text of the document to be audited, and recording the times as D 3 '; calculating the ratio D of the number of times that the part of speech of the word is noun to the total number of times that the word is used 1 ’/T 1 '; calculating the ratio D of the number of times of verbs to the total number of times of using the ' ground ' in the whole text according to the part of speech of the participle following the ' ground 2 ’/T 2 '; calculating the ratio D of the number of verbs of the participles following the 'get' to the total number of uses of the 'get' in the whole text 3 ’/T 3 '; calculating "ground", "ground" and "d" distinguishes the coefficients DC _ TPT. The value of the discrimination coefficient DC _ TPT is greater than or equal to 0 and less than or equal to 3.
The user writing style structure auxiliary word judging module; the offset DC-SC between the discrimination coefficient DC _ TD and the discrimination coefficient DC _ TPT is calculated, that is, the absolute value of the difference between the discrimination coefficient DC _ TD and the discrimination coefficient DC _ TPT is normalized.
When the value of the DC _ SC is less than or equal to the judgment threshold of the offset DC-SC, the user writing style structure auxiliary word judgment module judges that the author of the document to be audited is consistent with the use style of the three structure auxiliary words of the test picture description text and/or the test article description text; when the value of the DC _ SC is larger than the judgment threshold of the offset DC-SC, the user writing style structure assistant word judgment module judges that the author of the document to be audited is inconsistent with the use style of the three structure assistant words of the test picture description text and/or the test article description text. The judgment threshold value of the offset DC-SC is set by the system in advance and can be adjusted at any time according to actual needs. According to experimental data of early-stage operation of the system, when the value of the DC _ SC is less than or equal to 10%, the situation that an author of a document to be audited is consistent with the use style of three structural auxiliary words of a test picture description text and/or a test article description text by a user can be well reflected; when the value of the DC _ SC is greater than 10%, it may be considered that the author of the document to be audited is inconsistent with the use of the three structural auxiliary words "of", "ground" and "d" by the user of the test picture description text and/or the test article description text.
The user writing style judging module is used for judging the similarity Sim of the writing style of the user T (USER) is higher than or equal to the second self-audit threshold and lower than or equal to the first self-audit threshold; further judging whether the document to be checked submitted by the current user is consistent with the writing style of the user through the offset DC-SC; when the deviation degree DC-SC is larger than the judgment threshold of the deviation degree DC-SC, the document to be audited submitted by the current user is considered to be inconsistent with the writing style of the user(ii) a And when the deviation degree DC-SC is less than or equal to the judgment threshold of the deviation degree DC-SC, the document to be audited submitted by the current user is considered to be consistent with the writing style of the user.
According to one embodiment of the invention, the user access pattern detection module prompts the user to upload the document to be authenticated.
When the user detection mode determining module judges that the current user detection mode is the common plagiarism identification mode, the document word segmentation module to be identified is used for segmenting words of a document to be identified to obtain word segmentation results; when the word segmentation processing is carried out on the document to be identified, the same processing flow as the word segmentation processing carried out on the materials of the comparison library is needed.
According to a specific embodiment of the invention, the document to be identified is divided into word parts and part of speech classification module; and the method is used for further obtaining the part of speech corresponding to the word segmentation result. And the word segmentation part-of-speech classification mode is consistent with the word segmentation classification mode of the materials recorded in the comparison library.
According to a specific embodiment of the invention, the document word segmentation characteristic value generation module to be identified is used for generating a document word segmentation characteristic value to be identified; counting the number of each participle appearing in the corresponding document to be identified to obtain a participle characteristic value WCV _ TBI = [ W _ ID, W _ N ] corresponding to each participle, wherein W _ ID represents the unique number of the participle in a participle library, and W _ N represents the total number of times of the participle appearing in the document to be identified. Preferably, in consideration of the part of speech of each participle, a participle part of speech characteristic value WCCV _ TBI = [ W _ ID, W _ N, W _ CHAR ], wherein W _ ID represents a unique number of the participle in the participle library, W _ N represents a total number of times of participles of the specific participle in the document to be identified, and W _ CHAR represents the part of speech of the participle.
According to one embodiment of the invention, the to-be-identified document word segmentation tight coefficient generation module is used for generating a to-be-identified document word segmentation tight coefficient. <xnotran> , WGC _ TBI = [ G _ W _ ID _1,G_W_ID_2, 5363 zxft 5363, G _ W _ ID _ (W _ N-1) ], , G _ W _ ID _1 , G _ W _ ID _2 , G _ W _ ID _ (W _ N-1) W _ N-1 W _ N ; </xnotran> G _ W _ ID _1, G _W _ID _2, …, G _ W _ ID _ (W _ N-1) are the word segmentation compact coefficients corresponding to the word segmentation. According to an embodiment of the present invention, the participle compactness coefficient corresponding to each participle may be further represented in a vector form as a participle compactness coefficient feature vector WGCVE _ TBI = [ W _ ID, W _ N, W _ CHAR, G _ W _ ID _1, G _W _ID _2, …, G _ W _ ID _ (W _ N-1), where W _ ID represents the unique number of the participle in the participle library, W _ N represents the total number of participles of the particular participle in the document to be authenticated, W _ CHAR represents the part of speech of the participle, G _ W _ ID _1 represents the number of participles separated between the first occurrence and the second occurrence of the participle in the document to be authenticated, G _ W _ ID _2 represents the number of participles separated between the second occurrence and the third occurrence of the participle in the document to be authenticated, and G _ W _ ID _ (W _ N-1) represents the number of participles separated between the W _ N-1 occurrence and the W _ N occurrence of the participle in the document to be authenticated. Wherein, G _ W _ ID _1, G _W _ID _2, …, G _ W _ ID _ (W _ N-1) are the word segmentation part-of-speech feature vector compact coefficients corresponding to the word segmentation. Through the compact coefficient of the word segmentation feature vector, the overall distribution condition of the specific word segmentation in the corresponding document to be identified can be obtained, so that under the condition that the overall space of the document to be identified is too long or the description viewpoints are dispersed, the condition that the key word segmentation feature value is omitted due to the fact that the word segmentation feature vector is screened according to the total word segmentation times W _ N or (W _ N/word segmentation free vector dimension WFV) is avoided. Preferably, a specific part in a certain document to be identified can be extracted for comparison according to the word segmentation feature vector compactness coefficient.
According to a specific embodiment of the invention, the module for determining the dimension of the word segmentation free vector of the document to be identified is used for determining the dimension of the word segmentation free vector WFV _ TBI according to the word segmentation result of the document to be identified. When the space of the document to be identified is shorter or the word segmentation result in the document is less, the obtained word segmentation free vector dimension WFV-TBI is less; when the space of the document to be identified is longer or the word segmentation result in the document is more, the obtained word segmentation free vector dimension WFV _ TBI is more.
When the user detection mode determining module judges that the current user detection mode is the extended plagiarism identification mode, the document to be identified is segmented by the word segmentation module to obtain a word segmentation result; the same or similar meaning of the participles form a group, and the group is used as a unit for numbering. A plurality of participles with the same or similar meanings correspond to a participle group number; when the word segmentation processing is carried out on the document to be identified, the same processing flow as the word segmentation processing carried out on the materials of the comparison library is needed.
According to a specific embodiment of the invention, the document to be identified is divided into groups and parts of speech classification modules; and the method is used for further obtaining the part of speech corresponding to the word segmentation result. And the word classification mode of the word groups is consistent with the word classification mode of the material recorded in the comparison library.
According to a specific embodiment of the invention, the document to be identified is divided into word group characteristic values, and the word group characteristic values are generated by a generating module; counting the number of each word segmentation group appearing in the corresponding document to be identified to obtain a word segmentation characteristic value WGCV _ TBI = [ WG _ ID, WG _ N ] corresponding to each word segmentation group, wherein the WG _ ID represents the unique number of the word segmentation group in a word segmentation library, and the WG _ N represents the total number of times of the word segmentation group appearing in the document to be identified. Preferably, in consideration of the part of speech of each word segmentation group, a word segmentation group part of speech characteristic value WGCCV _ TBI = [ WG _ ID, WG _ N, WG _ CHAR ], wherein WG _ ID represents a unique number of the word segmentation group in a word segmentation library, WG _ N represents a total number of times of word segmentation of the specific word segmentation group in the document to be authenticated, and WG _ CHAR represents the part of speech of the word segmentation group.
According to a specific embodiment of the invention, the document word segmentation group compactness coefficient generation module to be identified is used for generating the document word segmentation compactness coefficient to be identified. According to an embodiment of the present invention, the participle compactness coefficient corresponding to each participle group may be represented as WGGC _ TBI = [ G _ WG _ ID _1, G _wg _ ID _2, …, G _ WG _ ID _ (WG _ N-1) ], where G _ WG _ ID _1 represents the number of participles separated between the first occurrence and the second occurrence of the participle group in the document to be authenticated, G _ WG _ ID _2 represents the number of participles separated between the second occurrence and the third occurrence of the participle group in the document to be authenticated, and G _ WG _ ID _ (WG _ N-1) represents the number of participles separated between the W _ N-1 occurrence and the W _ N occurrence of the participle group in the document to be authenticated; g _ WG _ ID _1, G _WG _ID _2, …, G _ WG _ ID _ (WG _ N-1) are the compact coefficients of the word-segmentation group corresponding to the word-segmentation group. According to an embodiment of the present invention, the phrase group compact coefficients corresponding to each phrase group can be further represented in a vector form as a phrase group compact coefficient feature vector WGGCVE _ TBI = [ WG _ ID, WG _ N, WG _ CHAR, G _ WG _ ID _1, G \uwg _id \u2, …, G _ WG _ ID _ (WG _ N-1) ], where WG _ ID represents a unique number of the phrase group in the phrase bank, WG _ N represents a total number of times of the phrase of the specific phrase group in the document to be authenticated, WG _ CHAR represents the part of speech of the participle group, G _ WG _ ID _1 represents the number of participles separated between the first occurrence and the second occurrence of the participle group in the document to be authenticated, G _ WG _ ID _2 represents the number of participles separated between the second occurrence and the third occurrence of the participle group in the document to be authenticated, and G _ WG _ ID _ (WG _ N-1) represents the number of participles separated between the W _ N-1 occurrence and the W _ N-th occurrence of the participle group in the document to be authenticated. Wherein, G _ WG _ ID _1, G _WG _ID _2, …, G _ WG _ ID _ (WG _ N-1) are the compact coefficients of the part-word characteristic vectors corresponding to the word groups. Through the compact coefficient of the word segmentation group feature vector, the overall distribution condition of a specific word segmentation group in the corresponding document to be identified can be obtained, so that under the condition that the overall space of the document to be identified is too long or the description viewpoints are scattered, the condition that the word segmentation feature vector is screened according to the total word segmentation times W _ N or (W _ N/word segmentation free vector dimension WFV) to omit key word segmentation feature values is avoided. Preferably, a specific part in a certain document to be identified can be extracted for comparison according to the word segmentation feature vector compactness coefficient.
According to a specific embodiment of the invention, the module for determining the dimension of the word segmentation free vector of the document to be identified is used for determining the dimension of the word segmentation free vector WGFV-TBI according to the word segmentation result of the document to be identified. When the space of the document to be identified is shorter or the word segmentation result in the document is less, the obtained word segmentation group free vector dimension WGFV-TBI is less; when the space of the document to be identified is longer or the word segmentation result in the document is more, the obtained word segmentation group free vector dimension WGFV-TBI is more.
When the user detection mode determining module judges that the current user detection mode is the multilingual plagiarism identification mode, the Chinese and foreign language word segmentation module of the document to be identified is used for segmenting words of the document to be identified to obtain Chinese and foreign language word segmentation results; the Chinese and foreign language participles with the same or similar meanings form a group, and the group is used as a unit for numbering. A plurality of Chinese and foreign language participles with the same or similar meanings correspond to a Chinese and foreign language participle group number. When the word segmentation processing is carried out on the document to be identified, the same processing flow as the word segmentation processing carried out on the materials of the comparison library is needed.
According to a specific embodiment of the invention, the document to be identified is divided into word classification modules; and the word property corresponding to the word segmentation result is further obtained. And the word classification mode of the word groups is consistent with the word classification mode of the material recorded in the comparison library.
According to a specific embodiment of the invention, the module for generating the characteristic values of the Chinese and foreign language word groups in the document to be identified is used for generating the characteristic values of the Chinese and foreign language word groups in the document to be identified; counting the number of the Chinese and foreign language word groups appearing in the corresponding document to be identified to obtain a word segmentation characteristic value WFGCV _ TBI = [ WFG _ ID, WFG _ N ] corresponding to each Chinese and foreign language word group, wherein the WFG _ ID represents the unique number of the Chinese and foreign language word group in a word segmentation library, and the WFG _ N represents the total number of the Chinese and foreign language word groups appearing in the document to be identified. Preferably, in consideration of the part of speech of each Chinese and foreign language word segmentation group, a Chinese and foreign language word segmentation group characteristic value WFGCCV _ TBI = [ WFG _ ID, WFG _ N, WFG _ CHAR ], where FWG _ ID represents a unique number of the Chinese and foreign language word segmentation group in a word segmentation library, WFG _ N represents a total number of times of segmentation of the specific Chinese and foreign language word segmentation group in the document to be authenticated, and WFG _ CHAR represents the part of speech of the Chinese and foreign language word segmentation group.
According to a specific embodiment of the present invention, the module for generating the compact coefficients of the Chinese and foreign word segmentation groups in the document to be identified is used for generating the compact coefficients of the Chinese and foreign word segmentation groups in the document to be identified. <xnotran> , WFGGC _ TBI = [ G _ WFG _ ID _1,G_WFG_ID_2, 5363 zxft 5363, G _ WFG _ ID _ (WFG _ N-1) ], , G _ WFG _ ID _1 , G _ WFG _ ID _2 , G _ WFG _ ID _ (WFG _ N-1) W _ N-1 W _ N ; </xnotran> G _ WFG _ ID _1, G _WFG _ID _2, …, G _ WFG _ ID _ (WFG _ N-1) are the Chinese and foreign phrase compactness coefficients corresponding to the Chinese and foreign phrases. According to an embodiment of the present invention, the chinese-foreign language phrase compact coefficient corresponding to each chinese-foreign language phrase may be further expressed in a vector form as a chinese-foreign language phrase compact coefficient feature vector WFGGCVE _ TBI = [ WFG _ ID, WFG _ N, WFG _ CHAR, G _ WFG _ ID _1, G _WFG _ID _2, …, G _ WFG _ ID _ (WFG _ N-1) ], wherein WFG _ ID represents the unique number of the Chinese and foreign word-separating phrase in the word-separating library, WFG _ N represents the total times of word-separating of the specific Chinese and foreign word-separating phrase in the document to be identified, WFG _ CHAR represents the part of speech of the Chinese and foreign language phrase, G _ WFG _ ID _1 represents the number of participles spaced between the first occurrence and the second occurrence of the Chinese and foreign language phrase in the document to be identified, G _ WFG _ ID _2 represents the number of participles spaced between the second occurrence and the third occurrence of the Chinese and foreign language phrase in the document to be identified, and G _ WFG _ ID _ (WG _ N-1) represents the number of participles spaced between the W _ N-1 occurrence and the W _ N occurrence of the Chinese and foreign language phrase in the document to be identified. Wherein, G _ WFG _ ID _1, G _WFG _ID _2, …, G _ WFG _ ID _ (WFG _ N-1) are the compact coefficients of the part-of-speech feature vectors corresponding to the Chinese and foreign word groups. Through the Chinese and foreign language word group feature vector compact coefficient, the overall distribution condition of a specific Chinese and foreign language word group in a corresponding document to be identified can be obtained.
According to a specific embodiment of the invention, the module for determining the free vector dimension of Chinese and foreign language word segmentation groups in the document to be identified is used for determining the free vector dimension WFGFV _ TBI of the Chinese and foreign language word segmentation groups according to the word segmentation result of the document to be identified. When the space of the document to be identified is shorter or the word segmentation result in the document is less, the obtained Chinese and foreign word segmentation group free vector dimension WFGFV _ TBI is less; when the space of the document to be identified is longer or the word segmentation result in the document is more, the obtained word segmentation group free vector dimension WFGFV _ TBI is more.
According to a specific embodiment of the invention, the word segmentation simplified vector dimension generation module of the document to be identified is used for simplifying the word segmentation free vector dimension WFV-TBI of the document to be identified and generating the word segmentation simplified vector dimension RWV-TBI of the document to be identified. The word segmentation reduced vector dimension RWV _ TBI is specified by the system. Preferably, the system specifies a participle reduced vector dimension RWV _ TBI of 500. Preferably, the system specifies a participle reduced vector dimension RWV _ TBI of 800. Preferably, the reduction system specifies a participle reduced vector dimension RWV _ TBI of 1000.
According to a specific embodiment of the invention, the word segmentation simplified vector dimension generation module of the document to be identified adopts an equal interval extraction method to simplify the word segmentation free vector dimension WFV-TBI of the document to be identified. The simplification process is as follows: judging whether the word segmentation free vector dimension WFV _ TBI of the document to be identified is larger than the word segmentation simplified vector dimension RWV _ TBI of the document to be identified, if so, dividing the word segmentation free vector dimension WFV _ TBI of the document to be identified by the word segmentation simplified vector dimension RWV _ TBI of the document to be identified specified by the simplification system, and performing rounding operation on the obtained quotient value to further obtain a simplification coefficient REDU _ TBI of the document to be identified; extracting one characteristic value at intervals of REDU _ TBI-1 from the characteristic values corresponding to the document word segmentation free vector dimension WFV _ TBI to be identified; after all the characteristic values are extracted, judging whether the number of the extracted characteristic values is equal to the word segmentation reduced vector dimension RWV _ TBI of the document to be identified; when the number of the extracted characteristic values is equal to the word segmentation simplified vector dimension RWV _ TBI of the document to be identified, the word segmentation free vector dimension WFV _ TBI simplification of the document to be identified is completed; when the number of the extracted characteristic values is smaller than the document word segmentation reduced vector dimension RWV _ TBI to be identified, calculating the difference value between the document word segmentation reduced vector dimension RWV _ TBI to be identified and the number of the characteristic values; and randomly extracting the characteristic values with the difference quantity equal to the word segmentation reduced vector dimension RWV _ TBI of the document to be identified and the characteristic values from the unextracted characteristic values to finish the reduction of the word segmentation free vector dimension WFV _ TBI of the document to be identified.
According to a specific embodiment of the invention, the word segmentation simplified vector dimension generation module of the document to be identified adopts a part-of-speech screening method to simplify the word segmentation free vector dimension WFV-TBI of the document to be identified. The simplification process is as follows: classifying the characteristic values according to the corresponding word segmentation parts of speech; according to an embodiment of the present invention, the feature values are divided into a class A1 real word feature value, a class A2 real word feature value, a class B real word feature value, a class C real word feature value, a class D real word feature value, and a class V particle feature value. It is generally considered that the similarity contrast of the feature values corresponding to the real words plays a greater role, wherein the term noun in the professional art is more effective than the general noun in representing the effective content of the document to be identified. The number of feature values in each category, i.e., estimate _ A1 (the number of real-word feature values in a category 1), estimate _ A2 (the number of real-word feature values in a category 2), estimate _ B (the number of real-word feature values in a category B), estimate _ C (the number of real-word feature values in a category C), estimate _ D (the number of real-word feature values in a category D), and estimate _ V (the number of real-word feature values in a category V) are counted. Calculating a value RWV _ TBI _ S _ V of a document word segmentation reduced vector dimension RWV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) to be identified; if the sum is more than 0, the simplification is quitted; if the sum is equal to 0, the simplification is finished; if the word segmentation reduced vector dimension is less than 0, further calculating a value RWV _ S _ D of a document to be identified word segmentation reduced vector dimension RWV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D); if the difference value is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWV _ TBI _ S _ D from the feature values corresponding to AMOUNT _ V to finish the simplification; if the sum is equal to 0, the simplification is finished; if the word segmentation reduced vector dimension is less than 0, further calculating a value RWV _ TBI _ S _ C of a document to be identified word segmentation reduced vector dimension RWV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C); if the difference value is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWV _ TBI _ S _ C from the feature values corresponding to AMOUNT _ D, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the word segmentation reduced vector dimension is less than 0, further calculating a value RWV _ TBI _ S _ B of the document to be identified word segmentation reduced vector dimension RWV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B); if the difference value is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWV _ TBI _ S _ B from the feature values corresponding to AMOUNT _ C to finish the simplification; if the sum is equal to 0, the simplification is finished; if the word segmentation reduced vector dimension is less than 0, further calculating the value RWV _ TBI _ S _ A2 of the word segmentation reduced vector dimension RWV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2) of the document to be identified; if the difference value is larger than 0, randomly extracting feature values with the same number as the difference value RWV _ TBI _ S _ A2 from the feature values corresponding to AMOUNT _ B to finish the simplification; if the sum is equal to 0, the simplification is finished; if the word segmentation is smaller than 0, further calculating the value RWV _ TBI _ S _ A1 of the word segmentation reduced vector dimension RWV _ TBI-AMOUNT _ A1 of the document to be identified; if the difference value is larger than 0, randomly extracting feature values with the same number as the difference value RWV _ TBI _ S _ A1 from the feature values corresponding to AMOUNT _ A2 to finish the simplification; if the sum is equal to 0, the simplification is finished; if the number of the word segmentation reduced vector dimension RWV _ TBI is smaller than 0, the feature values with the number equal to that of the word segmentation reduced vector dimension RWV _ TBI of the document to be identified are randomly extracted from the feature values corresponding to AMOUNT _ A1, and the reduction is completed.
For the case that the value RWV _ TBI _ S _ V of the reduced vector dimension RWV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) for calculating the word segmentation of the document to be authenticated is greater than 0, that is, the document to be authenticated has a smaller size or less information content, and therefore, the document to be authenticated is not suitable for comparison by using the feature value.
When the word segmentation free vector dimension WFV-TBI of the document to be identified is smaller than the word segmentation reduced vector dimension RWV-TBI of the document to be identified, the dimension of the word segmentation free vector dimension WFV-TBI is small, the magnitude values under other dimensions are equal to 0, and the word segmentation free vector dimension WFV-TBI and the magnitude values can be directly labeled in a system and independently recorded and processed.
According to a specific embodiment of the invention, the word segmentation group reduced vector dimension generation module of the document to be identified is used for reducing the word segmentation group free vector dimension WGFV _ TBI of the document to be identified and generating the word segmentation group reduced vector dimension RGWV _ TBI of the document to be identified. The reduced set of words vector dimension RWGV _ TBI is specified by the system. Preferably, the system specifies a reduced word-group vector dimension RWGV _ TBI of 500. Preferably, the system specifies a reduced word-group vector dimension RWGV _ TBI of 800. Preferably, the reduction system specifies a reduced vector dimension RWGV _ TBI of 1000.
According to a specific embodiment of the invention, the word group reduction vector dimension generation module of the document to be identified adopts an equal-interval extraction method to reduce the free vector dimension WGFV-TBI of the word group of the document to be identified. The simplification process is as follows: judging whether the free vector dimension WGFV-TBI of the word-segmentation group of the document to be identified is larger than the reduced vector dimension RWGV-TBI of the word-segmentation group of the document to be identified, if so, dividing the free vector dimension WGFV-TBI of the word-segmentation group of the document to be identified by the reduction system to specify the reduced vector dimension RWGV-TBI of the word-segmentation group of the document to be identified, and performing rounding operation on the obtained quotient to further obtain a reduced coefficient REDU-TBI; extracting a characteristic value at intervals of REDU _ TBI-1 from the characteristic values corresponding to the free vector dimension WGFV of the word segmentation group of the document to be identified; after all the characteristic values are extracted, judging whether the quantity of the extracted characteristic values is equal to the word group reduced vector dimension RWGV-TBI of the document to be identified; when the number of the extracted characteristic values is equal to the word group reduced vector dimension RWGV-TBI of the document to be identified, finishing word group free vector dimension WGFV-TBI reduction of the document to be identified; when the number of the extracted characteristic values is smaller than the simplified vector dimension RWGV-TBI of the document word group to be identified, calculating the difference value between the simplified vector dimension RWGV-TBI of the document word group to be identified and the number of the characteristic values; and randomly extracting characteristic values with the same quantity as the difference value of the simplified vector dimension RWGV-TBI of the word group of the document to be identified and the characteristic values from the unextracted characteristic values to finish the simplification of the free vector dimension WGFV-TBI of the word group of the document to be identified.
According to a specific embodiment of the invention, the word group reduction vector dimension generation module of the document to be identified adopts a part-of-speech screening method to reduce the free vector dimension WGFV-TBI of the word group of the document to be identified. The simplification process is as follows: classifying the characteristic values according to corresponding word segmentation and word formation; according to an embodiment of the present invention, the feature values are divided into a class A1 real word feature value, a class A2 real word feature value, a class B real word feature value, a class C real word feature value, a class D real word feature value, and a class V particle feature value. It is generally considered that the similarity of the feature values corresponding to the real words plays a greater role in comparison, wherein the term noun is more effective in representing the effective content of the document to be authenticated than the general term noun. The number of feature values, AMOUNT _ A1 (the number of class A1 real-word feature values), AMOUNT _ A2 (the number of class A2 real-word feature values), AMOUNT _ B (the number of class B real-word feature values), AMOUNT _ C (the number of class C real-word feature values), AMOUNT _ D (the number of class D real-word feature values), and AMOUNT _ V (the number of class V real-word feature values) in each category are counted. Calculating a value RWGV _ TBI _ S _ V of a reduced vector dimension RWGV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) of the document word segmentation group to be identified; if the sum is more than 0, the simplification is quitted; if the sum is equal to 0, the simplification is finished; if the word group reduced vector dimension RWGV _ TBI of the document to be authenticated is smaller than 0, further calculating the value RWGV _ S _ D of the word group reduced vector dimension RWGV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D); if the difference value RWGV is larger than 0, randomly extracting the characteristic values with the quantity equal to that of the difference value RWGV _ TBI _ S _ D from the characteristic values corresponding to the AMOUNT _ V, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the word group reduced vector dimension RWGV _ TBI is smaller than 0, further calculating the value RWGV _ TBI _ S _ C of the document to be identified, namely the word group reduced vector dimension RWGV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C); if the difference value RWGV _ TBI _ S _ C is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWGV _ TBI _ S _ C from the feature values corresponding to the AMOUNT _ D, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the word group number of the document to be authenticated is smaller than 0, further calculating the value RWGV _ TBI _ S _ B of the reduced vector dimension RWGV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B); if the difference value is larger than 0, randomly extracting the feature values with the same quantity as the difference value RWGV _ TBI _ S _ B from the feature values corresponding to AMOUNT _ C to finish the simplification; if the sum is equal to 0, the simplification is finished; if the word group number of the document to be identified is smaller than 0, further calculating a value RWV _ TBI _ S _ A2 of a reduced vector dimension RWGV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2) of the word group of the document to be identified; if the difference value RWGV _ TBI _ S _ A2 is larger than 0, randomly extracting the feature values with the number equal to that of the difference value RWGV _ TBI _ S _ A2 from the feature values corresponding to the AMOUNT _ B, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the word group number of the document to be identified is smaller than 0, further calculating the value RWGV _ TBI _ S _ A1 of the simplified vector dimension RWGV _ TBI-AMOUNT _ A1 of the word group of the document to be identified; if the difference value is larger than 0, randomly extracting feature values with the number equal to that of the difference value RWGV _ TBI _ S _ A1 from the feature values corresponding to the AMOUNT _ A2 to finish the simplification; if the sum is equal to 0, the simplification is finished; if the number of the characteristic values is smaller than 0, randomly extracting the characteristic values with the number equal to that of the reduced vector dimension RWGV _ TBI of the word group of the document to be identified from the characteristic values corresponding to the AMOUNT _ A1, and finishing the reduction.
For the case that the value RWGV _ TBI _ S _ V of the reduced vector dimension RWGV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) of the word segmentation group of the document to be authenticated is greater than 0, it means that the document to be authenticated has a small space or a small information AMOUNT, and therefore, it is not suitable for comparison by using the feature value.
When the word group free vector dimension WGFV-TBI of the document to be identified is smaller than the word group reduced vector dimension RWGV-TBI of the document to be identified, the dimension of the document to be identified is small, and the magnitude values under other dimensions are equal to 0, and the document to be identified can be directly marked in the system and independently collected and recorded.
According to a specific embodiment of the invention, the Chinese and foreign language word segmentation reduced vector dimension generation module is used for reducing the Chinese and foreign language word segmentation free vector dimension WFGFV _ TBI of the document to be identified and generating the Chinese and foreign language word segmentation reduced vector dimension RFGWV _ TBI of the document to be identified. The Chinese-foreign word-group reduced vector dimension RWFFV _ TBI is specified by the system. Preferably, the system specifies a Chinese-foreign participle-group reduced vector dimension RWFFVVTBI of 500. Preferably, the system specifies a Chinese-foreign participle-group reduced vector dimension RWFFVVTBI of 800. Preferably, the reduction system specifies a Chinese-foreign participle-group reduced vector dimension RWFFVVTBI of 1000.
According to a specific embodiment of the invention, the Chinese and foreign language word segmentation and reduction vector dimension generation module in the document to be identified adopts an equal interval extraction method to reduce the free vector dimension WFGFV _ TBI of the Chinese and foreign language word segmentation in the document to be identified. The simplification process is as follows: judging whether the free vector dimension WFGFV _ TBI of the Chinese and foreign language word segmentation groups in the document to be identified is larger than the simplified vector dimension RWFFVTBI of the Chinese and foreign language word segmentation groups in the document to be identified, if so, dividing the free vector dimension WFGFV _ TBI of the Chinese and foreign language word segmentation groups in the document to be identified by the simplifying system to specify the simplified vector dimension RWFFVTBI of the Chinese and foreign language word segmentation groups in the document to be identified, and performing rounding operation on the obtained quotient value to further obtain a simplified coefficient RED _ TBI; extracting a characteristic value from characteristic values corresponding to the free vector dimension WFGFV of the Chinese and foreign language word groups in the document to be identified at intervals of REDU _ TBI-1; after all the characteristic values are extracted, judging whether the quantity of the extracted characteristic values is equal to the Chinese and foreign word segmentation reduced vector dimension RWFFV _ TBI of the document to be identified; when the number of the extracted characteristic values is equal to the Chinese and foreign language word group reduced vector dimension RWFFV _ TBI of the document to be identified, the Chinese and foreign language word group free vector dimension WFGFV _ TBI reduction of the document to be identified is completed; when the number of the extracted characteristic values is smaller than the simplified vector dimension RWFFVBV _ TBI of the Chinese word-segmentation groups in the document to be identified, calculating the difference value between the simplified vector dimension RWFBV _ TBI of the Chinese word-segmentation groups in the document to be identified and the number of the characteristic values; and randomly extracting characteristic values with the number equal to the difference value between the simplified vector dimension RWFFGV _ TBI of the Chinese word segmentation group in the document to be identified and the characteristic values from the unextracted characteristic values, and completing the simplification of the free vector dimension WFGFV _ TBI of the Chinese word segmentation group in the document to be identified.
According to a specific embodiment of the invention, the Chinese and foreign language word segmentation reduced vector dimension generation module in the document to be identified adopts a part-of-speech screening method to reduce the free vector dimension WFGFV _ TBI of the Chinese and foreign language word segmentation in the document to be identified. The simplification process is as follows: classifying the characteristic values according to the corresponding Chinese and foreign language word classification; according to an embodiment of the present invention, the feature values are divided into a class A1 real word feature value, a class A2 real word feature value, a class B real word feature value, a class C real word feature value, a class D real word feature value, and a class V particle feature value. It is generally considered that the similarity of the feature values corresponding to the real words plays a greater role in comparison, wherein the term noun is more effective in representing the effective content of the document to be authenticated than the general term noun. The number of feature values in each category, i.e., estimate _ A1 (the number of real-word feature values in a category 1), estimate _ A2 (the number of real-word feature values in a category 2), estimate _ B (the number of real-word feature values in a category B), estimate _ C (the number of real-word feature values in a category C), estimate _ D (the number of real-word feature values in a category D), and estimate _ V (the number of real-word feature values in a category V) are counted. Calculating a value RWFFV _ TBI _ S _ V of a reduced vector dimension RWFFVFVFVFVSV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) of the document word group to be identified; if the sum is larger than 0, the simplification is quitted; if the sum is equal to 0, the simplification is finished; if the dimension RWFVV _ TBI is smaller than 0, further calculating the value RWFVV _ S _ D of the simplified vector dimension RWFVV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D) of the Chinese segmentation group in the document to be identified; if the difference value RWFFV _ TBI _ S _ D is larger than 0, randomly extracting the characteristic values with the same quantity as the difference value RWFFV _ TBI _ S _ D from the characteristic values corresponding to AMUNT _ V, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension RWFVG _ TBI is smaller than 0, the value RWFVG _ TBI _ S _ C of the Chinese word segmentation group reduced vector dimension RWFVG _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C) in the document to be identified is further calculated; if the difference value RWFFV _ TBI _ S _ C is larger than 0, randomly extracting the characteristic values with the same number as the difference value RWFFV _ TBI _ S _ C from the characteristic values corresponding to the AMOUNT _ D, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension RWFFVVTBI is smaller than 0, the value RWFVTBI _ S _ B of the simplified vector dimension RWFFVVTBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B) of the Chinese word segmentation group in the document to be identified is further calculated; if the difference value RWFFV _ TBI _ S _ B is larger than 0, randomly extracting the characteristic values with the quantity equal to the difference value RWFFV _ TBI _ S _ B from the characteristic values corresponding to AMUNT _ C to finish the simplification; if the sum is equal to 0, the simplification is finished; if the dimension RWFFV _ TBI is less than 0, the value RWV _ TBI _ S _ A2 of the Chinese word segmentation group reduced vector dimension RWFFV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2) in the document to be identified is further calculated; if the difference value RWFFV _ TBI _ S _ A2 is larger than 0, randomly extracting the feature values with the same quantity as the difference value RWFFV _ TBI _ S _ A2 from the feature values corresponding to AMUNT _ B, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the dimension is less than 0, further calculating the value RWGV _ TBI _ S _ A1 of the simplified vector dimension RWGFV _ TBI-AMOUNT _ A1 of the Chinese word segmentation group in the document to be identified; if the difference value is larger than 0, randomly extracting the feature values with the quantity equal to that of the difference value RWFFV _ TBI _ S _ A1 from the feature values corresponding to AMOUNT _ A2, and finishing the simplification; if the sum is equal to 0, the simplification is finished; if the number of the feature values is smaller than 0, the feature values with the number equal to that of the reduced vector dimension RWFFV _ TBI of the word-group of the document to be identified are randomly extracted from the feature values corresponding to AMUNT _ A1, and the reduction is finished.
For the case that the value RWFGV _ TBI _ S _ V of the text participle group reduced vector dimension RWFGV _ TBI- (AMOUNT _ A1+ AMOUNT _ A2+ AMOUNT _ B + AMOUNT _ C + AMOUNT _ D + AMOUNT _ V) in the document to be authenticated is calculated to be greater than 0, it means that the document to be authenticated is small in space or information AMOUNT, and therefore, the document to be authenticated is not suitable for comparison by using the feature value.
When the free vector dimension WFGFV _ TBI of the Chinese and foreign word segmentation groups in the document to be identified is smaller than the reduced vector dimension RWFFGV _ TBI of the Chinese and foreign word segmentation groups in the document to be identified, the dimension of the free vector dimension WFGFV _ TBI is small, and the magnitude values under other dimensions are equal to 0, and can be directly marked in the system and independently recorded and processed.
Preferably, in order to facilitate similarity comparison, the material word segmentation reduced vector dimension RWV selected in the system is equal to the word segmentation reduced vector dimension RWV _ TBI of the document to be identified; the material word group reduced vector dimension RWGV is equal to the word group reduced vector dimension RWGV-TBI of the document to be identified; the Chinese and foreign language word segmentation group reduced vector dimension RWFFV of the material and the Chinese and foreign language word segmentation group reduced vector dimension RWFFV _ TBI of the document to be identified should be equal.
According to a specific embodiment of the invention, a to-be-identified document word segmentation feature vector generation module extracts a feature value corresponding to a to-be-identified document word segmentation reduced vector dimension RWV _ TBI in each to-be-identified document according to the word segmentation reduced vector dimension RWV _ TBI to generate a to-be-identified document word segmentation feature vector WVE _ RWV _ TBI, wherein the to-be-identified document word segmentation feature vector generation module generates the to-be-identified document word segmentation feature vector WVE _ RWV _ TBI
WVE_RWV_TBI=[W_ID 1 ,W_N 1 ,...,W_ID i ,W_N i ,...,W_ID RWV_TBI ,W_N RWV_TBI ]
Wherein W _ ID i Representing a unique number of a participle in a thesaurus, W _ N i And representing the total times of the occurrences of the participles in the document to be identified, and taking the times as the characteristic value of the participle.
According to a specific embodiment of the invention, when the user detection mode determining module judges that the current user detection mode is the common plagiarism identification mode and carries out similarity comparison, the document word segmentation feature vector generation module to be identified generates a word segmentation feature vector WVE _ RWV _ TBI of a document to be identified; WVE _ RWV _ TBI = [ W _ ID = 1 ,W_N 1 ,...,W_ID i ,W_N i ,...,W_ID RWV_TBI ,W_N RWV_TBI ]The dimension of the word segmentation feature vector of the document to be identified is RWV _ TBI; word segmentationThe feature vector generation module generates a participle feature vector WVE _ RWV of the material in the comparison library; WVE _ RWV = [ W _ ID = 1 ,W_N 1 ,...,W_ID i ,W_N i ,...,W_ID RWV ,W_N RWV ](ii) a The dimension RWV _ TBI of the word segmentation feature vector of the document to be identified is equal to the dimension RWV of the word segmentation feature vector.
Note that although W _ ID is used in both the participle feature vectors WVE _ RWV _ TBI and WVE _ RWV i Representing a unique number of a participle in a thesaurus, W _ N i Representing the total times of the occurrences of the participle in the document to be identified and using the times as the feature value of the participle, but it should be noted that the W _ ID in the participle feature vector WVE _ RWV _ TBI i There is a great possibility of the W _ ID being equal to that in WVE _ RWV i And are not the same. Therefore, when performing similarity comparison, the dimensions of the two word segmentation feature vectors need to be adjusted to be consistent.
According to a specific embodiment of the invention, the document feature vector adjustment module to be identified is used for adjusting the W _ ID corresponding to all feature values in the participle feature vector WVE _ RWV _ TBI i The values are sorted in ascending or descending order according to the numbers in the word-dividing bank, and the missing W _ ID is sorted i Value insertion, inserted participle number W _ ID i The corresponding characteristic value is 0; assuming that the total number of word segmentation numbers in the word segmentation library is W, the number of the word segmentation numbers needing to be inserted is W-RWV _ TBI, and thus the expanded document word segmentation feature vector to be identified WVE _ RWV _ TBI _ EXT = [ W _ ID = TBI_EXT_1 ,W_N TBI_EXT_1 ,...,W_ID TBI_EXT_i ,W_N TBI_EXT_i ,...,W_ID TBI_EXT_RWV_TBI ,W_N TBI_EXT_RWV_TBI ,...,W_ID W ,W_N W ]。
According to a specific embodiment of the present invention, the material feature vector adjusting module is configured to adjust W _ IDs corresponding to all feature values in the participle feature vector WVE _ RWV i The values are arranged in ascending order or descending order according to the numbers in the word segmentation library, and the missing W _ ID is arranged i Value insertion, inserted participle number W _ ID i The corresponding characteristic value is 0; assuming that the total number of the word segmentation numbers in the word segmentation library is W, the number of the word segmentation numbers needing to be insertedIs W-RWV, thus obtaining the expanded participle feature vector WVE _ RWV _ EXT = [ W _ ID = EXT_1 ,W_N EXT_1 ,...,W_ID EXT_i ,W_N EXT_i ,...,W_ID EXT_RWV ,W_N EXT_RWV ,...,W_ID W ,W_N W ]。
By the method, the dimensionality of the word segmentation feature vectors of the documents to be identified and the materials in the comparison library is expanded to W, and the two word segmentation feature vectors are uniformly arranged in an ascending order or a descending order according to the serial numbers in the word segmentation library, so that the dimensionality of the feature values corresponding to the two word segmentation feature vectors is consistent.
The common plagiarism identification similarity calculation module is used for calculating the similarity between the document to be identified and any material in the comparison library; calculated by the following formula:
according to a specific embodiment of the invention, when the user detection mode determining module judges that the current user detection mode is the extended plagiarism identification mode and carries out similarity comparison, the document to be identified word-group feature vector generating module generates a word-group feature vector WVE _ RWGV _ TBI of the document to be identified; WVE _ RWGV _ TBI = [ WG _ ID ] 1 ,WG_N 1 ,...,WG_ID i ,WG_N i ,...,WG_ID RWGV_TBI ,WG_N RWGV_TBI ]The dimension of the word segmentation group feature vector of the document to be identified is RWGV-TBI; the participle group feature vector generation module generates a participle group feature vector WVE _ RWGV of the material in the comparison library; WVE _ RWGV = [ WG _ ID ] 1 ,WG_N 1 ,...,WG_ID i ,WG_N i ,...,WG_ID RWGV ,WG_N RWGV ](ii) a Wherein WG _ ID i Representing the unique number of a phrase in the lexicon of phrases, WG _ N i And representing the total times of the occurrence of the word segmentation group in the document to be identified, and taking the times as the characteristic value of the word segmentation group. The dimension RWGV _ TBI of the participle group feature vector of the document to be identified is equal to the dimension RWGV of the participle group feature vector.
Identification model for common plagiarismAccording to a specific embodiment of the invention, the expansion plagiarism identification document feature vector adjusting module adjusts the obtained expansion document word group feature vector to be identified WVE _ RWGV _ TBI _ EXT = [ WG _ ID ] to be identified TBI_EXT_1 ,WG_N TBI_EXT_1 ,...,WG_ID TBI_EXT_i ,WG_N TBI_EXT_i ,...,WG_ID TBI_EXT_RWV_TBI ,WG_N TBI_EXT_RWGV_TBI ,...,WG_ID W ,WG_N W ](ii) a The material characteristic vector adjusting module is used for adjusting the expanded participle group characteristic vector WVE _ RWGV _ EXT = [ WG _ ID = EXT_1 ,WG_N EXT_1 ,...,WG_ID EXT_i ,WG_N EXT_i ,...,WG_ID EXT_RWV ,WG_N EXT_RWGV ,...,WG_ID W ,W_N W ]. Expanded participle group feature vector WVE _ RWGV _ TBI _ EXT = [ WG _ ID = TBI_EXT_1 ,WG_N TBI_EXT_1 ,...,WG_ID TBI_EXT_i ,WG_N TBI_EXT_i ,...,WG_ID TBI_EXT_RWGV_TBI ,WG_N TBI_EXT_RWGV_TBI ,...,WG_ID W ,WG_N W ]。
By the method, the dimensionality of the word segmentation feature vectors of the documents to be identified and the materials in the comparison library is expanded to W, and the two word segmentation feature vectors are uniformly arranged in an ascending order or a descending order according to the serial numbers in the word segmentation library, so that the dimensionality of the feature values corresponding to the two word segmentation feature vectors is consistent.
The extended plagiarism identification similarity calculation module is used for calculating the similarity between the document to be identified and any material in the comparison library; calculated by the following formula:
according to a specific embodiment of the invention, when the user detection mode determining module judges that the current user detection mode is the multilingual plagiarism identification mode and performs similarity comparison, the Chinese and foreign language word segmentation feature vector generation module in the document to be identified generates the Chinese and foreign language word segmentation feature vector WVE _ RWFVGV _ TBI of the document to be identified; WVE _ RWFFV _ TBI = [ WFG _ ID ] 1 ,WFG_N 1 ,...,WFG_ID i ,WFG_N i ,...,WFG_ID RWFGV_TBI ,WFG_N RWFGV_TBI ]The dimension of the Chinese and foreign language phrase feature vector of the document to be identified is RWFFGV-TBI; the word group feature vector generation module generates Chinese and foreign language word group feature vectors WVE _ RWFDV of the materials in the comparison library; WVE _ RWFFGV = [ WFG _ ID) 1 ,WFG_N 1 ,...,WFG_ID i ,WFG_N i ,...,WFG_ID RWFGV ,WFG_N RWFGV ](ii) a Wherein WFG _ ID i The unique number of Chinese and foreign language phrase in the phrase library is shown, WFG _ N i And representing the total times of the Chinese and foreign language word groups appearing in the document to be identified, and taking the times as the characteristic values of the Chinese and foreign language word groups. The dimension RWFFGV _ TBI of the Chinese and foreign language word segmentation characteristic vector of the document to be identified is equal to the dimension RWFFGV of the Chinese and foreign language word segmentation characteristic vector.
Similar to the processing procedure of the common plagiarism identification mode, according to a specific embodiment of the present invention, in the multilingual plagiarism identification mode, the feature vector adjustment module of the document to be identified adjusts and obtains the expanded feature vector WVE _ RWFGV _ TBI _ EXT = [ WFG _ ID ] of the Chinese and foreign word segmentation group in the document to be identified TBI_EXT_1 ,WFG_N TBI_EXT_1 ,...,WFG_ID TBI_EXT_i ,WFG_N TBI_EXT_i ,...,WFG_ID TBI_EXT_RWFGV_TBI ,WFG_N TBI_EXT_RWFGV_TBI ,...,WFG_ID W ,WFG_N W ](ii) a The material characteristic vector adjusting module adjusts the expanded participle group characteristic vector WVE _ RWFFV _ EXT = [ WFG _ ID = EXT_1 ,WFG_N EXT_1 ,...,WFG_ID EXT_i ,WFG_N EXT_i ,...,WFG_ID EXT_RWV ,WFG_N EXT_RWFGV ,...,WFG_ID W ,WFG_N W ]. Expanded participle feature vector WVE _ RWFGV _ TBI _ EXT = [ WFG _ ID = TBI_EXT_1 ,WFG_N TBI_EXT_1 ,...,WFG_ID TBI_EXT_i ,WFG_N TBI_EXT_i ,...,WFG_ID TBI_EXT_RWFGV_TBI ,WFG_N TBI_EXT_RWFGV_TBI ,...,WFG_ID W ,WFG_N W ]。
By the method, the dimensionalities of the word segmentation feature vectors of the documents to be identified and the materials in the comparison library are expanded to W, and are uniformly arranged in an ascending order or a descending order according to the serial numbers in the word segmentation library, so that the dimensionalities of the feature values corresponding to the two word segmentation feature vectors are consistent.
The multi-language plagiarism identification similarity calculation module is used for calculating the similarity between the document to be identified and any material in the comparison library; calculated by the following formula:
according to a specific embodiment of the present invention, in order to avoid excessive expanded dimensions, all the participle IDs in the participle feature vector WVE _ RWV _ TBI may also be used as a set; and the participle ID in WVE _ RWV is used as another set; or all participle IDs in the participle group feature vector WVE _ RWGV _ TBI are taken as a set; and the participle ID in WVE _ RWGV is used as another set; or all participle IDs in the Chinese and foreign word segmentation group feature vector WVE _ RWFFV _ TBI are used as a set; and the participle IDs in WVE _ RWFDV are used as another set; taking a union set of the two sets to obtain a total participle ID set; expanding the dimension of the participle feature vector of the material in the document to be identified and the comparison base according to the total participle ID set, arranging the participle IDs corresponding to all feature values in ascending or descending order according to the serial numbers in the participle base, and inserting the W _ ID which is contained in the total participle ID set but not contained in the original self-set i Value, inserted participle number W _ ID i The corresponding eigenvalue is 0; or insert WG _ IDs contained in the total participle group ID set but not in the original set itself i Value, inserted participle number WG _ ID i The corresponding eigenvalue is 0; or inserting WFG _ ID contained in the total Chinese and foreign phrase group ID set but not contained in the original self set i Value, inserted participle number WFG _ ID i The corresponding eigenvalue is 0.
According to the access mode of a user, providing materials of different sub-libraries in a comparison library for similarity comparison, wherein the comparison adopts a traversal mode, namely extracting the feature vectors of all the materials in a selected range, and performing similarity comparison with the document to be identified; and comparing the calculated similarity value with a preset threshold value, and when the similarity value is higher than the preset threshold value, recording the corresponding material as a suspected material for later use.
And after the comparison between the document to be identified and all the materials is finished, extracting all the suspected materials, and further comparing the document to be identified and the suspected materials.
According to a preferred embodiment of the invention, all materials in the proverbal custom language library, the celebrity and famous language library and the poem library can be selected as suspected materials.
According to a preferred embodiment of the present invention, the material with the segmentation free vector dimension WFV smaller than the segmentation reduced vector dimension RWV can be selected as the suspect material.
According to a preferred embodiment of the present invention, the material with the word-group free vector dimension WGFV smaller than the reduced word-group vector dimension RWGV can be selected as the suspect material.
According to a preferred embodiment of the invention, the material with the Chinese and foreign language word segmentation free vector dimension WFGFV smaller than the Chinese and foreign language word segmentation reduced vector dimension RWFFGV can be selected as the suspected material.
According to a preferred embodiment of the present invention, the suspected material may be further selected by a word segmentation compactness factor.
According to a specific embodiment of the invention, under a common plagiarism identification mode, suspected materials can be screened according to the word segmentation compact coefficient of the document to be identified and the word segmentation compact coefficient of the materials. The document to be authenticated compactness coefficient statistical module is used for performing statistics on the compactness coefficient characteristic vectors WGCVE _ TBI = [ W _ ID, W _ N, W _ CHAR, G _ W _ ID _1, G _W _ID _2, …, G _ W _ ID _ i, … and G _ W _ ID _ (W _ N-1) corresponding to the participles in the document to be authenticated]Extracting high-density word segmentation and corresponding positions. The document compactness coefficient statistical module to be identified selects the participles with parts of speech being real words according to the part of speech W _ CHAR in the participle compactness coefficient feature vector, and counts the total quantity of the interval participles of a preset number of adjacent participles:wherein n is a predetermined adjacent number, when the total number of interval participles of the predetermined adjacent number of participles is less than a predetermined tight threshold value TH G And recording the ID and the corresponding position of the participle.
According to a specific embodiment of the invention, under the extended plagiarism identification mode, suspected materials can be screened according to the word group compactness coefficient of the document to be identified and the word group compactness coefficient of the materials. The document to be authenticated compact coefficient statistical module is used for determining a compact coefficient characteristic vector WGGCVE _ TBI = [ WG _ ID, WG _ N, WG _ CHAR, G _ WG _ ID _1, G _WG _ID _ _2, …, G _ WG _ ID _ i, … and G _ WG _ ID _ (W _ N-1)) according to the corresponding word segmentation compact coefficient characteristic vector WGGCVE _ TBI = [ WG _ ID, WG _ N, WG _ CHAR, G _ WG _ ID _1 and G _ WG _ ID _)]And extracting high-density phrase and corresponding position. The document close coefficient statistical module to be identified selects word segmentation groups with the part of speech being real words according to the word segmentation group word properties WG _ CHAR in the feature vectors of the close coefficients of the word segmentation groups, and counts the total quantity of the interval word segmentation of the preset number of adjacent word segmentation groups:wherein n is a predetermined adjacent number, when the total number of the interval participles of the predetermined adjacent number participle group is less than a predetermined tight threshold value TH G Then, the ID and corresponding position of the participle group are recorded.
According to a specific embodiment of the invention, in the multilingual plagiarism identification mode, suspected materials can be screened according to the Chinese and foreign language word segmentation compact coefficients of the document to be identified and the Chinese and foreign language word segmentation compact coefficients of the materials. The document to be identified compact coefficient statistic module is used for identifying a document to be identified according to a participle compact coefficient feature vector WFGGCVE _ TBI = [ WFG _ ID, WFG _ N, WFG _ CHAR, G _ WFG _ ID _1, G _WFG _ID _ _2, …, G _ WFG _ ID _ i, …, G _ WFG _ ID _ (W _ N-1)]And extracting high-density phrase and corresponding position. The document to be identified compactness coefficient statistical module selects the participle group with the part of speech being a real word according to the participle group part of speech WFG _ CHAR in the Chinese and foreign language participle group compactness coefficient characteristic vector, and counts the interval participle total quantity of the predetermined adjacent part of participle groups:wherein n is a predetermined adjacent number, when the total number of the interval participles of the predetermined adjacent number participle group is less than a predetermined tight threshold value TH G And recording the ID and the corresponding position of the Chinese and foreign word group.
The value of the predetermined number n of neighbors and the tight threshold value TH G All the devices are preset by the system and can be adjusted according to actual needs; when the total number of interval participles of a predetermined number of adjacent participles is less than a predetermined tight threshold value TH G In the process, the real word segmentation is considered to be relatively dense in corresponding positions, so that a certain viewpoint is probably set forth in a centralized manner, and important attention needs to be paid.
Under the common plagiarism identification mode, the close coefficient suspected material extraction module extracts the close coefficient suspected material according to the fact that the total quantity of the interval participles of the preset adjacent quantity participles is smaller than a preset close threshold value TH G Then, the recorded word segmentation ID extracts and compares all materials containing the word segmentation ID in the library; respectively calculating the participle compact coefficient eigenvector WGCVE = [ W _ ID, W _ N, W _ CHAR, G _ W _ ID _1, G _W _ID _, 2, …, G _ W _ ID _ i, …, G _ W _ ID _ (W _ N-1)]Counting the total quantity of interval participles of a preset adjacent quantity of participles:wherein n is a predetermined adjacent number, when the total number of interval participles of the predetermined adjacent number of participles is less than a predetermined tight threshold value TH G And if so, selecting the material as a suspected material. The segmentation ID is one or more, and one or more materials containing the one or more segmentation IDs are extracted from the one or more segmentation IDs.
Under the expansion plagiarism identification mode, the close coefficient suspected material extraction module divides the total quantity of the spaced words of the word groups according to the preset adjacent quantity to be less than the preset close threshold value TH G Then, the recorded segmentation group ID extracts all the materials containing the segmentation group ID in the comparison library; respectively calculating the segment group compact coefficient eigenvectors WGGCVE = [ WG _ ID, WG _ N, WG _ CHAR, G _ WG _ ID _1, G _WG _ID _2, …, G _ WG _ ID _ i, …, G _ WG _ ID _ (WG _ N-1) corresponding to the segment group ID in the material]Counting the number of word groups between predetermined adjacent numbersTotal number of segmented words:wherein n is a predetermined adjacent number, when the total number of the interval participle groups of the predetermined adjacent number of participles is less than a predetermined tight threshold value TH G And if so, selecting the material as a suspected material. And one or more of the phrase group IDs are extracted, and one or more of the materials containing the one or more phrase group IDs are extracted according to the one or more phrase group IDs.
Under the multi-language plagiarism identification mode, the close coefficient suspected material extraction module extracts the close coefficient suspected material according to the fact that the total quantity of the interval participles of Chinese and foreign language participles in the preset adjacent quantity is smaller than the preset close threshold value TH G Then, recording Chinese and foreign language segmentation word group ID, and extracting all materials containing the Chinese and foreign language segmentation word group ID in a comparison library; respectively calculating the Chinese and foreign language phrase compact coefficient eigenvectors WFGGCVE = [ WFG _ ID, WFG _ N, WFG _ CHAR, G _ WFG _ ID _1, G \uWFG \uID \u2, …, G _ WFG _ ID _ i, …, G _ WFG _ ID _ (WFG _ N-1) corresponding to the Chinese and foreign language phrase ID in the material]Counting the total quantity of interval word segmentation of Chinese word segmentation groups in preset adjacent quantity:wherein n is a predetermined adjacent number, and when the total number of the Chinese word segmentation groups in the interval of the predetermined adjacent number of the word segmentation groups is less than a predetermined tight threshold value TH G And if yes, selecting the material as a suspected material. The Chinese and foreign language phrase IDs are one or more, and one or more materials containing the one or more Chinese and foreign language phrase IDs are extracted from the one or more Chinese and foreign language phrase IDs.
By the extraction mode, certain real word segmentation possibly described in certain position sets and corresponding positions in the document to be identified, which has a low total occurrence number, can be extracted for further comparison.
According to a specific embodiment of the invention, under a formula plagiarism authentication mode, a formula extraction module is used for extracting a formula in a document to be authenticated; a formula decomposition module for decomposing the variable parameters and dependent variable parameters of the formula into operationExtracting symbols, specific meanings, dimensions and value ranges of the parameters respectively; the formula comparison module is used for comparing the respective variable parameters, dependent variable parameters, operation symbols and the specific meanings, dimensions and value ranges of the formulas extracted from the documents to be identified with the respective variable parameters, dependent variable parameters, operation symbols and the specific meanings, dimensions and value ranges of the parameters of the formulas stored in the formula library one by one; when the coincidence degree of the respective variable parameters, dependent variable parameters, operation symbols, dimensions and value ranges of the formulas in the document to be identified and the respective variable parameters, dependent variable parameters, operation symbols, dimensions and value ranges of the formulas stored in the formula library exceeds a formula comparison threshold value TH MATH And taking the material in the formula library associated with the currently compared formula as the suspected material. The contact ratio is the ratio of the sum of independent variable parameters, dependent variable parameters, operation symbols and dimension numbers of the formula in the document to be identified to the sum of the independent variable parameters, the dependent variable parameters, the operation symbols and the dimension numbers of the current formula in the document to be identified compared with the formula in the formula library.
According to one embodiment of the present invention, a sliding window may be used to compare the document to be authenticated with the suspect material in full text. The size of the sliding window may be set by the system. The size of the sliding window directly influences the comparison effect, erroneous judgment is easily caused when the sliding window is selected too small, and missed judgment is easily caused when the sliding window is selected too large. The sliding step of the sliding window is also preset by the system. As shown in fig. 2, step S0: starting; s1: sliding window setting module initialized similar window counter CT 1 CT (computed tomography) with walking-step counter of =0 2 =0; step S2: the sliding window setting module sets the sliding windows of the document to be authenticated and the suspected material to be at the initial position of the document; and step S3: the sliding window comparison module compares the sliding window of the document to be identified with the sliding window of the suspected material and counts the number of the same real word segmentation; and step S4: the sliding window comparison module judges whether the number of the same real word participles is larger than or equal to a threshold value TH W (ii) a Counting when greater than or equal to a threshold valueThe value of the device plus one, i.e. CT 1 =CT 1 +1, recording the current positions of the sliding window of the identification document and the sliding window of the suspected material and the content in the sliding window; step S5: the sliding window setting module sets a sliding step length for sliding the sliding window of the suspected material; step S6: the sliding window setting module judges whether the document is positioned at the document end position; if not, returning to step S3: if it is the end position, go to step S11; step S11: the sliding window setting module judges whether a sliding window of the document to be authenticated is positioned at the document end position; if not, go to step S12, if it is, go to step S13; step S12: the sliding window setting module sets the sliding window of the suspected material to return to the initial position of the document; sliding the sliding window of the document to be authenticated by a sliding step length, CT 2 =CT 2 +1 goes to step S3; step S13: sliding window comparison module calculates similar window counter CT 1 Numerical value and sliding walking counter CT 2 The ratio of the values M; s14: the sliding window comparison module judges whether the ratio M is greater than or equal to a preset threshold value TH m When M is greater than or equal to TH M If so, the document to be identified is considered to be similar to the suspected material; when M is<TH M If so, the document to be identified is considered to be dissimilar to the suspected material; s15: the sliding window comparison module judges whether suspected materials need to be compared, and if yes, the step S1 is returned; if not go to step S16; step S16: the comparison report generation module generates and outputs a comparison report, wherein the comparison report comprises a similar window counter CT of the identification document and all similar suspected materials 1 Numerical value, sliding walking counter CT 2 The numerical value and the ratio of the two, the specific position and the specific content of the similar part of the identification document and the similar suspected material; step S17: and finishing the comparison.
According to an embodiment of the present invention, step S3: the sliding window comparison module compares the sliding window of the document to be identified with the sliding window of the suspected material and counts the number of the same real word segmentation; under a common plagiarism identification mode, the same real word segmentation means that the IDs of the real word segmentation in the segmentation library are the same; under the extended plagiarism identification mode, the same real word segmentation means that the IDs of the real word segmentation groups in the segmentation library are the same; under the multilingual plagiarism identification mode, the same real word segmentation means that IDs of Chinese and foreign language segmentation groups in a segmentation library are the same.
According to an embodiment of the present invention, step S16: the comparison report generation module outputs a comparison report, and further comprises that the content of the comparison report is different according to different identification modes. Under a common plagiarism identification mode, comparing the specific position and the specific content of the similar part of the document to be identified and similar suspected materials in a report; the document to be identified adopts an expression mode consistent with the similar part in the similar suspected material; namely, the adopted words are completely consistent; only individual word orders may be adjusted; if the document to be authenticated is rewritten over the document that it is plagiarized, when the rewriting degree is large, the document that it is plagiarized may not be found by the ordinary plagiarism authentication mode. Under the extended plagiarism identification mode, the comparison report contains the specific position and the specific content of the similar part of the document to be identified and similar suspected materials; if the identified document carries out synonym or near synonym rewriting on the document copied by the identified document, when the document structure is not rewritten greatly, the extended copy identification mode can possibly find the document copied by the identified document. Under the multilingual plagiarism identification mode, the comparison report contains the specific position and the specific content of the similar part of the document to be identified and similar suspected materials; if the authenticated document performs translation rewriting on the document plagiarized by the authenticated document, when the document structure is not rewritten to a large extent, the extended plagiarism authentication mode may also be capable of finding the document plagiarized by the authenticated document.
According to an embodiment of the present invention, the sliding window being located at the document starting position means that the leftmost side of the sliding window coincides with the document starting position; the sliding window being at the document end position means that the rightmost side of the sliding window coincides with the document end position.
According to the system early operation test, the sliding window is selected to be of a proper size for the four real word segmentation, and the size of the sliding window can be selected to be of other sizes as required. When in comparison, the sliding window slides the step length of one real word segmentation at a time; and in the comparison process, when three or more real word segmentations appearing in the sliding window are the same (the sequence of the real word segmentations is not considered), recording the current position and content of the sliding window in the document to be authenticated and the suspected material.
Although the present invention has been described with reference to a preferred embodiment, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.