Disclosure of Invention
In view of the above disadvantages in the prior art, the present invention provides a method for detecting a paper similarity for a user, which overcomes the drawback of paper similarity detection in the prior art.
The invention provides a method for detecting thesis similarity, wherein the method comprises the following steps:
the method comprises the steps that a server obtains a thesis file to be subjected to similarity detection and input by a user, and text content contained in the thesis file is divided into a plurality of texts;
the server compares the character information of each text with the character information of the thesis in the paper library in sequence to obtain a first text of which the similarity between the text information of each text and the character information in the thesis library exceeds a first preset threshold;
the server acquires a plurality of papers of which the similarity with the character information in the first text exceeds a second preset threshold value from a paper library, and merges the acquired papers into a second text;
and the server judges the similarity value between the first text and the second text as the detection similarity value of the paper to be detected.
Optionally, the step of determining the similarity value between the first text and the second text as the detection similarity value of the paper to be detected further includes:
the server disassembles the first text to obtain a plurality of candidate sentences;
the server determines the importance scores of the candidate sentences;
the server extracts a target sentence with an importance score larger than a preset value as key information of the first text;
and the server compares the similarity of the key information of the first text with the key information of the second text, and judges the compared similarity value as the similarity value between the first text and the second text.
Optionally, in the step of parsing the first text by the server to obtain a plurality of candidate sentences, the method for parsing the first text includes:
disassembling according to punctuation marks; and when the punctuations are the pause signs, the colon signs and the quotation marks, the punctuations are not disassembled.
Optionally, the step of the server determining the importance score of each candidate sentence includes:
judging whether the candidate sentences contain Chinese sentences and/or webpage link addresses;
if only the Chinese sentence is contained, taking the sum of the weights of all phrases in the Chinese sentence as the importance score of the candidate sentence;
if only the webpage link address is contained, taking the sum of the weights of the page elements contained in the webpage corresponding to the webpage link address as the importance score of the candidate sentence;
and if the candidate sentences contain the Chinese sentences and the webpage link addresses, taking the weighted average of the sum of the weights of all phrases in the Chinese sentences and the sum of the weights of the page elements contained in the webpage corresponding to the webpage link addresses as the importance scores of the candidate sentences.
Optionally, the step of taking the sum of the weights of the phrases in the chinese sentence as the importance score of the candidate sentence includes:
splitting each candidate sentence into a plurality of phrases according to a semantic analysis mode;
carrying out full-text retrieval, and calculating the occurrence times of each phrase;
sequencing the phrases according to the sequence of the occurrence times from high to low, wherein each phrase is endowed with a corresponding weight according to the occurrence times, and the higher the occurrence times, the higher the weight;
and calculating the importance score of each candidate sentence according to the weight of each phrase, wherein the importance score is the sum of the weights of each phrase in the candidate sentence.
Optionally, the step of taking the sum of the weights of the page elements included in the web page corresponding to the web page link address as the importance score of the candidate sentence includes:
the server background opens a target webpage corresponding to the webpage link address;
and the server determines the importance score of the target webpage according to the page elements contained in the target webpage.
Optionally, the step of determining, by the server, the importance score of the target web page according to the page elements included in the target web page includes:
determining an importance score for the target web page using the following formula;
where S (Vi) is the importance score of the target web page, d is a damping coefficient, typically set to 0.85, and in (Vi) is the set of web pages for which there is a link to the target web page. out (Vj) is the set of web pages pointed to by the links in web page j, out (Vj) is the absolute value used to represent the number of elements in the set of web pages, and S (Vj) is the importance score of web page j.
Optionally, the step of comparing, by the server, the similarity between the key information of the first text and the key information of the second text includes:
calculating cosine similarity of a first sentence in the key information of the first text and a second sentence in the key information of the second text;
and if the cosine similarity is higher than the preset value, determining that the first text is approximate to the second text.
Optionally, the cosine similarity calculation method includes:
splitting the first sentence into a plurality of phrases;
splitting the second sentence into a plurality of phrases;
comparing the two groups of phrases one by one, if the phrases exist, recording the phrases as 1, if the phrases do not exist, recording the phrases as 0, and obtaining a first sequence and a second sequence;
and calculating the cosine similarity between the first sequence and the second sequence and taking the cosine similarity as the cosine similarity between the first sentence and the second sentence.
Optionally, the calculation of the cosine similarity between the first sequence and the second sequence is calculated using the following formula:
wherein ab represents the integral addition of multiplication of the middle elements of the a sequence and the corresponding elements of the b sequence, and the denominator represents the multiplication of the square sum of the root of all the elements in the a sequence and the square sum of the root of all the elements in the a sequence.
The method has the advantages that by acquiring the paper file to be subjected to similarity detection, the text content in the paper file is divided into a plurality of texts and then compared with the text information of the papers in the paper library, so that the first text with the similarity between the text information in the paper library and the text information in the paper library exceeding a first preset threshold is obtained; the server acquires a plurality of papers of which the similarity with the character information in the first text exceeds a second preset threshold value from a paper library, and merges the acquired papers into a second text; and the server judges the similarity value between the first text and the second text as the detection similarity value of the paper to be detected. The process of the invention can be automatically completed by the server without manual reading, thereby improving the working efficiency.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention provides a method for detecting the similarity of papers, which comprises the following steps of:
step S101, a server acquires a paper file to be subjected to similarity detection, which is input by a user, and divides text content contained in the paper file into a plurality of texts.
The method comprises the steps that a user uploads a paper file to be detected to a server, the server divides the content contained in the paper file according to the page or the chapter of the paper file after obtaining the paper file, a plurality of texts are obtained, and the texts are stored respectively.
In the step, a plurality of texts are divided for longer papers, so that the comparison result can be obtained quickly, and the corresponding similarity can be obtained for each text, so that the effect better than that of similarity detection of a single paper can be obtained.
And S102, the server compares the character information of each text with the character information of the papers in the paper library in sequence to obtain a first text of which the similarity with the character information in the paper library exceeds a first preset threshold value.
And comparing the character information of each text with the character information of the papers in the paper library in sequence to obtain the similarity between each text and each paper in the paper library, screening out the papers with the similarity exceeding a preset first preset threshold value between the texts and the papers to be detected, and integrating the documents with the similarity exceeding the first preset threshold value in the papers to be detected and the papers in the papers library to obtain the first text. In practice, the first predetermined threshold may be customized or set by default, and preferably, the first predetermined threshold is set to zero-five percent.
Step S103, the server obtains a plurality of papers in a paper library, the similarity of which to the character information in the first text exceeds a second preset threshold, and merges the obtained papers into a second text.
And integrating the thesis with the content in the first text, wherein the similarity of the thesis and the content in the first text exceeds a preset second threshold value to obtain a second text. While a plurality of papers are integrated, repeated parts in the papers can be deleted or non-text contents in the files can be deleted.
And step S104, the server judges the similarity value between the first text and the second text as the detection similarity value of the paper to be detected.
And calculating the similarity between the first text obtained in the step S102 and the second text obtained in the step S103, and determining the similarity between the first text and the second text as the similarity value of the paper to be detected, that is, the repetition rate value.
Specifically, in order to obtain a more accurate similarity value between the first text and the second text, the step of determining the similarity value between the first text and the second text as the detection similarity value of the paper to be detected further includes:
the server disassembles the first text to obtain a plurality of candidate sentences; the method for splitting the first text comprises the following steps: disassembling according to punctuation marks; and when the punctuations are the pause signs, the colon signs and the quotation marks, the punctuations are not disassembled.
The server determines the importance scores of the candidate sentences;
the server extracts a target sentence with an importance score larger than a preset value as key information of the first text;
and the server compares the similarity of the key information of the first text with the key information of the second text, and judges the compared similarity value as the similarity value between the first text and the second text.
Further, since each candidate sentence may include information with different attributes, that is, the candidate sentence may include a chinese sentence or a web page link address, before the calculating of the importance score, the step of determining the importance score of each candidate sentence by the server includes:
judging whether the candidate sentences contain Chinese sentences and/or webpage link addresses;
if only the Chinese sentence is contained, taking the sum of the weights of all phrases in the Chinese sentence as the importance score of the candidate sentence;
if only the webpage link address is contained, taking the sum of the weights of the page elements contained in the webpage corresponding to the webpage link address as the importance score of the candidate sentence;
and if the candidate sentences contain the Chinese sentences and the webpage link addresses, taking the weighted average of the sum of the weights of all phrases in the Chinese sentences and the sum of the weights of the page elements contained in the webpage corresponding to the webpage link addresses as the importance scores of the candidate sentences.
The step of taking the sum of the weights of all phrases in the Chinese sentence as the importance scores of the candidate sentences comprises the following steps:
splitting each candidate sentence into a plurality of phrases according to a semantic analysis mode;
carrying out full-text retrieval, and calculating the occurrence times of each phrase;
sequencing the phrases according to the sequence of the occurrence times from high to low, wherein each phrase is endowed with a corresponding weight according to the occurrence times, and the higher the occurrence times, the higher the weight;
and calculating the importance score of each candidate sentence according to the weight of each phrase, wherein the importance score is the sum of the weights of each phrase in the candidate sentence.
For example, one paper contains the following:
today XX association held a work meeting in beijing with good weather, about 30 degrees centigrade, no rain and good traffic, on the work meeting, the chairman summarized the work of XX association in the last year and also showed excellent employees of XX association.
The candidate sentences include:
A. today XX associations held a conference in beijing;
B. weather is good;
C. approximately 30 degrees celsius;
D. no rain is present;
E. the traffic situation is also good;
F. on a working meeting;
G. the Zhang Congress summarized the past year work of the XX Association;
H. also show excellent employees of the XX association.
The words obtained by disassembling include:
today: appear 1 time and have a weight of 1
XX Association: appear 3 times and have a weight of 3
Beijing: 1 time, weight 1
And (3) opening: 1 time, weight 1
And (4) working meeting: 2 times, weight 2
Weather: 1 time, weight 1
30 ℃ of: 1 time, weight 1
Rain: 1 time, weight 1
Traffic conditions are as follows: 1 time, weight 1
Lengthening the sheet: 1 time, weight 1
The work in the last year: 1 time, weight 1
To summarize: 1 time, weight 1
Carrying out exterior recognition: 1 time, weight 1
Excellent staff: 1 time, weight 1
The importance scores of the above candidate sentences are respectively: no. 8, No. 1, No. 2, No. 6 and No. 5.
Assuming that the preset value is 2 points, the target sentences are No. A, No. F, No. G and No. H, and the final key information is as follows: today XX associations held a conference in beijing; on a working meeting; the Zhang Congress summarized the past year work of the XX Association; also show excellent employees of the XX association.
Further, the step of taking the sum of the weights of the page elements contained in the web page corresponding to the web page link address as the importance score of the candidate sentence includes:
the server background opens a target webpage corresponding to the webpage link address;
and the server determines the importance score of the target webpage according to the page elements contained in the target webpage.
The server determines the importance score of the target webpage according to the page elements contained in the target webpage, and the method comprises the following steps:
determining an importance score for the target web page using the following formula;
where S (Vi) is the importance score of the target web page, d is a damping coefficient, typically set to 0.85, and in (Vi) is the set of web pages for which there is a link to the target web page. out (Vj) is the set of web pages pointed to by the links in web page j, out (Vj) is the absolute value used to represent the number of elements in the set of web pages, and S (Vj) is the importance score of web page j.
Specifically, the step of comparing the similarity between the key information of the first text and the key information of the second text by the server includes:
calculating cosine similarity of a first sentence in the key information of the first text and a second sentence in the key information of the second text;
and if the cosine similarity is higher than the preset value, determining that the first text is approximate to the second text.
Specifically, the cosine similarity calculation method includes:
splitting the first sentence into a plurality of phrases;
splitting the second sentence into a plurality of phrases;
comparing the two groups of phrases one by one, if the phrases exist, recording the phrases as 1, if the phrases do not exist, recording the phrases as 0, and obtaining a first sequence and a second sequence;
and calculating the cosine similarity between the first sequence and the second sequence and taking the cosine similarity as the cosine similarity between the first sentence and the second sentence.
For example:
the first sentence is: today the association held meetings in Beijing.
The second sentence is: the association held a conference of the general law in beijing.
The first sequence a is (1, 1, 1, 1, 0, 1) and the second sequence b is (0, 1, 1, 1, 1, 1).
Preferably, the method step calculates the cosine similarity between the first sequence and the second sequence using the following formula:
wherein ab represents the integral addition of multiplication of the middle elements of the a sequence and the corresponding elements of the b sequence, and the denominator represents the multiplication of the square sum of the root of all the elements in the a sequence and the square sum of the root of all the elements in the a sequence.
For example, the two sentences above calculate the results as:
the final calculation result is: 0.8.
the invention discloses a method for detecting the similarity of a paper, which comprises the steps of obtaining a paper file to be detected in similarity, dividing text contents in the paper file into a plurality of texts, and comparing the texts with character information of the paper in a paper library to obtain a first text of which the similarity between the text information and the character information in the paper library exceeds a first preset threshold; the server acquires a plurality of papers of which the similarity with the character information in the first text exceeds a second preset threshold value from a paper library, and merges the acquired papers into a second text; and the server judges the similarity value between the first text and the second text as the detection similarity value of the paper to be detected. The process of the invention can be automatically completed by the server without manual reading, thereby improving the working efficiency.
It should be understood that equivalents and modifications of the technical solution and inventive concept thereof may occur to those skilled in the art, and all such modifications and alterations should fall within the scope of the appended claims.