TWM623980U - System of screening for text data relevance - Google Patents

Abstract

A screening and association system for text data, after word segmentation, screening, correlation processing and integration processing are performed on multiple contrasting text data (such as papers, etc.) The related index file is used to quickly sort out the brief information of the text data to be compared, and the originality of the text data to be compared can be further analyzed according to the related index file.

Description

Screening and Association System for Text Data

This creation is about a screening and correlation system of text data; it especially uses a kind of screening and segmentation based on the adjacent adjacent words, which can quickly organize and analyze text data, and can analyze the original text of the text data to be compared against the text data. Data filtering and correlation system.

In recent years, plagiarism incidents have emerged one after another, and the public has begun to have doubts about the originality of papers. Although there are many detection systems for plagiarism comparison of papers and articles on the market, most of these systems are used to detect copyright holders who publish research papers. It is unfair to the copyright owner to conduct plagiarism comparison detection with a skeptical attitude. In addition, some units even require the copyright owner to submit the plagiarism comparison results first, and require a certain degree of similarity before the copyright owner of the paper can publish the paper. Therefore, the copyright owner needs to use this method to prove that his document has not been copied from others. The practice is to take an attitude of distrust to the copyright owner, which is very inappropriate. The creators believe that they should think backwards and provide tools for detecting originality for the publication of the copyright owner's papers, as a reference for the publication of their papers, and the publishing unit may formulate the originality ratio as a reference for the quality management of papers.

Regarding the plagiarism comparison system, in recent years, in academic research, the issue of plagiarism has become more and more serious. Plagiarism Detection has been paid more and more attention. Plagiarism issues are mainly divided into the following categories: 1. Copy/paste/clone plagiarism without modification. 2. Paraphrasing plagiarism: By copying paragraphs, switching vocabulary, or rewriting sentence structure or grammatical style. 3. Metaphor plagiarism: A way of expressing the ideas of others better through clarity. 4. Idea plagiarism: An idea or solution is borrowed from other sources as one's own research paper. 5. Self/recycled plagiarism: Use your own published articles and republish them as new research results. 6. Citing plagiarism: citing references from appropriate sources, but whose descriptions are similar to the original content in terms of words, sentences, and even structural grammar.

Among these types of plagiarism, "unmodified copy and paste or fragment plagiarism" and "paragraph rewriting" are the most concerned. These two plagiarism methods can be clearly seen by comparing the paper with the plagiarized literature. Seeing plagiarism, these two are the most criticized.

In 1995, some scholars conducted research. The paper carried out copy detection on digital files. With the evolution of natural language processing and hardware equipment, many different methods have been introduced in recent years. In the field of plagiarism detection , mainly divided into several methods: 1. Character-Based Methods: This method is the most common method for the detection of plagiarism in papers. The papers to be compared are compared with the existing papers database, and by searching for matching strings , and then determine the proportion of plagiarism in the paper, and therefore can tell the system users, plagiarized paragraphs and sentences. Shrestha and Solorio published in 2013, by combining stop words, named entities, and all words with In the n-grams method, plagiarism is detected by considering whether the detected paper and the text database article have articles with a high degree of n-gram conformity. In 2016, Nguyen et al. proposed to detect plagiarism in Vietnamese articles through plagiarism detection. This method uses the method of substring n-gram. This kind of method has the following three defects: First, if the paper has words that are not in the paper database, it will lead to no similar texts and sentences, so plagiarized papers cannot be detected; second, users can change the vocabulary or Swap the order of words, and then avoid this method of detection, resulting in no similar words and sentences to be detected; 3. Because this method is to compare strings, if the length of the input string is too long, it is easy to dilute the input paper, thereby reducing plagiarism similarity. 2. Vector-Based Methods: This method extracts lexical and grammatical features and classifies them as vectors instead of strings. This similarity is usually measured by methods such as the Jaccard coefficient, the Dice coefficient, the Overlap coefficient or the Cosine Similarity to measure the similarity between papers and papers degree. Mahdavi et al. published a vector space model to detect plagiarism in Persian articles, and compare the similarity of the articles by converting the articles to TF-IDF. Jiffriya et al. proposed in 2013 to convert the article into a vector and then group it through the K-means algorithm. After the grouping, the article is based on tri-gram for plagiarism detection. The disadvantage of this method is that the importance of a word in the article is measured by word frequency. Sometimes important words may not appear enough times, which will lead to poor comparison results, and this calculation cannot reflect location information and The importance of words in context. 3. Syntax-Based Methods: This method uses syntactic special Signs are parts of speech, dependency trees of sentences, and words in different statements to detect plagiarism, use parts of speech to represent word structures and calculate similarity. This method can find paragraphs with similar sentence structure, but can not find paragraph rewriting, swapping vocabulary and changing sentence structure plagiarism. Grammar-based methods have several flaws. First, Chinese grammar is much more complicated than English grammar. If we use our Chinese plagiarism system to detect plagiarism by grammar, it will result in extremely poor comparison results; second, this method Detecting plagiarized content through syntactic features will result in finding words with similar syntactic features, but without plagiarism, only the syntax is the same, resulting in a wrong judgment. 4. Semantic-Based Methods: This method converts the article into a vector by letting the system understand the semantics of the paragraph, which can be used to detect changing the order and changing the active and passive, but this method cannot find plagiarized paragraphs and sentences. . Torres proposed to help detect plagiarism by building a dictionary in 2009, and Resnik in 1999 used external resources to help detect plagiarism using semantics. Solving plagiarism detection by means of semantics will find papers with similar semantics, but cannot know the plagiarized paragraphs and vocabulary, and there is no way to verify plagiarism.

In view of this, the creators thought hard, actively researched, and after years of experience in related product research, and through continuous experiments and improvements, finally developed this creation.

The purpose of this creation is to provide a method for filtering and correlating text data that can quickly sort out the brief information of text data.

The method of this creation to achieve the above purpose includes the following steps: S11. Based on a word segmentation vocabulary database, perform word segmentation processing on a text data to generate a piece of word segmentation information; S12. perform a screening process on the word segmentation information to generate a piece of word segmentation information; the filtered word segmentation information has more than two selected word segmentations; S13. perform correlation on the filtered word segmentation information Processing to generate a plurality of related sequence information; the related sequence information is respectively composed of two or more adjacent screening segmented words.

Preferably, before performing the step S11, a step S110 may be performed first; the step S110 is: collecting the author-defined keywords in the text data to establish a professional keyword vocabulary database, and using the professional keyword vocabulary The library imports the word segmentation vocabulary library, so as to obtain related sequence information that is closer to the original meaning of the text data.

Preferably, in this step S12, after the screening process, the synonymous word processing may be performed first, and then the subsequent steps are performed; the synonymous word processing is: performing a text synonym check on the screened segmented words after the screening process, and Synonyms and synonyms are converted into standard characters.

Another purpose of this creation is to provide a text data screening and correlation system that can quickly sort out the brief information of text data.

The structure of this creation to achieve the above purpose includes: a storage module for storing a word segmentation vocabulary database; a word segmentation processing module for performing word segmentation processing on a text data to generate word segmentation information; a screening processing module A group is used to filter the segmented information to generate a filtered segmented information; a correlation processing module is used to perform correlation processing on the filtered segmented information to generate a plurality of related sequence information.

Another purpose of this creation is to provide a method that can quickly sort out the brief information of multiple comparison text data, and combine the information of each reference text data. The brief information is integrated together to facilitate the analysis of the originality of the text data to be compared and the method of screening and correlation of text data.

The method of this creation to achieve the above-mentioned purpose includes the following steps: S21. Create a comparison set of information with more than two copies of the comparison text data; S22. Based on a word segmentation vocabulary database, perform word segmentation processing on the comparison text data to separate them. Generate a contrast word segmentation information; S23. Perform screening processing on the comparison word segmentation information to respectively generate a comparison filter word segmentation information; the comparison filter word segmentation information has two or more filter segmentation information respectively; S24. The Perform correlation processing on the comparison and screening segment information to respectively generate a plurality of comparison-related sequence information; the comparison-related sequence information is respectively composed of two or more adjacent screening segmented words; S25. All comparison The relational sequence information is integrated together to create a relational index file.

Preferably, before the step S22 is performed, a step S220 is performed first; the step S220 is: collecting a part or all of the author-defined keywords in the comparison text data and the to-be-compared text data to establish a professional A keyword vocabulary database is imported, and the professional keyword vocabulary database is imported into the word segmentation vocabulary database, so as to obtain a related index file that is closer to the original meaning of the text data.

Preferably, after step S25, steps S26-S29 are performed; step S26 is: performing word segmentation processing, screening processing and correlation processing on a text data to be compared to generate a plurality of related sequence information to be compared; Step S27 is: aligning the related sequence information to be compared with the related index file, respectively, to find each reference text information that has the same related sequence information to be compared with the related sequence information to be compared. Step S28 is: establishing an intersection sequence, and arranging all the relative sequence information that is the same as the related sequence information to be compared; Step S29 is: analyzing each piece of the text data to be compared with the same related sequence information The comparison text data, in order to analyze the originality of the text data to be compared.

Preferably, in this step S23, after the screening process, the synonym processing may be performed first, and then the subsequent steps may be performed to increase the effect of correlation comparison.

Another purpose of this creation is to provide a text that can quickly sort out the brief information of multiple comparison text data, and integrate the brief information of each reference text data, so as to facilitate the analysis of the originality of the text data to be compared. Data filtering and correlation system.

The structure of this creation to achieve the above-mentioned purpose includes: a storage module for storing a word segmentation vocabulary database and a comparison set information;

a word segmentation processing module for performing word segmentation processing on each comparison text data of the comparison set information to generate a comparison word segmentation information respectively; a screening processing module for screening the comparison word segmentation information processing to respectively generate a comparison screening segment information; a correlation processing module for performing correlation processing on the comparison screening segmentation information to respectively generate a plurality of comparison correlation sequence information; an integration module for Integrate all reference related sequence information together to create a related index file.

Preferably, the word segmentation processing module, the screening processing module and the correlation processing module perform word segmentation processing, screening processing and correlation processing on a text data to be compared to generate a plurality of related sequence information to be compared , and the screening and association system for the text data further includes: a comparison module, which is used for the waiting The alignment related sequence information is compared with the related index file respectively, and each reference text data with the same comparative related sequence information as the related sequence information to be compared is found out; The sequence information of the related sequence information to be compared is the same as the related sequence information, so as to establish the intersection sequence; an analysis module analyzes each control text data with the same related sequence information as the text data to be compared.

In order to achieve the above and other purposes, the technical means, components and their effects adopted by this creation are described below with a preferred embodiment in conjunction with the diagrams.

100, 100a: Screening and association system for text data

1. 1a: storage module

2, 2a: word segmentation processing module

3, 3a: Screening processing module

4, 4a: Association processing module

5a: Integrate modules

6a: Comparison module

7a: Intersection module

8a: Analysis module

9, 9a: word break system

[FIG. 1] A flowchart of a method for filtering and correlating text data according to the first embodiment of the present invention.

[FIG. 2] A block diagram of one of the specific embodiments of the method of the present invention that can automatically execute the first embodiment.

[FIG. 3] A flowchart of a method for filtering and correlating textual data according to the second embodiment of the present invention.

[FIG. 4] A block diagram of one of the specific embodiments of the method of the present invention that can automatically execute the second embodiment.

1 to 2 are the first embodiment of this creation. As shown in FIG. 1 , the method for screening and correlating written text data includes the following steps: S11. Based on a word segmentation vocabulary database, perform word segmentation processing on a text data to generate word segmentation information; S12. The word segmentation information Perform screening processing to generate a screening segmentation information; the screening segmentation information has more than two screening segmentations; S13. Perform correlation processing on the screening segmentation information to generate multiple correlation sequences Column information; the related sequence information is composed of two or more adjacent screening segmented words; by this method, the brief information of the text data can be quickly sorted out. This will be explained in detail below.

Step S11 is to perform word segmentation processing on a word data based on a word segmentation vocabulary database to generate word segmentation information.

The text data can be various published text data, such as doctoral and master theses, academic papers, general articles or sentences. In addition, for large-length text materials such as papers, the paper can be directly regarded as one text data, or multiple text materials can be formed after the paper is processed into sections. There are many ways of segmentation processing, which are illustrated as follows. When performing segmentation processing, you can use symbols such as line breaks, continuous spaces, exclamation points (!), semicolons (:), tildes (~), question marks (?), commas (,), periods (.)...etc. On the basis of this, divide a piece of written data into multiple pieces of written data with its length not less than the appropriate length as the dividing point. When performing segmentation processing, it is also possible to divide a text data into multiple text data based on each chapter and section of the text data. When performing segmentation processing, it can also be used together with the word segmentation vocabulary database. Based on the method of screening a predetermined number of word segmentations such as ten, twenty, etc. as a segment, a text data is divided into multiple text data. .

Word segmentation processing is to convert text data into word segmentation information based on multiple words recorded in the word segmentation vocabulary database. Multiple words in the word segmentation vocabulary database can be classified according to part of speech, such as common noun (Na), foreign language (FW), action transitive verb (VC), action intransitive verb (VA), local word (Nc), Proper Noun (Nb), Verb of Condition (VHC), Colon (COLONCATEGORY) ... and other part-of-speech classifications.

Step S12 is to filter the segmented information to generate a segmented segmented information; the segmented segmented information has more than two segmented segments. The filtering process is to retain some meaningful parts of speech in the segmentation information, and remove other parts of speech, such as retaining common noun (Na), foreign language (FW), action transitive verb (VC), action intransitive verb (VA) , local words (Nc), proper nouns (Nb), state verbs (VHC)...etc. All words retained after the screening process are collectively referred to as screening segmented words.

Step S13 performs correlation processing on the filtered segment information to generate a plurality of related sequence information; the related sequence information is respectively composed of two or more adjacent filtered segment information. Combining two or more adjacent filtered and segmented words through correlation processing can distinguish text data in the same field but different technical characteristics to a certain extent, especially between text data with the same keywords. difference.

The first embodiment of the present creation is a method of quickly sorting out word segmentation information that is closer to the original meaning of text data than keywords. Whether it is used to analyze other people's text data or your own text data, it can quickly sort out text data. The purpose of brief information is to facilitate the analysis and utilization of text data.

As shown in FIG. 1 , before step S11 is performed, step S110 may be performed first; step S110 is: collecting author-defined keywords in the text data to establish a professional keyword vocabulary database, and storing the professional keyword vocabulary database Import the word segmentation vocabulary. Generally speaking, text materials such as papers have keywords customized by the author, and the keywords include, for example, proper names, scientific and technological The key words customized by the author are added to the word segmentation vocabulary database, and then word segmentation processing and subsequent steps are performed, so as to obtain related sequence information that is closer to the original meaning of the text data.

FIG. 2 shows one specific embodiment of a text data screening and correlation system that can automatically execute the text data screening and correlation method of the first embodiment. As shown in FIG. 2, the present creation provides a text data screening and association system 100, which includes: a storage module 1 for storing a word segmentation vocabulary database; a word segmentation processing module 2 for performing a text data processing word segmentation processing to generate word segmentation information; a screening processing module 3 for performing screening processing on the word segmentation information to generate a screening segmentation information; a correlation processing module 4 for the screening segmentation Correlation processing is performed on the word information to generate a plurality of related sequence information. The storage module 1, the word segmentation processing module 2, the filtering processing module 3, and the correlation processing module 4, etc. can be established in one or more computers and/or cloud servers. When the text data screening and correlation system 100 is established in a cloud server, a corresponding web page may be provided, and the user can obtain a plurality of correlation sequence information (not shown in the figure) after inputting the text data.

3 to 4 are the second embodiment of the present creation. As shown in Figures 3 to 4, the method for screening and correlating textual data of this creation includes the following steps: S21. Create a set of contrasting information based on two or more contrasting textual data; S22. Based on a word segmentation vocabulary database, compare these Perform word segmentation processing against the text data to respectively generate a comparison word segmentation information; S23. Perform a screening process on the comparison word segmentation information to respectively generate a comparison screening word segmentation information; the comparison screening word segmentation information has two or more pieces respectively The screening segment; S24. Screen these comparisons Perform correlation processing on the selected segmented word information to generate a plurality of comparison related sequence information respectively; the comparison related sequence information is respectively composed of two or more adjacent screening segmented words; S25. Relevant all comparisons The sequence information is integrated together to create a related index file; this method can quickly sort out the brief information of multiple reference text data, and integrate the brief information of each reference text data together, so as to facilitate the analysis of the text data to be compared originality.

Step S21 is to create a set of comparison information with more than two pieces of comparison text data. The reference set information can contain various text data, for example, some or all of the theses in the Taiwan Ph. In addition, when creating the comparison collection information, different photo collection information can be created in different scopes, such as electronic, mechanical, text data within 10 years, etc. The text data for comparison and the text data to be compared in the second embodiment are the same as the text data in the first embodiment, and can be various published text data, such as doctoral and master theses, academic papers, general articles or sentences etc., the difference lies in that in the second embodiment, the text data to be compared need to be compared and analyzed with each reference text data one by one, so there are different names to facilitate the distinction.

Steps S22 to S24 are to perform word segmentation processing, screening processing and correlation processing on each piece of reference text data in the reference collection information, respectively, to generate reference word segmentation information, comparison screening word segmentation information, and multiple comparison correlation sequences. Information.

Step S25 is to integrate all the reference related sequence information to create a related index file. Integrate the established relational index file to It is then compared with the text data to be compared, so as to facilitate the analysis of the originality of the text data to be compared.

As shown in FIG. 3 , before step S22 is performed, step S220 may be performed first; step S220 is: collecting the comparison text data and a part or all of the author-defined keywords in the text data to be compared to establish a professional key The word vocabulary database is imported, and the professional keyword vocabulary database is imported into the word segmentation vocabulary database, so as to obtain a related index file that is closer to the original meaning of the text data. In addition, the work of removing duplicates can be added to the sorting of the professional keyword vocabulary database, so as to increase the processing efficiency.

The second embodiment of the present creation can quickly sort out the brief information of the reference text data, and further integrate the brief information of each reference text data together, so as to facilitate the user to compare and analyze the text data to be compared. For example, the following steps S26-S29 are used to analyze the originality of the text data to be compared.

Step S26 is to perform word segmentation processing, screening processing and correlation processing on a text data to be compared to generate a plurality of related sequence information to be compared. The processing methods of steps S22-S24 and S26 are the same as those of steps S11-S13, so the generated comparison related sequence information and the related sequence information to be compared have corresponding types, which can facilitate the comparison. In addition, in S12, S23/or S26, after the screening process, the synonym process may be performed first, and then the subsequent steps may be performed. The processing of synonyms is: check the text synonyms of the screened and segmented words after screening, and convert some or all synonyms and synonyms (except some special words that are not suitable for synonym processing) into standard text, which can increase the correlation comparison. Effect. For example, change "air conditioner" and "air conditioner" to " In addition, the related sequence information to be compared and the related sequence information to be compared can be composed of two or more adjacent screening segments. In the related sequence information to be compared and the related sequence information to be compared, The greater the number of screening segmented words, the easier it is for the relative sequence information to be compared and the related sequence information to be compared to reflect the concept of the corresponding text data, but it may also result in too many restrictions and the text to be compared cannot be found. The data is similar to the comparison of text data. Therefore, basically two adjacent screening segments are used to form the comparison related sequence information and the related sequence information to be compared. In many cases, in order to speed up the analysis, three or more adjacent screening segments can be used to form the related sequence information for comparison and the related sequence information to be compared.

Step S27 is to compare the correlation sequence information to be compared with the correlation index file, respectively, and to find each reference text data that has the same correlation sequence information to be compared with the correlation sequence information to be compared. With the above-mentioned method for screening and correlating textual data, the correlation between the textual data to be compared and each comparison textual data can be quickly analyzed, thereby facilitating the analysis of the originality of the textual data to be compared. In addition, the relational index file has a simple format, which can easily add new reference relational sequence information, which can overcome the shortcoming of the conventional reverse database requiring frequent system reorganization due to new data.

The following example outlines how hyphenation, etc. processing works. The numbering of each example is only for the convenience of description, and should not limit the meaning of this creation. Step S21 is to create a set of comparison information, and each comparison text data can be numbered in sequence, for example, the comparison text data with the number 1 is recorded as ID1. The reference set information is a set of storing ID1, ID2, . . . , IDn.

Step S22 performs word segmentation processing.

In step S23, a screening process is performed, and each screening segment can be numbered in sequence, for example, the first reserved screening segment of ID1 is recorded as ID1tp1.

In step S24, correlation processing is performed, and each reference correlation sequence information may be sequentially numbered, for example, the first reference correlation sequence information of ID1 is recorded as ID1S1.

Step S25 establishes a relevancy index file, and each comparison and screening word segmentation information can be regarded as an index (namely called Index or Key) of the relevant index file, and the number of the comparison filter word segmentation information can be used as the data of the relevant index file (Data). When creating a related index file, any one of the comparison filter segmentation information may be the same as another comparison filter segmentation information (eg ID1S2, ID2S1). Therefore, an index can be compared with a plurality of different data, the amount of the data is large, and the total data length stored in it increases as more reference text data is added.

In step S26, word segmentation is performed on the text data to be compared, and the text data to be compared can be recorded as IDx.

Step S27: Use the related sequence information to be compared as an index To search, read all data with the same index in the associated index file.

Step S28 is to create an intersection sequence, which can arrange all the related related sequence information that is the same as the related related sequence information to be compared (ie, sorting).

Step S29 is to analyze each piece of reference text data having the same related sequence information as the text data to be compared, so as to generate an originality analysis result of the text data to be compared with respect to each piece of reference text data. There are many comparison methods. For example, statistical analysis methods are used to analyze the similarity reference ratio of each comparison text data in the comparison set information of the text data to be compared, and commonly used theories such as the Dice Coefficient rule can be used. In addition, simple analysis can also be performed in an easy-to-understand and generalized way.

By means of the above-mentioned method for screening and correlating textual data, the correlation between the textual data to be compared and each reference textual data can be quickly analyzed, and the originality of the textual data to be compared can be further analyzed.

FIG. 4 shows one specific embodiment of a text data screening and correlation system that can automatically execute the text data screening and correlation method of the second embodiment. As shown in FIG. 4 , the present invention provides a text data screening and association system 100a, which includes: a storage module 1a for storing a word segmentation vocabulary database and a comparison set information; a word segmentation processing module 2a for using performing word segmentation processing on each comparison text data of the comparison set information to respectively generate a comparison segmentation information; a screening processing module 3a is used to perform screening processing on the comparison segmentation information to respectively generate a comparison screening word segmentation information; a correlation processing module 4a for performing correlation processing on the comparison and screening word segmentation information to generate a plurality of comparison correlation sequence information respectively; an integration module 5a for correlating all comparisons The sex sequence information is integrated together to create a relational index file. In addition, the word segmentation processing module 2a, the screening processing module 3a and the correlation processing module 4a can further perform word segmentation processing, screening processing and correlation processing on a text data to be compared to generate a plurality of correlations to be compared Sex sequence information, and the screening association system for the text data 100a further includes: an alignment module 6a, respectively aligning the related sequence information to be compared with the related index file to find out the related sequence information that has the same relative sequence information to be compared with the related sequence information to be compared Each of the comparison text data; an intersection module 7a, arranging all the comparison related sequence information that is the same as the related sequence information to be compared, so as to establish an intersection sequence; an analysis module 8a, analyze each copy and the related sequence information to be compared. A comparison text that has the same sequence information of relevance to the text.

The storage module 1a, the word segmentation processing module 2a, the screening processing module 3a, the correlation processing module 4a, the integration module 5a, the comparison module 6a, the intersection module 7a, and the analysis module 8a can be established in one or multiple computers and/or cloud servers. When the text data screening and association system 100a is established in a cloud server, a corresponding web page can be provided, and the user can obtain originality analysis results after inputting the text data to be compared (not shown in the figure).

In addition, the aforementioned parts related to word segmentation processing, such as steps S11, S22 and word segmentation vocabulary database, etc., can use conventional word segmentation systems such as CKIP developed by Taiwan Academia Sinica or Jaba which has published computer code. , 9a, in order to save costs.

As mentioned above, the written data can be various kinds of written data that have been published, and for large-scale written data such as papers, the paper can be directly regarded as a written data, or it can be formed after the dissertation is processed in sections. Multiple texts. These multiple pieces of text data formed by segmental processing can be additionally correlated with each other so as to make a unified original analysis result. For example, the number of a paper is IDa1, and the number of the paper after being segmented (for example, by chapters) is IDa2~IDan, that is, Not only the thesis is regarded as a written material, but each subsection (each chapter) of the thesis can also be regarded as a written material. In this way, after analysis, not only the originality analysis results of the text to be compared relative to the paper can be obtained, but also the results of the original analysis of the text to be compared relative to each section (each chapter) of the paper. Originality analysis results.

The above-mentioned examples of this creation are only for the convenience of description, and should not limit the meaning of this creation, that is, all kinds of transformation designs based on the listed patent application scope should be included in the patent scope of this creation. middle.

100a: Screening and Association System for Text Data

1a: storage module

2a: Word segmentation processing module

3a: Screening processing module

4a: Association processing module

5a: Integrate modules

6a: Comparison module

7a: Intersection module

8a: Analysis module

9a: Hyphenation system

Claims (3)

A screening and association system for text data, including: a storage module for storing a word segmentation vocabulary; A word segmentation processing module for performing word segmentation processing on a text data to generate word segmentation information; a filtering processing module for filtering the segmented information to generate a filtered segmented information; A correlation processing module is used to perform correlation processing on the filtered word segmentation information to generate a plurality of correlation sequence information. A screening and association system for text data, including: a storage module for storing a word segmentation vocabulary database and a comparison set information; a word segmentation processing module for performing word segmentation processing on each comparison text data of the comparison set information to generate a comparison word segmentation information respectively; a screening processing module for performing screening processing on the comparison segment information to generate a comparison filter segment information respectively; a correlation processing module for performing correlation processing on the comparison screening segment information to generate a plurality of comparison correlation sequence information respectively; An integration module is used to integrate all the reference related sequence information together to create a related index file. According to the text data screening and correlation system of claim 2, wherein the word segmentation processing module, the screening processing module and the correlation processing module perform word segmentation processing, screening processing and correlation processing on a text data to be compared The method is used to generate a plurality of related sequence information to be compared, and the screening and correlation system for the text data further includes: an alignment module, which is used to compare the related sequence information to be compared with the related index file respectively, Find out each comparison text data with the same comparison-related sequence information as the waiting-aligned-related sequence information; an intersection module, arranging all the comparison-related sequence information that is the same as the to-be-aligned related sequence information, Thereby, an intersection sequence is established; an analysis module analyzes each control text data with the same related sequence information as the text data to be compared.

Images