ES2320511B1 - NEW METHOD TO DETERMINE THE REPRESENTATIVITY OF A CORPUS. - Google Patents

Abstract

New method to determine the representativeness of a corpus.

The present invention is an effective solution to retrospectively determine the minimum size of a corpus or collection textual, regardless of the language or text type of the collection, establishing therefore the minimum threshold representation through an algorithm (N- Cor) of lexical density analysis based on the incremental increase of the corpus. From this premise, we have reached a proposal for computer implementation that has been specified in an application developed in Java, and that we have called ReCor. This system has the following main classes: a) Words (algorithm of computation, reading and writing to file); b) Gui (user interface); and c) Graphic window (adapter for graphic representation).

Description

       \ global \ parskip0.900000 \ baselineskip
    

New method to determine the representativeness of a corpus.

Technical sector

The present invention relates to a method of data processing implemented in computer, particularly linguistic data and information

State of the art The question of representativeness is still one of the most important aspects controversial of the linguistics of the corpus. In the case of specialized corpus, which usually have a much more size reduced than the so-called "general corpus" or "of reference ", the question of representativeness is really key, moreover, is one of its defining characteristics.

Leaving aside that the representativeness of a corpus depends, first of all, on having applied the appropriate external and internal design criteria, in practice the quantification of the minimum size that a specialized corpus must have has not yet been objectively addressed . And there is no consensus on what is the minimum number of documents or words that a given corpus must have in order to be considered valid and representative of the population that you want to represent. The figures vary dramatically from one author to another. Thus, if for Biber (1995. Dimensions of Register Variation: A cross-linguistic comparison . Cambridge University Press), 1000 words and 10 documents are sufficient to ensure the representativeness of a specialized corpus; according to Friedblicher and Friedblicher (2000. The Argument for Using English Specialized Corpora to Understand Academic and Professional Language . Discourse in the Professions: Perspectives From Corpus Linguistics. John Benjamins), the size ranges from 500,000 to 5,000,000 words; while McEnery and Wilson (2006 [2000]. ICT4LT Module 3.4. Corpus Linguistics. <http://www.ict4lt.org/en/en_mod3-4.htm> [09/11/2006]) set the limit at 1,000 .000 words. But all these figures do not solve the problem of calculating the representativeness of a corpus, since they are a priori established figures, lacking objective, measurable and quantifiable basis.

Detailed description of the invention

The present invention is an effective solution to retrospectively determine the minimum size of a corpus or collection textual, regardless of the language or text type of the collection, establishing therefore the minimum threshold representation through an algorithm (N- Cor) of lexical density analysis based on the incremental increase of the corpus.

From this premise has reached a proposal of implementation in computer that has been specified in an application developed in Java, and that we have called ReCor. This system has the following main classes: a) Words (algorithm of computation, reading and writing to file); b) Gui (user interface); and c) Graphic Window (adapter for graphic representation).

Description of the drawings

Figure 1: System use life cycle

Figure 2: Exemplification of representations graphs A and B

Figure 3: Window implementation graph

Figure 4: Order class Frequency defined in the compareTo method of the Comparable interface

Figure 5: Gui class for the creation of graphical user interface

Figure 6: Word class for the analysis of corpus

Figure 7: Controller class that specifies the action associated with each event

Figure 8: Route class with main method.

Embodiments of the invention N-Cor algorithm

As set forth above, the present method calculate the minimum size of a corpus by analyzing the lexical density (d) in relation to the incremental increases of corpus (C) document by document, as shown in the following equation:

one

To do this, all the files that make up the corpus are analyzed gradually, extracting information about the frequency of the type words ( types ) and the occurrences or different words ( tokens ) of each file of the corpus. In this operation, two file selection criteria are used, namely in alphabetical order and randomly,

2

where:

       \ global \ parskip1.000000 \ baselineskip
    

Ty: It refers to types , that is, the number of different words up to that time.

To: Shows the tokens , that is, the total number of words up to that time.

N: Number of documents that make up the corpus

The life cycle of using the system can be as follows (Figure 1): Each file that integrates the corpus must be uniquely identified by an alphanumeric code name (for example, A001, A002, A003 ... A00n). The algorithm operates first in alphabetical order and then randomly, in order to ensure that the order in which the files are selected does not affect the result. When the documents are selected in alphabetical order, the algorithm analyzes the first file (for example, A001) and for this the tokens (To) and types (Ty), and the corresponding lexical density are calculated. With this, a point is already obtained in the graphic representation that is intended to be extracted. Then, following the same selection criteria as in the first one, the following document of the corpus is taken (for example, A002) and the To and Ty are calculated again for it, but adding the results to the Ty and To of the previous iteration (in this case those of the first document analyzed), the lexical density is calculated and with this a second point for the graphic representation is obtained. This algorithm is followed until all the documents that make up the corpus being studied have been treated, which in this example would be the A00n. The second phase of the analysis takes the documents in random order, for example the first A003, then the A00n, and so on until all the documents of the corpus have been analyzed.

This is the same algorithm for the analysis of n-grams, that is, the option to perform an analysis of the frequency of occurrence of word sequences (1-gram, 2-gram, ..., n-gram). The application offers the possibility of computing these sequences considering a range of sequence lengths (number of words) defined by the user. As is done with independent words ( tokens ), a graphic with the representative information of the corpus is shown both for a random order of the files and for an alphabetical order by their name. On the horizontal axis the number of files consulted is maintained, and on the vertical axis the ratio (number of different n-grams) / (number of total n-grams). For these purposes, an n-gram is considered a token . Likewise, the generated output files indicate the n-grams.

Both in the analysis in alphabetical order and in the random one there comes a time when a certain document does not contribute just types to the corpus, which indicates that it has reached an adequate size, that is, that the analyzed corpus can already be considered a representative sample of the population in statistical terms. In a graphic representation we would be at the point where the lines of types and tokens stabilize and approach zero (Figure 2).

If the corpus is really representative, the graph tends to descend exponentially because the tokens (To) grow in each iteration much more than the types (Ty), because, in theory, fewer new words are appearing that are not stored in the data structures used by the program. Thus, we can say that the corpus is representative when the graph is constant at values close to zero, because, in practice, it is impossible to achieve the incorporation of zero types in the corpus since the documents will always contain variables of the type number , proper names, etc.

If a corpus produces this graphic representation we can affirm that it is representative and that X is enough for us files (those that correspond to the point of the horizontal axis where the graph stabilizes around zero). In this way we will have identified the minimum size of the collection, from which It can be considered representative.

       \ newpage
    

Below is the source code of the N-Cor algorithm:

3

5

       \ newpage
    

Implementation Proposal

The N-Cor method and algorithm described above have been implemented in computer using the ReCor application. The tool used has been Java 2 SDK, Standard Edition (J2SE), plus the Java library for graphics and JFreeChart diagrams. As editor and compiler of Java has been resorted to the JCreator Pro environment. As already stated, the system is not structured in packages, and has the following main classes:

- Words: Computation algorithm, reading and writing to file

- Gui: User interface.

- VentanaGrafica: Adapter for graphic representation.

In this section we deal with UML design (unified modeling language), while addressing the main classes created for the application and software of chosen development.

       \ vskip1.000000 \ baselineskip
    

Class Diagram

In this section the variables and the methods of each class created for the application; also be they treat the interactions of different classes with each other.

       \ vskip1.000000 \ baselineskip
    

1. Graphic Window Class

This class (Figure 3) is responsible for present on screen the graphic representation of the corpus. For the implementation of this class library methods have been used  JFreeChart Extends the java class called Frame.

       \ vskip1.000000 \ baselineskip
    

2. Order Frequency class

This class (Figure 4) is necessary to order the different words that appear throughout the corpus depending on the total number of occurrences they have. This is essential for the creation of one of the output files. This class defines in the compareTo method of the Comparable interface.

       \ vskip1.000000 \ baselineskip
    

3. Gui class

This class (Figure 5) creates the graphical user interface and serves to define the buttons, labels, drop-down lists, checks , etc. Extends the java class called JFrame.

       \ vskip1.000000 \ baselineskip
    

4. Word class

It is the most important class; in it (Figure 6) the corpus is analyzed (for example, the pairs [word, number of occurrences] in a Hash table), the files of output and the different points of the functions are calculated for the graphic representation.

       \ vskip1.000000 \ baselineskip
    

5. Controller Class

In this class (Figure 7) the action associated with each event that occurs in the graphical user interface is specified. This class defines the actionPerformed method of the ActionListener interface.

       \ vskip1.000000 \ baselineskip
    

6. Route class

This class (Figure 8), which has the main method, it prepares everything to start the execution of the application and creates an object of the Gui class (affected by the Controller class).

       \ vskip1.000000 \ baselineskip
    

specs

Given the set of documents that make up the corpus, the program extracts information from it in several files, in addition to the two graphs, which are used to study the corpus selected.

Input data

to)

Corpus files: Selection of set of files that make up the (sub) corpus.

b)

File word filter: File entry, in which are collected the different words that are not you want to analyze, that is here are all the words you want filter (separated by space, comma, period, semicolon, two points or line break).

C)

Choice of parameters: Choice of word group size (1, 2, ..., 10), or n-grams For the selection of each group, it use the sliding window method, regardless comma contexts, points, paragraphs, etc. You can also choose if for the corpus to be analyzed you want to filter or not numbers.

Output data

The output data comprises a graphic representation of the corpus. The number of files selected up to that moment is represented on the horizontal axis and the types / tokens ratio on the vertical axis. Two functions have been represented, one for files sorted by name, and another for files randomly chosen. Both functions tend to decrease exponentially the more documents we take. When the functions are stabilized, it can be affirmed that the corpus is representative and it can be determined approximately from what number of documents this is produced.

Graphic representation

The graphic representation is what allows the user to decide whether a corpus is representative or not. Thus, there are two interesting graphical representations: the first - graphical representation A - is the one that places the vertical axis Ty / To and on the horizontal axis To; the second - graphical representation B - places the Ty / To on the vertical axis and on the horizontal axis the number of files analyzed up to that point. The first graphic representation would indicate the minimum number of words that the collection should have, while the second one specifies the number of documents or texts.

It can be said that the corpus is representative when the graph stabilizes around zero values, and tends to descend exponentially because the tokens (To) will grow in each iteration much more than the types (Ty), because in theory every time fewer new words will appear as the lexical density of the incremental subset of documents analyzed is analyzed.

Output files

In addition to the graphic representation, it is also extract corpus information in several output files:

       \ vskip1.000000 \ baselineskip
    

a) Output file No. 1

This file contains five columns:

?

Ty: Shows the types (different words) until that moment.

?

To: Shows the tokens (number of words in total) until that moment.

?

Ty / To: quotient between types and tokens.

?

 V1: The number of words with so Only one appearance so far.

?

 V2: The number of words with so Only two appearances so far.

Show the results of two analyzes different, one for files sorted alphabetically by name and other for files sorted in random order. For each of these analyzes will be as many lines as files have The selected corpus. The number columns will be five (those above described).

       \ vskip1.000000 \ baselineskip
    

b) Output file nº 2

This file is made up of two columns:

?

 Word: All are collected here the different words that make up the corpus (ordered alphabetically by the name of the document).

?

 Appearances: Refers to number of occurrences in the word corpus in question.

c) Output file nº 3

This file is made up of two columns:

?

 Word: All are collected here the different words that make up the corpus (sorted by number of appearances).

?

 Appearances: Refers to number of occurrences in the word corpus in question.

Claims (5)

1. Method implemented in a computer to determine the representativeness of a corpus by executing a program characterized by:

?

 It is language independent or textual type of the collection of documents analyzed,

?

 Set the minimum threshold of representativeness through an algorithm (N-Cor) of lexical density analysis as a function of the increase incremental corpus,

?

 Understand input data, output data, graphic representation, and files exit,

?

It includes the gradual analysis of all the files that make up the corpus, extracting information on the frequency of such words (types) and various occurrences or words (tokens) of each file corpus.

?

Each file that integrates the corpus must be uniquely identified by an alphanumeric code name; proceeding first to an analysis in alphabetical order and then to a random analysis; calculating in each case and for each document the tokens , types , and corresponding lexical density; which finally allows to obtain a graphic representation indicative of the representativeness of the analyzed corpus.

2. Computer-based method of determining the representativeness of a corpus by executing a program according to the preceding claim characterized in that, based on said N-Cor algorithm, it is possible to perform an analysis of the frequency of occurrence of word sequences, being able to the computation of said sequences be made considering a range of lengths defined by the user. 3. A computer-based method of determining the representativeness of a corpus by executing a program according to any of the preceding claims, characterized in that said computer application (ReCor) has been developed in Java 2 SDK Standard Edition using the Java library for graphics and JFreeChart diagrams; and using the JCreator Pro environment as a Java editor and compiler. 4. A computer-based method of determining the representativeness of a corpus by executing a program according to the preceding claim characterized in that it comprises the following classes:

?

 Graphic Window Class, whose function is to present on screen the graphic representation of corpus and its implementation is done using library methods JFreeChart;

?

 Class Order Frequency, involved in the ordering of the words that appear in the corpus depending on the total number of occurrence of the themselves;

?

 Gui class, whose function is create the graphical user interface;

?

 Word class, involved in the Corpus analysis, creation of output files, and the calculation of the different points of the functions for the graphic representation.

?

 Controller Class, whose function is specify specifies the action associated with an event in the graphical user interface;

?

 Route class, which allows you to start running the application and creates an object of the class Gui.

5. Electronic device programmed to determine the representativeness of a corpus according to any of the preceding claims characterized in that it allows establishing the minimum threshold of representativeness through the N-Cor algorithm by means of the execution of the ReCor computer application.