RU2386167C1 - Device of information processing for information searching - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: device comprises memory (1) of initial array, unit (2) of symbol text processing, patch board (3), unit (4) of display, unit (5) of input-output, buffer memory (6) of input-output, unit (8) of text structural analysis, control unit (13), unit of archive card synthesis (14), memory (15) of archive cards, field bus (20). Technical result is achieved by introduction of long-term memory (7), unit of service information identification (9), unit (10) of stop-vocabulary, unit (11) of quasimorphological analysis, unit (12) of word bases inverted index generation, unit (17) for generation of criteria for sentences selection for archive card, unit (18) for sentences selection for archive card and counter (19) of symbols in text annotation with according links.

EFFECT: wider area of application and functional resources of information processing device due to provision of the possibility to process and search text information of theme and semantic varieties, and automatic adaptation of device to changes in subject area of processed information on the basis of complete exclusion of a human being from the process of analysis, reading, annotation and cataloguing of texts.

8 dwg

Description

The invention relates to technical means of computer science and computer technology and can be used to solve problems of symbolic processing of text information and preliminary processing of text data for information retrieval.

A device for information retrieval (patent RU 2039376, IPC G06F 17/30), comprising a memory interface unit, a source array memory, an input-output unit, a dial-up field, a display unit, an input-output buffer memory, a control unit, a buffer memory, trunk bus, including address, information and control buses.

A disadvantage of the known device is that the original array of textual information for processing must be converted to a special form used in the device, the result of processing the information does not make it possible to evaluate the content of the text and cannot be used as an annotation to the processed text, the device does not provide the ability to select data .

The closest in essence to the claimed invention is an information processing device for information retrieval (patent RU 2096825, IPC G06F 17/00, 17/30), which contains a memory interface unit, a source array memory, an input-output unit, a type-setting field, display unit, input / output buffer memory, control unit, buffer memory, text fragment synthesis unit, text fragment structural analysis unit, text fragment symbol processing unit, word dictionary corrector adjustment unit, primary synthesis unit of the first dictionary, filtering unit of the primary dictionary, block for synthesizing secondary dictionaries, block for analyzing secondary dictionaries, block for synthesizing an archive card, block for archiving text fragments, block for adjusting system dictionaries, memory for an archive card, a trunk bus that includes address, information, and control buses.

In the device corresponding to the prototype, the process of processing information and forming an archive card is divided into two stages.

At the first stage, the device operates under the control of the operator. The result of the first stage of work is another text fragment, which has a trivial meaning: a separate chapter in a document, a separate paragraph, a separate article in a collection of articles, etc.

At the second stage, the device operates in automatic mode as follows.

The device operator generates a device activation command in the typed field. From the typesetting field to the input of the I / O unit, a signal is received that the device starts to work in automatic mode. From the output of the input-output unit via the bus, the signal is transmitted to the input of the control unit, where it is converted into a sequence of device control commands. The control unit sequentially activates the blocks of structural analysis of a text fragment, symbolic processing of a text fragment, corrections of a dictionary of word combinations, synthesis of a primary dictionary, filtering of a primary dictionary, synthesis of secondary dictionaries, analysis of secondary dictionaries, synthesis of an archive card, archiving of text fragments, adjustment of system dictionaries by issuing control commands to the inputs of these blocks. The signal of the result of the operation of the device (archive card of a text fragment) is fed to the memory input of archive cards, where it is recorded for future use.

The principle of operation of all of these blocks is the same. At the input of block No. k, the signals from the information array of the buffer memory formed by block No. (k-1) are sequentially received via the bus line. From the output of block No. k, the signals are sequentially fed into the information array of the buffer memory, and from it to block No. (k + 1).

The block for structural analysis of a text fragment selects information signals about the presence in the input data of strings with a certain percentage of digital characters, as well as signals of a character information array of a given length. The signal structure of the specified output is stored in the block of structural analysis of the text fragment when setting up the device. The device is set up before it starts.

In the block of symbolic processing of the text fragment, the input signals are processed corresponding to the decomposition of the source information into separate words and phrases (a set of words limited in quotation marks in the source text information). From the output of the symbol processing block of a text fragment on the buffer memory, an information array signal is received, the structure of which allows to read a single word or phrase from the buffer memory, as well as to distinguish when searching for words and phrases, while data signals on the number of lines and words processed in the buffer memory are written text fragment information.

In the correction block of the dictionary of phrases, the signals of the number of characters in the information read from the block are compared with the reference signal. If the number of characters of the reference value is exceeded, the signal read from the buffer memory is subjected to processing in the corrector of the dictionary of phrases corresponding to the decomposition of phrases into individual words. The results of processing information signals in block 1 of the dictionary corrections are stored in its information array, the structure of which allows you to search for the desired word and store, in addition to symbolic information, also numeric information associated with the addressing of data to a specific word.

The primary dictionary synthesis block generates a signal of a new information array of the same structure, but does not contain the same words and phrases. The numerical data signals associated with each word of the specified information array contain information about the number of such words in the input information signal of the primary dictionary synthesis block.

The primary dictionary filtering unit is designed to generate a signal of a new information array of the same structure. In the filter block of the primary dictionary, a signal is generated that corresponds to the result of the operation of deleting from the input information array information about those words that coincide with the words of the information array of the same structure recorded in the memory of the filter block of the primary dictionary when setting up the device. If the word signal from the input information of the filtering unit of the primary dictionary satisfies the comparison conditions with the words of the internal memory of the block, the data about it does not fall into the output information array. The comparison operation is carried out in two stages. At the first stage, the word signals are compared according to the scheme of complete coincidence, at the second stage, the word signals from the input information are evaluated for the coincidence of the symbolic structure of words (for example, the sequences of the characters "worked" and "works." Coincide in the symbolic structure).

In the block for synthesizing secondary dictionaries, the input information signal is processed corresponding to the division of the data of the input information array into four independent information arrays of the same structure. The signal of each word from the input information passes symbol-by-symbol processing in the block for synthesizing secondary dictionaries to determine the type of each character ("lowercase", "uppercase", "digital", "alphabetical", "special", "significant", "space"). Signals of words consisting only of digital and special characters are excluded from further processing. The signals of the remaining words (and the data corresponding to them) are summed with the signals of one of the output information arrays: "proper names", "abbreviations", "quotation marks", "ordinary words".

The secondary dictionaries analysis unit analyzes the signal of the identification code of the information array, which is read from the output of the buffer memory, and performs one of three processing options for the input information signal depending on the value of the identification code. The first processing option is to read the input signal and write it to one of the output information arrays, the other two options are based on the selection of the sequence of characters of the word base from the word signal by removing suffix and ending characters and assessing the coincidence of the character structure of the obtained sequence of characters with the signals of the remaining words current information array.

The synthesis card synthesis unit generates an output signal, addresses for storing in the archive card memory of a new archive card, addresses the input information for information retrieval, generates an output information signal for a new archive card.

The block for archiving text fragments carries out processing of input signals corresponding to the operations of determining the address of the archive region of the buffer memory for storing a fragment, verifying its existence and, if necessary, creating and identifying it, compressing the initial information.

In the block for adjusting system dictionaries, the address of access to the last archive card is determined, moreover, the output signal of the address from the output of the block for adjusting system dictionaries is fed to the input of the control unit. The control unit generates a command to read the archive card and, together with the signal of the address of access to the card, directs it to the memory input of the archive card. At the same time, the control unit supplies the preparation control signal for recording the area of system dictionaries to the input of the buffer memory.

From the output of the archive card memory, the signals of six information arrays recorded in the archive card memory from the output of the archive card synthesis block are removed. Information signals from the output of the archive card memory are fed to the input of the system dictionaries correction block.

The system dictionaries adjustment unit performs processing of input information signals corresponding to the conversion of input information to the form of a word list without any additional numerical information, and the generated information signal from its output is fed to the input of the buffer memory. After performing this operation, the system dictionaries correction unit sends a command to shut down the device to the input of the control unit.

The control unit generates a sequence of control commands corresponding to clearing the buffer memory, with the exception of the memory areas of system dictionaries and archive areas for storing signals of compressed information of text fragments, and after the execution of the specified sequence of commands ends the operation of the device.

The disadvantage of the prototype is the schematic and conventional display of the contents of a text document, the involvement of the human operator in the preliminary processing of information,

The aim of the invention is to expand the scope and functionality of the device by providing the ability to process and search for textual information of various thematic and semantic directions, as well as automatically adapt the device to change the subject area of the processed information based on the complete exclusion of a person from the process of analysis, reading, annotation and cataloging texts.

The goal is achieved by the fact that in the known information processing device for information retrieval, containing a sequentially connected typesetting field, a display unit, which is the output of the device as a whole, an input-output unit, a trunk bus, to which the inputs and outputs of the control unit and archive card memory are connected the second output of which through the input-output unit is connected to the second input of the display unit, and the input of the archive card memory is connected to the output of the synthesis card synthesis unit, the memory of the original array, the first the first output of which through the symbol processing unit is connected to the input of the text structural analysis unit, the input-output buffer memory, the input of which is the input of the device as a whole, according to the invention, long-term memory, an auxiliary information allocation unit, a stop dictionary storage and correction unit, a block are introduced quasimorphological analysis, a block for generating an inverted index of the foundations of words, a block for forming the associated basics of words, a block for generating signs of selecting proposals for an archive card, a block for selecting proposals for the archive card and a character counter in the annotation of the text, the output of the I / O buffer memory via a long-term memory connected to the main bus connected to the memory input of the source array, the second output of which is connected through the service information allocation unit to the first input of the archive synthesis unit cards, and through series-connected the block of selection of offers for the archive card and the character counter in the annotation of the text is connected to the second input of the synthesis block of the archive card, to the third input which has a serially connected block for storing and adjusting the stop dictionary, a quasimorphological analysis block, an inverted index base word formation block, and an output of a text structural analysis block through a quasimorphological analysis block and a serially connected block word formation block and sentence selection formation block for the archive card is connected to the second input of the selection block for the archive card.

A comparative analysis of the technical solution with the device selected as a prototype shows that the novelty of the technical solution consists in maintaining new circuit elements in the claimed device: long-term memory, service information allocation unit, stop dictionary storage and adjustment unit, quasimorphological analysis unit, formation unit inverted index of word stems, block of formation of related word stems, block of formation of signs of selection of sentences for an archive card, block of selection of pre expansions for archival card counter and characters in the text annotations, with respective connections.

Thus, the claimed technical solution meets the criteria of the invention of "novelty."

Analysis of known technical solutions in the studied and related fields allows us to conclude that the introduced functional units are known. However, their introduction into a known device with the indicated connections gives this device new properties. The introduced functional units interact in such a way that they expand the scope and functionality of the processing device, automatically adapt the device to change the subject area of the processed information, and exclude a person from the process of analysis, reading, annotation and cataloging of texts.

Thus, the technical solution meets the criterion of "inventive step", since it does not explicitly follow from the prior art for a specialist.

The invention can be used to solve the problems of symbolic processing of text information and preliminary processing of text data for information retrieval.

Thus, the invention meets the criterion of "industrial applicability".

Figure 1 presents a block diagram of a device for processing information for information retrieval, figure 2 is a block diagram of a functioning algorithm of a quasi-morphological analysis unit, figure 3 is a block diagram of a functioning algorithm of a unit for generating related pairs of word stems, in figure 4 - a block diagram of a functioning algorithm of a block for generating signs of selecting proposals, Fig. 5 is a block diagram of a functioning algorithm for a block for selecting offers, Fig. 6 is a block diagram of a functioning algorithm for a block for generating an inverted index, in fi .7 - a flowchart block allocation operation of the service information in Figure 8 - a flowchart of the functioning of the storage unit and adjusting the stop dictionary.

Figure 1:

1 - memory of the source array;

2 - block character processing text;

3 - typesetting field;

4 - display unit;

5 - input-output unit;

6 - input / output buffer memory;

7 - long-term memory;

8 - block structural analysis of the text;

9 - block allocation of service information;

10 - block storage and adjustment of the stop dictionary;

11 - block quasimorphological analysis;

12 - block forming an inverted index of the basics of words;

13 - control unit;

14 - block synthesis archive cards;

15 - memory of archive cards;

16 is a block forming the associated word stems;

17 - block forming the signs of the selection of proposals for an archive card;

18 - block selection proposals for an archive card;

19 - counter characters in the annotation of the text;

20 - trunk bus

moreover, the input of the input / output buffer memory 6 is the input of the device as a whole, and the display unit 4 is the output of the device as a whole, while the output of the input / output buffer memory 6 via a long-term memory 7, which is connected to the bus 20, is connected to the memory input 1 an initial array, the first output of which is connected through the block 2 of character processing of text to the input of the block 8 of structural analysis of the text, and the second output of which is connected through the block 9 for extracting service information to the first input of the synthesis synthesis block 14 of the archive card, and through the series-connected block 18 of the selection of proposals for the archive card and the counter 19 characters in the annotation of the text connected to the second input block 14 of the synthesis of the archive card, the third input of which is connected in series to the block 10 for storage and adjustment of the stop dictionary, block 11 of the quasi-morphological analysis, block 12 the formation of an inverted index of the basics of words, while the output of block 8 structural analysis of the text through block 11 quasi-morphological analysis and sequentially connected block 16 forming links of the word basics and the block 17 for generating signs for selecting offers for an archive card is connected to the second input of the block 18 for selecting offers for an archive card, while the inputs and outputs of the control unit 13 and the memory 15 of archive cards are connected to the main bus 20, the second output of which is through block 5 the input-output is connected to the second input of the display unit 4, and the memory input 15 of the archive cards is connected to the output of the synthesis card synthesis unit 14,

The operation of the specified device is as follows.

From the output of the device converting the source file to the format. t × t text files are written to the non-volatile memory 7 through the I / O buffer memory 6, and the attributes of these files from the non-volatile memory 7 via the main bus 20, the input-output block 5 are sent to the display unit 4, according to which the operator uses a typeset field 3 selects the file to be processed. The control signals from the dialing field 3 through the display unit 4, the input-output unit 5 and the bus line 20 are sent to the control unit 13, from where a signal is sent to the long-term memory 7 to read the corresponding file into the memory 1 of the original array. From the memory 1 of the source array, the text file is sent to the service information allocation block 9, the sentence selection block 18 for the archive card and, through the character processing block 2, to the text structural analysis block 8

The work of the memory 1 of the original array, block 2 character processing, typing field 3, block 4 display, block 5 I / O, buffer memory 6 I / O, block 8 structural analysis of the text, block 13 control is completely similar to the prototype.

From block 8 of the structural analysis of the text to block 11 of the quasi-morphological analysis, a text file is received in which all service characters are deleted and individual words are highlighted. The quasimorphological analysis unit 11 operates in accordance with the algorithm shown in FIG. 2, according to which words containing less than three characters are deleted from the text file, insignificant words that are in the stop dictionary storage and adjustment unit 10 are deleted, and the remaining words of the text file are formed word stems by cutting off the endings, then the resulting strings containing less than three characters are deleted.

As an example, the conversion of the first sentence of the description of this device can be used.

Initial proposal: “The invention relates to technical means of computer science and computer technology and can be used to solve the problems of symbolic processing of text information and preliminary processing of text data for information retrieval”

Proposal after quasi-morphological processing:

"Invented relates technical means computer science computer technician used solved problems symbol word processing information preliminary word processing data information search"

From the block 11 of the quasi-morphological analysis of the basics of significant words with a length of three or more characters, they arrive at the block 16 for forming the associated word stems and the block 12 for forming the inverted index of the word stems. Block 16 of the formation of related word stems operates in accordance with the algorithm presented in figure 3, according to which dibasic combinations are formed from a sequence of individual stems of significant words. An example of the results of such a conversion is given in Table 1.

Table 1 invented relates relates technical technical means computer science tools computer scientist computer technician technician used used resolved solved problems task symbol processing symbol word processing texts of information pre-information pretreatment word processing data texts this information information search

From the block 16 for the formation of the associated word stems, the dibasic combinations in the form similar to that presented in Table 1 are sent to the block 17 for generating signs of selecting proposals for the archive card. Block 17 of the formation of signs of selection of proposals for an archive card operates in accordance with the algorithm presented in figure 4, which calculates the number of occurrences of dibasic combinations from Table 1, and dibasic combinations are sorted in descending order of the quantities of their appearance in a text file. Then dibasic combinations having the maximum number of occurrences are defined as signs of selection of proposals for the annotation included in the archive card. In this example, this is a two-basic combination of “word processing” that appears twice in the text (Table 2).

table 2 word processing 2 computer technician one this information one task symbol one invented relates one computer scientist one pre-information one information search one used resolved one relates technical one pretreatment one solved problems one processing symbol one computer science tools one data texts one texts of information one technician used one technical means one

Signs of selection from block 17 of the formation of signs of selection of proposals for an archive card, in this case, this is a two-basic combination of “word processing”, transmitted to block 18 of selection of offers for an archive card, to the second input of which from the memory 1 of the source array the source text for processing is received. Block 18 selection of proposals for the archive card operates in accordance with the algorithm presented in figure 5, according to which for the archive card offers are selected containing two-word combinations that are signs of selection. The selected offers from block 18 of the selection of proposals for the archive card through the counter 19 characters in the annotation, which limits the volume of each selected proposal to the first six words in this proposal and limits the total volume of the archive card, in accordance with GOST 7.9-95 SIBID "Abstract and annotation. General requirements ”, 500 characters, are sent to the archive card synthesis block 14, the second input of which receives service information from the service information allocation block 9 and keywords from the block 12 for generating an inverted word base index.

Keywords are formed as follows. From the block 11 of the quasimorphological analysis, the stems of significant words with a length of three or more characters are sent to the block 12 for generating an inverted index of the stems of words, where the indicated stems of words are ordered in decreasing order of frequency of their appearance in the analyzed text, which forms the inverted index of the analyzed text. It is believed that words that are more often found in the text generally reflect its content more fully than words that are rarely found in the text. Six word stems having the maximum frequency of occurrence in the analyzed text from block 12 of the formation of the inverted index of the word strings as “keywords” are received at the third input of the archive card synthesis block 14.

The work of block 14 synthesis archive card is completely similar to the prototype. Block 9 allocation of service information operates in accordance with the algorithm presented in Fig.7. Block 12 forming an inverted index of the basics of words operates in accordance with the algorithm presented in Fig.6.

The archive cards formed in the synthesis card synthesis block 14 are stored in the memory of 15 archive cards. The archived cards accumulated in the memory of 15 archive cards are used in the compilation of thematic indexes of texts, cataloging of texts and in the information search by keywords and attributes.

Thus, the positive effect compared to the prototype is to expand the scope and functionality of the information processing device of various thematic and semantic directions, as well as automatically adapt the device to change the subject area of the processed information based on the complete exclusion of a person from the analysis, reading, annotation and cataloging texts.

An information processing device for information retrieval is implemented based on input signal conditioners 74LCX16 245; reprogrammable logic integrated circuit (FPGA) VirtexII FG676 and 7C1380 RAM module (RAM1); FPGA XC9572XL-VQ6, line of indicators KRS3216, generator 50 MHz; digital signal processing processor (DSP) TMS320C6416, read-only memory 93LC66B and three modules 7C1380 RAM (OZU2-OZU4). The device is installed in the standard PCI bus of a personal computer, from which the working configurations of the FPGA and the DSP processor are loaded. Signals of text messages are received at the input of the device through the formers on the FPGA.

Claims (1)

An information processing device for information retrieval, comprising a sequentially connected typesetting field, a display unit, which is the output of the device as a whole, an input-output unit, a bus line to which the inputs and outputs of the control unit and archive card memory are connected, the second output of which is through the input unit -the output is connected to the second input of the display unit, and the input of the archive card memory is connected to the output of the synthesis card synthesis unit, the memory of the original array, the first output of which is symbolically through the unit text processing is connected to the input of the text structural analysis unit, the input / output buffer memory, the input of which is the input of the device as a whole, characterized in that it has long-term memory, an auxiliary information allocation unit, a stop dictionary storage and adjustment unit, a quasi-morphological analysis unit , block for the formation of an inverted index of the foundations of words, block for the formation of related word stems, block for the formation of signs for selecting proposals for an archive card, block for selecting sentences for an archive card and a character counter in the text annotation, wherein the output of the input / output buffer memory via a long-term memory that is connected to the main bus is connected to the memory input of the original array, the second output of which is connected to the first input of the archive card synthesis unit through the auxiliary information allocation unit, and through a series-connected selection block for archival cards and a character counter in the text annotation is connected to the second input of the synthesis card synthesis block, to the third input of which there are connected properly connected block for storing and adjusting the stop dictionary, a block of quasi morphological analysis, a block for generating an inverted index of word stems, while the output of a block for structural analysis of the text through a block of quasi morphological analysis and sequentially connected a block for forming related word strings and a block for generating signs for selecting sentences for an archive card with the second input of the proposal selection block for the archive card.

Images