RU2305314C1 - Method for finding and selecting information in various databases - Google Patents

Abstract

FIELD: technology for finding and identifying documents based on their descriptions, present in various databases and information resources with different document creation standards.

SUBSTANCE: in accordance to the invention, search requests formed by user are dispatched into search system of server, which processes aforementioned requests by selecting documents from various databases, searching system combines all selected documents in single list, sorts aforementioned selected documents based on topics, creates folders, which contains aforementioned documents of one topic, then aforementioned sorted documents are sorted again with consideration of final rating. After that on basis of user request sections of future report are determined, by means of searching system, text signs of beginning and end of sections are determined, text of documents selected for greatest final rating is marked up, inside each section text segments are selected, segments are sorted according to publishing data, final report is prepared, in which text segments, sorted according to publishing date of original document, are combined in single text array, after that final report is dispatched to user terminal through telecommunication communication means.

EFFECT: increased precision of searching and analyzing of received information.

4 dwg, 1 tbl

Description

The invention relates to means for searching and identifying documents according to their descriptions, which are in various databases and information resources with various standards for generating documents, and can also be used when filling a database with fragments of initially unstructured texts.

Known methods for identifying documents by their descriptions, which include converting natural language texts in given areas of knowledge into signals suitable for machine processing, generating a query in the form of a selection of keywords and comparing a selection of query keywords with thesauri of texts stored in a database ( see RF patents No. 2107942, 2167450, US patent No. 6460034, Yandex search database).

A disadvantage of the known methods is the limitedness of one database with a known standard for the formation.

The closest analogue adopted for the prototype is the method of searching and retrieving information from databases described in patent RU 2236699, in accordance with which they carry out: forming by the user at his workplace, which is any personal computer that has access to various databases, at least one search query; transmitting a user-generated request to a search engine; Search engine processing user-generated search queries by selecting documents from databases; moreover, the search system sorts the mentioned selected documents by subject and generates folders, each of which contains the mentioned documents, sorted by one subject; for each sorted document, features characterizing this document are distinguished; inside each folder, the search system determines the rating of each feature contained in each sorted document; after which the search system determines the number of matches of signs of individual sorted documents in one folder with signs of other documents contained in other folders; determines the final rating of each sorted document, taking into account the number of matches of signs and taking into account the weight coefficient of the database; after which the search system again sorts the mentioned sorted documents taking into account the final rating and sends the documents sorted in accordance with the final rating to the user's workplace.

The disadvantage of the prototype is the lack of structural processing and analysis of the received documents by their significance in relation to a given query element. The equivalence of all selected objects and documents leads to an increase in the volume of selected information and an increase in information noise, which ultimately increases the cost of intellectual labor for processing the selected information by the user.

In addition, in the case of working with many document repositories with different standards for document generation, the identification of objects becomes difficult to do.

The technical result of the claimed invention is the expansion of functionality by increasing the accuracy of the search and analysis of the information received.

The technical result is achieved due to the fact that in the method of searching and retrieving information from various databases, including the generation by the user on the user computer of at least one search query, the transmission of the user-generated request through telecommunication means to the server’s search system, processing by the search system user-generated search queries by selecting documents from various databases, in addition, the search system combines all the selected documents copes into a single list, sorts the mentioned selected documents by topics, forms folders, each of which contains the mentioned documents, sorted by one topic, for each folder determines the relevance assessment parameters, for each sorted document of each folder, selects the characteristics that characterize each document, inside of each folder determines the rating of each feature contained in each sorted document, determines the final rating of each selected document, taking into account the rating of each of the document’s characteristic and the folder relevance rating parameter, it sorts the mentioned sorted documents again based on the final rating, stores the documents sorted according to the final rating in the memory of the search engine, selects the number of documents specified by the user in the request with the highest final rating, performs structural processing of the selected number documents for the preparation of the final report, forms a graph showing the dependence of the number of sort documents in accordance with the final rating of documents from the current time, transmits to the user terminal via telecommunication means the final report and the generated schedule.

When analyzing trends and monitoring a particular subject area, based on the analysis of unstructured information (books, articles, reviews, etc.), the user encounters the following main problems:

1. A large amount of information. In traditional subject areas, publications are measured in hundreds of thousands or even millions of publications. The user cannot physically read such a volume of materials. Thus, he has to draw conclusions about the state of the subject area based on the sample. Moreover, no user can be completely sure that there are no significant materials that refute his point of view, which he simply could not find.

2. Relevance of materials. Almost all search engines (Google, Yahoo, Yandex, etc.) give search results in a single list. In these lists, according to formal principles (number of links, word concentration, traffic, etc.), the relevance (value) of the materials is determined. However, in everyday activities, the user never uses these principles when comparing dissimilar objects. The book is not compared with a person, but a company with an event, a patent with news. In some cases, the user is interested in one information, for example an article, in other cases it may be other information - conference materials or a patent.

3. Incompleteness of information. As a rule, the user is dealing with a set of diverse information (texts): articles, books, conference materials, news, etc. The materials selected for work often relate to different periods of time for the development of the subject area, written by different authors representing different organizations, with different interests, goals, objectives, level of development and competence. In addition, these texts differ in their structure, depth, terminology, and data volume. The differences may be fundamental, the points of view may contradict one another even for one author, whose position often changes from work to work depending on time.

The claimed method is aimed at solving the above problems.

The essence of the claimed invention lies in the fact that after a user forms a search query on a user computer and transmits it through telecommunication means to the server’s search system, the search engine processes the search query generated by the user by selecting documents from various databases, the following steps are performed.

Stage 1. The selection of information.

Initially, a user in various search engines, for example Yandex, Yahoo, Google, etc., selects an arbitrary number of documents in the order that the search engine provides him in accordance with its own rules. As noted above, this list of links contains materials of diverse nature: articles, patents, companies, news, etc. Obviously, in real life we do not compare a person and a patent or book with a company. The value of a particular material depends on various subjective factors, the interests of the user in this particular case.

The method proposes:

Combine using the search engine all the materials found in a single list without taking into account the order of their appearance in the lists of various search engines;

Sort documents by subject using a search engine, namely, to collect materials of homogeneous nature into separate folders (groups) (all books in the Books folder, all patents in the Patents folder, information about specialists in the Personalities folder, etc. .d.).

The number of folders can be arbitrary, but each folder should include homogeneous documents that exist as objects in real life.

An example of a set of folders:

Popular materials (Introduction to the topic)

news

News sources

Developments

The organization

Personalities

Portals

Periodicals

Books

Reviews

Define for each folder the parameters for assessing the relevance (importance) of a particular material.

For example, the “Books” folder. Evaluation parameters are: citation index, readers rating in online stores, year of publication, fame of the author, language of the work, etc.

In the “Companies” folder, the parameters can be: the company’s popularity, specialization in a given subject area, capitalization, region, country, etc.

The number of parameters that a user can enter is unlimited. However, in practice, the number of important parameters rarely exceeds the number 7. Empirically it has been established that in most practically significant cases the optimal number of parameters is 4.

Each parameter in each folder is assigned a certain degree of importance, called the weight of the parameter and taking values from 0 to 1.

For each parameter in each folder, integer level values are set depending on its real value.

For example, for the “Books” folder.

Citation Index

more than or equal to 10-5 points,

more than 2, but less than 10-4 points,

1-2-3 points;

0-2 points.

Publication Language Option

"Russian" - 5 points;

"English" - 4 points;

“German” - 3 points;

"Japanese" - 2 points;

"Arabic" -1 point.

Year of publication option

2004-2006 - 5 points;

2000-2003 - 4 points;

1995-1999 - 3 points

1980-1995 - 2 points;

until 1980 - 1 point.

Each user can customize the parameter system and assign weights in accordance with their preferences. The parameter level for each document in the folder directly depends on the actual parameter values for this document.

Determine the rating of the i-th document in the j-th folder according to the formula:

m _j is the number of parameters in the j-th folder;

a _k ^j - weight value of the k-th parameter in the j-th folder;

p _k ^ij is the really determined level value of the k-th parameter for the i-th element in the j-th folder;

with _i ^j - the number of groups in which the i-th document in the j-th folder is mentioned.

Then, in each folder, only the first few documents with the maximum relevance indicators in the folder are selected.

Documents sorted by the search engine in accordance with the final rating are stored in the memory of the search engine.

The initial selection of documents, sorting them into folders (groups), calculating the relevance of each element and selecting the final number of documents with high relevance indicators complete the 1st stage of work.

2 stage. Structural processing.

Documents sorted by the search engine in accordance with the final rating are stored in the memory of the search engine. To implement the 2 stages, you must:

1. Identify sections of the future final report. For example, Goals, Tasks, Forecasts, Current Results and others.

2. Identify keywords specific to each section.

3. Identify the text signs of the beginning and end of the section.

4. Mark the text of the documents selected at the first stage in accordance with the sections. Select text segments, transfer these segments to the database in the corresponding section.

5. Inside each section, sort the segments. Segments are sorted according to the publication date of the original document. Segments in which reciprocal quotes are present are placed sequentially one after another, starting from a segment with an earlier date.

6. Carry out replenishment of the database with new documents (carry out monitoring).

7. Generate a report on the problem.

At any time, the user can generate a summary report of the problem. When generating a report, all segments of the section are combined into a single text array with links to the source.

The above method of selecting information and its structural processing can also be used when filling the database with fragments of initially unstructured texts.

3 stage. Trend analysis.

Most subject areas develop in accordance with the G-curve (Fig. 1.). This curve was first published by the analytic corporation Gartner Group in 1995.

The G-curve shows the dependence of the number of information messages of technology documents on time and the level of technology development. Information messages mean any mention of technology in the media, literature, the Internet and other sources of information. The text of each reference is considered a document.

The curve is conditionally divided into five sections:

1 site - the start of technology (the appearance of a new concept).

On this site there is a rapid increase in the number of documents. This is due to a significant increase in investment in this technology, an increase in advertising, an increase in the number of participants in the development and promotion of technology.

Section 2 - the peak of excessive expectations.

At this site, the growth in the number of publications stops, the volume of advertising decreases, the growth in the number of participants stops.

Section 3 is a disappointment.

The number of published documents is reduced, the volume of advertising is reduced, many participants leave the business. Investments are leaving the subject area. Costs are directed to the search for qualitative changes in technology.

Section 4 - the bias of education.

Found quality solutions to problems. Investments are starting to grow. The volume of documents is increasing. The volume of advertising is growing.

Section 5 - productivity plateau.

A technology with optimal parameters has been developed. The number of documents is steadily increasing. Optimized advertising budgets. The volume of investments is growing.

Starting to analyze a new subject area, the user is faced with the fact that he cannot initially determine which section of the G-curve corresponds to this information material (document) (figure 2).

To solve this problem, it would be good to compare information materials related to different time intervals (figure 3). However, in practice it is difficult to realize this task, since most information materials have a different structure and completeness of description. For example, in some reports the goals, objectives and development prospects are discussed, in others - problems and technical characteristics, in the third they consider the investment attractiveness of the project. It is very difficult to determine the development trend using individual materials in the financial, technical, scientific fields related to different time periods. In addition, in the selected materials, the numerical values of any characteristics of the technology under study may be absent altogether.

The method allows you to build a G-curve, identify a trend in the presence of fragmentary data of a verbal description of the characteristics of the state of technology development.

To solve the problem of constructing a G-curve, it is necessary to do the following operations:

1. In each section created at the 2nd stage of work, an assessment of the relevant characteristics is carried out. As noted above, all collected segments of the original documents within the section are placed sequentially in accordance with the date of publication of the original document. To formalize the verbal characteristics of the technology, pairwise comparison of adjacent segments is carried out.

If the qualitative characteristics of the subject area are amplified, then +1 is entered in the table of processing results; if they are weakened, then -1. After filling out the table according to its data, a G-curve is constructed (figure 4).

2. To simplify the formalization process, a table of rules is used:

No. p.p. Formalization rule Value one. The number of points in the subsequent segment of the document increased +1 2. The text of the documents is repeated. -one 3. Event Dates Appear +1 four. Indicated numerical parameters +1 5. There are words “transferred”, “do not satisfy”, “do not match” -one ... ...... n If A ... then B ...

If the conditions of the Rules from the table have the opposite meaning, then the numerical value of the segment changes to the opposite.

A user computer is any personal computer, for example, of IBM, consisting of a system unit to which a monitor, keyboard, and mouse-type manipulator are connected.

The user computer must have access to databases, which can be both remote and local. Access to databases can be achieved by connecting a user computer via telecommunication means to a search engine of a global Internet server or local area network, for example Intranet.

Databases can be both homogeneous, each of which contains documents on only one subject, for example, a patent database, and heterogeneous, which contain documents on various topics, for example Yandex.

Databases are stored in the memory of a computer or server, for example, on a hard disk.

The search engine is a regular 32-bit machine (for example, Linux, Solaris, Free BSD, Win32).

As a search engine, for example, the Fast search engine is used, which implements the well-known direct search logic. Fast search engine is designed and marketed by the Norwegian company Fast Search & Transfer ASA.

The application of the method also allows to reduce machine search time, increase the relevance of the selection of documents to the query, reduce the cost of intellectual labor in the analysis of the selection of documents.

Claims (1)

A method for searching and retrieving information from various databases, including generating at least one search query by a user on a user computer, transmitting a user-generated request via telecommunication means to a server search system, processing a search system by user-generated search queries by selecting documents from various databases, the search engine combines all the selected documents into a single list, sorts the mentioned selected documents by those to mathematicians, forms folders, each of which contains the mentioned documents, sorted by one topic, for each folder determines the relevance assessment parameters, for each sorted document of each folder, selects the characteristics that characterize each document, inside each folder determines the rating of each characteristic contained in each sorted document, determines the final rating of each selected document, taking into account the rating of each attribute of the document and the parameter for assessing the relevance of the folder, again with sorts the mentioned sorted documents taking into account the final rating, stores the documents sorted in accordance with the final rating in the memory of the search engine, selects the number of documents specified by the user in the request with the highest final rating, characterized in that they perform structural processing of the selected number of documents containing steps which the user in the search query determines sections of the future final report and keywords specific to of each section, using the search engine, determine the text signs of the beginning and end of the sections, carry out the markup of the text of the documents selected with the highest indicators of the final rating, select segments of text within each section, sort the segments according to the publication date, prepare a final report in which the segments text sorted in accordance with the publication date of the original document, combined into a single text array, and then passed to the user term Inal through the telecommunication means of communication the final report.

Images