EA038241B1 - Method and system for searching for relevant news - Google Patents

Abstract

The invention relates to the field of information technology, and more particularly to search engines designed to identify relevant information from different types of data sources. The technical result is that of enabling the formation of a related set of information from news sources with grouping according to companies that are the subject of news or given types of events. In a first preferred embodiment of the claimed invention, a computer-implemented method for searching for relevant news is proposed in which a set of news is received on a control server from at least one news aggregator server; an analysis of the received set of news is performed on the control server, which includes lemmatizing the texts of each piece of news from said set of news, processing the resulting lemmas of the news texts with the aid of a machine learning model containing a predetermined set of company data and a list of events, wherein for each event in the machine learning model a given set of lemmas is predetermined, identifying news containing lemmas that characterize given events, and forming links between the identified events and at least one company; whereupon, a list of relevant news is generated on the basis of the analysis performed.

Description

Technology area

This technical solution generally relates to the field of information technology, and in particular to search engines designed to identify relevant information from heterogeneous data sources.

State of the art

Currently, data mining is an important component for various business areas, especially in the areas of analytics and forecasting. Often the source of information about information on topics of interest are public resources on the Internet, such as news resources (websites, channels in instant messengers, etc.).

When analyzing data, the main problem is the aggregation of an array of news sources, in particular, linking actual events to companies for subsequent search. As a rule, today there are no effective tools for filtering the collected news content to create aggregated arrays of information with reference to news objects, for example, companies. Various algorithms for collecting data are known in the prior art, for example the solution described in WO 1999005614 (by Louis Gay et al., Published 02/04/1999), which allows data to be aggregated from multiple sources and to track the historical relevance of the collected data. From the patent RU 2382401 (patent holder Microsoft Corporation, published 02/20/2010) an approach is known for analyzing and comparing collections of documents, according to which documents can be presumably organized into groups according to their content or source and analyzed for intergroup and intragroup differences and commonalities. For example, comparing two sets of documents on the same topic but obtained from two different sources, such as an informational overview of an incident in different parts of the world, can reveal interesting differences of opinion and general interpretations of situations. By moving content from static collections to sets of articles generated over time, its evolution can be considered. For example, a stream of news articles by general description can be viewed over time to highlight really informative news stories and filter out the many articles that pretty much convey much the same thing. A common disadvantage of the existing approaches is the lack of a way to identify relevant news in relation to binding to a news object, for example, a company and the corresponding event associated with it, which does not allow efficient collection of relevant information from a variety of data sources.

Disclosure of invention

The technical problem or technical problem to be solved using the stated approach is to ensure the process of searching and forming a set of news with reference to a given set of company names as objects of news and events about which information appears in open data sources. The technical result achieved when solving the above technical problem is to ensure the formation of a related set of information from news sources with a grouping by companies that are news objects and specified types of events.

An additional technical result is an increase in the accuracy of identifying information about companies for a given type of event in publicly available information sources.

The specified technical result is achieved due to the implementation of a computer-implemented method for searching for relevant news, in which a set of news from at least one server of the news aggregator is received on the control server;

the analysis of the received set of news is carried out on the control server, which includes lemmatization of the texts of each news from the said set of news;

processing the obtained lemmas of news texts using a machine learning model, which contains an established set of company data and a list of events, and for each event in the machine learning model a given set of lemmas is set;

defining news containing lemmas identifying the specified events and forming the connection of the identified events with at least one company;

form a list of relevant news based on the analysis performed.

In one particular embodiment of the method, when a set of news is received, duplicate news is filtered.

In another particular embodiment of the method, filtering is performed by calculating the Jacquard measure between news signatures.

In another particular embodiment of the method, the events assigned to the company news are stored in a database.

In another particular embodiment of the method, the news aggregator updates the news list using information channels.

In another particular embodiment of the method, the information channels are Internet websites and / or messenger channels.

- 1 038241

In another particular example of implementation of the method, during the analysis of news, it is determined whether the event belongs to the main or subsidiary company.

In another particular embodiment of the method, the ownership of the company mentioned in the news is determined using a decision tree algorithm.

In another particular embodiment of the method, during the lemmatization of news texts, they are cleared of punctuation marks, stop words and named entities.

In another particular embodiment of the method, a statistical measure is calculated for each lemma of the news text during lemmatization.

In another particular example of the implementation of the method, the machine learning algorithm is a logical regression that classifies the belonging of the news to the event based on the analysis of the statistical measure of the lemmas.

In another particular embodiment of the method, for each news text, the definition of frequency phrases with a length of 2 to 10 lemmas is performed.

In another particular embodiment of the method, the machine learning algorithm is a gradient boosting trained to classify an event based on the number of sentences containing lemmas that identify an event from a search query.

In another particular embodiment of the method, after assigning an event from a company news, the extraction of lemmas and / or sentences containing lemmas identifying the mentioned event is performed.

In another preferred embodiment of the claimed solution, a relevant news search system is provided comprising at least one processor and at least one memory that contains machine-readable instructions that, when executed by at least one processor, perform the above method.

Brief Description of Drawings

The features and advantages of the present technical solution will become apparent from the following detailed description and accompanying drawings.

FIG. 1 illustrates the interaction of the elements included in the claimed solution.

FIG. 2 illustrates the general flow of the method.

FIG. 3 illustrates the processing of text data.

FIG. 4 - an example of a graphical user interface is presented when interacting with a service for the selection of relevant news.

FIG. 5 illustrates a general view of a computing device.

Implementation of the invention

FIG. 1 shows the general computing architecture (100) of the presented solution. The main functionality for collecting and processing information is performed on the control server (110), which, through the data transmission channel, receives information from the server (120) of the news aggregator, which is connected via the Internet (150) with a plurality of news resources (130). The server (110) provides interaction with users (10) to display data on the collected news information, as well as additional functionality, which will be disclosed later in the application materials.

The Internet or Intranet can be used as a data transmission channel between the control server (110) and the server of the news aggregator (120). In this case, the server of the news aggregator (120) can represent several devices that are part of a different network environment, for example, a set of servers, routers, clusters, etc. A data transmission channel can be organized using various types of known data transmission protocols, both wired and wireless, for example TCP / IP, 802.11, Ethernet, FTP, etc., providing the formation of various network interactions, in particular LAN, WAN, PAN, WLAN, etc. The control server (110) performs the main processing of information received from the news aggregator server (120), stores and generates data for display to users (10). Information display can be formed using a specialized graphical user interface. Users (10) can interact with the management server (110) using a web portal or other type of software application that provides access to aggregated news information. Access can be provided, for example, through an API. The interaction of users (10) can be carried out using various electronic devices, which can be, for example, a computer, laptop, smartphone, tablet, game console, smart wearable electronic device, thin client, as well as devices of augmented, mixed or virtual reality, and dr.

The server of the news aggregator (120) is connected via the Internet (150) with various information resources (130) or information channels providing news information. Such resources (130) can be, for example, websites, messenger channels (Telegram ™, WatsApp ™, Viber ™, etc.), social networks (Facebook ™, Vkontakte ™, etc.). The storage of the received information on the server (110) can be carried out in JSON format in a data store, for example, a database. In this case, the source of obtaining news information and the date of its posting on the corresponding resource can be taken into account (130).

- 2 038241

FIG. 2 shows the general process of performing the claimed method of searching for relevant news information (200). Information from news sources, collected and stored on the server of the news aggregator (120), is transmitted (201) to the control server (110). Information from the server of the news aggregator (120) can be transmitted online or offline. In online mode, data from the Internet (150) is transmitted as soon as it appears on a web resource to which the news aggregator server has a connection (120). In offline mode, news is stored on the news aggregator server (120), for example, in a database, and at a set time (for example, every hour, once a day, etc.) or upon request from the control server (110) are transmitted to it.

Data from the news aggregator server (120) can be transmitted in various formats, for example, xml, html, txt, etc. The data format for transmission can also change depending on the mode of information transmission to the control server (110). In addition to the text of the news itself, the data contains information about the companies mentioned in the text.

On the control server (110) there is a generated list containing the names of companies (2021) and events (2022), for which the analysis of incoming news information from the server of the news aggregator (120) is carried out. The specified data is stored in the database of the control server (110). Events can be, for example, the arrest / freezing of company accounts, bankruptcy of the company, the presence of claims against the company, the collapse / growth of shares, etc. The list of events (2022) and companies (2021) can be updated or changed over time. The search for relevant information according to the data received from the server of the news aggregator (120) is carried out by processing (202) the obtained data array using a machine learning model, which is trained to search by company names (2021) and corresponding events (2022) in the text array. information and make judgments about the relevance of the relevant information. Data processing on the server (110) is performed upon receipt of a new data array from the news aggregator server (120) or according to a predetermined scenario. As a script, an automatic script can be configured, which at a set time activates a machine learning model for data processing (202).

When the processing step (202) is performed, the information store of the control server (110) is accessed, which contains data received from the news aggregator server (120) from news sources (130). When accessing the information stored on the control server (110), it is processed (202) to identify relevant data and bind data (203) from news to the corresponding types of events during information processing using a machine learning model.

FIG. 3 shows a process (300) for processing news data received from a news aggregator server (120), which is performed during steps (202) - (203). At the first step (301), the news text data received from the news aggregator server (120) is lemmatized, during which the text corpus of each news is divided into lemmas. From the received data, news text and metadata from files are extracted. During the process of lemmatization of texts (301), the body of the news is divided into words using all punctuation separators, and then reduced to normal form, for example, using the pymorphy2 library. Then the text is converted, in particular, the text is cleared of punctuation marks, stop words (prepositions, conjunctions, pronouns) and nominal entities. In this case, a nominal entity is considered to be any word that begins with a capital letter and is not the first word in a sentence. The process of N-gramming (https://ru.wikipedia.org/wiki/N-gram) can also be performed, in which the most frequent phrases of length from 2 to 10 lemmas are highlighted in the text. The list of the most frequent phrases was obtained by automatic analysis of a large body of text and contains more than 9 million objects.

Also, incoming news goes through a deduplication procedure, during which duplicate news is filtered out. During the deduplication procedure, the signature MinHash is calculated for each news (see https://en.wikipedia.org/wiki/MinHash), after which the signature similarity is calculated for each pair of news in the Jaccard measure (sometimes the Jaccard coefficient). If the similarity of a pair of news exceeds the specified threshold, for example, 0.7, then the shorter news from the pair of text corpuses is considered duplicate and is not subjected to further processing. In the next step (302), after lemmatization of the news texts, the normalized text is processed. The text of the news is searched for the names of companies that do not have homonyms (for example, Sberbank ™). All phrases with a capital letter and in quotes are found, after which the lemmas of the found phrases are searched in the list of companies stored in the server database (110). Found company names are classified according to the main or additional company (that is, which is indirectly mentioned in the text of the news). A company is considered primary if it is the subject of the news, and secondary if the name of the company is simply mentioned in the TV news. The classification is carried out using a machine learning model, in particular a decision-making algorithm, for example, using decision trees. The decision tree features are as follows.

- 3 038241

1) Number of the sentence of the first mention of the company (0, if it is a title).

2) Number of the sentence of the first mention of the company, normalized to the number of sentences.

3) The length of the text in characters; the classification threshold is 0.5.

To determine the relevance of an event for the companies indicated in the TV news, the obtained lemmas from the news body are processed at step (303) using machine learning models.

As one example of a machine learning model, logical regression can be applied by calculating the TF-IDF statistical measure for text lemmas (see https://ru.wikipedia.org/wiki/TFIDF). For each lemma, a statistical measure is calculated in the text, after which a pre-trained logistic regression judgment is made on the obtained features. In addition to processing with a machine learning model, a list of given lemmas is compiled, for example, the list may contain 3040 lemmas that have the highest weight in logistic regression. The list is built for each event after the logistic regression learning process. At the output of the model, the weight of each lemma is determined, according to which the selection of lemmas for the list is carried out based on the values of their weights.

If the obtained value of the probability of judgment of the model (303) is higher than a predetermined threshold and at least one lemma from the mentioned list occurs in the news text, then at least one event is assigned to news (304) for the company name identified in the text.

Additionally, for each event, a set of lemmas can be specified, for example, 10-15 lemmas that most correspond to the event, which are selected from the previously defined list of lemmas, and if the event was assigned to news at step (304), then all the lemmas found in the text from the mentioned set are selected in text. The second example of the application of the machine learning model is a classifying algorithm in the form of gradient boosting, for example LightGBM (https://lightgbm.readthedocs.io). For each news text, the number of sentences containing pairs of lemmas characteristic of the event is counted. Pairs of characteristic lemmas are selected for each event during the training of the classifier. Typical lemmas (and their number) are selected automatically during training.

On the features obtained in this way (pairs of lemmas), a judgment is made using the aforementioned machine learning model (303). If the obtained probability value is higher than a predetermined threshold, then the event is assigned to news (304) for one or more companies specified in the news. Additionally, sentences containing pairs of lemmas characteristic of the event can be highlighted in each text. If, in the course of processing news data, the identification of relevant events for the specified names of companies is not carried out, then such information is not taken into account (305).

FIG. 4 shows an example of a graphical user interface (400) for interacting with a service for the selection of relevant news information. The interface (400) provides functionality for displaying and managing the content of the provided data. The formation of a search query is performed using the information input panel about the name of the company (401). In the main field (404) for displaying current or found information, a list of companies is presented for which processing of identifying relevant information from the server database (110) is carried out.

Companies in the field (404) can be displayed in a different hierarchical order, for example, alphabetically, by the number of news, etc. The information can be filtered by the time range, which is set in the date input field (402).

Also, the interface (400) contains a control panel for setting parameters of search queries (403). Using the control panel (403), you can configure the detection of certain types of events, link companies, configure service parameters, etc. The field (405) displays a list of identified news sources in accordance with the specified events for the companies.

Users (10) can also set the alert function for selected company names. Notifications about the arrival of new news can be sent via e-mail messages, PUSH notifications, SMS notifications, etc. When setting up the notification function, the user (10) can configure the required parameters, for example, the name of the company, the type of events related to companies.

The generated information on processed news can also be displayed using a filter configured with respect to the role of the user (10) interacting with the interface (400). Taking into account the parameters of the user account (10), only those news that contain the type of events associated with his role can be displayed to him.

FIG. 5 shows an example of a general view of the device (500), which provides the implementation of the presented solution. A different range of computing devices can be implemented on the basis of the device (500), for example, a control server (110), a news aggregator server (120), user devices (10), etc. In general, the device (500) contains one or more processors (501) united by a common bus of information exchange, memory means such as RAM (502) and ROM (503), input / output interfaces (504), input / output devices (505 ) and a device for networking (506).

Processor (501) (or multiple processors, multi-core processor, etc.) may be selected from

- 4,038241 assortment of devices currently widely used, for example from manufacturers such as Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™ and the like. Under the processor or one of the processors used in the device (500), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU or Graphcore, the type of which is also suitable for full or partial execution of the method (200), and can also be used for training and applying machine learning models in various information systems.

RAM (502) is a random access memory and is intended for storing machine-readable instructions executed by the processor (501) for performing the necessary operations for logical data processing. RAM (502), as a rule, contains executable instructions of the operating system and corresponding software components (applications, software modules, etc.). In this case, the available memory of the graphics card or graphics processor can act as RAM (502).

ROM (503) is one or more means for permanent storage of data, such as a hard disk drive (HDD), solid state data storage device (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

Various types of I / O interfaces (504) are used to organize the operation of device components (500) and to organize the operation of external connected devices. The choice of appropriate interfaces depends on the specific design of the computing device, which can be, but are not limited to PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini , type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc. To ensure user interaction with the computing system (500), various means (505) of I / O information are used, for example, a keyboard, display (monitor), touch display, touchpad, joystick, mouse manipulator, light pen, stylus, touch panel, trackball, speakers , microphone, augmented reality, optical sensors, tablet, light indicators, projector, camera, biometric identification (retina scanner, fingerprint scanner, voice recognition module), etc.

The networking tool (506) provides data transmission via an internal or external computer network, such as an Intranet, Internet, LAN, and the like. As one or more means (506), an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and / or BLE module, a Wi-Fi module, and etc. Additionally, satellite navigation aids can be used as part of the device (500), for example, GPS, GLONASS, BeiDou, Galileo.

The specific choice of device elements (500) for the implementation of various software and hardware architectural solutions can vary while maintaining the required functionality provided from a particular type of device. The presented materials of the invention disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation, which do not go beyond the scope of the claimed legal protection, which are obvious to specialists in the relevant field of technology.

Claims (16)

CLAIM 1. A computer-implemented method of searching for relevant news, containing the stages, which receive on the control server a set of news from at least one server of the news aggregator; the analysis of the received set of news is carried out on the control server, which includes the lematization of the texts of each news from the said set of news; processing the obtained lemmas of news texts using a machine learning model, which contains an established set of company data and a list of events, and for each event in the machine learning model a given set of lemmas is set; defining news containing lemmas identifying specified events, and forming a connection between the identified events and at least one company; generate a list of relevant news based on the analysis of the news set. 2. The method according to claim 1, characterized in that when a set of news is received, duplicate news is filtered. 3. A method according to claim 2, characterized in that the filtering is performed by calculating the Jaccard measure between news signatures. 4. A method according to claim 1, characterized in that the events assigned to company news are stored in a database. 5. The method according to claim 1, characterized in that the news aggregator updates the news list using information channels. 6. The method according to claim 5, characterized in that the information channels are websites on the Internet and / or messenger channels. 7. The method according to claim 1, characterized in that during the analysis of news, the event belongs to the main or subsidiary company. 8. The method according to claim 7, characterized in that the affiliation of the company mentioned in the news is determined using a decision tree algorithm. 9. The method according to claim 1, characterized in that during the lemmatization of news texts, they are cleared of punctuation marks, stop words and nominal entities. 10. The method according to claim 1, characterized in that during the lemmatization for each lemma of the text of the news, a statistical measure is calculated. 11. The method according to claim 10, characterized in that the machine learning algorithm is a logical regression classifying the belonging of the news to the event based on the analysis of the statistical measure of the lemmas. 12. The method according to claim 1, characterized in that for each news text the definition of frequency phrases with a length of 2 to 10 lemmas is performed. 13. The method of claim 1, wherein the machine learning algorithm is a gradient boosting trained to classify an event based on the number of sentences containing lemmas identifying the event from a search query. 14. The method according to claim 1, characterized in that after assigning an event from the company news, extraction of lemmas and / or sentences containing lemmas identifying said event is performed. 15. A device for searching relevant news, comprising at least one processor and at least one memory containing machine-readable instructions, which, when executed by at least one processor, execute the method according to any one of claims 1-14. 16. System for searching relevant news, containing at least one control server; at least one news aggregator server configured to receive news data from at least one news source, the control server configured to receive news data from at least one news aggregator server; analysis of the obtained data, during which the lemmatization of the texts of each news from the mentioned set of news is performed; processing the obtained lemmas of news texts using a machine learning model, which contains an established set of company data and a list of events, and for each event in the machine learning model a given set of lemmas is set; defining news containing lemmas identifying the specified events, and forming a connection of the identified events with at least one company; formation of a list of relevant news based on the analysis of the set of news.