RU2530672C1 - Checking method of web pages for availability in them of real time multimedia flows, and computer-implemented system for method's implementation - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: checking method of web pages for availability in them of real time multimedia flows involves the following: loading of a web page for time schedule checking, analysis of loaded web pages for availability in them of references to multimedia flows, checking of flow data for determination of their type, whether a multimedia flow is a real time flow or not, periodic checking of references to real time flows contained in database of flows, for detection of variations in a flow type, and/or state of flow, and/or in flow characteristics. New references are added to the time schedule for the web pages on which there have been detected references to real time multimedia flows, and/or checking period is changed for web page references existing in the time schedule, for which changes in a flow have been detected, and/or time of the beginning of the next check is changed.

EFFECT: improving efficient detection of real time multimedia flows and reducing the time required for saving of data and maintenance of its current status.

24 cl, 11 tbl, 7 dwg

Description

FIELD OF THE INVENTION

The present invention relates to computer and network technologies, namely to search engines on the Internet, the purpose of which is to download, analyze, save and index web pages containing targeted content, which is, for example, a real-time multimedia stream or the so-called " live stream or live content.

More specifically, the invention relates to a technology for rechecking web pages previously found by search robots on the Internet, which host real-time multimedia streams. In this case, rechecking is carried out according to a schedule that determines the order (using the coefficient of significance for the page) and the period (frequency) of rechecking. All web pages containing real-time multimedia streams should be double-checked for a certain period in order to store up-to-date data in the search system in order to further provide the user with the ability to search.

The invention can be applied both to search for specific media objects (online radio, streams from webcams, video streams, etc.), and to search for objects in the form of links to external sources of a certain type, signaling the presence of target audio and / or real-time video content, for example, data transfer schemes - rtmp, rtsp, mms, etc.

State of the art

Currently, search engines are widely known and massively used in the world, which provide users with the ability to search the Internet for web pages containing the information they need for the search queries they enter. Popular search engines are, in particular, Yahoo !, Google, Yandex, Rambler.

The general principle of operation of known search engines is based on the collection of information on web pages on the Internet, its processing and indexing to further provide the user with the opportunity to search for the necessary information to the extent that has been processed by the search engine. Each search engine includes search robots, the purpose of which is to scan web pages on the Internet and load them. After the search robot contacts the specified web page address, it scans, for example, http-headers, checking when the page was last modified. If the search robot has already viewed this web page, and the date of the last modification of the page has changed, then it will load it again for processing, if the web page it browses has not been viewed at all, then it will be immediately loaded for processing.

Web pages loaded by the search robot are processed by the corresponding software and hardware components of the search engine. The purpose of this processing is to analyze the page: as a rule, the title is first extracted from the web page, since it carries general information about the web page. Next, all the text is extracted and processed, which is somehow highlighted, for example, in italics, underline or size (in particular, its font size is larger than the font size of the main text), since the search engine assumes that these are key places in the text and they are emphasized .

Some search engines look at the meta tags of web pages, suggesting that they contain keywords or phrases on the page. At the same time, since the content of meta tags is often given inaccurate information, some search engines do not use them to determine the keywords of the page.

Also, the entire text of the web page is fully processed. For example, those search engines that do not use meta tags to determine the keywords of a web page search for keywords by checking for how often a particular word appears in the text, and for this all “stop words” are removed from the text such as <a>, <he>, <you>, <b>, as well as all the characters and numbers, as they create noise when searching for keywords.

Finally, the processed text of the web page is properly indexed by the search engine so as to provide the user with a web interface (for example, a browser) the ability to conveniently search the search engine database based on the input of search queries.

The structure and basic principles of the functioning of search engines are described in more detail, in particular, in "The anatomy of a large-scale hypertextual Web search engine", Brin, S., Page, L., Computer Networks and ISDN Systems, 30 (1-7 ): 107-117, 1998; "Effective Web Crawling", Castillo, C., PhD thesis, University of Chile, 2004; "Crawling the Web." Web Dynamics: Adapting to Change in Content, Size, Topology and Use, ed. by M. Levene, A. Poulovassilis, 153-178, Pant, G., Srinivasan, P., Menczer, F., 2004.

However, search engines constructed in this way cease to meet the requirements of today because of the ever-increasing volume and variety of information presented on the Internet. In particular, when conducting such a text-based search of web pages that contain pictures or videos of interest to the user, an extensive resulting list of web pages is produced in which the proportion of pages that truly meet the requirements and requirements of the user is small, since this list , due to the specifics of the search engines described, those web pages that contain mentions, discussions, advertisements, reviews, etc. related to the required pictures or videos, but not directly from keep themselves required picture or video. As the aggregate volume and variety of web content on the Internet grows, the proportion of relevant web pages during this kind of specific searches will only decrease, and as a result, users will be forced to build clever, most likely repeated searches and spend time sifting large arrays of results search.

This problem determines the relevance of creating specialized (so-called vertical) search engines that are strictly focused on searching on thematic Internet resources, including a system for re-checking found objects according to a special schedule.

In particular, technical solutions are known from the prior art that provide scheduling re-checking of web pages in order to store and index relevant data posted on the Internet (patent for invention US 7886042 B2). This solution relates to search engines capable of loading web pages, their further storage in the database and indexing in order to provide the end user with the ability to search the generated database. Re-checking web pages keeps the current data in the index. Since the amount of data on the Internet is huge, the task of re-checking requires finding more optimal solutions in order to save resources and identify landing pages for double-checking.

Closest to the claimed technical solution is a method and system for re-checking web documents according to the schedule presented in the patent for invention US 8386459 B1 - "Scheduling a recrawl". The invention provides a method for constructing a re-check schedule for web documents based on information about the document being checked. According to the present invention, for constructing a web document rechecking schedule, the frequency of change of the web document itself is determined based on the history of its previous check, which allows one to determine the time interval within which the web document has changed, and based on this information, calculate the optimal time interval for double-checking it.

However, the known solution is based on double-checking all possible web documents on the Internet. This decision does not take into account the possibility of excluding non-target data from verification. Also, such a solution requires storing the history of the previous verification of web documents, which is a high-cost solution, given the amount of data available on the Internet. Moreover, in the known technical solution, there is no possibility of making a decision on rechecking dynamically, since the construction of a rechecking schedule is based on the history of the previous page check.

SUMMARY OF THE INVENTION

The objective of the present invention is to provide a method and system focused on identifying web pages with link (s) to the real-time multimedia stream based on the results of checking an array of web pages according to a specific schedule.

The technical result of the invention is to increase the efficiency (including performance) of detecting multimedia streams in real time, the links to which are contained in the checked web pages, as well as reducing the number of resource-intensive operations (optimization of the used computing and network resources) used in checking the web pages that do not contain such content, as well as reducing the time to save and maintain data up to date, while increasing the reliability of the results.

Thus, the inventive method can be implemented with significantly lower costs, including temporary (in comparison with well-known counterparts) required to search for web pages with target content while increasing the relevance of the detected web pages to the presence of the target content.

When using the proposed method of double-checking web pages, the results produced by the search program, practically do not contain information noise. Thus, the results obtained correspond to the search criteria of the real-time multimedia content set by the user and contain only reliable content, so the user spends less time filtering the search results.

The problem is solved in that the method of checking web pages for the presence of multimedia streams in real time, according to the invention, is as follows:

- download web pages for checking according to a schedule that determines the period and procedure for checking links to web pages,

- analyze the loaded web pages for the presence of links to multimedia streams in them, followed by saving the found links to multimedia streams in the stream database, while information about the membership of the multimedia stream to the web page is stored in the relationship database (or, in other words, the relationship of links to web pages and their associated links to multimedia streams),

- when identifying links to multimedia streams that are not in the stream database, add to the stream database, after which they check the data streams to determine their type, whether the multimedia stream is a real-time stream or not, and also periodically check links to real-time streams, flows in the database, to detect changes in the type of stream, and / or the state of the stream, and / or in the characteristics of the stream, followed by saving information about changes in the database of streams, while in the database of relationships do etc. about the fact of changes,

- at the same time, new links to web pages are added to the schedule, on which links to real-time multimedia streams are identified, and / or the check period is changed for existing links to web pages in the schedule, for which changes to the stream are detected, and / or are changed the start time of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to the current one, if changes in the type and / or status of the stream are detected due to an increase in the number of links on the web page ok for real-time multimedia streams, the verification period is reduced, and when the number of links to real-time multimedia streams on a web page decreases, the verification period is increased, and in the case when the number of links to real-time multimedia streams becomes zero, this link to the web the page is excluded from the scan schedule.

In the claimed invention, a web page is a file directly containing the text of the web page and / or a script file associated with this web page.

Downloading a web page by reference can be done by emulating the operation of a web browser by building a model of a web document and creating all objects that potentially contain links to multimedia streams.

The period of checking links to streams from the database of streams having the status of a multimedia stream of real time to detect changes in the type of stream, and / or the state of the stream, and / or in the characteristics of the stream, can be selected from the interval of values of 2-5 minutes.

The state of the stream is determined based on whether it is on or off.

The relationship database has a structure showing that a link to a web page belongs to one or more links to multimedia streams, while in the relationship database, streams that are of the type of real-time multimedia streams are noted.

As characteristics of the stream, a description of the multimedia stream and technical characteristics of the stream can be used. As a description of a multimedia stream, a textual description of the stream, the title of the stream, an indication of the owner of the stream, a link to the site of the stream, or any other data transmitted within the stream and displaying its essence are used. As the technical characteristics of the stream, bitrate, format, information about audio or video codecs or any other technical characteristics of the stream are used.

Based on the analysis of the loaded web pages, the availability of web pages is additionally determined, and if inaccessible web pages are detected, an appropriate check mark is made in the check schedule. If an unavailable web page is in this state for a week, the link to this web page is excluded from the scan schedule.

In one embodiment, for each link to a web page, a significance coefficient (K) can be determined, which can be chosen equal to the number of links to real-time multimedia streams on this page, and the verification period can be calculated based on the relation T = T _const / K, where T _const is the specified verification period, for example 24 hours, and in case T <T _min , then T = T _min , where T _min is the minimum allowable verification period, for example 1 hour, followed by saving the results for (K) and (T) in the check schedule. In this case, the loading of web pages and their analysis is carried out starting with a link to a web page from the schedule that has the highest coefficient (K).

The problem is also solved by the fact that a computer-implemented system for checking web pages for the presence of multimedia streams in real time includes:

Schedule database, including a list of links to web pages with a period for each link and the procedure for checking it,

Stream database, including a list of links to multimedia streams, as well as information about the type, status and characteristics of streams,

Relationship database storing information about the affiliation of the multimedia stream to the corresponding web page, as well as the type of multimedia stream,

a data loading module configured to download web pages via a link from the schedule database and analyze downloaded web pages for links to multimedia streams in them,

a data management module, configured to save the links found to the multimedia streams found by the download module to the streams database, and also to save and / or change information about the relationship between the multimedia stream and the web page in the relationships database,

a module for checking streams from the database of streams, configured to determine the type of multimedia stream, as well as periodically check references to multimedia streams in real time to detect changes in the type of stream and / or in the state and / or characteristics of the stream, followed by storing the received information in the stream database

a flow control module configured to detect changes made to the flow database, followed by recording information about the presence of changes in the relationship database,

a schedule management module, configured to change the schedule in the schedule database by adding new links to web pages into it, on which links to real-time multimedia streams are identified, and / or by changing the check period for existing links to web pages in the schedule, for which there are changes in the stream, and / or changes in the start time of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to while maintaining its verification period; when detecting changes in the type and / or status of the stream associated with an increase in the number of links to real-time multimedia streams on a web page, the check period is reduced, and when the number of links to real-time multimedia streams on a web page decreases, the check period is increased, and when the number of links to real-time multimedia streams becomes equal to zero, this link to a web page is excluded from the scan schedule.

The data loading module is also configured to emulate the operation of a web browser by building a model of a web document and creating all objects potentially containing links to multimedia streams. In addition, the data loading module is configured to determine the availability of a web page, and in case of unavailable web pages, information about this is recorded in the schedule database. If an unavailable web page is in this state for a week, the schedule management module is configured to exclude links to this web page from the schedule database.

The module for checking streams is also configured to change the period for checking links to streams from the database of streams having the status of a multimedia stream of real time from 2 to 5 minutes. The module for checking flows when checking the status of a real-time stream determines whether a given stream is on or off.

In the stream database, as a stream characteristic, a description of the multimedia stream and technical characteristics of the stream are present; as a description of a multimedia stream, there is a text description of the stream, the heading of the stream, an indication of the owner of the stream, a link to the site of the stream or any other data transmitted within the stream and displaying its essence; as the technical characteristics of the stream there is bitrate, format, information about audio or video codecs or any other technical characteristics of the stream.

The schedule management module is configured to determine a significance coefficient (K) for each link to a web page, which is chosen equal to the number of links on this page to real-time multimedia streams, and to calculate the verification period based on the relation T = T _const / K, where T _const is the specified verification period, for example 24 hours, and in the case where T <T _min , then T = T _min , where T _min is the minimum allowable verification period, for example 1 hour, followed by saving the results for (K) and (T) in the schedule database. At the same time, the data loading module is configured to download web pages and analyze them starting from a link to a web page from the schedule database that has the highest coefficient K.

A distinctive feature of the claimed technical solution is that the criterion for rechecking the web page (or web document) is not the web page itself, but the connection of this web page with a link to the real-time multimedia stream (or information about the multimedia stream belonging to the web -page). The criterion for checking a web page is the presence in this page of a link to a multimedia server that broadcasts the multimedia stream in real time. Moreover, the claimed solution allows you to dynamically exclude or include from the scan web pages that link to multimedia streams in case of data changes in the type, state or characteristics of the stream, for example, if the real-time multimedia stream is turned off, then all web pages that link to given stream are excluded from re-checking. This decision is based not on the dynamics of changes in these web pages, but on an independent resource, such as a multimedia server, changes in which lead to a double-check of the web page that refers to this multimedia resource.

Description of drawings

The invention is illustrated by drawings, where figure 1 presents a block diagram of the inventive system for checking web pages for the presence of a multimedia stream of real time; figure 2 presents a flowchart showing a sequence of operations for calculating the parameters of checking web pages; figure 3 presents a display of the link links to a web page with links to multimedia streams; figure 4 presents an example of linking links to web pages and links to streams; figure 5 is an example of data communication after re-verification; figure 6 presents the algorithm for storing threads in the database of threads; figure 7 presents the display of changes in the structure of relations;

The positions in figure 1 indicate: 1 - a schematic representation of the movement of data between the modules of the system; 2 - data loading module, receiving links to web pages from the database of schedule 9; 3 - a data management module that creates and modifies associative links between links to web pages and links to multimedia streams; 4 - stream database, which contains links to multimedia streams and all information about them; 5 - stream verification module, which determines the type, status and characteristics of the multimedia stream; 6 is a database of relationships, which stores relevant associative links between links to web pages and links to multimedia streams; 7 is a flow control module that detects changes in the type, condition, or characteristics of flows in a stream database 4, followed by a record of changes in the relationship database 6; 8 - schedule management module, which makes changes to the schedule database 9 by adding new records to it or by modifying existing records; 9 is a schedule database that contains a list of links to web pages with a period for checking it and a verification procedure for each link.

DETAILED DESCRIPTION OF THE INVENTION

The claimed invention allows to optimize the scan schedule of web pages that contain links to real-time multimedia streams by calculating the optimal scan period. Changing the validation period of a web page is based on the change:

- type of multimedia stream - is the link to the multimedia stream a real-time stream; and / or

- state of the multimedia stream - the stream is on or off; and / or

- Characteristics or descriptions of the real-time multimedia stream.

The basis for calculating the period of checking the web page are multimedia streams of real time, located on the web page.

Search bots find web pages on the Internet that host multimedia streams. All web pages containing real-time multimedia streams should be re-checked for a certain period in order to store up-to-date data related to the checked web page in the search system in order to further provide the user with search capabilities. To double-check the web pages, the schedule is used, which is located in the database of schedule 9, in which the coefficient of significance for the checked link to the web page is set, as well as the period of verification and the time of the end of the last check and the start of the next check. All links to web pages containing multimedia streams are stored in the system in the schedule database 9. The streams located on the page are checked to determine their membership in real-time multimedia streams. The purpose of re-checking web pages is to search the web pages for new links to multimedia streams and confirm the presence on the web page of links to multimedia streams found during the previous check of the web page, as well as updating the availability of the page and the information contained on it that displays it the essence.

Data loading module 2 receives a list of links to web pages, which must be checked in accordance with the schedule from schedule database 9. The loaded pages of data loading module 2 are analyzed, which searches for multimedia links to streams. In addition to links from the page, information related to the multimedia stream is extracted in the form of a text description, which is further used as part of the multimedia stream description. The received information of the downloaded and analyzed web page and the links to the multimedia streams found in it is transmitted to the data management module 3. The data management module stores the found links to the multimedia streams in the stream database 4. For all the transferred links to the multimedia streams, the data management module receives information from database streams 4 about the current type of multimedia stream to mark in the database of relationships 6. After receiving information for the found links to multimedia streams m modulus data management 3 begins to check and make changes to the database linkages 6:

- checks the previous associative link of this link to a web page and links to multimedia streams in the database of relationships 6;

- makes changes in the associative relations between the link to the web page and links to multimedia streams based on the new associative links between the link to the web page and the links to multimedia streams found on the web page in the relationship database 6;

- saves the changes or, in the absence of changes, confirms the previous associative relationship in the relationship database 6.

All multimedia streams located in the stream database 4 are checked by the stream verification module 5. All new streams are checked to identify real-time multimedia streams, as well as all multimedia streams that are defined as real-time streams and are in working (on) state or were on, but currently turned off. A regular check of multimedia streams in real-time status is carried out in order to store relevant information about streams, in the framework of which it is determined:

- type of multimedia stream, in order to confirm that the link to the stream belongs to multimedia streams in real time;

- the status of the multimedia stream (for example, the server transmitting the multimedia stream is on or off);

- technical specifications and description of the multimedia stream, as well as track their changes.

The flow control module 7 receives from the stream database 4 a list of real-time multimedia streams that have changes in type, and / or status, and / or technical characteristics, and / or description. Next, the flow control module 7 makes a note of the changes in the relationship database 6 for all links to web pages that have associative links with the resulting list of multimedia streams. It should be noted that with one link to the multimedia stream can be associated with many links to web pages. Schedule Management Module 8 retrieves a list of links to web pages from the relationship database 6, for which there is a mark on changes in real-time multimedia streams. For the received links to web pages, the degree of changes in the stream is determined, the significance coefficient is calculated, and the verification period is calculated. Based on the received data, the calculation of the start time of the next check is performed. Schedule management module 8 saves the results to the database of schedule 9 for the received list of links to web pages, changing the start time of the next check for links to web pages, and also adds or excludes links to web pages from re-checking. Links available for download on schedule 9 are sent for download and analysis to data loading module 2.

Below is a more detailed description of the structure of the module for checking threads 6 and the algorithm of its operation.

The algorithm for checking links to streams from the database of streams for the presence of multimedia streams in them in real time includes the following steps:

- connection to the media server by reference,

- receiving (downloading) from the media server information about the media stream, including the characteristics of the stream in a given format and / or part of the stream intended for playback on the client side,

- analysis of the information received about the media stream, consisting in the search for signs indicating that the analyzed stream is a multimedia source, the broadcast of which is carried out in real time,

however, any sequence of characters and / or bytes in the media stream is used as signs, on the basis of which it is concluded that the media stream meets the criterion of a real-time multimedia stream (“live” stream).

As information about the media stream, protocol headers can additionally be used.

When a sequence of bytes is received from the server, their analysis is carried out in continuous mode until the data intended for direct reproduction is received, and upon receipt of a message with information that the stream is recorded, it is concluded that the stream being tested is not a multimedia source, the broadcast of which carried out in real time.

As signs can be used:

- a parameter characterizing the duration of the stream (Duration), and / or

- a parameter characterizing the position from which playback starts in the stream (Start Time), and / or

- parameter of the ability to rewind within the transmitted stream (Seekable).

If the value of the parameter characterizing the duration of the stream (Duration) is negative, or zero, or more than the specified limit, the value of the parameter of the possibility of rewinding within the transmitted stream (Seekable) is analyzed, if it indicates the prohibition of rewinding in the stream , conclude that the analyzed stream is a multimedia source, the broadcast of which is carried out in real time.

If the value of the parameter characterizing the duration of the stream (Duration) is in the range from zero to the specified limit, reconnect to the media server and determine the values of this parameter and the parameter characterizing the position at which playback starts (Start Time), which compare with the values of similar parameters obtained during the initial connection, and in case of mismatch of at least one of the parameter values, it is concluded that the analyzed stream is the source ultimedia, broadcast which is carried out in real time; if the values of the parameters coincide, they search for signs of the multimedia stream in the server response headers, upon detection of which they conclude that the stream being tested is a multimedia source broadcast in real time.

The established limit of the values of the parameter characterizing the duration of the flow is selected experimentally and can be in the range of values from 5 to 9 hours.

If the values of the stream duration and / or playback position parameters have not been received from the server, it is concluded that the stream being tested is a multimedia source broadcast in real time.

The thread verification module for implementing the above algorithm contains:

- a multimedia client, configured to connect to the media server by reference and download information about the media stream, including the characteristics of the stream in a given format, and / or a specific part of the stream intended for playback on the client side, and / or information about the protocol headers received from the server ,

- a unit for analyzing information about the media stream, which is configured to check the received information about the media stream, which consists in searching for signs indicating that the analyzed stream is a multimedia source broadcast in real time, where any sequence of characters is used as signs or bytes in the media stream, based on which they conclude that the media stream meets the criterion of "live" stream.

As a multimedia client, such applications as MPlayer or VLC media player can be used, as well as any other product, including a self-developed multimedia client made with the ability to communicate, process and provide the necessary information.

Thus, the technology for determining the type of stream, whether it is a real-time stream or a static file of a fixed duration, consists in analyzing meta-information obtained from the media stream itself. The media client connects to the media server, after which it receives meta-information about the stream in the specified format, as well as a certain part of the stream intended for playback on the client side. The received meta-information, as well as the transmitted media stream buffer, go through the verification stage in order to determine the type of stream. The main purpose of the check is to analyze the data and search for signs that indicate that the analyzed stream is a multimedia source, the broadcast of which is carried out in real time. At the same time, a characteristic feature of a “live” stream (content) is the inability to perform “fast forward” with respect to it using the means of a client playback application.

Typical examples of “live” AV content on the Internet are television (TV) and broadcasting of on-air studios, special Internet broadcasting by professional and amateur studios, and images from a webcam for streaming broadcasting.

The principle of building a schedule 9 is presented in more detail below.

To link to a web page, Schedule Management Module 8 determines the changes in the stream. If there are changes in the characteristics of the stream, which may include data such as a description of the stream, the title of the stream, an indication of the owner of the stream, a link to the site of the stream or any other data transmitted within the stream and showing its essence, as well as changes in technical stream characteristics, such as bit rate, format, information about audio or video codecs or any other technical characteristics of the stream, then for the checked link to the stream, the start time of the next check is set equal to the current time. Setting the start time of the next link check to a web page equal to the current time causes the link to be checked immediately. If a link or a state that can take two values changes for a link to a real-time multimedia stream, the stream is turned on or the stream is turned off, then Schedule Control Module 8 changes the significance factor, period and time of the next check of the link to web pages. If the multimedia stream is in the on state, then the significance coefficient of the link to the web page is increased by one; if the stream was turned on and now it is off, the significance coefficient is reduced by one. For example, for one link to a web page there can be more than one “working” stream, if there are three, then the significance coefficient will be equal to three, respectively. In a particular case, the significance coefficient is fully consistent with the number of links to real-time multimedia streams located on a web page. The conditions for determining the rules for selecting a significance factor for checking links on a web page may not depend on the number of links to real-time multimedia streams located at the specified address of the web page, and, as a result, can be determined by other conditions for choosing the significance coefficient. If the checked link to the web page has, for example, two links to real-time multimedia streams, and both links to the streams have state changes, for example, the streams have stopped working (are off), in this case, the link check value is per web page will be zero, which will exclude the page from scanning. Based on the significance coefficient, the period for checking the link to the web page is calculated, and the sequence of loading the web pages in the loading module 2, for which the time has come to check, is established. The rule for calculating the validation period may vary depending on the events that are the reason for the re-checking of the web page. In a particular case, for each link to a web page, a significance coefficient (K) is determined, which is chosen equal to the number of links to real-time multimedia streams on this web page, and the verification period is calculated based on the ratio T = Tconst / K, where Tconst is a specified verification period, for example 24 hours, and if T <Tmin, then T = Tmin, where Tmin is the minimum validation period, for example 1 hour. All changes are recorded in the database of schedule 9, from which later the data goes to the data loading module 2.

The proposed system can be implemented on one or more server computers, combined to jointly implement the prescribed functionality, while the above modules can be implemented in software and hardware components of these server computers, known to specialists and widely used in technology. In particular, the aforementioned databases can be implemented on one or more widely known computer-readable media, for example, hard disk drives, RAID arrays, solid state memory, and the like. The data download module can be connected and can interact with the Internet on the basis of widely known wired and / or wireless network technologies and equipment, in particular, on the basis of the protocol stack http / tcp / ip. To interact with database modules, the operator can use any known terminal equipment that supports the ability to execute commands of the database interaction language (for example, SQL). Such equipment may be, for example, a suitably configured personal / laptop / handheld computer.

The following are specific examples of the schedule for rechecking links to web pages with real-time multimedia streams placed on them.

Example No. 1:

The first example shows the appearance of new links on a web page in the schedule database 9.

The search engine found a new web page at Reference_1, after analyzing which it was revealed that the page has two links to the multimedia streams Stream_1 and Stream_2. Data management module 3 is transmitted information about the web page, which contains various meta-information about the page itself and the detected links to multimedia streams (see figure 3).

The data management module sends links to the stream to the database of streams 4, where along with it it requests status for the transferred streams. If this link to the multimedia stream has already been transferred to the database of streams 4, then the data management module will receive information about it, if the link to the stream is new, then the stream information will remain unknown until it is verified by the streaming check module 5. Next data management module 3 checks the information about the reference Reference_1 in the relationship database 6 to determine the associative relationship of this link to a web page with links to multimedia streams obtained from the previous check of this page. If this link to the web page is not in the database of relationships (that is, new), then it will be added to it, where the link of this link to the web page and links to multimedia streams will be indicated (see figure 3). Information about this link to a web page will remain unchanged in the database of relationships until links to the multimedia streams that were found on it are checked. Changes to the relationship database 6 and further work with this link to the web page will be carried out only if after checking the links to the streams it will be determined that at least one of them refers to links to real-time multimedia streams. The module for checking streams 5 takes references to the verification from the database of streams 4, after which it determines that one of the links refers to multimedia streams in real time (see Table 1).

Table 1 Stream link Stream Information Stream_1 RTS (Live Stream Link) Stream_2 File (Link to a static file)

The thread control module 7 requests information from the thread database 4 about real-time streams that have changes in type, state or characteristics. After the request, the module 7 will receive a link to the stream Stream_1, where it will be indicated that the stream has switched to the type of streams that belong to live streams (see table 2).

table 2 Stream link Change Stream_1 RTS (Live Stream Link)

The flow control module 7 makes a mark in the relationship database 6 for all links to web pages that have a link to this link to a multimedia stream that this stream has switched to the status of a "live" stream. Next, the schedule management module 8 takes from the relationship database 6 all links to web pages that have changes in the type, state or characteristics of links to real-time streams. Schedule management module 8 will receive the Reference_1 link, for which the number of real-time streams and a mark on changes in the streams will be indicated (see table 3).

Table 3 Web link Stream link Change Reference_1 Stream_1 RTS (live stream link)

Since for this link to a web page there is only one link to a real-time stream, it will be set to a significance factor equal to one, as well as a page verification period, for example, equal to 24 hours, after which the data will be stored in the schedule database 9 (see table 4). Since Reference_1 did not yet participate in the rechecking schedule, a separate record will be created for it, which will indicate the time after which it will be rechecked. The next check time for a new link to a web page is defined as the current time plus period.

Table 4 Web link Significance coefficient Period Last check time Next check start time Reference_1 one 24 hours 0000-00-00 00:00:00 Current time +24 hours

Example No. 2

Consider the situation of re-checking web pages that are already on the schedule. Data loading module 2 receives from the database of schedule 9 (see table 5) three links to web pages for which the time has come to start the scan.

Table 5 Web link Significance coefficient Period Last check time Next check start time Reference_5 2 12 hours 2013-07-05 15:34:27 2013-07-06 3:34:27 Reference_6 one 24 hours 2013-07-05 15:34:11 2013-07-06 15:34:11 Reference_7 one 24 hours 2013-07-05 15:34:18 2013-07-06 15:34:11

In the relationship database 6, until the moment of verification, links to web pages have the following relationships with multimedia links to real-time streams (see Fig. 4).

The reference_5 web page contains two links to real-time streams, while Reference_6 has a common stream with it, and the reference_7 web page contains an independent stream link that has no intersections with other web pages. Data Download Module 2 downloads web pages from the specified links. After analyzing the content, links to streams that have the connection shown in FIG. 5 were found in the downloaded documents.

The found data is transmitted to the data management module 3. Next, the data management module sends the found links to the streams to the database of streams 4 in order to save new links to the streams and obtain information for already known streams (see Fig. 6).

The received information about streams from the stream database 4 (see table 6) indicates that the links to the stream Stream_10 and Stream_50 are checked and are real-time streams, the Stream_11 link pointed to a real-time stream, is off, and Stream_51 is new reference to the multimedia stream and requires verification by the streaming verification module 5.

Table 6 Stream link Stream Information Stream_10 RTS (Live Stream Link) Stream_11 RTS Off Air (Link has stopped working) Stream_50 RTS (Live Stream Link) Stream_51 New (New link to the multimedia stream)

Next, the data management module starts checking the previous association for the data of links to web pages in the relationship database 6. Based on the check, it is determined that some web pages have changed links to multimedia streams, which leads to changes in the associations in the relationship database 6 for these web page links (see FIG. 7).

The data management module notes that the web page at Reference_7 no longer contains a link to the Stream_12 stream and assigns new links to it, indicating that it contains Stream_50 and Stream_51 streams. Along with making changes in associative relations, information about the streams is received, which was obtained from the stream database 4. It is noted that the state of the link to the stream Stream_11 has been changed, where it is indicated that the stream is turned off, and the links to streams Stream_10 and Stream_50 are in working condition and are multimedia streams of real time. Since Stream_51 is a new link to a stream, there is no information for it that can affect changes in the schedule 9. Schedule management module 8 requests links to web pages that have changes in real-time streams from the relationship database 6 (see table 7).

Table 7 Web link Stream link Change Reference_5 Stream_10 RTS (Live Stream Link) Reference_5 Stream_11 RTS Off Air (Link has stopped working) Reference_6 Stream_11 RTS Off Air (Link has stopped working) Reference_7 Stream_50 RTS (Live Stream Link)

For the Reference_5 link, a significance coefficient equal to one will be calculated, since one of the two available threads has stopped working, for the Reference_6 link, the significance coefficient will be zero, because it does not currently have real-time streams in the on state, for the Reference_7 link coefficient significance will remain equal to one. After calculating the significance coefficient and calculating the validation period for current links to web pages, the schedule management module makes changes to the schedule 9 database (see table 8).

Table 8 Web link Significance coefficient Period Last check time Next check start time Reference_5 one 24 hours 2013-07-06 3:34:27 2013-07-07 3:34:27 Reference_6 0 0 hours 2013-07-0615: 34: 11 0000-00-00 00:00:00 Reference_7 one 24 hours 2013-07-0615: 34: 11 2013-07-06 15:34:11

The changes concern only two records, for Reference_5 the significance coefficient decreases and, as a result, the period increases. Reference_6 is excluded from the scan because it does not currently have real-time streams working. There are currently no changes for Reference_7, since before the scheduled check this link to a web page had an associative connection with Stream_12, after checking it began to point to two streams, but the type of stream was known only for one link to the stream, the second link to the stream was new, so the significance coefficient for this link currently remains equal to 1. Data loading module 2 revealed a new link to Stream_51, for which the type was not determined, after it was checked by the thread verification module 5 о Roedel that the link points to a live stream. The flow control module 7 requests data from the database of streams 4 and receives information (see table 9) that Stream_51 refers to multimedia links to real-time streams.

Talitsa 9 Stream link Change Stream_51 RTS (Live Stream Link)

Then, the flow control module makes changes to the relationship database 6, where for all links to web pages that have a connection with the stream Stream_51, a mark is made about the change in the type of stream. Schedule management module 8 again requests information about links to web pages that have changes in the type, state, or characteristics of real-time streams from the relationship database 6 and receives data (see table 10) about the changes for Reference_7.

Table 10 Web link Stream link Change Reference_7 Stream_50 RTS (Live Stream Link) Reference_7 Stream_51 RTS (Live Stream Link)

For reference_7, the significance coefficient, the verification period and the time of the next verification start are calculated, after which changes are made to the schedule 9 database (see table 11).

Table 11 Web link Significance coefficient Period Last check time Next check start time Reference_5 one 24 hours 2013-07-06 3:34:27 2013-07-07 3:34:27 Reference_6 0 0 hours 2013-07-06 15:34:11 0000-00-00 00:00:00 Reference_7 2 12 hours 2013-07-06 15:34:11 2013-07-07 3:34:11

Since Reference_7 now has 2 links to real-time streams, it accordingly increases the significance coefficient and the recheck period changes, which leads to a change for the next time that the link to the web page is checked.

Claims (24)

1. A method of checking web pages for the presence of multimedia streams in real time, according to which
load web pages for checking according to a schedule that includes links to web pages and determines the period and procedure for checking links to web pages,
analyze downloaded web pages for the presence of links to multimedia streams in them, followed by saving the found links to multimedia streams in the stream database, while information about the membership of the multimedia stream to the web page is stored in the relationship database,
then check the data streams to determine their type, whether the multimedia stream is a real-time stream or not, and also periodically check links to real-time streams in the stream database to detect changes in the type of stream and / or stream status, and / or in the characteristics of the stream, with subsequent storage of information about changes in the database of flows, while in the database of relationships make a note about the fact of changes,
at the same time, new links to web pages on which links to real-time multimedia streams are identified are added to the schedule, and / or the check period for existing links to web pages for which changes in the stream are detected are changed and / or time is changed the beginning of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to the current one, if changes in the type and / or status of the stream are detected due to an increase in the number of links on the web page to real-time multimedia streams, the verification period is reduced, and when the number of links to real-time multimedia streams on a web page decreases, the verification period is increased, and in the case when the number of links to real-time multimedia streams becomes zero, this link to the web the page is excluded from the scan schedule. 2. The method according to claim 1, characterized in that the web page is a file directly containing the text of the web page, and / or a script file associated with this web page. 3. The method according to claim 1, characterized in that the download of the web page by reference is carried out by emulating the operation of the web browser by building a model of the web document and creating all objects that potentially contain links to multimedia streams. 4. The method according to claim 1, characterized in that the period of checking links to streams from the database of streams having the status of a multimedia stream of real time, to detect changes in the type of stream, and / or the state of the stream, and / or in the characteristics of the stream, is selected from an interval of values of 2-5 minutes. 5. The method according to claim 1, characterized in that the state of the stream is determined based on whether it is in the on state or off. 6. The method according to claim 1, characterized in that the relationship database has a structure that displays the link of a web page to one or more links to multimedia streams, while in the relationship database mark streams that are of the type of multimedia streams in real time. 7. The method according to claim 1, characterized in that the characteristics of the stream use the description of the multimedia stream and the technical characteristics of the stream. 8. The method according to claim 1, characterized in that as a description of the multimedia stream using a text description of the stream, the title of the stream, an indication of the owner of the stream, a link to the site of the stream or any other data transmitted within the stream and displaying its essence. 9. The method according to claim 1, characterized in that as the technical characteristics of the stream using bitrate, format, information about audio or video codecs or any other technical characteristics of the stream. 10. The method according to claim 1, characterized in that according to the results of the analysis of the loaded web pages, the availability of the web pages is determined, and in the case of unavailable web pages, an appropriate check mark is made in the check schedule. 11. The method according to claim 10, characterized in that if the unavailable web page is in this state for a week, the link to this web page is excluded from the check schedule. 12. The method according to claim 1, characterized in that for each link to a web page, a significance coefficient (K) is determined, which is chosen equal to the number of links on this page to real-time multimedia streams, and the verification period is calculated based on the relation T = T _const / K, where T _const is the specified test period, for example 24 hours, and in the case T <T _min , then T = T _min , where T _min is the minimum allowable test period, for example 1 hour, followed by saving the received results for (K) and (T) in the check schedule. 13. The method according to p. 12, characterized in that the loading of web pages and their analysis is carried out starting with a link to a web page from the schedule that has the highest coefficient K. 14. A computer-implemented system for checking web pages for the presence of real-time multimedia streams, including:
Schedule database, including a list of links to web pages with a period for each link and the procedure for checking it;
Database of streams, including a list of links to multimedia streams, as well as information about the type, status and characteristics of streams;
Relationship database, including information on the affiliation of the multimedia stream to the corresponding web page, as well as the type of multimedia stream;
a data loading module configured to download web pages via a link from the schedule database and analyze downloaded web pages for links to multimedia streams;
a data management module, configured to save the links found to the multimedia streams found by the download module to the streams database, as well as to save and / or change information about the relationship between the multimedia stream and the web page in the relationships database;
a module for checking streams from the database of streams, configured to determine the type of multimedia stream, as well as periodically check references to multimedia streams in real time to detect changes in the type of stream and / or in the state and / or characteristics of the stream, followed by storing the received information in the stream database;
a flow control module configured to detect changes made to the flow database, followed by recording information about the presence of changes in the relationship database;
a schedule management module, configured to change the schedule in the schedule database by adding new links to web pages into it, on which links to real-time multimedia streams are identified, and / or by changing the check period for existing links to web pages in the schedule, for which there are changes in the stream, and / or changes in the start time of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to while maintaining its verification period; when detecting changes in the type and / or status of the stream associated with an increase in the number of links to real-time multimedia streams on a web page, the check period is reduced, and when the number of links to real-time multimedia streams on a web page decreases, the check period is increased, and when the number of links to real-time multimedia streams becomes equal to zero, this link to a web page is excluded from the scan schedule. 15. The system of claim 14, characterized in that the data loading module is configured to emulate a web browser by building a model of a web document and creating all objects potentially containing links to multimedia streams. 16. The system of claim 14, characterized in that the module for checking streams is configured to change the period for checking links to streams from the database of streams having the status of a multimedia stream of real time from 2 to 5 minutes. 17. The system according to 14, characterized in that the flow check module, when checking the status of the real-time stream, determines whether a given stream is on or off. 18. The system according to 14, characterized in that in the stream database as a stream characteristics there is a description of the multimedia stream and technical characteristics of the stream. 19. The system according to 14, characterized in that in the stream database as a description of the multimedia stream there is a text description of the stream, the stream title, an indication of the owner of the stream, a link to the site of the stream or any other data transmitted within the stream and displaying its essence . 20. The system of claim 14, characterized in that the stream database contains bitstream, format, information about audio or video codecs or any other technical characteristics of the stream as technical characteristics of the stream. 21. The system according to 14, characterized in that the data loading module is configured to determine the availability of a web page, and in the event that inaccessible web pages are detected, information about this is recorded in the schedule database. 22. The system according to item 21, characterized in that if the unavailable web page is in this state for a week, the schedule management module is configured to exclude links to this web page from the schedule database. 23. The system of claim 14, characterized in that the schedule management module is configured to determine a significance coefficient (K) for each link to a web page that is selected equal to the number of links on this page to real-time multimedia streams, and calculating a period checks based on the ratio T = T _const / K, where T _const is the specified test period, for example 24 hours, and if T <T _min , then T = T _min , where T _min is the minimum allowable test period, for example 1 hour, with the subsequent preservation of the received results atov for (R) and (T) in the schedule database. 24. The system according to claim 23, characterized in that the data loading module is configured to download web pages and analyze them starting from a link to a web page from the schedule database having the highest coefficient K.

Images