JP6875579B2 - Machine learning-based identification of corrupted network connections - Google Patents

Description

Background The Internet (registered trademark) facilitates information exchange and transactions between users around the world. Through such information exchange, the data provider can provide data to various users. Data can be provided in a variety of ways, including publishing information on electronic documents (eg, web pages) and within applications (eg, mobile applications). For example, an application can include an application page that presents information to the user.

Links can be used to help users navigate to resources of interest. The link can point to an electronic document or a specific location within the application (eg, an application page). However, some links may not point correctly to the document or application page. Such links are commonly referred to as broken links. For example, if you move the document or application page pointed to by a link to a new address or remove it from the domain that hosted the document or application page, the link may become a corrupted link. In addition, application links can be corrupted after an application update or in response to the server hosting the application content becoming unavailable.

Summary This specification uses machine learning based on how long the content was presented after initiating (or completing) a network connection (for example, in response to a user interaction with a corrupted application link). Describes techniques for identifying corrupted network connections (eg, corrupted application links).

In general, one innovative aspect of the subject matter described herein can be implemented in a system, which is a plurality of different user interactions with one or more application links that link to a given application. For one or more fronts that receive data indicating a presentation period indicating the period during which the application content linked by the application link was presented, following the plurality of different user interactions with the one or more application links. It has an end server. The system may further include one or more back-end servers that communicate with one or more front-end servers and perform operations, which operations are based on the application of a machine learning model for the presentation period for each application link. Includes classifying the above application links as broken or functioning normally. The machine learning model can be generated using the labeled training data, and the labeled training data is for each training application link in the set of training application links, for the presentation period for the training application links. The training application link is corrupted or is functioning properly based on whether the content pointed to by the training application link was successfully presented as a result of user interaction between the set and the training application link. Includes a label indicating whether or not it is present. The back-end server also performs actions that include generating and outputting an alarm that identifies a given application link as a corrupted link based on that it was classified as corrupted by a machine learning model. Can be done. Other embodiments of this aspect include corresponding devices, methods, and computer programs encoded on computer storage devices that are configured to perform method actions.

Each of these and other embodiments may optionally include one or more of the following features: In some aspects, the machine learning model classifies application links by assigning each presentation period for the application link to a period group in a set of period-based period groups. Each period group can be for a range of time periods and may include the presentation time for the application link within the range of the time period for the time period group. The machine learning model may classify the application links as corrupted or functioning properly, based on a pattern defined by the number of presentation periods assigned to each period group. The pattern, defined by the number of presentation periods, may include the distribution of the presentation periods.

In some aspects, the action performed by the system identifies the set of application links that contain the one or more application links that link to a given application, and the application links, the set of application links. Based on the characteristics of the application links within, grouping into a set of application link groups, and applying each application link group to the presentation period for the application link in the application link group, based on the application of the machine learning model. It may include classifying as having a broken link or a link that is functioning normally.

In some aspects, each application link contains a string that identifies a particular location within a particular application. The operation performed by the system may include identifying one or more application link prefix patterns for a set of application links containing the one or more application links, based on at least the string of the application links. Each application link prefix pattern may indicate a string common to two or more application links. For each application link prefix pattern, the action performed by the system is to identify the group of application links that contain the string indicated by that application link prefix pattern and the presentation period for each application link within the group of application links. Identifying a set of, applying the machine learning model to the presentation period for a group of application links, and applying the machine learning model to a given group of application links for a given application link prefix pattern. May include generating and outputting an alarm that identifies the set of application links as corrupted based on classifying the set of application links as corrupted.

In some aspects, the actions performed by the system include identifying one or more of the application link prefix patterns based on the number of presentation periods available for each application link prefix pattern. In some aspects, the machine learning model includes a linear regression model.

Specific embodiments of the subject matter described herein can be realized to achieve one or more of the following advantages: Selecting a corrupted link to the application's content can cause the application to crash or malfunction, so identifying and removing the corrupted link to the application's content can cause the application to crash and / or improperly. It can function or prevent the device from running the application. By using machine learning to identify bad application links, it is possible to identify bad application links that point to application content that cannot be properly crawled by the application or web crawler. User dissatisfaction is alleviated by directing the user's device to incorrect application content or by reducing the number of bad application links that cause applications running on the user's device to crash or malfunction. ..

Details of one or more embodiments of the subject matter described herein are described in the accompanying drawings and the detailed description below. Other features, aspects and advantages of the subject will become apparent from the detailed description, drawings and claims.

A corrupted link detection is an exemplary environment in which a corrupted application link automatically detects a corrupted application link. It is a graph which shows the representation of the exemplary distribution of the presentation period about an application link. It is a graph which shows the representation of the exemplary distribution of the presentation period about an application link. It is a flowchart of an exemplary process for training a machine learning model to predict whether an application link is a corrupted link. It is a flowchart of an exemplary process of identifying a corrupted application link. It is a flowchart of an exemplary process identifying an application link prefix pattern of a corrupted application link.

Similar reference numbers and designations in various drawings indicate similar elements.
Detailed Description The systems and technologies described herein cannot be tested by selecting an application link (eg, manually or automatically) to determine if the content pointed to by the application link is available. Allows detection of corrupted application links, including links. For example, an application crawler used to test an application link in this aspect has access to the application content pointed to by the application link because the application may require login information to access the application content. May not be available. In another example, testing an application link using an application crawler can result in longer network timeouts than testing a web link to a web page using a web crawler, making the application crawler less effective. .. By using machine learning instead of application crawlers, such network timeouts can be avoided, thereby improving the technique of identifying corrupted application links, for example by providing more accurate corrupted link analysis. can do.

The system can train a machine learning model to predict whether an application link is a corrupted link based on the presentation period for the application link. The presentation period for the application link indicates the period during which the application content linked by the application link is presented following the user's interaction with the application link. The presentation period may be measured from the time when the user interaction with the application link occurs to, for example, the time when the application is closed or navigates to other content in response to the user interaction with the application. If the application crashes, the presentation period may be measured from the time when the user interaction with the application link occurs and the time when the application crashes.

In some implementations, the machine learning model classifies the application link as corrupted or functioning properly, based on the distribution of presentation periods for the application link. For example, application links have a substantially higher number of short presentation periods (eg, less than the threshold) (eg, a specified number or percentage) than a number of long presentation periods (longer than the threshold period). The application link may have a distribution pattern suggesting that the application link is corrupted. In another example, if an application link has a naturally distributed presentation period with a small number of short presentation periods (eg, below the threshold), then the application link works fine and it works. It may have a distribution pattern suggesting that it is not damaged.

Machine learning models can also predict (eg, determine) whether a group of application links is corrupted. Application links can be grouped into application link groups based on the application link prefix pattern. Each application link can contain a string that identifies a particular location within a particular application, such as a particular application page for an application, a particular video for a video application, or a particular restaurant page for a restaurant review application. .. Each application link prefix pattern can indicate a character string common to a plurality of application links. For example, the character string of the first application link may be "appname: itemcategory1 / item1", and the character string of the second application link may be "appname: itemcategory1 / item2". The application link prefix pattern link for these two applications may be "appname: itemcategory /" because this string is common (eg, the same) for both application links. Applications that have a matching prefix pattern (for example, that is the same or is within a certain similarity criterion based on the number of matching characters) are often corrupted at the same time or for the same reason, so the application By grouping links by prefix pattern, the machine learning model can predict whether the links within the group are broken without having to run separately during the presentation period for each application link. This grouping can be categorized more quickly, using less computational resources, compared to having to run the model separately on each application link. Grouping can also increase the amount of selection period data used by the machine learning model for a particular classification, reducing the impact of noisy or incorrect data, thus providing more accurate predictions. Can bring.

FIG. 1 is an exemplary environment 100 in which the broken link detection system 110 automatically detects broken application links. The broken link detection system 110 includes one or more front-end servers 112 and one or more link classification servers 114. The front-end server 112 can receive data indicating the presentation period from the client device 142 (for example, the client devices 142-A and 142-B) via the network 130. The front-end server 112 may also send notifications or alerts to the client device 142 over the network 130, for example, in response to the application link being classified as a corrupted link. The network 130 can include a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof.

Each client device 142 can include one or more applications 146 and period monitors 144. For example, client device 142-A includes application 146-A and period monitor 144-A. Similarly, client device 142-B includes application 146-B and period monitor 144-B. Application 146 may include native applications developed for a particular platform or a particular type of device.

The user accesses the content of application 146 by interacting with (eg, selecting) an application link that points to application 146 or a specific location within that application (eg, links to that application's network location). You may. The application link contains a string that causes the client device 142 to open the application 146 (if the application 146 is not already open) and navigates to the specific location pointed to by the application link in response to user interaction with the application link. be able to. For example, the string can identify a particular location within the application 146 and the application 146 pointed to by the application link. The application link can be a deep link in the form of a link to a specific page (or user interface screen) other than the application's top page (eg, welcome page or sign-in page) or in a unique resource identifier (URI). ..

Application links can be presented on web pages, within applications, on search results pages, and / or on other types of electronic resources. The same application link may be presented on multiple different resources, allowing you to interact with different users. As mentioned above, when application 146 is updated, the server hosting the application content becomes unavailable (eg, goes offline), or other events that may not be readily apparent to the user. If so, the application link may be corrupted. In such cases, user interaction with the application link (on any resource for which the application link is presented) may crash the application pointed to by the application link or cause that application to present incorrect content. unknown. Therefore, identifying corrupted application links and removing them from electronic resources can prevent such crashes and the presentation of incorrect content and improve the user experience with the application.

Corrupted links are links that are not properly linked to a particular application or to a particular location within the application. For example, a corrupted application link may link to a location within the application that does not exist. In another example, a corrupted application link is a location within an application that previously existed, but is now unavailable to the server hosting the content at that location, or, for example, of that application. Due to an update, the content has been moved to a different location and may be linked to a location that does not exist.

Application links can also be corrupted if they are properly linked to incorrect content. For example, an application link may be intended to link to a particular news story, but actually links to a different news story. In this example, a user interacting with an application link may be presented with a different news story and may close the application or return to the application or content containing the link without reading the different news story. Identifying and removing such corrupted application links improves the user experience, reduces the number of network requests initiated to retrieve content, improves network performance, and reduces the demand for network resources. Will be done.

The period monitor 144 can monitor the presentation period of the application content. For example, the period monitor 114 can monitor the presentation period of the content presented following the user dialogue (eg, selection thereof) with the application link that the user points to the application content. The presentation period can be measured from the time when the user interaction with the application link occurs to the time when the application is closed or the application navigates to other content, for example, in response to the user interaction with the application. it can. For example, the user may select an application link (eg, presented on a search results page) to a particular location within the application at 12:01 pm, and the content at that particular location is the application. It may be presented to the user via. In response to the selection, the period monitor 144 (or the search system that provided the search results page) may log the time of the selection (eg, search log). If the user selects the back button or exits the application, the client device 142 may return to the search results page. The period monitor 144 (or search system) may log the time when the client device 142 closes the application or returns to the search results page, eg 12:05. In this example, the period monitor 144 may obtain, for example, two times from the log and subtract 12:01 from 12:05 to determine the presentation period. Therefore, in this example, the presentation period is 4 minutes. If the application crashes, the presentation period can be measured from the time when the user interaction with the application link occurs and the time when the application crashes. For example, the period monitor 144 may receive a message indicating the time when the application crashed, for example, from the operating system of the application or client device.

The period monitor 144 may monitor and record the presentation period for a plurality of different application links that receive the user dialogue on the client device 142. The period monitor 144 may also transmit data indicating the presentation period for each application link to the front-end server 112 of the broken link detection system 110. The front-end server 112 can receive presentation periods from a plurality of different client devices 142 associated with a plurality of different users. For example, the front-end server 112 receives a presentation period from client device 142-A associated with the first user and client device 142-B associated with a second user different from the first user. May be good. In this way, the selection period for a given application link may be aggregated across multiple different users.

The front-end server 112 can provide the received presentation period to the link classification server 114. The link classification server 114 can then store, for example, the presentation period in the presentation period database 118 stored in the data storage device. Each presentation period can be stored with an association to the application link in which the presentation period was recorded.

The link classification server 114 can be implemented as one or more backend servers to predict if an application link is corrupted and / or if an application link within a group of application links is corrupted. , Machine learning model 116 can be trained. An exemplary technique for training a machine learning model is shown in FIG. 4 and described below. The machine learning model 116 can classify an application link as corrupted or functioning normally (eg, not corrupted) based on the presentation period for the application link. For example, the link classification server 114 may apply the machine learning model 116 to a set of presentation periods for application links and classify the application links based on the set of presentation periods. The set of presentation periods used to classify an application link may be for a specific period following a user interaction with the application link, eg, content presentation that occurred the day before or the previous week.

In some implementations, the machine learning model 116 classifies the application link as broken or functioning properly, based on the distribution of presentation periods for the application link. In this example, the link classification server 114 may have a set of period groups, and each period group may correspond to a range of time periods. For example, the group may include a first group for a period of 0 to 3 seconds, a second group for a period of 3 to 6 seconds, a third group for a period of 6 to 9 seconds, and so on. Good. For each presentation period, the link classification server 114 can identify a range of periods within which the presentation period falls and assign the presentation period to a period group corresponding to the range of the period. For example, the link classification server 114 may allocate a 7 second presentation period to a period group having a range of 6-9 seconds.

The link classification server 114 can determine the count of the number of presentation periods assigned to each period group. The count for each period group defines the distribution of presentation periods for application links. Specifically, the distribution for application links indicates the number of presentation periods for application links assigned to a period group for each set of period groups. An exemplary distribution of application links is shown in FIG. 2 and described below. The machine learning model 116 can be applied to the count of each period group for the application link and can classify the application link based on the count. For example, the link classification server 114 may generate a feature vector containing vector elements for each period group. The value of each vector element may be equal to (or proportional to) the count of the number of presentation periods for the application links assigned to the period group corresponding to that vector element. For example, the feature vector may include vector elements corresponding to a period group of 6-9 seconds. The value of this vector element may be the number of presentation periods for the application link, which was between 6 and 9 seconds. The link classification server may apply the machine learning model 116 to the feature vector generated for the application link to classify whether the application link is corrupted or functioning properly.

By classifying the application links based on the distribution of presentation periods, the machine learning model 116 can take into account the pattern of presentation periods over multiple requests for the application by the plurality of different client devices 142. For example, a broken application link can result in a shorter duration group being assigned a larger number of presentation periods and a longer duration group being assigned a significantly smaller number of presentation periods. In contrast, a functioning application link can have a significant number of presentation periods (eg, more than a threshold number) that are assigned to many different period groups, including those with longer periods. .. The machine learning model 116 is designed to identify different different distribution patterns for corrupted application links and different different distribution patterns for functioning application links, as described in more detail below. Can be trained.

The machine learning model 116 can classify groups of application links in a manner similar to individual application links. For example, the link classification server 114 can identify the presentation period for the application link for each application link included in the application link group, and can aggregate the presentation period for the group. The link classification server 114 then applies the machine learning model 116 to the presentation period for a group of application links and outputs a classification as to whether the application links within the group are corrupted as a whole. For example, a "corrupted" classification for a group may indicate that each application link within that group is corrupted. In response, the link classification server 114 may generate an alarm indicating that the application links in the group may be corrupted and send it to the front-end server 112. The front-end server 112 may then send an alert to the system administrator's client device, which may further investigate the application links within the group.

To classify a group of application links, the link classification server 114 may generate a distribution of presentation periods for application links within the group and apply a machine learning model to that distribution. For example, the link classification server 114 may generate feature vectors for groups in a manner similar to feature vectors for individual application links. The link classification server 114 may generate a feature vector containing vector elements for each period group within the set of period groups. The value of each vector element may be equal to (or proportional to) the count of the number of presentation periods for the group of application links assigned to the period group corresponding to that vector element. For example, the feature vector may include vector elements corresponding to a period group of 1-3 seconds. The value of this vector element may be the total number of presentation periods of the group, which was 1-3 seconds. As an example, a group may have two application links as members. One of the two application links may have 10 presentation periods between 1 and 3 seconds, and the other application link has 5 presentation periods between 1 and 3 seconds. You may have it. In this example, the vector element corresponding to the period group of 1-3 seconds may have a value of 15. The link classification server may apply the machine learning model 116 to the feature vectors generated for a group of application links to classify the application links within the group as corrupted or functioning properly.

The link classification server 114 can assign a set of application links to a group based on one or more characteristics of the application link and / or one or more characteristics of the application or application content pointed to by the application link. The link classification server 114 can store data identifying those groups and members of each group in the link group database 120. In some implementations, the link classification server 114 can assign application links to groups based on the strings contained in the application links. As mentioned above, application links can be grouped based on the application link prefix pattern. In this example, the link classification server 114 can identify strings found in one or more of the application links that generate the group. In a particular example, the link classification server 114 identifies a string that is common (eg, the same) to two or more of the application links and generates a group for the application link prefix pattern defined by that common string. You may.

In some implementations, the application link prefix pattern is based on a string that begins at the beginning of the application link string. For example, the character string of the first application link may be "appname: itemcategory1 / item1", and the character string of the second application link may be "appname: itemcategory1 / item2". In this example, the link classification server 114 may evaluate the string from left to right and determine that the two application links have a common prefix of "appname: itemcategory1 /". Accordingly, the link classification server 114 may generate a group for the application link prefix pattern "appname: itemcategory1 /" and include each application link having a character string starting with the application link prefix pattern in the group.

The link classification server 114 may generate groups based on the number of application links and / or the number of presentation periods available for each potential group. For example, the link classification server 114 may identify a character string that is common to two or more application links. For each identified string, the link classification server 114 may identify the number of application links containing that string. The link classification server 114 may then generate a group for each character string contained in at least a threshold number of application links. In another example, the link classification server 114 may rank the strings based on the number of application links containing the strings and generate a group for the top N strings, where. "N" is the specified number (for example, specified by the administrator of the corrupted link detection system and can be retrieved from a data store or digital memory device). For each group, the link classification server 114 can populate that group with application links that contain the strings generated by that group.

Similarly, the link classification server 114 may identify strings that are common (eg, the same) to two or more different application links. For each identified string, the link classification server 114 may identify the total number of presentation periods stored in the presentation period database 118 for application links containing that string. When the link classification server 114 uses the machine learning model 116 to classify application links based on the presentation period for a particular time period (eg, the previous day), the link classification server includes the string and the presentation period. May identify the total number of presentation periods for application links that have occurred, at least in part, within a particular time period. For example, the presentation period may occur partially within that particular period if the presentation of the application content started within that particular period but continued until after that particular period ended. The link classification server 114 can generate a group for each string available for application links that include at least a threshold number of presentation periods. In another example, the link classification server 114 ranks the strings based on the number of presentation periods available and produces a group of strings with the top N (where "N" is the specified number). You may. For each group, the link classification server 114 can populate the group with an application link containing the string from which the group was generated.

The link classification server 114 may also generate groups for a particular application. For example, a link classification server may create a group for a particular application, and the group for a particular application may include each application link pointing to that particular application or a particular location within that particular application. ..

The link classification server 114 can also classify application links and / or groups of applications based on changes in the presentation period over time. For example, if the presentation period for an application link is significantly reduced (for example, by at least a threshold amount), the application link may be corrupted. To classify an application link, the link classification server 114 can determine the average presentation period for that application link for a particular time period, eg, each day over the previous week (or another suitable time period). The application server can also determine the average duration of application link presentation for the day (or any other suitable period). The link classification server 114 can then compare the average presentation period for the current day with the average presentation period for the previous day (or the combined average for all previous days under consideration). If the average presentation period for the current day is at least a threshold amount less than the average presentation period for the previous day, the link classification server 114 may classify the application link as a corrupted link.

In some implementations, the link classification server 114 does not have to classify the application links if the standard deviation of the average presentation period for the previous day exceeds the threshold. For example, a high standard deviation (eg, above a threshold) may indicate that the data is not stable enough to be used to classify application links, which causes the link classification server 114 to erroneously. It may be prevented from outputting the classification.

FIG. 2 is a graph 200 showing an exemplary distribution representation of the presentation period for application links. Graph 200 shows a presentation period group and, for each period group, a count of the number of presentation periods assigned to each period group. In this example, the period groups are for a period range of 2 seconds, respectively. For example, the first period group is for a period range of 0 to 2 seconds, and the first period group is assigned 10 presentation periods. That is, the content of the application linked by the application link was viewed 10 times over a period of 0 to 2 seconds. In some implementations, the distribution may be based on the percentage of presentation periods assigned to each period group, rather than counting the number of presentation periods assigned to each period group.

Machine learning models can be applied to distributions to predict (eg, determine) whether application links are functioning or broken. This exemplary distribution may represent a presentation period for a broken link, because most of the presentation period is less than 20 seconds and less than 30 seconds. Therefore, the machine learning model may output a classification of "corruption" for this exemplary distribution.

FIG. 3 is a graph 300 showing an exemplary distribution representation of the presentation period for application links. Similar to Graph 200 in FIG. 2, Graph 300 shows a presentation period group and, for each period group, a count of the number of presentation periods assigned to each period group. Machine learning models can be applied to distributions to predict (eg, determine) whether application links are functioning or broken. This exemplary distribution may represent the presentation period for a functioning link, because the presentation period is distributed over a range of periods and is not heavily weighted towards shorter presentation periods. is there. Therefore, the machine learning model may output the classification "working normally" for this exemplary distribution.

FIG. 4 is a flow chart of an exemplary process 400 for training a machine learning model to predict whether an application link is a broken link. The operation of process 400 can be performed by a system that includes one or more data processing devices, such as the broken link detection system 110 of FIG. The process 400 can also be executed by the instructions stored in the computer storage medium, and the execution of the instructions by the system including the data processing device causes the data processing device to execute the operation of the process 400.

The system identifies a set of training application links (402). For example, the system may access a database or index that contains data identifying training application links and labeled training data for each training application link. The training application link can be an application link for which labeled training data is available. For example, a training application link has at least a threshold number of actual operating states (eg, functioning or corrupted) that are known and available to use when training a machine learning model. It may be an application link that has a presentation period of. Training application links can be identified such that the set contains at least a threshold number of normally functioning application links and at least a threshold number of corrupted application links.

The system acquires labeled training data for each training application link (404). For example, the system may access the database, identify the labeled training data that is indexed for each training application link, and retrieve that labeled training data for those training application links. The labeled training data for each training application link can include a set of presentation periods for that training application link, and a label. The label for the training application link can indicate whether the training application link is corrupted or functioning properly. For example, a label for an application link is either corrupted or successful based on whether the device navigates to the content pointed to by the training application link through user interaction with the training application link. Can indicate if it is working.

In some implementations, one or more system administrators and / or one or more users test the training application link and update the label accordingly. For example, the system administrator may interact with the training application link (for example, select it) and monitor to see if the content linked by the training application link is presented. If the correct content is presented after interacting with the training application link, the system administrator may update the label for the training application link to indicate that the training application link is working properly. If the application linked by the training application link does not open, or if the content linked by the training application link is not presented after interacting with the training application link, the system administrator updates the label to update the training application link. May indicate that is damaged. In another example, a user interacting with a training application link was prompted, for example, to automatically use a user interface element to choose whether the training application was working or corrupted.

The system uses the training data to train the machine learning model to predict whether the application link is functioning properly or corrupted (406). In some implementations, the system may generate a machine learning model by partitioning a portion of the training application link and the corresponding presentation period and label into a training set and a test set. The machine learning model may then be trained using the presentation period of each training application link in the training set and the label for each training application link in the training set. In some implementations, the machine learning model has one or more weights associated with a particular feature of the presentation period, such as a weight for each period group, a weight for the size of the presentation period, and / or each period group. May include weights for the presentation period of relative numbers in. Training a predictive model may include adjusting the weights associated with features and feature combinations that have been found to predict whether the application link is broken or functioning properly. Various methods and techniques can be used to correlate features with labels, including clustering, linear regression, logistic regression, and non-linear regression.

After training the predictive model, the model may be tested using the presentation period of the training application link within the test set or within another training set. The weights associated with each feature in the machine learning model may then be adjusted based on the results of the test.

FIG. 5 is a flowchart of an exemplary process 500 for identifying a broken application link. The operation of process 500 can be performed by a system that includes one or more data processing devices, such as the broken link detection system 110 of FIG. The process 500 can also be executed by an instruction stored in a computer storage medium, and the execution of the instruction by the system including the data processing apparatus causes the data processing apparatus to execute the operation of the process 500.

The system receives data indicating the presentation period for one or more application links for a given application (502). Each of the one or more application links may be linked to a particular location or specific content within a given application. Each presentation period may indicate the period during which the application content linked by the application link was presented following the interaction with the user's application link.

The data can include presentation periods for a plurality of different user interactions with each of the one or more application links. For example, each application link may receive dialogue from a number of different users. In response to each dialogue, the application content linked by the application link may be presented to the user interacting with the application link. For each user dialogue, the system can monitor the period during which the application content is presented following the user dialogue and record that period as the presentation period for the application link.

If the application link is broken, the application content may not be presented to the user, or incorrect content may be presented to the user. If the application crashes, the presentation period may indicate the amount of time between the user interaction with the application and the application crash. If the application does not open, the presentation period may be zero. If incorrect content is presented, the presentation period may be short (eg, less than the threshold period), because the user can navigate away from the incorrect content or close the application.

The system classifies each application link as corrupted or functioning normally, based on the application of the machine learning model to the presentation period for the application link (504). For example, the system can identify a machine learning model trained to predict whether an application link is functioning properly or corrupted based on the observed presentation period for the application link. The machine learning model may be trained using, for example, the exemplary process 400 shown in FIG. 4 and described above.

In some implementations, the machine learning model predicts whether the application link is functioning or broken based on the distribution of presentation periods. For example, the system may assign each presentation period to a period group that has a range of time periods within which the presentation period falls. The system can determine the count of the number of presentation periods assigned to each period group. The count for each period group defines the distribution of presentation periods for application links. Machine learning models can be applied to counts for each period group, and machine learning models are classified as "working" or "corrupted" for application links based on that count. Can be output.

In some implementations, the machine learning model can be applied to feature vectors for application links. In this example, the feature vector can include vector elements for each period group. The value of each vector element may be equal to (or proportional to) the count of the number of presentation periods for the application links assigned to the period group corresponding to that vector element.

The system generates and outputs an alert for each application link classified as a broken link by the machine learning model (506). An alert for a given application link may identify a given application link as a broken link based on that it was classified as broken by a machine learning model. The system may send an alert to the system administrator's client device. In response, the system administrator may further investigate the application link. If the system administrator determines that the application link is corrupted, the system administrator may remove the corrupted link from one or more electronic resources that contain the link and / or remove the link. It may be repaired.

FIG. 6 is a flowchart of an exemplary process 600 for identifying an application link prefix pattern for a corrupted application link. The operation of process 600 can be performed by a system that includes one or more data processing devices, such as the broken link detection system 110 of FIG. The process 600 can also be executed by an instruction stored in a computer storage medium, and the execution of the instruction by the system including the data processing apparatus causes the data processing apparatus to execute the operation of the process 600.

The system identifies one or more application link prefix patterns for a set of application links, at least based on the application link string (602). For example, as mentioned above, each application link can include a string to identify a particular location within a particular application. The system may identify a string that is common to two or more of the application links and generate a group for the application link prefix pattern defined by that common string. As mentioned above, application link groups may be generated based on the number of application links and / or the number of presentation periods available for each potential group.

For each application link prefix pattern, a group of application links containing the string indicated by the application link prefix pattern is identified (604). For example, the system may compare the string indicated by the application link prefix pattern with the string for each application link. Each application link containing a set of characters that matches the string indicated by the application link prefix pattern may be included in the application link group for that application link prefix pattern. For example, if the first string is the same as the second string, or at least has a certain degree of similarity, then the two strings may be considered to match. The similarity between strings may be based on the number of characters (or percentage of characters) that are the same in those strings and at the same position in the strings. In some implementations, the system simply includes an application link within an application link group for an application link prefix pattern that has a string beginning with the string indicated by that application link prefix pattern.

For each application link prefix pattern, the system identifies a set of presentation periods for each application link within the group of application links (606). For example, each application link may have a set of presentation periods. The system may aggregate the presentation periods for a group into a set of presentation periods.

For each application link prefix pattern, the system applies a machine learning model to a set of presentation periods for a group of application links (608). The system may apply the machine learning model to the set of presentation periods in the same or similar manner as described above with reference to FIG. For example, the system may assign each presentation period in a set of presentation periods to a period group that has a range of periods within which the presentation period falls. The system can determine the count of the number of presentation periods in the set of presentation periods assigned to each period group. The system can apply a machine learning model to the count for each period group, and the machine learning model is "working" or "corrupted" for the application link based on that count. The classification of can be output.

In some implementations, the machine learning model can be applied to feature vectors for application link groups. In this example, the feature vector can include vector elements for each period group. The value of each vector element may be equal to (or proportional to) the count of the number of presentation periods for the application link group assigned to the period group corresponding to that vector element.

For a given application link prefix pattern, the system generates an alert that identifies a given group of application links as corrupted links based on the machine learning model classifying the set of application links as corrupted. And output (610). For example, the machine learning model may output a classification of "working properly" or "corrupted" for each application link group. For those with the classification "corrupted", the system can generate an alert that identifies the application link in the application group as a corrupted link. The system may send an alert to the system administrator's client device, and the system administrator may further investigate, repair, or remove the broken link.

Embodiments of the subject matter and operations described herein are in digital electronic circuitry, or in computer software, firmware or hardware, including the structures disclosed herein and their structural equivalents, or them. It can be realized in one or more combinations of. Embodiments of the subject matter described herein are encoded as one or more computer programs, i.e., for execution by a data processor or on a computer storage medium to control the operation of the data processor. It can be realized as one or more modules of computer program instructions. Alternatively, or in addition, program instructions are generated to encode information for transmission to a receiving device suitable for execution by a data processing device, such as a machine-generated electrical signal, optical. It can be encoded on an artificially generated propagated signal, such as a signal or electromagnetic signal. The computer storage medium can be or include a computer-readable storage device, a computer-readable storage board, a random or serial access memory array or device, or a combination thereof. Further, while some computer storage media are not propagated signals, some computer storage media can be the source or destination of computer program instructions encoded by artificially generated propagated signals. The computer storage medium may further be, or may be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

The operations described herein can be implemented as operations performed by a data processor on data stored in one or more computer-readable storage devices or data received from other sources.

The term "data processor" refers to all types of devices, devices and machines for processing data, including, for example, programmable processors, computers, systems on chips, or multiples or combinations of those mentioned above. Include. The apparatus may include special purpose logic circuits such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). In addition to hardware, the device further includes, for example, processor firmware, protocol stacks, database management systems, operating systems, cross-platform run-time environments, virtual machines, or code that constitutes one or more combinations thereof. It can contain code that creates an operating environment for a computer program. The device and execution environment can implement a variety of different computing model infrastructures, including web services, distributed computing and grid computing infrastructures.

Computer programs (also known as programs, software, software applications, scripts or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, as stand-alone programs. , Or can be deployed in any form, including as a module, component, subroutine, object, or other unit suitable for use in a computing environment. Computer programs may, but need not, support files in the file system. A program may be another program or in a single file dedicated to that program, or in multiple coordinated files (eg, a file that contains one or more modules, subprograms, or parts of code). It can be stored as part of a file that holds data (eg, one or more scripts stored in a markup language document). Computer programs may be deployed to run on one or more computers located in one location or distributed across multiple locations and interconnected by communication networks.

The processes and logical flows described herein can be performed by one or more programmable processors running one or more computer programs to perform actions by manipulating input data and producing output. .. Execution of the process and logic flow, as well as implementation of the device, can also be performed by a special purpose logic circuit system such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

Processors suitable for executing computer programs may include, for example, both general purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. In general, the processor will receive instructions and data from read-only memory and / or random access memory. An essential element of a computer is a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or receives data from the one or more mass storage devices. It is operably coupled to transfer data to the one or more mass storage devices, or both. However, the computer need not have such a device. In addition, computers are, for example, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (eg, universal serial bus (USB) flash drives). Can be embedded in another device such as. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices such as EPROM, EEPROM and flash memory elements; for example. Includes magnetic disks such as internal hard disks or removable disks; magneto-optical disks; as well as CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by special purpose logic circuits or incorporated into special purpose logic circuits.

To provide interaction with the user, embodiments of the subject matter described herein include display devices for displaying information to the user, such as a CRT (cathode tube) or LCD (liquid crystal display) monitor. It can be implemented on a computer having a keyboard and a pointing device that allows the user to provide input to the computer, such as a mouse, trackball. Other types of devices can be used to provide interaction with the user as well; for example, the feedback provided to the user is any form of sensory feedback, such as visual feedback, auditory feedback or tactile feedback. Gain; input from the user can be received in any form, including acoustic input, spoken input, or tactile input. In addition, the computer sends the document to the device used by the user and receives the document from the device used by the user, for example, in response to a request received from the web browser, the web browser on the user's client device. You can interact with the user by sending a web page to.

Embodiments of the subject matter described herein include a backend component, eg, as a data server, or a middleware component, eg, an application server, for example, implementations and users of the subject matter described herein. Calculations that include front-end components such as a client computer with a graphical user interface or web browser that can interact, or include any combination of one or more such back-end components, middleware components, or front-end components. It can be realized in the system. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LAN”) and wide area networks (“WAN”), internetworks (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks).

Computational systems can include clients and servers. Clients and servers are generally remote from each other and typically interact through communication networks. The client-server relationship is generated by computer programs that run on their respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (eg, an HTML page) to the client device (for example, to display data to a user interacting with the client device and to receive user input from that user). .. Data generated on the client device (eg, the result of user interaction) can be received on the server from the client device.

Although the present specification contains details of many specific embodiments, they should not be construed as a limitation to the scope of any invention or the scope of what can be claimed, and specific embodiments of a particular invention. It should be interpreted as a description of the characteristics peculiar to. Certain features described herein in the context of separate embodiments may also be realized in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can be realized separately in multiple embodiments or in any suitable partial combination. Further, features are described above as acting in a combination and may even be initially claimed as such, but one or more features from the claimed combination may in some cases be said combination. Combinations that can be removed from and claimed from can be directed to partial combinations or variants of partial combinations.

Similarly, the actions are shown in the figure in a particular order, but such actions need to be performed in that particular order or in a contiguous order to achieve the desired result. It should not be understood, or that all indicated actions need to be performed. In some situations, multitasking and parallelism can be advantageous. Moreover, the separation of the various system components in the embodiments described above should not be understood to require such separation in all embodiments, and the program components and systems described are generally single. It should be understood that they can be integrated together or packaged into multiple software products.

Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims may be performed in a different order and still achieve the desired result. In addition, the process shown in the attached figure does not necessarily have to be in the particular order or sequence shown to achieve the desired result. In some implementations, multitasking and parallelism can be advantageous.

Claims (20)

A method implemented by a data processor
A step that identifies multiple application links with the same prefix pattern, where each application link is a link to an application.
For each application in the plurality of application links, a step of acquiring data indicating a presentation period indicating a period during which the application content linked by the application link is presented, and a step of acquiring data indicating a presentation period.
A step of determining whether an application link having the prefix pattern is corrupted or functioning normally based on the distribution of the presentation period for the plurality of application links.
A step of generating and providing a notification that the application link having the prefix pattern is corrupted in response to determining that the application link having the prefix pattern is corrupted.
A method. Steps to receive data that identifies a set of application links, and
A step of determining that each application link in the plurality of application links has the same character string containing at least a specific number of characters.
A step of determining that the plurality of application links have the same prefix pattern in response to the determination that each application link in the plurality of application links has the same character string containing at least a specific number of characters.
The method of claim 1, further comprising. The first application link in the plurality of application links has the first one or more characters following the prefix pattern.
The second application link in the plurality of application links has a second one or more characters following the prefix pattern.
The first one or more characters are different from the second one or more characters,
The method according to claim 1. The step of determining whether an application link having the prefix pattern is corrupted or functioning normally based on the distribution of the presentation period for the plurality of application links is corrupted based on the distribution of the presentation period. The method of claim 1, comprising the step of presenting the distribution of said presentation periods as input to a machine learning model trained to classify application links as being or functioning normally. The machine learning model was generated using the labeled training data, and the labeled training data was used for each training application link in the set of training application links.
A set of presentation periods for the training application link,
Whether the training application link is corrupted or functioning properly, based on whether the content pointed to by the training application link was successfully presented as a result of user interaction with the training application link. With the label to identify
4. The method of claim 4. The method of claim 1, wherein each presentation period indicates a period in which the application content linked by the application link is presented following a user dialogue using the application link. Based on the distribution of the presentation period for the plurality of application links, a step of determining whether the application link having the prefix pattern is corrupted or functioning normally is
A step of assigning the presentation period to a period group in a set of period groups based on the size of the presentation period, where each period group is for a range of periods and for the period group. A step and a step that includes the presentation period for the application link that is within the period.
With steps to classify said application links with said prefix pattern as corrupted or functioning properly based on the presentation period pattern defined by the number of presentation periods assigned to each period group.
The method of claim 1. Data processing device and
A memory storage device that is in data communication with the data processing device, and the memory storage device stores instructions that can be executed by the data processing device, and at the time of execution, the data processing device stores the instructions.
Identifying multiple application links that have the same prefix pattern, where each application link is a link to an application.
For each application in the plurality of application links, acquisition of data indicating a presentation period indicating the period during which the application content linked by the application link was presented, and
Determining whether an application link having the prefix pattern is corrupted or functioning normally based on the distribution of the presentation period for the plurality of application links.
To generate and provide a notification that the application link having the prefix pattern is corrupted in response to determining that the application link having the prefix pattern is corrupted.
With a memory storage device that performs the operation
The system. The above operation
Receiving data that identifies a set of application links and
Determining that each application link in the plurality of application links has the same character string containing at least a specific number of characters.
In response to determining that each application link in the plurality of application links has the same character string containing at least a specific number of characters, it is determined that the plurality of application links have the same prefix pattern.
8. The system of claim 8. The first application link in the plurality of application links has the first one or more characters following the prefix pattern.
The second application link in the plurality of application links has a second one or more characters following the prefix pattern.
The first one or more characters are different from the second one or more characters,
The system according to claim 8. Determining whether an application link having the prefix pattern is corrupted or functioning normally based on the distribution of the presentation period for the plurality of application links is corrupted based on the distribution of the presentation period. The system of claim 8, comprising presenting the distribution of said presentation periods as input to a machine learning model trained to classify application links as being or functioning normally. The machine learning model was generated using the labeled training data, and the labeled training data was used for each training application link in the set of training application links.
A set of presentation periods for the training application link,
Whether the training application link is corrupted or functioning properly, based on whether the content pointed to by the training application link was successfully presented as a result of user interaction with the training application link. With the label to identify
11. The system of claim 11. The system according to claim 8, wherein each presentation period indicates the period during which the application content linked by the application link is presented following the user dialogue using the application link. Determining whether an application link with the prefix pattern is corrupted or functioning normally based on the distribution of the presentation periods for the plurality of application links can be determined.
Assigning each presentation period to a period group in a set of period groups based on the size of the presentation period, where each period group is for a range of periods and for said period group. Allocation, including the presentation period for the application link, which is within the period.
To classify said application links with said prefix pattern as corrupted or functioning properly based on the offer period pattern defined by the number of presentation periods assigned to each period group.
8. The system of claim 8. A computer storage medium encoded by a computer program that, when executed by the data processing apparatus, causes the data processing apparatus to perform.
Identifying multiple application links that have the same prefix pattern, where each application link is a link to an application.
For each application in the plurality of application links, acquisition of data indicating a presentation period indicating the period during which the application content linked by the application link was presented, and
Determining whether an application link having the prefix pattern is corrupted or functioning normally based on the distribution of the presentation period for the plurality of application links.
To generate and provide a notification that the application link having the prefix pattern is corrupted in response to determining that the application link having the prefix pattern is corrupted.
A computer storage medium containing instructions for performing an operation. The above operation
Receiving data that identifies a set of application links and
Determining that each application link in the plurality of application links has the same character string containing at least a specific number of characters.
In response to determining that each application link in the plurality of application links has the same character string containing at least a specific number of characters, it is determined that the plurality of application links have the same prefix pattern.
The computer storage medium of claim 15, further comprising. The first application link in the plurality of application links has the first one or more characters following the prefix pattern.
The second application link in the plurality of application links has a second one or more characters following the prefix pattern.
The first one or more characters are different from the second one or more characters,
The computer storage medium of claim 15. Determining whether an application link having the prefix pattern is corrupted or functioning normally based on the distribution of the presentation period for the plurality of application links is corrupted based on the distribution of the presentation period. 15. The computer storage medium of claim 15, comprising presenting the distribution of said presentation periods as input to a machine learning model trained to classify application links as being or functioning normally. The machine learning model was generated using the labeled training data, and the labeled training data was used for each training application link in the set of training application links.
A set of presentation periods for the training application link,
Whether the training application link is corrupted or functioning properly, based on whether the content pointed to by the training application link was successfully presented as a result of user interaction with the training application link. With the label to identify
18. The computer storage medium of claim 18. The computer storage medium of claim 15, wherein each presentation period indicates the period during which the application content linked by the application link is presented following a user dialogue using the application link.

Images