CN112527611A - Product health degree assessment method and system - Google Patents
Product health degree assessment method and system Download PDFInfo
- Publication number
- CN112527611A CN112527611A CN202011018489.7A CN202011018489A CN112527611A CN 112527611 A CN112527611 A CN 112527611A CN 202011018489 A CN202011018489 A CN 202011018489A CN 112527611 A CN112527611 A CN 112527611A
- Authority
- CN
- China
- Prior art keywords
- data
- crash
- index
- monitoring
- statistical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to a method and a system for evaluating the health degree of a product, wherein the method comprises the following steps: collecting product application data of corresponding types according to the evaluation types; classifying the application data to obtain corresponding evaluation index data; and analyzing the evaluation index data according to a health evaluation strategy matched with the evaluation index to obtain health data of the product. The system includes at least a blast evaluation module configured to include one or more of the following: the device comprises a classification unit, a statistic unit, a monitoring unit, a tracking management unit and an event analysis unit. The invention evaluates the health degree of the product from multiple aspects, and provides a basis for solving the product, improving the product performance and improving the user satisfaction degree.
Description
Technical Field
The invention relates to the technical field of network application, in particular to a method and a system for evaluating product health degree.
Background
With the development of network Application technology, various functions and types of applications (apps) bring great convenience to production and life of people. Network application enterprises can develop various different apps according to user requirements, and for the enterprises, the different apps can be called products of corresponding services. Whether the product can be designed to operate healthily after being brought online is one of important factors for the product to survive. In order to ensure that the product can run healthily after being brought online, the method generally starts from the following aspects: firstly, in the stage of design and development, secondly in the stage of test and debugging, and thirdly, the gray level test is carried out. In the prior art, although the product after the above several stages is satisfactory for various reasons, various problems may still occur after the product is online, such as a crash for various reasons, and various bad experiences reflected by the user and occurring in cold start, page loading, and the like. The staff needs to find the reason and solve the problem according to the problem. At present, no technology for comprehensively evaluating whether an on-line product can run healthily exists, and a worker can only solve the problem passively when the problem occurs and cannot acquire the overall health state of the product to actively improve the product performance.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides a method and a system for evaluating the health degree of a product, which are used for evaluating the health degree of the product from multiple aspects and provide a basis for solving the product, improving the product performance and improving the user satisfaction degree.
In order to solve the above technical problem, according to an aspect of the present invention, there is provided a product health assessment method, including the steps of:
collecting product application data of corresponding types according to the evaluation types;
classifying the application data to obtain corresponding evaluation index data; and
analyzing the assessment index data according to a health assessment strategy matched with the assessment index to obtain health data of the product.
According to another aspect of the invention, the invention provides a product health assessment system comprising a blast assessment module configured to include one or more of the following:
the classification unit is configured to analyze the collected Crash data of the corresponding product according to a plurality of preset classification dimensions to obtain corresponding classification dimension index data of the Crash data;
the statistic unit is configured to count one or more classified dimension index data of the Crash data according to the statistic index to obtain statistic data of corresponding dimensions;
a monitoring unit configured to monitor the statistical data according to a corresponding monitoring index, and to alarm when an alarm condition is reached;
a trace management unit configured to trace manage the created defects; and
an event analysis unit; configured to acquire events that may cause Crash to occur and generate a link of events in a service association or chronological order.
The product health assessment system and the method provided by the invention can assess internet products from multiple aspects and multiple dimensions, so that accurate and detailed product health data can be obtained, and reliable data support is provided for product improvement and problem solving.
Drawings
Preferred embodiments of the present invention will now be described in further detail with reference to the accompanying drawings, in which:
FIG. 1 is a functional block diagram of a product health assessment system according to one embodiment of the present invention;
FIG. 2 is a detailed information schematic of a portion of Crash data according to one embodiment of the invention;
FIG. 3 is a schematic diagram of an interface for creating defects according to one embodiment of the invention;
FIG. 4 is a graph illustrating the Crash delta data for defect occurrence for each hour over 48 hours according to one embodiment of the invention;
FIG. 5 is a Crash sorted list according to one embodiment of the invention;
FIG. 6 is a diagram of Crash statistical data obtained by performing statistics according to two dimension indexes of "create thread" and "SDK" according to an embodiment of the present invention;
FIG. 7 is a graphical representation of alarm information and statistical data for when the Crash hour ring ratio increment exceeds a threshold value, in accordance with one embodiment of the present invention;
FIG. 8 is a schematic diagram of an alarm notification when the total number of Crash exceeds a corresponding threshold, according to one embodiment of the invention;
FIG. 9 is a schematic diagram of an alarm notification alerting when the number of Crash peering increments exceeds a threshold value in accordance with one embodiment of the invention;
FIG. 10 is a statistical information diagram of event information according to one embodiment of the invention;
FIG. 11 is a graph illustrating statistics of performance indicators, according to an embodiment of the present invention;
FIG. 12 is a time consuming diagram of various modules during a cold start according to one embodiment of the invention;
FIG. 13 is a graphical representation of comparative data of the time spent by various modules during a cold start of a terminal of different models in accordance with one embodiment of the present invention;
FIG. 14 is a customer feedback question representation intent according to one embodiment of the present invention;
FIG. 15 is a statistical schematic of a thermal problem according to one embodiment of the present invention;
FIGS. 16A-16B are schematic diagrams of alarm information issued when the amount of feedback for a high sensitivity problem exceeds a threshold in accordance with one embodiment of the present invention;
FIG. 17 is a schematic diagram of issuing a reminder when a defect created from an incident is unresolved, according to one embodiment of the invention;
FIG. 18 is a core primary index data comparison data schematic of a gray scale packet and its reference packets according to one embodiment of the invention;
FIG. 19 is a functional block diagram of a Crash evaluation module according to one embodiment of the present invention;
FIG. 20 is a functional block diagram of a performance evaluation module according to one embodiment of the present invention;
FIG. 21 is a functional block diagram of a customer service feedback module according to one embodiment of the present invention;
FIG. 22 is a functional block diagram of an incident evaluation module according to one embodiment of the present invention;
FIG. 23 is a functional block diagram of a grayscale admission evaluation module according to one embodiment of the present invention; and
FIG. 24 is a schematic diagram of the overall data display according to one embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the following detailed description, reference is made to the accompanying drawings that form a part hereof and in which is shown by way of illustration specific embodiments of the application. In the drawings, like numerals describe substantially similar components throughout the different views. Various specific embodiments of the present application are described in sufficient detail below to enable those skilled in the art to practice the teachings of the present application. It is to be understood that other embodiments may be utilized and structural, logical or electrical changes may be made to the embodiments of the present application.
The invention provides a system and a method for evaluating health degree of a product, wherein the product comprises various applications (Application). In the actual situation, an application may comprise a plurality of versions due to the reasons of increasing functions, improving performances and the like, and the invention can select or set the version to be evaluated according to needs when evaluating the health degree of a product.
FIG. 1 is a schematic block diagram of a product health assessment system according to an embodiment of the present invention. In the embodiment, the health degree of a product is evaluated according to Crash generated by the product, performance, accidents or problems generated on line, customer feedback and the like. The system comprises an application data collection module 1, a preprocessing module 2 and an analysis module 3, wherein the data collection module 1 comprises various data acquisition units and data interfaces, and the data acquisition units can be integrated in a client or/and a server of a product application package as a section of data acquisition codes. When the user downloads the client of the application, the codes for collecting the data are installed on the client terminal along with the application, and various application data are collected at the client. The collected data codes integrated in the server can obtain various application data of the server. The application data collection module 1 further includes a data interface connected to existing platforms such as other data platforms and monitoring systems to acquire corresponding data.
The application data comprises Crash data, performance index data, various client feedback data, accident data and the like. The Crash data comprises user ID, generated version, corresponding business module, generated time, generated times, user quantity, threads which are being created when the user is generated, pages of which basic types (such as OOM, non-OOM, Native or non-Native or Java) generate Crash and the like. And recording the information of the Crash as a Crash record every time the Crash occurs, and numbering the Crash record for convenient management.
The performance index data comprises a cold start index, a hot start index or a page loading index. Wherein the page load index data comprises a load time mean and/or P90. And the data for the cold start indicator also includes the time spent by each module during the cold start. In addition, when collecting the performance index data, in order to obtain more comprehensive data, in addition to collecting the performance data of the full version and the latest market package, the performance data of the comparison package and the experiment package when being fully opened or fully closed can be set and collected, so that more comprehensive and contrastive performance data can be obtained.
The customer feedback data comprises various feedback data which are provided by a user for evaluating the product in an online mode, a telephone mode and the like.
The accident data includes various online accidents or problems occurring in the evaluation product, and can be acquired from other systems or platforms through a data interface.
The invention can also provide gray test evaluation for a certain product, so the application data can also be various monitoring data during gray test.
Because different types of application data have different evaluation indexes, the collected application data are classified by the preprocessing module 2, corresponding evaluation index data are obtained according to the different types of application data, and the current health data of the product are obtained by the analysis module 3 according to the health evaluation strategy matched with the evaluation indexes.
Blast data
Specifically, when the application data is blast data, the preprocessing module 2 is configured to analyze the blast data according to a plurality of preset classification dimensions to obtain corresponding classification dimension index data of the blast data as evaluation index data. The classification dimension comprises a Crash appearance stage, an SDK type, a Crash basic type, a Crash appearing page and a Crash state. Wherein, the Crash appearance phase refers to whether the current Crash is the current latest version or the previous version of the product. In the dimension of the "Crash appearance stage", in one embodiment, Crash occurring for the first time in the current latest version is set as "new", Crash occurring in other versions for the last N days is set as "recent", and Crash occurring in other versions before N days is set as "history". The Crash is classified through the dimension of the Crash appearance stage, whether the Crash is a newly added problem or not can be quickly determined, so that the historical problem of repeated analysis is avoided, and the analysis efficiency is further improved.
The classification dimension index of "SDK type" is used to represent a corresponding service module of a service line where Crash occurs, for example, for an enterprise or a company, products of the enterprise or the company include various applications of different types, such as applications of advertisement type, live broadcast type, reading type, and the like, so that a person in charge of a corresponding service department can quickly follow up when viewing various health data.
The classification dimension index of "Crash basic type" includes OOM (out of memory)/non-OOM, system Native/non-system Native, Java or creating thread to distinguish the type of Crash.
The classification dimension index of "Crash appeared page" is used for marking the page type of Crash occurrence, such as game page, advertisement page, application service page, etc.
The classification dimension index of the 'Crash state' is used for marking whether the Crash creates defects according to the Crash and the processing state of the created defects, such as the processed state, the to-be-processed state and the like.
And classifying and marking a piece of Crash data according to the classification dimension, thereby obtaining corresponding classification dimension index data. As shown in fig. 2, a diagram of detailed information of the corresponding portion Crash data is shown. In fig. 2, the viewed classification dimension may be selected, and the corresponding classification type and general Crash data are displayed in the Crash data table according to the selected classification dimension. For example, the first Crash data with an ID of 92906286 has a "recent" appearance phase, meaning that the Crash occurs in other versions within N days. The defect state is 'to be built', namely the defect is not built yet, and compared with the next Crash data, the state is 'repaired'. In addition, in this embodiment, an interface for creating/checking a defect is reserved, through which a defect can be created and the state of the created defect can be checked, as shown in fig. 3, a title is created for the defect, and after creation, an ID number is automatically generated.
Correspondingly, the analysis module is configured to count the corresponding classification dimension index data of the Crash data according to the statistical index to obtain statistical data of a corresponding dimension, and the statistical data is used as health data. The statistical indexes can be various and have different statistical indexes according to different classification dimensions. For example, all Crash data are simply counted, and some corresponding general statistical indexes are as follows: the number of users, the user proportion, the number of times of Crash occurrence, the trend compared with the previous statistics, etc.
In addition, Crash data corresponding to a certain classification index or a defect created from the Crash data may be counted with the classification index as a statistical index. As shown in fig. 4, all Crash of the selected version is acquired, and Crash increment data for the occurrence of defects in each hour within 48 hours is counted with the defects as a statistical unit. When Crash suddenly increases on the line, the specific Crash with problems can be quickly judged through statistical data.
The Crash can also be sorted according to the occurrence frequency, and during statistics, corresponding indexes are calculated, such as proportion, fluctuation comparison with the previous day, and the like, as shown in fig. 5, for the Crash sorting table provided in this embodiment, the sorting number can be determined as required, for example, 20-200. In sorting, a specific classification dimension index may be selected for sorting. In the example shown in fig. 5, by selecting "non-Native" in the "basic Crash type" index, the problem of non-systematic Native can be screened out, and by calculating the increment data of each day Crash, the abnormally changed Crash can be identified, so that the rapid positioning is convenient.
Since each service party names a thread when starting the thread, abnormal service use behaviors can be identified by counting thread creation conditions during Crash, as shown in FIG. 6, current Crash data is counted by selecting a classification dimension index of 'create thread' and 'SDK' in the classification dimension index of 'Crash basic type', for example, Crash frequency, thread number and a corresponding thread list corresponding to a certain service module are generated, and detailed information of the current thread is recorded in the thread list.
Furthermore, the invention provides a monitoring alarm function for some scenes, namely, the invention also comprises a monitoring alarm module which is used for monitoring corresponding statistical data according to the monitoring indexes of corresponding evaluation types and giving an alarm when corresponding alarm conditions are reached. Such as Crash alarms, accident alarms, alarms that feed back problems, etc. Specifically, in one embodiment, the present invention monitors a single Crash, in one case, with hours as a unit of monitoring time, counts and calculates a Crash hour ring ratio increment, compares it to a threshold, and alarms if the Crash hour ring ratio increment exceeds the threshold. FIG. 7 shows an alarm notification and a 48-hour statistical chart for the same. In the alarm notification, the corresponding Crash ID and the defect (or problem) thereof are recorded, and the online of the new version, the plug-in change and the configuration change can be better monitored by the monitoring method. In another case, taking a day as a monitoring time unit, counting the total number of times of a Crash, comparing the total number of times of the Crash with a threshold value, and alarming when the total number of times of the Crash exceeds the corresponding threshold value. As shown in fig. 8, the alarm notification is a notification when the total number of times of one Crash exceeds the corresponding threshold, and the notification describes the corresponding Crash ID and the related information. In another case, counting the geometric increase quantity of the Crash times of one hour of a certain version by taking the hour as a monitoring unit, comparing the geometric increase quantity with a corresponding threshold value, and alarming if the geometric increase quantity exceeds the threshold value. As shown in fig. 9, the alarm notification is an alarm that alarms when the number of increases exceeds the threshold. The corresponding Crash ID and related information, such as a threshold, specific parity data, etc., are recorded in the notification.
Since Crash occurs to some extent, in this embodiment, the defect management module 4 is further included, and the defect may be created by other systems or the present system. The Crash state in the classification dimension index as described above may determine whether a defect is created from the Crash. When a defect is created, the corresponding service and the module thereof can be determined according to the SDK generated by Crash, so that the attribution and the responsible person of the corresponding module are associated in the defect, and the related description information is created for the defect. Creating windows is shown in fig. 3, and creating interfaces are shown in fig. 2 and 5. The defect management module 4 periodically synchronizes the state of the defect, such as the state to be repaired and the repaired state in fig. 2, according to the processing information of the defect, periodically counts the defect, and generates statistical data of the defect so as to view, for example, by using the item "defect" in fig. 2 and 5, the specific defect information can be viewed, and also all statistical data of the defect within a period of time can be viewed. The ID, the number of occurrences, the responsible person, the corresponding service or service module, the latest occurrence time, and the like of each defect can be marked in a list form, and the number of defects to be solved by a specific responsible person can be shown in a chart (column, pie) form. The statistical data may be presented in various ways, and is not limited to the form described in this embodiment.
In another embodiment, the application data collection module further obtains event link information which can cause Crash to occur, wherein the event link information comprises event type, event content and occurrence time. As shown in fig. 10, the statistical information of the acquired event information is shown.
Performance index data
When the application data collection module 1 collects performance index data of each product and each version according to the performance type. In order to obtain more comprehensive performance data, the application data collection module 1 respectively collects performance index data of an application market packet, a comparison packet and an experiment packet thereof, wherein the experiment packet comprises one or more new function modules relative to the comparison packet, and controls which modules are collected through a switch arranged at a server end. For example, when a switch for controlling a certain function is turned on at the server side, the functional module is in an operating state, and corresponding performance data can be collected at this time, and when the switch for controlling a certain function is turned off at the server side, the functional module is not operated, and corresponding performance data cannot be collected at this time.
The preprocessing module 2 classifies the performance index data according to the types of the application packages to obtain the performance index data of different application packages, such as the performance index data of a comparison package, the performance index data of an experiment package, and the like.
The analysis module 3 obtains the statistical data of the single performance index and the comparison data of different corresponding packets according to the preset statistical index. The statistical indexes comprise standard reaching rate of performance indexes, trends of the performance indexes and the like. The method presets corresponding baseline threshold values for different performance indexes according to specific conditions, and compares each performance index data with the corresponding baseline threshold value when statistics is carried out; and calculating the standard reaching rate and the trend of each performance index. As shown in fig. 11, the upper half is a statistical data graph for each performance index on a certain day, and the lower half is a statistical data graph for a cold start index over a period of time.
For the cold start index, the time consumption of each module in the cold start process is also included, and correspondingly, in the statistics, only the time consumption data of a plurality of cold start modules of a single application package may be counted, as shown in fig. 12; the comparison data of the time consumed by each module in the same cold start process of the control packet and the experiment packet thereof can also be counted, as shown in fig. 13, the comparison data of the time consumed by each module in the cold start process of the terminals of different models is included in fig. 13.
Customer feedback data
The application data collected by the application data collecting module 1 is various types of client feedback data, such as a client question, a reflected question, and the like, for example, a client feedback question table shown in fig. 14. Correspondingly, the preprocessing module 2 determines specific individual feedback problems and classifications thereof according to various types of client feedback data. In fig. 14, the preprocessing module 2 can help classify the problem by segmenting the client feedback data, such as determining the large category first, then determining the sub-categories belonging to the large category, and so on. The large category may be specific services in a service, such as advertisement and game, live broadcast, or specific activities in a certain service, such as gold coin activities. The analysis module 3 obtains the feedback problem quantity and the trend of the corresponding index according to the preset statistical index. For example, the overall trend of customer service feedback data changes, and the trend of single problems changes, and the number of feedback of customers is monitored. For example, when a certain problem or a certain class of problems occurs for 3 consecutive days, the class of problems is classified as a heat-sensitive problem and a heat-sensitive label is set thereto, and since the class of problems is more general than others, in order to monitor whether the class of problems is effectively solved, a specific statistical time limit is set thereto, such as once a day. FIG. 15 is a statistical chart of the "page stuck/flashed" class of problems.
And the feedback quantity/day is more than or equal to 300; or compared with the previous day, the feedback quantity is more than or equal to 60; or the number of newly added customer service feedback on the day is more than or equal to 50, the feedback problems of the customers meeting the indexes are determined as high-sensitivity problems, and hot spot labels are determined for the feedback problems. By setting these thresholds, when a feedback problem is greater than these thresholds, it is shown that such a problem is a serious problem and needs to be dealt with in a timely manner. To monitor such problems, shorter statistical and monitoring times are set, for example, in hours and 10 minutes. An alarm is issued if the amount of feedback monitoring this type of high sensitivity problem still exceeds a threshold. Such as the alarm messages shown in fig. 16A-16B. And aiming at the alarm information, customer service personnel create defects in the special module. As can be seen from the foregoing, the defect management module 4 of the present invention can count and track defects, thereby facilitating the solution of the problem.
Accident data
When the application data collected by the application data collection module 1 is accident data, the preprocessing module 2 classifies the accident data to obtain different dimensions according to different accident data, for example, according to the importance level in the accident data, and according to the time of the accident. And the analysis module 3 counts the collected accident data according to different accident statistical dimensions to obtain accident statistical data with corresponding dimensions. For example, the total number of accidents is counted in a fixed statistical time period, such as the total number of accidents occurring each day and the difference and the increasing trend compared with the previous day, the total number of accidents occurring each month and the difference and the increasing trend compared with the previous month; or the number of occurrences of the same accident; or the maximum number of days without accidents in a fixed time period, such as the maximum number of days without accidents in the current month; the maximum number of days without accidents in the first half year, etc.
The defect management module 4 of the present invention counts defects created according to an accident, and issues a reminder in the form of a mail, a message, or the like when the defects are still unresolved within a predetermined time, as shown in fig. 17.
Gray scale admittance monitoring data
When the application data collection module 1 collects monitoring data of an application gray level package according to a gray level entry type, the preprocessing module 2 divides the monitoring data into classified data of three dimensions of Crash data, performance data and core first-level index data according to the monitoring type. The analysis module 3 counts the Crash data according to the Crash statistical index; and acquiring the abnormal performance index and the abnormal core first-level index of the application gray level package according to the corresponding reference index data.
Specifically, when the gray monitoring data is the Crash data of a gray package, analyzing the Crash data from multiple dimensions to obtain Crash index data of multiple dimensions, wherein the dimensions comprise basic types, such as system Native/non-system Native, OOM/non-OOM, Java and the like; and counting the Crash index data of the multiple dimensions to obtain Crash statistical data of corresponding dimensions. The statistical data comprises the number of Crash users, user proportion, Crash times, number proportion, Crash number, Crash time percentage, Crash rate and the like, and also comprises basic information of the application gray level package, such as development responsibility, creation time, gray level time, version number and the like.
And monitoring the Crash statistical data, and sending alarm data when the Crash times or Crash rate of a certain Crash exceeds a threshold value in a preset time so as to improve the gray level packet in time.
Alarming when adding Crash data in the gray level package, creating defects according to the Crash data, and counting and monitoring by the defect management module 4.
The gray monitoring data also comprises plug-in package data, the plug-in package is monitored in the process, namely, the gray time of the plug-in package is monitored, and a reminding message is sent after the preset gray time threshold value is met, such as 8 hours. And after receiving the reminding message, the staff detects whether the plug-in has a problem or not and whether the plug-in has a change according to the problem or not. If there are no problems within 8 hours, the plug-in may be incorporated into the master version of the application. If there is a change, then 8 hours of monitoring is required.
The gray scale monitoring data for the application gray scale package also includes performance index data. And counting the performance data of the gray level packet on the current day by taking the day as a counting time period, and determining whether the gray level packet is abnormal or not according to the reference value of the corresponding performance index.
The gray level monitoring data of the application gray level package also comprises a plurality of core first-level index data, wherein in order to obtain whether the core first-level index data of the current gray level package reaches the design index, when the performance data is collected, the core first-level index data of the gray level package and the reference package are collected at the same time, and the core first-level index data is suitable for the application, such as the per-capita use time, the secondary retention rate, the per-capita reading time and the like. As shown in fig. 18, the data is compared with the core primary index data of the reference packet for the counted gray level packet. According to the core first-level index data comparing the current gray-scale version with the reference version, the abnormal index data is displayed in a floating mode, and the function that the indexes are seriously affected is conveniently identified.
In order to perform corresponding processing on the various types of application data to obtain corresponding health indicators, in one embodiment, the evaluation system provided by the present invention includes modules that can perform various functions, such as: the system comprises a Crash evaluation module 10, a performance evaluation module 20, a customer service feedback module 30, an accident evaluation module 40 and a gray scale admission evaluation module 40. As shown in fig. 19, a functional block diagram of a Crash evaluation module is shown, wherein the Crash evaluation module 10 is configured to include one or more of the following units: a classification unit 11, a statistical unit 12, a monitoring unit 13, a trace management unit 14 and an event analysis unit 15. The classification unit 11 analyzes collected Crash data of corresponding products according to a plurality of preset classification dimensions to obtain corresponding classification dimension index data of the Crash data; for example, "Crash appearance phase", such as new addition, near term, history; such as advertising, live broadcasting, etc.; "Crash basic type", such as OOM/non-OOM, system Native/non-Native or create thread; "page where Crash appears", such as advertisement page, application service page; "Crash state" such as established defect, unestablished, treated, untreated, etc.
The statistical unit 12 performs statistics on the corresponding classification dimension index data of the Crash data according to the corresponding statistical index to obtain statistical data of the corresponding dimension. For example, according to general statistical indicators such as: counting Crash or defect counting correspondingly established according to Crash, such as the number of users, user proportion, Crash occurrence frequency, trend compared with the previous counting and the like; or counting the number of threads according to the SDK type; or counting the Crash increment data aiming at a certain defect in each hour within 48 hours by taking the defect as a statistical unit, and the like; or sorting Crash.
The monitoring unit 13 monitors the statistical data of the corresponding dimension according to the corresponding monitoring index, and gives an alarm when the alarm condition is reached. And monitoring each Crash, taking hours as a monitoring time unit, counting and calculating a Crash hour ring ratio increment, comparing the Crash hour ring ratio increment with a threshold value, and alarming if the Crash hour ring ratio increment exceeds the threshold value. Or taking the day as a monitoring time unit, counting the total number of times of one Crash, comparing the total number of times of one Crash with a threshold value, and alarming when the total number of times of one Crash exceeds the corresponding threshold value; or counting the geometric increase quantity of the Crash times of one hour of a certain version by taking the hour as a monitoring unit, comparing the geometric increase quantity with a corresponding threshold value, and alarming if the geometric increase quantity exceeds the threshold value.
The trace management unit 14 is used to trace manage the created defects. The defects are created by a person in a dedicated module, the status of the created defects, e.g., processed or unprocessed, is monitored, and the status of the defects is periodically synchronized to generate statistics of the defects for review.
The event analysis unit 15 is configured to obtain an event that may cause Crash to occur, and generate an event link according to the service association relationship. The event link may be an event sequence generated in a time sequence, or may also be a call link generated in an event call relationship. And statistics is carried out on event link information such as event type, event content and occurrence time.
As shown in fig. 20, the performance evaluation module 20 includes one or more of the following units: a monitoring unit 21, a statistical unit 22 and a testing unit 23. The monitoring unit 21 is configured to monitor corresponding performance index data of different application packages according to corresponding baseline thresholds, and determine a standard reaching rate and a trend of each performance index. Such as the achievement rate and trend of cold start time, hot start time, or page load time.
The statistical unit 22 is used for counting the performance index data and the comparison data of different application packages. For example, the comparison data of various performance indexes of different versions of application packages (such as experiment packages, reference packages, latest version market packages, and the like) of terminals of different brands and different models.
The test unit 23 is configured to test the cold start process of at least two comparison application packages to obtain cold start comparison data of the comparison application packages. For the cold start performance data, in addition to the use time of the whole process of the cold start, the use time of each module in the cold start process is also included, and more detailed performance data can be obtained through the refinement of the cold start and the comparison with a reference packet and the like.
As shown in fig. 21, the customer service feedback module 30 includes one or more of the following elements: a statistic module 31, a process supervision module 32, a hot spot processing unit 33 and a high sensitivity processing unit 34. The statistical module 31 is configured to classify and count various types of collected customer feedback data to obtain a total customer feedback amount and a trend, and a number and a trend of a single feedback problem. For example, the category of the feedback question, the overall trend change of the customer service feedback data, and the trend change of the individual question are determined. The process supervision module 32 is configured to determine a feedback problem meeting a supervision index according to the statistical data, and determine a corresponding supervision label for the feedback problem; the supervision indexes at least comprise heat-sensitive indexes and high-sensitivity indexes, and the corresponding supervision labels are at least hot spot labels and high-sensitivity labels. For example, when a certain problem or class of problems occurs for 3 consecutive days, the class of problems is classified as a heat-sensitive problem. If the feedback quantity/day is more than or equal to 300; or compared with the previous day, the feedback quantity is more than or equal to 60; or the number of newly added customer service feedback on the day is more than or equal to 50, the feedback problems of the customers meeting the indexes are determined as high-sensitivity problems, and hot spot labels are determined for the feedback problems. Correspondingly, the heat-sensitive processing unit 33 is configured to count the problem amount and the trend of the hot spot label with a preset heat-sensitive dimension counting time period. Such as the problem of counting hot tags once per day. The high-sensitivity processing unit 34 is used for counting the problem quantity and the trend of the high-sensitivity label in a preset high-sensitivity dimension statistical time period. For example, once per hour. And monitoring whether the statistical feedback problem quantity exceeds a threshold value, and sending alarm information if the statistical feedback problem quantity exceeds the threshold value. In addition, when a defect is created according to the customer service feedback problem, the trace management may be performed by the trace management unit 14 in the blast evaluation module 10.
As shown in fig. 22, the incident evaluation module 40 includes one or more of the following elements: an accident statistics unit 41 and an accident monitoring unit 42. The accident statistics unit 41 is configured to count the collected accident data according to different accident statistics dimensions to obtain accident statistics data of corresponding dimensions. For example, the total number of accidents is counted in a fixed statistical time period, such as the total number of accidents occurring each day and the difference and the increasing trend compared with the previous day, the total number of accidents occurring each month and the difference and the increasing trend compared with the previous month; or the number of occurrences of the same accident; or the maximum number of days without accidents in a fixed time period, such as the maximum number of days without accidents in the current month; the maximum number of days without accidents in the first half year, etc. The accident monitoring unit 42 is configured to monitor and/or alarm the accident data and/or the accident statistic in accordance with an accident monitoring indicator. For example, when an accident occurs and a problem is created in a special module, the problem is monitored and reminded according to a reminding time limit, information of problem solution or problem unsolved information is synchronized in real time or periodically, and the problem is still unsolved and a mail reminding is sent within a preset time, such as more than 2 days.
As shown in fig. 23, the grayscale admission evaluation module 50 includes one or more of the following units: a blast data unit 51, a defect monitoring unit 52, a plug-in package monitoring unit 53, a performance monitoring unit 54, and a core index unit 55. The Crash data unit 51 is configured to count the collected grayscale packet Crash data according to the grayscale statistic index. For example, the number of Crash users, the user proportion, the Crash frequency, the frequency proportion, the number of Crash, the Crash frequency percentage, the Crash rate, and the like are counted, and the basic information of the application gray package, such as the development responsibility, the creation time, the gray time, the version number, and the like, is also included.
The defect monitoring unit 52 is configured to follow up monitoring defects created according to the newly added Crash data, and functions as the tracking management unit 14 in the Crash evaluation module 10, so that the defect monitoring unit 52 may also be the tracking management unit 14.
The plug-in package monitoring unit 53 is configured to obtain and count plug-in package monitoring data according to the gray monitoring index. Such as gray-scale time, change data, etc. for the plug-in package.
The performance monitoring unit 54 is configured to count performance data of the grayscale packets, and determine an abnormal performance index according to a reference value corresponding to the performance index. For example, the cold start P90 data is counted and the difference from the reference value is calculated. The core first-level index unit 55 is used for counting core index comparison data of the gray level packet and the reference packet, and determining an abnormal core index according to the reference packet data.
The modules can be correspondingly processed according to the selected product to obtain the health data of the corresponding aspect of the product, and each module can independently form an evaluation report according to the respective data of the module, and can also be integrated together to form a complete health report. As shown in fig. 24, the overall display data of the system according to the present invention includes the partial health data of the remaining 4 modules except the grayscale admission evaluation module 50. Therefore, the product health assessment system and the method provided by the invention can assess an internet product from multiple aspects and multiple dimensions, so that the health state of the product can be accurately and specifically known, and reliable data support is provided for improvement and problem solving of the product.
The above embodiments are provided only for illustrating the present invention and not for limiting the present invention, and those skilled in the art can make various changes and modifications without departing from the scope of the present invention, and therefore, all equivalent technical solutions should fall within the scope of the present invention.
Claims (24)
1. A product health assessment method comprising:
collecting product application data of corresponding types according to the evaluation types;
classifying the application data to obtain corresponding evaluation index data; and
analyzing the assessment index data according to a health assessment strategy matched with the assessment index to obtain health data of the product.
2. The method of claim 1, wherein the evaluation type is a Crash type and the application data is Crash data, further comprising: analyzing the Crash data according to a plurality of preset classification dimensions to obtain a plurality of classification dimension index data of the Crash data as evaluation index data.
3. The method of claim 2, wherein the health assessment policy comprises: and counting the corresponding classification dimension index data according to the statistical indexes to obtain statistical data.
4. The method of claim 3, wherein the statistics include one or more of single Crash statistics, Crash statistics of the same defect, Crash statistics of different defects, Crash statistics of different traffic modules, Crash or defect ordering, and/or thread statistics.
5. The method of claim 4, further comprising: the tracking manages the defects that have been created.
6. The method of claim 3, further comprising: and monitoring corresponding statistical data according to the monitoring indexes, and giving an alarm when an alarm condition is reached.
7. The method of claim 1, further comprising: event information which can cause Crash to occur is obtained, and an event link is generated according to the event information.
8. The method of claim 1, wherein the assessment type is a performance type, the application data is performance indicator data for a specified product and/or a specified version; the step of obtaining health data for the product further comprises:
comparing the performance index data with a baseline threshold corresponding thereto; and
and calculating the standard reaching rate and the trend of the performance index.
9. The method of claim 8, wherein in collecting the corresponding types of application data, collecting performance indicator data for an experimental package and a corresponding control package, respectively, wherein the experimental package includes one or more new functional modules relative to the control package; the step of obtaining product health data further comprises:
and counting one or more performance index data, standard reaching rate and trend data of the experiment package and the comparison package thereof.
10. The method of claim 8, wherein the performance indicator data comprises a cold start indicator and a plurality of cold start module elapsed times during a cold start, the corresponding health data comprising: time consumption data of a plurality of cold start modules of the experiment package; and comparing the time-consuming cold start modules of the experimental package and the control package thereof.
11. The method of claim 1, wherein the assessment type is a customer service type, and the application data is one or more types of customer feedback data; the evaluation indexes comprise total feedback quantity and single-type feedback problem quantity;
correspondingly, the step of processing the application data to obtain corresponding evaluation index data comprises:
counting the total amount of more than one type of customer feedback data in a preset counting time period to obtain the feedback total amount;
determining the feedback problem quantity of one or more types of feedback problems and single type of feedback problems in a preset statistical time period from more than one type of client feedback data;
correspondingly, the step of obtaining product health data comprises:
and obtaining feedback total amount trend data and single-type feedback problem trend data according to the feedback total amounts of a plurality of preset statistical time periods.
12. The method of claim 11, further comprising:
responding to one or more types of feedback problems to accord with preset hotspot indexes, and determining hotspot labels for the feedback problems; and
counting the problem quantity and the trend of the hotspot label in a preset hotspot dimension counting time period; and/or
Responding to one or more types of feedback problems to accord with preset high-sensitivity indexes, and determining a high-sensitivity label for the feedback problems;
counting the problem quantity and the trend of the high-sensitivity label in a preset high-sensitivity dimension counting time period; and
and responding to the problem quantity of the high-sensitivity label, and alarming when the problem quantity exceeds a threshold value within the high-sensitivity dimension statistical time period.
13. The method of claim 12, further comprising:
defects are created from alarms that respond to high sensitivity tags with a quantity of problems exceeding a threshold and associated with the relevant module and the responsible person.
14. The method of claim 1, wherein the assessment type is an incident type and the collected application data is incident data; further comprising: and counting the collected accident data according to different accident counting dimensions to obtain accident counting data of corresponding dimensions.
15. The method of claim 14, further comprising: and monitoring the accident statistical data and/or the accident data according to accident monitoring indexes, and responding to the fact that the accident statistical data and/or the accident data meet alarm conditions to give an alarm.
16. The method of claim 1, wherein the evaluation type is a gray scale admission type, the application data comprises monitoring data of an application gray scale package; further comprising: classifying the monitoring data of the gray level packet to obtain classified data; the classification data comprises one or more of Crash type monitoring data, plug-in package data, performance data and core primary index data.
17. The method of claim 16, wherein when the classification data is blast-type monitoring data, further comprising: classifying the monitoring data of the Crash type according to a plurality of preset dimensions, and carrying out classified statistics on the monitoring data of the Crash type to obtain statistical data of corresponding types.
18. The method of claim 17, further comprising: and creating defects according to the newly added Crash data, and monitoring the defects.
19. The method of claim 16, wherein when the classification data is plug-in package data, further comprising: monitoring the gray time and the change data of the plug-in package;
when the classification data is the performance index data of the application gray level package, the method further comprises the following steps: counting performance data and determining abnormal performance indexes;
when the application data comprises core level one index data, the method further comprises the following steps: and acquiring core first-level index comparison data of the comparison packet, and determining abnormal core first-level indexes.
20. A product health assessment system comprising a Crash assessment module configured to include one or more of the following:
the classification unit is configured to analyze the collected Crash data of the corresponding product according to a plurality of preset classification dimensions to obtain corresponding classification dimension index data of the Crash data;
a statistical unit configured to count one or more classification dimension index data of the Crash data according to the statistical index to obtain statistical data;
a monitoring unit configured to monitor the statistical data according to the monitoring index, and to alarm when an alarm condition is reached;
a trace management unit configured to trace manage the created defects; and
an event analysis unit; configured to acquire events that may cause Crash to occur and generate a link of events in a service association or chronological order.
21. The evaluation system of claim 20, further comprising a performance evaluation module configured to include one or more of the following:
a monitoring unit configured to monitor the collected performance index data according to a performance index data baseline threshold, and determine an achievement rate and a trend of the performance index;
the statistical unit is configured to count the comparison data of each performance index of different application packages; and
a test unit configured to test a cold start process of at least two application packages to obtain cold start comparison data of the two application packages.
22. The evaluation system of claim 20, further comprising a customer service feedback module configured to include one or more of the following:
the statistical module is configured to classify and count various types of collected customer feedback data to obtain the total customer feedback amount and trend and the number and trend statistical data of single type feedback problems;
a process supervision module configured to determine a feedback problem meeting a supervision index according to the statistical data and determine a corresponding supervision label for the feedback problem; the supervision indexes at least comprise hot spot indexes and high-sensitivity indexes, and the corresponding supervision labels are hot spot labels and high-sensitivity labels;
the hot spot processing unit is configured to count the problem amount and the trend of the hot spot label in a preset hot spot dimension counting time period; and
the high-sensitivity processing unit is configured to count the problem quantity and the trend of the high-sensitivity label by a preset high-sensitivity dimension counting time period; and responding to the problem quantity of the high-sensitivity label, and alarming when the problem quantity exceeds a threshold value within the high-sensitivity dimension statistical time period.
23. The evaluation system of claim 20, further comprising an incident evaluation module configured to include one or more of the following:
the accident statistic unit is configured to count the collected accident data according to different accident statistic dimensions to obtain accident statistic data;
and the accident monitoring unit is configured to monitor the accident data and/or the accident statistical data according to an accident monitoring index, and alarm when the accident data and/or the accident statistical data meet an alarm condition.
24. The evaluation system of claim 20, further comprising a grayscale admission evaluation module configured to include one or more of the following units:
a Crash data unit configured to count the collected gray level packet Crash data according to the gray level statistical index;
a defect monitoring unit configured to follow up monitoring a defect created according to the newly added blast data;
a plug-in package monitoring unit; configured to obtain and count plugin package monitoring data according to the gray level monitoring index;
the performance monitoring unit is configured to count performance data of the gray level packet and determine an abnormal performance index according to a reference value corresponding to the performance index; and
and the core primary index unit is configured to count the core primary index comparison data of the gray level packet and the reference packet, and determine the abnormal core primary index.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011018489.7A CN112527611A (en) | 2020-09-24 | 2020-09-24 | Product health degree assessment method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011018489.7A CN112527611A (en) | 2020-09-24 | 2020-09-24 | Product health degree assessment method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112527611A true CN112527611A (en) | 2021-03-19 |
Family
ID=74980336
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011018489.7A Pending CN112527611A (en) | 2020-09-24 | 2020-09-24 | Product health degree assessment method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527611A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115357468A (en) * | 2022-10-19 | 2022-11-18 | 广东省科技基础条件平台中心 | Software performance evaluation method, device, equipment and storage medium |
Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902442A (en) * | 2012-12-25 | 2014-07-02 | 中国移动通信集团公司 | Method and system for evaluating cloud software health degree |
| CN104657278A (en) * | 2015-03-13 | 2015-05-27 | 百度在线网络技术（北京）有限公司 | Client side performance evaluation method and system |
| CN104679655A (en) * | 2015-03-06 | 2015-06-03 | 国网四川省电力公司电力科学研究院 | Software system state evaluation method based on parameter correlation |
| CN104993946A (en) * | 2015-06-01 | 2015-10-21 | 华为技术有限公司 | Evaluation method, device and system for gated launch |
| CN105095747A (en) * | 2014-04-15 | 2015-11-25 | 阿里巴巴集团控股有限公司 | Method and system for assessing health degree of Java application |
| CN106373014A (en) * | 2016-08-24 | 2017-02-01 | 腾讯科技（深圳）有限公司 | Method and apparatus for assessing health degree of application |
| CN106453549A (en) * | 2016-10-08 | 2017-02-22 | 中国工商银行股份有限公司 | Distributed software version release method and system thereof |
| CN106878965A (en) * | 2015-12-14 | 2017-06-20 | 中国移动通信集团浙江有限公司 | A kind of method and apparatus for assessing mobile terminal performance |
| CN107656872A (en) * | 2017-10-09 | 2018-02-02 | 平安普惠企业管理有限公司 | Method for testing software, device, equipment and computer-readable storage medium |
| CN107943677A (en) * | 2017-10-13 | 2018-04-20 | 东软集团股份有限公司 | Application performance monitoring method, device, readable storage medium storing program for executing and electronic equipment |
| CN108829597A (en) * | 2018-06-13 | 2018-11-16 | 中国平安人寿保险股份有限公司 | Software crowd surveys method and device, computer installation and readable storage medium storing program for executing |
| US20190041457A1 (en) * | 2018-03-12 | 2019-02-07 | GAVS Technologies Pvt. Ltd. | Method and system of determining application health in an information technology environment |
| CN110413513A (en) * | 2019-07-11 | 2019-11-05 | 北京百度网讯科技有限公司 | Publication quality control system, method and the device of software |
| CN110580214A (en) * | 2018-06-07 | 2019-12-17 | 百度在线网络技术（北京）有限公司 | application program test evaluation method and device, storage medium and terminal equipment |
| CN110597699A (en) * | 2019-08-21 | 2019-12-20 | 珠海国津软件科技有限公司 | Service evaluation method |
| CN110928764A (en) * | 2019-10-10 | 2020-03-27 | 中国人民解放军陆军工程大学 | Automated mobile application crowdsourcing test report evaluation method and computer storage medium |
| US10613525B1 (en) * | 2017-07-26 | 2020-04-07 | Numerify, Inc. | Automated health assessment and outage prediction system |
| CN111552607A (en) * | 2020-03-27 | 2020-08-18 | 深圳壹账通智能科技有限公司 | Health evaluation method, device and equipment of application program and storage medium |
-
2020
- 2020-09-24 CN CN202011018489.7A patent/CN112527611A/en active Pending
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902442A (en) * | 2012-12-25 | 2014-07-02 | 中国移动通信集团公司 | Method and system for evaluating cloud software health degree |
| CN105095747A (en) * | 2014-04-15 | 2015-11-25 | 阿里巴巴集团控股有限公司 | Method and system for assessing health degree of Java application |
| CN104679655A (en) * | 2015-03-06 | 2015-06-03 | 国网四川省电力公司电力科学研究院 | Software system state evaluation method based on parameter correlation |
| CN104657278A (en) * | 2015-03-13 | 2015-05-27 | 百度在线网络技术（北京）有限公司 | Client side performance evaluation method and system |
| CN104993946A (en) * | 2015-06-01 | 2015-10-21 | 华为技术有限公司 | Evaluation method, device and system for gated launch |
| CN106878965A (en) * | 2015-12-14 | 2017-06-20 | 中国移动通信集团浙江有限公司 | A kind of method and apparatus for assessing mobile terminal performance |
| CN106373014A (en) * | 2016-08-24 | 2017-02-01 | 腾讯科技（深圳）有限公司 | Method and apparatus for assessing health degree of application |
| CN106453549A (en) * | 2016-10-08 | 2017-02-22 | 中国工商银行股份有限公司 | Distributed software version release method and system thereof |
| US10613525B1 (en) * | 2017-07-26 | 2020-04-07 | Numerify, Inc. | Automated health assessment and outage prediction system |
| CN107656872A (en) * | 2017-10-09 | 2018-02-02 | 平安普惠企业管理有限公司 | Method for testing software, device, equipment and computer-readable storage medium |
| CN107943677A (en) * | 2017-10-13 | 2018-04-20 | 东软集团股份有限公司 | Application performance monitoring method, device, readable storage medium storing program for executing and electronic equipment |
| US20190041457A1 (en) * | 2018-03-12 | 2019-02-07 | GAVS Technologies Pvt. Ltd. | Method and system of determining application health in an information technology environment |
| CN110580214A (en) * | 2018-06-07 | 2019-12-17 | 百度在线网络技术（北京）有限公司 | application program test evaluation method and device, storage medium and terminal equipment |
| CN108829597A (en) * | 2018-06-13 | 2018-11-16 | 中国平安人寿保险股份有限公司 | Software crowd surveys method and device, computer installation and readable storage medium storing program for executing |
| CN110413513A (en) * | 2019-07-11 | 2019-11-05 | 北京百度网讯科技有限公司 | Publication quality control system, method and the device of software |
| CN110597699A (en) * | 2019-08-21 | 2019-12-20 | 珠海国津软件科技有限公司 | Service evaluation method |
| CN110928764A (en) * | 2019-10-10 | 2020-03-27 | 中国人民解放军陆军工程大学 | Automated mobile application crowdsourcing test report evaluation method and computer storage medium |
| CN111552607A (en) * | 2020-03-27 | 2020-08-18 | 深圳壹账通智能科技有限公司 | Health evaluation method, device and equipment of application program and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115357468A (en) * | 2022-10-19 | 2022-11-18 | 广东省科技基础条件平台中心 | Software performance evaluation method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10102056B1 (en) | Anomaly detection using machine learning | |
| CN110824270A (en) | Electricity stealing user identification method and device combining transformer area line loss and abnormal events | |
| US20050216793A1 (en) | Method and apparatus for detecting abnormal behavior of enterprise software applications | |
| US10802057B2 (en) | Systems and methods for monitoring a power system | |
| CN110728525B (en) | Delimitation processing method and device for network batch user complaints | |
| CN101925039A (en) | Prewarning method and device of billing ticket | |
| CN111984442A (en) | Method and device for detecting abnormality of computer cluster system, and storage medium | |
| CN111401973A (en) | Commodity demand prediction information prediction system and method under multiple influence factors | |
| CN116664238A (en) | Retail industry risk order auditing management method and system | |
| CN112527611A (en) | Product health degree assessment method and system | |
| CN114881305B (en) | Canteen prediction early warning system and prediction method thereof | |
| CN111080353B (en) | Product recommendation method, system and readable storage medium based on vehicle data | |
| US8543552B2 (en) | Detecting statistical variation from unclassified process log | |
| CN112036725A (en) | Electric energy meter fault identification method | |
| CN108664405B (en) | Automatic monitoring method and terminal based on funnel model | |
| CN114493238A (en) | Power supply service risk prediction method, system, storage medium and computer equipment | |
| CN111428963B (en) | Data processing method and device | |
| CN112306828A (en) | Product Crash data monitoring method and system | |
| CN115378143B (en) | Intelligent inspection system for electric power marketing | |
| CN110619572A (en) | Method for monitoring high fault tolerance growth of enterprise public data | |
| CN113469584B (en) | Risk management method and device for business service operation | |
| CN112306860A (en) | Product gray level packet access evaluation method and system | |
| CN115358336A (en) | Power utilization abnormity detection method and device and electronic equipment | |
| CN115018434A (en) | Remote operation and maintenance management system for new energy power station | |
| CN114971887A (en) | Credit system operation monitoring method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210319 |
|
| WD01 | Invention patent application deemed withdrawn after publication |