CN113485889B - Buried data verification method and device, electronic equipment and storage medium - Google Patents
Buried data verification method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113485889B CN113485889B CN202110733001.7A CN202110733001A CN113485889B CN 113485889 B CN113485889 B CN 113485889B CN 202110733001 A CN202110733001 A CN 202110733001A CN 113485889 B CN113485889 B CN 113485889B
- Authority
- CN
- China
- Prior art keywords
- verification
- data
- rule
- buried
- historical data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
- G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to data processing and discloses a buried data verification method, which comprises the following steps: acquiring first historical data corresponding to a target application program, acquiring second historical data corresponding to a control type in the target application program, and generating an initial verification rule based on the first and second historical data; generating a check rule configuration interface, wherein the check rule configuration interface comprises an initial check rule modification and/or confirmation area and a check rule supplementing area, and generating a rule set corresponding to the target application program based on rules configured by a user in the check rule configuration interface; when the target application program is monitored to generate buried point data, the generated buried point data is checked by utilizing the rule set, and a target check result is obtained. The invention also provides a buried data verification device, electronic equipment and a storage medium. The invention ensures the integrity of the verification rule and improves the accuracy of the verification result.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method and apparatus for verifying buried data, an electronic device, and a storage medium.
Background
Buried point data acquisition is an important data acquisition mode, and user behavior tracks can be analyzed and application program performance can be optimized through buried point data. Because of numerous sources, huge data volume and various formats of the buried data, the condition that the data is not standard inevitably exists, and in order to ensure the subsequent and smooth analysis of the buried data, the buried data needs to be checked first.
Currently, the verification rule is usually set manually, however, the manually set verification rule is not comprehensive enough, resulting in low accuracy of the verification result. Therefore, there is a need for a buried data verification method to ensure the integrity of the verification rule and improve the accuracy of the verification result.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a buried data verification method, which aims to ensure the integrity of the verification rule and improve the accuracy of the verification result.
The buried point data verification method provided by the invention comprises the following steps:
analyzing a buried data verification request aiming at a target application program and sent by a client to acquire a control type of a control in the target application program carried by the buried data verification request;
acquiring first historical data corresponding to the target application program in a first time period, acquiring second historical data corresponding to the control type, and generating an initial verification rule based on the first historical data and the second historical data;
Generating a check rule configuration interface, wherein the check rule configuration interface comprises an initial check rule modification and/or confirmation area and a check rule supplementing area, the check rule configuration interface is displayed on the client interface, and a rule set corresponding to the target application program is generated based on rules configured by a user on the check rule configuration interface;
when the target application program is monitored to generate buried point data, the generated buried point data is checked by utilizing the rule set to obtain a target check result, and the generated buried point data and the target check result are stored in a preset database.
Optionally, the first historical data carries first labeling information, the first labeling information includes success and failure of verification, and generating the initial verification rule based on the first historical data and the second historical data includes:
determining index values of preset index items corresponding to each field and/or control in the target application program based on the first marking information in the first historical data as buried point data which is successfully verified, and generating a first verification rule corresponding to the target application program based on the index values;
Acquiring third historical data corresponding to the target application program in a second time period, wherein the third historical data carries second labeling information, constructing a verification model based on the first historical data, verifying the third historical data by adopting the verification model, and generating a second verification rule corresponding to the target application program based on an initial verification result output by the verification model and the second labeling information;
acquiring an original check rule corresponding to each piece of buried point data in the second historical data, and taking a designated rule as a third check rule corresponding to the target application program if the ratio of the number of the buried point data corresponding to the designated rule in the original check rule to the total number of the buried point data in the second historical data is larger than a first threshold;
and taking the set of the first, second and third check rules as initial check rules corresponding to the target application program.
Optionally, the building a verification model based on the first historical data includes:
extracting a sample set from the first historical data, and splitting the sample set into a training set, a verification set and a test set;
establishing an initial model by using the training set, and adjusting model parameters of the initial model by using the verification set to obtain an updated model;
And calculating the performance index value of the updated model by using the test set, and taking the updated model as a verification model when the performance index value is greater than a second threshold value.
Optionally, the extracting the sample set from the first historical data includes:
obtaining the corresponding score of each piece of buried point data in the first historical data, and screening buried point data with the score lower than a third threshold value;
determining pages and/or controls corresponding to the screened embedded point data, and taking the determined pages and/or controls as the pages and/or controls to be optimized;
and taking the set of buried point data corresponding to the page to be optimized and/or the control in the first historical data as a sample set.
Optionally, before the generated buried point data and the target verification result are stored in a preset database, the method further includes:
acquiring a sensitive word stock corresponding to the target application program, and identifying sensitive fields in the generated buried point data according to the sensitive word stock;
and acquiring a mapping relation between a pre-configured sensitive field and a desensitization measure, and executing desensitization processing on a value corresponding to the sensitive field in the generated buried point data according to the mapping relation.
Optionally, after the generated buried point data and the target verification result are stored in a preset database, the method further includes:
and acquiring fourth historical data corresponding to the target application program in the third time period every third time, acquiring fifth historical data corresponding to the control type, and updating a rule set corresponding to the target application program based on the fourth historical data and the fifth historical data.
Optionally, before the verifying the generated buried data using the rule set, the method further includes:
and executing cleaning processing on the generated buried point data, wherein the cleaning processing comprises removing data which is not in a first preset format and data which is in a second preset format from the generated buried point data.
In order to solve the above problems, the present invention also provides a buried data verification apparatus, the apparatus comprising:
the analysis module is used for analyzing a buried data verification request aiming at a target application program and sent by a client to acquire a control type of a control in the target application program carried by the buried data verification request;
the first generation module is used for acquiring first historical data corresponding to the target application program in a first time period, acquiring second historical data corresponding to the control type and generating an initial verification rule based on the first historical data and the second historical data;
The second generation module is used for generating a check rule configuration interface, the check rule configuration interface comprises an initial check rule modification and/or confirmation area and a check rule supplement area, the check rule configuration interface is displayed on the client interface, and a rule set corresponding to the target application program is generated based on rules configured by a user on the check rule configuration interface;
and the verification module is used for verifying the generated buried point data by using the rule set when the buried point data generated by the target application program is monitored, so as to obtain a target verification result, and storing the generated buried point data and the target verification result into a preset database.
In order to solve the above-mentioned problems, the present invention also provides an electronic apparatus including:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores a buried data verification program executable by the at least one processor, the buried data verification program being executable by the at least one processor to enable the at least one processor to perform the buried data verification method described above.
In order to solve the above-mentioned problems, the present invention also provides a computer-readable storage medium having stored thereon a buried data verification program executable by one or more processors to implement the above-mentioned buried data verification method.
Compared with the prior art, the method and the device have the advantages that first historical data corresponding to the target application program are obtained, second historical data corresponding to the control type is obtained, and an initial verification rule is generated based on the first historical data and the second historical data; then, generating a check rule configuration interface, wherein the check rule configuration interface comprises an initial check rule modification and/or confirmation area and a check rule supplementing area, and generating a rule set corresponding to the target application program based on rules configured by a user in the check rule configuration interface; and finally, when the target application program is monitored to generate buried point data, checking the generated buried point data by using a rule set to obtain a target checking result. The rule set comprises the initial check rule generated by the historical data and the specific rule supplemented by the user, so that the comprehensiveness and the integrity of the check rule are ensured, and the accuracy of the check result is improved. Therefore, the invention ensures the integrity of the checking rule and improves the accuracy of the checking result.
Drawings
FIG. 1 is a flow chart of a method for verifying buried data according to an embodiment of the present invention;
FIG. 2 is a schematic block diagram of a buried data verification device according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an electronic device for implementing a method for verifying buried data according to an embodiment of the present invention;
the achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that the description of "first", "second", etc. in this disclosure is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
The invention provides a buried data verification method. Referring to fig. 1, a flow chart of a method for verifying buried data according to an embodiment of the invention is shown. The method may be performed by an electronic device, which may be implemented in software and/or hardware.
In this embodiment, the buried data verification method includes:
s1, analyzing a buried data verification request aiming at a target application program and sent by a client to acquire a control type of a control in the target application program carried by the buried data verification request.
In general, when a user operates an application, the embedded data reported by the application includes an application identifier, a page theme, a field name in a page, a control identifier in the page, a control coordinate, and user information. The content of the embedded point data to be reported can also be set in advance according to the service type, for example, if the service type is the conversion rate of the calculation search result, the reported embedded point data comprises equipment information, user information, search terms, click term information, click position information, click source page information and skip page information. The buried data is typically Json format data.
In this embodiment, it may be that a developer of the target application program sends a request for verifying the embedded data through the client, where the request includes a control type of a control in the target application program, where the control type includes a button, an input box, and a hyperlink, and the button includes a submit type button and a select type button.
S2, acquiring first historical data corresponding to the target application program in a first time period, acquiring second historical data corresponding to the control type, and generating an initial verification rule based on the first historical data and the second historical data.
In this embodiment, the first historical data carries first labeling information, where the first labeling information includes verification success and verification failure, and the first labeling information may be labeled according to a manual verification result.
And the second historical data is check-passed buried point data corresponding to the control with the same control type in a plurality of different application programs, for example, the control type of the control 1 in the page 1 of the target application program is a hyperlink, and the check-passed buried point data corresponding to the hyperlink in the plurality of application programs is obtained and is used as the second historical data corresponding to the hyperlink.
The parameter value range of each field and/or control in the target application program can be obtained through analyzing the first historical data, and then the verification rule corresponding to each field and/or control can be generated; the control type corresponding to the second historical data is the same as the control type in the target application program, and the rule corresponding to the second historical data can be recommended to serve as a verification rule of the target application program.
The generating an initial verification rule based on the first historical data and the second historical data includes:
a11, determining index values of all preset index items corresponding to each field and/or control in the target application program based on the first marking information in the first historical data as buried point data which is successfully verified, and generating a first verification rule corresponding to the target application program based on the index values;
in this embodiment, a plurality of index items are set in advance for the field according to the data format of the field, and for the numeric field, the preset index items include a maximum value, a minimum value, a display form and a few bits after displaying the decimal point, for example, through embedding point data successfully verified in the first historical data, the maximum value of the value corresponding to the field 1 is 50, the minimum value is 10, the display form is the decimal point form and the display form is 2 bits after displaying the decimal point, and then the verification rule corresponding to the field 1 is: it is determined whether the value of field 1 ranges between 10 and 50, whether the presentation form of field 1 is in the form of a decimal point, and whether field 1 is displayed to the 2 bits after the decimal point.
For a character type field, the preset index item includes a field length.
In this embodiment, a preset index item of the control is determined according to the function of the control, for example, for a recommended bit control, the preset index item includes an item click sequence, if the click sequence of the recommended bit control 1 is analyzed to be 1-5 (only 5 items are recommended at a time due to the recommended bit) by the successfully verified buried point data in the first historical data, then the verification rule corresponding to the control 1 is if (division=1) the next clk_pos in [0,5].
And generating corresponding check rules for the fields and/or the controls in the target application program through the buried point data successfully checked in the first historical data.
A12, acquiring third historical data corresponding to the target application program in a second time period, wherein the third historical data carries second labeling information, constructing a verification model based on the first historical data, verifying the third historical data by adopting the verification model, and generating a second verification rule corresponding to the target application program based on an initial verification result output by the verification model and the second labeling information;
the first historical data and the third historical data are manually verified, manually set verification rules are often not comprehensive enough, the first historical data can be split into positive samples and negative samples according to the first labeling information, verification models can be obtained through training according to the positive samples and the negative samples (the building process of the verification models is refined in the steps B11-B13), unqualified buried point data which are not verified manually in the third historical data can be verified through the verification models, and the verification rules which can be verified to be used as second verification rules corresponding to target application programs.
In an alternative embodiment, the part of unqualified buried point data and the initial verification result and the second labeling information output by the verification model may be sent to the preset client, and the user of the preset client supplements the verification rule for the user, so that the part of unqualified buried point data may be verified according to the supplemented verification rule, and the supplemented verification rule is used as the second verification rule.
A13, acquiring an original check rule corresponding to each piece of buried point data in the second historical data, and taking a specified rule as a third check rule corresponding to the target application program if the ratio of the number of the buried point data corresponding to the specified rule in the original check rule to the total number of the buried point data in the second historical data is larger than a first threshold;
for example, if the control 2 of the target application program is an input box, second historical data corresponding to the input box is obtained, if 1000 pieces of buried point data are included in the obtained second historical data, an original check rule corresponding to each piece of buried point data is obtained, if the buried point data corresponding to the original check rule 1 is 880 pieces, the ratio corresponding to the original check rule 1 is 0.88, and if the first threshold is 0.8, the original check rule 1 is used as the check rule corresponding to the control 2.
A14, taking the set of the first, second and third verification rules as initial verification rules corresponding to the target application program.
And summarizing the first, second and third verification rules to obtain the initial verification rule corresponding to the target application program.
The constructing a verification model based on the first historical data includes:
b11, extracting a sample set from the first historical data, and splitting the sample set into a training set, a verification set and a test set;
in this embodiment, a page and/or a control with incomplete verification rules in a target application program is used as a page and/or a control to be optimized, buried data corresponding to the page and/or the control to be optimized is extracted from first historical data to be used as a sample set (the extraction process of the sample set is refined in steps C11-C13, which are described below), a sample with marking information failing to be verified in the sample set is used as a positive sample, a sample with marking information successful to be verified in the sample set is used as a negative sample, and the sample set is split into a training set, a verification set and a test set according to a preset proportion.
For example, if there are 10 ten thousand pieces of buried point data in the first historical data, 100 pieces of the first historical data are positive samples (the labeling information is verification failure), the splitting manner may be:
training set: negative samples 6 ten thousand;
Verification set: 2 ten thousand negative samples and 50 positive samples;
test set: negative samples 2 ten thousand, positive samples 50.
B12, establishing an initial model by using the training set, and adjusting model parameters of the initial model by using the verification set to obtain an updated model;
in this embodiment, feature modeling is performed on each index item of a page and/or a control to be optimized by using a multivariate gaussian distribution, for example, a modeling process is described by using a "control click range" of an index item of a control 3 of a target application program as a feature, and if a click coordinate is x (x includes an abscissa and an ordinate), feature engineering processing is performed on the click coordinate, so that the click coordinate is subjected to the multivariate gaussian distribution p (x; μ, Σ), and the process is as follows:
fitting to obtain a first parameter using a training setSecond parameter-> (μ∈R n ,Σ∈R n*n ) Wherein x is i The ith click coordinate in the training set is obtained, and n is the number of the click coordinates in the training set.
The multivariate gaussian distribution formula is:
where p (x; μ, Σ) is a gaussian distribution value, x is a click coordinate. Substituting the first parameter mu and the second parameter sigma into the formula to obtain an initial model, substituting the click coordinates in the verification set into the formula to predict whether each piece of buried data in the verification set passes the verification, and expressing a prediction result by y, wherein the calculation formula of y is as follows:
When y=1, the predicted result is a verification failure, and when y=0, the predicted result is a verification success, epsilon is a model parameter, an initial value can be set for epsilon, and epsilon is adjusted to enable the predicted result of the embedded point data in the verification set to be consistent with the first labeling information, and at the moment, the value of epsilon is fixed, so that an updated model is obtained.
And B13, calculating the performance index value of the updated model by using the test set, and taking the updated model as a verification model when the performance index value is greater than a second threshold value.
The performance indexes comprise error rate, precision and recall ratio, the updated model is input with click coordinates in the test set, whether each piece of buried data in the test set passes verification can be predicted, the index value corresponding to each performance index can be calculated according to the prediction result of each piece of data and the first labeling information, the weight and the index value corresponding to each performance index item are weighted and summarized to obtain a total performance index value, and when the total performance index value is larger than a second threshold value, the updated model is used as a verification model.
The extracting a sample set from the first historical data includes:
c11, obtaining the score corresponding to each piece of buried point data in the first historical data, and screening buried point data with the score lower than a third threshold value;
The scoring may be performed by an analyst performing user behavior trace analysis or application performance analysis on the buried data in the first historical data, and in general, the score corresponding to the buried data with accurate verification result is high, the score corresponding to the buried data with inaccurate verification result is low, so that it is known that the verification rule corresponding to the buried data with the score lower than the third threshold is incomplete or inaccurate.
C12, determining pages and/or controls corresponding to the screened embedded point data, and taking the determined pages and/or controls as the pages and/or controls to be optimized;
according to the specific content of the embedded point data, it can be known which page and/or which control of the target application program is described by each piece of the screened embedded point data, and the page and/or the control can be used as the page and/or the control to be optimized.
And C13, taking the set of buried point data corresponding to the page to be optimized and/or the control in the first historical data as a sample set.
For example, if the control to be optimized is control 3, the set of buried point data corresponding to control 3 in the first historical data is used as a sample set.
S3, generating a check rule configuration interface, wherein the check rule configuration interface comprises an initial check rule modification and/or confirmation area and a check rule supplement area, the check rule configuration interface is displayed on the client interface, and a rule set corresponding to the target application program is generated based on rules configured by a user on the check rule configuration interface.
And after obtaining the initial verification rule corresponding to the target application program, generating a verification rule configuration interface, wherein the verification rule configuration interface comprises an initial verification rule modification and/or confirmation area and a verification rule supplement area.
The specific content, the modification button and the confirmation button of the initial verification rule are displayed in the initial verification rule modification and/or confirmation area, and a user can modify or confirm the displayed initial verification rule.
A user can supplement a verification rule for a target application program through the verification rule supplementing area, for example, for a recommendation bit control, recommendation bits are generally fixed on a certain page, so that for buried point data of the recommendation bit control, whether the page mark after jumping is a specific value needs to be verified, and null transmission or error transmission cannot be carried out.
In this embodiment, after a user configures a rule on the checking rule configuration interface, the configured rule is converted into an averager expression and stored in the rule set, and the averager is a high-performance and lightweight evaluator, which can be customized, easily expanded, and supports a large number of operations and high-precision operations.
And S4, when the fact that the target application program generates buried point data is monitored, checking the generated buried point data by using the rule set to obtain a target check result, and storing the generated buried point data and the target check result into a preset database.
When the target application program is monitored to generate buried point data, the generated buried point data is verified through the Aviator expression in the rule set, verification failure is indicated when the evaluation result of the Aviator expression is false, and verification success is indicated when the evaluation result is true.
After the verification is completed, the generated buried point data and the target verification result are stored in a preset database, and buried point data failing to verify is sent to the client at intervals of preset time (for example, one hour). In this embodiment, the preset database is an ES database, and the data is queried from the ES database later, so that the data query efficiency can be improved.
Before the generated buried point data and the target verification result are stored in a preset database, the method further comprises:
d11, acquiring a sensitive word stock corresponding to the target application program, and identifying sensitive fields in the generated buried point data according to the sensitive word stock;
in this embodiment, a sensitive word stock is set for a target application program in advance, and the sensitive word stock is matched with generated buried point data, so that a sensitive field in the buried point data can be identified.
And D12, acquiring a mapping relation between a pre-configured sensitive field and a desensitization measure, and executing desensitization processing on a value corresponding to the sensitive field in the generated buried point data according to the mapping relation.
In this embodiment, a mapping relationship between a sensitive field and a desensitization measure is preconfigured, for example, for an identification card number, the desensitization measure is to mask the first three bits and the last four bits; for the mobile phone number, the desensitization measure is to mask the last eight bits. And according to the mapping relation, desensitizing the value corresponding to the sensitive field.
After the generated buried point data and the target verification result are stored in a preset database, the method further comprises:
and acquiring fourth historical data corresponding to the target application program in the third time period every third time, acquiring fifth historical data corresponding to the control type, and updating a rule set corresponding to the target application program based on the fourth historical data and the fifth historical data.
In this embodiment, at intervals of a third time, for example, three months, buried point data and verification results (i.e., fourth historical data) corresponding to the target application program in the three months, buried point data (i.e., fifth historical data) passing verification corresponding to a control of the same type as the control of the target application program in other application programs, and the fourth and fifth historical data are analyzed to update rule sets of the target application program, wherein the analysis process is the same as step S2 and is not described herein.
Before the verifying the generated buried data using the rule set, the method further includes:
and executing cleaning processing on the generated buried point data, wherein the cleaning processing comprises removing data which is not in a first preset format and data which is in a second preset format from the generated buried point data.
The first preset format is Json format, the second preset format is messy code, namely, the cleaning processing comprises eliminating buried point data which is not Json format and eliminating buried point data with messy code.
Before the generated buried point data and the target verification result are stored in a preset database, the method further comprises:
and if the ratio of the number of verification failures to the total number of verification in the target verification result corresponding to a certain specified rule in the rule set is greater than a fourth threshold, sending early warning information to the client.
If the ratio of the number of failed verification to the total verification number in the target verification result corresponding to a certain specified rule is large (for example, greater than 90%), early warning information needs to be sent to the client to remind the user of the client to timely check the abnormal reasons, which may be unreasonable in rule setting or that the page identifier or the control identifier is privately changed by the application program, and the abnormal reasons need to be timely corrected.
As can be seen from the above embodiments, in the buried data verification method provided by the present invention, first historical data corresponding to a target application program is obtained, second historical data corresponding to a control type is obtained, and an initial verification rule is generated based on the first and second historical data; then, generating a check rule configuration interface, wherein the check rule configuration interface comprises an initial check rule modification and/or confirmation area and a check rule supplementing area, and generating a rule set corresponding to the target application program based on rules configured by a user in the check rule configuration interface; and finally, when the target application program is monitored to generate buried point data, checking the generated buried point data by using a rule set to obtain a target checking result. The rule set comprises the initial check rule generated by the historical data and the specific rule supplemented by the user, so that the comprehensiveness and the integrity of the check rule are ensured, and the accuracy of the check result is improved. Therefore, the invention ensures the integrity of the checking rule and improves the accuracy of the checking result.
Fig. 2 is a schematic block diagram of a buried data verification device according to an embodiment of the invention.
The buried data verification apparatus 100 of the present invention may be installed in an electronic device. The buried data verification device 100 may include an parsing module 110, a first generating module 120, a second generating module 130, and a verification module 140 according to the implemented functions. The module of the invention, which may also be referred to as a unit, refers to a series of computer program segments, which are stored in the memory of the electronic device, capable of being executed by the processor of the electronic device and of performing a fixed function.
In the present embodiment, the functions concerning the respective modules/units are as follows:
the parsing module 110 is configured to parse a buried data verification request for a target application program sent by a client, and obtain a control type of a control in the target application program carried by the buried data verification request.
The first generation module 120 is configured to obtain first historical data corresponding to the target application program in a first period of time, obtain second historical data corresponding to the control type, and generate an initial verification rule based on the first historical data and the second historical data.
The first historical data carries first labeling information, the first labeling information comprises verification success and verification failure, and the generating an initial verification rule based on the first historical data and the second historical data comprises the following steps:
a21, determining index values of all preset index items corresponding to each field and/or control in the target application program based on the first marking information in the first historical data as buried point data which is successfully verified, and generating a first verification rule corresponding to the target application program based on the index values;
a22, acquiring third historical data corresponding to the target application program in a second time period, wherein the third historical data carries second labeling information, constructing a verification model based on the first historical data, verifying the third historical data by adopting the verification model, and generating a second verification rule corresponding to the target application program based on an initial verification result output by the verification model and the second labeling information;
A23, acquiring an original check rule corresponding to each piece of buried point data in the second historical data, and taking a specified rule as a third check rule corresponding to the target application program if the ratio of the number of the buried point data corresponding to the specified rule in the original check rule to the total number of the buried point data in the second historical data is larger than a first threshold;
a24, taking the set of the first, second and third verification rules as initial verification rules corresponding to the target application program.
The constructing a verification model based on the first historical data includes:
b21, extracting a sample set from the first historical data, and splitting the sample set into a training set, a verification set and a test set;
b22, establishing an initial model by using the training set, and adjusting model parameters of the initial model by using the verification set to obtain an updated model;
and B23, calculating the performance index value of the updated model by using the test set, and taking the updated model as a verification model when the performance index value is greater than a second threshold value.
The extracting a sample set from the first historical data includes:
c21, obtaining the score corresponding to each piece of buried point data in the first historical data, and screening buried point data with the score lower than a third threshold value;
C22, determining pages and/or controls corresponding to the screened embedded point data, and taking the determined pages and/or controls as the pages and/or controls to be optimized;
and C23, taking the set of buried point data corresponding to the page to be optimized and/or the control in the first historical data as a sample set.
The second generating module 130 is configured to generate a verification rule configuration interface, where the verification rule configuration interface includes an initial verification rule modification and/or confirmation area and a verification rule supplement area, display the verification rule configuration interface on the client interface, and generate a rule set corresponding to the target application program based on rules configured by a user on the verification rule configuration interface.
And the verification module 140 is configured to, when it is monitored that the target application program generates buried point data, verify the generated buried point data by using the rule set to obtain a target verification result, and store the generated buried point data and the target verification result into a preset database.
Before the generated buried point data and the target verification result are stored in a preset database, the verification module 140 is further configured to:
d21, acquiring a sensitive word stock corresponding to the target application program, and identifying sensitive fields in the generated buried point data according to the sensitive word stock;
And D22, acquiring a mapping relation between a pre-configured sensitive field and a desensitization measure, and executing desensitization processing on a value corresponding to the sensitive field in the generated buried point data according to the mapping relation.
After the generated buried point data and the target verification result are stored in a preset database, the verification module 140 is further configured to:
and acquiring fourth historical data corresponding to the target application program in the third time period every third time, acquiring fifth historical data corresponding to the control type, and updating a rule set corresponding to the target application program based on the fourth historical data and the fifth historical data.
Before the verifying the generated buried data using the rule set, the verification module 140 is further configured to:
and executing cleaning processing on the generated buried point data, wherein the cleaning processing comprises removing data which is not in a first preset format and data which is in a second preset format from the generated buried point data.
Before the generated buried point data and the target verification result are stored in a preset database, the verification module 140 is further configured to:
and if the ratio of the number of verification failures to the total number of verification in the target verification result corresponding to a certain specified rule in the rule set is greater than a fourth threshold, sending early warning information to the client.
Fig. 3 is a schematic structural diagram of an electronic device for implementing a method for verifying embedded data according to an embodiment of the present invention.
The electronic device 1 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a preset or stored instruction. The electronic device 1 may be a computer, a server group formed by a single network server, a plurality of network servers, or a cloud formed by a large number of hosts or network servers based on cloud computing, wherein the cloud computing is one of distributed computing, and is a super virtual computer formed by a group of loosely coupled computer sets.
In the present embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13, which are communicably connected to each other via a system bus, and the memory 11 stores therein a buried data check program 10, and the buried data check program 10 is executable by the processor 12. Fig. 3 shows only the electronic device 1 with the components 11-13 and the buried data verification program 10, it will be understood by those skilled in the art that the structure shown in fig. 3 is not limiting of the electronic device 1 and may include fewer or more components than shown, or may combine certain components, or a different arrangement of components.
Wherein the storage 11 comprises a memory and at least one type of readable storage medium. The memory provides a buffer for the operation of the electronic device 1; the readable storage medium may be a storage medium such as flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the nonvolatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the electronic device 1. In this embodiment, the readable storage medium of the memory 11 is generally used to store an operating system and various application software installed in the electronic device 1, for example, store codes of the embedded data verification program 10 in one embodiment of the present invention. Further, the memory 11 may be used to temporarily store various types of data that have been output or are to be output.
Processor 12 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 12 is typically used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices, etc. In this embodiment, the processor 12 is configured to execute the program code stored in the memory 11 or process data, such as running the buried data check program 10.
The network interface 13 may comprise a wireless network interface or a wired network interface, the network interface 13 being used for establishing a communication connection between the electronic device 1 and a client (not shown).
Optionally, the electronic device 1 may further comprise a user interface, which may comprise a Display (Display), an input unit such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device 1 and for displaying a visual user interface.
It should be understood that the embodiments described are for illustrative purposes only and are not limited to this configuration in the scope of the patent application.
The embedded data verification program 10 stored in the memory 11 of the electronic device 1 is a combination of a plurality of instructions, and the above embedded data verification method may be implemented when the processor 12 runs, and specifically, the specific implementation method of the embedded data verification program 10 by the processor 12 may refer to the description of the related steps in the corresponding embodiment of fig. 1, which is not repeated herein.
Further, the modules/units integrated in the electronic device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. The computer readable medium may be nonvolatile or nonvolatile. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
The computer-readable storage medium has stored thereon a buried data verification program 10, the buried data verification program 10 being executable by one or more processors to implement the buried data verification method described above.
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be other manners of division when actually implemented.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.
The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.
Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the system claims can also be implemented by means of software or hardware by means of one unit or means. The terms second, etc. are used to denote a name, but not any particular order.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (9)
1. A method for verifying buried data, the method comprising:
analyzing a buried data verification request aiming at a target application program and sent by a client to acquire a control type of a control in the target application program carried by the buried data verification request;
acquiring first historical data corresponding to the target application program in a first time period, acquiring second historical data corresponding to the control type, and generating an initial verification rule based on the first historical data and the second historical data, wherein the second historical data is buried point data which passes verification and corresponds to the control with the same control type in a plurality of different application programs;
generating a check rule configuration interface, wherein the check rule configuration interface comprises a modification and/or confirmation area of an initial check rule and a supplement area of the check rule, the check rule configuration interface is displayed on the client interface, and a rule set corresponding to the target application program is generated based on the rule configured by a user on the check rule configuration interface;
When the target application program is monitored to generate buried point data, the generated buried point data is checked by utilizing the rule set to obtain a target check result, and the generated buried point data and the target check result are stored into a preset database;
the first historical data carries first labeling information, the first labeling information comprises verification success and verification failure, and the generating the initial verification rule based on the first historical data and the second historical data comprises the following steps: determining index values of preset index items corresponding to each field and/or control in the target application program based on the first marking information in the first historical data as buried point data which is successfully verified, and generating a first verification rule corresponding to the target application program based on the index values; acquiring third historical data corresponding to the target application program in a second time period, wherein the third historical data carries second labeling information, constructing a verification model based on the first historical data, verifying the third historical data by adopting the verification model, and generating a second verification rule corresponding to the target application program based on an initial verification result output by the verification model and the second labeling information; acquiring an original check rule corresponding to each piece of buried point data in the second historical data, and taking a designated rule as a third check rule corresponding to the target application program if the ratio of the number of the buried point data corresponding to the designated rule in the original check rule to the total number of the buried point data in the second historical data is larger than a first threshold; and taking the set of the first, second and third check rules as initial check rules corresponding to the target application program.
2. The method of claim 1, wherein constructing a verification model based on the first historical data comprises:
extracting a sample set from the first historical data, and splitting the sample set into a training set, a verification set and a test set;
establishing an initial model by using the training set, and adjusting model parameters of the initial model by using the verification set to obtain an updated model;
and calculating the performance index value of the updated model by using the test set, and taking the updated model as a verification model when the performance index value is greater than a second threshold value.
3. The method of verifying buried data according to claim 2, wherein said extracting a sample set from said first historical data comprises:
obtaining the corresponding score of each piece of buried point data in the first historical data, and screening buried point data with the score lower than a third threshold value;
determining pages and/or controls corresponding to the screened embedded point data, and taking the determined pages and/or controls as the pages and/or controls to be optimized;
and taking the set of buried point data corresponding to the page to be optimized and/or the control in the first historical data as a sample set.
4. The method of claim 1, wherein prior to storing the generated buried point data and the target verification result in a predetermined database, the method further comprises:
acquiring a sensitive word stock corresponding to the target application program, and identifying sensitive fields in the generated buried point data according to the sensitive word stock;
and acquiring a mapping relation between a pre-configured sensitive field and a desensitization measure, and executing desensitization processing on a value corresponding to the sensitive field in the generated buried point data according to the mapping relation.
5. The method of claim 1, wherein after storing the generated buried point data and the target verification result in a predetermined database, the method further comprises:
and acquiring fourth historical data corresponding to the target application program in a third time period every third time, acquiring fifth historical data corresponding to the control type, and updating a rule set corresponding to the target application program based on the fourth historical data and the fifth historical data.
6. The method of claim 1, wherein prior to said verifying the generated buried data using the rule set, the method further comprises:
And executing cleaning processing on the generated buried point data, wherein the cleaning processing comprises removing data which is not in a first preset format and data which is in a second preset format from the generated buried point data.
7. A buried data verification apparatus for implementing the buried data verification method according to any one of claims 1 to 6, characterized in that said apparatus comprises:
the analysis module is used for analyzing a buried data verification request aiming at a target application program and sent by a client to acquire a control type of a control in the target application program carried by the buried data verification request;
the first generation module is used for acquiring first historical data corresponding to the target application program in a first time period, acquiring second historical data corresponding to the control type and generating an initial verification rule based on the first historical data and the second historical data;
the second generation module is used for generating a check rule configuration interface, the check rule configuration interface comprises an initial check rule modification and/or confirmation area and a check rule supplement area, the check rule configuration interface is displayed on the client interface, and a rule set corresponding to the target application program is generated based on rules configured by a user on the check rule configuration interface;
And the verification module is used for verifying the generated buried point data by using the rule set when the buried point data generated by the target application program is monitored, so as to obtain a target verification result, and storing the generated buried point data and the target verification result into a preset database.
8. An electronic device, the electronic device comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores a buried data verification program executable by the at least one processor, the buried data verification program being executable by the at least one processor to enable the at least one processor to perform the buried data verification method of any one of claims 1 to 6.
9. A computer-readable storage medium having stored thereon a buried data verification program executable by one or more processors to implement the buried data verification method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110733001.7A CN113485889B (en) | 2021-06-29 | 2021-06-29 | Buried data verification method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110733001.7A CN113485889B (en) | 2021-06-29 | 2021-06-29 | Buried data verification method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113485889A CN113485889A (en) | 2021-10-08 |
| CN113485889B true CN113485889B (en) | 2023-08-15 |
Family
ID=77936864
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110733001.7A Active CN113485889B (en) | 2021-06-29 | 2021-06-29 | Buried data verification method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113485889B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115484189A (en) * | 2022-08-31 | 2022-12-16 | 未来电视有限公司 | Data compliance testing method and device, electronic equipment and storage medium |
| CN115576849A (en) * | 2022-11-21 | 2023-01-06 | 智者四海（北京）技术有限公司 | Client-side embedded point correctness automatic verification method and device based on scene comparison |
| CN116522000B (en) * | 2023-05-23 | 2024-01-23 | 上海任意门科技有限公司 | Training method and device for recommendation model for recommending content to user |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105321047A (en) * | 2015-11-10 | 2016-02-10 | 中国电力科学研究院 | Multi-dimensional verification method for schedule plan data |
| CN107330641A (en) * | 2017-08-18 | 2017-11-07 | 广发证券股份有限公司 | A kind of real-time risk control system of financial derivatives based on Storm stream process framework and regulation engine and method |
| CN107688530A (en) * | 2017-04-06 | 2018-02-13 | 平安科技（深圳）有限公司 | Method for testing software and device |
| CN108764674A (en) * | 2018-05-16 | 2018-11-06 | 普信恒业科技发展（北京）有限公司 | A kind of risk control method and device of rule-based engine |
| US10489387B1 (en) * | 2018-05-24 | 2019-11-26 | People.ai, Inc. | Systems and methods for determining the shareability of values of node profiles |
| CN111459811A (en) * | 2020-03-26 | 2020-07-28 | 广州虎牙科技有限公司 | Buried point data checking method and device |
-
2021
- 2021-06-29 CN CN202110733001.7A patent/CN113485889B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105321047A (en) * | 2015-11-10 | 2016-02-10 | 中国电力科学研究院 | Multi-dimensional verification method for schedule plan data |
| CN107688530A (en) * | 2017-04-06 | 2018-02-13 | 平安科技（深圳）有限公司 | Method for testing software and device |
| CN107330641A (en) * | 2017-08-18 | 2017-11-07 | 广发证券股份有限公司 | A kind of real-time risk control system of financial derivatives based on Storm stream process framework and regulation engine and method |
| CN108764674A (en) * | 2018-05-16 | 2018-11-06 | 普信恒业科技发展（北京）有限公司 | A kind of risk control method and device of rule-based engine |
| US10489387B1 (en) * | 2018-05-24 | 2019-11-26 | People.ai, Inc. | Systems and methods for determining the shareability of values of node profiles |
| CN111459811A (en) * | 2020-03-26 | 2020-07-28 | 广州虎牙科技有限公司 | Buried point data checking method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113485889A (en) | 2021-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113485889B (en) | Buried data verification method and device, electronic equipment and storage medium | |
| CN109471857B (en) | SQL statement-based data modification method, device and storage medium | |
| CN113327136B (en) | Attribution analysis method, attribution analysis device, electronic equipment and storage medium | |
| CN109933502B (en) | Electronic device, user operation record processing method and storage medium | |
| CN110046155B (en) | Method, device and equipment for updating feature database and determining data features | |
| CN112597135A (en) | User classification method and device, electronic equipment and readable storage medium | |
| CN113178071B (en) | Driving risk level identification method and device, electronic equipment and readable storage medium | |
| CN113344433A (en) | Product matching method and device, electronic equipment and readable storage medium | |
| CN113706253A (en) | Real-time product recommendation method and device, electronic equipment and readable storage medium | |
| CN112948705A (en) | Intelligent matching method, device and medium based on policy big data | |
| CN111831708A (en) | Missing data-based sample analysis method and device, electronic equipment and medium | |
| CN113610580B (en) | Product recommendation method and device, electronic equipment and readable storage medium | |
| CN111429085A (en) | Contract data generation method and device, electronic equipment and storage medium | |
| CN114818685B (en) | Keyword extraction method and device, electronic equipment and storage medium | |
| CN114281991A (en) | Text classification method and device, electronic equipment and storage medium | |
| CN114139530A (en) | Synonym extraction method and device, electronic equipment and storage medium | |
| CN113537806A (en) | Abnormal user identification method and device, electronic equipment and readable storage medium | |
| CN113360505B (en) | Time sequence data-based data processing method and device, electronic equipment and readable storage medium | |
| CN114742060B (en) | Entity identification method, entity identification device, electronic equipment and storage medium | |
| CN117171816B (en) | Data verification method based on industrial identification and related equipment | |
| CN113656586B (en) | Emotion classification method, emotion classification device, electronic equipment and readable storage medium | |
| CN113792539B (en) | Entity relationship classification method and device based on artificial intelligence, electronic equipment and medium | |
| CN114896363B (en) | Data management method, device, equipment and medium | |
| CN113657910B (en) | Real name authentication method, device, electronic equipment and readable storage medium | |
| CN114168836A (en) | Webpage data analysis and visualization method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |