CN117762770A - Buried point data real-time verification method, system, electronic equipment and storage medium - Google Patents
Buried point data real-time verification method, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117762770A CN117762770A CN202311433851.0A CN202311433851A CN117762770A CN 117762770 A CN117762770 A CN 117762770A CN 202311433851 A CN202311433851 A CN 202311433851A CN 117762770 A CN117762770 A CN 117762770A
- Authority
- CN
- China
- Prior art keywords
- real
- point
- buried point
- buried
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012795 verification Methods 0.000 title claims abstract description 106
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000004590 computer program Methods 0.000 claims description 15
- 230000008439 repair process Effects 0.000 claims description 15
- 238000004458 analytical method Methods 0.000 claims description 10
- 238000004140 cleaning Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 5
- 230000004931 aggregating effect Effects 0.000 claims description 4
- 238000012217 deletion Methods 0.000 claims description 2
- 230000037430 deletion Effects 0.000 claims description 2
- 238000012360 testing method Methods 0.000 abstract description 19
- 238000012545 processing Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 9
- 230000002776 aggregation Effects 0.000 description 6
- 238000004220 aggregation Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 238000013524 data verification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Abstract
The invention provides a real-time verification method, a system, electronic equipment and a storage medium for embedded point data, wherein the method responds to an embedded point verification request to acquire reported embedded point logs; obtaining an error type according to comparison of the historical error buried point log and buried point registration data; and checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information to a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repairing suggestion corresponding to the error type. The reported buried point log is checked based on the error type through the real-time check task to obtain buried point check information, and the buried point check information is stored in the real-time storage engine, so that the buried point check is automatically and real-time completed, the testing efficiency is improved, the timeliness is ensured, the buried point check covers all error scenes, and the accuracy of the buried point test result is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, a system, an electronic device, and a storage medium for verifying buried point data in real time.
Background
With the rapid development of computer technology, more and more users finish online service or online transaction through the internet, and embedded point data verification generally appears in a test link before online in a service demand life cycle, and quality of online embedded points can be effectively ensured by carrying out standard verification on embedded point data which is already registered and developed, so that inaccuracy of the online embedded point data caused by incorrect embedded points is avoided, and analysis and decision of the service are affected. Aiming at verification of embedded data, currently, packet capturing verification is mainly carried out on a client by a tester, or inquiry of an original report log is carried out on the total embedded data, and then manual comparison is carried out. The buried point verification service for checking the full buried point information only checks whether the actual buried point ID is matched with the registered buried point ID, if so, the full buried point data is given, and whether the service is correct or not and the error cause are required to be compared by itself. Because tens to hundreds of buried points possibly exist in one demand, the cost of the time for triggering and comparing one by one is high, the timeliness is poor, the testing efficiency is low, and meanwhile, all scenes are difficult to cover by manual comparison, so that the accuracy of the buried point verification result is low.
Disclosure of Invention
The invention provides a real-time verification method, a system, electronic equipment and a storage medium for embedded point data, which are used for solving the defects that in the prior art, the cost of comparing verification time one by one is high, the timeliness is poor, the test efficiency is low, all scenes are difficult to cover by manual comparison, and the accuracy of embedded point test results is low.
The invention provides a buried point data real-time verification method, which comprises the following steps:
responding to a buried point verification request, and acquiring a reported buried point log;
obtaining an error type according to comparison of the historical error buried point log and buried point registration data;
and checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information into a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repair suggestion corresponding to the error type.
According to the real-time verification method for the embedded point data provided by the invention, the reported embedded point log is verified based on the error type through the real-time verification task, and the method comprises the following steps:
acquiring real-time updated buried point registration data through redis;
and carrying out real-time check on the reported buried point log and the buried point registration data updated in real time piece by piece according to the error types through a Flink task to obtain the error types of the reported buried point log and the repairing suggestions corresponding to the error types.
According to the real-time verification method for the embedded point data provided by the invention, the embedded point registration data comprises the registered embedded point meta information and the registered custom information, and the error type comprises the following steps:
a value error, wherein the value error comprises that the value of the embedded point meta information in the reported embedded point log is different from the value of the registered embedded point meta information, and the value of the custom information in the reported embedded point log is different from the value of the registered custom information;
a combination error, wherein the combination error comprises that the field combination of the embedded point meta information in the reported embedded point log is different from the field combination of the registered embedded point meta information;
the self-defining information is deleted, wherein the self-defining information deletion comprises the fact that the registered self-defining information parameter does not exist in the self-defining information parameter in the report embedded point log;
the user-defined information multi-report comprises the fact that the user-defined information parameters in the report embedded point log do not exist in the registered user-defined information parameters;
the custom information type error comprises that the custom information parameter type in the report embedded point log is different from the registered custom information parameter type;
the custom information range error comprises that the custom information parameter value range in the report embedded point log is different from the registered custom information parameter value range;
and the custom information regular error comprises a regular expression in which the parameter value of the custom information in the report embedded point log does not accord with the registered custom information.
According to the real-time verification method for the embedded point data, the embedded point verification data is stored in a real-time storage engine, and the method comprises the following steps:
and (5) adopting ClickHouse to store the buried point verification data in a distributed mode through a MergeTree structure table.
According to the real-time verification method for the embedded point data provided by the invention, the reported embedded point log is obtained, and the method comprises the following steps:
carrying out format analysis on the initially reported buried point log to obtain a buried point log after format conversion;
and carrying out data cleaning and filtering on the buried point log after format conversion to obtain a reported buried point log.
According to the real-time verification method for the embedded point data provided by the invention, after the reported embedded point log is verified based on the error type through the real-time verification task and the embedded point verification data is stored in the real-time storage engine, the method comprises the following steps:
acquiring buried point logs meeting the query conditions from the real-time storage engine according to the buried point verification query conditions;
classifying the buried point logs meeting the query conditions according to the error types, and aggregating corresponding repair suggestions and log examples to obtain the error types of the buried points, the repair suggestions corresponding to the error types and the corresponding buried point logs.
According to the real-time verification method for the embedded point data, the embedded point registration data are obtained by registering according to the embedded point registration rules, and the embedded point registration rules are determined according to the business embedded point requirements.
The invention also provides a real-time verification system for the embedded point data, which comprises the following steps:
the acquisition module is used for responding to the embedded point verification request and acquiring reported embedded point logs;
the comparison module is used for comparing the historical error buried point log with buried point registration data to obtain an error type;
the verification module is used for verifying the reported buried point log based on the error type through a real-time verification task to obtain buried point verification information, and storing the buried point verification information into a real-time storage engine, wherein the buried point verification information comprises the error type of the reported buried point log and a repair suggestion corresponding to the error type.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the real-time verification method of the embedded point data according to any one of the above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of real-time verification of embedded point data as described in any one of the above.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements a method of real-time verification of embedded point data as claimed in any one of the preceding claims.
The invention provides a real-time verification method, a system, electronic equipment and a storage medium for embedded point data, wherein the method responds to an embedded point verification request to acquire reported embedded point logs; obtaining an error type according to comparison of the historical error buried point log and buried point registration data; and checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information to a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repairing suggestion corresponding to the error type. The reported buried point log is checked based on the error type through the real-time check task to obtain buried point check information, and the buried point check information is stored in the real-time storage engine, so that the buried point check is automatically and real-time completed, the testing efficiency is improved, the timeliness is ensured, the buried point check covers all error scenes, and the accuracy of the buried point test result is improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for verifying embedded point data in real time according to the present invention;
FIG. 2 is a second flow chart of the real-time verification method of embedded point data according to the present invention;
FIG. 3 is a third flow chart of the real-time verification method of embedded point data according to the present invention;
FIG. 4 is a flow chart of a real-time verification method for embedded point data according to the present invention;
FIG. 5 is a flow chart of a real-time verification method for embedded point data according to the present invention;
FIG. 6 is a schematic diagram of a real-time verification system for embedded point data according to the present invention;
fig. 7 is a schematic diagram of the physical structure of the electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flowchart of a real-time verification method for embedded data provided by an embodiment of the present invention, as shown in fig. 1, where the real-time verification method for embedded data provided by the embodiment of the present invention includes:
step 101, responding to a buried point verification request, and acquiring a reported buried point log;
step 102, obtaining an error type according to comparison of the historical error buried point log and buried point registration data;
and step 103, checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information to a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repairing suggestion corresponding to the error type.
In the embodiment of the invention, the life cycle of the buried point is divided into the following four stages:
1. in the buried point information generation stage, a business party generates buried point demands, such as PV (page view, browsing times), UV (ultraviolet) and the like of a button with a new function according to product functions and index requirements, corresponding buried point information is registered on a buried point platform according to buried point rules, the buried point information comprises business, pages, functions, transmission parameters and a manner of triggering buried points, which need to be buried, and the buried point information is integrated into a buried point demand document and a development document in one demand, and buried point development is carried out based on the documents.
2. And a buried data generation stage. After the service research and development is buried in the corresponding function, when the user triggers the target behavior, buried data can be reported to a data warehouse or other storage engines through a data link, converted into a format and cleaned and filtered.
3. Buried data use stage. Index development or impromptu query of buried point data is carried out through offline digital bins, and real-time analysis can be carried out through a real-time engine.
4. And (5) a buried point data offline stage. After the embedded point data is online, if the analysis demand of the business on the embedded point data is ended or the new version covers the old version, the business is pushed to be offline after the traffic of the embedded point of the business is reduced to a certain range.
In the buried point data generation stage, two stages of testing and online are generally performed, and in the testing stage before online, the verification of the buried point data directly affects the quality of the online buried point and the correctness of service analysis, so that the verification of the buried point data in the testing stage is very important.
In the embodiment of the invention, after a user triggers a target task, a buried point on a service end collects SDK (Data processing tool) to collect buried point Data, and according to http request formats and strategies of the buried point Data uniformly packaged in the buried point collection SDK, the buried point Data is uniformly reported to a log receiving service at the service end in an http request or other modes, meanwhile, the performance pressure of the service end is relieved through modes such as regular reporting, quantitative reporting and the like, the log receiving service generally analyzes and converts the format of an originally reported buried point log, outputs the format into a processable format for downstream service, generally in json format, and a Data-pipe cleaning service processes the format-converted Data and mainly cleans and filters general service scene fields or functions such as abnormal value processing, token analysis and other general service functions. The cleaned data are stored in the kafka, the embedded point verification task performs error verification on the embedded point data in the kafka according to the embedded point registration rule, error information and restoration suggestions are written into an embedded point real-time storage engine, and the embedded point data are aggregated through error types by the real-time storage engine, so that a tester can conveniently and quickly group and aggregate the embedded point data.
At present, because tens to hundreds of buried points possibly exist in one demand, the cost of triggering comparison verification time one by one is high, the test efficiency is low, meanwhile, because the buried point registration standard is relatively complex, the manual comparison is difficult to cover all scenes, the problem buried points are on line, and the data accuracy is low.
The invention provides a real-time verification method, a system, electronic equipment and a storage medium for embedded point data, wherein the method responds to an embedded point verification request to acquire reported embedded point logs; obtaining an error type according to comparison of the historical error buried point log and buried point registration data; and checking the reported buried point log based on the error type through a real-time checking task, and storing buried point checking data to a real-time storage engine, wherein the buried point checking data comprises the error type of the reported buried point log and a repairing suggestion corresponding to the error type. The reported buried point log is checked based on the error type through the real-time check task, and buried point check data are stored in the real-time storage engine, so that the buried point check is automatically and real-time completed, the testing efficiency is improved, the timeliness is ensured, the buried point check covers all error scenes, and the accuracy of the buried point test result is improved.
Based on any one of the above embodiments, the present invention provides a real-time verification method for embedded point data, where embedded point registration data includes registered embedded point meta information and registered custom information, and error types include, but are not limited to:
the value error comprises that the value of the embedded point meta information in the reported embedded point log is different from the value of the registered embedded point meta information, and the value of the custom information in the reported embedded point log is different from the value of the registered custom information;
the combination error comprises that the field combination of the embedded point meta information in the reported embedded point log is different from the field combination of the registered embedded point meta information;
the method comprises the steps of deleting custom information, wherein the deleting of the custom information comprises registering the custom information parameters and not existing in the custom information parameters in the report embedded point log;
the method comprises the steps of a user-defined information multi-report, wherein the user-defined information multi-report comprises the fact that user-defined information parameters in a report embedded point log do not exist in registered user-defined information parameters;
the custom information type error comprises that the custom information parameter type in the report embedded point log is different from the registered custom information parameter type;
the custom information range error comprises that the custom information parameter value range in the reported buried point log is different from the registered custom information parameter value range;
the custom information regular error comprises a regular expression in which the parameter value of the custom information in the report embedded point log does not accord with the registered custom information.
In the embodiment of the invention, the buried point registration data is obtained by registering according to the buried point registration rule, and the buried point registration rule is determined according to the service buried point requirement. The embedded point platform registers embedded point meta information and custom information according to embedded point registration rules, the embedded point meta information describes the identification, service attribute, page attribute and type attribute of one embedded point, and the embedded point meta information can know which embedded point is produced in which mode under which page of which service scene. The custom information of the buried points describes which parameters are needed by a buried point in a specific business scenario to calculate the desired index. The embedded point registration specification does not include the embedded point environment information because the parameters of the general class do not need to be registered, the embedded point environment information, the configuration information and the like of the embedded point can be automatically acquired through the embedded point acquisition SDK, the embedded point cost of the service is reduced, and the complexity of the service embedded point is reduced.
Based on the embedded point registration rule, the error type checking rule and the embedded point registration rule are unified in caliber to obtain the seven types of error types, all error types of embedded point meta-information and custom information are covered, wherein the error types comprise parameter value errors, parameter missing, parameter multi-report, parameter value types, parameter range, parameter regularization and the like, the quality of embedded point data is guaranteed, the prior art generally only supports checking whether the embedded point ID is matched with the actual embedded point ID and the registered embedded point ID, and if the embedded point ID is matched with the actual embedded point ID, the total embedded point data is given, and whether the service self-comparison is correct or not and the error cause is needed. The embodiment of the invention can provide rich error prompts and correction suggestions based on error types, give out complete embedded point verification reports, ensure the accuracy of single embedded point data and improve the working efficiency of testers.
Based on any one of the above embodiments, the present invention provides a real-time verification method for embedded point data, as shown in fig. 2, the obtaining reporting embedded point log includes:
step 201, performing format analysis on the initially reported buried point log to obtain a buried point log after format conversion;
in the embodiment of the invention, the buried point SDK is used for collecting buried point data and reporting the buried point data to the buried point log receiving service, and then the buried point log receiving service analyzes and converts the format of the generated reported buried point log and outputs the format to a processable format for downstream service, which is generally json format.
Step 202, cleaning and filtering data of the buried point log after format conversion to obtain a reported buried point log.
In the embodiment of the invention, the Data after the Data-pi pline cleaning service format conversion is used for cleaning and filtering the Data, including cleaning and filtering general service scene fields or functions, such as abnormal value processing, token parsing and other general service functions.
Based on any of the above embodiments, as shown in fig. 3, checking the reported buried point log based on the error type by a real-time checking task includes:
step 301, acquiring real-time updated buried point registration data through redis;
in the embodiment of the invention, the embedded point information is acquired through the redis, the embedded point meta information and the custom information are synchronously updated to the redis for storage, and the redis real-time data is adopted for comparison and verification, so that the real-time performance of verification is ensured.
And 302, checking the reported buried point log and the buried point registration data updated in real time one by one according to the error types through the link task to obtain the error types of the reported buried point log and the repair suggestions corresponding to the error types.
In the embodiment of the invention, the flank task consumes the buried point data subjected to format processing in real time, compares the buried point data with buried point information in redis, checks the seven error types one by one, generates a set of error types and corresponding repairing suggestions, and merges the set of error types and the corresponding repairing suggestions into an original log to be written into a downstream storage engine ClickHouse sink stream. By pre-arranging the error type and the repair suggestion package, functional decoupling is realized, and the problem that the error type and the package repair suggestion need to be analyzed when the downstream aggregation is carried out according to the error type is avoided.
The Flink architecture comprises 3 layers, namely a Client layer, a task manager layer and a JobManager layer, wherein the Client layer comprises a component connected with kafka, a map component of Flink processing check logic and a component connected with ClickHouse, as 3 nodes, a source operator is built in the component connected with kafka and used for receiving upstream data, a map operator is built in the Flink and used for processing check logic, and a sink operator is built in the component connected with ClickHouse and used for transmitting check results to the downstream. By deploying TaskManager on different computers for executing tasks corresponding to 3 nodes, slots in the TaskManager are used for controlling how much memory is used for controlling parallelism, by deploying JobManager on computers for managing TaskManager and allocating which operators are executed by the slots of which TaskManager in particular.
Based on any one of the above embodiments, the present invention provides a real-time verification method for embedded point data, storing embedded point verification data into a real-time storage engine, including:
and (5) adopting ClickHouse to store the buried point verification data in a distributed mode through a MergeTree structure table.
In the embodiment of the invention, because the flow rate of the buried data is large, an OLAP (Online Analytical Processing, online analysis and processing) engine is generally used for storing, for example ClickHouse, doris or drud, so as to ensure the test efficiency. The ClickHouse is an open source column database for online analysis processing, the Doris is a distributed SQL database based on a massive parallel processing technology, and the Druid is an open source, distributed and column-oriented real-time analysis data storage system.
The ClickHouse uses a MergeTree structure table to store buried point verification data in a distributed mode, the field type of the distributed table comprises a user id type, a time type, a buried point registration data type and a buried point verification result type, wherein the time type is the starting and stopping time of user verification, the buried point result type comprises two upstream task important fields, an error type and a hit restoration suggestion, the error type field is used for storing all error types of a single buried point log, the restoration suggestion field is used for storing all restoration suggestions of the single buried point log, k-v format is used for storing k-v, k is a specific suggestion, and when a user inquires according to an inquiry condition, the distributed table is used for obtaining a structure table for locally storing the data, and further obtaining the buried point log meeting the inquiry condition.
Most of the prior art does not support real-time engine storage inquiry, adopts a message queue consumption mode, and has limited aggregation rate; even if the real-time engine storage is adopted, only a full error log is generally given, and the test efficiency is low. The embodiment of the invention supports second-level rapid grouping aggregation query by using a real-time engine to store check data, realizes aggregation according to the embedded point meta-information and error types, enumerates all error types, classifies the total error logs, focuses error reasons and improves the test efficiency.
Based on any of the above embodiments, as shown in fig. 4, the present invention provides a real-time verification method for embedded point data, which verifies reported embedded point logs based on error types through a real-time verification task, and stores the embedded point verification data in a real-time storage engine, and then includes:
step 401, acquiring buried point logs meeting the query conditions from a real-time storage engine according to the buried point verification query conditions;
step 402, classifying the buried point logs meeting the query conditions according to the error types, and aggregating the corresponding repair suggestions and log examples to obtain the error types of the buried points, the repair suggestions corresponding to the error types and the corresponding buried point logs.
The method comprises the steps of inputting the equipment id of a triggering buried point, verifying the start and stop time and the buried point meta information to inquire the actual report log, classifying all the buried point logs hit by the inquiry condition according to the error types, and aggregating the repair suggestions and the corresponding log examples so as to inform a user of all the error types, the repair suggestions and the corresponding buried point log examples of each buried point. According to the error type aggregation, the tester can conveniently focus the error type, the identification efficiency is improved, and each buried point log does not need to be displayed.
The invention provides a real-time verification method for buried data, as shown in fig. 5, which specifically comprises the following steps:
after a user triggers a target task, a buried point on a service end collects SDK (Data processing tool) to collect buried point Data, the buried point Data is uniformly reported to a buried point log receiving service in an http request or other modes according to an http request format and a strategy of the buried point Data uniformly packaged by the SDK collected at the buried point, meanwhile, performance pressure of the service end is relieved through modes such as regular reporting and quantitative reporting, the buried point log receiving service generally analyzes and converts an originally reported buried point log into a format and outputs the format into a processable format for downstream service, the Data is processed by a json format, and general service scene fields or functions such as abnormal value processing and token analysis are mainly cleaned and filtered. The cleaned data are stored in the kafka, the embedded point verification Flink task consumes the upstream kafka data in real time, error verification is conducted on the embedded point data in the kafka according to embedded point registration rules, error information and restoration suggestions are written into an embedded point real-time storage engine, the embedded point data are aggregated through error types by the real-time storage engine, so that rapid grouping aggregation query by test staff is facilitated, and timeliness of the whole verification process can reach the second level.
The embedded point data reporting to the real-time data verification link is established through the embedded point acquisition SDK, the embedded point log receiving service, the data cleaning service, the kafka center, the embedded point verification Flink task and the ClickHouse storage engine, so that the testing efficiency of the tester is improved, and the accuracy of the embedded point data is ensured.
The embedded point data real-time verification system provided by the invention is described below, and the embedded point data real-time verification system described below and the embedded point data real-time verification method described above can be referred to correspondingly.
Fig. 6 is a schematic structural diagram of a real-time verification system for embedded data according to an embodiment of the present invention, as shown in fig. 6, where the real-time verification system for embedded data according to an embodiment of the present invention includes:
an acquisition module 601, configured to respond to a buried point verification request, and acquire a reported buried point log;
the comparison module 602 is configured to compare the historical error buried point log with the buried point registration data to obtain an error type;
the verification module 603 is configured to verify, through a real-time verification task, the reported buried point log based on the error type, obtain buried point verification information, and store the buried point verification information to the real-time storage engine, where the buried point verification information includes the error type of the reported buried point log and a repair suggestion corresponding to the error type.
Fig. 7 illustrates a physical schematic diagram of an electronic device, as shown in fig. 7, which may include: processor 710, communication interface (Communications Interface) 720, memory 730, and communication bus 740, wherein processor 710, communication interface 720, memory 730 communicate with each other via communication bus 740. Processor 710 may invoke logic instructions in memory 730 to perform a method of real-time verification of embedded data, the method comprising: responding to a buried point verification request, and acquiring a reported buried point log; obtaining an error type according to comparison of the historical error buried point log and buried point registration data; and checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information to a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repairing suggestion corresponding to the error type.
Further, the logic instructions in the memory 730 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the method for real-time verification of embedded point data provided by the above methods, the method comprising: responding to a buried point verification request, and acquiring a reported buried point log; obtaining an error type according to comparison of the historical error buried point log and buried point registration data; and checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information to a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repairing suggestion corresponding to the error type.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of performing a method for real-time verification of embedded point data provided by the methods described above, the method comprising: responding to a buried point verification request, and acquiring a reported buried point log; obtaining an error type according to comparison of the historical error buried point log and buried point registration data; and checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information to a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repairing suggestion corresponding to the error type.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product, which may be stored in a computer-readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the various embodiments or methods of some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for verifying buried data in real time, comprising:
responding to a buried point verification request, and acquiring a reported buried point log;
obtaining an error type according to comparison of the historical error buried point log and buried point registration data;
and checking the reported buried point log based on the error type through a real-time checking task to obtain buried point checking information, and storing the buried point checking information into a real-time storage engine, wherein the buried point checking information comprises the error type of the reported buried point log and a repair suggestion corresponding to the error type.
2. The real-time verification method of embedded point data according to claim 1, wherein the verifying the reported embedded point log based on an error type by a real-time verification task comprises:
acquiring real-time updated buried point registration data through redis;
and carrying out real-time check on the reported buried point log and the buried point registration data updated in real time piece by piece according to the error types through a Flink task to obtain the error types of the reported buried point log and the repairing suggestions corresponding to the error types.
3. The method for real-time verification of embedded point data according to claim 1, wherein the embedded point registration data includes registered embedded point meta information and registered custom information, and the error type includes:
a value error, wherein the value error comprises that the value of the embedded point meta information in the reported embedded point log is different from the value of the registered embedded point meta information, and the value of the custom information in the reported embedded point log is different from the value of the registered custom information;
a combination error, wherein the combination error comprises that the field combination of the embedded point meta information in the reported embedded point log is different from the field combination of the registered embedded point meta information;
the self-defining information is deleted, wherein the self-defining information deletion comprises the fact that the registered self-defining information parameter does not exist in the self-defining information parameter in the report embedded point log;
the user-defined information multi-report comprises the fact that the user-defined information parameters in the report embedded point log do not exist in the registered user-defined information parameters;
the custom information type error comprises that the custom information parameter type in the report embedded point log is different from the registered custom information parameter type;
the custom information range error comprises that the custom information parameter value range in the report embedded point log is different from the registered custom information parameter value range;
and the custom information regular error comprises a regular expression in which the parameter value of the custom information in the report embedded point log does not accord with the registered custom information.
4. The method for real-time verification of embedded point data according to claim 1, wherein storing the embedded point verification information to a real-time storage engine comprises:
and (5) adopting ClickHouse to store the buried point verification data in a distributed mode through a MergeTree structure table.
5. The method for real-time verification of embedded point data according to claim 1, wherein the obtaining the report embedded point log comprises:
carrying out format analysis on the initially reported buried point log to obtain a buried point log after format conversion;
and carrying out data cleaning and filtering on the buried point log after format conversion to obtain a reported buried point log.
6. The real-time verification method of embedded point data according to claim 1, wherein the verifying the reported embedded point log based on the error type by the real-time verification task to obtain embedded point verification information, and storing the embedded point verification information in a real-time storage engine comprises:
acquiring buried point logs meeting the query conditions from the real-time storage engine according to the buried point verification query conditions;
classifying the buried point logs meeting the query conditions according to the error types, and aggregating corresponding repair suggestions and log examples to obtain the error types of the buried points, the repair suggestions corresponding to the error types and the corresponding buried point logs.
7. The method for real-time verification of embedded point data according to claim 1, wherein the embedded point registration data is obtained by registering according to an embedded point registration rule, and the embedded point registration rule is determined according to business embedded point requirements.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of real-time verification of embedded data as claimed in any one of claims 1 to 7 when the program is executed by the processor.
9. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the buried data real-time verification method according to any of claims 1 to 7.
10. A computer program product comprising a computer program which, when executed by a processor, implements the embedded point data real-time verification method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311433851.0A CN117762770A (en) | 2023-10-31 | 2023-10-31 | Buried point data real-time verification method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311433851.0A CN117762770A (en) | 2023-10-31 | 2023-10-31 | Buried point data real-time verification method, system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117762770A true CN117762770A (en) | 2024-03-26 |
Family
ID=90313285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311433851.0A Pending CN117762770A (en) | 2023-10-31 | 2023-10-31 | Buried point data real-time verification method, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117762770A (en) |
-
2023
- 2023-10-31 CN CN202311433851.0A patent/CN117762770A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11829365B2 (en) | Systems and methods for data quality monitoring | |
| US10031829B2 (en) | Method and system for it resources performance analysis | |
| US20170109657A1 (en) | Machine Learning-Based Model for Identifying Executions of a Business Process | |
| US20170109676A1 (en) | Generation of Candidate Sequences Using Links Between Nonconsecutively Performed Steps of a Business Process | |
| CN111027615B (en) | Middleware fault early warning method and system based on machine learning | |
| US20180046956A1 (en) | Warning About Steps That Lead to an Unsuccessful Execution of a Business Process | |
| CN112181955B (en) | Data standard management method for information sharing of heavy haul railway comprehensive big data platform | |
| CN111459698A (en) | Database cluster fault self-healing method and device | |
| CN108108445A (en) | A kind of data intelligence processing method and system | |
| CN106708965A (en) | Data processing method and apparatus | |
| JP2019114241A (en) | Sql tuning automation method and system via statistical sql pattern analysis | |
| CN113806343B (en) | Evaluation method and system for Internet of vehicles data quality | |
| CN111767193A (en) | Server data anomaly detection method and device, storage medium and equipment | |
| CN117251414B (en) | Data storage and processing method based on heterogeneous technology | |
| CN117762770A (en) | Buried point data real-time verification method, system, electronic equipment and storage medium | |
| CN116248393A (en) | Intranet data transmission loophole scanning device and system | |
| CN107273293B (en) | Big data system performance test method and device and electronic equipment | |
| CN113377801A (en) | Data inspection method, data inspection device, electronic equipment and computer storage medium | |
| CN105590224A (en) | Method for determining failure node in transaction process | |
| CN111352818A (en) | Application program performance analysis method and device, storage medium and electronic equipment | |
| CN111953544B (en) | Fault detection method, device, equipment and storage medium of server | |
| CN113037550B (en) | Service fault monitoring method, system and computer readable storage medium | |
| CN116308135A (en) | Big data auditing and statistics system for quotation quotes | |
| CN117707943A (en) | Data regression testing method and device | |
| CN114036143A (en) | Repeated detection method, equipment and medium for multi-source heterogeneous data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |