CN113642047A - Buried point data verification method and system - Google Patents

Buried point data verification method and system Download PDF

Info

Publication number
CN113642047A
CN113642047A CN202110931151.9A CN202110931151A CN113642047A CN 113642047 A CN113642047 A CN 113642047A CN 202110931151 A CN202110931151 A CN 202110931151A CN 113642047 A CN113642047 A CN 113642047A
Authority
CN
China
Prior art keywords
buried point
point data
transmission processing
data
processing platform
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110931151.9A
Other languages
Chinese (zh)
Inventor
袁敏
未燕
沙艳波
张建奇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Bilibili Technology Co Ltd
Original Assignee
Shanghai Bilibili Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Bilibili Technology Co Ltd filed Critical Shanghai Bilibili Technology Co Ltd
Priority to CN202110931151.9A priority Critical patent/CN113642047A/en
Publication of CN113642047A publication Critical patent/CN113642047A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
    • G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
    • G06K7/10544—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation by scanning of the records by radiation in the optical part of the electromagnetic spectrum
    • G06K7/10821—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation by scanning of the records by radiation in the optical part of the electromagnetic spectrum further details of bar or optical code scanning devices

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Security & Cryptography (AREA)
  • Toxicology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Bioethics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

The application discloses a buried point data verification method, wherein the reporting of the buried point data comprises two channels, namely an online big data transmission processing platform and an independent transmission processing platform, and the method comprises the following steps: providing the configuration information to the mobile terminal through the two-dimensional code; after the mobile terminal scans the two-dimensional code, redirecting the buried point data reported by the mobile terminal to an independent transmission processing platform according to the configuration information for transmission processing; and the buried point data is consumed from a push target of the independent transmission processing platform and is automatically verified. The application also discloses a buried point data verification system, an electronic device and a computer readable storage medium. Therefore, the method and the device can support one-key code scanning, reduce the use threshold, support real-time reporting of the buried point data, ensure the end-to-end correctness of the data, perform automatic verification and improve the verification efficiency.

Description

Buried point data verification method and system
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method, a system, an electronic device, and a computer-readable storage medium for verifying buried data.
Background
The embedded point is a common data acquisition mode, and is a related technology for capturing and reporting information such as user behaviors or events by implanting codes in products. The buried point data is a very important data source of a company product and is an important basis for the company to perform user behavior analysis and product decision, so that the quality of the buried point data is also important, and the buried point data verification is an essential product guarantee link.
Currently, there are two verification schemes commonly used in the industry: the first is to set an agent on the mobile device, and then to check the reported data from the request received by the agent in a packet capturing mode by the agent; the second is to wait for the reported data to be stored in the data warehouse and then verify the reported data by inquiring the data in the data warehouse.
However, the above first scheme is cumbersome to configure the agent, and not all APP (application) versions support setting the agent, and this method does not cover the data processing process from the sending of the data reporting request to the storage of the data warehouse, and in addition, as the requirement of the company for security increases, the request sent by the product gradually starts to encrypt or use the data packet of the custom protocol, so that the scenarios that this method can verify become less and less. The second scheme has delay of T +1, real-time verification cannot be achieved, and the query efficiency of the data warehouse is low. Meanwhile, the two schemes are not friendly to non-technical users, have a certain use threshold, have no method for carrying out additional automatic verification, and have poor expandability.
It should be noted that the above-mentioned contents are not intended to limit the scope of protection of the application.
Disclosure of Invention
The present application mainly aims to provide a method, a system, an electronic device and a computer readable storage medium for verifying buried point data, and aims to solve the problems of high use cost, low efficiency and incapability of performing real-time end-to-end verification in the conventional scheme.
In order to achieve the above object, an embodiment of the present application provides a method for verifying buried point data, where reporting of the buried point data includes two channels, namely an online big data transmission processing platform and an independent transmission processing platform, and the method includes:
providing the configuration information to the mobile terminal through the two-dimensional code;
after the mobile terminal scans the two-dimensional code, redirecting the buried point data reported by the mobile terminal to the independent transmission processing platform according to the configuration information for transmission processing;
and the buried point data is consumed from a push target of the independent transmission processing platform and is automatically verified.
Optionally, the configuration information in the two-dimensional code includes: setting a jump address of a page, a redirected target IP address, a unique identifier allocated to the equipment of the mobile terminal, a time interval reported by the buried point data and a callback address.
Optionally, the redirecting, after the mobile terminal scans the two-dimensional code, the buried point data reported by the mobile terminal to the independent transmission processing platform according to the configuration information to perform transmission processing includes:
after receiving an instruction of scanning the two-dimensional code by the mobile terminal, controlling the mobile terminal to jump to the setting page according to the jump address in the two-dimensional code;
automatically filling the configuration information in the two-dimensional code into the setting page, and receiving a successful code scanning notification sent by the mobile terminal according to the callback address in the two-dimensional code;
redirecting the mobile terminal to the independent transmission processing platform corresponding to the target IP address according to the buried point data reported by the time interval in the two-dimensional code;
and the independent transmission processing platform carries out transmission processing on the buried point data and pushes the buried point data to the external Kafka.
Optionally, the independent transmission processing platform is laid in an independent environment for simulating the online big data transmission processing platform, and the transmission processing process and logic of the online big data transmission processing platform on the buried point data are consistent, so as to verify the buried point data.
Optionally, the consuming the buried point data from the push target of the independent transmission processing platform and performing automatic verification includes:
consuming and parsing the buried point data from the external Kafka;
acquiring metadata corresponding to the buried point data from the online big data transmission processing platform;
assembling the buried point data and the metadata obtained by analysis into readable data;
acquiring a verification rule from the online big data transmission processing platform;
and verifying the packaged readable data according to the verification rule to obtain a verification result.
Optionally, the method further comprises:
acquiring a corresponding connection channel according to the unique identifier contained in the buried point data;
and sending the verification result of the buried point data to a front end page for displaying through the connecting channel.
Optionally, the time interval is a real-time reporting period of one second.
In addition, to achieve the above object, an embodiment of the present application further provides a buried point data verification system, where the system includes:
the configuration module is used for providing the configuration information to the mobile terminal through the two-dimensional code;
the processing module is used for redirecting the buried point data reported by the mobile terminal to an independent transmission processing platform for transmission processing according to the configuration information after the mobile terminal scans the two-dimensional code, wherein the reporting of the buried point data comprises two channels of an on-line big data transmission processing platform and the independent transmission processing platform;
and the verification module is used for consuming the buried point data from a push target of the independent transmission processing platform and performing automatic verification.
In order to achieve the above object, an embodiment of the present application further provides an electronic device, including: the embedded data verification program can be used for realizing the embedded data verification method when being executed by the processor.
To achieve the above object, an embodiment of the present application further provides a computer-readable storage medium, on which a buried point data verification program is stored, and when executed by a processor, the buried point data verification program implements the buried point data verification method as described above.
The method, the system, the electronic device and the computer readable storage medium for verifying the buried point data can support one-key code scanning and reduce the use threshold; after code scanning, the buried point data can be reported in real time through specific configuration, and various versions of APP are supported; the reported buried point data is redirected to an independent transmission processing platform consistent with the on-line data for transmission processing, so that the buried point data undergoes a transmission processing process consistent with the on-line data, and the end-to-end correctness of the data is guaranteed; and then, automatic verification is carried out according to the consumed data, so that the verification efficiency is improved.
Drawings
FIG. 1 is a diagram of an application environment architecture in which various embodiments of the present application may be implemented;
fig. 2 is a flowchart of a buried point data verification method according to a first embodiment of the present application;
FIG. 3 is a detailed flowchart of step S202 in FIG. 2;
FIG. 4 is a schematic diagram of an on-line big data transmission processing platform and an independent transmission processing platform according to the present application;
FIG. 5 is a detailed flowchart of step S204 in FIG. 2;
fig. 6 is a flowchart of a buried point data verification method according to a second embodiment of the present application;
FIG. 7 is a detailed flowchart of step S306 in FIG. 6;
fig. 8 is a schematic hardware architecture diagram of an electronic device according to a third embodiment of the present application;
fig. 9 is a block diagram of a buried point data verification system according to a fourth embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the descriptions relating to "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a diagram illustrating an application environment architecture for implementing various embodiments of the present application. The application can be applied to application environments including, but not limited to, mobile terminal 2, service terminal 4, and network 6.
The mobile terminal 2 is configured to display the two-dimensional code for a user to scan the code, and data of the embedded point in the mobile terminal 2 is reported to the server terminal 4. The mobile terminal 2 may be a mobile terminal device such as a mobile phone and a tablet computer.
The server 4 is configured to provide the configuration information to the mobile terminal 2 through the two-dimensional code, redirect the reported buried point data to an independent transmission processing environment for transmission processing after the mobile terminal 2 scans the code, and then consume the buried point data from a push target of the independent transmission processing environment and perform verification. In each embodiment of the present application, the reporting of the buried point data includes two channels, namely, an online big data transmission processing platform and an independent transmission processing platform. The independent transmission processing platform is used for simulating the on-line big data transmission processing platform to lay in an independent environment, is consistent with the transmission processing process and logic of the on-line big data transmission processing platform to the buried point data, and is used for verifying the buried point data. The server 4 may be composed of one or more servers and other electronic devices. The server may be a rack server, a blade server, a tower server or a cabinet server, or may be an independent server or a server cluster formed by a plurality of servers.
The network 6 may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, and the like. The service end 4 and one or more mobile ends 2 are connected through the network 6 for data transmission and interaction.
Example one
Fig. 2 is a flowchart of a buried point data verification method according to a first embodiment of the present application. It is to be understood that the flow charts in the embodiments of the present method are not intended to limit the order in which the steps are performed. Some steps in the flowchart may be added or deleted as desired. The method will be described below with the server 4 as the execution subject.
The method comprises the following steps:
and S200, providing the configuration information to the mobile terminal through the two-dimensional code.
The mobile terminal APP can generate buried point data in the using process, the buried point data is sent to a big data transmission processing link (an online big data transmission processing platform) through an HTTP request, the link comprises a gateway layer (Lancer gateway), a data buffer layer (internal Kafka), a distribution layer (Lancer Collector) and the like, and the online big data transmission processing platform can finally distribute the buried point data to pushing targets such as Hive (a data warehouse tool) and external Kafka (an open source stream processing platform). In this embodiment, the HTTP request is redirected to another independent transmission processing platform that is consistent with the environment of the online big data transmission processing platform, and the independent transmission processing platform performs a transmission processing process that is consistent with the online environment on the reported buried point data.
And the data of the embedded point generated by the mobile terminal APP can be cached in the mobile terminal under the normal condition, and a data reporting request can be triggered when the data is accumulated to a certain quantity or reaches a certain time interval, so that the data of the embedded point can not be reported in real time under most conditions. In this embodiment, the setting may be completed by scanning the two-dimensional code, so that the buried point data is reported in real time.
The two-dimensional code mainly comprises the following configuration information:
(1) and setting the jump address of the page. And after the user uses the mobile terminal to scan the two-dimensional code, jumping to an APP setting page where the jumping address is located.
(2) And reporting the redirected target IP address, namely the IP address of the independent transmission processing platform, by the buried point data.
(3) And Unique Identification (UUID) distributed to the equipment of the mobile terminal.
(4) Time interval (reporting period) for reporting buried point data. For example, the reporting period of the buried point data is adjusted to a real-time reporting period of one second level.
(5) And the callback address is the interface address corresponding to the test tool of the server. According to the callback address, after the code scanning of the mobile terminal is successful, a message that the code scanning of the mobile terminal is successful is notified to the testing tool of the server terminal, so that the buried point data is consumed from the testing tool to the push target Kafka of the independent transmission processing platform.
The two-dimensional code can be displayed through the front end (Web page) of the testing tool, so that the two-dimensional code is provided for a mobile terminal to scan the code.
And S202, after the mobile terminal scans the codes, redirecting the buried point data reported by the mobile terminal to an independent transmission processing platform according to the configuration information for transmission processing.
Specifically, further refer to fig. 3, which is a schematic view of the detailed flow of step S202. It is to be understood that the flow chart is not intended to limit the order in which the steps are performed. Some steps in the flowchart may be added or deleted as desired. In this embodiment, the step S202 specifically includes:
and S2020, after receiving an instruction that the mobile terminal scans the two-dimensional code, controlling the mobile terminal to jump to a setting page according to the jump address in the two-dimensional code.
S2022, automatically filling the configuration information in the two-dimensional code into the setting page, and receiving a successful code scanning notification sent by the mobile terminal according to the callback address in the two-dimensional code.
And after the user scans the two-dimensional code by using the mobile terminal, the user jumps to the setting page, and simultaneously, the configuration information in the two-dimensional code is automatically filled into the setting page for the user to check. And after the setting is finished, the test tool is informed of the success of code scanning through a preset interface (the callback address).
And S2024, redirecting the buried point data reported by the mobile terminal according to the time interval in the two-dimensional code to an independent transmission processing platform corresponding to the target IP address.
After returning from the setting page, the mobile terminal APP redirects the domain name reported by the buried point data to the target IP address configured in the setting page, the reporting period is adjusted from the original period (for example, quantitative or five-minute level) to real-time reporting of one-second level, and the reported buried point data is brought with the unique identifier allocated to the mobile terminal device.
After that, the HTTP request corresponding to the buried point data may be redirected to the independent transmission processing platform according to the target IP address.
S2026, the independent transmission processing platform performs transmission processing on the buried point data and pushes the buried point data to an external Kafka.
The independent transmission processing platform simulates the online big data transmission processing platform by the server to lay, and the transmission processing process of the data buried in the data is consistent with that of the online big data transmission processing platform when the environment of the online big data transmission processing platform is consistent with that of the online big data transmission processing platform. Fig. 4 is a schematic diagram of the online big data transmission processing platform and the independent transmission processing platform in the present application.
And finally, the independent transmission processing platform distributes the buried point data to Hive, external Kafka and other push targets.
S204, the buried point data is consumed from the push target of the independent transmission processing platform and is automatically verified.
The test tool consumes the buried point data from the external Kafka of one of the push targets, thereby enabling the test tool to support end-to-end data assurance.
Specifically, further refer to fig. 5, which is a schematic view of the detailed flow of step S204. It is to be understood that the flow chart is not intended to limit the order in which the steps are performed. Some steps in the flowchart may be added or deleted as desired. In this embodiment, the step S204 specifically includes:
and S2040, consuming and analyzing the buried point data from the external Kafka.
According to the transmission processing process of the independent transmission processing platform, it can be known that buried point data reported by a mobile terminal can be pushed to an external Kafka through transmission processing, and the back end (which can be a Java server) of the test tool consumes the buried point data from the external Kafka and then analyzes the data.
S2042, obtaining metadata corresponding to the buried point data from the online big data transmission processing platform.
Because the buried point data (reported by the mobile terminal) obtained by analysis only has values without metadata information, the rear end of the test tool can simultaneously acquire the metadata corresponding to the buried point data from the corresponding service of the on-line big data transmission processing platform. The metadata is descriptive data of data and information resources, and is mainly defined by some buried points in the embodiment.
S2044, the buried point data and the metadata obtained through analysis are assembled into readable data.
S2046, obtaining a verification rule from the online big data transmission processing platform.
In this embodiment, in order to improve the testing efficiency, the back end of the testing tool may further obtain the verification rule from the online big data transmission processing platform. The verification rule is a Data Quality Check (DQC) rule, which is a rule configurable on the Data Quality platform and can be used to Check the Data format and value of the buried Data reported by the mobile terminal.
S2048, verifying the packaged readable data according to the verification rule to obtain a verification result.
And the rear end of the test tool automatically verifies the readable data according to the DQC rule, and obtains data to be sent according to a verification result after verification is completed.
The embedded data verification method provided by the embodiment can support a low threshold mode used by one-key code scanning, so that any user can use the embedded data verification method without a threshold; after code scanning, the buried point data can be reported in real time through specific configuration, and various versions of APP are supported; the reported buried point data is redirected to an independent transmission processing platform consistent with the on-line data for transmission processing, so that the buried point data undergoes a transmission processing process consistent with the on-line data, and the end-to-end correctness of the data is guaranteed; and DQC rule real-time automatic verification and time-consuming monitoring of each link of data processing are supported, and verification efficiency is improved.
Example two
Fig. 6 is a flowchart of a buried point data verification method according to a second embodiment of the present application. In the second embodiment, the buried point data verification method further includes step S306 based on the first embodiment. It is to be understood that the flow charts in the embodiments of the present method are not intended to limit the order in which the steps are performed. Some steps in the flowchart may be added or deleted as desired.
The method comprises the following steps:
and S300, providing the configuration information to the mobile terminal through the two-dimensional code.
The method comprises the steps that a mobile terminal APP generates buried point data in the using process, the buried point data is sent to a big data transmission processing link (an online big data transmission processing platform) through an HTTP request, and the online big data transmission processing platform finally distributes the buried point data to pushing targets such as Hive and external Kafka. In this embodiment, the HTTP request is redirected to another independent transmission processing platform that is consistent with the environment of the online big data transmission processing platform, and the independent transmission processing platform performs a transmission processing process that is consistent with the online environment on the reported buried point data.
And the data of the embedded point generated by the mobile terminal APP can be cached in the mobile terminal under the normal condition, and a data reporting request can be triggered when the data is accumulated to a certain quantity or reaches a certain time interval, so that the data of the embedded point can not be reported in real time under most conditions. In this embodiment, the setting may be completed by scanning the two-dimensional code, so that the buried point data is reported in real time.
The two-dimensional code mainly comprises the following configuration information:
(1) and setting the jump address of the page. And after the user uses the mobile terminal to scan the two-dimensional code, jumping to an APP setting page where the jumping address is located.
(2) And reporting the redirected target IP address, namely the IP address of the independent transmission processing platform, by the buried point data.
(3) And Unique Identification (UUID) distributed to the equipment of the mobile terminal.
(4) Time interval (reporting period) for reporting buried point data. For example, the reporting period of the buried point data is adjusted to a real-time reporting period of one second level.
(5) And the callback address is the interface address corresponding to the test tool of the server. According to the callback address, after the code scanning of the mobile terminal is successful, a message that the code scanning of the mobile terminal is successful is notified to the testing tool of the server terminal, so that the buried point data is consumed from the testing tool to the push target Kafka of the independent transmission processing platform.
The two-dimensional code can be displayed through the front end (Web page) of the testing tool, so that the two-dimensional code is provided for a mobile terminal to scan the code.
S302, after the mobile terminal scans the codes, redirecting the buried point data reported by the mobile terminal to an independent transmission processing platform according to the configuration information for transmission processing.
And after receiving an instruction of scanning the two-dimensional code by the mobile terminal, controlling the mobile terminal to jump to a set page according to a jump address in the two-dimensional code. And then, automatically filling the configuration information in the two-dimensional code into the setting page, and receiving a successful code scanning notification sent by the mobile terminal according to the callback address in the two-dimensional code. And redirecting the buried point data reported by the mobile terminal according to the time interval in the two-dimensional code to an independent transmission processing platform corresponding to the target IP address.
After returning from the setting page, the mobile terminal APP redirects the domain name reported by the buried point data to the target IP address configured in the setting page, the reporting period is adjusted from the original period (for example, quantitative or five-minute level) to real-time reporting of one-second level, and the reported buried point data is brought with the unique identifier allocated to the mobile terminal device.
After that, the HTTP request corresponding to the buried point data may be redirected to the independent transmission processing platform according to the target IP address. And the independent transmission processing platform carries out transmission processing on the buried point data and pushes the buried point data to the external Kafka.
S304, the buried point data is consumed from the push target of the independent transmission processing platform and is automatically verified.
The test tool consumes the buried point data from the external Kafka of one of the push targets, thereby enabling the test tool to support end-to-end data assurance. According to the transmission processing process of the independent transmission processing platform, it can be known that buried point data reported by a mobile terminal can be pushed to an external Kafka through transmission processing, and the back end (which can be a Java server) of the test tool consumes the buried point data from the external Kafka and then analyzes the data.
Because the buried point data (reported by the mobile terminal) obtained by analysis only has values without metadata information, the rear end of the test tool can simultaneously acquire the metadata corresponding to the buried point data from the corresponding service of the on-line big data transmission processing platform. The metadata is descriptive data of data and information resources, and is mainly defined by some buried points in the embodiment.
And then assembling the buried point data and the metadata obtained by analysis into readable data, and acquiring a verification rule from the online big data transmission processing platform.
In this embodiment, in order to improve the testing efficiency, the back end of the testing tool may further obtain the verification rule from the online big data transmission processing platform. The verification rule is a DQC rule which is a rule which is configurable on a data quality platform and can be used for checking the data format and the value of buried point data reported by a mobile terminal.
And finally, verifying the packaged readable data according to the verification rule to obtain a verification result. In this embodiment, the back end of the test tool automatically verifies the readable data according to the DQC rule, and obtains data to be sent according to a verification result after verification is completed.
S306, sending the verification result to a front-end page for displaying.
The back end of the test tool pushes the verification result to a front end (Web end) page through an independent channel for display, so that a user can only see the data reported by the equipment after code scanning.
Specifically, further refer to fig. 7, which is a schematic view of the detailed flow of step S306. It is to be understood that the flow chart is not intended to limit the order in which the steps are performed. Some steps in the flowchart may be added or deleted as desired. In this embodiment, the step S306 specifically includes:
s3060, obtaining a corresponding connection channel according to the unique identifier contained in the buried point data.
In this embodiment, when the user uses the test tool (code scanning), the user opens a front-end page of the test tool, establishes a Websocket (a protocol for performing full duplex communication on a single TCP connection) connection with the back-end, and the back-end provides a unique Session identifier (Session ID) for the connection channel, and stores the Session ID and the corresponding Websocket connection channel as a keyword (key) and a value (value) in the lookup table (Map). Meanwhile, the Session ID is also transmitted to the front end to be used as the UUID in the two-dimensional code. And the UUID is carried by the reported buried point data after the mobile terminal scans the code.
And after the rear end of the test tool processes the consumed data to obtain data to be sent, finding a corresponding Websocket connecting channel from the Map according to the UUID in the buried point data.
S3062, the verification result of the embedded data is sent to a front end page through the connecting channel to be displayed.
In this embodiment, each mobile terminal APP corresponds to a different independent Websocket connection channel, and after the Websocket connection channel is obtained, data is pushed to a front-end page through the connection channel, so that a user of the mobile terminal can only see data reported by the device.
And finally, displaying the real-time pushed buried point data on a front-end page, and displaying a DQC verification result, timeout information of each link and the like.
The embedded data verification method provided by the embodiment can support a low threshold mode used by one-key code scanning, so that any user can use the embedded data verification method without a threshold; after code scanning, the buried point data can be reported in real time through specific configuration, and various versions of APP are supported; the reported buried point data is redirected to an independent transmission processing platform consistent with the on-line data for transmission processing, so that the buried point data undergoes a transmission processing process consistent with the on-line data, and the end-to-end correctness of the data is guaranteed; the DQC rule real-time automatic verification and time-consuming monitoring of each link of data processing are supported, and the verification efficiency is improved; different mobile end APPs use independent Websocket connection channels to push data, so that users can only see data reported by the devices, and data safety is guaranteed.
EXAMPLE III
Fig. 8 is a schematic diagram of a hardware architecture of an electronic device 20 according to a third embodiment of the present application. In the present embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23, which are communicatively connected to each other through a system bus. It is noted that fig. 8 only shows the electronic device 20 with components 21-23, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. In this embodiment, the electronic device 20 may be the server 4.
The memory 21 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 21 may be an internal storage unit of the electronic device 20, such as a hard disk or a memory of the electronic device 20. In other embodiments, the memory 21 may also be an external storage device of the electronic apparatus 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the electronic apparatus 20. Of course, the memory 21 may also include both an internal storage unit and an external storage device of the electronic apparatus 20. In this embodiment, the memory 21 is generally used for storing an operating system installed in the electronic device 20 and various application software, such as program codes of the embedded data verification system 60. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is generally used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is configured to run the program codes stored in the memory 21 or process data, for example, run the buried point data verification system 60.
The network interface 23 may include a wireless network interface or a wired network interface, and the network interface 23 is generally used for establishing a communication connection between the electronic apparatus 20 and other electronic devices.
Example four
Fig. 9 is a block diagram of a buried point data verification system 60 according to a fourth embodiment of the present invention. The buried data verification system 60 may be partitioned into one or more program modules, which are stored in a storage medium and executed by one or more processors to implement embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of performing specific functions, and the following description will specifically describe the functions of each program module in the embodiments.
In this embodiment, the buried point data verification system 60 includes:
the configuration module 600 is configured to provide the configuration information to the mobile terminal through the two-dimensional code.
The method comprises the steps that a mobile terminal APP generates buried point data in the using process, the buried point data are sent to an online big data transmission processing platform through HTTP requests, and the online big data transmission processing platform finally distributes the buried point data to pushing targets such as Hive and external Kafka. In this embodiment, the HTTP request is redirected to another independent transmission processing platform that is consistent with the environment of the online big data transmission processing platform, and the independent transmission processing platform performs a transmission processing process that is consistent with the online environment on the reported buried point data.
And the data of the embedded point generated by the mobile terminal APP can be cached in the mobile terminal under the normal condition, and a data reporting request can be triggered when the data is accumulated to a certain quantity or reaches a certain time interval, so that the data of the embedded point can not be reported in real time under most conditions. In this embodiment, the setting may be completed by scanning the two-dimensional code, so that the buried point data is reported in real time.
The two-dimensional code mainly comprises the following configuration information:
(1) and setting the jump address of the page. And after the user uses the mobile terminal to scan the two-dimensional code, jumping to an APP setting page where the jumping address is located.
(2) And reporting the redirected target IP address, namely the IP address of the independent transmission processing platform, by the buried point data.
(3) And Unique Identification (UUID) distributed to the equipment of the mobile terminal.
(4) Time interval (reporting period) for reporting buried point data. For example, the reporting period of the buried point data is adjusted to a real-time reporting period of one second level.
(5) And the callback address is the interface address corresponding to the test tool of the server. According to the callback address, after the code scanning of the mobile terminal is successful, a message that the code scanning of the mobile terminal is successful is notified to the testing tool of the server terminal, so that the buried point data is consumed from the testing tool to the push target Kafka of the independent transmission processing platform.
The two-dimensional code can be displayed through the front end (Web page) of the testing tool, so that the two-dimensional code is provided for a mobile terminal to scan the code.
A processing module 602, configured to redirect, according to the configuration information, the buried point data reported by the mobile terminal to an independent transmission processing platform for transmission processing after the mobile terminal scans the code.
And after the user scans the two-dimensional code by using the mobile terminal, the user jumps to the setting page, and simultaneously, the configuration information in the two-dimensional code is automatically filled into the setting page for the user to check. And after the setting is finished, the test tool is informed of the success of code scanning through a preset interface (the callback address).
After returning from the setting page, the mobile terminal APP redirects the domain name reported by the buried point data to the target IP address configured in the setting page, the reporting period is adjusted from the original period (for example, quantitative or five-minute level) to real-time reporting of one-second level, and the reported buried point data is brought with the unique identifier allocated to the mobile terminal device.
After that, the HTTP request corresponding to the buried point data may be redirected to the independent transmission processing platform according to the target IP address. The independent transmission processing platform simulates the online big data transmission processing platform by the server to lay, and the transmission processing process of the data buried in the data is consistent with that of the online big data transmission processing platform when the environment of the online big data transmission processing platform is consistent with that of the online big data transmission processing platform. And finally, the independent transmission processing platform distributes the buried point data to Hive, external Kafka and other push targets.
A verification module 604, configured to consume the buried point data from the push target of the independent transmission processing platform and perform automatic verification.
The verification module 604 consumes the buried point data from the external Kafka that is one of the push targets and then parses it so that the test tool supports end-to-end data assurance.
Since the buried point data (reported by the mobile terminal) obtained by the analysis only has a value without metadata information, the verification module 604 may obtain the metadata corresponding to the buried point data from the corresponding service of the online big data transmission processing platform at the same time. The metadata is descriptive data of data and information resources, and is mainly defined by some buried points in the embodiment.
In this embodiment, in order to improve the testing efficiency, the verification module 604 further obtains the verification rule from the online big data transmission processing platform. The verification rule is a Data Quality Check (DQC) rule, which is a rule configurable on the Data Quality platform and can be used to Check the Data format and value of the buried Data reported by the mobile terminal. The verification module 604 performs automatic verification on the readable data according to the DQC rule, and obtains data to be transmitted according to a verification result after the verification is completed.
The embedded data verification system provided by the embodiment can support a low threshold mode used by one-key code scanning, so that any user can use the embedded data verification system without a threshold; after code scanning, the buried point data can be reported in real time through specific configuration, and various versions of APP are supported; the reported buried point data is redirected to an independent transmission processing platform consistent with the on-line data for transmission processing, so that the buried point data undergoes a transmission processing process consistent with the on-line data, and the end-to-end correctness of the data is guaranteed; and DQC rule real-time automatic verification and time-consuming monitoring of each link of data processing are supported, and verification efficiency is improved.
EXAMPLE five
The present application further provides another embodiment, which is to provide a computer readable storage medium storing a buried point data verification program, which is executable by at least one processor to cause the at least one processor to perform the steps of the buried point data verification method as described above.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different from that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications that can be made by the use of the equivalent structures or equivalent processes in the specification and drawings of the present application or that can be directly or indirectly applied to other related technologies are also included in the scope of the present application.

Claims (10)

1. A buried point data verification method is characterized in that reporting of the buried point data comprises two channels, namely an online big data transmission processing platform and an independent transmission processing platform, and the method comprises the following steps:
providing the configuration information to the mobile terminal through the two-dimensional code;
after the mobile terminal scans the two-dimensional code, redirecting the buried point data reported by the mobile terminal to the independent transmission processing platform according to the configuration information for transmission processing;
and the buried point data is consumed from a push target of the independent transmission processing platform and is automatically verified.
2. The method for verifying buried point data as claimed in claim 1, wherein the configuration information in the two-dimensional code comprises: setting a jump address of a page, a redirected target IP address, a unique identifier allocated to the equipment of the mobile terminal, a time interval reported by the buried point data and a callback address.
3. The method for verifying the buried point data according to claim 2, wherein the redirecting the buried point data reported by the mobile terminal to the independent transmission processing platform according to the configuration information for transmission processing after the mobile terminal scans the two-dimensional code comprises:
after receiving an instruction of scanning the two-dimensional code by the mobile terminal, controlling the mobile terminal to jump to the setting page according to the jump address in the two-dimensional code;
automatically filling the configuration information in the two-dimensional code into the setting page, and receiving a successful code scanning notification sent by the mobile terminal according to the callback address in the two-dimensional code;
redirecting the mobile terminal to the independent transmission processing platform corresponding to the target IP address according to the buried point data reported by the time interval in the two-dimensional code;
and the independent transmission processing platform carries out transmission processing on the buried point data and pushes the buried point data to the external Kafka.
4. The method for verifying the data of the buried points as claimed in claim 1 or 3, wherein the independent transmission processing platform is laid in an independent environment for simulating the online big data transmission processing platform, and the transmission processing process and logic of the online big data transmission processing platform to the data of the buried points are consistent, so as to verify the data of the buried points.
5. The buried point data verification method according to claim 3 or 4, wherein the consuming the buried point data from the push target of the independent transmission processing platform and performing automatic verification includes:
consuming and parsing the buried point data from the external Kafka;
acquiring metadata corresponding to the buried point data from the online big data transmission processing platform;
assembling the buried point data and the metadata obtained by analysis into readable data;
acquiring a verification rule from the online big data transmission processing platform;
and verifying the packaged readable data according to the verification rule to obtain a verification result.
6. The buried point data verification method of claim 2, further comprising:
acquiring a corresponding connection channel according to the unique identifier contained in the buried point data;
and sending the verification result of the buried point data to a front end page for displaying through the connecting channel.
7. The method of claim 2, wherein the time interval is a real-time reporting period on the order of one second.
8. A buried point data verification system, the system comprising:
the configuration module is used for providing the configuration information to the mobile terminal through the two-dimensional code;
the processing module is used for redirecting the buried point data reported by the mobile terminal to an independent transmission processing platform for transmission processing according to the configuration information after the mobile terminal scans the two-dimensional code, wherein the reporting of the buried point data comprises two channels of an on-line big data transmission processing platform and the independent transmission processing platform;
and the verification module is used for consuming the buried point data from a push target of the independent transmission processing platform and performing automatic verification.
9. An electronic device, comprising: a memory, a processor and a buried point data verification program stored on the memory and executable on the processor, the buried point data verification program implementing the buried point data verification method as claimed in any one of claims 1 to 7 when executed by the processor.
10. A computer-readable storage medium, having stored thereon a buried point data verification program which, when executed by a processor, implements the buried point data verification method according to any one of claims 1 to 7.
CN202110931151.9A 2021-08-13 2021-08-13 Buried point data verification method and system Pending CN113642047A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110931151.9A CN113642047A (en) 2021-08-13 2021-08-13 Buried point data verification method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110931151.9A CN113642047A (en) 2021-08-13 2021-08-13 Buried point data verification method and system

Publications (1)

Publication Number Publication Date
CN113642047A true CN113642047A (en) 2021-11-12

Family

ID=78421557

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110931151.9A Pending CN113642047A (en) 2021-08-13 2021-08-13 Buried point data verification method and system

Country Status (1)

Country Link
CN (1) CN113642047A (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108304176A (en) * 2017-12-14 2018-07-20 广东数果科技有限公司 Point methods are buried in a kind of visualization of cross-platform mobile terminal
WO2018202171A1 (en) * 2017-05-05 2018-11-08 平安科技（深圳）有限公司 Buried point verification system and method
CN111488261A (en) * 2020-03-11 2020-08-04 北京健康之家科技有限公司 User behavior analysis system, method, storage medium and computing device
CN111506489A (en) * 2019-01-30 2020-08-07 阿里巴巴集团控股有限公司 Test method, system, device, server and storage medium
CN111611140A (en) * 2020-04-30 2020-09-01 广州华多网络科技有限公司 Reporting verification method and device of buried point data, electronic equipment and storage medium
CN111736918A (en) * 2020-06-12 2020-10-02 北京字节跳动网络技术有限公司 Buried point configuration method and device and electronic equipment
CN111797008A (en) * 2020-06-18 2020-10-20 北京每日优鲜电子商务有限公司 Automatic verification method, equipment and storage medium for mobile terminal buried point data

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018202171A1 (en) * 2017-05-05 2018-11-08 平安科技（深圳）有限公司 Buried point verification system and method
CN108304176A (en) * 2017-12-14 2018-07-20 广东数果科技有限公司 Point methods are buried in a kind of visualization of cross-platform mobile terminal
CN111506489A (en) * 2019-01-30 2020-08-07 阿里巴巴集团控股有限公司 Test method, system, device, server and storage medium
CN111488261A (en) * 2020-03-11 2020-08-04 北京健康之家科技有限公司 User behavior analysis system, method, storage medium and computing device
CN111611140A (en) * 2020-04-30 2020-09-01 广州华多网络科技有限公司 Reporting verification method and device of buried point data, electronic equipment and storage medium
CN111736918A (en) * 2020-06-12 2020-10-02 北京字节跳动网络技术有限公司 Buried point configuration method and device and electronic equipment
CN111797008A (en) * 2020-06-18 2020-10-20 北京每日优鲜电子商务有限公司 Automatic verification method, equipment and storage medium for mobile terminal buried point data

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
QINGLIANGHU: "大厂经验：埋点数据质量之埋点验证", Retrieved from the Internet <URL:https://zhuanlan.zhihu.com/p/300744170> *

Similar Documents

Publication Publication Date Title
CN108460115B (en) Message pushing method and device, computer equipment and storage medium
CN107239701B (en) Method and device for identifying malicious website
US20140019957A1 (en) Method, apparatus, and system for sharing software among terminals
CN109391673A (en) A kind of method, system and the terminal device of management update file
CN111106983B (en) Method and device for detecting network connectivity
CN112437078A (en) File storage method, device, equipment and computer readable storage medium
CN112367345A (en) Data processing method, server device and computer readable storage medium
CN109982152A (en) P2P CDN method for processing business, intelligent gateway and CDN platform
US8880108B2 (en) Short message processing method and apparatus
CN110851818A (en) Verification code display method and device, computer equipment and readable storage medium
CN113434293A (en) Method and apparatus for processing repeat request, storage medium, and electronic apparatus
CN103530343A (en) Structural data interactive system, data receiving terminal and structural data interactive method
CN112818270B (en) Data cross-domain transfer method and device and computer equipment
CN107509097B (en) Video sharing method and device and sharing server
CN111597559B (en) System command injection vulnerability detection method and device, equipment and storage medium
CN113055348B (en) Cross-platform data request method and device and electronic equipment
CN111143650B (en) Method, device, medium and electronic equipment for acquiring page data
CN111052072B (en) Installation-free application program downloading method and device
CN113642047A (en) Buried point data verification method and system
CN110933070A (en) User identification method, system, equipment and computer readable storage medium
CN110866196A (en) Printer network information acquisition method and device and electronic equipment
CN104809122A (en) Method and device for visiting web pages
CN112711518B (en) Log uploading method and device
CN110727554A (en) Data monitoring method, device and system
CN105677453B (en) Task processing method and device for task platform

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination