CN114417133A - Business data processing method and device, electronic equipment and computer storage medium - Google Patents

Business data processing method and device, electronic equipment and computer storage medium Download PDF

Info

Publication number
CN114417133A
CN114417133A CN202111584706.3A CN202111584706A CN114417133A CN 114417133 A CN114417133 A CN 114417133A CN 202111584706 A CN202111584706 A CN 202111584706A CN 114417133 A CN114417133 A CN 114417133A
Authority
CN
China
Prior art keywords
data
information
service
reporting information
subdata
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111584706.3A
Other languages
Chinese (zh)
Inventor
曹勇
邱德军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Dajia Internet Information Technology Co Ltd
Original Assignee
Beijing Dajia Internet Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dajia Internet Information Technology Co Ltd filed Critical Beijing Dajia Internet Information Technology Co Ltd
Priority to CN202111584706.3A priority Critical patent/CN114417133A/en
Publication of CN114417133A publication Critical patent/CN114417133A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • G06F16/9535Search customisation based on user profiles and personalisation

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The present disclosure relates to a method, an apparatus, an electronic device and a computer-readable storage medium for processing service data, wherein the method comprises: determining target reporting information to be verified from the reporting information; the reporting information is generated based on a service request aiming at the online recommendation service; generating statistical data for detection based on the target reporting information; and detecting the online data generated by the online recommendation service based on the statistical data to obtain a detection result. Therefore, before the Browse Set component receives the problem feedback of the user, the accuracy of the Browse Set can be quickly and accurately detected by the method, the problems that the accuracy of the Browse Set is slow to find and difficult to find due to the problem feedback of the user are solved, the timeliness, the stability and the accuracy of detection are greatly improved, and the user experience is improved.

Description

Business data processing method and device, electronic equipment and computer storage medium
Technical Field
The present disclosure relates to the field of internet technologies, and in particular, to a method and an apparatus for processing service data, an electronic device, and a computer-readable storage medium.
Background
In recent years, with the ecological development of internet content, the user experience of a user can be greatly reduced by pushing repeated content in a recommendation system, and the recommendation system naturally selects similar content in the browsing history of the user to push, so that the repeated content is very easy to appear. Therefore, it is a common practice of most recommendation systems to record user browsing history data, and then remove the recorded user browsing history data from the recommendation result, and the stored user browsing history data is called Browse Set.
However, as the recommended content pool and the number of users increase, the data volume of Browse Set will increase at a very high rate, and the data structure will become more and more complex to enable efficient querying. In this context, the accuracy of Browse Set cannot be effectively guaranteed.
In response to the above problem, it is currently possible to detect the problem by a general Browse Set component. However, most Browse Set components can only find the problem of the component itself when the user feeds back the bad case, and the user often affects many users when feeding back the bad case, and the timeliness is low. In addition, because the data volume of the Browse Set is large, the bad case fed back by the user is difficult to position quickly, and the accuracy is poor.
Disclosure of Invention
In order to overcome the problems in the related art, the present disclosure provides a service data processing method, an apparatus, an electronic device, and a storage medium. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, a method for processing service data is provided, including:
determining target reporting information to be verified from the reporting information; the reporting information is generated based on a service request aiming at the online recommendation service;
generating statistical data for detection based on the target reporting information;
and detecting the online data generated by the online recommendation service based on the statistical data to obtain a detection result.
Optionally, the generating statistical data for detection based on the target reporting information includes:
carrying out format conversion on the target reporting information to obtain reporting data;
verifying the reported data to obtain a verification result;
and generating statistical data for detection based on the reported data and the verification result.
Optionally, the determining, from the reporting information, target reporting information to be verified includes:
acquiring first reported information from the reported information according to a preset sampling rate;
acquiring second reported information from the reported information according to a preset white list;
and taking the first reporting information and the second reporting information as target reporting information to be verified.
Optionally, the format conversion of the target reporting information to obtain the reporting data includes:
formatting first reporting information and second reporting information in the target reporting information respectively by adopting a preset data structure to obtain reporting data with the preset data structure; the preset data structure comprises a standard data structure and/or a custom data structure.
Optionally, the reported data includes at least one piece of subdata;
the verifying the reported data to obtain a verification result comprises:
acquiring corresponding service data from online data based on a first preset field in the reported data; the first preset field is used for recording the position information of the service data;
and detecting whether each subdata in the at least one subdata belongs to the service data or not to obtain a detection result corresponding to each subdata in the at least one subdata.
Optionally, any sub-data in the at least one sub-data has a corresponding sub-data identifier; the service data comprises an identification list, and the identification list comprises at least one subdata identification;
the detecting whether each sub data in the at least one sub data belongs to the service data to obtain a detection result corresponding to each sub data in the at least one sub data includes:
matching the subdata identifier of any subdata in the at least one subdata with the identifier list;
if the matching item exists, judging that any subdata belongs to the service data, and generating a first detection result of successful verification;
and if the matching item does not exist, judging that any subdata does not belong to the service data, and generating a second detection result of verification failure.
Optionally, the generating statistical data for detection based on the reported data and the verification result includes:
splicing the verification result and the reported data based on a second preset field to obtain at least one group of target reported data; any one group of target reported data in the at least one group of target reported data comprises at least one subdata; the second preset field is a repeated field in the verification result and the reported data;
respectively counting a first number of subdata with detection results being successful detection in each group of target reported data and a second number of subdata with detection results being failed detection;
and taking the first quantity and the second quantity corresponding to each group of target reported data as statistical data.
Optionally, the detecting, based on the statistical data, the online data generated by the online recommendation service to obtain a detection result includes:
calculating at least one dimension analysis data corresponding to each preset feature in at least one preset feature by adopting the statistical data;
detecting whether the analysis data of any dimension of the at least one dimension exceeds an analysis data threshold of any dimension;
and if the analysis data of any dimension in the at least one dimension does not exceed the analysis data threshold of any dimension, judging that the on-line data generated by the on-line recommendation service is normal.
Optionally, the method further includes:
and responding to the access request, and accessing the online data through a preset data interface.
According to a second aspect of the embodiments of the present disclosure, there is provided a service data processing apparatus, including:
the data acquisition unit is configured to determine target report information to be verified from the report information; the reported information is obtained based on a service request aiming at the online recommendation service;
a statistical data generating unit configured to generate statistical data for detection based on the target report information;
and the monitoring unit is configured to detect the online data generated by the online recommendation service based on the statistical data to obtain a detection result.
Optionally, the statistical data generating unit includes:
the formatting subunit is configured to perform format conversion on the target reporting information to obtain reporting data;
the calling subunit is configured to verify the reported data to obtain a verification result;
a multidimensional statistics subunit configured to generate statistics data for detection based on the reported data and the verification result.
Optionally, the data obtaining unit includes:
the on-line data sampling subunit is configured to acquire first reporting information from the reporting information according to a preset sampling rate;
the white list data subunit is configured to acquire second reporting information from the reporting information according to a preset white list;
the first processing subunit is configured to use the first reporting information and the second reporting information as target reporting information to be verified.
Optionally, the formatting unit is specifically configured to:
formatting first reporting information and second reporting information in the target reporting information respectively by adopting a preset data structure to obtain reporting data with the preset data structure; the preset data structure comprises a standard data structure and/or a custom data structure.
Optionally, the reported data includes at least one piece of subdata;
the calling unit comprises:
the service data subunit is configured to acquire corresponding service data from online data based on a first preset field in the reported data; the first preset field is used for recording the position information of the service data;
a first detecting subunit, configured to detect whether each sub data in the at least one sub data belongs to the service data, to obtain a detection result corresponding to each sub data in the at least one sub data.
Optionally, any sub-data in the at least one sub-data has a corresponding sub-data identifier; the service data comprises an identification list, and the identification list comprises at least one subdata identification;
the first detection subunit is specifically configured to:
matching the subdata identifier of any subdata in the at least one subdata with the identifier list; if the matching item exists, judging that any subdata belongs to the service data, and generating a first detection result of successful verification; and if the matching item does not exist, judging that any subdata does not belong to the service data, and generating a second detection result of verification failure.
Optionally, the multidimensional statistic unit includes:
the splicing subunit is configured to splice the verification result and the reported data based on a second preset field to obtain at least one group of target reported data; any one group of target reported data in the at least one group of target reported data comprises at least one subdata; the second preset field is a repeated field in the verification result and the reported data;
the statistical subunit is configured to respectively count a first number of the subdata with detection results being successful detection in each group of target reported data, and count a second number of the subdata with detection results being failed detection;
and the second processing subunit is configured to use the first quantity and the second quantity corresponding to each group of target report data as statistical data.
Optionally, the monitoring unit includes:
the calculation subunit is configured to calculate, by using the statistical data, analysis data of at least one dimension corresponding to each preset feature in at least one preset feature;
a second detection subunit configured to detect whether the analysis data of any dimension of the at least one dimension exceeds an analysis data threshold of the any dimension;
the judging subunit is configured to judge that the online data generated by the online recommendation service is normal if the analysis data of any dimension of the at least one dimension does not exceed the analysis data threshold of any dimension.
Optionally, the method further includes:
and the access unit is configured to access the online data through a preset data interface.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the business data processing method as in the first aspect.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, wherein instructions of the storage medium, when executed by a processor of a mobile terminal, enable an electronic device to perform the service data processing method according to the first aspect.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program or computer instructions which, when executed by a processor, implement the business data processing method of the first aspect.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: the Browse Set verification system determines target report information to be verified from the report information; the reporting information is generated based on a service request aiming at the online recommendation service; then, generating statistical data for detection based on the target report information, and detecting the online data generated by the online recommendation service based on the statistical data to obtain a detection result of whether the online data generated by the online recommendation service is abnormal or not. Therefore, before the Browse Set component receives the problem feedback of the user, the accuracy of the Browse Set can be quickly and accurately detected by the method, the problems that the accuracy of the Browse Set is slow to find and difficult to find due to the problem feedback of the user are solved, the timeliness, the stability and the accuracy of detection are greatly improved, and the user experience is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a flowchart illustrating a business data processing method according to an exemplary embodiment.
FIG. 2 is a system framework diagram shown in accordance with an exemplary embodiment.
Fig. 3 is a first block diagram illustrating the structural framework of the Browse Set authentication system according to an exemplary embodiment.
Fig. 4 is a second structural framework diagram of the Browse Set verification system according to an exemplary embodiment.
Fig. 5 is a third block diagram illustrating the structural framework of the Browse Set authentication system according to an exemplary embodiment.
Fig. 6 is a fourth block diagram illustrating the structural framework of the Browse Set authentication system according to an exemplary embodiment.
Fig. 7 is a block diagram illustrating a traffic data processing apparatus according to an example embodiment.
Fig. 8 is a block diagram illustrating an apparatus for business data processing in accordance with an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
Fig. 1 is a flowchart illustrating a business data processing method according to an exemplary embodiment, where the business data processing method is used in a Browse Set verification system, as shown in fig. 1, and includes the following steps.
In step S11, determining target report information to be verified from the report information; the reporting information is generated based on a service request for an online recommendation service.
Referring to fig. 2, there is shown a schematic diagram of a system framework of the present embodiment, which includes an online recommendation service, a Browse Set verification system, and a Browse Set component. The online recommendation service is used for sampling online flow according to a service request of a user and reporting the sampling to a Browse Set verification system; the Browse Set verification system is used for verifying the accuracy of the Browse Set; the Browse Set component is used for reporting related information of the white list user, providing Browse Set SDK (Software Development Kit) verification, and storing complete Browse Set.
Specifically, a user may initiate a service request to the online recommendation service through a terminal device (e.g., a mobile phone or a tablet computer), so an APP for initiating the service request may be installed in the terminal device. Therefore, when a user initiates a service request at an APP in the terminal device, the service request can carry information such as a user ID, a device ID, a requested service, a request platform, an APP version and the like.
After receiving the service request, the online recommendation service generates reporting information (e.g., server show information) based on the information in the service request, where the reporting information includes, but is not limited to, a user ID (Identity document) of the service request, a device ID, a requested service, a request platform, and an APP (Application) version, and then sends the reporting information to the Browse Set verification system.
In practical application, because the amount of the reported information reported by the online recommendation service is huge, in order to reduce the load of the Browse Set verification system, a part of the reported information can be selected from all the reported information for verification, and for convenience of description, the selected reported information is recorded as the target reported information to be verified.
In this embodiment, the determining target reporting information to be verified from the reporting information includes:
acquiring first reported information from the reported information according to a preset sampling rate;
acquiring second reported information from the reported information according to a preset white list;
and taking the first reporting information and the second reporting information as target reporting information to be verified.
Referring to fig. 3, a first structural framework diagram of the Browse Set verification system of the present embodiment is shown, where the Browse Set verification system includes a data acquisition module and a verification center, and the data acquisition module includes an online data sampling module and a white list data module.
Specifically, the online data sampling module may obtain a part of the report information from all the report information according to a preset sampling rate, and record the part of the report information as "first report information". For example, the acquisition may be performed at a sampling rate of 0.1%, that is, 1 reporting information is acquired from every 1000 reporting information as the first reporting information. Of course, the sampling rate may also be obtained according to other sampling rates, for example, 1%, 10%, and the like, in practical applications, the sampling rate may be inversely adjusted according to the number of the reported information, and the specific value of the sampling rate is not limited in this embodiment.
Further, the online recommendation service may report the report information to the Browse Set verification system, and also report the report information to the Browse Set component, and the Browse Set component may store all the report information, and then the white list data module acquires a part of the report information from the Browse Set component according to the white list, and records the part as "second report information".
And then, taking the acquired first reporting information and the acquired second reporting information as final target reporting information.
For example, the online recommendation service generates 100 pieces of reported information, the preset sampling rate is 1%, and the number of the white lists is 10, then, the online data sampling module acquires 1 piece of first reported information from 100 pieces of reported information, the white list data module acquires 10 pieces of second reported information from 100 pieces of reported information, and finally, the target reported information to be verified is 11 pieces of reported information.
Further, after acquiring the first report information, the online data sampling module may send the first report information to a preset message system, such as kafka, according to a preset data format, such as a proto data format, so that the Browse Set verification center acquires the first report information from kafka.
It should be noted that the Browse Set component may store all the obtained reporting information in a preset database management system, such as clickhouse, and then the white list data module obtains the second reporting information from the database management system according to the white list. Besides, the database management system can store the reported information, and can also store all requests of the white list users as the full request link logs of the white list users for subsequent problem investigation, thereby realizing online data real-time sampling and white list full data storage, overcoming the singleness of verification data sources and enabling the verification service to cover all logics.
Further, the message system in this embodiment may adopt kafka, the database management system may adopt clickhouse, and other message systems and database management systems are also applicable to this embodiment, and may be set according to actual requirements in actual application, which is not limited in this embodiment.
In step S12, statistical data for detection is generated based on the target report information.
After the target report information is acquired, statistical data used for detecting online data generated by the online recommendation service may be generated according to the target report information.
In this embodiment, the generating statistical data for detection based on the target report information includes:
carrying out format conversion on the target reporting information to obtain reporting data;
verifying the reported data to obtain a verification result;
and generating statistical data for detection based on the reported data and the verification result.
Specifically, after the verification center acquires the target report information, format conversion can be performed on the target report information to obtain report data with a uniform format, so that subsequent data verification is facilitated.
In this embodiment, the performing format conversion on the target reporting information to obtain reporting data includes:
formatting first reporting information and second reporting information in the target reporting information respectively by adopting a preset data structure to obtain reporting data with the preset data structure; the preset data structure comprises a standard data structure and/or a custom data structure.
Referring to fig. 4, a schematic diagram of a structural framework of the Browse Set authentication system of the present embodiment is shown, in which the authentication center includes a formatting module, a Browse Set authentication module, and a monitoring alarm module.
Specifically, since the target reporting information includes the first reporting information and the second reporting information, the formatting module may adopt a preset data structure, where the preset data structure may include, but is not limited to, a standard data structure and a custom data structure. For example, if the preset data structure is a proto data structure, the first reporting information and the second reporting information are formatted respectively to obtain the reporting data with the data structure, so that the Browse Set verification module verifies the reporting data.
Wherein, the proto data structure is defined as follows:
browsed _ photo _ list: including a list of identifications of the photo IDs that need to be verified. For on-line data sampling, one request per sample is written as a group. For white list users, in order to reduce the service pressure of the online Browse Set, aggregation may be performed by user ID on a daily basis, for example, one user's one-day report information may be aggregated into one proto. That is, a proto (report data) is report data of a user in one day, and a proto includes a plurality of sets of data, each set of data includes a plurality of photo (sub data), and each photo has a unique identifier, which is denoted as a "sub data identifier" (photo ID).
extension _ info: an extension information field storing unique information of the data source. For on-line data sampling, a configurable extended information field is added. In addition, the server show is proto format data, which contains caller information, and the system supports adding configurable extended information fields, so that a field path of the server show is added into a configuration file, and the system can read a value of a corresponding field from the proto through a reflection mechanism of the proto and write the value into extension _ info, so that statistics can be performed based on the extension _ info in subsequent verification.
It should be noted that, in addition to converting the report information into the report data of the proto data structure, the report information may also be converted into other custom data structures, for example, the administrator may preset the data structure to be converted and the corresponding conversion method. Therefore, the embodiment provides a general data structure and a self-defined data structure, and ensures the degree of freedom of expansion while being compatible with personalized data of various different services.
Further, after the verification center formats the target report information to obtain the report data, the verification center can verify the report data to obtain a verification result.
In this embodiment, the verifying the reported data to obtain a verification result includes:
acquiring corresponding service data from the generated online data based on a first preset field in the reported data;
and detecting whether each subdata in the at least one subdata belongs to the service data or not to obtain a detection result corresponding to each subdata in the at least one subdata.
Specifically, the verification center may obtain service data of a service requested by a user from the generated online data according to a first preset field in the reported data. Wherein, the generated online data can be all generated Browse Set; the first preset field records the position information of the service data, so that the verification center can acquire the service data of the service requested by the user from the Browse Set according to the position information.
Further, since the reported data includes at least one subdata, it is possible to detect whether each subdata belongs to the service data, thereby obtaining a detection result corresponding to each subdata.
The service data comprises an identification list, wherein the identification list comprises at least one subdata identification;
the detecting whether each sub data in the at least one sub data belongs to the service data to obtain a detection result corresponding to each sub data in the at least one sub data includes:
matching the subdata identifier of any subdata in the at least one subdata with the identifier list;
if the matching item exists, judging that any subdata belongs to the service data, and generating a first detection result of successful verification;
and if the matching item does not exist, judging that any subdata does not belong to the service data, and generating a second detection result of verification failure.
Specifically, the service data includes an identifier list, and the identifier list includes at least one subdata identifier, so that, for any subdata in the reported data, the subdata identifier of the subdata can be matched with all subdata identifiers in the identifier list, if a matching item exists, it is determined that the subdata belongs to the service data, and a first detection result of successful verification is generated; if the matching item does not exist, judging that any subdata does not belong to the service data, and generating a second detection result of verification failure. And repeating the steps to obtain the detection result of each subdata in the reported data, and taking the detection result of each subdata as the final verification result of the reported data.
Further, referring to fig. 5, a third schematic diagram of a structural framework of the Browse Set verification system of the embodiment is shown, in which the Browse Set verification module includes a Browse Set calling module, a multidimensional statistics module, and a dotting reporting module. Furthermore, as described above, the Browse Set component provides Browse Set SDK verification, because the Browse Set SDK has determination logic, and the Browse Set component may include the Browse Set SDK, so that when the verification center detects sub-data, the Browse Set calling module may call the Browse Set SDK in the Browse Set component to detect each sub-data.
Furthermore, after the verification result of the reported data is obtained, the reported data is combined with the verification result, so that the number of the sub-data which is successfully verified and the number of the sub-data which is failed to be verified in the reported data can be determined, and the statistical data for detection is generated according to the number of the sub-data and the number of the sub-data.
In this embodiment, the generating statistical data based on the reported data and the verification result includes:
combining the verification result with the reported data based on a second preset field to obtain at least one group of target reported data; any one group of target reported data in the at least one group of target reported data comprises at least one subdata;
respectively counting a first number of subdata with detection results being successful detection in each group of target reported data and a second number of subdata with detection results being failed detection;
and taking the first quantity and the second quantity corresponding to each group of target reported data as statistical data.
Specifically, the verification result and the reported data each include a corresponding field, and the field of the verification result and the field of the reported data may have partial duplication, so the multidimensional statistics module can combine the verification result and the reported data based on the duplicated field (marked as a "second preset field") to obtain at least one set of target reported data. The set of target reporting data comprises at least one piece of sub-data, and each piece of sub-data comprises a second preset field.
Then, for any group of target reported data, the multidimensional counting module counts the number of subdata with detection results of successful detection, which is recorded as a first number, and the number of subdata with detection results of failed detection, which is recorded as a second number, and so on, i.e. the first number and the second number corresponding to each group of target reported data can be counted, and all the first number and the second number are taken as final statistical data.
Further, after the statistical data are obtained, a dotting reporting module can be adopted to report the statistical data by dotting. Specifically, the dotting reporting module supports configuration of self-defined multi-dimensional reporting, an administrator can configure field names needing dotting in advance, and the dotting reporting module can obtain corresponding fields from extension _ info according to the field names when dotting reporting is performed. For the indirect field, the info stored in the extension _ info field can be automatically parsed, and the field needing dotting is obtained from json. After the field is determined, the statistical data can be reported to a preset database management system, such as clickhouse, according to the determined field, for subsequent use by the monitoring alarm module.
In step S13, the online data generated by the online recommendation service is detected based on the statistical data, and a detection result is obtained.
After the dotting reporting module performs dotting reporting on the statistical data, the monitoring alarm module can acquire the statistical data from the database management system, and perform accuracy detection on the service based on the statistical data to obtain a service detection result of whether the service accuracy is abnormal or not.
In this embodiment, the detecting the service based on the statistical data to obtain a service detection result includes:
calculating at least one dimension analysis data corresponding to each preset feature in at least one preset feature by adopting the statistical data;
detecting whether the analysis data of any dimension of the at least one dimension exceeds an analysis data threshold of any dimension;
and if the analysis data of any dimension in the at least one dimension does not exceed the analysis data threshold of any dimension, judging that the on-line data generated by the on-line recommendation service is normal.
Specifically, the administrator may preset a plurality of features to be detected, including but not limited to a requested service, a requested platform, an APP version, and the like, and a dimension of analysis data corresponding to each feature, including but not limited to an average value, a maximum value, a geometric increase value, a ring geometric increase value, and the like. Therefore, when the monitoring alarm module detects, the analysis data of each feature in each dimension can be calculated by adopting statistical data, for example, the analysis data of four dimensions such as an average value, a maximum value, a same-ratio growth value and a ring-ratio growth value of sub-data verification failure in reported data in an APP version are calculated by adopting statistical data.
Then, whether the analysis data of each dimension exceeds the respective corresponding analysis data threshold is detected, for example, whether the average value of the sub-data verification failure in the reported data exceeds the average value threshold, whether the maximum value exceeds the maximum value threshold, whether the same-ratio growth value exceeds the same-ratio growth value threshold, and whether the ring-ratio growth value exceeds the ring-ratio growth value threshold under the APP version is detected. If the analysis data of each dimension under each characteristic does not exceed the analysis data threshold of each dimension, judging that the service requested by the user is normal; otherwise, judging that the service requested by the user is abnormal, and carrying out alarm processing by the monitoring alarm module. Therefore, through multi-dimensional detailed monitoring and alarming, problems can be found rapidly and alarming is carried out, and the detection timeliness is greatly improved.
Further, after the analysis data of at least one dimension corresponding to each feature is obtained through calculation, the analysis data may be visualized, for example, drawn into a graph, a bar graph, or the like, for viewing.
In this embodiment, the method further includes:
and responding to the access request, and accessing the online data through a preset data interface.
Specifically, referring to fig. 6, a fourth structural framework diagram of the Browse Set verification system of the present embodiment is shown, where the verification center further includes a storage review module, and the Browse Set further includes an underlying storage. The storage review module is provided with a preset data interface, such as a web interface, all browser sets (namely generated online data) are stored in the bottom-layer storage, and after an alarm occurs, an administrator can directly access all browser sets through the web interface to perform abnormal positioning, so that access through a browser Set SDK is not needed, and the problem of low accuracy of abnormal positioning caused by bug interference access of the browser Set SDK is solved.
In this embodiment, the Browse Set verification system determines target report information to be verified from the report information; the reporting information is generated based on a service request aiming at the online recommendation service; then, generating statistical data for detection based on the target report information, and detecting the online data generated by the online recommendation service based on the statistical data to obtain a detection result of whether the online data generated by the online recommendation service is abnormal or not. Therefore, before the Browse Set component receives the problem feedback of the user, the accuracy of the Browse Set can be quickly and accurately detected by the method, the problems that the accuracy of the Browse Set is slow to find and difficult to find due to the problem feedback of the user are solved, the timeliness, the stability and the accuracy of detection are greatly improved, and the user experience is improved.
Fig. 7 is a block diagram illustrating a traffic data processing apparatus according to an example embodiment. Referring to fig. 7, the apparatus includes a data acquisition unit 71, a statistical data generation unit 72, and a monitoring unit 73.
A data obtaining unit 71, configured to determine target report information to be verified from the report information; the reporting information is generated based on a service request aiming at the online recommendation service;
a statistical data generating unit 72 configured to generate statistical data for detection based on the target report information;
and the monitoring unit 73 is configured to detect the online data generated by the online recommendation service based on the statistical data, and obtain a detection result.
Optionally, the statistical data generating unit includes:
the formatting subunit is configured to perform format conversion on the target reporting information to obtain reporting data;
the calling subunit is configured to verify the reported data to obtain a verification result;
a multidimensional statistics subunit configured to generate statistics data for detection based on the reported data and the verification result.
Optionally, the data obtaining unit includes:
the on-line data sampling subunit is configured to acquire first reporting information from the reporting information according to a preset sampling rate;
the white list data subunit is configured to acquire second reporting information from the reporting information according to a preset white list;
the first processing subunit is configured to use the first reporting information and the second reporting information as target reporting information to be verified.
Optionally, the formatting unit is specifically configured to:
formatting first reporting information and second reporting information in the target reporting information respectively by adopting a preset data structure to obtain reporting data with the preset data structure; the preset data structure comprises a standard data structure and/or a custom data structure.
Optionally, the reported data includes at least one piece of subdata;
the calling unit comprises:
the service data subunit is configured to acquire corresponding service data from online data based on a first preset field in the reported data; the first preset field is used for recording the position information of the service data;
a first detecting subunit, configured to detect whether each sub data in the at least one sub data belongs to the service data, to obtain a detection result corresponding to each sub data in the at least one sub data.
Optionally, any sub-data in the at least one sub-data has a corresponding sub-data identifier; the service data comprises an identification list, and the identification list comprises at least one subdata identification;
the first detection subunit is specifically configured to:
matching the subdata identifier of any subdata in the at least one subdata with the identifier list; if the matching item exists, judging that any subdata belongs to the service data, and generating a first detection result of successful verification; and if the matching item does not exist, judging that any subdata does not belong to the service data, and generating a second detection result of verification failure.
Optionally, the multidimensional statistic unit includes:
the splicing subunit is configured to splice the verification result and the reported data based on a second preset field to obtain at least one group of target reported data; any one group of target reported data in the at least one group of target reported data comprises at least one subdata; the second preset field is a repeated field in the verification result and the reported data;
the statistical subunit is configured to respectively count a first number of the subdata with detection results being successful detection in each group of target reported data, and count a second number of the subdata with detection results being failed detection;
and the second processing subunit is configured to use the first quantity and the second quantity corresponding to each group of target report data as statistical data.
Optionally, the monitoring unit includes:
the calculation subunit is configured to calculate, by using the statistical data, analysis data of at least one dimension corresponding to each preset feature in at least one preset feature;
a second detection subunit configured to detect whether the analysis data of any dimension of the at least one dimension exceeds an analysis data threshold of the any dimension;
the judging subunit is configured to judge that the online data generated by the online recommendation service is normal if the analysis data of any dimension of the at least one dimension does not exceed the analysis data threshold of any dimension.
Optionally, the method further includes:
and the access unit is configured to access the online data through a preset data interface.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 8 is a block diagram illustrating an apparatus 800 for business data processing in accordance with an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.
Referring to fig. 8, the apparatus 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.
The processing component 802 generally controls overall operation of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
The memory 804 is configured to store various types of data to support operation at the device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
Power components 806 provide power to the various components of device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the apparatus 800.
The multimedia component 808 includes a screen that provides an output interface between the device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the device 800. For example, the sensor assembly 814 may detect the open/closed state of the device 800, the relative positioning of the components, such as a display and keypad of the apparatus 800, the sensor assembly 814 may also detect a change in position of the apparatus 800 or a component of the apparatus 800, the presence or absence of user contact with the apparatus 800, orientation or acceleration/deceleration of the apparatus 800, and a change in temperature of the apparatus 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 816 is configured to facilitate communications between the apparatus 800 and other devices in a wired or wireless manner. The apparatus 800 may access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 804 comprising instructions, executable by the processor 820 of the device 800 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims (10)

1. A method for processing service data is characterized by comprising the following steps:
determining target reporting information to be verified from the reporting information; the reporting information is generated based on a service request aiming at the online recommendation service;
generating statistical data for detection based on the target reporting information;
and detecting the online data generated by the online recommendation service based on the statistical data to obtain a detection result.
2. The method of claim 1, wherein the generating statistical data for detection based on the target reporting information comprises:
carrying out format conversion on the target reporting information to obtain reporting data;
verifying the reported data to obtain a verification result;
and generating statistical data for detection based on the reported data and the verification result.
3. The method of processing service data according to claim 1, wherein the determining the target reporting information to be verified from the reporting information includes:
acquiring first reported information from the reported information according to a preset sampling rate;
acquiring second reported information from the reported information according to a preset white list;
and taking the first reporting information and the second reporting information as target reporting information to be verified.
4. The method according to claim 2 or 3, wherein the performing format conversion on the target reporting information to obtain the reporting data includes:
formatting first reporting information and second reporting information in the target reporting information respectively by adopting a preset data structure to obtain reporting data with the preset data structure; the preset data structure comprises a standard data structure and/or a custom data structure.
5. The method of claim 2, wherein the report data includes at least one sub data;
the verifying the reported data to obtain a verification result comprises:
acquiring corresponding service data from online data based on a first preset field in the reported data; the first preset field is used for recording the position information of the service data;
and detecting whether each subdata in the at least one subdata belongs to the service data or not to obtain a detection result corresponding to each subdata in the at least one subdata.
6. The method according to claim 5, wherein any sub-data of the at least one sub-data has a corresponding sub-data identifier; the service data comprises an identification list, and the identification list comprises at least one subdata identification;
the detecting whether each sub data in the at least one sub data belongs to the service data to obtain a detection result corresponding to each sub data in the at least one sub data includes:
matching the subdata identifier of any subdata in the at least one subdata with the identifier list;
if the matching item exists, judging that any subdata belongs to the service data, and generating a first detection result of successful verification;
and if the matching item does not exist, judging that any subdata does not belong to the service data, and generating a second detection result of verification failure.
7. A service data processing apparatus, comprising:
the data acquisition unit is configured to determine target report information to be verified from the report information; the reporting information is generated based on a service request aiming at the online recommendation service;
a statistical data generating unit configured to generate statistical data for detection based on the target report information;
and the monitoring unit is configured to detect the online data generated by the online recommendation service based on the statistical data to obtain a detection result.
8. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the traffic data processing method according to any of claims 1 to 6.
9. A computer-readable storage medium, in which instructions, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the traffic data processing method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program or computer instructions, characterized in that the computer program or computer instructions, when executed by a processor, implement the business data processing method of any one of claims 1 to 6.
CN202111584706.3A 2021-12-22 2021-12-22 Business data processing method and device, electronic equipment and computer storage medium Pending CN114417133A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111584706.3A CN114417133A (en) 2021-12-22 2021-12-22 Business data processing method and device, electronic equipment and computer storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111584706.3A CN114417133A (en) 2021-12-22 2021-12-22 Business data processing method and device, electronic equipment and computer storage medium

Publications (1)

Publication Number Publication Date
CN114417133A true CN114417133A (en) 2022-04-29

Family

ID=81267400

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111584706.3A Pending CN114417133A (en) 2021-12-22 2021-12-22 Business data processing method and device, electronic equipment and computer storage medium

Country Status (1)

Country Link
CN (1) CN114417133A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114885013A (en) * 2022-05-06 2022-08-09 北京达佳互联信息技术有限公司 Method and device for reporting package information, electronic equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114885013A (en) * 2022-05-06 2022-08-09 北京达佳互联信息技术有限公司 Method and device for reporting package information, electronic equipment and storage medium
CN114885013B (en) * 2022-05-06 2024-03-12 北京达佳互联信息技术有限公司 Method and device for reporting package information, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
JP6240363B2 (en) Number marking method, apparatus, program, and recording medium
CN105303412A (en) Method, device and system for layout of content items
CN113259226B (en) Information synchronization method and device, electronic equipment and storage medium
CN105446854B (en) Consumption detection method, system and terminal
CN114417133A (en) Business data processing method and device, electronic equipment and computer storage medium
CN106506808B (en) Method and device for prompting communication message
CN111079040B (en) Resource sniffing method, device, terminal, server and storage medium
CN113420205B (en) Method and device for determining shared source account, terminal equipment and server
CN110764847A (en) User information processing method and device, electronic equipment and storage medium
CN115314426A (en) Data acquisition method, system, electronic device and storage medium
CN113067757B (en) Information transmission and storage method, device and medium
CN111988669B (en) Video interaction data processing method and device, electronic equipment and storage medium
CN114239531A (en) Template recommendation method and device, electronic equipment and storage medium
CN104536966B (en) Data uploading method, data receiver method, client and server
CN113849638A (en) Information processing method, information processing device, electronic equipment and storage medium
CN112819061A (en) Password information identification method, device, equipment and storage medium
CN107517128B (en) Data transmission method, device and equipment
CN112486770A (en) Client dotting reporting method and device, electronic equipment and storage medium
CN111526084A (en) Information processing method, device, equipment and storage medium
CN111143184B (en) Data processing method, device and system
CN113206772B (en) Method, device, equipment, medium and product for judging correctness of response message
CN113473233B (en) Log splicing method and device, electronic equipment, storage medium and product
CN117170958A (en) Index acquisition method, device, electronic equipment and computer readable storage medium
CN117009177A (en) Information processing method and device, electronic equipment and storage medium
CN114385380A (en) Message processing method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination