CN116069838A

CN116069838A - Data processing method, device, computer equipment and storage medium

Info

Publication number: CN116069838A
Application number: CN202111289099.8A
Authority: CN
Inventors: 李娜; 朱兰; 漆洪凯; 黄泽涛; 许炳相
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-11-02
Filing date: 2021-11-02
Publication date: 2023-05-05

Abstract

The embodiment of the application discloses a data processing method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: when the service data to be tested is obtained, the initial data format of the service data to be tested is identified; acquiring a protocol processing class associated with the initial data format based on a mapping relation table associated with the service data to be tested, and analyzing the service data to be tested based on the protocol processing class to obtain a field to be standardized; based on a standardized protocol associated with a standard data format, carrying out standardized processing on a field to be standardized to obtain standardized data information with the standard data format; and determining standard service data corresponding to the service data to be tested based on the standardized data information. By adopting the embodiment of the application, the unification of the data formats can be realized, so that the complexity of data processing is reduced.

Description

Data processing method, device, computer equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, a data processing device, a computer device, and a storage medium.

Background

In the traditional data reporting scheme, sources of reporting data are different, data formats of service data are different, and natural reporting modes are different. Based on the above, when the data reporting terminal obtains the service data to be reported to the data acquisition and exchange platform, the data reporting terminal can report the service data with different data formats according to the reporting logic of the data reporting terminal. Because of excessive data reporting protocols and various data formats, the data caliber of the reported service data is difficult to be kept consistent, so that the complexity of data processing is increased.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device, computer equipment and a storage medium, which can realize unification of data formats so as to reduce complexity of data processing.

An aspect of an embodiment of the present application provides a data processing method, including:

when the service data to be tested is obtained, the initial data format of the service data to be tested is identified;

based on a mapping relation table associated with the service data to be tested, acquiring a protocol processing class associated with an initial data format, and analyzing the service data to be tested based on the protocol processing class to obtain a field to be standardized;

Based on a standardized protocol associated with a standard data format, carrying out standardized processing on a field to be standardized to obtain standardized data information with the standard data format;

and determining standard service data corresponding to the service data to be tested based on the standardized data information.

An aspect of an embodiment of the present application provides a data processing apparatus, including:

the data format identification module is used for identifying the initial data format of the service data to be tested when the service data to be tested is acquired;

the analysis processing module is used for acquiring a protocol processing class associated with the initial data format based on a mapping relation table associated with the service data to be tested, and analyzing the service data to be tested based on the protocol processing class to obtain a field to be standardized;

the standardized processing module is used for carrying out standardized processing on the field to be standardized based on a standardized protocol associated with the standard data format to obtain standardized data information with the standard data format;

and the standard data determining module is used for determining standard service data corresponding to the service data to be tested based on the standardized data information.

Wherein the apparatus further comprises:

The test program starting module is used for starting a test program for testing the service data and starting the service receiver based on the test program;

the data receiving module is used for receiving the service data which is used for reporting to the data acquisition and exchange platform through the service receiver, and taking the received service data as the service data to be reported;

the message queue acquisition module is used for determining a service theme corresponding to the service data to be reported and acquiring a distributed message queue associated with the service theme from the data acquisition and exchange platform;

the data adding module is used for adding the service data to be reported to the distributed message queue; the service data to be reported comprises the service data to be tested.

Wherein, the data format recognition module includes:

the consumption request acquisition unit is used for acquiring a data consumption request carrying a service identifier when acquiring the consumption right aiming at the data acquisition and exchange platform; the data acquisition and exchange platform comprises N business data; n is a positive integer;

the data query unit is used for querying service data corresponding to the service identifier from the N service data;

the consumption filtering unit is used for carrying out consumption filtering on the queried service data according to the consumption identifier indicated by the consumption authority when the service data corresponding to the service identifier is queried, so as to obtain the service data to be tested;

The format identification unit is used for identifying the data format of the service data to be tested and taking the identified data format as an initial data format.

The analysis processing module comprises:

the storage protocol acquisition unit is used for acquiring a data storage protocol associated with the initial data format, and determining a service theme of the service data to be tested and a service interface corresponding to the service data to be tested based on the data storage protocol;

the mapping table acquisition unit is used for acquiring a mapping relation table associated with the service data to be tested, and searching a protocol processing class with a mapping relation with the service theme and the service interface from the mapping relation table;

and the analysis processing unit is used for analyzing the service data to be tested based on the protocol processing class to obtain the field to be standardized.

The service data to be tested has a target object identifier;

the standard data determination module includes:

the real-time verification result determining unit is used for acquiring a legal identification list for storing legal object identifications, and carrying out real-time verification on standardized data information based on the legal identification list and the target object identifications to obtain a real-time verification result;

the coarse granularity verification result determining unit is used for performing coarse granularity verification on a first key field in the standardized data information based on the standardized protocol to obtain a coarse granularity verification result;

The verification result determining unit is used for determining a verification result corresponding to the standardized data information based on the real-time verification result and the coarse granularity verification result;

the standard data determining unit is used for taking the standardized data information as standard service data corresponding to the service data to be tested when the verification result indicates that the verification is successful;

and the data deleting unit is used for deleting the standardized data information when the verification result indicates that the verification fails.

Wherein the real-time check result determining unit includes:

a legal identifier list obtaining subunit, configured to obtain a legal identifier list for storing M legal object identifiers; m is a positive integer;

the identification searching subunit is used for searching the target object identification from M legal object identifications to obtain a searching result;

and the first real-time result determining subunit is used for obtaining a real-time verification result for indicating successful verification if the search result indicates that the legal object identifier matched with the target object identifier exists in the M legal object identifiers.

And the second real-time result determining subunit is used for obtaining a real-time verification result for indicating verification failure if the search result indicates that the legal object identifier matched with the target object identifier does not exist in the M legal object identifiers.

Wherein the coarse grain verification result determining unit includes:

the key value acquisition subunit is used for acquiring a first key value corresponding to a first key field in the standardized data information;

an enumeration value interval obtaining subunit, configured to obtain an enumeration value interval corresponding to the first key field from the standardized protocol;

the comparison unit is used for comparing the first key value with each enumeration value in the enumeration value interval corresponding to the first key field;

and the coarse granularity result determining subunit is configured to obtain a coarse granularity verification result that indicates that the coarse granularity verification is successful if an enumeration value that matches the first key value exists in the enumeration value interval corresponding to the first key field.

Wherein the apparatus further comprises:

the data storage module is used for taking the standard service data stored in the offline database as service data to be checked when the standard service data are stored in the offline database for offline checking;

and the rule configuration module is used for configuring the check rule for the second key field in the business data to be checked and storing the check rule into the rule base.

Wherein the apparatus further comprises:

the offline analysis module is used for acquiring the service data to be checked from the offline database, performing offline analysis on the service data to be checked, and determining a second key value corresponding to a second key field in the service data to be checked;

The rule acquisition module is used for acquiring the verification rule configured for the second key field from a rule base associated with the business data to be verified; the check rule includes a configuration value;

and the offline verification result determining module is used for determining the offline verification result of the business data to be verified based on the second key value and the configuration value.

Wherein the apparatus further comprises:

the standard data sending module is used for sending the standard service data to the management terminal so that the management terminal analyzes the standard service data according to a standardized protocol to obtain a key field which is used for being displayed on a terminal interface of the management terminal; the key field on the terminal interface is used for indicating the management object corresponding to the management terminal to carry out service detection.

In one aspect, a computer device is provided, including: a processor and a memory;

the processor is connected to the memory, wherein the memory is configured to store a computer program, and when the computer program is executed by the processor, the computer device is caused to execute the method provided in the embodiment of the application.

In one aspect, the present application provides a computer readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in the embodiments of the present application.

In one aspect, the present application provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the embodiments of the present application.

In the embodiment of the application, when the computer equipment with the format standardization function acquires the service data to be tested, the initial data format of the service data to be tested can be identified. The data formats of the reported data are different due to different reporting modes of the service data to be tested, which are acquired by the computer equipment. Further, the computer device may obtain a protocol processing class associated with the initial data format based on the mapping relation table associated with the service data to be tested, and may further perform parsing processing on the service data to be tested based on the protocol processing class, so as to obtain a field to be standardized. At this time, the computer device may perform standardization processing on the field to be standardized based on a standardization protocol associated with the standard data format, so as to obtain standardized data information having the standard data format, and further may determine standard service data corresponding to the service data to be tested based on the standardized data information. It can be seen that the computer device can perform unified data processing on the service data to be tested in various different data formats based on the standardized protocol associated with the standard data format, so as to obtain the standard service data. Since the data format of the standard service data is the standard data format, this means that the computer device can achieve unification of the data formats, so that the complexity of data processing is reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic structural diagram of a network architecture according to an embodiment of the present application;

fig. 2 is a schematic view of a scenario in which service data to be tested is subjected to standardization processing according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of a data processing method according to an embodiment of the present application;

fig. 4 is a schematic flow chart of data reporting according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 6 is a schematic flow chart of establishing a data path according to an embodiment of the present application;

FIG. 7 is a block diagram of test service data according to an embodiment of the present application;

FIG. 8 is a schematic flow chart of a data test according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

fig. 10 is a schematic diagram of a computer device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

Referring to fig. 1, fig. 1 is a schematic structural diagram of a network architecture according to an embodiment of the present application. As shown in fig. 1, the network architecture may include a server 10F and a cluster of user terminals. The cluster of user terminals may comprise one or more user terminals, the number of which will not be limited here. As shown in fig. 1, the

user terminals

100a, 100b, 100c, …, and 100n may be specifically included. As shown in fig. 1, the

user terminals

100a, 100b, 100c, …, 100n may respectively perform network connection with the server 10F, so that each user terminal may perform data interaction with the server 10F through the network connection. The network connection is not limited to a connection manner, and may be directly or indirectly connected through a wired communication manner, may be directly or indirectly connected through a wireless communication manner, or may be other manners, which is not limited herein.

Wherein each user terminal in the user terminal cluster may include: smart terminals with business data processing functions such as smart phones, tablet computers, notebook computers, desktop computers, smart speakers, smart watches, vehicle-mounted terminals, smart televisions and the like. It should be appreciated that each user terminal in the user terminal cluster shown in fig. 1 may be provided with a target application (i.e. application client), which may interact with the server 10F shown in fig. 1, respectively, when the application client is running in each user terminal. The application client may include a social client, a multimedia client (e.g., a video client), an entertainment client (e.g., a game client), an educational client, a live client, etc. with a service data processing function. The application client may be an independent client, or may be an embedded sub-client integrated in a client (for example, a social client, an educational client, and a multimedia client), which is not limited herein.

As shown in fig. 1, the server 10F in the embodiment of the present application may be a server corresponding to the application client. The server 10F may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services.

For easy understanding, in the embodiment of the present application, one user terminal may be selected from the plurality of user terminals shown in fig. 1 as a data reporting terminal. For example, the embodiment of the present application may use the user terminal 100a shown in fig. 1 as a data reporting terminal, where a target application (i.e., an application client) may be integrated in the data reporting terminal. At this time, the data reporting terminal may implement data interaction between the service data platform corresponding to the application client and the server 10F.

For example, the data reporting terminal may report service data (e.g., advertisement data) having various data formats to a data collection and exchange platform (tdbank) through the application client. It is understood that the service type of the service data herein may include service pull data, service exposure data, service click data, service conversion data, service play data, and the like. It should be understood that the reporting sources of the service data are different, and the data formats of the service data are different. The service data may include first service data, second service data, third service data, and fourth service data. For example, the first service data may refer to service data reported by the application client, where a data format of the first service data may include a text format separated by commas. The second service data may refer to service data reported by the application client advertisement, where the data format of the second service data may include text format, comma separation, and Protocol Buffers format (pb format for short). The third service data may refer to service data returned by the advertiser, and the data format of the third service data may include a text format, a JavaScript Object Notation format (json format for short, a lightweight data exchange format). The fourth business data herein may refer to advertisement library static data, which may include advertisement material as well as advertisement plans.

The protocol processing class according to the embodiment of the application refers to a protocol type for analyzing and processing service data to be tested with an initial data format. The protocol processing class may include a first protocol processing class and a second protocol processing class. The first protocol processing class may be a java processing class (for example, a Proto precompiled java class) acquired according to a data storage protocol (for example, a protocol buf protocol), and the second protocol processing class may be a java custom processing class (for example, a KV custom method class) written according to a corresponding data storage protocol (for example, a Key-Value type protocol). The protocol buf protocol is a platform-independent, language-independent, extensible, lightweight and efficient protocol of a serialized data structure, and can be used for network communication and data storage.

In this embodiment of the present application, a computer device having a format standardization function (for example, the server 10F shown in fig. 1) may perform unified processing and processing on the service data (for example, advertisement data having various data formats) that has been reported to the data collection and exchange platform (for example, perform parsing processing on a certain service data to be tested according to a corresponding protocol processing class, and perform standardization processing on the service data to be tested according to a standardized protocol related to a standard data format), so as to obtain standard service data having a standard data format. The computer equipment in the embodiment of the application can uniformly process the data to be tested with various data formats into standard service data with standard data formats, so that the uniformity of the data formats is realized, the understanding cost is reduced, and the complexity of data processing is further reduced.

For easy understanding, further, please refer to fig. 2, fig. 2 is a schematic diagram of a scenario in which service data to be tested is normalized according to an embodiment of the present application. As shown in fig. 2, the computer device with the format normalization function in the embodiment of the present application may be the server 10F shown in fig. 1, or any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100b.

The data collection and exchange platform 20P shown in fig. 2 may be configured to store N service data, where N is a positive integer. For example, the data collection switching platform 20P may include business data S ₁ Service data S ₂ Service data S ₃ …, service data S _N . It will be appreciated that the traffic dataThe reporting sources are different, the data formats of the service data are different, and the reporting modes are different. For example, business data S ₁ The data format of (2) may be text format, separated by commas; service data S ₂ The data format of (a) may be pb format; service data S ₁ The data format of (a) may be json format; …; service data S _N The data format of (c) may be in text format, separated by commas.

It should be understood that the computer device may acquire certain service data from the data acquisition and exchange platform 20P for performing data testing, and may further perform data testing on the acquired service data (e.g., service data S shown in fig. 2) ₁ ) As the traffic data to be tested. At this time, the computer device can process the service data S ₁ The data format of (2) is identified, and the identified text format can be used as the service data S ₁ Is used to determine the initial data format of the data.

Further, the computer device can acquire and service data S ₁ The mapping table (e.g., the mapping table 20X shown in fig. 2) associated with the service data to be tested, and further, the protocol processing class associated with the initial data format may be acquired based on the mapping table 20X. For example, the computer device, when determining the protocol processing class, may be based on the data S ₁ Data storage protocol associated with the initial data format of (e.g., key-Value type protocol associated with text format) to determine a lookup parameter (e.g., business data S) for looking up a corresponding protocol processing class ₁ Is a business topic and business data S ₁ Based on the two determined lookup parameters), a protocol processing class (i.e., the protocol processing class 1L shown in fig. 2, for example, a java custom processing class) having a mapping relationship with the two lookup parameters may be determined in the mapping relationship table 20X shown in fig. 2. Wherein a traffic topic (topic) is a distributed, data write operation unit based on a publish/subscribe messaging system (e.g., kafka), a traffic topic may correspond to a message queue. Wherein the service interface (tid) is a service interface name defined in the data acquisition and exchange platform.

Further, the computer device can process the class 1L based on the protocol, and process the service data S ₁ And performing analysis processing so as to obtain the field to be standardized. The computer device may then obtain a standardized protocol associated with a standard data format (e.g., a standard pb format), and may then perform a normalization process on the fields to be normalized based on the standardized protocol, such that standardized data information having the standard data format may be obtained. For example, the computer device may combine fields to be normalized based on a normalization protocol to generate normalized data information having a standard data format (e.g., binary data having a standard pb format). Herein, the standard pb format refers to a new data format redefined by a development object (for example, a development user) for an original pb format.

It will be appreciated that the computer device may determine the service data S based on standardized data information ₁ Corresponding standard service data (e.g., service data B shown in fig. 2 ₁ ). Wherein the computer device can directly take the standardized data information as service data S when generating the standardized data information with standard data format ₁ Corresponding standard service data. Optionally, the computer device may perform preliminary verification on the standardized data information when generating the standardized data information having the standard data format, and use the standardized data information as the service data S when the verification is successful ₁ Corresponding standard service data. The preliminary verification herein may include, among other things, a real-time checksum coarse-grained verification. The real-time verification herein refers to verifying standardized data information based on a legal identification list for storing legal object identifications (i.e., legal user identifications). Coarse-grained verification herein refers to verifying key fields (e.g., common protocol fields such as account information, system information, network information, etc.) in standardized data information based on a standardized protocol. The method means that the computer equipment can effectively identify reporting problems of missing data information, dirty data, repeated data and the like, and further can effectively ensure the integrity of the reported data and improve the accuracy of the data.

Similarly, the computer device may also obtain another service data (e.g., service data S) from the data collection switching platform 20P shown in FIG. 2 ₂ ) As the service data to be tested, the service data S can be used as the service data to be tested ₂ For the service data S ₂ Analyzing to obtain service data S ₂ In turn, the service data S may be based on a standardized protocol associated with a standard data format ₂ The field to be standardized of the (2) is standardized so as to be convenient for determining service data S based on standardized data information obtained after the standardized processing ₂ Standard service data of (2). And so on, the computer device may process all the service data in the data collection and exchange platform 20P to obtain the standard service data corresponding to each. Wherein the computer device is used for processing business data S ₂ Specific embodiments of data processing can be seen from the above description of the service data S ₁ Specific embodiments of data processing will not be described herein.

It can be seen that the computer device in the embodiment of the present application recognizes the service data S ₁ To-be-tested service data of the initial data format) of the data processing system, the service data B with the standard data format can be obtained by performing data processing on the to-be-tested service data ₁ (i.e. standard service data corresponding to the service data to be tested), based on which the computer device can collect all service data in the switching platform 20P according to the service data S ₁ When the processing mode is used for data processing, the unification of the data formats can be realized, so that the complexity of the data processing is reduced.

The specific implementation manner of the computer device for performing data processing on the service data to be tested with the initial data format through the standardized protocol associated with the standard data format to obtain the standard service data with the standard data format can be seen in the embodiments corresponding to fig. 3 to 8 below.

Further, referring to fig. 3, fig. 3 is a flow chart of a data processing method according to an embodiment of the present application. As shown in fig. 3, the method may be performed by a computer device having a format normalization function, and the computer device may be a user terminal (e.g., the user terminal 100b shown in fig. 1) or a server (e.g., the server 10F shown in fig. 1), which is not limited herein. For ease of understanding, embodiments of the present application will be described with the method being performed by a server as an example, and the method may include at least the following steps S101 to S104:

step S101, when the service data to be tested is obtained, the initial data format of the service data to be tested is identified.

Specifically, when the computer device with the format standardization function obtains the consumption right for the data collection exchange platform, the computer device can obtain the data consumption request carrying the service identifier (for example, service ID). The data acquisition and exchange platform can comprise N business data; and N is a positive integer. Further, the computer device may query the N service data for service data corresponding to the service identifier. When service data corresponding to the service identifier is queried, the computer equipment can perform consumption filtering on the queried service data according to the consumption identifier (for example, consumption target service ID) indicated by the consumption authority so as to obtain service data to be tested. Further, the computer device may identify a data format of the service data to be tested, and take the identified data format as an initial data format.

It should be appreciated that embodiments of the present application may be applied to business scenarios (e.g., advertisement scenarios) associated with application clients, including essentially testing all business data related to client advertisements. Such as clicking, exposing, converting, and returning business data for a plurality of ad slots. It can be understood that when generating service data with various data formats, the data reporting terminal can report the service data to the data acquisition and exchange platform through the application client.

For ease of understanding, further, please refer to fig. 4, fig. 4 is a schematic flow chart of data reporting according to an embodiment of the present application. As shown in fig. 4, the data collection and exchange platform 40P in the embodiment of the present application is mainly used for real-time collection, distribution and configuration management of data, and aims to unify data entry, provide various data access modes to the outside, and distribute data in real time with high efficiency. The data collection and exchange platform 40P is mainly used for acquiring data from a service data source, distributing the acquired data to an offline/online processing platform, constructing a bridge between the data source and a data processing system, and decoupling the data processing system from the data source on the same service side.

It should be appreciated that the business data to be reported into the data collection exchange platform 40P in embodiments of the present application may include object behavior data 41s (e.g., user behavior data) and advertisement library static data 42s. The object behavior data 41s herein may include service data 411s (e.g., client report data), service data 412s (e.g., client advertisement report data), and service data 413s (e.g., advertiser return data), among others. The advertisement library static data 42s herein may include business data 421s (e.g., advertisement material data) and business data 422s (e.g., advertisement plan data). The service types of the service data may include service pull data, service exposure data, service click data and service conversion data. It will be appreciated that a service type may correspond to a service identity. Different reporting sources will cause different data formats of the service data, and different reporting modes of the service data.

For example, the data format of the service data 411s generated by the data generating terminal (e.g., any one of the user terminals in the user terminal cluster shown in fig. 1, e.g., the user terminal 100 a) may include data format 1 (e.g., text format, comma separated). When the service data 411s is reported, the data generating terminal may determine the service type of the service data 411s in response to a trigger operation for the application client. The service data 411s may include two reporting manners. It should be appreciated that when the data generating terminal determines that the service type of the service data 411s is service click data, the data generating terminal may invoke a client data storage service (e.g., a distributed key-value data storage service, simply referred to as a clientKV service). Further, the data generating terminal may send the service data 411s to a data reporting terminal (e.g., client server 4F shown in fig. 4 based on the client data storage service ₂ ) So that the client server 4F ₂ The service data 411s is reported to the data collection and exchange platform 40P shown in fig. 4. Optionally, when the data generating terminal reports the service data 411s, the data generating terminal may respond to a triggering operation for the application client, and further may send the service data 411s to the data collection and exchange platform 40P shown in fig. 4 based on the client background service, which will not be described herein.

For another example, the data format of the service data 412s generated by the data generating terminal includes data format 1 (e.g., text format, comma separated) and data format 2 (e.g., pb format). When the service data 412s is reported, the data generating terminal may determine the service type of the service data 411s in response to a trigger operation for the application client. When the data generating terminal determines that the service type of the service data 412s is any one of the service type of the service pull data, the service exposure data or the service click data, the data generating terminal may be based on transmitting the service data 412s to a corresponding data reporting terminal (e.g., the advertisement service server 4F shown in fig. 4) ₁ ) So that the advertising service server 4F ₁ The service data 412s is reported to the data collection switching platform 40P. The service data 412s may include front-end data, pull data, and general report data.

For another example, the data format of the service data 413s generated by the data generating terminal includes a data format 1 (e.g., text format) and a data format 3 (e.g., json format). When reporting the service data 413s, the data generating terminal may send the service data 413s to the corresponding data reporting terminal (e.g., the partner server 4F shown in fig. 4 ₃ ) So that the partner server 4F ₃ ) The service data 413s is reported to the data collection switching platform 40P.

For another example, the data generating terminal may directly send the advertisement library static data 42s to the data collection exchange platform 40P when creating the generated advertisement library static data 42 s. Therefore, the data mode terminal in the embodiment of the application has complex reporting logic, so that the data caliber is difficult to keep consistent, and the understanding cost is increased. In order to reduce the complexity of data processing, the embodiment of the application can perform data testing on the service data reported to the data acquisition and exchange platform, namely perform data processing on the service data to obtain service data with a uniform data format so as to facilitate subsequent verification.

It should be understood that, before testing the service data reported to the data collection and exchange platform, the computer device in the embodiment of the present application may generate the rights application request for the data collection and exchange platform. Further, the computer device may send the rights application request to an approval terminal having a network connection relationship with the computer device. The approval terminal may be a terminal device having a permission granting function (i.e. granting a consumption permission for a computer device that sends a permission application request), the approval terminal may be another server having a network connection relationship with the computer device, and the approval terminal may also be any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100c.

It may be appreciated that when the approval terminal receives the permission application request, the approval object (for example, a superior leader of the consumption object corresponding to the computer device) may perform the triggering operation for the authorization control on the audit interface of the approval terminal. The triggering operation may include a touch operation such as clicking, long pressing, or a non-touch operation such as voice, gesture, or the like, which will not be limited herein. Further, the approval terminal may generate an authorization notification for returning to the computer device based on the consumption identification in response to the triggering operation. The consumption identifier can be used for indicating the computer equipment to perform consumption filtering on the service data in the data acquisition and exchange platform.

When the computer equipment receives the authorization notification, the computer equipment can be considered to have the consumption right for the data acquisition and exchange platform. At this time, the computer device may obtain a data consumption request carrying a service identifier of the service data to be tested, and may further perform data consumption on the service data in the data acquisition and exchange platform based on the data consumption request. Wherein the data consumption request can be determined by the computer device when responding to the triggering operation of the consumption object for certain service data; alternatively, the data consumption request may be sent by a consumption terminal (e.g., any one of the user terminals shown in fig. 1, e.g., the user terminal 100 a) having a network connection relationship with the computer device, and the source of the data consumption request will not be limited herein. Wherein the data consumption may refer to acquiring/using service data.

Further, the computer device may query the service data corresponding to the service identifier from the N service data in the data collection and exchange platform. When the service data corresponding to the service identifier is queried, the computer equipment can perform consumption filtering on the queried service data according to the consumption identifier indicated by the consumption authority, so that the service data to be tested can be obtained. At this time, the computer device may identify a data format of the service data to be tested, and further may use the identified data format as an initial data format.

As shown in fig. 2, the data collection switching platform 20P may include service data S ₁ Service data S ₂ Service data S ₃ …, service data S _N The N pieces of traffic data. The service data corresponding to the service identifier is inquired in the computer equipment as service data S ₁ When the computer equipment is used, the business data S can be processed according to the consumption identification indicated by the consumption authority ₁ Consumption filtering is performed, so that service data S can be obtained ₁ . At this time, the service data in the data collection and exchange platform 20P may include service data S ₂ Service data S ₃ …, service data S _N These (N-1) pieces of traffic data. Further, the computer device may identify the service data S ₁ Is a data format of (a). Wherein the present applicationThe data format of the service data to be tested may be referred to as an initial data format in the embodiments.

Step S102, based on the mapping relation table associated with the service data to be tested, acquiring a protocol processing class associated with the initial data format, and based on the protocol processing class, analyzing the service data to be tested to obtain a field to be standardized.

Specifically, the computer device may acquire a data storage protocol associated with the initial data format, and further may determine, based on the acquired data storage protocol, a service theme of the service data to be tested and a service interface corresponding to the service data to be tested. Further, the computer device may obtain a mapping relation table associated with the service data to be tested, and may further search a protocol processing class having a mapping relation with the service theme and the service interface from the mapping relation table. At this time, the computer device may analyze the service data to be tested based on the protocol processing class, so as to obtain a field to be standardized corresponding to the service data to be tested.

For ease of understanding, further, please refer to table 1, table 1 is a mapping table provided in the embodiment of the present application.

TABLE 1

The mapping relationship table shown in table 1 is used for quickly searching a protocol processing class of service data to be tested, where the first protocol processing class may be a java processing class (for example, a proco precompiled java class) obtained according to a data storage protocol (for example, a protocol buf protocol), and the second protocol processing class may be a java custom processing class (for example, a KV custom method class) written according to a corresponding data storage protocol (for example, a Key-Value type protocol).

It should be appreciated that if the traffic data to be tested (e.g., traffic data S ₁ ) Is the first data format (e.g., pb format), then the computer device obtains the initial data format associated with the initial data formatThe data storage protocol may be a protocol buf protocol. The service data to be tested having the first data format may be referred to as first type test data in the embodiment of the present application. At this time, the computer device may determine the service data S based on the protocol buf protocol ₁ Is a business topic and business data S ₁ And a corresponding service interface. For example, the service data S ₁ The business topic of (a) can be topic a, business data S ₁ The corresponding service interface may be service interface 1, and then the computer device may find a protocol processing class (for example, a first protocol processing class) having a mapping relationship with the theme a and the service interface 1 from the above table 1, and further may perform the mapping on the service data S based on the first protocol processing class ₁ The analysis processing is carried out, and the business data S after the analysis processing can be further processed ₁ As a field to be normalized.

Alternatively, if the service data to be tested (e.g., service data S ₂ ) The initial data format of (a) is a second data format (e.g., other data formats such as text format), the data storage protocol associated with the initial data format acquired by the computer device may be a Key-Value type protocol. The service data to be tested having the second data format may be referred to as second type test data herein. At this time, the computer device may determine the service data S based on the Key-Value type protocol ₂ Is a business topic and business data S ₂ And a corresponding service interface. For example, the service data S ₂ The business topic of (a) can be topic b, business data S ₂ The corresponding service interface may be the service interface 3, and then the computer device may find the protocol processing class (for example, the second protocol processing class) having a mapping relationship with the theme b and the service interface 3 from the above table 1, and may further perform the mapping on the service data S based on the second protocol processing class ₂ The analysis processing is carried out, and the business data S after the analysis processing can be further processed ₂ As a field to be normalized.

Step S103, based on the standardized protocol associated with the standard data format, the field to be standardized is standardized, and standardized data information with the standard data format is obtained.

In particular, the computer device may combine fields to be standardized based on a standardized protocol associated with a standard data format to obtain standardized data information having the standard data format.

Step S104, based on the standardized data information, determining standard service data corresponding to the service data to be tested.

Specifically, the computer device may verify the standardized data information, and when the verification is successful, determine the standardized data information as standard service data corresponding to the service data to be tested. The verification herein may include, among other things, real-time verification as well as coarse-grained verification. For example, the computer device may obtain a legal identifier list for storing legal object identifiers, and perform real-time verification on standardized data information based on the legal identifier list and a target object identifier (user information, abbreviated as uin) of service data to be tested, so as to obtain a real-time verification result. Meanwhile, the computer equipment can perform coarse-granularity verification on key fields in standardized data information based on a standardized protocol so as to obtain a coarse-granularity verification result. In this embodiment of the present application, the key field in the standardized data information may be referred to as a first key field, and the key value corresponding to the first key field may be referred to as a first key value. Further, the computer device may determine a verification result corresponding to the standardized data information based on the real-time verification result and the coarse-granularity verification result. The computer device may delete the standardized data information when the verification result indicates that the verification fails. Optionally, when the verification result indicates that the verification is successful, the computer device may use the standardized data information as standard service data corresponding to the service data to be tested. The order in which the computer device determines the real-time verification result and the coarse-grain verification result is not limited herein. The embodiment of the application can take the example of determining the real-time check result and then determining the coarse granularity check result, so as to illustrate the process of checking the standardized data information by the computer equipment.

For ease of understanding, further, please refer to table 2Table 2 is a legal identifier list generated based on a test object provided in an embodiment of the present application. It may be understood that the legal identifier list may refer to a white list generated by the computer device after testing an object identifier corresponding to the test object (i.e. a test account ID corresponding to the test user). In this embodiment of the present application, the object identifier corresponding to the legal object may be referred to as a legal object identifier. As shown in table 2, the legal identifier list may include object identifiers corresponding to M legal objects, where M may be a positive integer. The M legal object identifications may include object identification D ₁ Object identification D ₂ …, object identification D _M 。

TABLE 2

Legal object identification
	Object identification D ₁
Object identification D ₂
	…
Object identification D _M

It should be appreciated that the computer device may obtain the list of legal identifications shown in table 2 above when it obtains standardized data information having a standard data format. Because the service data to be tested carries the target object identifier, the computer equipment can acquire the target object identifier, and then can search the target object identifier from M legal object identifiers in the legal object identifier list shown in the table 2 so as to obtain a search result.

If the search result indicates that the legal object identifier matched with the target object identifier does not exist in the M legal object identifiers, the computer equipment can obtain a real-time verification result for indicating verification failure. At this time, the computer device may delete the standardized data information so as to improve the accuracy of reporting the data.

Optionally, if the search result indicates that a legal object identifier matched with the target object identifier exists in the M legal object identifiers, the computer device may obtain a real-time verification result for indicating that verification is successful. At this time, the computer device may acquire a first key value corresponding to the first key field in the normalized data information. Meanwhile, the computer equipment also needs to acquire an enumeration value interval corresponding to the first key field from the standardized protocol. Further, the computer device may compare the first key value with each of the enumerated values in the enumerated value interval corresponding to the first key field. If the enumeration value interval corresponding to the first key field has an enumeration value matched with the first key value, the computer equipment can obtain a coarse granularity verification result for indicating that the coarse granularity verification is successful. At this time, the computer device may determine that the verification result of the standardized data information is a legal verification result (i.e., a verification result indicating that the verification is successful), and at this time, the computer device may use the standardized data information as standard service data corresponding to the service data to be tested. Optionally, if the enumeration value interval corresponding to the first key field does not have an enumeration value matching the first key value, the computer device may obtain a coarse granularity verification result for indicating a coarse granularity failure. At this time, the computer device may determine that the verification result of the standardized data information is an illegal verification result (i.e., a verification result indicating that the verification failed), and at this time, the computer device may delete the standardized data information.

The key fields in the embodiments of the present application may refer to common protocol fields such as system information, network information, account information, and the like. For example, in a standardized protocol associated with a standard data format, it may be pre-configured: enumeration value interval Q corresponding to key field of system information ₁ For operating system A, operating system B, operating system C]Network information thisEnumerated value interval Q corresponding to key field ₂ For [ communication technology 1, communication technology 2, communication technology 3]. For example, the communication technology 1 may be a second Generation mobile phone communication technology specification (second Generation, abbreviated as 2G), the communication technology 2 may be a third Generation mobile communication technology (3 rd Generation, abbreviated as 3G), and the communication technology 3 may be a wireless communication technology (Wi-Fi).

For example, for the key field of the system information, the computer device may obtain the key value corresponding to the system information from the standardized data information, and may further obtain the enumeration value interval Q from the standardized protocol associated with the standard data format ₁ . At this time, the computer device can compare the key value with the enumerated value interval Q ₁ Is compared to each enumerated value in (c). If the key value corresponding to the system information is the operating system D, the computer device may determine an enumeration value interval Q ₁ There is no enumerated value that matches the key value, meaning that the computer device may obtain a coarse-grain verification result indicating that the coarse-grain verification failed. If the key value corresponding to the system information is the operating system A, the computer device can determine an enumeration value interval Q ₁ When there is an enumerated value matched with the key value, the computer device may continue to verify the key value corresponding to the key field of the network information until the verification of the common protocol field is completed, so as to obtain a coarse-granularity verification result for indicating that the coarse-granularity verification is successful. It can be appreciated that when the coarse-granularity verification result indicates that the verification is successful, the computer device may use the standardized data information as standard service data corresponding to the service data to be tested.

Further, referring to fig. 5, fig. 5 is a flow chart of a data processing method according to an embodiment of the present application. The method may relate to a computer device having a format normalization function, which may be a user terminal (e.g., the user terminal 100b shown in fig. 1) or a server (e.g., the server 10F shown in fig. 1), and a management terminal having a service detection function, which is not limited herein. The management terminal may be any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100c. For ease of understanding, embodiments of the present application will be described with the method being performed by a server as an example, and the method may include at least the following steps S201-S206:

in step S201, when the computer device with the format standardization function obtains the service data to be tested, the initial data format of the service data to be tested is identified.

In step S202, the computer device obtains a protocol processing class associated with the initial data format based on the mapping relation table associated with the service data to be tested, and performs parsing processing on the service data to be tested based on the protocol processing class, so as to obtain a field to be standardized.

In step S203, the computer device performs normalization processing on the field to be normalized based on the normalization protocol associated with the standard data format, to obtain normalized data information having the standard data format.

In step S204, the computer device determines standard service data corresponding to the service data to be tested based on the standardized data information.

Specifically, the computer device may verify the standardized data information, and when the verification is successful, determine the standardized data information as standard service data corresponding to the service data to be tested. The verification herein may include, among other things, real-time verification as well as coarse-grained verification. For example, the computer device may obtain a legal identifier list for storing legal object identifiers, and perform real-time verification on standardized data information based on the legal identifier list and a target object identifier (user information, abbreviated as uin) of service data to be tested, so as to obtain a real-time verification result. Meanwhile, the computer equipment can perform coarse-granularity verification on key fields in standardized data information based on a standardized protocol so as to obtain a coarse-granularity verification result. In this embodiment of the present application, the key field in the standardized data information may be referred to as a first key field, and the key value corresponding to the first key field may be referred to as a first key value. Further, the computer device may determine a verification result corresponding to the standardized data information based on the real-time verification result and the coarse-granularity verification result. The computer device may delete the standardized data information when the verification result indicates that the verification fails. Optionally, when the verification result indicates that the verification is successful, the computer device may use the standardized data information as standard service data corresponding to the service data to be tested. The order in which the computer device determines the real-time verification result and the coarse-grain verification result is not limited herein.

The specific implementation of the steps S201 to S204 may be referred to the description of the steps S101 to S104 in the embodiment corresponding to fig. 3, and will not be repeated here.

For ease of understanding, further, please refer to fig. 6, fig. 6 is a schematic flow chart for establishing a data path according to an embodiment of the present application. As shown in fig. 6, the computer device in the embodiment of the present application may be a terminal device with a format standardization function, and the terminal device may be the server 10F shown in fig. 1 and may also be any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100b, which will not be limited herein. The process of establishing the data path comprises a data consumption stage, a standardized processing stage and a data filtering stage.

It should be appreciated that the computer device may initiate a test program for testing the service data and the kernel of the computer device may initiate the service receiver based on the test program. When the data reporting terminal acquires the service data (for example, service data S) for reporting to the data acquisition and exchange platform shown in fig. 6 and the object identifier corresponding to the service data S, the service data S and the object identifier corresponding to the service data S may be sent to the computer device together based on the network connection relationship between the data reporting terminal and the computer device. At this time, the computer device may receive the service data S and the object identifier of the service data S, and may further use the received service data S as the service data to be reported.

Further, the computer device may determine a service topic corresponding to the service data to be reported, and may further obtain a distributed message queue associated with the service topic from the data collection and exchange platform. At this point, the computer device may add the traffic data to be reported to the distributed message queue. The service data to be reported here includes service data to be tested.

For example, the data collection exchange platform in the embodiment of the present application may include a plurality of distributed message queues, which may specifically take 3 as an example, for example, distributed message queue 1, distributed message queue 2, and distributed message queue 3. Wherein the distributed message queue 1 may refer to a distributed message queue associated with a service topic a, the distributed message queue 2 may refer to a distributed message queue associated with a service topic b, and the distributed message queue 3 may refer to a distributed message queue associated with a service topic c. If the service theme corresponding to the service data to be reported acquired by the computer equipment is the service theme a, the computer equipment can directly add the data to be reported to the distributed message queue 1, so that the efficiency of data consumption in the data acquisition and exchange platform is improved.

It will be appreciated that in the data consumption stage, the computer device may perform data consumption on the service data to be tested from the data collection and exchange platform shown in fig. 6, and further may add the service data filtered by consumption to the processing queue shown in fig. 6. Further, in the standardized processing stage and the data filtering stage, the computer device may obtain certain service data from the processing queue as service data to be tested. At this time, the computer device may determine, according to the data storage protocol of the to-be-tested service data, a service theme of the to-be-tested service data and a service interface of the to-be-tested service data. At this time, the computer device may search a mapping relationship table for a protocol processing class for parsing the service data to be tested based on the service theme and the service interface. At this time, the computer device may perform a preprocessing operation on the data to be tested based on the protocol processing class, so as to store the standard service data corresponding to the service data to be tested and the verification result into an offline database (for example, a TPG database for performing operation result display and small data size analysis) shown in fig. 6. It can be understood that the standard service data in the embodiment of the present application may be used in the scenarios of service data testing, service effect analysis, service pulling, etc.

It can be understood that, if the service data to be tested is first type test data (for example, data to be tested having pb format), the protocol processing class acquired by the computer device may be a first protocol processing class (for example, proco precompiled java class), and further, the computer device may analyze the first type test data based on the first protocol processing class, so as to obtain a field to be standardized corresponding to the first type test data. If the service data to be tested is the second type data (for example, the data to be tested with a text format), the protocol processing class acquired by the computer device may be the second protocol processing class (for example, KV custom method class), and further, the computer device may analyze the second type test data based on the second protocol processing class, so as to obtain a field to be standardized corresponding to the second type test data.

At this time, when the computer device obtains the field to be standardized, the standardized protocol associated with the standard data format may be obtained, and further, based on the standardized protocol, the field to be standardized may be subjected to standardization processing, so as to obtain standardized data information having the standard data format. Further, the computer device may verify the standardized data information to obtain a verification result.

For example, the computer device may obtain the target object identifier of the test service data, and further may perform real-time verification on the standardized data information based on the target object identifier and the legal identifier list for storing the legal object identifier, so as to obtain a real-time verification result. In addition, the computer device can perform coarse-granularity verification on the first key field in the standardized data information based on the standardized protocol associated with the standard data format, so that a coarse-granularity verification result can be obtained. Further, the computer device may determine a verification result corresponding to the standardized data information based on the real-time verification result and the coarse-granularity verification result.

If the real-time check result indicates that the check fails, or the coarse-granularity check result indicates that the check fails, the computer device may determine that the check result corresponding to the standardized data information is an illegal check result, and at this time, the computer device may delete the illegal check result and the standardized data information corresponding to the service data to be tested together. Optionally, if the real-time verification result indicates that the verification is successful and the coarse-granularity verification result indicates that the verification is successful, the computer device may determine that the verification result corresponding to the standardized data information is a legal verification result, and may further use the standardized data information as standard service data corresponding to the service data to be tested. At the same time, the computer device may also store the legal verification result and the standard service data in the offline database shown in fig. 6.

In step S205, the computer device transmits the standard service data to the management terminal.

The computer equipment and the management terminal in the embodiment of the application have a network connection relationship. The network connection may be a direct or indirect connection via wired communication, may be a direct or indirect connection via wireless identity, or may be another manner, which is not limited herein. Specifically, the computer device may transmit the standard service data to the management terminal based on the network connection relationship. Here, the management terminal may refer to a user terminal corresponding to a management object for performing service detection (e.g., manual detection) on standard service data.

And S206, the management terminal analyzes the standard service data according to the standardized protocol to obtain key fields for displaying on a terminal interface of the management terminal.

Specifically, when the management terminal receives standard service data, the standard service data can be analyzed according to a standardized protocol associated with a standard data format, so that original service data corresponding to the standard service data can be obtained, and further, the computer equipment can acquire key fields in the original service data, so that the key fields in the original service data can be displayed on a terminal interface of the management terminal.

It should be understood that the management object (i.e. the management user) corresponding to the management terminal can perform service detection on the key field displayed on the terminal interface, which means that the service data reported to the data acquisition platform can be preliminarily verified through a legal list and a standardized protocol in the data test process, and can be further verified through manual detection after the preliminary verification is successful, so that the accuracy of the standard service data can be effectively ensured. In addition, when the standard service data corresponding to the service data to be tested is manually detected, the data format of the standard service data is the standard data format, so that the complexity of data processing is reduced, the unification of the follow-up detection mode is realized, and the difficulty of data testing is also reduced.

It should be appreciated that the management object may verify the key value corresponding to the key field on the terminal interface based on the detection rule defined by the management object, so as to determine the service detection result (e.g., the manual detection result) of the standard service data. For example, if the detection rule defined by the management object includes that the value corresponding to the client version information in the standard service data is not null.

Then, when the key value corresponding to the key field (e.g., the client version) on the terminal interface is not null, the management object may perform a triggering operation on the service detection control (e.g., "qualified" control) on the terminal interface, so that the management terminal may respond to the triggering operation to obtain a service detection result (i.e., a detection success result) for indicating that the standard service data is successfully detected. At this time, the management terminal may generate a service notification for returning to the computer device based on the service success result to inform the computer device that the standard service data detection is successful.

Optionally, if the key value corresponding to the key field (e.g., the client version) on the terminal interface is null, the management object may perform a triggering operation on the service detection control (e.g., "failed" control) on the terminal interface, so that the management terminal may respond to the triggering operation to obtain a service detection result (i.e., a detection failure result) for indicating that the standard service data detection fails. At this time, the management terminal may generate a service notification for returning to the computer device based on the service failure result to inform the computer device that the standard service data detection fails.

For ease of understanding, further, please refer to fig. 7, fig. 7 is a frame diagram of test service data provided in an embodiment of the present application. As shown in fig. 7, the frame diagram in the embodiment of the present application may include a data collection exchange platform, an offline database, and a rule base. The computer device according to the embodiment of the present application may be a terminal device with a format standardization function, and the terminal device may be the server 10F shown in fig. 1 and may also be any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100b, which will not be limited herein.

The frame diagram shown in fig. 7 has a plurality of pages as supports for visual display and operation, so that one-stop one-touch data testing service can be realized. For example, the plurality of pages may include a data viewing page (a page for viewing business data in real time), a check result presentation page (a page for presenting a check result of the business data), a legal list management page (a page for managing a white list), an offline analysis page (a page for offline analysis of standard business data stored into an offline database), a data monitoring page (a page for monitoring business data), and a rule management page (a page for configuring check rules for standard business data stored into an offline database), among others.

It should be understood that the computer device may perform data consumption on all the service data in the data collection and exchange platform in sequence, so as to obtain service data to be tested. It will be appreciated that when the computer device obtains service data to be tested, the computer device needs to perform data processing (including parsing processing and normalization processing, for example) on the service data to be tested according to the same data protocol (for example, a standardized protocol associated with a standard data format), so as to obtain standardized data information with the standard data format. Further, the computer device may perform preliminary verification on the standardized data information to obtain a verification result of the standardized data information. For example, the computer device may perform real-time verification on the standardized data information according to the target object identifier of the service data to be tested, so as to obtain a real-time verification result. Further, the computer device may perform coarse-granularity verification on the key field in the standardized data information according to the enumerated value interval corresponding to the key field defined in the standardized protocol, so as to obtain a coarse-granularity verification result. At this time, the computer device may determine a verification result corresponding to the standardized data information based on the real-time verification result and the coarse-granularity verification result. It should be appreciated that the computer device may delete the standardized data information when the verification result indicates that the verification failed. Optionally, when the verification result indicates that the verification is successful, the computer device may use the standardized data information as standard service data corresponding to the service data to be tested, and further may store the standard service data and the verification result together into an offline database shown in fig. 7.

It can be understood that, in order to improve the accuracy of data verification, the computer device may further configure a verification rule for the standard service data stored in the offline database, so that the computer device may automatically perform offline verification on the standard service data stored in the offline database, so as to obtain an offline verification result, so that the possibility of occurrence of an anomaly in the data is reduced.

For example, when storing standard service data in an offline database for offline verification, the computer device may use the standard service data stored in the offline database as service data to be verified. Further, the computer device may configure a verification rule for the key field in the service data to be verified, and store the verification rule to the rule base shown in fig. 7. In this embodiment of the present application, the key field in the service data to be checked may be referred to as a second key field, and the key value corresponding to the second key field may be referred to as a second key value. For example, the rule base may include a check rule 1 (e.g., a check rule configured for a key field such as a reporting timestamp), a check rule 2 (e.g., a check rule configured for a key field such as network information), a check rule 3 (e.g., a check rule configured for a key field such as a client version), a check rule 4 (e.g., a check rule configured for a key field such as a path).

For example, check rule 1 is: "reporting timestamp of traffic data of a certain traffic type is before configuration timestamp (e.g., 12:00 a day)"; the check rule 2 may be: "network information is a wireless communication technology"; the check rule 3 may be: "client version is not empty"; the check rule 4 may be: "path of applet landing page: pages/xxx).

It should be understood that the computer device may automatically obtain certain standard service data from the offline database through the data verification service as service data to be verified, and further may perform offline analysis on the service data to be verified, to determine a second key value corresponding to a second key field in the service data to be verified. The second key field herein may include behavior information, system information, network information, service type, etc. Further, the computer device may obtain the verification rule configured for the second key field from a rule base (e.g., a rule base shown in fig. 7) associated with the service data to be verified; the check rule here includes a configuration value. At this time, the computer device may determine an offline verification result of the service data to be verified based on the second key value and the configuration value. It can be appreciated that the computer device may compare the second key value with the configuration value, and when the second key value matches the configuration value, the computer device may determine that the offline verification result of the service data to be verified indicates that the verification is successful. Optionally, when the second key value does not match the configuration value, the computer device may determine that the offline verification result of the service data to be verified indicates that verification fails, and may further delete the service data to be verified from the offline database.

Further, for ease of understanding, please refer to fig. 8, fig. 8 is a schematic flow chart of a data test according to an embodiment of the present application. As shown in fig. 8, the computer device according to the embodiment of the present application may be a terminal device with a format standardization function, and the terminal device may be the server 10F shown in fig. 1 and may also be any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100b, which will not be limited herein.

It should be understood that, when executing step S81, the data generating terminal may generate service data with various initial data formats, and then may send the service data to the data reporting terminal, so that the data reporting terminal may execute step S82 to report the service data as data to be reported to the data collecting and exchanging platform.

Further, the computer device may execute step S83 to consume data of a certain service data in the data collection and exchange platform, so as to obtain the service data to be tested. At this time, the computer device may perform step S84 to perform data processing on the acquired service data to be tested to obtain standardized data information having a standard data format. The data processing may include identifying a data format, analyzing the service data to be tested, and performing standardization processing on the parsed field to be standardized.

The computer device may perform steps S85 to S86 to perform preliminary verification of the standardized data information, thereby obtaining a verification result of the standardized data information. For example, the computer device may perform real-time verification on the standardized data information based on the target object identifier of the service data to be tested, so as to obtain a real-time verification result. Further, the computer device may perform coarse-granularity verification on the key field in the standardized data information according to the enumerated value interval corresponding to the key field defined in the standardized protocol, so as to obtain a coarse-granularity verification result. At this time, the computer device may determine a verification result corresponding to the standardized data information based on the real-time verification result and the coarse-granularity verification result, and when the verification result indicates that the verification fails, the computer device may delete the standardized data information. Optionally, when the verification result indicates that the verification is successful, the computer device may use the standardized data information as standard service data corresponding to the service data to be tested, and further may store the standard service data and the verification result together into the offline database.

Further, in order to implement automation of data verification and reduce the possibility of abnormality in the data, the computer device may execute step S87 to perform offline verification on the standard service data stored in the offline database. In addition, in order to improve accuracy of data verification, the computer device may further send standard service data to the management terminal, so that the management terminal executes step S88, and performs parsing processing on the standard service data based on a standardized protocol, so as to obtain a key field for displaying on a terminal interface. At this time, the key terminal may perform step S89 to perform service detection (i.e., manual detection) on the key field displayed on the terminal interface. The specific implementation manner of step S81 to step S89 in the present application can refer to the description of step S101 to step S104 in the embodiment corresponding to fig. 3 and the description of step S201 to step S206 in the embodiment corresponding to fig. 5, which will not be repeated here.

In the embodiment of the application, when the computer equipment with the format standardization function acquires the service data to be tested, the initial data format of the service data to be tested can be identified. The data formats of the reported data are different due to different reporting modes of the service data to be tested, which are acquired by the computer equipment. Further, the computer device may obtain a protocol processing class associated with the initial data format based on the mapping relation table associated with the service data to be tested, and may further perform parsing processing on the service data to be tested based on the protocol processing class, so as to obtain a field to be standardized. At this time, the computer device may perform standardization processing on the field to be standardized based on a standardization protocol associated with the standard data format, so as to obtain standardized data information having the standard data format, and further may determine standard service data corresponding to the service data to be tested based on the standardized data information. It can be seen that the computer device can perform unified data processing on the service data to be tested in various different data formats based on the standardized protocol associated with the standard data format, so as to obtain the standard service data. Since the data format of the standard service data is the standard data format, this means that the computer device can achieve unification of the data formats, so that the complexity of data processing is reduced. In addition, in order to improve the accuracy of data verification, the computer equipment can automatically perform offline verification on standard service data stored in an offline database based on the verification rules in the rule base, and a test object does not need to add a dotting log in a code for reporting the service data in advance, so that not only is a mode for automatically verifying the data provided, but also the possibility of data abnormality is reduced. Due to unification of data formats, the computer equipment can verify the service data in all reporting modes, so that the difficulty of manual testing is reduced, and the accuracy of automatic detection is improved.

Further, referring to fig. 9, fig. 9 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 9, the data processing apparatus 1 may be a computer program (including program code) running in a computer device, for example, the data processing apparatus 1 is an application software; the data processing device 1 may be adapted to perform the respective steps of the method provided by the embodiments of the present application. As shown in fig. 9, the data processing apparatus 1 may be operated on a computer device having a format normalization function, and the computer device may be the server 10F shown in fig. 1 or any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100b, which is not limited herein. The data processing apparatus 1 may include: the system comprises a data format identification module 11, an analysis processing module 12, a standardization processing module 13, a standard data determination module 14, a test program starting module 15, a data receiving module 16, a message queue acquisition module 17, a data adding module 18, a data storage module 19, a rule configuration module 20, an offline analysis module 21, a rule acquisition module 22, an offline verification result determination module 23 and a standard data transmission module 24.

The data format identifying module 11 is configured to identify an initial data format of the service data to be tested when the service data to be tested is obtained.

Wherein the data format recognition module 11 includes: a consumption request acquisition unit 111, a data query unit 112, a consumption filtering unit 113, and a format recognition unit 114.

The consumption request acquiring unit 111 is configured to acquire a data consumption request carrying a service identifier when acquiring a consumption right for the data acquisition and exchange platform; the data acquisition and exchange platform comprises N business data; n is a positive integer;

the data query unit 112 is configured to query service data corresponding to a service identifier from N service data;

the consumption filtering unit 113 is configured to, when service data corresponding to a service identifier is queried, perform consumption filtering on the queried service data according to the consumption identifier indicated by the consumption authority, to obtain service data to be tested;

the format identifying unit 114 is configured to identify a data format of the service data to be tested, and take the identified data format as an initial data format.

The specific implementation manner of the consumption request obtaining unit 111, the data query unit 112, the consumption filtering unit 113 and the format identifying unit 114 may be referred to the description of step S101 in the embodiment corresponding to fig. 3, and the detailed description will not be repeated here.

The parsing module 12 is configured to obtain a protocol processing class associated with the initial data format based on a mapping relation table associated with the service data to be tested, and parse the service data to be tested based on the protocol processing class, so as to obtain a field to be standardized.

Wherein, this analysis processing module 12 includes: a storage protocol acquisition unit 121, a mapping table acquisition unit 122, and an analysis processing unit 123.

The storage protocol obtaining unit 121 is configured to obtain a data storage protocol associated with an initial data format, and determine a service theme of service data to be tested and a service interface corresponding to the service data to be tested based on the data storage protocol;

the mapping table obtaining unit 122 is configured to obtain a mapping relation table associated with service data to be tested, and find a protocol processing class having a mapping relation with a service theme and a service interface from the mapping relation table;

the parsing unit 123 is configured to parse the service data to be tested based on the protocol processing class to obtain a field to be standardized.

The specific implementation manner of the storage protocol obtaining unit 121, the mapping table obtaining unit 122, and the parsing unit 123 may refer to the description of step S102 in the embodiment corresponding to fig. 3, and the detailed description will not be repeated here.

The normalization processing module 13 is configured to perform normalization processing on the field to be normalized based on a normalization protocol associated with the standard data format, so as to obtain normalized data information having the standard data format.

The standard data determining module 14 is configured to determine standard service data corresponding to the service data to be tested based on the standardized data information.

The service data to be tested has a target object identifier;

the standard data determination module 14 includes: a real-time check result determination unit 141, a coarse granularity check result determination unit 142, a check result determination unit 143, a standard data determination unit 144, and a data deletion unit 145.

The real-time verification result determining unit 141 is configured to obtain a legal identifier list for storing legal object identifiers, and perform real-time verification on the standardized data information based on the legal identifier list and the target object identifier, so as to obtain a real-time verification result.

Wherein the real-time check result determining unit 141 includes: a legal identification list acquisition subunit 1411, an identification lookup subunit 1412, a first real-time result determination subunit 1413, and a second real-time result determination subunit 1414.

The legal identifier list obtaining subunit 1411 is configured to obtain a legal identifier list for storing M legal object identifiers; m is a positive integer;

The identifier searching subunit 1412 is configured to search for a target object identifier from M legal object identifiers, to obtain a search result;

the first real-time result determining subunit 1413 is configured to obtain a real-time verification result for indicating that the verification is successful if the search result indicates that a legal object identifier matched with the target object identifier exists in the M legal object identifiers.

The second real-time result determining subunit 1414 is configured to obtain a real-time verification result for indicating a verification failure if the search result indicates that a legal object identifier matched with the target object identifier does not exist in the M legal object identifiers.

The specific implementation manner of the legal identifier list obtaining subunit 1411, the identifier searching subunit 1412, the first real-time result determining subunit 1413 and the second real-time result determining subunit 1414 may refer to the description of the real-time verification result in the embodiment corresponding to fig. 3, and will not be further described herein.

The coarse granularity verification result determining unit 142 is configured to perform coarse granularity verification on the first key field in the standardized data information based on the standardized protocol, to obtain a coarse granularity verification result.

Wherein the coarse grain verification result determining unit 142 includes: a key value acquisition subunit 1421, an enumerated value interval acquisition subunit 1422, a comparison unit 1423, and a coarse granularity result determination subunit 1424.

The key value obtaining subunit 1421 is configured to obtain a first key value corresponding to a first key field in the standardized data information;

the enumeration value interval obtaining subunit 1422 is configured to obtain an enumeration value interval corresponding to the first key field from the standardized protocol;

the comparing unit 1423 is configured to compare the first key value with each enumerated value in the enumerated value interval corresponding to the first key field;

the coarse grain result determining subunit 1424 is configured to obtain a coarse grain verification result for indicating that the coarse grain verification is successful if an enumeration value matched with the first key value exists in the enumeration value interval corresponding to the first key field.

The specific implementation manners of the key value obtaining subunit 1421, the enumerated value interval obtaining subunit 1422, the comparing unit 1423 and the coarse granularity result determining subunit 1424 may be referred to the description of the coarse granularity check result in the embodiment corresponding to fig. 3, and will not be further described herein.

The verification result determining unit 143 is configured to determine a verification result corresponding to the standardized data information based on the real-time verification result and the coarse granularity verification result.

The standard data determining unit 144 is configured to, when the verification result indicates that the verification is successful, use the standardized data information as standard service data corresponding to the service data to be tested.

The data deleting unit 145 is configured to delete the standardized data information when the verification result indicates that the verification fails.

The specific implementation manner of the real-time verification result determining unit 141, the coarse granularity verification result determining unit 142, the verification result determining unit 143, the standard data determining unit 144 and the data deleting unit 145 may be referred to the description of step S104 in the embodiment corresponding to fig. 3, and the detailed description will not be repeated here.

The test program starting module 15 is used for starting a test program for testing the service data and starting the service receiver based on the test program;

the data receiving module 16 is configured to receive, through a service receiver, service data for reporting to the data collection and exchange platform, and take the received service data as service data to be reported;

the message queue obtaining module 17 is configured to determine a service topic corresponding to service data to be reported, and obtain a distributed message queue associated with the service topic from the data acquisition and exchange platform;

the data adding module 18 is configured to add service data to be reported to a distributed message queue; the service data to be reported comprises the service data to be tested.

The data storage module 19 is configured to, when storing standard service data in an offline database for offline verification, use the standard service data stored in the offline database as service data to be verified;

the rule configuration module 20 is configured to configure a verification rule for a second key field in the service data to be verified, and store the verification rule in the rule base.

The offline analysis module 21 is configured to obtain service data to be checked from an offline database, perform offline analysis on the service data to be checked, and determine a second key value corresponding to a second key field in the service data to be checked;

the rule obtaining module 22 is configured to obtain, from a rule base associated with service data to be verified, a verification rule configured for the second key field; the check rule includes a configuration value;

the offline verification result determining module 23 is configured to determine an offline verification result of the service data to be verified based on the second key value and the configuration value.

The standard data sending module 24 is configured to send standard service data to the management terminal, so that the management terminal analyzes the standard service data according to a standardized protocol to obtain a key field for displaying on a terminal interface of the management terminal; the key field on the terminal interface is used for indicating the management object corresponding to the management terminal to carry out service detection.

The specific implementation manner of the data format identifying module 11, the parsing module 12, the standardized processing module 13, the standard data determining module 14, the test program starting module 15, the data receiving module 16, the message queue obtaining module 17, the data adding module 18, the data storing module 19, the rule configuring module 20, the offline analyzing module 21, the rule obtaining module 22, the offline checking result determining module 23 and the standard data transmitting module 24 may be referred to the description of the steps S201-S206 in the embodiment corresponding to fig. 5, and the description thereof will not be repeated here.

Further, referring to fig. 10, fig. 10 is a schematic diagram of a computer device according to an embodiment of the present application. As shown in fig. 10, the computer device 1000 may be the server 10F shown in fig. 1 and any one of the user terminals in the user terminal cluster shown in fig. 1, for example, the user terminal 100b, which is not limited herein. The computer device 1000 may include: at least one processor 1001, e.g., a CPU, at least one network interface 1004, a user interface 1003, memory 1005, at least one communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display (Display), a Keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), among others. The memory 1005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1005 may also optionally be at least one storage device located remotely from the aforementioned processor 1001. As shown in fig. 10, the memory 1005, which is one type of computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

In the computer device 1000 shown in fig. 10, the network interface 1004 is mainly used for network communication with a management terminal, an approval terminal, and the like; while user interface 1003 is primarily used as an interface for providing input to a user; and the processor 1001 may be used to invoke a device control application stored in the memory 1005 to implement:

It should be understood that the computer device 1000 described in the embodiment of the present application may perform the description of the data processing method in the embodiment corresponding to fig. 3 and 5, and may also perform the description of the data processing apparatus 1 in the embodiment corresponding to fig. 9, which is not repeated herein. In addition, the description of the beneficial effects of the same method is omitted.

The embodiment of the present application further provides a computer readable storage medium, where a computer program is stored, where the computer program includes program instructions, and when executed by a processor, implement a data processing method provided by each step in fig. 3 and 5, and specifically refer to an implementation manner provided by each step in fig. 3 and 5, which is not described herein again.

The computer readable storage medium may be the data transmission apparatus provided in any of the foregoing embodiments or an internal storage unit of a computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (flash card) or the like, which are provided on the computer device. Further, the computer-readable storage medium may also include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

In one aspect, the present application provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device may execute the description of the data processing method in the embodiment corresponding to fig. 3 or fig. 5, which is not described herein. In addition, the description of the beneficial effects of the same method is omitted.

The terms first, second and the like in the description and in the claims and drawings of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprise," "include," and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, article, or device that comprises a list of steps or elements is not limited to the list of steps or modules but may, in the alternative, include steps or modules not listed or inherent to such process, method, apparatus, article, or device.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The methods and related devices provided in the embodiments of the present application are described with reference to the method flowcharts and/or structure diagrams provided in the embodiments of the present application, and each flowchart and/or block of the method flowcharts and/or structure diagrams may be implemented by computer program instructions, and combinations of flowcharts and/or blocks in the flowchart and/or block diagrams. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart block or blocks of one or more of the flowcharts and/or block diagrams. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart block or blocks of one or more of the flowcharts and/or block diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks and/or block diagram block or blocks.

The foregoing disclosure is only illustrative of the preferred embodiments of the present application and is not intended to limit the scope of the claims herein, as the equivalent of the claims herein shall be construed to fall within the scope of the claims herein.

Claims

1. A method of data processing, comprising:

when obtaining service data to be tested, identifying an initial data format of the service data to be tested;

acquiring a protocol processing class associated with the initial data format based on a mapping relation table associated with the service data to be tested, and analyzing the service data to be tested based on the protocol processing class to obtain a field to be standardized;

based on a standardized protocol associated with a standard data format, carrying out standardized processing on the field to be standardized to obtain standardized data information with the standard data format;

2. The method according to claim 1, wherein the method further comprises:

starting a test program for testing service data, and starting a service receiver based on the test program;

Receiving service data for reporting to a data acquisition and exchange platform through the service receiver, and taking the received service data as service data to be reported;

determining a service theme corresponding to the service data to be reported, and acquiring a distributed message queue associated with the service theme from the data acquisition and exchange platform;

adding the service data to be reported to the distributed message queue; the service data to be reported comprises the service data to be tested.

3. The method of claim 1, wherein the identifying the initial data format of the traffic data to be tested when the traffic data to be tested is acquired comprises:

when the consumption right aiming at the data acquisition and exchange platform is obtained, a data consumption request carrying a service identifier is obtained; the data acquisition and exchange platform comprises N business data; the N is a positive integer;

inquiring service data corresponding to the service identifier from the N service data;

when the service data corresponding to the service identification is inquired, carrying out consumption filtering on the inquired service data according to the consumption identification indicated by the consumption authority to obtain service data to be tested;

And identifying the data format of the service data to be tested, and taking the identified data format as an initial data format.

4. The method according to claim 1, wherein the obtaining a protocol processing class associated with the initial data format based on the mapping relation table associated with the service data to be tested, and the parsing the service data to be tested based on the protocol processing class, to obtain the field to be standardized, includes:

acquiring a data storage protocol associated with the initial data format, and determining a service theme of the service data to be tested and a service interface corresponding to the service data to be tested based on the data storage protocol;

acquiring a mapping relation table associated with the service data to be tested, and searching a protocol processing class with a mapping relation between the service theme and the service interface from the mapping relation table;

and analyzing the service data to be tested based on the protocol processing class to obtain a field to be standardized.

5. The method according to claim 1, wherein the business data to be tested has a target object identification;

The determining, based on the standardized data information, standard service data corresponding to the service data to be tested includes:

acquiring a legal identification list for storing legal object identifications, and carrying out real-time verification on the standardized data information based on the legal identification list and the target object identifications to obtain a real-time verification result;

performing coarse-granularity verification on a first key field in the standardized data information based on the standardized protocol to obtain a coarse-granularity verification result;

determining a verification result corresponding to the standardized data information based on the real-time verification result and the coarse granularity verification result;

when the verification result indicates that verification is successful, the standardized data information is used as standard service data corresponding to the service data to be tested;

and deleting the standardized data information when the verification result indicates that verification fails.

6. The method of claim 5, wherein the obtaining a legal identifier list for storing legal object identifiers, and performing real-time verification on the standardized data information based on the legal identifier list and the target object identifier to obtain a real-time verification result, includes:

Acquiring a legal identification list for storing M legal object identifications; m is a positive integer;

searching the target object identifiers from the M legal object identifiers to obtain a searching result;

if the searching result indicates that legal object identifiers matched with the target object identifier exist in the M legal object identifiers, a real-time verification result for indicating that verification is successful is obtained;

and if the searching result indicates that the legal object identifiers matched with the target object identifier do not exist in the M legal object identifiers, obtaining a real-time verification result for indicating verification failure.

7. The method according to claim 5, wherein performing coarse-grain verification on the first key field in the standardized data information based on the standardized protocol to obtain a coarse-grain verification result includes:

acquiring a first key value corresponding to a first key field in the standardized data information;

acquiring an enumeration value interval corresponding to the first key field from the standardized protocol;

comparing the first key value with each enumeration value in an enumeration value interval corresponding to the first key field;

And if the enumeration value interval corresponding to the first key field has the enumeration value matched with the first key value, obtaining a coarse granularity verification result for indicating that the coarse granularity verification is successful.

8. The method according to claim 1, wherein the method further comprises:

when the standard service data are stored in an offline database for offline verification, the standard service data stored in the offline database are used as service data to be verified;

and configuring a check rule for a second key field in the service data to be checked, and storing the check rule into a rule base.

9. The method of claim 8, wherein the method further comprises:

acquiring the service data to be checked from the offline database, performing offline analysis on the service data to be checked, and determining a second key value corresponding to a second key field in the service data to be checked;

acquiring the verification rule configured for the second key field from a rule base associated with the service data to be verified; the check rule comprises a configuration value;

and determining an offline verification result of the service data to be verified based on the second key value and the configuration value.

10. The method according to claim 1, wherein the method further comprises:

the standard service data is sent to a management terminal, so that the management terminal analyzes the standard service data according to the standardized protocol to obtain key fields for displaying on a terminal interface of the management terminal; and the key field on the terminal interface is used for indicating the management object corresponding to the management terminal to carry out service detection.

11. A data processing apparatus, comprising:

the standardized processing module is used for carrying out standardized processing on the field to be standardized based on a standardized protocol associated with a standard data format to obtain standardized data information with the standard data format;

12. A computer device, comprising: a processor and a memory;

the processor is connected to a memory, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program to cause the computer device to perform the method of any of claims 1-10.

13. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any of claims 1-10.

14. A computer program product or computer program, characterized in that it comprises computer instructions stored in a computer-readable storage medium, which are adapted to be read and executed by a processor to cause a computer device with the processor to perform the method of any of claims 1-10.