CN114780434A - Data processing method and device, electronic equipment and computer readable storage medium - Google Patents
Data processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN114780434A CN114780434A CN202210522944.XA CN202210522944A CN114780434A CN 114780434 A CN114780434 A CN 114780434A CN 202210522944 A CN202210522944 A CN 202210522944A CN 114780434 A CN114780434 A CN 114780434A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- information
- task
- processing method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
Abstract
The present disclosure relates to a data processing method, an apparatus, an electronic device, and a computer-readable storage medium, the data processing method comprising: acquiring online motion flow data; determining object screening information and a target field of a target acquisition task; screening target data from the online action flow data according to the object screening information and the target field, wherein the object screening information is used for screening data of corresponding objects from the online action flow data; the target data is stored. The client normally reports the buried point data and forms on-line action flow data, and then the target data required by the test is obtained from the on-line action flow data, so that the client does not need to be additionally developed, and the development cost can be effectively reduced.
Description
Technical Field
The present disclosure relates to the field of computer application technologies, and in particular, to a data processing method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the development of mobile internet, functions which can be realized by mobile end apps are more and more diverse, and after a certain function is on line, a product manager usually needs to check the influence of the function on the use condition of the apps through buried point data. In the current embedded point test, the embedded point data reported by the client is usually intercepted in an agent mode, and the scheme needs to perform additional operation on the client, so that the client reports the data through the agent, and the development cost of the client is brought.
Disclosure of Invention
The present disclosure provides a data processing method, apparatus, electronic device, and computer-readable storage medium to solve at least the problem of high development cost in the related art, and may not solve any of the above problems.
According to a first aspect of the present disclosure, there is provided a data processing method, including: acquiring online motion flow data; determining object screening information and a target field of a target acquisition task, wherein the object screening information is used for screening data of a corresponding object from the online action flow data; screening target data from the online action flow data according to the object screening information and the target field; and storing the target data.
Optionally, the data processing method further includes: acquiring data verification information, wherein the data verification information is related to the target acquisition task; and verifying the target data according to the data verification information.
Optionally, the verifying the target data according to the data verification information includes: acquiring client configuration information corresponding to the target data; selecting configuration verification parameters from the client configuration information according to the data verification information; and checking the target data by contrasting the configured checking parameters.
Optionally, the obtaining client configuration information corresponding to the target data includes: acquiring client configuration information corresponding to the target data from a service agent; the client configuration information is obtained through the following steps: and the service agent responds to the identification of a configuration request, and acquires client configuration information fed back by a server aiming at the configuration request, wherein the configuration request is sent by the equipment generating the target data.
Optionally, the data processing method further includes: acquiring equipment information of equipment generating the target data; wherein, the acquiring data verification information includes: requesting a callback interface before the target acquisition task is finished to acquire task parameters of the target acquisition task, wherein the task parameters comprise task identification and equipment information; comparing the device information corresponding to the target data with the device information in the task parameters, determining a task identifier corresponding to the same device information as the target data, and recording the task identifier as a target task identifier; and acquiring the data verification information according to the target task identifier.
Optionally, the device information is a device serial number, and the target data includes a device identifier, where the obtaining the device information of the device that generates the target data includes: inquiring mapping information, wherein the mapping information records the mapping relation between the equipment serial number and the equipment identification; and determining the equipment serial number of the equipment generating the target data according to the mapping information and the equipment identification in the target data.
Optionally, the target collection task is configured with at least one collection use case, the task parameters further include a use case identifier of each collection use case and an occurrence period of each collection use case, and the target data further includes time information, where acquiring the data verification information according to the target task identifier includes: comparing the time information of the target data with the occurrence time period of each acquisition case corresponding to the target task identifier, determining the case identifier of the acquisition case corresponding to the target data in time, and recording the case identifier as the target case identifier; and acquiring the data verification information according to the target task identifier and the target case identifier.
Optionally, the data processing method further includes: and dynamically displaying the trend curve of the target data based on the stored target data.
Optionally, the data processing method further includes: screening data of corresponding objects from the online action flow data according to the object screening information, and marking the data as primary screening data; storing the prescreening data separately from the target data.
According to a second aspect of the present disclosure, there is provided a data processing apparatus comprising: an acquisition unit configured to: acquiring online action flow data; a determination unit configured to: determining object screening information and a target field of a target acquisition task, wherein the object screening information is used for screening data of a corresponding object from the online action flow data; a screening unit configured to: screening target data from the online action flow data according to the object screening information and the target field; a storage unit configured to: and storing the target data.
Optionally, the obtaining unit is further configured to: acquiring data verification information, wherein the data verification information is related to the target acquisition task; the data processing apparatus further comprises a verification unit configured to: and verifying the target data according to the data verification information.
Optionally, the verification unit is further configured to: acquiring client configuration information corresponding to the target data; selecting configuration verification parameters from the client configuration information according to the data verification information; and checking the target data by contrasting the configured checking parameters.
Optionally, the verification unit is further configured to: acquiring client configuration information corresponding to the target data from a service agent; wherein the client configuration information is obtained by the following steps: and the service agent responds to the identification of a configuration request, and acquires client configuration information fed back by a server aiming at the configuration request, wherein the configuration request is sent by the equipment generating the target data.
Optionally, the obtaining unit is further configured to: acquiring equipment information of equipment generating the target data; requesting a callback interface before the target acquisition task finishes running so as to acquire task parameters of the target acquisition task, wherein the task parameters comprise task identifiers and equipment information; comparing the equipment information corresponding to the target data with the equipment information in the task parameters, determining a task identifier corresponding to the same equipment information with the target data, and recording the task identifier as a target task identifier; and acquiring the data verification information according to the target task identifier.
Optionally, the device information is a device serial number, the target data includes a device identifier, and the obtaining unit is further configured to: inquiring mapping information, wherein the mapping information records the mapping relation between the equipment serial number and the equipment identification; and determining the equipment serial number of the equipment generating the target data according to the mapping information and the equipment identification in the target data.
Optionally, the target collection task is configured with at least one collection use case, the task parameters further include a use case identifier of each collection use case and an occurrence time period of each collection use case, the target data further includes time information, and the obtaining unit is further configured to: comparing the time information of the target data with the occurrence time period of each acquisition case corresponding to the target task identifier, determining the case identifier of the acquisition case corresponding to the target data in time, and recording the case identifier as the target case identifier; and acquiring the data verification information according to the target task identifier and the target case identifier.
Optionally, the data processing apparatus further comprises a presentation unit configured to: and dynamically displaying the trend curve of the target data based on the stored target data.
Optionally, the screening unit is further configured to: screening data of corresponding objects from the online action flow data according to the object screening information, and marking the data as primary screening data; the storage unit is further configured to: storing the prescreening data separately from the target data.
According to a third aspect of the present disclosure, there is provided an electronic apparatus comprising: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform a data processing method according to the present disclosure.
According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium in which instructions, when executed by at least one processor, cause the at least one processor to perform a data processing method according to the present disclosure.
According to a fifth aspect of the present disclosure, there is provided a computer program product comprising computer instructions which, when executed by at least one processor, implement a data processing method according to the present disclosure.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
according to the data processing method and the data processing device, the client normally reports the buried point data and forms the on-line action flow data, and then the target data required by the test is obtained from the on-line action flow data, so that the client does not need to be additionally developed, and the development cost can be effectively reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
Fig. 1 is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure.
Fig. 2 is an overall architecture diagram illustrating a data processing method according to an exemplary embodiment of the present disclosure.
Fig. 3 is a schematic diagram illustrating a visualization presentation in accordance with an exemplary embodiment of the present disclosure.
Fig. 4 is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present disclosure.
Fig. 5 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.
Detailed Description
In order to make the technical solutions of the present disclosure better understood, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the disclosure, as detailed in the appended claims.
In this case, the expression "at least one of the items" in the present disclosure means a case where three types of parallel expressions "any one of the items", "a combination of any plural ones of the items", and "the entirety of the items" are included. For example, "include at least one of a and B" includes the following three cases in parallel: (1) comprises A; (2) comprises B; (3) including a and B. For another example, "at least one of the first step and the second step is performed", which means that the following three cases are juxtaposed: (1) executing the step one; (2) executing the step two; (3) and executing the step one and the step two.
With the development of mobile internet, functions which can be realized by mobile end apps are more and more diverse, and after a certain function is on line, a product manager usually needs to check the influence of the function on the use condition of the apps through buried point data. In short, a large number of operation nodes exist on the app, one or some of the operation nodes have strong association with the functions to be tested, and the buried data is operation data of the user on the strongly associated operation nodes. The method comprises the steps of data reporting, data analysis, data storage, data verification and data statistics, wherein the data of the embedded point is produced and reported by a client, and the reported data is encrypted in the process. A common embedded point reporting test mode is that an enterprise intercepts reported data, decrypts the intercepted data, and displays a data plaintext on a web page, and a QA (Quality assurance) verifies the data on the web page during embedded point testing to ensure the correctness and integrity of the embedded data.
Online apps often have a large user amount and many operation nodes, and it is difficult to ensure that all acquired data can be suitable for testing of functions to be tested, so in order to improve data acquisition efficiency, enterprises often use devices to register accounts by themselves, and use the devices to operate the functions to be tested, so as to generate buried point data. In the current embedded point test, a general solution is that in the reporting process, embedded point data is intercepted and decrypted in an agent mode, the intercepted and decrypted data is visually displayed in a plaintext, and then the QA goes to a visual platform to manually check the data.
This solution has the following drawbacks: 1. to implement data interception, a report button needs to be configured on a client, and the report button is triggered by operation, so that the client reports data via an agent, which brings development cost of the client. 2. The decryption mode of proxy configuration and timely updating of buried data is required, which brings the maintenance cost of proxy. 3. The operation of intercepting data by the agent interferes with the original reporting process of the data, so that the intercepted data cannot realize original data storage, if the problem of data burying exists in verification, the intercepted data can only be inquired in the storage of the agent, the inquiry of an online data analysis platform cannot be realized, the data storage realized by the agent occupies additional resources, and the storage efficiency and the use efficiency of the data are reduced.
According to the data processing scheme of the exemplary embodiment of the disclosure, the client normally reports the buried point data and forms the online action stream data, and then the target data required by the test is acquired from the online action stream data, so that the client does not need to be additionally developed, the development cost can be effectively reduced, the conventional decryption process of the online action stream data can be executed, the additional agent maintenance is not needed, and the cost control is facilitated. In addition, all the data of the embedded points generated by the client can normally enter the online process, the plaintext data of the embedded points can be obtained under the condition that the production and consumption processes of the data of the embedded points on the line are not interfered, specifically, the execution of the process of the scheme is parallel to the online process, no overlapping part exists, and no mutual influence exists.
Hereinafter, a data processing method and a data processing apparatus according to an exemplary embodiment of the present disclosure will be described in detail with reference to fig. 1 to 5.
Fig. 1 is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure. Fig. 2 is an overall architecture diagram illustrating a data processing method according to an exemplary embodiment of the present disclosure.
Referring to fig. 1, in step 101, on-line action stream data is acquired. Referring to fig. 2, the online action stream data is, for example, online kafka queue data (kafka is a high throughput distributed publish-subscribe messaging system, which can process all action stream data of a consumer in a website), a client (corresponding to the data collection layer shown in fig. 2) can issue a buried point report through an http request, the reported buried point data are collected together, and the online action stream data can be formed and delivered to a service layer to perform online consumption services. Specifically, the online action stream data may be cleaned in real time, and data decryption and data format conversion (for example, converting pb (protocol buffer) data into json (JS Object Notation) data) may be completed for subsequent use. For some scenes with large data volume, random sampling or white list sampling can be performed to reduce the data volume. It should be understood that multiple parallel consuming services may be executed simultaneously, without interaction. As an example, consuming services may include recommending content for a user that may be of interest based on a user's browsing history.
Referring back to fig. 1, in step 102, object filtering information and object fields of the target collection task are determined, wherein the object filtering information is used for filtering data of corresponding objects from the online motion flow data. The collection task refers to a task of collecting buried point data in order to realize a specific functional test, and the target collection task is a collection task concerned currently. As mentioned above, in order to improve data acquisition efficiency, enterprises often use devices to register accounts and use the devices to operate functions to be tested to generate buried point data, and the object screening information is, for example, a data table or an array recording the devices or accounts used by the enterprises for testing. As an example, referring to fig. 2, a configuration management platform inside the app may maintain a did (device identity document, for identifying one physical device) white list and a uid (user identity document) white list, that is, object screening information. It should be understood that the device/user whitelist is different from the whitelist used for the whitelist sampling in step 101.
In step 103, target data is screened from the online action stream data according to the target screening information and the target field. Each buried point data in the online action flow data generally comprises an equipment identifier and a user identifier, and the data hitting the object screening information in the online action flow data can be screened out by combining the object screening information, so that the data acquisition efficiency is ensured. Meanwhile, in order to meet the data requirements of different acquisition tasks, the buried data usually includes a large number of fields, such as user data of user gender, user age group, and user city, such as device data of device operating system and client version, and content data of video resolution, code rate, and frame rate. By combining the target fields, the fields required to be used can be screened out, the data volume of target data is reduced, the data pertinence is improved, and the data acquisition efficiency is ensured. It should be appreciated that the device identification and user identification will generally continue to be retained in the target data for ongoing data statistics under test.
At step 104, the target data is stored. By storing the target data acquired in step 103, QA can perform data verification, and then perform data statistics, thereby implementing functional testing. As an example, referring to fig. 2, target data, such as a QOS (Quality of Service) indicator, may be stored in a mysql database.
Optionally, after step 104, the data processing method according to the exemplary embodiment of the present disclosure may further include a step of automatically verifying the target data, so as to replace or assist manual verification, thereby improving data processing efficiency. The step of automatically verifying the target data can be further refined into an information acquisition step and a verification processing step. The information acquisition step is to acquire data verification information, and the data verification information is related to the target acquisition task. And the checking processing step is to check the target data according to the data checking information. By acquiring the data verification information related to the target acquisition tasks, the data of a plurality of different target acquisition tasks can be acquired simultaneously, and automatic verification processing is performed according to the respective data verification information, so that the data processing efficiency is improved, the data of only one target acquisition task does not need to be acquired at each time, and automatic verification is performed according to single data verification information.
Next, the information acquisition step will be described.
Optionally, the data processing method further includes: acquiring device information of a device generating target data; the information acquiring step may include: requesting a callback interface before the target acquisition task finishes running so as to acquire task parameters of the target acquisition task, wherein the task parameters comprise task identifiers (task ids) of corresponding acquisition tasks and equipment information of equipment for acquiring data when the corresponding acquisition tasks are executed; comparing the equipment information corresponding to the target data with the equipment information in the task parameters, determining a task identifier corresponding to the same equipment information with the target data, and recording the task identifier as a target task identifier; and acquiring data verification information according to the target task identifier. The target data is acquired when the target acquisition task is executed, but the information of the target acquisition task is not added into the target data in the acquisition process, and in order to realize the automatic verification of the target data, the target acquisition task needs to be determined first, so that the used data verification information is acquired. By requesting a callback interface before the target acquisition task finishes data processing but finishes a task, task parameters including equipment information and a task identifier can be acquired, the equipment information can be used as an intermediate bridge and is compared with the equipment information corresponding to the target data, so that the task identifier corresponding to the target data, namely the task identifier of the target acquisition task, can be found, data verification information can be acquired according to the task identifier, and smooth acquisition of the data verification information is guaranteed. Specifically, the target task identification may be stored in a mysql database and associated with corresponding target data.
Alternatively, the device information may be used to identify a device at the system level, the app platform often manages the device through the device information, the device information is, for example, a serial number of the device, and the target data often includes the device identification at the hardware level and does not directly include the device information. The step of acquiring device information of a device that generates the target data may include: inquiring mapping information, and recording the mapping relation between the equipment serial number and the equipment identification by the mapping information; and determining the equipment serial number of the equipment generating the target data according to the mapping information and the equipment identification in the target data. By combining the mapping information, the device information corresponding to the target data can be successfully determined, and the reliable acquisition of the target task identifier and the data verification information is ensured. Referring to fig. 2, a did map, that is, mapping information, may also be maintained in the configuration management platform inside the app, and the device serial number may be determined by comparing the device identifier in the target data with the mapping information. Specifically, the device serial number may be stored in a mysql database and associated with corresponding target data.
Optionally, the target collection task is configured with at least one collection use case, for example, the collection use case may be an operating system of the device or a client version, and different collection use cases of the same target collection task may have different configurations and thus have different data verification information. The task parameters further comprise use case identification of each acquisition use case and occurrence time period of each acquisition use case, and the target data further comprise time information. Correspondingly, the step of obtaining the data verification information according to the target task identifier may include: comparing the time information of the target data with the occurrence time period of each acquisition case corresponding to the target task identifier, determining the case identifier of the acquisition case corresponding to the target data in time, and recording the case identifier as the target case identifier; and acquiring data verification information according to the target task identifier and the target case identifier. By further comparing the generation time of the target data with the generation time period of each acquisition case when the target acquisition task is executed, which acquisition case corresponds to the target data can be determined, so that the corresponding target case identification is determined, the corresponding data verification information is acquired, and smooth and reliable acquisition of the information is guaranteed.
The verification processing steps will be described next.
Referring to fig. 2, in a data processing method according to an exemplary embodiment of the present disclosure, the verification rule may include at least one of a timing verification, an integrity verification, a reference verification, and a configuration read verification.
One complete operation flow has one session id, different embedded points of the same flow have the same session id, and different flows have different session ids. The time sequence check is to check whether the occurrence time and the front-back relation of each embedded point data on one session id link are normal, and the data check information can record the normal occurrence time and the front-back relation of each embedded point data. The integrity check is to check whether the data of the embedded points associated on one session id link is missing, if the embedded points are imported, exported and uploaded, and check whether the fields of the data of the embedded points are complete, the data check information can record the embedded points of the complete link and the complete fields of the data of the embedded points.
For reference verification, the data verification information may record a plurality of reference fields in the embedded data, and if the embedded data simultaneously has the reference fields, the reference verification is passed.
For configuration reading verification, a scheme exists in the related art for making a json verification rule in advance and performing verification judgment on the json verification rule and the obtained target data, for example, when resolution buried point data is judged, the json verification rule can be configured: the video height is "720" & & source video width is "1080", i.e. the video height is 720 and the width is 1080. By parsing the json check rule and then taking the target field in the target data, it can be checked whether the target data meets expectations. However, if the resolution is actually upgraded due to the change of the requirement, and the new resolution is 1080 × 1920, the original check rule is not applicable, and a new check rule needs to be written, which results in maintenance cost.
Optionally, the verification processing step according to an exemplary embodiment of the present disclosure may include: acquiring client configuration information corresponding to target data; selecting configuration verification parameters from the client configuration information according to the data verification information; and checking the target data by contrasting the configured checking parameters. By actively acquiring the client configuration information corresponding to the target data, selecting the parameters which are consistent with the data verification information as the configuration verification parameters, verification processing can be performed according to the latest configuration verification parameters, automatic updating of the configuration verification parameters is realized, the verification process is more flexible, and the maintenance cost is fully reduced. As an example, for the case that the function to be tested is a video uploading function, the data verification information may include imported/exported transcoding soft and hard codes, a code rate, and a resolution gear, and accordingly, the configuration verification parameter is a configuration value of the imported/exported transcoding soft and hard codes, the code rate, and the resolution gear, which constitutes a specific transcoding strategy for importing/exporting a video during the video uploading, and the verification processing is to compare a value of a corresponding parameter in the target data with the configuration verification parameter, and verify whether the value of the corresponding parameter in the target data conforms to the configuration.
Optionally, referring to fig. 2, the step of obtaining the client configuration information corresponding to the target data includes: acquiring client configuration information corresponding to target data from a service agent; the client configuration information is obtained through the following steps: and the service agent responds to the identification of the configuration request, and acquires the client configuration information fed back by the server for the configuration request, wherein the configuration request is sent by the equipment generating the target data. Because the client configuration information is usually notified to the client by the server through configuration issue, a specific configuration policy can be obtained by obtaining configuration issue content. Specifically, the client is connected to a service agent, such as a submirroxy (man-in-the-middle agent tool), and the service agent filters the http request, so that the configuration request, that is, the request for configuration delivery, is identified, and the client configuration information fed back by the server, that is, the configuration delivery, can be acquired.
Optionally, the data processing method according to the exemplary embodiment of the present disclosure further includes: and dynamically displaying the trend curve of the target data based on the stored target data. The stored target data are displayed by utilizing the trend curve, so that the change trend of the related target data can be visually displayed, and workers can know the change of the target data. It should be understood that the dynamic display means that when the target data does not pass the verification and is correspondingly corrected, the trend curve can be correspondingly changed, the trend curve is dynamically updated, and the timeliness of information transmission is guaranteed. As an example, data can be read from a database storing target data and a data set can be created, after the data set is created, part of dirty data is filtered out through an original value filter, and a trend curve and detailed data are created according to the display requirement.
Fig. 3 is a schematic diagram illustrating a visualization presentation in accordance with an exemplary embodiment of the present disclosure.
Referring to fig. 3, the abscissa of the trend curve represents different client versions, the ordinate represents a certain parameter, and the two curves represent two devices. Most of the data in the graph is historical data, since the same device uses the same client version for a short time. When one curve is in the second client version, the parameters suddenly drop and gradually rise later, and the subsequent client versions can be considered to have an optimization effect on the problem; but the other curve has fluctuations indicating that the optimization effect varies from device to device.
Optionally, regarding data storage, the data processing method according to an exemplary embodiment of the present disclosure further includes: screening data of corresponding objects from online action stream data according to the object screening information, and recording the data as primary screening data; storing the prescreening data separately from the target data. As can be seen from the foregoing description, the target data is the data corresponding to the target field in the prescreening data. Through storing complete prescreening data alone, can enough provide the reference for data correction after the data check, guarantee the reliable going on of data correction, can guarantee to go on high-efficiently again to the processing of target data. Referring to fig. 2, the prescreened data (i.e., the complete buried point data corresponding to the object screening information) may be stored in an ES (electronic search, a distributed full-text search engine with multi-user capability) database, so as to implement convenient and fast search of data.
By way of example, and referring generally to FIG. 2, a data processing method in accordance with an exemplary embodiment of the present disclosure involves a data collection layer, a service layer, a storage layer, and an interaction layer.
The data collection layer is composed of a plurality of client devices, and can report buried point data and other data generated in the running process of the app together to form online kafka queue data. The service layer represents a service end, for online kafka queue data, the service layer executes consumption service in S11, acquires online kafka queue data, then screens out embedded data (namely primary screened data) from the acquired online kafka queue data in combination with a did white list or a uid white list in a configuration management platform in the app in S12, stores the complete embedded data in an ES database of the storage layer in S13, screens out QOS indexes (corresponding to target data) from the embedded data in combination with target fields in S14, and stores the screened QOS indexes in a mysql database of the storage layer to realize acquisition, screening and storage of the embedded data.
The interaction layer comprises user equipment and a visual presentation platform, and can be used for inputting and outputting information. The user equipment can query the corresponding information from the ES database according to the query input request query interface of the user. And the visual display platform requests a callback interface at S21 before the target acquisition task finishes data processing and ends the task, acquires the task parameters of the target acquisition task, and determines the data verification information of the target acquisition task by contrasting the task parameters and the QOS index in the mysql database at S22, so that S23 can complete the automatic verification of the QOS index. In addition, at S30, the visualization presentation platform may obtain QOS metrics in the mysql database and perform a visualization presentation on the platform such as that shown in fig. 3.
Fig. 4 is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present disclosure.
Referring to fig. 4, data processing apparatus 400 includes acquisition unit 401, determination unit 402, filtering unit 403, and storage unit 404.
The acquisition unit 401 may acquire online motion stream data.
The determining unit 402 may determine object filtering information and a target field of the target collection task, wherein the object filtering information is used for filtering data of a corresponding object from the online motion flow data.
The filtering unit 403 may filter out target data from the online motion stream data according to the object filtering information and the target field.
The storage unit 404 may store target data.
Optionally, the obtaining unit 401 may further obtain data verification information, where the data verification information is related to the target collection task; the data processing apparatus 400 further includes a verifying unit, which can perform a verifying process on the target data according to the data verifying information.
Optionally, the verification unit may further obtain client configuration information corresponding to the target data; selecting configuration verification parameters from the client configuration information according to the data verification information; and checking the target data by contrasting the configured checking parameters.
Optionally, the verification unit is further configured to: acquiring client configuration information corresponding to target data from a service agent; the client configuration information is obtained through the following steps: and the service agent responds to the identification of the configuration request, and acquires the client configuration information fed back by the server for the configuration request, wherein the configuration request is sent by the equipment for generating the target data.
Alternatively, the acquisition unit 401 may also acquire device information of a device that generates the target data; requesting a callback interface before the target acquisition task is finished to acquire task parameters of the target acquisition task, wherein the task parameters comprise task identification and equipment information; comparing the equipment information corresponding to the target data with the equipment information in the task parameters, determining a task identifier corresponding to the same equipment information with the target data, and recording the task identifier as a target task identifier; and acquiring data verification information according to the target task identifier.
Optionally, the device information is a device serial number, the target data includes a device identifier, the obtaining unit 401 may further query mapping information, and the mapping information records a mapping relationship between the device serial number and the device identifier; and determining the equipment serial number of the equipment generating the target data according to the mapping information and the equipment identification in the target data.
Optionally, the target acquisition task is configured with at least one acquisition use case, the task parameters further include a use case identifier of each acquisition use case and an occurrence time period of each acquisition use case, the target data further includes time information, and the acquisition unit 401 may further compare the time information of the target data with the occurrence time period of each acquisition use case corresponding to the target task identifier, determine a use case identifier of an acquisition use case corresponding to the target data in time, and record the use case identifier as the target use case identifier; and acquiring data verification information according to the target task identifier and the target case identifier.
Optionally, the data processing apparatus further includes a presentation unit, and the presentation unit may dynamically present the trend curve of the target data based on the stored target data.
Optionally, the screening unit 403 may also screen data of a corresponding object from the online motion stream data according to the object screening information, and record the data as primary screening data; the storage unit 404 may also store the prescreening data separately from the target data.
The operations of the respective units in the data processing apparatus 400 can be understood in conjunction with the data processing methods described with reference to fig. 1 to 3.
Fig. 5 is a block diagram of an electronic device according to an example embodiment of the present disclosure.
Referring to fig. 5, the electronic device 500 comprises at least one memory 501 and at least one processor 502, the at least one memory 501 having stored therein a set of computer-executable instructions that, when executed by the at least one processor 502, perform a data processing method according to an exemplary embodiment of the present disclosure.
By way of example, the electronic device 500 may be a PC computer, tablet device, personal digital assistant, smartphone, or other device capable of executing the set of instructions described above. Here, the electronic device 500 need not be a single electronic device, but can be any collection of devices or circuits that can execute the above instructions (or sets of instructions), either individually or in combination. The electronic device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote (e.g., via wireless transmission).
In the electronic device 500, the processor 502 may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a programmable logic device, a special-purpose processor system, a microcontroller, or a microprocessor. By way of example, and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.
The processor 502 may execute instructions or code stored in the memory 501, wherein the memory 501 may also store data. The instructions and data may also be transmitted or received over a network via the network interface device, which may employ any known transmission protocol.
The memory 501 may be integrated with the processor 502, for example, by having RAM or flash memory disposed within an integrated circuit microprocessor or the like. Further, memory 501 may comprise a stand-alone device, such as an external disk drive, storage array, or any other storage device usable by a database system. The memory 501 and the processor 502 may be operatively coupled or may communicate with each other, such as through I/O ports, network connections, etc., so that the processor 502 can read files stored in the memory.
In addition, the electronic device 500 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 500 may be connected to each other via a bus and/or a network.
According to an exemplary embodiment of the present disclosure, there may also be provided a computer-readable storage medium, in which instructions, when executed by at least one processor, cause the at least one processor to perform a data processing method according to an exemplary embodiment of the present disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD + R, CD-RW, CD + RW, DVD-ROM, DVD-R, DVD + R, DVD-RW, DVD + RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk memory, Hard Disk Drive (HDD), solid-state disk drive (SSD), card-type memory (such as a multimedia card, a Secure Digital (SD) card or an extreme digital (XD) card), tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a magnetic tape, a magneto-optical data storage device, a hard disk, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic tape, a magnetic data storage device, a magnetic disk, a magnetic data storage device, a magnetic disk, A solid state disk, and any other device configured to store and provide a computer program and any associated data, data files, and data structures to a processor or computer in a non-transitory manner such that the processor or computer can execute the computer program. The computer program in the computer-readable storage medium described above can be run in an environment deployed in a computer apparatus, such as a client, a host, a proxy device, a server, and the like, and further, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by one or more processors or computers.
According to an exemplary embodiment of the present disclosure, there may also be provided a computer program product comprising computer instructions which, when executed by at least one processor, cause the at least one processor to perform a data processing method according to an exemplary embodiment of the present disclosure.
According to the data processing method, the data processing device, the electronic equipment and the computer readable storage medium, the client normally reports the buried point data and forms the online action flow data, and then the target data required by the test is obtained from the online action flow data, so that the client does not need to be additionally developed, the development cost can be effectively reduced, the conventional decryption process of the online action flow data can be executed, the additional proxy maintenance is not needed, and the cost control is facilitated. In addition, all the data of the embedded points generated by the client can normally enter the online process, the plaintext data of the embedded points can be obtained under the condition that the production and consumption processes of the data of the embedded points on the line are not interfered, particularly, the execution of the process of the scheme is parallel to the online process, no overlapping part exists, and no influence exists.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice in the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (10)
1. A method of data processing, comprising:
acquiring online motion flow data;
determining object screening information and a target field of a target acquisition task, wherein the object screening information is used for screening data of a corresponding object from the online action flow data;
screening target data from the online action flow data according to the object screening information and the target field;
and storing the target data.
2. The data processing method of claim 1, wherein the data processing method further comprises:
acquiring data verification information, wherein the data verification information is related to the target acquisition task;
and according to the data verification information, verifying the target data.
3. The data processing method according to claim 2, wherein the verifying the target data according to the data verification information includes:
acquiring client configuration information corresponding to the target data;
selecting configuration verification parameters from the client configuration information according to the data verification information;
and checking the target data by contrasting the configured checking parameters.
4. The data processing method of claim 3, wherein the obtaining client configuration information corresponding to the target data comprises:
acquiring client configuration information corresponding to the target data from a service agent;
wherein the client configuration information is obtained by the following steps:
and the service agent responds to the identification of a configuration request, and acquires client configuration information fed back by a server aiming at the configuration request, wherein the configuration request is sent by the equipment generating the target data.
5. The data processing method of claim 2, wherein the data processing method further comprises: acquiring equipment information of equipment generating the target data;
wherein, the acquiring data verification information includes:
requesting a callback interface before the target acquisition task is finished to acquire task parameters of the target acquisition task, wherein the task parameters comprise task identification and equipment information;
comparing the device information corresponding to the target data with the device information in the task parameters, determining a task identifier corresponding to the same device information as the target data, and recording the task identifier as a target task identifier;
and acquiring the data verification information according to the target task identifier.
6. The data processing method of claim 5, wherein the device information is a device serial number, the target data includes a device identification, and wherein the obtaining the device information of the device that generated the target data comprises:
inquiring mapping information, wherein the mapping information records the mapping relation between the equipment serial number and the equipment identification;
and determining the equipment serial number of the equipment generating the target data according to the mapping information and the equipment identification in the target data.
7. A data processing apparatus, comprising:
an acquisition unit configured to: acquiring online action flow data;
a determination unit configured to: determining object screening information and a target field of a target acquisition task;
a screening unit configured to: screening target data from the online action flow data according to the object screening information and the target field, wherein the object screening information is used for screening data of corresponding objects from the online action flow data;
a storage unit configured to: and storing the target data.
8. An electronic device, comprising:
at least one processor;
at least one memory storing computer-executable instructions,
wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the data processing method of any of claims 1 to 6.
9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by at least one processor, cause the at least one processor to perform the data processing method of any one of claims 1 to 6.
10. A computer program product comprising computer instructions, characterized in that the computer instructions, when executed by at least one processor, implement the data processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210522944.XA CN114780434A (en) | 2022-05-13 | 2022-05-13 | Data processing method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210522944.XA CN114780434A (en) | 2022-05-13 | 2022-05-13 | Data processing method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114780434A true CN114780434A (en) | 2022-07-22 |
Family
ID=82436557
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210522944.XA Pending CN114780434A (en) | 2022-05-13 | 2022-05-13 | Data processing method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114780434A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116502054A (en) * | 2023-05-12 | 2023-07-28 | 上海邮电设计咨询研究院有限公司 | Flow data analysis method, system, medium and electronic equipment |
-
2022
- 2022-05-13 CN CN202210522944.XA patent/CN114780434A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116502054A (en) * | 2023-05-12 | 2023-07-28 | 上海邮电设计咨询研究院有限公司 | Flow data analysis method, system, medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200250024A1 (en) | High-volume distributed script error handling | |
| US10733079B2 (en) | Systems and methods for end-to-end testing of applications using dynamically simulated data | |
| KR101828506B1 (en) | Visualization of jvm and cross-jvm call stacks | |
| US9070086B2 (en) | Data driven component reputation | |
| US8490055B2 (en) | Generating dependency maps from dependency data | |
| US10776244B2 (en) | Consolidation planning services for systems migration | |
| CN110928772B (en) | Test method and device | |
| CA2798759C (en) | Bug clearing house | |
| US8924942B1 (en) | Identifying user interface improvements from observed user behavior | |
| US20160092516A1 (en) | Metric time series correlation by outlier removal based on maximum concentration interval | |
| US10362086B2 (en) | Method and system for automating submission of issue reports | |
| US10229028B2 (en) | Application performance monitoring using evolving functions | |
| DE112016002120T5 (en) | Development and sales platform for software | |
| KR100803889B1 (en) | Method and system for analyzing performance of providing services to client terminal | |
| US9229845B1 (en) | Testing using production data in scalable pre-production environments | |
| US9910726B2 (en) | System dump analysis | |
| CN111539756B (en) | System and method for identifying and targeting users based on search requirements | |
| CN113760641A (en) | Service monitoring method, device, computer system and computer readable storage medium | |
| US20190227824A1 (en) | Systems and methods for capturing and visualizing user interactions across devices | |
| US11843526B2 (en) | Automatic automation recommendation | |
| US9164746B2 (en) | Automatic topology extraction and plotting with correlation to real time analytic data | |
| CN114780434A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US20190266136A1 (en) | Data sampling in a storage system | |
| US11615363B2 (en) | Digital chat conversation and virtual agent analytics | |
| Wang et al. | Analyzing and monitoring kubernetes microservices based on distributed tracing and service mesh |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |