CN114900445A

CN114900445A - Flow recording method and device and electronic equipment

Info

Publication number: CN114900445A
Application number: CN202210462089.8A
Authority: CN
Inventors: 孙宇博; 曲中鑫; 金鑫
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-04-28
Filing date: 2022-04-28
Publication date: 2022-08-12
Anticipated expiration: 2042-04-28
Also published as: CN114900445B

Abstract

The disclosure provides a traffic recording method and device and electronic equipment, relates to the technical field of computers, and particularly relates to the technical field of traffic recording of online services. The specific implementation scheme is as follows: monitoring a target task queue subscribed by M task execution processes aiming at a target service, wherein M is a positive integer; under the condition that the target task queue comprises a target flow recording task issued by a task issuing process, determining a target task execution process from the M task execution processes, wherein the target flow recording task is used for recording the flow generated by the on-line service of the target service; and recording the traffic generated by the online service of the target service through the target task execution process based on the target traffic recording task.

Description

Flow recording method and device and electronic equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a traffic recording method and apparatus for online services, and an electronic device.

Background

In the daily test process, because the on-line real traffic can reflect the real situation of the service after being on-line better than the artificially constructed traffic, the on-line real traffic is often required to be used for testing the performance, the function and the like of the service.

Currently, for obtaining the online real traffic of the service, an online instance log is usually pulled manually to perform conversion, or the online traffic of a single instance is recorded in a middleware manner.

Disclosure of Invention

The disclosure provides a traffic recording method and device and electronic equipment.

According to a first aspect of the present disclosure, there is provided a traffic recording method, including:

monitoring a target task queue subscribed by M task execution processes aiming at a target service, wherein M is a positive integer;

under the condition that the target task queue comprises a target flow recording task issued by a task issuing process, determining a target task execution process from the M task execution processes, wherein the target flow recording task is used for recording the flow generated by the on-line service of the target service;

and recording the traffic generated by the online service of the target service through the target task execution process based on the target traffic recording task.

According to a second aspect of the present disclosure, there is provided a traffic recording apparatus, comprising:

the monitoring module is used for monitoring a target task queue subscribed by M task execution processes aiming at a target service, wherein M is a positive integer;

the determining module is used for determining target task execution processes from the M task execution processes under the condition that the target task queue is monitored to comprise a target flow recording task issued by a task issuing process, wherein the target flow recording task is used for recording the flow generated by the online service of the target service;

and the recording module is used for recording the flow generated by the online service of the target service through the target task execution process based on the target flow recording task.

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the methods of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform any one of the methods of the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements any of the methods of the first aspect.

According to the technology disclosed by the invention, the problems of complex flow and poor expansibility of recording the online flow of the business service are solved, the flow of recording the online flow of the business service is simplified, and the expansibility of recording the online flow of the business service is improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a schematic flow chart of a traffic recording method according to a first embodiment of the present disclosure;

fig. 2 is a schematic flowchart of a task execution process executing a traffic recording task;

fig. 3 is a schematic diagram of the architecture of a traffic recording center;

fig. 4 is a schematic structural diagram of a traffic recording apparatus according to a second embodiment of the present disclosure;

FIG. 5 is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

First embodiment

As shown in fig. 1, the present disclosure provides a traffic recording method, including the following steps:

step S101: and monitoring a target task queue subscribed by the M task execution processes aiming at the target service.

Wherein M is a positive integer.

In the embodiment, the traffic recording method relates to the technical field of computers, in particular to the technical field of traffic recording of online services, and can be widely applied to map service scenes. The traffic recording method according to the embodiment of the present disclosure may be executed by the traffic recording apparatus according to the embodiment of the present disclosure. The traffic recording apparatus of the embodiment of the present disclosure may be configured in any electronic device, and the electronic device may be deployed in a traffic recording center to execute the traffic recording method of the embodiment of the present disclosure.

The target service may be any service running on-line, such as a map service. In the process of the target service running on line, various service requests of users on line can be received, the service is provided for the users on line based on the service requests, and the data interacted with the users on line can form the on-line flow generated after the service of the target service is on line.

Because the online traffic can reflect the real state of the service after being online compared with the manually constructed traffic, the online traffic is usually recorded to perform performance and function tests on the service. The purpose of this embodiment is to implement recording of traffic generated by an online service of a target service by using a distributed architecture of a publish-subscribe mode.

The traffic recorder can deploy a task publishing process (which can be called as a task distributor Master), M task executing processes (which can be called as a task subscriber Worker) and a target task queue in a distributed framework of a publish-subscribe mode. If M is greater than 1, that is, under the condition that the traffic recording device deploys at least two task execution processes, the at least two task execution processes may be deployed in a distributed manner.

The M task execution processes may subscribe to the same task queue for the target service, and the subscribed same task queue is the target task queue. The target task queue can be used for storing a traffic recording task issued by the task issuing process for subscribing the M task executing processes, and the traffic recording task is used for recording traffic generated by the online service of the target service. The target task queue may be implemented by Redis or Kafka.

The flow recording device can monitor whether a target task queue has a flow recording task issued by the task issuing process in real time or periodically. The monitoring mode includes but is not limited to: reading a target task queue, judging whether the target task queue is empty or not, if not, monitoring that the target task queue comprises a flow recording task issued by a task issuing process, and if so, monitoring that the flow recording task is not issued by the task issuing process.

The distributed architecture of the publish-subscribe mode can be implemented based on the distributed system Celery framework of the programming language Python, such as Python 3.

Step S102: and under the condition that the target task queue comprises a target flow recording task issued by a task issuing process, determining a target task execution process from the M task execution processes, wherein the target flow recording task is used for recording the flow generated by the online service of the target service.

In this step, when it is monitored that the target task queue includes the target traffic recording task issued by the task issuing process, a target task execution process may be determined from the M task execution processes, where the target task execution process is used to execute the target traffic recording task issued by the task issuing process.

When M is 1, the target task execution process may serially execute the traffic recording task issued by the task issuance process, and the traffic recording device may determine the target task execution process from the M task execution processes after receiving completion of task execution returned by the target task execution process, so as to allow the target task execution process to take away the target traffic recording task issued first from the target task queue.

When M is greater than 1, if the number of the task execution processes is greater than or equal to the number of the target traffic recording tasks issued by the task issuing process at the time, at this time, the M task execution processes may execute the target traffic recording tasks issued by the task issuing process in parallel.

When M is greater than 1, if the number of the task execution processes is less than the number of the target traffic recording tasks issued by the task issuing process at the time, the M task execution processes can execute the target traffic recording tasks issued by the task issuing process in parallel, sacrifice the timeliness of task execution, and enable the same task execution process to execute a plurality of target traffic recording tasks in series.

By controlling the flexible expansion and contraction of the number of task execution processes, a self-defined balance between resource use and execution efficiency can be achieved. When the execution efficiency and the execution timeliness are emphasized, the deployment number of the task execution processes can be increased, and the recording flow tasks in the queue at the same time can be immediately claimed by the idle task execution processes; when the resource use is emphasized, the deployment of the task execution process can be properly reduced, the timeliness of the task execution is sacrificed, and the same task execution process can serially execute a plurality of flow recording tasks.

In an optional embodiment, the traffic recording device may determine, from the M task execution processes, a target task execution process for executing the target traffic recording task according to factors such as process performance and network bandwidth. For example, the M task execution processes can be sorted according to the sequence from good to bad according to factors such as process performance and network bandwidth, and the task execution processes with better process performance and network bandwidth can take away the flow recording task at the head of the target task queue in sequence.

In another optional implementation manner, M task execution processes may contend to execute a target traffic recording task issued by a task issuing process in a contention mode, the M task execution processes may monitor a target task queue, and send a task pull request when it is determined that the target task queue has the target traffic recording task, and after the task issuing process issues the target traffic recording task, the traffic recording device may allocate a traffic recording task located at the head of the queue in the target task queue to a target task execution process that sends the task pull request earliest (corresponding to the task pull request that is obtained earliest by the traffic recording device). Accordingly, one or more target task execution processes can sequentially contend for the target traffic recording task issued by the task issuing process.

And under the condition that the task execution process takes away the flow recording task at the head of the target task queue, the flow recording task at the head of the target task queue is updated until all the flow recording tasks in the target task queue are taken away.

In the case where the target task execution process is determined from the M task execution processes, the traffic recording device may allow the target task execution process to take away the target traffic recording task.

It should be noted that, the traffic recording device may determine a target traffic recording task from the task execution processes in the idle state among the M task execution processes, so as to ensure that one task execution process executes one traffic recording task at a time.

Step S103: and recording the traffic generated by the online service of the target service through the target task execution process based on the target traffic recording task.

In this step, the target traffic recording task in the target task queue may be pulled through the target task execution process, and the target traffic recording task may be executed through each target task execution process, and executed offline, so as to record a flow generated by an online service of the target service.

In an application scenario, such as a map service, an online instance can be recorded based on a recording function of a Remote Procedure Call (RPC) framework of the map service, the RPC framework used by the map service can have a capability of recording a service request to a local disk, and based on the capability, an individual online instance or a batch of online instances can be controlled to start RPC request recording by a HyperText Transfer Protocol (HTTP) instruction, and the online service request can be recorded to the disk. And an online transmission channel can be established, and the recorded data of each instance is transmitted to a distributed File System (AFS) cluster for temporary storage. Correspondingly, the flow data matched with the target flow recording task can be pulled from the corresponding position in the AFS cluster through the target task execution process.

In an optional implementation manner, the configuration information of the target traffic recording task may be obtained from the configuration database through the target task execution process, and the configuration information is used to instruct the target task execution process to execute the target traffic recording task. And executing the target flow recording task through the target task execution process based on the configuration information so as to record the dump data of the online service with the granularity of an example from a preset online data storage library of the target service.

It may include, but is not limited to, the following information content: the method comprises the steps of triggering time of a target flow recording task, the proportion of flow recorded based on the target flow recording task to flow generated by the online service, the time generated based on the flow recorded by the target flow recording task, identification information of the online service corresponding to the flow recorded based on the target flow recording task, the unloading address of the flow recorded based on the target flow recording task in a preset online data storage library and the like.

In another optional implementation manner, the target traffic recording task itself may be associated with task related information, and when the target traffic recording task is pulled by the target task execution process, the target traffic recording task may be executed by the target task execution process based on the task related information in the target traffic recording task, so as to record the dump data of the online service with the instance as the granularity from the preset online data repository of the target service.

In this embodiment, based on the mode of the task publisher and the task subscriber, distributed deployment of the task subscriber Worker for traffic recording can be realized. The distribution subscription mode based on Master and Worker can realize the customization of the traffic recording task, and the deployment is simple. Compared with a mode of manually pulling an online instance log for conversion or recording the traffic of an online single instance in a middleware mode, after the system is deployed, the deployed processes can be highly multiplexed to execute each issued traffic recording task, and programs do not need to be repeatedly developed and redeployed for each traffic recording task, so that the flow is simple, the reusability is high, and the execution efficiency is high. And moreover, the flow recording task is issued through the task issuing process, so that the flow recording task can be adapted to the online service of a plurality of product lines in the target service, and the expansibility is strong.

Optionally, the step S102 specifically includes:

acquiring task pull requests sent by N first task execution processes aiming at the target flow recording task, wherein the N first task execution processes are task execution processes in an idle state in the M task execution processes, and N is a positive integer smaller than or equal to M;

determining the target task execution process from the N first task execution processes, where the target task execution process is a task execution process that sends a target task pull request from the N first task execution processes, and the target task pull request is: and taking the issuing time of the target flow recording task as a reference, and obtaining the task pull request at the earliest.

In this embodiment, when the task execution process is in an idle state, the flow recording task may be requested to be executed, and specifically, a task pull request may be sent to request the flow recording task to be executed.

The N first task execution processes in the idle state can perform the target flow recording task issued by the task issuing process in a contention mode, monitor the target task queue, send a task pull request under the condition that the target flow recording task exists in the target task queue, and after the task issuing process issues the target flow recording task, the flow recording device can distribute the flow recording task at the head of the queue in the target task queue to the target task execution process which sends the task pull request (corresponding to the task pull request which is obtained by the flow recording device at the earliest time) in the N first task execution processes. Correspondingly, one or more target task execution processes can be determined from the N first task execution processes, and the one or more target task execution processes can sequentially contend for a target traffic recording task issued by the task issuing process.

For example, the 5 first task execution processes are a task execution process a1, a task execution process a2, a task execution process A3, a task execution process a4 and a task execution process a5, the task issuing process issues two traffic recording tasks at a certain time, which are a traffic recording task B1 and a traffic recording task B2, respectively, and the traffic recording task B1 is located at the head of the target task queue. And then task pulling requests sent by the task execution process A3, the task execution process A4, the task execution process A5, the task execution process A2 and the task execution process A1 are respectively acquired in time sequence. Since the task pull request sent by the task execution process A3 is obtained earliest, the task execution process A3 takes the traffic recording task B1, and the head of the target task queue is updated to the traffic recording task B2. In the case where task execution process A3 takes traffic recording task B1, task execution process a4 will become the task execution process that sent the task pull request the earliest, taking traffic recording task B2 accordingly.

In this embodiment, the M task execution processes compete for the target traffic recording task issued by the task issuance process in the contention mode, so that the target traffic recording task issued by the task issuance process can be recorded, and the execution efficiency of the traffic recording task is further improved.

Optionally, step S103 specifically includes:

acquiring configuration information of the target flow recording task from a configuration database through the target task execution process;

and executing the target flow recording task through the target task execution process based on the configuration information so as to record the dump data of the online service with the granularity of an example from a preset online data storage library of the target service.

In this embodiment, the configuration information is used to instruct the target task execution process to execute the target traffic recording task.

The user can configure different flow recording tasks through the visual task configuration front end, the visual task configuration front end can be directly connected with the configuration database, the configuration database can be a relational database management system MySQL, and the MySQL can store the corresponding relation between the flow recording tasks and the configuration information of the flow recording tasks. The flow recording request can be sent to the flow recording device, and various configuration information of the flow recording task is sent to the configuration database for the task execution process to use when executing the flow recording task. The parameters configurable by the user at the front end can include: the service of recording the flow, the time period of recording the flow, the triggering time of the task of recording the flow, the proportion of recording the flow, whether to execute the screening strategy, the screening rule, the accumulation request rule, etc.

Correspondingly, when the target task execution process pulls the target traffic recording task, the configuration information of the target traffic recording task can be read from the configuration database through the target task execution process.

The preset online data storage library can be an AFS cluster, and for map services (other services are also applicable), the online instance recording can be realized based on the recording function of the RPC frame of the map service, the RPC frame used by the map service can have the capability of recording the service request to a local disk, and based on the capability, the RPC request recording can be started by controlling individual online instances or batch online instances through an offline HTTP instruction, and the online service request is recorded to the disk. And an online transmission channel can be established, and the recorded data of each instance is transmitted to the AFS cluster for temporary storage.

Correspondingly, the flow recording device can record the unloading data matched with the configuration information from the AFS cluster through the target task execution process based on the configuration information. The data in the preset online data storage library is transferred and stored by taking an online service example as granularity.

In the embodiment, the front-end configuration database and the back-end configuration database are configured through the visual task, so that user customization of various recording scenes can be realized, the flexibility of flow recording can be realized, and the operation flow of a user is shortened. The visual task configuration front end supports one-click operation of a user, the flow is simple, for example, the user can submit task setting parameters at the front end to realize one-click access of a flow recording task, online flow recording at different time intervals and different proportions can be realized by changing the front end parameters, and compared with a log conversion and middleware recording mode, the method can greatly reduce the execution cost of the user and save testing manpower resources.

Moreover, the universal applicability of flow recording can be realized through the recording function of the service RPC framework, and the problem of incomplete flow coverage caused by log conversion or middleware recording only recording individual instances can be avoided based on the flow extraction of the online full instances, so that the coverage rate of the recorded online flow of the service can be improved.

Optionally, the configuration information includes at least one of:

recording identification information of an online service corresponding to the traffic recorded by the target traffic recording task;

the address of the data storage library on the preset line based on the flow recorded by the target flow recording task;

recording the time generated by the recorded flow of the task based on the target flow;

recording the triggering time of the task by the target flow;

and recording the flow of the target flow based on the target flow recording task in proportion to the flow generated by the online service.

In the embodiment, the triggering time of the flow recording task, the time and proportion of the flow required to be recorded by the flow recording task, the identification of the corresponding on-line service, the address of the preset on-line data storage library and other information are configured to indicate the task execution process to execute the flow recording task.

Optionally, when the configuration information includes the trigger time of the target traffic recording task, the executing the target traffic recording task through the target task execution process based on the configuration information includes:

under the condition that the current time reaches the trigger time, marking the state of the target flow recording task issued at the current time as an executed state, and recording dump data of the online service with the granularity of an example from a preset online data storage library of the target service through a target task execution process based on the configuration information to obtain a target data set of the online service;

and when the current time does not reach the trigger time, exiting the target task execution process to record the flow of the target flow recording task.

In this embodiment, the triggering time may be a time for executing a target traffic recording task, and may be immediate triggering or timing triggering.

Under the condition that the trigger time indication of the target traffic recording task is triggered immediately, namely the current time reaches the trigger time, the state of the target traffic recording task issued at the current time needs to be marked as an executed state, so as to prevent other task execution processes from executing for the second time. And based on the configuration information, through a target task execution process, unloading data of the online service with the instance as the granularity is recorded from a preset online data repository of the target service, and a target data set of the online service is obtained.

And under the condition that the triggering time of the target flow recording task indicates timing triggering, namely the current time does not reach the triggering time, exiting the target task execution process to record the flow of the target flow recording task, namely ending the flow recording process.

In the embodiment, when the current time reaches the trigger time, the state of the target flow recording task issued at the current time is marked as the executed state, so that the secondary execution of other task execution processes can be prevented, and when the current time does not reach the trigger time, the flow recording process is ended, so that the resource utilization rate of the task execution processes can be improved.

Optionally, the recording, by the target task execution process based on the configuration information, dump data of the online service with granularity of an instance from a preset online data repository of the target service to obtain a target data set of the online service includes:

based on the configuration information, through the target task execution process, recording dump data of the online service with the example as the granularity from a preset online data storage library of the target service;

aggregating the recorded unloading data to obtain a first data set;

and performing vocabulary screening on the first data set to obtain a target data set of the online service.

Fig. 2 is a schematic flow chart of a task execution process executing a traffic recording task, and as shown in fig. 2, a normal work flow of the task execution process may be: and reading the configuration database, pulling the dump data which corresponds to the recording time period and takes the example as the granularity from the AFS cluster under the condition that the current time reaches the trigger time, aggregating the dump data, screening word lists, uploading the word lists, notifying mails, and ending. Therefore, the flow recording task can be executed through the task execution process so as to record the flow of the service on the service line.

And under the condition that the current time does not reach the trigger time, ending the flow recording process. Where a vocabulary refers to a collection of service requests.

Reading a configuration database: after the task execution process takes a certain online service traffic recording task from the target task queue, various configurations corresponding to the online service or the traffic recording task, including expected trigger time set by a user, can be read from the configuration database, and the traffic recording process can be directly quitted when the trigger time is not reached. If the triggering time is up, the state of the flow recording task issued at the current time is marked as an executed state, and other task execution processes are prevented from executing for the second time. And reading configuration information such as storage paths, sampling ratios and the like of the traffic needing to be recorded in the AFS cluster from the configuration database.

Pulling dump data which corresponds to the recording time period and takes the example as granularity from the AFS cluster: because the full-time recorded data of the online instance is transferred to the designated AFS path through the RPC frame of the online service, the task execution process only needs to pull the original recorded data (namely the transferred data) of the corresponding time period from the AFS according to the configured recording time period. And if the user configures the sampling proportion, randomly sampling and pulling the data according to the sampling proportion.

Aggregating the dump data: the original recorded data with the granularity of the online example is aggregated into a data collection.

And (3) word list screening: and screening the aggregated dump data based on various screening methods (such as static screening, dynamic screening and the like), so that the request coverage rate is ensured, the vocabulary level is reduced as much as possible, and the pressure measurement efficiency is improved. For example, 200 ten million service requests can be selected from 5000 ten million service requests by vocabulary selection.

Uploading a word list: after the vocabulary is manufactured, the vocabulary can be automatically uploaded to a pressure sending platform for a user to use in pressure measurement.

E, mail notification: after the flow recording task is completed, a notification mail can be automatically sent to the user to notify the user that the recording of the pressure-measured data is completed.

Optionally, the method further comprises at least one of:

regularly issuing a flow recording task to the target task queue through the task issuing process to obtain the target flow recording task;

when the state of a preset service interface of the target service is monitored to be changed through the task issuing process, issuing a flow recording task corresponding to a target service into the target task queue through the task issuing process to obtain the target flow recording task, wherein the target service is an online service corresponding to the preset service interface;

and under the condition that a flow recording request sent by a task configuration front end is received through the task issuing process, a flow recording task corresponding to the flow recording request is issued to the target task queue through the task issuing process so as to obtain the target flow recording task.

In this embodiment, the task issuing process may issue the traffic recording task to the target task queue through three ways, namely, timing issue, polling issue, and front-end triggering of user visual task configuration, so that the issue of the traffic recording task may be realized. The task issuing process is only responsible for issuing the traffic recording task to the target task queue, and does not care when the traffic recording task is executed.

And (3) timing release: a schedule may be maintained in the task issuing process, for example, the traffic recording device may issue the traffic recording task to the target task queue for the task executing process to subscribe to by issuing the traffic recording task once every 5 minutes or 10 minutes.

And (3) polling and issuing: the flow recording device can poll a fixed interface through the task issuing process, when the state of the interface changes (such as data version updating of online service, online of new service and the like), the task issuing process can monitor the state change, and can issue a flow recording task corresponding to the service to the target task queue.

User visualization task configuration front-end trigger: the user can click an execution button at the front end of the visual task configuration, the front end of the visual task configuration correspondingly sends the flow recording request to the flow recording device, and the flow recording device can send the flow recording task corresponding to the flow recording request to the target task queue after receiving the flow recording request of the user through the task issuing process.

Optionally, the method further includes:

pulling a service request of an online service of the target service from a preset online data storage library of the target service through a task execution process in an idle state in the M task execution processes;

and based on a preset rule configured by a user, screening the service request of the on-line service of the target service to obtain the rare flow of the on-line service of the target service, wherein the rare flow is the flow of which the ratio of the flow generated by the on-line service is less than a preset threshold value.

In this embodiment, when the task execution process does not contend for the traffic recording task in the target task queue, or the target task queue does not have the traffic recording task, the traffic recording device may execute the task with the accumulated special requests through the task execution process, and the task execution process may pull a large number of service requests from the AFS cluster at idle time, and screen the service requests according to preset rules (including screening rules, accumulation request rules, and the like) configured by the user through the visual task configuration front end, so as to screen rare traffic of the online service of the target service. The rare traffic may be traffic with a smaller occupation ratio than a preset threshold in traffic generated by the online service, and the preset threshold may be set according to an actual situation, where the representation occupation ratio is usually small.

The flow recording device can upload the service request obtained by screening to the AFS cluster for the second time through the task execution process. The task execution process can repeatedly execute the task when the task execution process is in an idle state every day until the screening is finished to the set number, and the mail informs the user that the screening is finished.

Correspondingly, the user can aggregate the rare flow and the recorded flow uploaded to the pressure sending platform for pressure measurement in the subsequent process, and thus high coverage of the service request set can be realized through the flow recording and special rare request accumulation functions.

Optionally, the method further includes:

receiving an update command issued by a client process through M server processes, wherein each task execution process in the M task execution processes corresponds to one server process;

updating the M task execution processes by the M server processes based on the update command.

In this embodiment, the traffic recording device may include a task issuing process and a large number of task execution processes, and these task execution processes need to be updated synchronously each time a system code is updated.

Specifically, a Client process Client can be deployed on an electronic device, code management is performed on the electronic device, the latest task execution process codes are maintained, each task execution process can correspond to one production container, and a server process Agent and a task execution process are deployed on each production container. The Agent is essentially a web server webserver and can receive various instructions from the Client, including commands for downloading, decompressing, starting and the like. And issuing an update command to all agents in the production container with the task execution process through the Client, and correspondingly receiving the update command issued by the Client process through the server process Agent. The Agent can execute a series of tasks such as pulling codes, unpacking, starting a task execution process and the like. The series of operations are packaged into a fixed tool, so that the agility of updating the task execution process can be realized, and the labor cost is greatly saved.

Optionally, the method further includes:

creating K server processes, wherein K is a positive integer;

receiving a deployment command sent by a client process through the K server processes;

and deploying K task execution processes through the K server processes based on the deployment command, wherein the K task execution processes are different from the M task execution processes, and the K task execution processes are used for executing the flow recording task issued by the task issuing process.

For each additional traffic recording task served on the line, a task execution process can be added for the sake of execution efficiency.

In order to implement one-click deployment of the task execution process, the code of the task execution process is rapidly deployed into a production container, K server processes corresponding to the K task execution processes to be deployed one by one can be created, and each server process is deployed into a new production container.

And issuing a deployment command to an Agent in a new production container through the Client, and correspondingly receiving the deployment command issued by the Client process through the server process Agent. The Agent can execute a series of tasks such as pulling codes, unpacking, starting a task execution process and the like, and the task execution process is deployed in a new production container, so that the rapid deployment of a large number of task execution processes can be realized, and the operation and maintenance cost is reduced.

The following describes the traffic recording method according to this embodiment with reference to fig. 3 as an example.

Fig. 3 is a schematic architecture diagram of a traffic recording center station, and as shown in fig. 3, the traffic recording center station may include a task issuing process, a task executing process, a schedule, a target task queue, a visual task configuration front end, a configuration database, a client process, and a server process.

The task issuing process can issue the flow recording task to a target task queue for a plurality of task execution processes to subscribe through three modes of timing issuing, polling issuing and front-end triggering of user visual task configuration.

After the task issuing process issues the flow recording task through the contention mode, the task executing processes take the flow recording task out of the target task queue, acquire the flow recording task or the configuration information of the online service configured by the user through the visual task configuration front end from the configuration database, and execute the flow recording task based on the configuration information so as to record the flow of the online service from the AFS cluster.

The method can be issued to a server process through a client process command, and a large number of task execution processes are deployed in batches or updated in batches through the server process.

Second embodiment

As shown in fig. 4, the present disclosure provides a traffic recording device 400, including:

a monitoring module 401, configured to monitor a target task queue subscribed by M task execution processes for a target service, where M is a positive integer;

a determining module 402, configured to determine a target task execution process from the M task execution processes when it is monitored that the target task queue includes a target traffic recording task issued by a task issuing process, where the target traffic recording task is used to record traffic generated by an online service of the target service;

a recording module 403, configured to record, based on the target traffic recording task, traffic generated by the online service of the target service through the target task execution process.

Optionally, the determining module 402 is specifically configured to:

Optionally, the recording module 403 includes:

the acquisition submodule is used for acquiring the configuration information of the target flow recording task from a configuration database through the target task execution process;

and the task execution submodule is used for executing the target flow recording task through the target task execution process based on the configuration information so as to record the unloading data of the online service with the granularity of an example from a preset online data storage library of the target service.

Optionally, the configuration information includes at least one of:

the address of the data storage library on the preset line based on the traffic recorded by the target traffic recording task;

recording the triggering time of the task by the target flow;

Optionally, when the configuration information includes the trigger time of the target traffic recording task, the task execution sub-module includes:

the marking unit is used for marking the state of the target flow recording task issued at the current time as an executed state under the condition that the current time reaches the trigger time;

a task execution unit, configured to record, through the target task execution process and based on the configuration information, dump data of the online service, where the dump data has a granularity of an instance, from a preset online data repository of the target service, so as to obtain a target data set of the online service;

Optionally, the task execution unit is specifically configured to:

based on the configuration information, through the target task execution process, the dump data of the online service with the granularity of an example is recorded from a preset online data repository of the target service;

aggregating the recorded dump data to obtain a first data set;

Optionally, the apparatus further comprises:

the first issuing module is used for issuing a flow recording task to the target task queue at regular time through the task issuing process so as to obtain the target flow recording task;

the second issuing module is used for issuing a flow recording task corresponding to a target service to the target task queue through the task issuing process under the condition that the state of a preset service interface of the target service is monitored to change through the task issuing process so as to obtain the target flow recording task, wherein the target service is an online service corresponding to the preset service interface;

and the third issuing module is used for issuing a traffic recording task corresponding to the traffic recording request to the target task queue through the task issuing process under the condition that the traffic recording request sent by the task configuration front end is received through the task issuing process so as to obtain the target traffic recording task.

Optionally, the apparatus further comprises:

an obtaining module, configured to pull, from a preset online data repository of the target service, a service request of an online service of the target service through a task execution process in an idle state among the M task execution processes;

and the screening module is used for screening the service request of the on-line service of the target service based on a preset rule configured by a user to obtain the rare flow of the on-line service of the target service, wherein the rare flow is the flow of which the ratio of the flow generated by the on-line service is less than a preset threshold value.

Optionally, the apparatus further comprises:

the first receiving module is used for receiving an update command issued by a client process through M server processes, wherein each task execution process in the M task execution processes corresponds to one server process;

and the updating module is used for updating the M task execution processes through the M server processes based on the updating command.

Optionally, the apparatus further comprises:

the system comprises a creating module, a selecting module and a processing module, wherein the creating module is used for creating K server processes, and K is a positive integer;

the second receiving module is used for receiving the deployment command sent by the client process through the K server processes;

and the deployment module is used for deploying K task execution processes through the K server processes based on the deployment command, wherein the K task execution processes are different from the M task execution processes, and the K task execution processes are used for executing the flow recording task issued by the task issuing process.

The traffic recording apparatus 400 provided by the present disclosure can implement each process implemented by the traffic recording method embodiment, and can achieve the same beneficial effects, and for avoiding repetition, the details are not repeated here.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 5 illustrates a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 5, the apparatus 500 comprises a computing unit 501 which may perform various appropriate actions and processes in accordance with a computer program stored in a Read Only Memory (ROM)502 or a computer program loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The calculation unit 501, the ROM 502, and the RAM 503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

A number of components in the device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, or the like; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

The computing unit 501 may be a variety of general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 501 performs the respective methods and processes described above, such as the traffic recording method. For example, in some embodiments, the traffic recording method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509. When loaded into RAM 503 and executed by the computing unit 501, a computer program may perform one or more of the steps of the traffic recording method described above. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the traffic recording method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A traffic recording method comprises the following steps:

2. The method of claim 1, wherein the determining a target task execution process from the M task execution processes comprises:

3. The method of claim 1, wherein the recording, by the target task execution process, traffic generated by an online service of the target service based on the target traffic recording task comprises:

4. The method of claim 3, wherein the configuration information comprises at least one of:

recording the triggering time of the task by the target flow;

5. The method according to claim 4, wherein, in a case that the configuration information includes a trigger time of the target traffic recording task, the executing, by the target task execution process, the target traffic recording task based on the configuration information includes:

6. The method according to claim 5, wherein the obtaining, by the target task execution process based on the configuration information, a target data set of the online service by recording, from a preset online data repository of the target service, dump data of the online service at a granularity of an instance includes:

aggregating the recorded dump data to obtain a first data set;

7. The method of any of claims 1 to 6, further comprising at least one of:

8. The method of any of claims 1 to 6, further comprising:

9. The method of any of claims 1 to 6, further comprising:

10. The method of any of claims 1 to 6, further comprising:

creating K server processes, wherein K is a positive integer;

11. A traffic recording apparatus comprising:

the determining module is used for determining target task execution processes from the M task execution processes under the condition that the target task queue is monitored to comprise a target flow recording task issued by a task issuing process, wherein the target flow recording task is used for recording flow generated by an online service of the target service;

12. The apparatus of claim 11, wherein the determining means is specifically configured to:

13. The apparatus of claim 11, wherein the recording module comprises:

14. The apparatus of claim 13, wherein the configuration information comprises at least one of:

recording the triggering time of the task by the target flow;

15. The apparatus according to claim 14, wherein in a case that the configuration information includes a trigger time of the target traffic recording task, the task execution sub-module includes:

16. The apparatus according to claim 15, wherein the task execution unit is specifically configured to:

aggregating the recorded dump data to obtain a first data set;

17. The apparatus of any of claims 11 to 16, further comprising:

18. The apparatus of any of claims 11 to 16, further comprising:

an obtaining module, configured to pull, through a task execution process in an idle state among the M task execution processes, a service request of an online service of the target service from a preset online data repository of the target service;

19. The apparatus of any of claims 11 to 16, further comprising:

20. The apparatus of any of claims 11 to 16, further comprising:

21. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-10.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-10.