CN116094959A

CN116094959A - Network data processing method and device, electronic equipment and storage medium

Info

Publication number: CN116094959A
Application number: CN202310036367.8A
Authority: CN
Inventors: 高佳
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2023-01-09
Filing date: 2023-01-09
Publication date: 2023-05-09

Abstract

The disclosure relates to a network data processing method, a network data processing device, an electronic device and a storage medium. The method comprises the following steps: responding to pressure test configuration operation triggered by aiming at the target service, and generating a test task corresponding to the target service; the test task comprises test information corresponding to each sub-service in the target service; acquiring a target data set and a candidate instance set corresponding to the sub-service according to each piece of test information; the candidate instance set is created by a cluster system and is used for providing resources for executing sub-services; performing pressure test on the target instance in the candidate instance set according to the target data set to obtain single instance service capacity of the target instance; the service capacity of the cluster system for the sub-service is determined based on the single instance service capacity and the total number of instances in the set of candidate instances. By adopting the method, the network data processing efficiency is improved.

Description

Network data processing method and device, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of server testing, and in particular relates to a network data processing method, a device, electronic equipment and a storage medium.

Background

In the server cluster, service capacity assessment is carried out according to various types of service requirements, and a service capacity result of the cluster system for various types of services is obtained.

In the current service capacity assessment method, a cluster system generally adopts a pressure test mode to determine a service capacity result aiming at the service. Specifically, in the stress test process, the test model of the tree structure includes nodes, each node corresponds to one sub-service included in the processing service, and when a certain node is subjected to service fusing, the service capacity result of the sub-service corresponding to the node is determined. And then, increasing the service capacity of the node to ensure that the node does not interfere with the pressure test process, repeating the execution process, determining the service capacity results of the sub-services corresponding to all the nodes, and ending the pressure test.

However, in the current service capacity assessment method, each time the stress test process needs to mobilize all nodes in the tree structure to perform target data processing, and then determine the service capacity of each type of sub-service one by one, which results in lower assessment efficiency of the service capacity.

Disclosure of Invention

The disclosure provides a network data processing method, a device, an electronic device and a storage medium, so as to at least solve the problem of low network data processing efficiency in the related art. The technical scheme of the present disclosure is as follows:

According to a first aspect of embodiments of the present disclosure, there is provided a network data processing method, the method comprising:

responding to pressure test configuration operation triggered by aiming at a target service, and generating a test task corresponding to the target service; the test task comprises test information corresponding to each sub-service in the target service;

acquiring a target data set and a candidate instance set corresponding to the sub-service according to each piece of test information; the candidate instance set is created by a cluster system and is used for providing resources for executing the sub-service;

performing pressure test on a target instance in the candidate instance set according to the target data set to obtain single instance service capacity of the target instance;

based on the single instance service capacity and the total number of instances in the set of candidate instances, a service capacity of the clustered system for the sub-service is determined.

In an exemplary embodiment, the obtaining the target data set corresponding to the sub-service includes:

acquiring target data corresponding to the sub-service;

and calling a preset data processing interface to process target data corresponding to the sub-service to obtain a target data set corresponding to the sub-service, wherein the target data set comprises a plurality of target data.

In an exemplary embodiment, the performing a pressure test on a target instance in the candidate instance set according to the target data set to obtain a single instance service capacity of the target instance includes:

determining a preset number of target examples in the candidate example set according to a preset example screening strategy;

performing pressure test on each target instance based on a target data set corresponding to the sub-service and a preset pressure test step length to obtain single instance service capacity of each target instance for the sub-service; the pressure test step size characterizes a variation of a data transmission amount of the target data transmitted to the target instance each time.

In an exemplary embodiment, the performing, based on the target data set corresponding to the sub-service and a preset pressure test step length, the pressure test on each target instance to obtain a single instance service capacity of each target instance for the sub-service includes:

according to the preset data transmission quantity, transmitting the target data in the target data set to a target instance;

in the process of processing the target data by the target instance, if service fusing does not occur in the target instance, updating the data transmission amount according to the current preset pressure measurement step length, and executing the step of transmitting the target data in the target data set to the target instance based on the updated data transmission amount;

If the target instance is subjected to service fusing, determining a target service fusing result corresponding to the target instance under the condition that a preset service fusing condition is met according to a preset pressure measurement step length, a service fusing callback strategy and the current data transmission quantity; and taking the target service fusing result as a single-instance service capacity of the target instance for the sub-service.

In an exemplary embodiment, the determining the service capacity of the cluster system for the sub-service based on the single instance service capacity and the total number of instances in the candidate instance set includes:

sequencing the single instance service capacity corresponding to each target instance according to the sequence from small to large to obtain a single instance service capacity sequence;

in the single-instance service capacity sequence, determining a target single-instance service capacity meeting a preset fractional number condition;

determining the service capacity of the cluster system for the sub-service based on the target single instance service capacity and the total number of instances in the candidate instance set.

In an exemplary embodiment, the method further comprises:

acquiring the total number of instances in the candidate instance set corresponding to the sub-service, the required service capacity of the sub-service and the single instance service capacity peak value of the target instance in the pressure test process;

Determining the planned service capacity of the cluster system for the sub-service according to the required service capacity of the cluster system for the sub-service and a preset redundancy parameter value;

determining a resource allocation result of the cluster system to the sub-service according to the total number of instances in the candidate instance set, the single instance service capacity peak value and the planned service capacity;

and carrying out resource configuration management on the total number of the instances in the candidate instance set in the cluster system based on the resource configuration result.

In an exemplary embodiment, the determining a resource allocation result of the cluster system to the sub-service according to the total number of instances in the candidate instance set, the single instance service capacity peak value, and the planned service capacity includes:

determining the number of target instance demands according to the ratio of the planned service capacity to the single instance service capacity peak value;

and determining a resource configuration result of the cluster system to the sub-service according to the difference value between the total number of the instances in the candidate instance set and the required number of the target instances.

In an exemplary embodiment, after the determining the resource configuration result of the cluster system for the sub-service, the method further includes:

And if the cluster system characterizes the insufficient resource allocation of the resource allocation result of the sub-service, generating and outputting prompt information of the insufficient resource allocation.

According to a second aspect of embodiments of the present disclosure, there is provided a network data processing apparatus, the apparatus comprising:

the generating unit is configured to execute a pressure test configuration operation triggered by responding to a target service and generate a test task corresponding to the target service; the test task comprises test information corresponding to each sub-service in the target service;

the first acquisition unit is configured to acquire a target data set and a candidate instance set corresponding to the sub-service according to each piece of test information; the candidate instance set is created by a cluster system and is used for providing resources for executing the sub-service;

the testing unit is configured to execute pressure testing on target examples in the candidate example set according to the target data set, and obtain single-example service capacity of the target examples;

a first determining unit configured to perform determining a service capacity of the cluster system for the sub-service based on the single instance service capacity and a total number of instances in the set of candidate instances.

In an exemplary embodiment, the first acquisition unit includes:

an acquisition subunit configured to perform acquisition of target data corresponding to the sub-service;

and the processing subunit is configured to execute the call of a preset data processing interface to process the target data corresponding to the sub-service to obtain a target data set corresponding to the sub-service.

In an exemplary embodiment, the test unit includes:

a determining subunit configured to perform a predetermined number of target instances in the candidate instance set according to a predetermined instance screening policy;

the processing subunit is configured to execute pressure test on each target instance based on a target data set corresponding to the sub-service and a preset pressure test step length to obtain single instance service capacity of each target instance for the sub-service; the pressure test step size characterizes a variation of a data transmission amount of the target data transmitted to the target instance each time.

In an exemplary embodiment, the processing subunit is specifically configured to send, according to a preset data sending amount, target data in the target data set to a target instance;

In an exemplary embodiment, the first determining unit includes:

a sorting subunit configured to perform sorting processing on the single-instance service capacities corresponding to the target instances according to the order from small to large, so as to obtain a single-instance service capacity sequence;

a first determining subunit configured to determine, in the single-instance service capacity sequence, a target single-instance service capacity that satisfies a preset fractional number condition;

A second determination subunit configured to perform determining a service capacity of the clustered system for the sub-service based on the target single instance service capacity and a total number of instances in the set of candidate instances.

In an exemplary embodiment, the apparatus further comprises:

the second acquisition unit is configured to acquire the total number of the instances in the candidate instance set corresponding to the sub-service, the required service capacity of the sub-service and the single instance service capacity peak value of the target instance in the pressure test process;

a second determining unit configured to determine a planned service capacity of the cluster system for the sub-service according to a required service capacity of the cluster system for the sub-service and a preset redundancy parameter value;

a third determining unit configured to perform determining a resource allocation result of the cluster system to the sub-service according to the total number of instances in the candidate instance set, the single instance service capacity peak, and the planned service capacity;

and the configuration management unit is configured to perform resource configuration management on the total number of the instances in the candidate instance set in the cluster system based on the resource configuration result.

In an exemplary embodiment, the third determining unit is specifically configured to determine the target number of instance demands according to a ratio of the planned service capacity to the peak value of the single instance service capacity;

In an exemplary embodiment, the apparatus further comprises:

and the prompting unit is configured to generate and output prompting information of insufficient resource configuration when the cluster system characterizes the insufficient resource configuration of the resource configuration result of the sub-service.

According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the network data processing method according to any of the first aspects above.

According to a fourth aspect of embodiments of the present disclosure, there is provided a computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the network data processing method as described in any one of the first aspects above.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product, which instructions, when executed by a processor of an electronic device, enable the electronic device to perform the network data processing method of any one of the first aspects above.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:

according to the method, a cluster system responds to pressure test configuration operation triggered by target service, test tasks corresponding to the target service are generated, each sub-service contained in the target service in each test task corresponds to test information, pressure test is independently conducted on each sub-service in the target service according to each test information, single-instance service capacity of a sub-service of a target instance is determined, and service capacity of the cluster system to the sub-service is determined based on the single-instance service capacity of the sub-service and the total number of instances in a candidate instance set. In the pressure test process of the sub-service, the pressure test of other sub-services in the linkage target service is not needed, and all the examples in the candidate example set are not needed to be tested, so that the network data processing efficiency of the sub-service is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.

Fig. 1 is a cluster system architecture diagram illustrating a network data processing method according to an exemplary embodiment.

Fig. 2 is a flow chart illustrating a method of network data processing according to an exemplary embodiment.

FIG. 3 is a flowchart illustrating a method of acquiring a target data set, according to an example embodiment.

FIG. 4 is a flow chart illustrating a method of pressure testing of a target instance, according to an example embodiment.

Fig. 5 is a flow chart of a single instance service capacity determination method for one target instance, according to one illustrative embodiment.

Fig. 6 is an example flow chart illustrating a single instance service capacity determination method according to an example embodiment.

Fig. 7 is a flowchart illustrating a method of determining service capacity of a sub-service according to an exemplary embodiment.

Fig. 8 is a flowchart illustrating a cluster system resource configuration management method, according to an example embodiment.

Fig. 9 is a flowchart illustrating a cluster system resource configuration management method, according to an example embodiment.

Fig. 10 is a block diagram of a network data processing apparatus according to an exemplary embodiment.

Fig. 11 is a block diagram of an electronic device, according to an example embodiment.

Detailed Description

In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

It should be further noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for presentation, analyzed data, etc.) related to the present disclosure are information and data authorized by the user or sufficiently authorized by each party.

The network data processing method provided by the disclosure can be applied to a cluster system as shown in fig. 1. The cluster system comprises 4 layers of a user layer, a strategy layer, a dependence layer and a storage layer. The user layer is used for configuring a test task (also called a network data processing task) of the pressure test of each target service, mobilizing relevant configuration information of other levels and inquiring information in the processing process of the test task. And a policy layer for storing stress test policies, stress test rules, analysis of stress test results (service capacity), and the like for various types of target services. The dependency layer comprises a data dependency module and a platform dependency module, wherein the data dependency module is used for providing data support for a cluster system, for example, related index information of each instance resource in the cluster system, and the type and the number of data in the data dependency module are not limited in the embodiment of the disclosure. A platform dependent module for providing an interface to invoke a third party platform (e.g., an interface to a processing platform) for completing an online pressure test service. And a storage layer, configured to store data during the pressure test, for example, service capacity of each single instance during the pressure test, which is not limited in the embodiments of the disclosure.

Fig. 2 is a flowchart illustrating a network data processing method for use in a cluster system to be tested, as shown in fig. 2, according to an exemplary embodiment, the method comprising the following steps.

In step S210, in response to the stress test configuration operation triggered for the target service, a test task corresponding to the target service is generated.

The test task comprises test information corresponding to each sub-service in the target service.

In implementation, a user triggers a pressure test configuration operation on a target service through a service capacity configuration interface provided by a cluster system, and the cluster system responds to the pressure test configuration operation triggered on the target service to generate a test task corresponding to the target service. Because the types of the target data are different, the target data correspond to each type of the target data, and the sub-service corresponding to the target data exists in the target service. The test task generated by the cluster system includes test information corresponding to a plurality of sub-services in the target service.

In step S220, a target data set and a candidate instance set corresponding to the sub-service are obtained according to each test information.

The candidate instance set is created by the cluster system and is used for providing virtual resources for executing the sub-services corresponding to the candidate instance set. Specifically, the candidate instance set includes at least one instance. Each instance may serve as an on-cloud virtual computing server executing the sub-service, providing virtual resources for executing the sub-service.

In implementation, target data required by various types of sub-services are pre-stored in a cluster system, candidate instance sets are pre-created for each type of sub-service in the cluster system, and the cluster system acquires the target data set corresponding to each sub-service and the candidate instance set corresponding to each sub-service according to test information corresponding to each sub-service. The target data set contains target data corresponding to the sub-service. The specific method for constructing the target data set will be described in detail below, and will not be described herein.

In step S230, according to the target data set, a pressure test is performed on the target instance in the candidate instance set, so as to obtain a single instance service capacity of the target instance.

In implementation, the cluster system screens out a preset number of target instances from the candidate instance set, takes the preset number of target instances as instance representatives in the candidate instance set, and performs pressure test on sub-services on each target instance, so that the service capacity of the candidate instance set for the sub-services is determined according to the single-instance service capacity of each target instance for the sub-services. Specifically, the cluster system sends target data in the target data set to the target instance according to a preset data sending amount, the target instance processes the target data, and in the data processing process, the cluster system continuously adjusts and changes the size of the data sending amount through a preset pressure test step length until the target instance generates service fusing when sending the target data to the target instance according to a certain adjusted data sending amount size, and the cluster system determines the single instance service capacity of the target instance aiming at sub-services.

In step S240, the service capacity of the cluster system for the sub-service is determined based on the single instance service capacity and the total number of instances in the candidate instance set.

In implementation, after determining the single-instance service capacity of each target instance for the sub-service, the cluster system performs sorting processing on the determined single-instance service capacities to obtain a single-instance service capacity sequence. And determining the single-instance service capacity meeting the preset quantile condition as the target single-instance service capacity based on the preset quantile condition in the single-instance service capacity sequence. The target single instance service capacity may be used as an estimate of the service capacity of any one instance in the candidate instance set. The cluster system then determines a service capacity of the cluster system for the sub-service based on the total number of instances included in the candidate instance set and the target single instance service capacity.

In the network data processing method, the cluster system responds to the pressure test configuration operation triggered by the target service to generate a test task corresponding to the target service. The test task comprises test information corresponding to each sub-service in the target service. And then, the cluster system acquires a target data set and a candidate instance set corresponding to the sub-service according to each piece of test information. And the cluster system performs pressure test on the target instance in the candidate instance set according to the target data set to obtain the single instance service capacity of the target instance. The cluster system then determines a service capacity of the cluster system for the sub-service based on the single instance service capacity and the total number of instances in the set of candidate instances. According to the method, sub-services in each target service are separated through a pre-configured test task, pressure test is conducted on target examples of each sub-service respectively to obtain single-example service capacity, and then the service capacity of a cluster system for the sub-service is determined through the single-example service capacity and the total number of examples of a candidate example set of the sub-service. Therefore, in the process of carrying out the pressure test on the sub-service, the pressure test process of other sub-services in the linkage target service is not needed, all the examples of the sub-services of various types in the cluster system are not needed to participate in the pressure test to determine the service capacity, and the efficiency of network data processing is improved.

In an exemplary embodiment, as shown in fig. 3, the obtaining the target data set corresponding to the sub-service in step S220 may be specifically implemented by the following steps:

in step S221, target data corresponding to the sub-service is acquired.

Wherein, for each type of sub-service included in the target service, each type of sub-service corresponds to target data of the type of sub-service, for example, the target data of different types of sub-services may include: display object target data, recommended information target data, and the like, and the type of target data is not limited in the embodiments of the present disclosure.

In implementation, the cluster system acquires target data corresponding to a sub-service to be subjected to pressure test currently. For example, for sub-service a, the cluster system obtains display object target data based on the correspondence between sub-service type a and the target data.

In step S222, a preset data processing interface is invoked to process the target data corresponding to the sub-service, so as to obtain a target data set corresponding to the sub-service.

Wherein the target data set comprises a plurality of target data. For example, the target data may be, but is not limited to, traffic data, and the target data set is a data set containing a large amount of traffic data.

In implementation, the cluster system schedules a preset data processing interface, and processes the target data corresponding to the sub-service to obtain a target data set corresponding to the sub-service. The target data set is used for providing tested data samples in the process of performing pressure test on the target instance.

In this embodiment, a target data set of a sub-service is obtained by processing target data corresponding to the sub-service, so as to perform a stress test on a target instance based on a sufficient amount of target data in the target data set.

In an exemplary embodiment, as shown in fig. 4, in step S230, performing a pressure test on a target instance in a candidate instance set according to a target data set, to obtain a single instance service capacity specific processing procedure of the target instance includes:

in step S402, a preset number of target instances are determined in the candidate instance set according to a preset instance screening policy.

In implementation, an instance screening policy for screening target instances is pre-stored in the cluster system, where the instance screening policy may be, but is not limited to, a random screening policy. Specifically, the cluster system randomly selects a preset number of examples in the candidate example set according to the random selection strategy, and the preset number of examples are used as target examples. Wherein the target instance is used as a representative instance in the candidate instance set for participating in the stress test. Optionally, the target instances with a preset number that are screened by the cluster system may form a target instance list, in which the cluster system processes each target instance in a circulating manner, so as to complete the pressure testing process for each target instance.

Optionally, the screening method in the random selection policy may include, but is not limited to, a random sampling method, a hierarchical sampling method, an entire group sampling method, and the like, and the specific selection method of the random selection policy is not limited in the embodiments of the present disclosure.

Optionally, the pre-stored instance filtering policy in the cluster system may be configured as other types of filtering policies besides a random selection policy. For example, a targeted instance selection policy is set according to the performance differences between instances. Accordingly, embodiments of the present disclosure are not limited to specific policy content of the instance screening policy.

In step S404, based on the target data set corresponding to the sub-service and the preset pressure test step, the pressure test is performed on the target instances, so as to obtain the single-instance service capacity of each target instance for the sub-service.

Wherein the pressure test step size characterizes a variation of a data transmission amount of the target data transmitted to the target instance each time.

In implementation, for each target instance, the cluster system sends target data to the target instance according to a data sending amount each time, the initial data sending amount may be equal to or greater than a preset pressure test step length, the target data is sent by the initial data sending amount to perform pressure test on service capacity of the target instance, and in the test process, if service fusing does not occur in the target instance, the cluster system updates the initial data sending amount according to the pressure test step length, until service fusing occurs in the target instance, and the size of the initial data sending amount is continuously increased. When the cluster system detects that the target instance is subjected to service fusing, judging whether the current service fusing of the target instance meets the preset service fusing condition, and if the target instance meets the current service fusing condition, using a pressure test result of the current target instance under the service fusing condition as a single-instance service capacity of the target instance for sub-service by the cluster system.

In this embodiment, a preset number of target instances are determined in a candidate instance set based on a preset instance screening policy, the target instances are used as representative instances of each instance in the candidate instance set, participate in a stress test, and a single instance service capacity of the target instances is determined.

In an exemplary embodiment, in a process of performing a pressure test on each target instance, a step length callback method of a dichotomy is adopted in the embodiment of the present disclosure to determine a single instance service capacity of each target instance, specifically, as shown in fig. 5, in step S404, based on a target data set corresponding to a sub-service and a preset pressure test step length, performing a pressure test on each target instance to obtain a single instance service capacity of each target instance for the sub-service, and a specific processing procedure includes:

in step S502, target data in the target data set is transmitted to the target instance according to the preset data transmission amount.

In implementation, at the beginning of the stress test, the cluster system sends the target data in the target data set to the target instance according to a preset data sending amount (i.e., an initial data sending amount). The preset data transmission amount is generally smaller, so that service fusing of the target instance can not be directly caused.

In step S504, in the process of processing the target data by the target instance, if service fusing does not occur in the target instance, the data transmission amount is updated according to the current preset pressure measurement step, and the step of transmitting the target data in the target data set to the target instance is performed based on the updated data transmission amount.

In implementation, in the process of processing target data by a target instance, the cluster system detects service availability of the target instance and judges whether the target instance is subjected to service fusing. The service availability of the target instance may be 99% of availability, that is, the target instance satisfies 99 times of availability of the target instance when performing 100 pressure tests. If it is determined that the target instance is not service fused based on the determination of service availability, the cluster system monitors a QPS (Query Per Second) index of the target instance in the target data processing process, where the Query Per Second index may reflect a current pressure test condition of the target instance.

Then, under the condition that the target instance can normally process the target data, the cluster system can update the initial data transmission amount based on the preset pressure measurement step length to obtain the updated data transmission amount, and then continue to transmit the target data in the target data set to the target instance based on the updated data transmission amount, and execute the pressure measurement process on the target instance (i.e. repeatedly execute step S502). Specifically, the target instance receives the target data sent in the updated data sending amount, continues to process the target data, and the cluster system continues to monitor the pressure test condition of the target instance, where the processing process is similar to the process of processing the target data sent in the initial data sending amount by the target instance, and only reflects the difference on the pressure test result (the pressure test result, that is, the QPS query rate per second) of the target instance, the service fusing of the target instance does not occur in the embodiment of the disclosure, and the process of continuing the pressure test on the target instance by the cluster system is not repeated.

In step S506, if service fusing occurs in the target instance, determining a target service fusing result corresponding to the target instance when the preset service fusing condition is satisfied according to the preset pressure measurement step length, the service fusing callback policy and the current data transmission amount.

And taking the target service fusing result as a single-instance service capacity of the target instance for the sub-service.

In implementation, if service fusing occurs in the target instance, the characterization that the target data is sent according to the size of the data sending amount at the moment causes service fusing to occur in the target instance. Under the condition that the target instance is subjected to service fusing, the cluster system determines a target service fusing result corresponding to the target instance under the condition that the preset service fusing condition is met according to a preset pressure measurement step length, a service fusing callback strategy and the current data transmission quantity.

Specifically, in the case that service fusing occurs for the first time in the target instance, because the pressure test step length is larger, the interval of data transmission amount is larger, so that in the case of service fusing for the first time, the determined service fusing result may not be accurate, and therefore, the cluster system appropriately reduces the data transmission amount according to the step length callback strategy of the dichotomy, and determines the target service fusing result of the target instance under the critical condition that service fusing occurs, so as to serve as the single instance service capacity of the target instance for sub-service.

Optionally, in order to improve efficiency of network data processing, a pressure test step length set by the cluster system is generally larger, so that after an initial data transmission amount is updated as few as possible for a limited time, target data is transmitted based on the updated data transmission amount, so that service fusing can occur in a target instance, but at the same time, the pressure test step length cannot be set to be too large, and after the target data is transmitted by the data transmission amount with the larger pressure test step length, the processing process of the target instance is caused to directly fail. The pressure measurement step length can be set according to the requirement of the actual pressure test, and the embodiment of the disclosure is not limited.

In this embodiment, by performing a stress test on the target instances, a single-instance service capacity of each target instance is determined, and the single-instance service capacity result is used to evaluate the overall service capacity of the candidate instance set created by the cluster system for the sub-service, thereby improving the network data processing efficiency of the sub-service.

In an alternative embodiment, as shown in fig. 6, an example of a single instance pressure testing method is provided, specifically comprising the steps of:

step S601, responding to pressure test configuration operation triggered on each target service, and generating a test task corresponding to each target service.

Step S602, carrying out concurrent processing on each test task, wherein each test task comprises test information corresponding to each sub-service in the target service;

step S603, according to the test information of each sub-service, the target data set corresponding to the sub-service is obtained. The target data set is obtained by processing target data through a data processing platform;

step S604, screening a plurality of target instances in the candidate instance set, forming a target instance list, and obtaining each target instance in the target instance list.

Step S605, transmitting target data to the target instance based on the preset data transmission amount, performing data processing on the target data by the target instance, calling a processing platform to play back the processed target data, and detecting a pressure test index value of the target instance in the process of playing back the target data.

Step S606, judging whether service fusing occurs in the target instance, if so, executing step S607; if service fusing does not occur, updating the data transmission amount of the target data according to the preset pressure measurement step length, and executing step S605;

step S607, determining whether a preset callback number threshold is reached, and if the preset callback number threshold is reached, executing step S609. If the preset callback number threshold is not reached, step S608 is executed.

Step S608, processing the current pressure test step based on the preset dichotomy, determining the callback pressure test step, using the difference between the current data transmission amount and the callback pressure test step as the updated data transmission amount, and executing step S605 based on the updated data transmission amount.

Step S609, determining a service fusing result corresponding to the current target instance, and generating a pressure test record aiming at the target instance.

In an exemplary embodiment, as shown in fig. 7, in step S230, the service capacity of the cluster system for the sub-service is determined based on the single instance service capacity and the total number of instances in the candidate instance set, and specifically includes the following processing steps:

in step S701, the single-instance service capacities corresponding to the target instances are sorted according to the order from small to large, so as to obtain a single-instance service capacity sequence.

In implementation, after determining the single-instance service capacity of each target instance for sub-service, the cluster system performs sorting processing on the determined single-instance service capacities from small to large to obtain a single-instance service capacity sequence.

In step S702, in the single instance service capacity sequence, a target single instance service capacity satisfying a preset fraction condition is determined.

In implementation, the cluster system determines the single-instance service capacity meeting the preset quantile condition as the target single-instance service capacity based on the preset quantile condition in the single-instance service capacity sequence.

Optionally, when determining the service capacity of the target single instance, the preset quantile condition may be the quantile satisfying the first 90% or the quantile of the first 50%, and the specific quantile condition may be determined based on the actual requirement of the service capacity of the sub-service in the actual test process.

In step S703, the service capacity of the cluster system for the sub-service is determined based on the target single instance service capacity and the total number of instances in the candidate instance set.

In implementation, the target single instance service capacity may be used as an estimate of the service capacity of any one instance in the candidate instance set. The cluster system then determines a service capacity of the cluster system for the sub-service based on the total number of instances included in the candidate instance set and the target single instance service capacity.

In this embodiment, the target single instance service capacity is determined in the single instance service capacities, and the single instance service capacity result is used to evaluate the overall service capacity of the candidate instance set created by the cluster system for the sub-service, so as to improve the network data processing efficiency of the sub-service.

In an exemplary embodiment, after determining the service capacity of the cluster system for the sub-service through the stress test, the cluster system may determine whether the resource configuration of the current cluster system is reasonable or not based on the service capacity (i.e. representing the service actual demand service capacity) and the service capacity pre-configured by the cluster system, so as to ensure that the cluster system is not insufficient for the type of sub-service, and is not wasted due to the overlarge service capacity. Specifically, as shown in fig. 8, the method for resource configuration management of the cluster system further includes:

in step S802, the total number of instances in the candidate instance set corresponding to the sub-service, the required service capacity of the sub-service, and the peak value of the single-instance service capacity of the target instance in the pressure test process are obtained.

In implementation, the cluster system acquires the total number of instances in the candidate instance set corresponding to the sub-service, the required service capacity of the sub-service, and the single instance service capacity peak value of the target instance in the pressure test process. Specifically, the total number of instances in the candidate instance set characterizes the actual number of instances allocated to the sub-service by the current cluster system. The single-instance service capacity peak value is peak value data in service fusing results of each target instance determined when the cluster system performs pressure test on each target instance in the candidate instance set, and is used for representing the best performance of the target instance. The required service capacity of the sub-service may be determined by the cluster system detecting a peak value of an actual service capacity of the sub-service in a historical time period, and the duration of the historical time period may be one week, one month or one quarter.

In step S804, the planned service capacity of the cluster system for the sub-service is determined according to the required service capacity of the cluster system for the sub-service and the preset redundancy parameter value.

In implementation, after determining the required service capacity of the cluster system for the sub-service, the cluster system performs appropriate redundancy on the service capacity of the sub-service determined after the pressure test according to the redundancy parameter value, that is, calculates the product of the service capacity of the sub-service and the preset redundancy parameter value, and determines the planned service capacity of the cluster system for the sub-service. The projected service capacity characterizes the amount of resources required for the sub-service after appropriate redundancy.

In step S806, a resource allocation result of the cluster system to the sub-service is determined according to the total number of instances in the candidate instance set, the single instance service capacity peak, and the planned service capacity.

In an implementation, the cluster system determines a resource allocation result of the cluster system to the sub-service according to the total number of instances in the candidate instance set, the single instance service capacity peak, and the planned service capacity. The numerical value of the resource allocation result reflects an instance number difference value between the current allocation instance number and the actual needed instance number, the instance number difference value carries sign information, a negative sign indicates that the current service capacity is insufficient, and a positive sign indicates that the current service capacity is redundant. Thus, the resource allocation result may be used to characterize whether the service capacity for the sub-service is sufficient in the current cluster system.

In step S808, resource configuration management is performed on the candidate instance set in the clustered system based on the resource configuration result.

In the implementation, the corresponding relation between the resource allocation result and the resource management strategy is prestored in the cluster system. Specifically, the cluster system determines a target resource management policy in each corresponding relation based on a resource allocation result, and performs resource allocation management on a candidate instance set in the cluster system based on the target resource management policy.

In this embodiment, according to the total number of instances in the candidate instance set, the peak value of the single instance service capacity and the planned service capacity, the resource allocation result of the sub-service by the cluster system is determined, and based on the resource allocation result, leveling configuration management can be performed on the current resource allocation situation in the cluster system, so that not only is the risk of insufficient resource allocation avoided, but also the resource waste caused by redundancy of the number of instances is reduced.

In an exemplary embodiment, as shown in fig. 9, in step S808, according to the total number of instances in the candidate instance set, the peak value of the service capacity of a single instance, and the planned service capacity, determining a resource allocation result of the cluster system for the sub-service, a specific processing procedure includes:

In step S902, a target instance demand number is determined according to a ratio of the planned service capacity to the single instance service capacity peak.

In implementation, the cluster system determines the number of target instance demands based on the ratio of the planned service capacity to the single instance service capacity peak. The planned service capacity characterizes the required service capacity of the sub-service, and further, the target instance demand number of the sub-service can be determined based on the calculation of the ratio between the required service capacity of the sub-service and the peak value of the single instance service capacity.

In step S904, a resource allocation result of the sub-service is determined according to a difference between the total number of instances in the candidate instance set and the target instance demand number.

In implementation, the cluster system obtains a resource allocation result of the sub-service according to the difference between the total number of the instances in the candidate instance set and the required number of the target instances. The resource allocation result is a specific value of the phase difference between the number of the instances, the specific value can be positive number, negative number and 0, the redundancy of the current instance is represented when the number is positive number, the deficiency of the current instance is represented when the number is negative number, and the number of the instances is reasonable, namely the number of the instances is not too large or too small.

In this embodiment, according to the total number of instances in the candidate instance set, the peak value of the single instance service capacity and the planned service capacity, the resource configuration result of the sub-service of the cluster system is determined, where the resource configuration result is the difference between the total number of instances of the sub-service actual and the number of target instances required by the cluster system, and by using the resource configuration result, resource management can be implemented on the instances in the candidate instance set of the cluster system.

In an exemplary embodiment, for the case that the resource allocation of the sub-service is insufficient in the cluster system, in addition to adjusting the amount of resources corresponding to the sub-service through resource allocation management to improve the service capacity of the cluster system for the sub-service to be a desired value, the method may also perform early warning on the case that the resource allocation is insufficient, and after step S904, the method further includes:

if the resource allocation result of the cluster system for the sub-service represents that the resource allocation is insufficient, generating and outputting prompt information of the insufficient resource allocation.

In the implementation, if the resource allocation result of the cluster system to the sub-service is insufficient, the cluster system generates and outputs prompt information of the insufficient resource allocation.

In this embodiment, by adding the prompt information of insufficient resource allocation, the user is prompted that the current cluster system cannot meet the resource requirement of the sub-service, so as to realize early warning of the running risk of the cluster system.

It should be understood that, although the steps in the flowcharts of fig. 2-9 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least a portion of the steps of fig. 2-9 may include multiple steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the steps or stages are performed necessarily occur sequentially, but may be performed alternately or alternately with at least a portion of the steps or stages in other steps or other steps.

It should be understood that the same/similar parts of the embodiments of the method described above in this specification may be referred to each other, and each embodiment focuses on differences from other embodiments, and references to descriptions of other method embodiments are only needed.

FIG. 10 is a block diagram of a network data processing device, according to an example embodiment. Referring to fig. 10, the apparatus 1000 includes a generating unit 1002, an acquiring unit 1004, a testing unit 1006, and a determining unit 1008.

The generating unit 1002 is configured to generate a test task corresponding to the target service in response to a stress test configuration operation triggered for the target service; the test task comprises test information corresponding to each sub-service in the target service.

The first obtaining unit 1004 is configured to obtain, according to each piece of test information, a target data set and a candidate instance set corresponding to the sub-service; a set of candidate instances is created by the cluster system for providing resources for executing the sub-service.

The test unit 1006 is configured to perform a pressure test on a target instance in the candidate instance set according to the target data set, resulting in a single instance service capacity of the target instance.

The first determining unit 1008 is configured to determine a service capacity of the cluster system for the sub-service based on the single instance service capacity and the total number of instances in the set of candidate instances.

In an exemplary embodiment, the first acquisition unit 1004 includes:

an acquisition sub-unit configured to perform acquisition of target data corresponding to the sub-service;

and the processing subunit is configured to execute the process of calling the preset data processing interface to the target data corresponding to the sub-service to obtain the target data set corresponding to the sub-service.

In an exemplary embodiment, the test unit 1004 includes:

a determining subunit configured to perform a determination of a preset number of target instances in the set of candidate instances according to a preset instance screening policy;

the processing subunit is configured to execute pressure test on each target instance based on the target data set corresponding to the sub-service and the preset pressure test step length to obtain single instance service capacity of each target instance for the sub-service; the pressure test step size characterizes the amount of change in the data transmission amount of each time the target data is transmitted to the target instance.

In an exemplary embodiment, the processing subunit is specifically configured to send, according to a preset data sending amount, target data in the target data set to the target instance;

in the process of processing target data by the target instance, if service fusing does not occur in the target instance, updating the data transmission amount according to the current preset pressure measurement step length, and executing the step of transmitting the target data in the target data set to the target instance based on the updated data transmission amount;

if the target instance is subjected to service fusing, determining a target service fusing result corresponding to the target instance under the condition that the preset service fusing condition is met according to a preset pressure measurement step length, a service fusing callback strategy and the current data transmission quantity; and taking the target service fusing result as a single-instance service capacity of the target instance for the sub-service.

In an exemplary embodiment, the first determining unit 1006 includes:

a second determination subunit configured to perform determining a service capacity of the cluster system for the sub-service based on the target single instance service capacity and the total number of instances in the set of candidate instances.

In an exemplary embodiment, the apparatus 1000 further includes:

a second determining unit configured to perform determining a planned service capacity of the cluster system for the sub-service according to the required service capacity of the cluster system for the sub-service and a preset redundancy parameter value;

In an exemplary embodiment, the third determining unit is specifically configured to determine the number of target instance demands according to a ratio of the planned service capacity to the peak value of the single instance service capacity;

and determining a resource allocation result of the cluster system to the sub-service according to the difference between the total number of the instances in the candidate instance set and the required number of the target instances.

In an exemplary embodiment, the apparatus 1000 further includes:

the prompting unit is configured to generate and output prompting information of insufficient resource allocation when the resource allocation result of the cluster system represents the insufficient resource allocation.

The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.

Fig. 11 is a block diagram illustrating an electronic device 1100 for network data processing according to an example embodiment. For example, the electronic device 1100 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.

Referring to fig. 11, an electronic device 1100 may include one or more of the following components: a processing component 1102, a memory 1104, a power component 1106, a multimedia component 1108, an audio component 1111, an input/output (I/O) interface 1112, a sensor component 1114, and a communication component 1116.

The processing component 1102 generally controls overall operation of the electronic device 1100, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 1102 can include one or more modules that facilitate interactions between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate interaction between the multimedia component 1108 and the processing component 1102.

The memory 1104 is configured to store various types of data to support operations at the electronic device 1100. Examples of such data include instructions for any application or method operating on the electronic device 1100, contact data, phonebook data, messages, pictures, video, and so forth. The memory 1104 may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

The power supply component 1106 provides power to the various components of the electronic device 1100. The power supply component 1106 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 1100.

The multimedia component 1108 includes a screen between the electronic device 1100 and a user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, multimedia component 1108 includes a front camera and/or a rear camera. When the electronic device 1100 is in an operational mode, such as a shooting mode or a video mode, the front-facing camera and/or the rear-facing camera may receive external multimedia data. Each front and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 1111 is configured to output and/or input audio signals. For example, audio component 1111 includes a Microphone (MIC) configured to receive external audio signals when electronic device 1100 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 1104 or transmitted via the communication component 1116. In some embodiments, audio component 1111 also includes a speaker for outputting audio signals.

The I/O interface 1112 provides an interface between the processing component 1102 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.

The sensor assembly 1114 includes one or more sensors for providing status assessment of various aspects of the electronic device 1100. For example, the sensor assembly 1114 may detect an on/off state of the electronic device 1100, a relative positioning of the components, such as a display and keypad of the electronic device 1100, the sensor assembly 1114 may also detect a change in position of the electronic device 1100 or a component of the electronic device 1100, the presence or absence of a user's contact with the electronic device 1100, an orientation or acceleration/deceleration of the device 1100, and a change in temperature of the electronic device 1100. The sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1114 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 1116 is configured to facilitate communication between the electronic device 1100 and other devices, either wired or wireless. The electronic device 1100 may access a wireless network based on a communication standard, such as WiFi, an operator network (e.g., 2G, 3G, 4G, or 5G), or a combination thereof. In one exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 1116 further includes a Near Field Communication (NFC) module to facilitate short range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 1100 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.

In an exemplary embodiment, a computer-readable storage medium is also provided, such as memory 1104 including instructions executable by processor 1120 of electronic device 1100 to perform the above-described method. For example, the computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

In an exemplary embodiment, a computer program product is also provided, comprising instructions executable by the processor 1120 of the electronic device 1100 to perform the above-described method.

It should be noted that the descriptions of the foregoing apparatus, the electronic device, the computer readable storage medium, the computer program product, and the like according to the method embodiments may further include other implementations, and the specific implementation may refer to the descriptions of the related method embodiments and are not described herein in detail.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of network data processing, the method comprising:

2. The network data processing method according to claim 1, wherein the obtaining the target data set corresponding to the sub-service includes:

acquiring target data corresponding to the sub-service;

3. The network data processing method according to claim 1, wherein the performing a stress test on a target instance in the candidate instance set according to the target data set to obtain a single instance service capacity of the target instance includes:

4. The network data processing method according to claim 3, wherein the performing the pressure test on each target instance based on the target data set corresponding to the sub-service and the preset pressure test step length to obtain a single instance service capacity of each target instance for the sub-service includes:

5. The network data processing method of claim 1, wherein the determining the service capacity of the clustered system for the sub-service based on the single instance service capacity and the total number of instances in the set of candidate instances comprises:

6. The network data processing method of claim 1, wherein the method further comprises:

7. The network data processing method of claim 6, wherein the determining the resource allocation result of the cluster system to the sub-service according to the total number of instances in the candidate instance set, the single instance service capacity peak, and the planned service capacity comprises:

8. The network data processing method according to claim 6 or 7, wherein after determining the resource allocation result of the cluster system to the sub-service, the method further comprises:

9. A network data processing apparatus, the apparatus comprising:

10. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the network data processing method of any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that instructions in the computer readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the network data processing method of any one of claims 1 to 8.