CN111611305A

CN111611305A - Data processing method, device and medium

Info

Publication number: CN111611305A
Application number: CN202010393279.XA
Authority: CN
Inventors: 李晓伟; 李明宽; 杨吉平
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-05-11
Filing date: 2020-05-11
Publication date: 2020-09-01

Abstract

The invention discloses a data processing method, a data processing device and a data processing medium. The method comprises the following steps: acquiring a plurality of service use data sets sent by a plurality of clients at a target reporting time point, wherein each service use data set represents the calling information of the corresponding client to the service object in a target time period; determining a data writing rule based on identification information carried by the plurality of service usage data sets; writing a plurality of service usage data sets into a message queue according to a data writing rule; reading a plurality of service use data sets from the message queue according to a preset reading rule; clustering a plurality of service use data sets by taking the service object identification of the service object as a central point to obtain a clustering result; and generating charging data based on the clustering result. The invention regularly acquires the calling information of the service object of each client in the target time period, introduces the message queue to store a plurality of service use data sets in order, and can improve the efficiency and the accuracy of generating the charging data.

Description

Data processing method, device and medium

Technical Field

The present invention relates to the field of internet communications technologies, and in particular, to a data processing method, apparatus, and medium.

Background

Cloud services are an augmentation, usage, and interaction model for internet-based related services, typically involving the provision of dynamically scalable and often virtualized resources over the internet. With the development of enterprise cloud services, the demand of enterprises for using cloud services is continuously rising at present, and the types of cloud services used by enterprises are more and more.

In the related art, the cloud service provider charges the enterprise user for using the cloud service often depends on the log file. Based on the use of the cloud service by the enterprise user, a business module at the cloud service provider side can generate a large amount of log data, an acquisition module downloads the log data at regular time to obtain a log file, and a statistical module determines the service use condition of the enterprise user according to the log file and generates a bill by combining with a charging rule. In the charging mode depending on the log file, the log data generated by one service module often points to the corresponding cloud service. When an enterprise user uses at least two cloud services simultaneously, the corresponding log data may be scattered corresponding to different business modules. Accordingly, cloud service providers are inefficient at determining service usage by enterprise users. Therefore, there is a need to provide a more efficient determination of service usage.

Disclosure of Invention

In order to solve the problems of low efficiency and the like when the prior art is applied to determining the service use condition of an enterprise user, the invention provides a data processing method, a device and a medium, wherein the data processing method comprises the following steps:

in one aspect, the present invention provides a data processing method, including:

acquiring a plurality of service use data sets sent by a plurality of clients at a target reporting time point, wherein each service use data set represents the calling information of the corresponding client to a service object in a target time period;

determining a data writing rule based on the identification information carried by the plurality of service usage data sets;

writing the plurality of service usage data sets into a message queue according to the data writing rule;

reading the plurality of service usage data sets from the message queue according to a preset reading rule;

clustering the service use data sets by taking the service object identification of the service object as a central point to obtain a clustering result;

and generating charging data based on the clustering result.

Another aspect provides a data processing apparatus, the apparatus comprising:

an acquisition module: the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring a plurality of service use data sets sent by a plurality of clients at a target reporting time point, and each service use data set represents the calling information of the corresponding client to a service object in a target time period;

a determination module: determining a data writing rule based on the identification information carried by the service usage data set;

a writing module: for writing the plurality of service usage data sets into a message queue according to the data write rule;

a reading module: the server is used for reading the plurality of service usage data sets from the message queue according to preset reading rules;

a clustering module: the service object identification is used for clustering the service use data sets by taking the service object identification of the service object as a central point to obtain a clustering result;

a charging module: for generating charging data based on the clustering result.

Another aspect provides an electronic device, which includes a processor and a memory, where at least one instruction or at least one program is stored in the memory, and the at least one instruction or the at least one program is loaded by the processor and executed to implement the data processing method as described above.

Another aspect provides a computer readable storage medium, in which at least one instruction or at least one program is stored, the at least one instruction or the at least one program is loaded by a processor and executed to implement the data processing method as described above.

The data processing method, the device and the medium provided by the invention have the following technical effects:

the invention obtains a plurality of service use data sets sent by a plurality of clients at the target reporting time point, and regularly obtains the calling information of the service object of each client in the target time period. The message queue is introduced to store the service use data sets in order and ensure that a reliable and stable data source is provided for clustering the service use data sets. The invention can improve the efficiency and the accuracy of generating the charging data.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a schematic diagram of an application environment provided by an embodiment of the invention;

FIG. 2 is a flow chart of a data processing method according to an embodiment of the present invention;

fig. 3 is a schematic flowchart of determining a target reporting time point and a target time period according to an embodiment of the present invention;

fig. 4 is a schematic flowchart of determining a data writing rule based on identification information carried by multiple service usage data sets according to an embodiment of the present invention;

FIG. 5 is a flowchart illustrating a process of reading a plurality of service usage data sets from a message queue according to a preset reading rule according to an embodiment of the present invention;

fig. 6 is a flowchart illustrating a data processing method according to an embodiment of the present invention;

fig. 7 is a schematic flow chart of reading a plurality of service usage data sets from a message queue according to a preset reading rule according to an embodiment of the present invention;

fig. 8 is a schematic flowchart of a process of circularly detecting whether there is a fragment to be processed according to an embodiment of the present invention;

FIG. 9 is a flow chart illustrating clustering of multiple service usage data sets according to an embodiment of the present invention;

fig. 10 is a schematic flow chart of settlement based on charging data according to an embodiment of the present invention;

FIG. 11 is a block diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.

It should be noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of the present invention and the above-described drawings, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Referring to fig. 1, fig. 1 is a schematic diagram of an application environment according to an embodiment of the present invention, which may include a client 01 and a server 02, where the client and the server may be directly or indirectly connected through wired or wireless communication. The plurality of clients send a plurality of service usage data sets to the server, each service usage data set characterizing invocation information of the service object by a corresponding client over a period of time. The server performs corresponding storage, clustering, charging and other processing on the received multiple service use data sets to obtain corresponding charging data. It should be noted that fig. 1 is only an example.

Specifically, the client 01 may include a smart phone, a desktop computer, a tablet computer, a notebook computer, an Augmented Reality (AR)/Virtual Reality (VR) device, a digital assistant, a smart speaker, a smart wearable device, and other types of physical devices, and may also include software running in the physical devices, such as a computer program. The operating system running on the client 01 may include, but is not limited to, an Android system (Android system), an IOS system (which is a mobile operating system developed by apple inc.), linux (an operating system), Microsoft Windows (Microsoft Windows operating system), and the like.

Specifically, the server 02 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. The server 02 may comprise a network communication unit, a processor and a memory, etc. The server 02 may provide background services for the clients.

In practical application, an enterprise user purchases cloud service products from a cloud service provider, the cloud service provider provides cloud services of corresponding types to the enterprise user, a server can correspond to a background of the cloud service provider, a client can correspond to a component reporting a service use data set to the background, and the enterprise user can carry the component in a product developed by the enterprise user. Correspondingly, the personal user uses the product provided by the enterprise user, the product calls the cloud service to provide the related service experience for the personal user in use, and the component carried in the product reports the service use data set to the background. The above-mentioned component may be a SaaS (Software as a Service) component, a PaaS (Platform as a Service) component, or the like. Further, the cloud service provider may provide a product integrated with a plurality of cloud services for enterprise users.

The Cloud service provided by the Cloud service provider to the enterprise user may include a Private Cloud (Private Cloud) service, a Hybrid Cloud (Hybrid Cloud) service, a Cloud conference service, and the like. The hybrid Cloud integrates Public Cloud (Public Cloud) and private Cloud, and is the main mode and development direction of Cloud computing in recent years. The private cloud is mainly oriented to enterprise users, for safety, enterprises prefer to store data in the private cloud, but meanwhile hope to obtain computing resources of the public cloud, in the situation, the hybrid cloud is adopted more and more, the public cloud and the private cloud are mixed and matched to obtain the best effect, and the personalized solution achieves the purposes of saving money and being safe. The cloud conference is an efficient, convenient and low-cost conference form based on a cloud computing technology. A user can share voice, data files and videos with teams and clients all over the world quickly and efficiently only by performing simple and easy-to-use operation through an internet interface, and complex technologies such as transmission and processing of data in a conference are assisted by a cloud conference service provider to operate.

While specific embodiments of a data processing method according to the present invention are described below, fig. 2 is a flow chart of a data processing method according to an embodiment of the present invention, and the present specification provides the method operation steps as described in the embodiments or the flow chart, but may include more or less operation steps based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. In practice, the system or server product may be implemented in a sequential or parallel manner (e.g., parallel processor or multi-threaded environment) according to the embodiments or methods shown in the figures. Specifically, as shown in fig. 2, the method may include:

s201: acquiring a plurality of service use data sets sent by a plurality of clients at a target reporting time point, wherein each service use data set represents the calling information of the corresponding client to a service object in a target time period;

in the embodiment of the invention, the server acquires a plurality of service use data sets sent by a plurality of clients at the target reporting time point. The server may correspond to a background on the cloud service provider side, and the client may correspond to a software component that reports a service usage data set to the server, may also correspond to software developed by an enterprise user that includes the software component, and may also be an entity device that runs the software. Based on a cloud service product purchased from a cloud service provider by an enterprise user (a specific type of cloud service related to the cloud service product corresponds to a service object), a server and a client point to the same cloud service provider, and each service uses a data set to represent calling information of the corresponding client to the service object (corresponding cloud service) in a target time period. The cloud service provider can provide a product integrated with various cloud services for enterprise users, when corresponding clients are software developed for the enterprise users, the software is used by individual users, the software calls cloud services (corresponding service objects) based on software components to provide related service experiences for the individual users, and service use data sets point to the called cloud services.

Based on the dimensions of the software components, a client sends a service usage data set that may indicate its invocation information for at least one service object (corresponding cloud service) of a corresponding cloud service product. Based on the dimensionality of the software or physical device, one client may send at least one service usage data set, each of which may indicate its invocation information for at least one service object (corresponding cloud service) of a corresponding cloud service product. At least one service object of the cloud service product corresponds to cloud services purchased by the enterprise user development software from the cloud service provider. The calling information may include a usage duration and a usage flow corresponding to the usage service object.

The target reporting time point indicates a time point when the server acquires a plurality of service use data sets, and the target time period determines that the calling information of the client to the service object in the time range is used as the service use data set.

In a specific embodiment, as shown in fig. 3, the method further includes determining the target reporting time point and the target time period:

s301: setting a heartbeat reporting interface and a heartbeat interval;

s302: acquiring a previous reporting time point;

s303: determining the target time period and the target reporting time point based on the previous reporting time point and the heartbeat interval;

the server provides a heartbeat reporting interface, the heartbeat reporting interface can support a plurality of cloud services and a plurality of modules to report at one time and can be distinguished, and the heartbeat reporting interface can support the plurality of clients to report simultaneously; the heartbeat reporting interface can support the cloud service to self-define and report the use data; the heartbeat reporting interface can support the server to forward (transmit) the reported use data of the cloud service A to the cloud service B for the use of the relevant business logic. The server sets a heartbeat interval, which may be a fixed value or a variable value. It should be noted that the heartbeat interval may be set by the cloud service provider side for a certain cloud service product purchased by the enterprise user, and further may be set by the cloud service provider side for a certain cloud service product purchased by a specific enterprise user.

Therefore, the previous reporting time point is obtained, and the target time period and the target reporting time point can be determined by combining the heartbeat interval. The target time period is consistent with the duration pointed by the heartbeat interval, and the starting point of the target time period is the last reporting time point. The target reporting time point may be regarded as a later reporting time point of the previous reporting time point.

Correspondingly, the server acquires the heartbeat messages respectively reported by the plurality of clients at the target reporting time point by using the heartbeat reporting interface, and takes the heartbeat messages respectively reported by the plurality of clients as the plurality of service use data sets. The server can regard all messages acquired at the target reporting time point as one message data, and the message data effectively records various cloud service use data. In practical applications, even if the cloud services are integrated in the same cloud service product, the use occasions of the various cloud services are different. Based on the related setting of the heartbeat reporting interface, the mixed heartbeat reporting of various cloud services can be realized, and the server can conveniently perform unified management on cloud service use data of related cloud service products of enterprise users. When software developed by an enterprise user is provided with a component pointing to a product integrated with various cloud services and different enterprise users correspondingly have different charging rules, the server can conveniently perform unified and flexible charging processing on the enterprise user on related cloud service products (for example, unified issued bills can include independent account list items of various cloud services integrated with cloud service products), and accurate bills are provided for the enterprise user.

S202: determining a data writing rule based on the identification information carried by the plurality of service usage data sets;

in the embodiment of the present invention, the identification information carried by each service usage data set may include a corresponding service object identification set, a service object identification, and a user identification. The service object identifier (which may be indicated by businesstype) corresponds to a service object, that is, corresponds to a specific type of cloud service to which a cloud service product relates. The service object identifier (which may be denoted by sdkappa) may correspond to an enterprise user, and specifically may correspond to a cloud service product purchased by the enterprise user, where the cloud service product may be integrated with multiple cloud services. The user identifier (which may be indicated by userid) corresponds to a personal user, and specifically may be an account of a registered user or a client identifier corresponding to a guest user. Wherein the registered users and guest users are users of the software developed by the enterprise users.

In a specific embodiment, as shown in fig. 4, the determining a data writing rule based on the identification information carried by the plurality of service usage data sets includes:

s401: analyzing each service use data set respectively to obtain a service object identification set, a service object identification and a user identification corresponding to each service use data set;

based on the dimensions of the software components, a client sends a set of service usage data. Based on the software or the dimension of the physical device, one client may send at least one service usage data set, that is, may support multiple service mix reports. The service usage data set a may characterize invocation information of cloud services in the cloud service product a (purchased by the enterprise user a developing software a to the cloud service provider) when the individual user a uses the software a developed by the enterprise user a within a target time period. After the analysis, a service object identification set (corresponding to the cloud service in the cloud service product a), a business object identification (corresponding to the enterprise user a, specifically, the cloud service product a purchased by the enterprise user a), and a user identification (corresponding to the personal user a) corresponding to the service usage data set a can be obtained.

S402: extracting service object data corresponding to each service object identification from the service usage data set based on the service object identification set corresponding to the service usage data set;

the cloud services integrated by the cloud service product a may include a cloud service a1, a cloud service a2, and a cloud service a 3. Accordingly, data corresponding to the service object identification of cloud service a1, data corresponding to the service object identification of cloud service a2, and data corresponding to the service object identification of cloud service a3 may be extracted from the service usage data set a as service object data a1, service object data a2, and service object data a3, based on the dimension of a specific cloud service.

S403: acquiring a service object registry corresponding to the message queue;

the service object registry is preset based on the message queue and records the service objects corresponding to the data allowed to be stored in the message queue. Of course, the service object registry can be flexibly updated. For example, the service objects described in the current service object registry include: a service object corresponding to the cloud service a1 and a service object corresponding to the cloud service a 2.

S404: comparing the service object identification corresponding to the service object data with the service object registry;

comparing the service object identifier corresponding to the service object data a1 with the service object registry, comparing the service object identifier corresponding to the service object data a2 with the service object registry, and comparing the service object identifier corresponding to the service object data a3 with the service object registry.

S405: when the service object registry is loaded with the service object identification corresponding to the service object data, taking the service object data as candidate data, and determining a subscription theme in the message queue for the candidate data based on the service object identification corresponding to the service object data;

based on the current service object registry in which the service object corresponding to the cloud service a1 and the service object corresponding to the cloud service a2 are stored, the service object data a1 and the service object data a2 are set as candidate data, and the service object data a3 is not set as candidate data. Then, a subscription topic in the message queue is determined for the service object identification based on the service object data a1, and a subscription topic in the message queue is determined for the service object identification based on the service object data a 2. For the service object data a3, an entry of the service object data a3 may be added in the service object registry, and then the subscription topic in the message queue is determined based on the service object identification of the service object data a 3. Otherwise, an error log is printed and the data is discarded. Data isolation of data written to the message queue may be performed based on the subscription topic.

S406: determining a writing area of the candidate data in the message queue based on the business object identification and the user identification corresponding to the candidate data;

the writing area can correspond to different partitions configured for each message queue, data written into the message queue needs to be stored in the corresponding partitions, and the data in each partition is ordered, so that data sources for clustering and charging processing of subsequent read data can be guaranteed to be ordered, and errors in clustering and charging processing are avoided. The writing area of the candidate data in the message queue is determined based on the service object identifier and the user identifier corresponding to the candidate data, so as to ensure that the candidate data is stored according to the dimension of the individual user group, that is, the candidate data corresponding to one individual user group is stored in each writing area, wherein each individual user group corresponds to a plurality of user identifiers.

For determining the writing area, the following steps may be performed: firstly, generating a key value of the candidate data based on the business object identification and the user identification corresponding to the candidate data. And then, acquiring a target hash function corresponding to the message queue. Further, the write zone is determined based on the key value of the candidate data and the target hash function. The message queue may correspond to a hash table (including a target hash function) recording the storage location, where a modulo operation is performed on a service object identifier (e.g., sdkappa) and a user identifier (e.g., userid) serving as a hash partition key to obtain a key value (key value), and the key value is substituted into the target hash function to obtain the write area.

S407: and obtaining the data writing rule based on the subscription theme and the writing area corresponding to each candidate data.

Correspondingly, a data writing rule is obtained based on the subscription theme and the writing area corresponding to each candidate data determined in the above steps.

S203: writing the plurality of service usage data sets into a message queue according to the data writing rule;

in the embodiment of the present invention, the message queue may correspond to a high-throughput distributed message system adopting a publish-subscribe mode, such as Kafka. The message queue can also correspond to high-performance, high-availability, high-concurrency-supporting message middleware which is expanded and developed and optimized based on Kafka.

Based on the determined data writing rules, the server may utilize the relevant components of the message queue to directly process the plurality of service usage data sets. Multiple service usage data sets can be accurately and seamlessly recorded in the message log of the message queue. In combination with the related content in the above steps S401 to S407, a message queue instance may be created, and the number of partitions may be configured for the message queue instance. And starting a related component of the message queue, and sending a plurality of service use data sets to the ordered storage in the message queue instance by an asynchronous publisher (producer) in the related component according to the data writing rule. For the case of a transmission failure, a transmission failure error message may be printed. By utilizing the advantages of decoupling of receiving and transmitting, peak clipping and valley filling, horizontal expansion support and the like of the message queue, a plurality of service use data sets can be stored in order, the probability of data loss can be reduced, and a reliable and stable data source can be provided for clustering and charging the plurality of service use data sets. When there is an anomaly in the billing data, the accuracy of the data from the message queue may support data validation. Accordingly, the introduction of the message queue may provide underlying support for the charging process.

S204: reading the plurality of service usage data sets from the message queue according to a preset reading rule;

in the embodiment of the present invention, the preset reading rule is used to define a time point when data reading starts from the message queue, whether to introduce a slicing idea to sort the read data, and a time interval when two adjacent slices are created when the slicing idea is introduced. The preset reading rule can be generated according to the following steps: 1) determining a starting time point of data writing to the message queue; 2) setting a target time interval for aggregating data extracted from the message queue; 3) generating the preset reading rule based on the starting time point and the target time interval.

As shown in fig. 5 and 7, the reading the plurality of service usage data sets from the message queue according to a preset reading rule includes:

s501: sequentially extracting data from the message queue based on the starting time point, and aggregating the data extracted from the message queue at intervals of the target time interval to obtain corresponding data fragments;

s502: acquiring a clustering triggering condition, and triggering the corresponding data fragments to be clustered according to the clustering triggering condition;

s503: repeating the steps from the data extraction from the message queue to the step of triggering the clustering processing of the corresponding data fragments, until the extraction of the plurality of service use data sets from the message queue is completed.

The method can be regarded as slicing and sorting a large amount of data in the message queue, and can improve the efficiency of subsequent clustering processing. And based on the starting time point, taking the data extracted from the message queue corresponding to each target time interval as a fragment, wherein the aggregated fragment can trigger the subsequent clustering step in real time or wait for the triggering time to trigger the subsequent clustering step. The aggregated shards may be cached in a Redis (a key value pair database) and stored in a List (List).

Specifically, a slice (at this time, there is no data extracted from the message queue) may be created based on the starting time point, then data is extracted from the message queue one by one based on the first-in-first-out principle, and the data extracted from the message queue corresponding to the target time interval is stuffed into the slice. For example, a slice is created at the starting time point, the target time interval is 5 minutes, if the time interval between the extraction time corresponding to the extracted data and the starting time point is greater than 5 minutes, the data is stuffed into the newly created slice, otherwise, the data is stuffed into the slice created at the starting time point.

As shown in fig. 8, the clustering trigger condition may determine the timing of clustering the slices. It can circularly detect whether there is fragment to be processed currently, when there are more than two fragments in the current fragment pool (corresponding to the position of the fragment whose cache has been aggregated, such as the above-mentioned Redis), it is considered that the earlier aggregated fragment needs to be processed, otherwise, the detection is continued. The loop traversal may be performed by using a sleep function (the sleep function may put a computer program (process, task or thread) into sleep and make the computer program inactive for a certain period of time.

In addition, when data is extracted from the message queue, the data needs to be subjected to validity check before being stuffed into the fragments, for example, if the age characteristic of a registered user extracted from the data corresponds to 10000 years old, an error log can be printed, and the data can be discarded. Correspondingly, the data meeting the requirement of the validity check is stuffed into the fragments.

S205: clustering the service use data sets by taking the service object identification of the service object as a central point to obtain a clustering result;

in the embodiment of the present invention, based on a plurality of service usage data sets read from the message queue, a service object identifier set corresponding to each service usage data set may be obtained, and the service object identifiers are used as central points to perform clustering processing on the service usage data sets, so as to obtain a clustering result. In the clustering result, all service object data corresponding to the same service object identifier (for example, corresponding to cloud service a1, and specifically, corresponding to cloud service a1 in cloud service product a) form a set, and the user identifier corresponding to the data in the set may include a user identifier corresponding to individual user a, a user identifier corresponding to individual user B, a user identifier corresponding to individual user C, and the like.

In a specific embodiment, a plurality of service usage data sets may be clustered by using a business object identifier (corresponding to an enterprise user, and specifically corresponding to a cloud service product purchased by the enterprise user) as a central point, so as to obtain a plurality of large groups of enterprise user dimensions and even cloud service product dimensions purchased by the enterprise user. And then clustering data in each large group by taking the service object identification (corresponding to the cloud service in the cloud service product purchased by the enterprise user) as a central point to obtain a plurality of middle groups of cloud service dimensionalities. And clustering the data in each group by taking the user identification (corresponding to the individual user) as a central point to obtain a plurality of groups of individual user dimensionality.

And with reference to the related content in the foregoing steps S501 to S503, traversing service usage data in a processing task that needs to be executed in a clustering manner, or traversing service usage data in a segment that needs to be subjected to clustering, grouping the data according to enterprise user dimensions, cloud service product dimensions, cloud service dimensions (including sub-cloud service dimensions), and user dimensions, and aggregating and sorting out usage data of each cloud service of an individual user under each enterprise user based on a task processing trigger condition. Therefore, the use conditions of all users of each cloud service are separately recorded in a mode of carrying out fragment aggregation and arrangement on a large amount of data. The method can rapidly and efficiently process a plurality of service use data sets by combining with the introduced message queue, accurately record the service use data of each user, provide reliable guarantee for the subsequent generation of charging data and provide convenience for the detailed query of charging. It should be noted that, a plurality of service usage data sets may be taken as a whole, and the whole may be copied based on different dimensions, and then separately and accurately counted and stored. This ensemble can also be individually and accurately counted and saved based on the hierarchy setting of different dimensions (see fig. 9). The two statistical modes can coexist, and the statistical mode can be flexibly set.

S206: and generating charging data based on the clustering result.

In the embodiment of the invention, the clustering result is counted based on the preset charging rule to generate the charging data. Different enterprise users correspond to different charging rules, the preset charging rules point to the target business object, and the preset charging rules correspond to specific enterprise users and further correspond to specific cloud service products purchased by the specific enterprise users.

In a specific embodiment, before generating charging data based on the clustering result, corresponding charging information may be configured for a plurality of service objects corresponding to the target service object, respectively, where the corresponding charging information includes at least one of: charging units (such as duration calculation units and flow units), charging unit price, charging mode (such as prepayment) and charging period. Different cloud services integrated in a cloud service product may correspond to different billing information. The preset charging rule may be determined based on the corresponding charging information.

Correspondingly, the generating of the charging data based on the clustering result includes: 1) acquiring the corresponding charging information; 2) and obtaining first statistical data of the service object dimension corresponding to the target service object based on the clustering result, and obtaining first charging data of the service object dimension corresponding to the target service object based on the corresponding charging information and the first statistical data. The first statistical data may represent usage of each cloud service in a certain cloud service product by all individual users, for example, the first statistical data may include service usage data of cloud service a1 by all individual users, service usage data of cloud service a2 by all individual users, and the like. The first billing data may represent usage consumption information of all individual users for each cloud service in a certain cloud service product, for example, the first billing data may include consumption amount data of all individual users for the cloud service a1, consumption amount data of all individual users for the cloud service a2, and the like. 3) And obtaining second statistical data corresponding to the user dimension of the target service object based on the clustering result, and obtaining second charging data corresponding to the user dimension of the target service object based on the corresponding charging information and the second statistical data. The second statistical data may represent usage of all cloud services (such as cloud service a1+ cloud service a2+ cloud service a3) in a certain cloud service product by each individual user, for example, the second statistical data may include service usage data of the individual user a on cloud service a1+ cloud service a2+ cloud service a3, service usage data of the individual user B on cloud service a1+ cloud service a2+ cloud service a3, and the like. The second billing data may represent usage consumption information of each individual user for all cloud services in a certain cloud service product, for example, the second billing data may include consumption amount data of the individual user a for the cloud service a1+ the cloud service a2+ the cloud service a3, consumption amount data of the individual user B for the cloud service a1+ the cloud service a2+ the cloud service a3, and the like.

In another specific embodiment, as shown in fig. 10, after generating the billing data based on the billing rule and the clustering result, a first unsettled value may be determined based on the billing data, and the first unsettled value may indicate a value to be settled (such as a monetary amount, a duration, etc.) resulting from usage of the corresponding cloud service product by a specific enterprise user. And then, acquiring a target numerical value corresponding to the target business object, wherein the target numerical value can correspond to a numerical value (such as currency amount, duration and the like) which is purchased in advance by a specific enterprise user from a cloud service provider and is not used, and the target numerical value can be specific to a specific cloud service product. And further 1) updating the target value based on the first unsettled value when the target value is equal to or greater than the first unsettled value. That is, when the pre-stored value is greater than or equal to the value to be settled, the pre-stored value can sufficiently withhold the value to be settled generated by the specific enterprise user using the corresponding cloud service product. And updating the target value based on the first unsettled value, and taking the value obtained by subtracting the first unsettled value from the current target value as the next target value. 2) When the target value is smaller than the first unsettled value, determining a second unsettled value based on the first unsettled value and the target value, and generating an instruction for stopping calling the service object corresponding to the target service object based on the second unsettled value. That is, when the pre-stored value is smaller than the value to be settled, the pre-stored value is not sufficient to withhold the value to be settled generated by a specific enterprise user using the corresponding cloud service product. The second outstanding value represents the value to be settled that has not been deducted after the target value is used up. At this time, an instruction for stopping calling the service object corresponding to the target business object may be generated, and service use of the corresponding cloud service product provided for the specific enterprise user may be suspended.

In practical applications, the "slices" described in steps S204 and S205 may be combined, and first, the service usage data of a single individual user under each slice is read, and the service usage of the single individual user is counted. Then, all the individual users corresponding to the same enterprise user are traversed, and the total service usage of an enterprise in the statistical time period is counted. And moreover, obtaining the prepaid meal package purchased by the enterprise user, carrying out usage settlement, and deducting the allowance of the meal package. Wherein, the service stopping treatment is carried out aiming at enterprise users with insufficient allowance. Reliable guarantee is provided for the subsequent generation of charging data and convenience is provided for the charging detail query. Therefore, the accuracy and timeliness of the charging settlement can be improved, and support is provided for subsequent multi-dimensional charging detail query (for example, the usage of a single individual user is provided, and the usage of all individual users corresponding to the same enterprise user is summarized and stored in a database). Further, the prepaid package purchased by the enterprise user can flexibly allocate the prepaid usage of the cloud service corresponding to the prepaid package based on the previous usage situation and the actual demand.

As can be seen from the technical solutions provided in the embodiments of the present specification, a plurality of service usage data sets sent by a plurality of clients are obtained at a target reporting time point, and call information of a service object in a target time period is obtained by each client in a timed and unified manner. The message queue is introduced to store the service use data sets in order and ensure that a reliable and stable data source is provided for clustering the service use data sets. The invention can improve the efficiency and the accuracy of generating the charging data and improve the commercialization capacity of providing product services by the cloud service provider.

An embodiment of the present invention further provides a data processing apparatus, as shown in fig. 11, the apparatus includes:

the obtaining module 1110: the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring a plurality of service use data sets sent by a plurality of clients at a target reporting time point, and each service use data set represents the calling information of the corresponding client to a service object in a target time period;

the determination module 1120: determining a data writing rule based on the identification information carried by the service usage data set;

the write module 1130: for writing the plurality of service usage data sets into a message queue according to the data write rule;

the reading module 1140: the server is used for reading the plurality of service usage data sets from the message queue according to preset reading rules;

the clustering module 1150: the service object identification is used for clustering the service use data sets by taking the service object identification of the service object as a central point to obtain a clustering result;

a charging module 1160: for generating charging data based on the clustering result.

Specifically, as shown in fig. 6, the obtaining module, the determining module, and the writing module may correspond to the heartbeat reporting module shown in fig. 6, the reading module may correspond to the usage collecting module shown in fig. 6, and the clustering module and the charging module may correspond to the metering and charging module shown in fig. 6. The clustering module reports the data classification to the charging module, and the charging module persistently stores each piece of consumption data for reconciliation.

It should be noted that the device and method embodiments in the device embodiment are based on the same inventive concept.

An embodiment of the present invention provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided in the foregoing method embodiment.

Further, fig. 12 is a schematic diagram of a hardware structure of an electronic device for implementing the data processing method provided by the embodiment of the present invention, and the electronic device may participate in forming or including the data processing apparatus provided by the embodiment of the present invention. As shown in fig. 12, the electronic device 120 may include one or more (shown here as 1202a, 1202b, … …, 1202 n) processors 1202 (the processors 1202 may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1204 for storing data, and a transmitting device 1206 for communication functions. Besides, the method can also comprise the following steps: a display, an input/output interface (I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a power source, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 12 is only an illustration and is not intended to limit the structure of the electronic device. For example, electronic device 120 may also include more or fewer components than shown in FIG. 12, or have a different configuration than shown in FIG. 12.

It should be noted that the one or more processors 1202 and/or other data processing circuitry described above may be referred to generally herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part in software, hardware, firmware, or any combination thereof. Further, the data processing circuit may be a single stand-alone processing module, or incorporated in whole or in part into any of the other elements in the electronic device 120 (or mobile device). As referred to in the embodiments of the application, the data processing circuit acts as a processor control (e.g. selection of a variable resistance termination path connected to the interface).

The memory 1204 can be used for storing software programs and modules of application software, such as program instructions/data storage devices corresponding to the methods described in the embodiments of the present invention, and the processor 1202 executes various functional applications and data processing by operating the software programs and modules stored in the memory 124, so as to implement one of the data processing methods described above. The memory 1204 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1204 may further include memory located remotely from the processor 1202, which may be connected to the electronic device 120 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmitting device 1206 is used for receiving or sending data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the electronic device 120. In one example, the transmitting device 1206 includes a network adapter (NIC) that can be connected to other network devices through a base station to communicate with the internet. In one embodiment, the transmitting device 1206 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the electronic device 120 (or mobile device).

Embodiments of the present invention also provide a storage medium, which may be disposed in an electronic device to store at least one instruction or at least one program for implementing a data processing method in the method embodiments, where the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided in the method embodiments.

Alternatively, in this embodiment, the storage medium may be located in at least one network server of a plurality of network servers of a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the device and electronic apparatus embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. A method of data processing, the method comprising:

and generating charging data based on the clustering result.

2. The method of claim 1, further comprising determining the target reporting time point and the target time period:

setting a heartbeat reporting interface and a heartbeat interval;

acquiring a previous reporting time point;

determining the target time period and the target reporting time point based on the previous reporting time point and the heartbeat interval;

correspondingly, the obtaining of multiple service usage data sets sent by multiple clients at a target reporting time point includes:

and acquiring heartbeat messages respectively reported by the plurality of clients at the target reporting time point by using the heartbeat reporting interface, and taking the heartbeat messages respectively reported by the plurality of clients as the plurality of service use data sets.

3. The method of claim 1, wherein the determining a data writing rule based on the identification information carried by the plurality of service usage data sets comprises:

analyzing each service use data set respectively to obtain a service object identification set, a service object identification and a user identification corresponding to each service use data set;

extracting service object data corresponding to each service object identification from the service usage data set based on the service object identification set corresponding to the service usage data set;

acquiring a service object registry corresponding to the message queue;

comparing the service object identification corresponding to the service object data with the service object registry;

when the service object registry is loaded with the service object identification corresponding to the service object data, taking the service object data as candidate data, and determining a subscription theme in the message queue for the candidate data based on the service object identification corresponding to the service object data;

determining a writing area of the candidate data in the message queue based on the business object identification and the user identification corresponding to the candidate data;

and obtaining the data writing rule based on the subscription theme and the writing area corresponding to each candidate data.

4. The method of claim 3, wherein the determining a writing area of the candidate data in the message queue based on the service object identifier and the user identifier corresponding to the candidate data comprises:

generating a key value of the candidate data based on the business object identification and the user identification corresponding to the candidate data;

acquiring a target hash function corresponding to the message queue;

determining the write zone based on the key value of the candidate data and the target hash function.

5. The method of claim 1, wherein before the reading the plurality of service usage data sets from the message queue according to a preset reading rule, the method further comprises:

determining a starting time point of data writing to the message queue;

setting a target time interval for aggregating data extracted from the message queue;

generating the preset reading rule based on the starting time point and the target time interval.

6. The method according to claim 5, wherein the reading the plurality of service usage data sets from the message queue according to a preset reading rule comprises:

sequentially extracting data from the message queue based on the starting time point, and aggregating the data extracted from the message queue at intervals of the target time interval to obtain corresponding data fragments;

acquiring a clustering triggering condition, and triggering the corresponding data fragments to be clustered according to the clustering triggering condition;

repeating the steps from the data extraction from the message queue to the step of triggering the clustering processing of the corresponding data fragments, until the extraction of the plurality of service use data sets from the message queue is completed.

7. The method of claim 1, wherein before generating charging data based on the clustering result, the method further comprises:

respectively configuring corresponding charging information for a plurality of service objects corresponding to the target service object;

correspondingly, the generating of the charging data based on the clustering result includes:

acquiring the corresponding charging information;

obtaining first statistical data of service object dimensions corresponding to the target service object based on the clustering result, and obtaining first charging data of the service object dimensions corresponding to the target service object based on the corresponding charging information and the first statistical data;

and obtaining second statistical data corresponding to the user dimension of the target service object based on the clustering result, and obtaining second charging data corresponding to the user dimension of the target service object based on the corresponding charging information and the second statistical data.

8. The method of claim 1, wherein after generating charging data based on the charging rules and the clustering results, the method further comprises:

determining a first unsettled value based on the charging data;

acquiring a target numerical value corresponding to a target service object;

updating the target value based on the first unsettled value when the target value is greater than or equal to the first unsettled value;

when the target value is smaller than the first unsettled value, determining a second unsettled value based on the first unsettled value and the target value, and generating an instruction for stopping calling the service object corresponding to the target service object based on the second unsettled value.

9. A data processing apparatus, characterized in that the apparatus comprises:

a charging module: for generating charging data based on the clustering result.

10. A computer-readable storage medium, in which at least one instruction or at least one program is stored, which is loaded and executed by a processor to implement the data processing method according to any one of claims 1 to 8.