CN116627670A - Data processing method, apparatus, device, storage medium, and program product - Google Patents
Data processing method, apparatus, device, storage medium, and program product Download PDFInfo
- Publication number
- CN116627670A CN116627670A CN202310572304.4A CN202310572304A CN116627670A CN 116627670 A CN116627670 A CN 116627670A CN 202310572304 A CN202310572304 A CN 202310572304A CN 116627670 A CN116627670 A CN 116627670A
- Authority
- CN
- China
- Prior art keywords
- computing
- computing node
- data processing
- rate
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 99
- 238000000034 method Methods 0.000 claims abstract description 31
- 230000010354 integration Effects 0.000 claims abstract description 18
- 230000005540 biological transmission Effects 0.000 claims description 54
- 238000012544 monitoring process Methods 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 20
- 238000005192 partition Methods 0.000 claims description 13
- 238000004064 recycling Methods 0.000 claims description 13
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000010586 diagram Methods 0.000 description 12
- 230000015654 memory Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 8
- 238000013468 resource allocation Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 101150096185 PAAS gene Proteins 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02W—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO WASTEWATER TREATMENT OR WASTE MANAGEMENT
- Y02W90/00—Enabling technologies or technologies with a potential or indirect contribution to greenhouse gas [GHG] emissions mitigation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a data processing method, relates to the technical field of cloud computing, and can be applied to the technical field of finance. The method comprises the following steps: obtaining recharging information in a first message queue, wherein the first message queue is subscribed and consumed by the first computing node; inquiring a cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information; calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and writing the integration processing result into a second message queue. The present disclosure also provides a data processing apparatus, device, storage medium, and program product.
Description
Technical Field
The present disclosure relates to the field of computer technology, and in particular, to the field of cloud computing technology, and more particularly, to a data processing method, apparatus, device, storage medium, and program product.
Background
In the related art, processing of business customer points generally adopts a batch processing mode, data such as user account opening, recharging actions and the like in the previous day are assembled into files in the end of the day, the files are imported into a data table, unified processing is carried out through batch programs, and the files are updated into a system for customers to consume in the next day. This approach does not meet the high-timeliness integration processing scenario, and the processing performance is poor, and the batch processing approach generates a large amount of intermediate data, requiring a large storage space, including files, data tables, and some intermediate data. The demand for storage is high.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
In view of the foregoing, the present disclosure provides a data processing method, apparatus, device, storage medium, and program product that improve the efficiency of integrated data processing.
According to a first aspect of the present disclosure, there is provided a data processing method applied to a distributed cloud computing platform, the distributed cloud computing platform including a first computing node, the method comprising:
obtaining recharging information in a first message queue, wherein the first message queue is subscribed and consumed by the first computing node;
inquiring a cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information;
calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and
and writing the integration processing result into a second message queue.
According to an embodiment of the disclosure, the distributed cloud computing platform further comprises a second computing node, the method further comprising:
Acquiring account opening information and service parameter information in a third message queue, wherein the third message queue is subscribed and consumed by the second computing node; and
and storing the account opening information and the service parameter information into a cache database in the form of key value pairs.
According to an embodiment of the disclosure, the distributed cloud computing platform further comprises a monitoring node, the method further comprising:
periodically detecting the data transmission rate and the data consumption rate of each partition in the first message queue and the third message queue; and
and dynamically distributing the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes.
According to an embodiment of the disclosure, the dynamically allocating computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate, and a data processing rate of the computing nodes includes:
recovering computing resources of the first computing node and the second computing node according to the data transmission rate and the data processing rate of the computing nodes;
And expanding the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes.
According to an embodiment of the disclosure, the reclaiming computing resources of the first computing node and the second computing node according to the data transmission rate and the data processing rate of the computing node includes:
when the data transmission rate is determined to be smaller than a first preset threshold value, determining the number of recoverable containers according to the data processing rate of the computing node and the current node number; and
and recycling the computing resources of the computing nodes according to the number of the recyclable containers.
According to an embodiment of the disclosure, the expanding the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate, and the data processing rate of the computing nodes includes:
when the data transmission rate is determined to be greater than a second preset threshold value and the data transmission rate is determined to be greater than the data processing rate of the corresponding computing node, determining the number of the expandable containers according to the data transmission rate, the data consumption rate and the data processing rate; and
And creating containers according to the expandable container number and subscribing corresponding consumption queues.
A second aspect of the present disclosure provides a data processing apparatus for application to a distributed cloud computing platform, the distributed cloud computing platform including a first computing node, the apparatus comprising:
a first obtaining module, configured to obtain refill information in a first message queue, where the first message queue is subscribed for consumption by the first computing node;
the determining module is used for inquiring the cache database according to the recharging information so as to determine account opening information and service parameter information associated with the recharging information;
the calculation module is used for calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and
and the sending module is used for writing the integration processing result into a second message queue.
According to an embodiment of the disclosure, the distributed cloud computing platform further comprises a second computing node, the apparatus further comprising: and the second acquisition module and the storage module.
The second acquisition module is used for acquiring account opening information and service parameter information in a third message queue, wherein the third message queue is subscribed and consumed by the second computing node; and
And the storage module is used for storing the account opening information and the service parameter information into a cache database in the form of key value pairs.
According to an embodiment of the disclosure, the distributed cloud computing platform further comprises a monitoring node, the apparatus further comprising: the monitoring module and the computing resource allocation module.
The monitoring module is used for periodically detecting the data transmission rate and the data consumption rate of each partition in the first message queue and the third message queue; and
and the computing resource allocation module is used for dynamically allocating the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes.
According to an embodiment of the disclosure, the computing resource allocation module includes a resource reclamation sub-module and a resource expansion sub-module.
The resource recycling sub-module is used for recycling the computing resources of the first computing node and the second computing node according to the data transmission rate and the data processing rate of the computing nodes;
and the resource expansion sub-module is used for expanding the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes.
According to an embodiment of the present disclosure, the resource reclamation sub-module includes: a first determination unit and a resource recovery unit.
A first determining unit, configured to determine, when it is determined that the data transmission rate is less than a first preset threshold, the number of recoverable containers according to the data processing rate of the computing node and the current number of nodes; and
and the resource recycling unit is used for recycling the computing resources of the computing nodes according to the number of the recyclable containers.
According to an embodiment of the present disclosure, the resource extension submodule includes: a second determination unit and a resource extension unit.
A second determining unit, configured to determine, when it is determined that the data transmission rate is greater than a second preset threshold and the data transmission rate is greater than a data processing rate of a corresponding computing node, an expandable container number according to the data transmission rate, the data consumption rate, and the data processing rate; and
and the resource expansion unit is used for creating containers according to the expandable container number and subscribing corresponding consumption queues.
A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data processing method described above.
A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described data processing method.
A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above-described data processing method.
According to the data processing method provided by the embodiment of the disclosure, a first computing node subscribes to consume a first message queue to obtain recharging information in the queue, and queries a cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information; calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and writing the integration processing result into a second message queue. Compared with the related art, the method provided by the embodiment of the disclosure has the advantages that the timeliness of integral processing is greatly improved by adopting the distributed computing nodes through the distributed message queues and the cache database, and the message queues and the cache are combined through different data structures, so that the method is clear in functional division, low in coupling and low in maintenance cost.
Drawings
The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates a system architecture diagram of a data processing apparatus according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates an application scenario diagram of a data processing method, apparatus, device, storage medium and program product according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a data processing method provided in accordance with an embodiment of the present disclosure;
FIG. 4 schematically illustrates a flow chart of a data processing method provided in accordance with another embodiment of the present disclosure;
FIG. 5 schematically illustrates one of the flowcharts of the method of dynamic allocation of computing resources provided in accordance with an embodiment of the present disclosure;
FIG. 6 schematically illustrates a second flowchart of a method for dynamic allocation of computing resources provided in accordance with an embodiment of the present disclosure;
FIG. 7 schematically illustrates a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and
fig. 8 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
At present, the integration processing of customers in the industry has the problems of frequent data updating and large data volume, and the industry basically adopts a batch processing mode to lead the data such as user account opening, recharging and the like in the previous day into a data table in the final day assembly file, uniformly process the data through batch programs and update the data into a system for customers to consume in the next day. At present, the solution has low requirement on customer point processing aging, the high-aging point processing scene cannot be met, meanwhile, the processing performance is poor, the customer point processing can not be completed until the next day or later, and the customer can not perform point exchange as early as possible. Batch processing produces a large amount of intermediate data at the same time, and requires a large storage space, including files, data tables, and some intermediate data. The demand for storage is high. In addition, the calculation and storage resources of the integral processing generally adopt a quota allocation mode, and the situation that the service peak often has performance bottleneck and the service valley period resource utilization rate is too low is encountered.
Based on the technical problems described above, an embodiment of the present disclosure provides a data processing method applied to a distributed cloud computing platform, where the distributed cloud computing platform includes a first computing node, the method includes: obtaining recharging information in a first message queue, wherein the first message queue is subscribed and consumed by the first computing node; inquiring a cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information; calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and writing the integration processing result into a second message queue.
Fig. 1 schematically illustrates a system architecture diagram of a data processing apparatus according to an embodiment of the present disclosure. As shown in fig. 1, the risk transaction identification device 100 includes a first message queue 101, a second message queue 102, a first computing node 105, and a cache database 106. The first message queue 101 stores recharging information by adopting a multi-part message queue, and writes the recharging information into the corresponding part message queue according to the offset when recharging occurs, so that the integration collection and consumption can be stably performed under the condition of ensuring a high peak value, and the second message queue 102 is a message queue of the customer integration. The first computing node 105 is deployed on a distributed PAAS platform and concurrently processes messages in a first message queue.
Fig. 2 schematically illustrates an application scenario diagram of a data processing method, apparatus, device, storage medium and program product according to an embodiment of the present disclosure.
As shown in fig. 2, the application scenario 200 according to this embodiment may include a customer point data processing scenario. The network 204 is the medium used to provide communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 205 via the network 204 using the terminal devices 201, 202, 203 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 201, 202, 203.
The terminal devices 201, 202, 203 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 205 may be a client point processing server, where the data processing method provided by the embodiments of the present disclosure is executed, and obtain the refill information in the first message queue, where the first message queue is subscribed to and consumed by the first computing node; inquiring a cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information; calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and writing the integration processing result into a second message queue.
It should be noted that the data processing method provided in the embodiments of the present disclosure may be generally performed by the server 205. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 205. The data processing method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 205 and is capable of communicating with the terminal devices 201, 202, 203 and/or the server 205. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster different from the server 205 and capable of communicating with the terminal devices 201, 202, 203 and/or the server 205.
It should be understood that the number of terminal devices, networks and servers in fig. 2 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
It should be noted that, the data processing method and the device determined by the embodiments of the present disclosure may be used in the field of cloud computing technology, or may be used in the field of financial technology, or may be used in any field other than the financial field, and the application field of the data processing method and the device determined by the embodiments of the present disclosure is not limited.
The data processing method according to the embodiment of the present disclosure will be described in detail below with reference to fig. 3 to 6 based on the system architecture described in fig. 1 and the application scenario described in fig. 2.
Fig. 3 schematically illustrates a flow chart of a data processing method provided according to an embodiment of the present disclosure. As shown in fig. 3, the data processing method of this embodiment includes operations S210 to S240, which may be performed by a server or other computing device. The data processing method provided by the embodiment of the disclosure is applied to a distributed cloud computing platform, wherein the distributed cloud computing platform comprises a first computing node.
In operation S210, the refill information in the first message queue is acquired.
According to an embodiment of the disclosure, the first message queue is subscribed to consumption by the first computing node.
In operation S220, the cache database is queried according to the recharging information to determine account opening information and service parameter information associated with the recharging information.
In operation S230, an integration result is calculated according to the recharging information, the account opening information and the service parameter information.
The integration processing result is written into a second message queue in operation S240.
In one example, when a user generates a recharging action, the upstream application sends recharging information to the first message queue, waits for the first settlement node to process, and the first calculation node subscribes to different partitions of the first message queue to continuously consume the recharging information. The first computing node includes a plurality of containers, each container independently executing corresponding processing logic. Inquiring a cache database according to the obtained recharging information, wherein account opening information and service parameters of a user are stored in the cache database, specifically, inquiring value in the cache database according to key according to the recharging information, and calculating.
In one example, the result of the integral processing is calculated according to the recharging information, the account opening information and the service parameter, and whether the integral issuing condition is satisfied is judged according to the service parameter, for example, whether the recharging amount satisfies the integral issuing condition is determined, and whether the recharging time and the account opening time interval satisfy the integral issuing condition is determined. And sending the integration processing result to a second message queue. For consumption by downstream applications.
According to the data processing method provided by the embodiment of the disclosure, a first computing node subscribes to consume a first message queue to obtain recharging information in the queue, and queries a cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information; calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and writing the integration processing result into a second message queue. Compared with the related art, the method provided by the embodiment of the disclosure has the advantages that the timeliness of integral processing is greatly improved by adopting the distributed computing nodes through the distributed message queues and the cache database, and the message queues and the cache are combined through different data structures, so that the method is clear in functional division, low in coupling and low in maintenance cost.
Fig. 4 schematically shows a flow chart of a data processing method provided according to another embodiment of the present disclosure. As shown in fig. 4, operations S310 and S320 are included.
In operation S310, account opening information and service parameter information in a third message queue that is subscribed for consumption by the second computing node according to an embodiment of the present disclosure are obtained.
In operation S320, the account opening information and the service parameter information are stored in the form of key value pairs in a cache database.
In one example, the third message queue is a message queue storing account opening information and service parameters, and each time an account opening or service parameter is newly added or modified, the third message queue is written into the message queue and waits for consumption by the second computing node. The second computing node deploys a plurality of containers as well, subscribes each partition of the third message queue, writes the consumed account opening information and service parameter information into a cache database in the form of key value pairs, and sets the life cycle according to different services and different data caching time, namely the cached data, and is cleared by memory management after the calculation is completed or the clearing cycle is reached.
Fig. 5 schematically illustrates one of the flowcharts of the method of dynamic allocation of computing resources provided in accordance with an embodiment of the present disclosure. Fig. 6 schematically illustrates a second flowchart of a method for dynamic allocation of computing resources provided in accordance with an embodiment of the present disclosure.
In order to improve the utilization rate of the computing resources, the embodiment of the disclosure further deploys monitoring nodes to monitor the processing condition of the message in real time, and dynamically allocates the computing resources according to the processing condition of the message. The monitoring nodes comprise monitoring main nodes and monitoring standby nodes. The monitoring master node is used for dynamically allocating computing resources of the whole processing device, reading metadata of the message queue, and calculating throughput of the message queue for a period of time, namely a data transmission rate dtr and a data consumption rate dcr. Meanwhile, the data processing rate dsr of each computing node is monitored, when a certain partition dtr of the message queue is detected to be far higher than dcr, the computing nodes needing to be newly added are calculated according to formula weighting, and resource allocation is realized; however, when dtr is at a low threshold set in advance, the number of computing nodes that can be recycled and released can be calculated according to the formula. And stopping the containers with corresponding quantity by the monitoring node according to the subscription condition through the service, and realizing resource recovery. The monitoring standby node is used for high availability processing of the monitoring node, detecting the state of the main node through fixed frequency heartbeat, guaranteeing the availability of the whole resource monitoring, sending monitoring alarm by the monitoring standby node when the monitoring main node fails and is unavailable, subscribing the original monitoring service including message queue throughput, message queue consumption ratio, calculated node processing rate and the like, sending successful message of switching the standby node when the standby node takes over successfully, attempting restarting by the main node, and manually intervening when the restarting frequency reaches the upper limit.
As shown in fig. 5, operations S410 to S420 are included.
The data transmission rate and the data consumption rate of each partition in the first message queue and the third message queue are periodically checked in operation S410.
In operation S420, computing resources of the first computing node and the second computing node are dynamically allocated according to the data transmission rate, the data consumption rate, and a data processing rate of the computing nodes.
As shown in fig. 6, operation S420 includes operation S421 and operation S422.
In operation S421, computing resources of the first computing node and the second computing node are reclaimed according to the data transmission rate and the data processing rate of the computing nodes.
According to an embodiment of the disclosure, when the data transmission rate is determined to be smaller than a first preset threshold value, determining the number of recoverable containers according to the data processing rate of the computing node and the current node number; and recycling the computing resources of the computing nodes according to the number of the recyclable containers.
In operation S422, computing resources of the first computing node and the second computing node are extended according to the data transmission rate, the data consumption rate, and a data processing rate of the computing nodes.
According to an embodiment of the disclosure, when the data transmission rate is determined to be greater than a second preset threshold and the data transmission rate is determined to be greater than a data processing rate of a corresponding computing node, determining a scalable number of containers according to the data transmission rate, the data consumption rate and the data processing rate; and creating containers according to the expandable container number and subscribing corresponding consumption queues.
In one example, the monitoring node periodically checks the transmission condition of each partition of each queue, judges whether the data transmission efficiency is smaller than a first preset threshold value set in advance, if the data transmission rate is determined to be smaller than the first preset threshold value, the current service is characterized as being in a valley period, the computing resources are more redundant, the number of containers to be stopped is calculated through the following formula (1), N is the number of current computing nodes, the sum of dtr of all the partitions is divided by the peak dsr of the computing nodes to obtain the minimum number of computing nodes, and the number of containers to be stopped is calculated.
And checking the container ids with the lowest corresponding number of dsr according to the number of the stopped containers obtained by the calculation module, stopping the corresponding containers by the container start-stop module, and recovering the calculation resources by the partitions which are needed to be consumed again by the residual calculation nodes.
In one example, when the data transmission rate of a partition is greater than a second preset threshold and the data transmission rate is greater than the data consumption rate, the system is characterized by insufficient computing resources at present, and the computing resources are required to be expanded. Calculating a need start according to formula (2)Number of containers moved. W is the weight of each computing node, and if there is a dependency between computing nodes, the node that is dependent is weighted higher. f (f) 1 (j) Is the difference in consumption of the partition, i.e., dtr minus dcr. f (f) 2 (i) The processing rate is consumed evenly for each compute node. And (3) obtaining a plurality of parts generating consumption bottlenecks according to the current data consumption rate dsr condition through a container start-stop module, creating a corresponding number of containers and subscribing to a message queue, so that the system can stably run in a service peak period.
Based on the data processing method, the disclosure also provides a data processing device. The device will be described in detail below in connection with fig. 7.
Fig. 7 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 7, the data processing apparatus 700 of this embodiment includes a first acquisition module 710, a determination module 720, a calculation module 730, and a transmission module 740.
The first obtaining module 710 is configured to obtain refill information in a first message queue, where the first message queue is subscribed for consumption by the first computing node. In an embodiment, the first obtaining module 710 may be configured to perform the operation S210 described above, which is not described herein.
The determining module 720 is configured to query the cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information. In an embodiment, the determining module 720 may be configured to perform the operation S220 described above, which is not described herein.
The calculating module 730 is configured to calculate an integration result according to the recharging information, the account opening information and the service parameter information. In an embodiment, the computing module 730 may be configured to perform the operation S230 described above, which is not described herein.
The sending module 740 is configured to write the integration processing result into the second message queue. In an embodiment, the sending module 740 may be configured to perform the operation S240 described above, which is not described herein.
According to an embodiment of the disclosure, the distributed cloud computing platform further comprises a second computing node, the apparatus further comprising: and the second acquisition module and the storage module.
And the second acquisition module is used for acquiring account opening information and service parameter information in a third message queue, wherein the third message queue is subscribed and consumed by the second computing node. In an embodiment, the second obtaining module may be configured to perform the operation S310 described above, which is not described herein.
And the storage module is used for storing the account opening information and the service parameter information into a cache database in the form of key value pairs. In an embodiment, the storage module may be used to perform the operation S320 described above, which is not described herein.
According to an embodiment of the disclosure, the distributed cloud computing platform further comprises a monitoring node, the apparatus further comprising: the monitoring module and the computing resource allocation module.
And the monitoring module is used for periodically detecting the data transmission rate and the data consumption rate of each partition in the first message queue and the third message queue. In an embodiment, the monitoring module may be configured to perform the operation S410 described above, which is not described herein.
And the computing resource allocation module is used for dynamically allocating the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes. In an embodiment, the computing resource allocation module may be configured to perform the operation S420 described above, which is not described herein.
According to an embodiment of the disclosure, the computing resource allocation module includes a resource reclamation sub-module and a resource expansion sub-module.
And the resource recycling sub-module is used for recycling the computing resources of the first computing node and the second computing node according to the data transmission rate and the data processing rate of the computing nodes. In an embodiment, the resource recycling sub-module may be used to perform the operation S421 described above, which is not described herein.
And the resource expansion sub-module is used for expanding the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes. In an embodiment, the resource extension sub-module may be used to perform the operation S422 described above, which is not described herein.
According to an embodiment of the present disclosure, the resource reclamation sub-module includes: a first determination unit and a resource recovery unit.
And the first determining unit is used for determining the number of recoverable containers according to the data processing rate of the computing node and the current node number when the data transmission rate is determined to be smaller than a first preset threshold value. In an embodiment, the first determining unit may be configured to perform the operation S421 described above, which is not described herein.
And the resource recycling unit is used for recycling the computing resources of the computing nodes according to the number of the recyclable containers. In an embodiment, the resource recycling unit may be configured to perform the operation S421 described above, which is not described herein.
According to an embodiment of the present disclosure, the resource extension submodule includes: a second determination unit and a resource extension unit.
And the second determining unit is used for determining the number of the expandable containers according to the data transmission rate, the data consumption rate and the data processing rate when the data transmission rate is determined to be larger than a second preset threshold value and the data transmission rate is determined to be larger than the data processing rate of the corresponding computing node. In an embodiment, the second determining unit may be configured to perform the operation S422 described above, which is not described herein.
And the resource expansion unit is used for creating containers according to the expandable container number and subscribing corresponding consumption queues. In an embodiment, the resource extension unit may be used to perform the operation S422 described above, which is not described herein.
According to an embodiment of the present disclosure, any of the first acquisition module 710, the determination module 720, the calculation module 730, and the transmission module 740 may be combined in one module to be implemented, or any of the modules may be split into a plurality of modules. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the first acquisition module 710, the determination module 720, the calculation module 730, and the transmission module 740 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable way of integrating or packaging the circuitry, or in any one of or a suitable combination of any of the three implementations of software, hardware, and firmware. Alternatively, at least one of the first acquisition module 710, the determination module 720, the calculation module 730, and the transmission module 740 may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
Fig. 8 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.
As shown in fig. 8, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage portion 908 into a Random Access Memory (RAM) 903. The processor 901 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 901 may also include on-board memory for caching purposes. Processor 901 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the present disclosure.
In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. The processor 901 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the program may be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the disclosure, the electronic device 900 may also include an input/output (I/O) interface 905, the input/output (I/O) interface 905 also being connected to the bus 904. The electronic device 900 may also include one or more of the following components connected to the I/O interface 905: an input section 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 908 including a hard disk or the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 910 so that a computer program read out therefrom is installed into the storage section 908 as needed.
The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium described above carries one or more programs, which when executed, implement a data processing method according to an embodiment of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 902 and/or RAM 903 and/or one or more memories other than ROM 902 and RAM 903 described above.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code means for causing a computer system to carry out the data processing methods provided by the embodiments of the present disclosure when the computer program product is run on the computer system.
The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed, and downloaded and installed in the form of a signal on a network medium, via communication portion 909, and/or installed from removable medium 911. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 909 and/or installed from the removable medium 911. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.
The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.
Claims (10)
1. A data processing method applied to a distributed cloud computing platform, the distributed cloud computing platform including a first computing node, the method comprising:
Obtaining recharging information in a first message queue, wherein the first message queue is subscribed and consumed by the first computing node;
inquiring a cache database according to the recharging information to determine account opening information and service parameter information associated with the recharging information;
calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and
and writing the integration processing result into a second message queue.
2. The method of claim 1, wherein the distributed cloud computing platform further comprises a second computing node, the method further comprising:
acquiring account opening information and service parameter information in a third message queue, wherein the third message queue is subscribed and consumed by the second computing node; and
and storing the account opening information and the service parameter information into a cache database in the form of key value pairs.
3. The method of claim 2, wherein the distributed cloud computing platform further comprises a monitoring node, the method further comprising:
periodically detecting the data transmission rate and the data consumption rate of each partition in the first message queue and the third message queue; and
And dynamically distributing the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes.
4. The method of claim 3, wherein dynamically allocating computing resources of the first computing node and the second computing node based on the data transfer rate, the data consumption rate, and a data processing rate of the computing nodes comprises:
recovering computing resources of the first computing node and the second computing node according to the data transmission rate and the data processing rate of the computing nodes;
and expanding the computing resources of the first computing node and the second computing node according to the data transmission rate, the data consumption rate and the data processing rate of the computing nodes.
5. The method of claim 4, wherein the reclaiming computing resources of the first computing node and the second computing node based on the data transmission rate and a data processing rate of the computing node comprises:
when the data transmission rate is determined to be smaller than a first preset threshold value, determining the number of recoverable containers according to the data processing rate of the computing node and the current node number; and
And recycling the computing resources of the computing nodes according to the number of the recyclable containers.
6. The method of claim 4, wherein expanding the computing resources of the first computing node and the second computing node according to the data transfer rate, the data consumption rate, and a data processing rate of the computing node comprises:
when the data transmission rate is determined to be greater than a second preset threshold value and the data transmission rate is determined to be greater than the data processing rate of the corresponding computing node, determining the number of the expandable containers according to the data transmission rate, the data consumption rate and the data processing rate; and
and creating containers according to the expandable container number and subscribing corresponding consumption queues.
7. A data processing apparatus for use in a distributed cloud computing platform, the distributed cloud computing platform comprising a first computing node, the apparatus comprising:
a first obtaining module, configured to obtain refill information in a first message queue, where the first message queue is subscribed for consumption by the first computing node;
the determining module is used for inquiring the cache database according to the recharging information so as to determine account opening information and service parameter information associated with the recharging information;
The calculation module is used for calculating an integral processing result according to the recharging information, the account opening information and the service parameter information; and
and the sending module is used for writing the integration processing result into a second message queue.
8. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data processing method of any of claims 1-6.
9. A computer readable storage medium having stored thereon executable instructions which when executed by a processor cause the processor to perform the data processing method according to any of claims 1 to 6.
10. A computer program product comprising a computer program which, when executed by a processor, implements a data processing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310572304.4A CN116627670A (en) | 2023-05-19 | 2023-05-19 | Data processing method, apparatus, device, storage medium, and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310572304.4A CN116627670A (en) | 2023-05-19 | 2023-05-19 | Data processing method, apparatus, device, storage medium, and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116627670A true CN116627670A (en) | 2023-08-22 |
Family
ID=87637601
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310572304.4A Pending CN116627670A (en) | 2023-05-19 | 2023-05-19 | Data processing method, apparatus, device, storage medium, and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116627670A (en) |
-
2023
- 2023-05-19 CN CN202310572304.4A patent/CN116627670A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109783237B (en) | Resource allocation method and device | |
| US20180027061A1 (en) | Method and apparatus for elastically scaling virtual machine cluster | |
| US9979631B2 (en) | Dynamic rerouting of service requests between service endpoints for web services in a composite service | |
| CN109117252B (en) | Method and system for task processing based on container and container cluster management system | |
| CN110784498B (en) | Personalized data disaster tolerance method and device | |
| CN112686528B (en) | Method, device, server and medium for distributing customer service resources | |
| CN115277566B (en) | Load balancing method and device for data access, computer equipment and medium | |
| CN112925607A (en) | System capacity expansion and contraction method and device and electronic equipment | |
| CN112650575A (en) | Resource scheduling method and device and cloud service system | |
| US8819239B2 (en) | Distributed resource management systems and methods for resource management thereof | |
| US8812578B2 (en) | Establishing future start times for jobs to be executed in a multi-cluster environment | |
| CN110781441A (en) | New retail business middle desk based on micro-service framework | |
| CN117435569A (en) | Dynamic capacity expansion method, device, equipment, medium and program product for cache system | |
| CN114070847A (en) | Current limiting method, device, equipment and storage medium of server | |
| CN116627670A (en) | Data processing method, apparatus, device, storage medium, and program product | |
| CN115629853A (en) | Task scheduling method and device | |
| CN111754117B (en) | Task issuing method and device, electronic equipment and storage medium | |
| CN114237858A (en) | Task scheduling method and system based on multi-cluster network | |
| CN112632074A (en) | Inventory allocation method and device for database, electronic equipment and medium | |
| CN110765136A (en) | Identifier management method, device, medium and equipment | |
| CN112015745B (en) | Data management method and device | |
| CN117785486B (en) | Environment resource allocation method, device, equipment and medium | |
| CN115391042B (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN115484149B (en) | Network switching method, network switching device, electronic equipment and storage medium | |
| US20240202039A1 (en) | Smart web cluster routing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |