CN114185701A

CN114185701A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN114185701A
Application number: CN202111515252.4A
Authority: CN
Inventors: 李飞飞
Original assignee: Agricultural Bank of China
Current assignee: Agricultural Bank of China
Priority date: 2021-12-13
Filing date: 2021-12-13
Publication date: 2022-03-15

Abstract

The embodiment of the invention discloses a data processing method and device, electronic equipment and a storage medium. Wherein, the method comprises the following steps: receiving data to be processed by at least one receiving node; determining data to be arranged from the data to be processed according to a preset data processing sequence determination rule, and storing the data to be arranged into a preset blocking queue; and sending the data to be arranged in the blocking queue to a data processing node for the data processing node to perform data processing on the data to be arranged. Multiple receiving nodes can realize self-adaptive expansion in different actual service scenes. The concurrent condition of data processing can be subjected to peak clipping and valley filling through the blocking queue, so that server breakdown caused by excessive data is avoided, and the data processing efficiency is improved.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

Embodiments of the present invention relate to computer technologies, and in particular, to a method and an apparatus for data processing, an electronic device, and a storage medium.

Background

In the development of projects with different business requirements, data between different systems often needs to be interacted and processed. For example, the system a is a data source generating party, and generates various types of service data, and the system B needs to use and process the data of the system a.

In the prior art, a data receiving node of a system B obtains data of a system a, and directly processes the obtained data. Under the complex conditions of high concurrency, large data volume, high data accuracy requirement and the like, the B system cannot receive the data in a zero-error manner, the data processing nodes of the B system are easy to crash, and the efficiency and the accuracy of data processing are low.

Disclosure of Invention

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium, and aims to improve the data processing efficiency.

In a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:

receiving data to be processed by at least one receiving node;

determining data to be arranged from the data to be processed according to a preset data processing sequence determination rule, and storing the data to be arranged into a preset blocking queue;

and sending the data to be arranged in the blocking queue to a data processing node for the data processing node to perform data processing on the data to be arranged.

In a second aspect, an embodiment of the present invention further provides an apparatus for data processing, where the apparatus includes:

the data processing device comprises a to-be-processed data receiving module, a processing module and a processing module, wherein the to-be-processed data receiving module is used for receiving to-be-processed data through at least one receiving node;

the data to be arranged determining module is used for determining data to be arranged from the data to be processed according to a preset data processing sequence determining rule and storing the data to be arranged into a preset blocking queue;

and the data processing module is used for sending the data to be arranged in the blocking queue to a data processing node, so that the data processing node can perform data processing on the data to be arranged.

In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the data processing method according to any embodiment of the present invention.

In a fourth aspect, embodiments of the present invention also provide a storage medium containing computer-executable instructions for performing a method of data processing according to any of the embodiments of the present invention when executed by a computer processor.

The data processing method and the device for the data processing are characterized in that the data to be processed are obtained through at least one receiving node, the data to be arranged in the data to be processed are stored in a blocking queue according to a data processing sequence, and the data to be arranged in the blocking queue are sent to a data processing node for data processing. All data to be processed are prevented from being processed, and peak clipping and valley filling of data processing amount are effectively achieved. The problem that in the prior art, the processing request is invalid or the data is lost due to overlarge data volume is solved, the phenomenon that the server crashes because the data adding and processing node cannot accept a huge number of processing requests is avoided, and the precision and the efficiency of data processing are improved.

Drawings

FIG. 1 is a flow chart illustrating a method of data processing according to a first embodiment of the present invention;

FIG. 2 is a flowchart illustrating a data processing method according to a second embodiment of the present invention;

fig. 3 is a block diagram of a data processing apparatus according to a third embodiment of the present invention;

fig. 4 is a schematic structural diagram of a data processing apparatus in a fourth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Example one

Fig. 1 is a flowchart illustrating a data processing method according to an embodiment of the present invention, where the embodiment is applicable to processing a large amount of data, and the method can be executed by a data processing apparatus. As shown in fig. 1, the method specifically includes the following steps:

step 110, receiving data to be processed by at least one receiving node.

In actual project development, data interaction among different systems is often required, and cross-system transmission of data is performed. For example, the system a is a data source generating party, and may generate various service data, such as personal transaction detail data, personal consumption record or personal credit record, etc., and the system B needs to use and process the data of the system a, where the data generated by the system a is to-be-processed data. The side receiving data, i.e. the B system, may adopt a two-layer architecture mode, where the first layer is a data receiving layer for receiving the data to be processed by Kafka, and the second layer is a data processing layer for performing data processing. Kafka is an open source streaming platform that can be used to build real-time data pipes and streaming applications, and can be deployed on one or more server nodes. The A system serves as a data producer and sends data to be processed to the Kafka, and the B system can register in the Kafka and then serves as a consumer to use the data to be processed of the A system. The data receiving logic and the data processing logic are decoupled by the two-layer architecture mode, so that the data processing layer is more flexible and easier to perform function expansion, and server resources are easier to laterally expand.

The system A is a credit card transaction system, has services of credit card development, consumption, expenditure, payment and the like, generates data such as credit card records, credit card consumption records and credit card payment records, and the system B is a credit card risk management system, and needs to analyze personal accounts by using the credit card related data generated by the system A through technologies such as data analysis, data mining and the like, constructs personal credit records and provides the personal credit records for a credit investigation center to use. In the cross-system data access, a Kafka message queue is used as a data storage intermediate party, the system A is used as a data generating party to send self data serving as data to be processed to the Kafka, and the system B is used as a consuming party to receive the data to be processed.

A plurality of receiving nodes can be arranged at the data receiving layer of the B system for receiving the data to be processed of the Kafka message bus. The number of receiving nodes can be dynamically adjusted by a user according to the data volume, for example, the a system can generate 2 hundred million pieces of source data every day, the average concurrency is 3000 pieces/second, the peak time is 6000 pieces/second, the B system needs to receive the data in real time and with zero error, and the data of the B system is formed after the data is subjected to complex logic processing. For such a service scenario, the problems of data timeliness, data loss prevention, data repetition, data processing efficiency, data accuracy and the like need to be considered, and meanwhile, the influence of network abnormality and server downtime on the system needs to be avoided. And 12 servers can be arranged at a data receiving layer as receiving nodes to receive the data to be processed of Kafka. The user can change the receiving nodes properly according to the maximum partition number of the actual Kafka, each node receives 50 pieces of data to be processed at the maximum by default once, and each receiving node determines uniqueness of each piece of received data to be processed through three parameters, namely the data type, the partition and the offset, so that repeated data is prevented from being received.

Step 120, determining data to be arranged from the data to be processed according to a preset data processing sequence determination rule, and storing the data to be arranged in a preset blocking queue.

The method comprises the steps of presetting a data processing sequence determination rule, determining the rule according to the data processing sequence, and continuously processing received data to be processed. The data processing order determination rule may be a reception time order in which the receiving node receives the data to be processed.

The data to be arranged is determined from the data to be processed according to the data processing order determination rule, and for example, the data to be processed received earliest from among the data to be processed may be determined as the data to be arranged according to the reception time order in which the data to be processed is received by the receiving node. The data to be queued is put into a pre-set congestion queue, and the size of the congestion queue can be set to be 50 at the maximum, that is, there are 50 pieces of data to be processed at most in the congestion queue. If the data in the blocking queue reaches the maximum value, that is, the receiving node cannot transmit the data to be processed to the blocking queue any more, the receiving process of the current data to be queued is blocked until the data to be processed in the blocking queue drops below 50 or until the data to be queued does not exist in the blocking queue, and the receiving process of the data to be queued can be restarted. When the data to be arranged in the blocking queue reaches the maximum value, the receiving node can also stop receiving the data to be processed. The receiving node that stops the reception of the pending data may be a receiving node that has received the pending data but failed to send the pending data to the blocking queue due to the blocking of the blocking queue. And after all the data to be processed in the receiving node is put into the blocking queue, starting the data receiving of the receiving node. By adopting the blocking queue, peak clipping and valley filling can be performed, data congestion is avoided, excessive data processing requests are prevented from being sent to the data processing layer, and the phenomenon that the data processing layer cannot accept a large number of requests and the server crashes is avoided.

In this embodiment, optionally, determining data to be arranged from the data to be processed according to a preset data processing sequence determination rule includes: and determining the data to be arranged in a preset batch size from the data to be processed according to the receiving time of the data to be processed.

Specifically, the batch size of the data to be queued is preset, for example, the congestion queue accommodates at most 50 pieces of data to be queued, and the batch of the data to be queued may be 50 pieces of data to be processed, that is, the receiving node sends 50 pieces of data to be processed with the earliest reception time as the data to be queued to the congestion queue each time. When there is no data to be queued in the congestion queue, the receiving node sends 50 data to be queued to the congestion queue again.

Each receiving node can contain 50 pieces of data to be processed at most, and when the number of the data to be processed of the receiving node reaches 50, the receiving of the data to be processed can be stopped. When all the data to be processed in the receiving node is sent to the blocking queue, the receiving node may receive the data to be processed again. The receiving node may also send the pending data to the congestion queue one by one, and each time, send the pending data with the earliest receiving time as the pending data. The beneficial effect who sets up like this lies in, carries out batch processing through the data of treating of presetting batch size, can carry out the load shifting of clipping peak, avoids data crowding. And by adopting the blocking queue, the asynchronous problem of the data receiving layer and the data processing layer is effectively solved, and the efficiency and the precision of data processing are improved.

And step 130, sending the data to be arranged in the blocking queue to a data processing node for the data processing node to perform data processing on the data to be arranged.

After the data to be arranged is placed in the blocking queue, the data to be arranged can be sent to a data processing node of the data processing layer, and the data processing node performs data processing according to a preset data processing rule. The data to be arranged can be sent to the data processing nodes one by one, and the data to be arranged can also be sent to the data processing nodes in a multi-thread mode, so that the parallel processing of the data is realized, and the data processing efficiency is improved.

In this embodiment, optionally, sending the to-be-queued data in the blocking queue to the data processing node includes: determining target data according to the queue sequence of the data to be arranged in the blocking queue; and sending the target data to the data processing node through the HTTP request.

Specifically, the target data may be selected from the to-be-queued data in the blocking queue, and the target data may be sent to the data processing node. The target data may be determined by a queue order of the data to be queued in the congestion queue, e.g., the data to be queued that enters the congestion queue earliest as the target data. The target data may be transmitted to the data processing node by an HTTP (Hyper Text Transfer Protocol) request. The beneficial effect of setting in this way lies in that through confirming the target data, can prevent that excessive HTTP from requesting to send to the data processing layer, avoid the data processing layer can't accept the phenomenon that huge request appears the server and collapses, improve data processing precision and efficiency.

In this embodiment, optionally, at least two data processing nodes are provided; correspondingly, the target data is sent to the data processing node through the HTTP request, and the method comprises the following steps: sending the target data to load balancing through an HTTP connection pool; and determining a target processing node from the at least two data processing nodes through load balancing, and sending target data to the target processing node.

Specifically, a plurality of data processing nodes may be provided in the data processing layer, for example, dozens of data processing nodes may be provided, and the horizontal expansion may be performed according to the actual service requirement and the complexity of data processing, so as to achieve the high efficiency of real-time processing. And determining a target processing node in the plurality of data processing nodes, and sending the target data in the blocking queue to the target processing node. The target data may be one or more, and the target processing nodes of the target data may be the same or different. And sending the plurality of target data to the respective target processing nodes. The target processing node may be determined from at least two data processing nodes through load balancing, for example, the working pressure of each data processing node may be obtained through load balancing, and a data processing node with a small working pressure may be determined as the target processing node.

The target data is transmitted through the HTTP request, and may be transmitted to the load balancing through the HTTP connection pool, and then transmitted to the target processing node through the load balancing. Because the establishment and the release of the HTTP connection consume resources, an HTTP connection pool can be constructed, a certain number of HTTP connections are always reserved in the HTTP connection pool, the HTTP connections are always kept in a connection state, connection reestablishment and connection release are not needed, and therefore time and resource consumption can be saved. The beneficial effect of the arrangement is that a multithreading mode is adopted, a plurality of target data in the blocking queue are sent to the target processing node of the data processing layer through HTTP requests, and the data transmission efficiency is improved. And by adding the HTTP connection pool, the time consumption caused by frequently establishing HTTP connection and disconnecting can be greatly reduced, and the data processing precision and efficiency are improved. Through load balancing, received data processing requests are distributed to the data processing node servers at the back end in a balanced mode, the situation that the requests of some servers are stressed and other servers do not have the requests can be avoided.

In this embodiment, optionally, after sending the target data to the load balancing through the HTTP connection pool, the method further includes: and judging whether the data receiving quantity of the load balance reaches a preset data quantity threshold value, and if so, storing the target data serving as the data to be arranged back to the blocking queue.

Specifically, a data volume threshold of the target data received by load balancing is preset, and if the data receiving volume of the target data received by load balancing exceeds the preset data volume threshold, the data processing nodes cannot be allocated to the excess target data, and the excess target data is discarded. And judging whether the data receiving quantity of the load balance reaches a preset data quantity threshold value or not in real time, and if the data receiving quantity of the load balance receiving reaches the data quantity threshold value, taking the discarded target data as the data to be arranged and storing the data back to the blocking queue again.

If the network is interrupted before the target data is sent to the target processing node from the load balance, the target data which is not distributed to the target processing node during interruption can be added into the blocking queue again, and the HTTP request is sent again after the data processing layer is normal.

The data receiving layer can comprise receiving nodes and blocking queues, and the data processing layer can comprise load balancing and data processing nodes. When the blocking queue sends target data to load balance through the HTTP connection pool, if abnormal conditions such as connection overtime abnormal conditions, request overtime abnormal conditions, refused requests and the like are met, the conditions that a data processing layer is in high load or a server is down and the like are described, at the moment, the current HTTP request is considered to be failed to send, the target data are continuously stored in the blocking queue, and then retransmission is carried out. The beneficial effect who sets up like this lies in, avoids the losing of pending data, and the reliability is high, improves data processing precision.

According to the technical scheme of the embodiment, the data to be processed is obtained through at least one receiving node, the data to be arranged in the data to be processed is stored in the blocking queue according to the data processing sequence, and the data to be arranged in the blocking queue is sent to the data processing node for data processing. All data to be processed are prevented from being processed, and peak clipping and valley filling of data processing amount are effectively achieved. The problem that in the prior art, the processing request is invalid or the data is lost due to overlarge data volume is solved, the phenomenon that the server crashes because the data adding and processing node cannot accept a huge number of processing requests is avoided, and the precision and the efficiency of data processing are improved.

Example two

Fig. 2 is a flowchart illustrating a data processing method according to a second embodiment of the present invention, which is further optimized based on the second embodiment of the present invention. As shown in fig. 2, the method specifically includes the following steps:

step 210, receiving data to be processed by at least one receiving node.

Step 220, determining data to be arranged from the data to be processed according to a preset data processing sequence determination rule, storing the data to be arranged in a preset blocking queue, and storing the received data to be processed in a preset data cache table.

The method comprises the steps of presetting a blocking queue and a data cache table, putting data to be processed with preset batch sizes in a receiving node into the blocking queue as data to be arranged, and storing all the data to be processed in the receiving node into the data cache table. For example, there are twelve receiving nodes, each receiving node can receive 50 pieces of data, that is, the receiving node can receive 600 pieces of data in total, and 50 pieces of data can be arranged in the blocking queue. 600 pieces of data received by the receiving node can be stored in a data cache table in real time, and every 50 pieces of 600 pieces of data are transmitted to the blocking queue. After the number of data in the blocking queue reaches 50, if the receiving node still has data to be processed, the receiving node does not receive new data to be processed any more. When the data to be processed does not exist in any receiving node, the receiving node can start to receive new data to be processed and continuously store the new data to be processed into the data cache table.

The received data to be processed is stored in the data cache table, data persistence can be carried out, data loss is prevented, and the storage time of the data to be processed can be recorded in the data cache table. Meanwhile, the received data to be processed is inserted into a blocking queue, the blocking queue is designed to cut peaks and fill valleys, excessive HTTP requests can be prevented from being sent to a data processing layer, and the phenomenon that the data processing layer cannot accept a large number of requests and a server is crashed is avoided.

And step 230, sending the data to be arranged in the blocking queue to the data processing node for the data processing node to perform data processing on the data to be arranged.

Step 240, judging whether data processing completion information fed back by the data processing node is received in a preset data processing period; and if so, deleting the data to be arranged from the data cache table.

After receiving the data to be arranged, the data processing node processes the data to be arranged according to the processing mode of the service requirement, and feeds back data processing completion information after the processing is completed. If the data is not processed completely or the processing fails, the data processing completion information may not be generated or the processing failure information may not be fed back. A data processing period is preset, and under a normal condition, the target processing node can complete data processing in the data processing period. And judging whether data processing completion information fed back by the data processing node is received in a preset data processing period, if the data processing completion information is received, determining that the data to be arranged is successfully processed, and deleting the data to be arranged from the data cache table.

If the data processing completion information is not received, it is determined that the data to be arranged is not successfully processed, and the processing may fail due to network interruption or service interruption at the data processing layer. The data to be arranged can be continuously kept in the data cache table, and the loss of unprocessed data is effectively avoided. A timing processing mechanism can be arranged on the data cache table, and data to be processed stored before the preset time in the data cache table is reprocessed according to the storage time recorded when the data cache table is stored. For example, if the preset time is two minutes, the data processing layer may have extreme situations such as a network sudden interruption or a server sudden crash, and the data being processed may fail, but the data is still stored in the data cache table. The data to be processed stored in the data cache table for two minutes can be inquired out through a timing processing mechanism, and the data to be processed is sent to the data processing layer again through the HTTP request for processing. The data to be processed stored in the data cache table for two minutes before can be directly sent to the data processing node through the HTTP request through load balancing, and the data to be processed stored in the data cache table for two minutes before can also be inserted into the blocking queue and then sent to the data processing node through the load balancing.

In the embodiment of the invention, the data to be processed is received at the data receiving layer and then stored in the data cache table to achieve the purpose of data persistence, and even if the data processing process fails due to extreme reasons such as abnormal downtime of a server or network interruption, the data cannot be lost. Because only after the data is processed successfully, the data is deleted from the data cache table, and zero loss of the data is guaranteed.

The data receiving layer is responsible for receiving the data to be processed, the data processing layer is responsible for processing and post-processing the data to be processed in all dimensions, and as the data receiving layer is limited by Kafka, at most 12 receiving nodes can be deployed for receiving the data. However, the data processing layer can be transversely expanded according to actual conditions, more data processing nodes are deployed for data processing, the data receiving layer can keep the structure unchanged, only the data processing layer needs to be expanded, and the flexibility of data processing is improved. The embodiment is suitable for most of cross-system data real-time access scenes, and is not limited to Kafka as a message bus. In practical application, adaptive expansion can be performed according to an actual service scene, for example, when the data computation complexity is high, a real-time stream processing technology and the like can be added in a data processing node.

By adopting an efficient HTTP connection pool strategy and a blocking queue, the receiving of more than 10 hundred million data per day can be easily achieved. And the concurrent processing of the data is carried out by increasing the number of nodes of the data processing layer. For example, the data receiving layer receives 50 pieces of data from the a system at a time, and if each piece of data requires 10ms, if sequential processing is adopted, 50 pieces of data are processed, which requires 500ms, which is 50 × 10 ms. In this embodiment, however, the data receiving layer may adopt multiple threads to simultaneously send 50 pieces of data to the data processing layer, the data processing layer may adopt 50 data processing nodes, each data processing node receives one piece of data, and the 50 pieces of data are processed simultaneously and concurrently, so that only 10ms is required for processing 50 pieces of data, and the real-time performance of data processing is ensured.

In addition, the blocking queue can effectively solve the asynchronous problem of the data receiving layer and the data processing layer. When abnormal conditions such as connection timeout abnormity, request refusal and the like occur, the data to be arranged can be added into the blocking queue again to wait for subsequent retransmission. By adopting a data cache table strategy, data to be processed is inserted first, and then deleted after being processed successfully, so that the data can be ensured not to be lost under extreme conditions of sudden network interruption, power failure or sudden server shutdown and the like.

EXAMPLE III

Fig. 3 is a block diagram of a data processing apparatus according to a third embodiment of the present invention, which is capable of executing a data processing method according to any embodiment of the present invention, and includes functional modules and beneficial effects corresponding to the execution method. As shown in fig. 3, the apparatus specifically includes:

a to-be-processed data receiving module 301, configured to receive to-be-processed data through at least one receiving node;

a to-be-arranged data determining module 302, configured to determine to-be-arranged data from the to-be-processed data according to a preset data processing sequence determining rule, and store the to-be-arranged data in a preset blocking queue;

the data processing module 303 is configured to send the data to be arranged in the blocking queue to a data processing node, so that the data processing node performs data processing on the data to be arranged.

Optionally, the to-be-arranged data determining module 302 is specifically configured to:

and determining the data to be arranged in a preset batch size from the data to be processed according to the receiving time of the data to be processed.

Optionally, the data processing module 303 includes:

the target data determining unit is used for determining target data according to the queue sequence of the data to be arranged in the blocking queue;

and the target data sending unit is used for sending the target data to the data processing node through the HTTP request.

Optionally, the number of the data processing nodes is at least two;

accordingly, the target data transmission unit includes:

the HTTP request sending subunit is used for sending the target data to the load balancing unit through an HTTP connection pool;

and the target processing node determining subunit is configured to determine a target processing node from the at least two data processing nodes through the load balancing, and send the target data to the target processing node.

Optionally, the target data sending unit further includes:

and the data receiving quantity judging subunit is configured to, after the target data is sent to the load balancing through the HTTP connection pool, judge whether the data receiving quantity of the load balancing reaches a preset data quantity threshold, and if so, store the target data as data to be arranged back to the blocking queue.

Optionally, the apparatus further comprises:

the data caching module is used for storing the received data to be processed into a preset data caching table after the data to be processed is received by at least one receiving node.

Optionally, the apparatus further comprises:

the data deleting module is used for judging whether data processing completion information fed back by the data processing node is received in a preset data processing period after the data to be arranged in the blocking queue is sent to the data processing node;

and if so, deleting the data to be arranged from the data cache table.

Example four

Fig. 4 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention. The data processing device is an electronic device and fig. 4 shows a block diagram of an exemplary electronic device 400 suitable for use in implementing embodiments of the present invention. The electronic device 400 shown in fig. 4 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.

As shown in fig. 4, electronic device 400 is embodied in the form of a general purpose computing device. The components of electronic device 400 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 that couples the various system components (including the system memory 402 and the processing unit 401).

Bus 403 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Electronic device 400 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by electronic device 400 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 402 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)404 and/or cache memory 405. The electronic device 400 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 406 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 403 by one or more data media interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 407 generally perform the functions and/or methods of the described embodiments of the invention.

The electronic device 400 may also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), with one or more devices that enable a user to interact with the electronic device 400, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device 400 to communicate with one or more other computing devices. Such communication may be through input/output (I/O) interface 411. Also, the electronic device 400 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 412. As shown in FIG. 4, the network adapter 412 communicates with the other modules of the electronic device 400 over the bus 403. It should be appreciated that although not shown in FIG. 4, other hardware and/or software modules may be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402, for example, implementing a data processing method provided by an embodiment of the present invention, including:

receiving data to be processed by at least one receiving node;

EXAMPLE five

The fifth embodiment of the present invention further provides a storage medium containing computer-executable instructions, where the storage medium stores a computer program, and when the computer program is executed by a processor, the method for implementing data processing provided by the fifth embodiment of the present invention includes:

receiving data to be processed by at least one receiving node;

Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method of data processing, comprising:

receiving data to be processed by at least one receiving node;

2. The method according to claim 1, wherein determining data to be arranged from the data to be processed according to a preset data processing order determination rule comprises:

3. The method of claim 1, wherein sending the data to be queued in the congestion queue to a data processing node comprises:

determining target data according to the queue sequence of the data to be arranged in the blocking queue;

and sending the target data to a data processing node through an HTTP request.

4. The method of claim 3, wherein there are at least two of said data processing nodes;

correspondingly, the sending the target data to the data processing node through the HTTP request comprises the following steps:

sending the target data to load balancing through an HTTP connection pool;

and determining a target processing node from the at least two data processing nodes through the load balancing, and sending the target data to the target processing node.

5. The method of claim 4, after sending the target data to load balancing over a pool of HTTP connections, further comprising:

and judging whether the data receiving quantity of the load balance reaches a preset data quantity threshold value, if so, storing the target data serving as data to be arranged back to the blocking queue.

6. The method of claim 1, after receiving the data to be processed by the at least one receiving node, further comprising:

and storing the received data to be processed into a preset data cache table.

7. The method of claim 6, further comprising, after sending the data to be queued in the congestion queue to a data processing node:

judging whether data processing completion information fed back by the data processing node is received in a preset data processing period or not;

and if so, deleting the data to be arranged from the data cache table.

8. An apparatus for data processing, comprising:

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of data processing according to any of claims 1-7 when executing the program.

10. A storage medium containing computer-executable instructions for performing the method of data processing according to any one of claims 1-7 when executed by a computer processor.