CN112181678A - Service data processing method, device and system, storage medium and electronic device - Google Patents
Service data processing method, device and system, storage medium and electronic device Download PDFInfo
- Publication number
- CN112181678A CN112181678A CN202010949620.5A CN202010949620A CN112181678A CN 112181678 A CN112181678 A CN 112181678A CN 202010949620 A CN202010949620 A CN 202010949620A CN 112181678 A CN112181678 A CN 112181678A
- Authority
- CN
- China
- Prior art keywords
- service data
- distributed message
- processing
- message queue
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/545—Interprogram communication where tasks reside in different layers, e.g. user- and kernel-space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application discloses a method, a device and a system for processing service data, a storage medium and an electronic device. Wherein, the method comprises the following steps: acquiring original service data in a first distributed message queue, wherein the first distributed message queue is used for storing the original service data from a data source; storing intermediate service data processed by the original service data into second distributed message queues, wherein each second distributed message queue is used for storing one type of intermediate service data; and processing the intermediate service data in the second distributed message queue through the stream processing system. The method and the device solve the technical problem that the data processing speed is low in the related technology.
Description
Technical Field
The present application relates to the field of internet, and in particular, to a method, an apparatus, a system, a storage medium, and an electronic apparatus for processing service data.
Background
In the information age of today, various production and business data of enterprises are produced in large quantities every day, and the demand for data services is increasing. Because part of data has timeliness, more and more service indexes need to be acquired in real time, so that each index trend can be better mastered, and the strategy can be quickly responded and timely adjusted.
At present, the main framework of big data processing is that real-time data in a Flink consumption message queue Kafka is loaded into a data warehouse, for example, a patent document with patent publication number CN111339073A discloses a real-time data processing method, a device, an electronic device and a readable storage medium, the real-time data processing method includes: acquiring real-time data; target data is determined through a real-time consumption task, and the target data is consumed into a data warehouse through the real-time consumption task. According to the method, the consumption task directly obtains the user data statistical requirement from the task configuration file, so that under the condition that the user data statistical requirement is changed, only the task configuration file is required to be changed, a new real-time consumption task is not required to be generated each time, and the time for consuming data can be effectively reduced. But only reduce the time to generate new real-time tasks when demand changes and do not fundamentally optimize each real-time task.
With the increase of data service requirements and the complexity of service logic, the Flink processing block in the single Kafka node processing flow is increasingly overstaffed, so that the processing speed is slow, the timeliness of data cannot be ensured, and the code maintenance tends to be complicated.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the application provides a method, a device and a system for processing service data, a storage medium and an electronic device, so as to at least solve the technical problem of low data processing speed in the related technology.
According to an aspect of an embodiment of the present application, a method for processing service data is provided, including: acquiring original service data in a first distributed message queue, wherein the first distributed message queue is used for storing the original service data from a data source; storing intermediate service data processed by the original service data into second distributed message queues, wherein each second distributed message queue is used for storing one type of intermediate service data; and processing the intermediate service data in the second distributed message queue through the stream processing system.
According to another aspect of the embodiments of the present application, there is also provided a device for processing service data, including: the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring original service data in a first distributed message queue, and the first distributed message queue is used for storing the original service data from a data source; the storage unit is used for storing the intermediate service data processed by the original service data into second distributed message queues, wherein each second distributed message queue is used for storing one type of intermediate service data; and the processing unit is used for processing the intermediate service data in the second distributed message queue through the stream processing system.
According to another aspect of the embodiments of the present application, there is also provided a system for processing service data, including: the first distributed message queue is used for storing original service data from a data source; and the stream processing system is used for storing the intermediate service data processed by the original service data into second distributed message queues and processing the intermediate service data in the second distributed message queues, wherein each second distributed message queue in the stream processing system is used for storing one type of intermediate service data.
According to another aspect of the embodiments of the present application, there is also provided a storage medium including a stored program which, when executed, performs the above-described method.
According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the above method through the computer program.
In the embodiment of the application, original service data in a first distributed message queue is obtained, intermediate service data obtained by processing the original service data are stored in second distributed message queues, and each second distributed message queue is used for storing one type of intermediate service data; the intermediate service data in the second distributed message queue is processed through the stream processing system, the processing speed is improved through the processing module of the split stream processing system Flink, the intermediate layer data pipeline Kafka (second distributed message queue) is increased, repeated consumption of subsequent services is facilitated, the code maintenance workload of the Flink module is reduced, and the technical problem of low data processing speed in the related technology can be solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a flowchart of an alternative service data processing method according to an embodiment of the present application;
fig. 2 is a schematic diagram of an alternative service data processing scheme according to an embodiment of the present application;
fig. 3 is a schematic diagram of an alternative service data processing apparatus according to an embodiment of the present application;
and
fig. 4 is a block diagram of a terminal according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
First, partial nouns or terms appearing in the description of the embodiments of the present application are applicable to the following explanations:
kafka is a high-throughput, distributed, publish-subscribe schema-based messaging system that is responsible for passing data from one application to another, with applications only having to focus on data, and not on how data is passed between two or more applications. Messaging is based on reliable message queues, asynchronously delivering messages between a client application and a messaging system. Kafka is a publish-subscribe messaging schema in which messages are persisted into a topic. The consumers can subscribe one or more topics, all data in the topics are consumed, the same data can be consumed by a plurality of consumers, and the data cannot be immediately deleted after being consumed.
Apache Flink is an open source stream processing framework created for distributed, high-performance, readily available and accurate stream processing applications, and the main scene to be processed is stream data, which is a relatively active distributed processing framework in recent years.
According to an aspect of embodiments of the present application, a system embodiment of a system for processing service data is provided. The system comprises: the first distributed message queue is used for storing original service data from a data source; and the stream processing system is configured to store the intermediate service data obtained by processing the original service data in second distributed message queues and process the intermediate service data in the second distributed message queues, where each of the second distributed message queues in the stream processing system is configured to store one type of intermediate service data.
The scheme provides a real-time data processing method based on multi-node Kafka, and the efficiency and the robustness of a real-time task are ensured by segmenting a Flink processing module and adding an intermediate data layer. The main idea of the scheme is to split the Flink computation module (as shown in the dashed box part in fig. 2) based on a data flow framework of Kafka to Flink, and put the intermediate result into Kafka. The advantages of this are: for the same data source service, the result of the intermediate layer Kafka can be directly called without carrying out data processing from the data source again, so that resources and time are saved for subsequent services; after the splitting, a large task is dispersed into small modules, and the small modules can independently perform parallel computing (refer to the content in the specific subsequent implementation mode), which is similar to distributed computing, so that resources are fully utilized, and the effect of improving the speed can be achieved.
According to an aspect of the embodiments of the present application, a method embodiment of a method for processing service data is provided. Fig. 1 is a flowchart of an optional method for processing service data according to an embodiment of the present application, and as shown in fig. 1, the method may include the following steps:
step S1, obtain the original service data in the first distributed message queue, where the first distributed message queue is used to store the original service data from the data source.
Step S2, storing the intermediate service data processed by the original service data into second distributed message queues, where each second distributed message queue is used to store one type of intermediate service data.
Optionally, the storing the intermediate service data processed by the original service data to the second distributed message queue includes: and a first service processing module of the call flow processing system processes the original service data and stores the obtained intermediate service data to a second distributed message queue.
For example, a first service processing module of the call flow processing system performs conversion processing on original service data, and stores obtained intermediate service data to a second distributed message queue; a first service processing module of the call flow processing system is used for performing connection processing on original service data and storing the obtained intermediate service data to a second distributed message queue; a first service processing module of the call flow processing system carries out aggregation processing on original service data, and stores the obtained intermediate service data to a second distributed message queue; and calling a first service processing module of the stream processing system, carrying out window operation on the original service data, and storing the obtained intermediate service data to a second distributed message queue.
And step S3, processing the intermediate service data in the second distributed message queue through the stream processing system.
Optionally, before acquiring the original service data in the first distributed message queue, a plurality of first service processing modules and a plurality of second service processing modules may be configured in the stream processing system, the plurality of first service processing modules being configured to allow parallel operation, and the plurality of second service processing modules being configured to allow parallel operation.
Optionally, the processing, by the stream processing system, the intermediate service data in the second distributed message queue includes: when any one of the second service processing modules in the stream processing system is triggered, processing the intermediate service data in the second distributed message queue through any one of the second service processing modules, and storing the processed result in the database.
Through the steps, the original service data in the first distributed message queue is obtained, and the intermediate service data processed by the original service data is stored in the second distributed message queues, wherein each second distributed message queue is used for storing one type of intermediate service data; the intermediate service data in the second distributed message queue is processed through the stream processing system, the processing speed is improved through the processing module of the split stream processing system Flink, the intermediate layer data pipeline Kafka (second distributed message queue) is increased, repeated consumption of subsequent services is facilitated, meanwhile, the maintenance workload of a Flink module code is reduced, the technical problem that the data processing speed is slow in the related technology can be solved, the speed of a real-time stream processing frame from the Kafka to the Flink is optimized, and the problems that the subsequent Flink processing process is too swollen and tedious and the processing speed is slow due to single Kafka node are solved.
As an alternative example, the technical solution of the present application is further described below with reference to specific embodiments. The scheme is a real-time data processing method based on multi-node Kafka, and the timeliness of real-time tasks is ensured by segmenting the Flink processing module and adding an intermediate data layer. As shown in fig. 2, the specific implementation steps are as follows:
step 1, collecting data to Kafka Ming and Wei layer.
When the data of the database or the service system changes, a response change log is generated, and the change log Binlog of the service library is captured, so that the Binlog data is synchronized and analyzed, and the real-time data can be acquired. And sending the real-time data to a Kafka message queue, wherein different tables correspond to different message queues. For different message queues, the data may be saved to a corresponding database.
Step 2, Flink consumes the kafka data.
The flag provides a specific Kafka connector to read and write data in the Kafka message queue, the flag can freely process events from one or more data streams, and the API provides general modules for data processing, such as transformation (transformation), connection (join), aggregation (aggregation), window manipulation (windows), and the like. The calculated and converted data stream can be output to a Kafka message queue of the next node through a Sink function in the Flink, or directly output to a real-time database or an off-line database for service processing.
Step 3, Kafka detail summary layer.
And the detail summary layer is obtained by window aggregation and join operation of the detail layer on each business fact detail table and dimension table data through the Flink in the step 2. Specifically, the primary key id and the time stamp of each table are extracted to carry out union operation, then the detail information of the primary key id query table is used for carrying out join operation, and the detail summary data mainly exists in a wide table form.
And 4, a Kafka index summary layer.
The index summary layer is obtained by aggregation calculation through the detail layer or the detail summary layer, for example, for order data in the e-commerce system, indexes such as sales volume, monthly sales amount, quarterly sales amount and the like of each category are counted. This layer yields the vast majority of real-time data processing business metrics.
Step 5, Kafka other summary layers.
In order to respond to the data processing requirements of the user, different dimensions can be preset for summarizing and counting, for example, indexes such as collection number, evaluation number and the like can be calculated from the perspective of the user, indexes such as click number, browsing times and the like can be calculated from the perspective of content, specific summarizing indexes are determined according to services, and the data reusability is improved while the pertinence of the services is guaranteed.
Each Kafka in the above steps is an independent and associated module. Independence is that the modules are deployed separately, with data being processed from Kafka to the next layer of Kafka via the flex, and also from Kafka to the storage database via the flex. The association is one data stream from each Kafka data layer, which may be the data source of another data layer. By segmenting the Flink processing module, the middle Kafka data layer is added, the real-time processing speed is improved, and the module division can also reduce the workload of later code maintenance. Meanwhile, data layering also increases the reusability of data, and can provide data support for other services.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.
According to another aspect of the embodiment of the present application, a device for processing service data is further provided, which is used for implementing the method for processing service data. Fig. 3 is a schematic diagram of an alternative service data processing apparatus according to an embodiment of the present application, and as shown in fig. 3, the apparatus may include:
an obtaining unit 31, configured to obtain original service data in a first distributed message queue, where the first distributed message queue is used to store the original service data derived from a data source;
a storing unit 33, configured to store the intermediate service data obtained by processing the original service data into second distributed message queues, where each of the second distributed message queues is used to store one type of intermediate service data;
and the processing unit 35 is configured to process the intermediate service data in the second distributed message queue through a stream processing system.
It should be noted that the acquiring unit 31 in this embodiment may be configured to execute step S1 in this embodiment, the saving unit 33 in this embodiment may be configured to execute step S2 in this embodiment, and the processing unit 35 in this embodiment may be configured to execute step S3 in this embodiment.
The method comprises the steps that original service data in a first distributed message queue are obtained through the modules, intermediate service data processed by the original service data are stored in second distributed message queues, and each second distributed message queue is used for storing one type of intermediate service data; the intermediate service data in the second distributed message queue is processed through the stream processing system, the processing speed is improved through the processing module of the split stream processing system Flink, the intermediate layer data pipeline Kafka (second distributed message queue) is increased, repeated consumption of subsequent services is facilitated, meanwhile, the maintenance workload of a Flink module code is reduced, the technical problem that the data processing speed is slow in the related technology can be solved, the speed of a real-time stream processing frame from the Kafka to the Flink is optimized, and the problems that the subsequent Flink processing process is too swollen and tedious and the processing speed is slow due to single Kafka node are solved.
Optionally, the saving unit is further configured to invoke a first service processing module of the stream processing system, process the original service data, and save the obtained intermediate service data to the second distributed message queue.
Optionally, the saving unit is further configured to: calling a first service processing module of the stream processing system, converting the original service data, and storing the obtained intermediate service data to the second distributed message queue; calling a first service processing module of the stream processing system, performing connection processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue; calling a first service processing module of the stream processing system, performing aggregation processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue; and calling a first service processing module of the stream processing system, performing window operation on the original service data, and storing the obtained intermediate service data to the second distributed message queue.
Optionally, the processing unit is further configured to, when any one of the plurality of second service processing modules in the stream processing system is triggered, process the intermediate service data in the second distributed message queue through the any one of the plurality of second service processing modules, and store a result obtained by the processing in a database.
Optionally, the obtaining unit is further configured to configure, in the stream processing system, before obtaining the original service data in the first distributed message queue, a plurality of first service processing modules and a plurality of second service processing modules, where the plurality of first service processing modules are set to allow parallel operation, and the plurality of second service processing modules are set to allow parallel operation.
The scheme provides a real-time data processing method based on multi-node Kafka, and the efficiency and the robustness of a real-time task are ensured by segmenting a Flink processing module and adding an intermediate data layer. The main idea of the scheme is to split the Flink computation module (as shown in the dashed box part in fig. 2) based on a data flow framework of Kafka to Flink, and put the intermediate result into Kafka. The advantages of this are: for the same data source service, the result of the intermediate layer Kafka can be directly called without carrying out data processing from the data source again, so that resources and time are saved for subsequent services; after the splitting, a large task is dispersed into small modules, and the small modules can independently perform parallel computing (refer to the content in the specific subsequent implementation mode), which is similar to distributed computing, so that resources are fully utilized, and the effect of improving the speed can be achieved.
It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules as a part of the apparatus may run in a corresponding hardware environment, and may be implemented by software, or may be implemented by hardware, where the hardware environment includes a network environment.
According to another aspect of the embodiment of the present application, a server or a terminal for implementing the above service data processing method is also provided.
Fig. 4 is a block diagram of a terminal according to an embodiment of the present application, and as shown in fig. 4, the terminal may include: one or more processors 201 (only one shown in fig. 4), a memory 203, and a transmission means 205. as shown in fig. 4, the terminal may further include an input-output device 207.
The memory 203 may be configured to store software programs and modules, such as program instructions/modules corresponding to the service data processing method and apparatus in the embodiment of the present application, and the processor 201 executes various functional applications and data processing by running the software programs and modules stored in the memory 203, that is, implements the service data processing method described above. The memory 203 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 203 may further include memory located remotely from the processor 201, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 205 is used for receiving or sending data via a network, and can also be used for data transmission between a processor and a memory. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 205 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 205 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
Wherein the memory 203 is specifically used for storing application programs.
The processor 201 may call the application stored in the memory 203 via the transmission means 205 to perform the following steps:
acquiring original service data in a first distributed message queue, wherein the first distributed message queue is used for storing the original service data from a data source;
storing the intermediate service data processed by the original service data into second distributed message queues, wherein each second distributed message queue is used for storing one type of intermediate service data;
and processing the intermediate service data in the second distributed message queue through a stream processing system.
The processor 201 is further configured to perform the following steps:
calling a first service processing module of the stream processing system, converting the original service data, and storing the obtained intermediate service data to the second distributed message queue;
calling a first service processing module of the stream processing system, performing connection processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue;
calling a first service processing module of the stream processing system, performing aggregation processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue;
and calling a first service processing module of the stream processing system, performing window operation on the original service data, and storing the obtained intermediate service data to the second distributed message queue.
By adopting the embodiment of the application, the original service data in the first distributed message queue is obtained, the intermediate service data processed by the original service data is stored in the second distributed message queues, and each second distributed message queue is used for storing one type of intermediate service data; the intermediate service data in the second distributed message queue is processed through the stream processing system, the processing speed is improved through the processing module of the split stream processing system Flink, the intermediate layer data pipeline Kafka (second distributed message queue) is increased, repeated consumption of subsequent services is facilitated, the code maintenance workload of the Flink module is reduced, and the technical problem of low data processing speed in the related technology can be solved.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.
It can be understood by those skilled in the art that the structure shown in fig. 4 is only an illustration, and the terminal may be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a Mobile Internet Device (MID), a PAD, etc. Fig. 4 is a diagram illustrating the structure of the electronic device. For example, the terminal may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 4, or have a different configuration than shown in FIG. 4.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
Embodiments of the present application also provide a storage medium. Optionally, in this embodiment, the storage medium may be used to execute a program code of a processing method of service data.
Optionally, in this embodiment, the storage medium may be located on at least one of a plurality of network devices in a network shown in the above embodiment.
Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:
acquiring original service data in a first distributed message queue, wherein the first distributed message queue is used for storing the original service data from a data source;
storing the intermediate service data processed by the original service data into second distributed message queues, wherein each second distributed message queue is used for storing one type of intermediate service data;
and processing the intermediate service data in the second distributed message queue through a stream processing system.
Optionally, the storage medium is further arranged to store program code for performing the steps of:
calling a first service processing module of the stream processing system, converting the original service data, and storing the obtained intermediate service data to the second distributed message queue;
calling a first service processing module of the stream processing system, performing connection processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue;
calling a first service processing module of the stream processing system, performing aggregation processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue;
and calling a first service processing module of the stream processing system, performing window operation on the original service data, and storing the obtained intermediate service data to the second distributed message queue.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.
Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including instructions for causing one or more computer devices (which may be personal computers, servers, network devices, or the like) to execute all or part of the steps of the method described in the embodiments of the present application.
In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (10)
1. A method for processing service data is characterized by comprising the following steps:
acquiring original service data in a first distributed message queue, wherein the first distributed message queue is used for storing the original service data from a data source;
storing the intermediate service data processed by the original service data into second distributed message queues, wherein each second distributed message queue is used for storing one type of intermediate service data;
and processing the intermediate service data in the second distributed message queue through a stream processing system.
2. The method of claim 1, wherein saving the intermediate traffic data processed on the raw traffic data to a second distributed message queue comprises:
and calling a first service processing module of the stream processing system, processing the original service data, and storing the obtained intermediate service data to the second distributed message queue.
3. The method of claim 2, wherein invoking a first service processing module of the stream processing system to process the original service data and saving the obtained intermediate service data to the second distributed message queue comprises at least one of:
calling a first service processing module of the stream processing system, converting the original service data, and storing the obtained intermediate service data to the second distributed message queue;
calling a first service processing module of the stream processing system, performing connection processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue;
calling a first service processing module of the stream processing system, performing aggregation processing on the original service data, and storing the obtained intermediate service data to the second distributed message queue;
and calling a first service processing module of the stream processing system, performing window operation on the original service data, and storing the obtained intermediate service data to the second distributed message queue.
4. The method of claim 1, wherein processing the intermediate traffic data in the second distributed message queue by a stream processing system comprises:
when any one of a plurality of second service processing modules in the stream processing system is triggered, processing the intermediate service data in the second distributed message queue through the any one service processing module, and storing the processed result in a database.
5. The method according to any of claims 1 to 4, wherein before obtaining the original traffic data in the first distributed message queue, the method further comprises:
configuring a plurality of first service processing modules and a plurality of second service processing modules in the stream processing system, wherein the plurality of first service processing modules are set to allow parallel operation, and the plurality of second service processing modules are set to allow parallel operation.
6. A device for processing service data, comprising:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring original service data in a first distributed message queue, and the first distributed message queue is used for storing the original service data from a data source;
a storage unit, configured to store intermediate service data obtained by processing the original service data into second distributed message queues, where each of the second distributed message queues is used to store one type of intermediate service data;
and the processing unit is used for processing the intermediate service data in the second distributed message queue through a stream processing system.
7. The apparatus of claim 6, wherein the saving unit is further configured to:
and calling a first service processing module of the stream processing system, processing the original service data, and storing the obtained intermediate service data to the second distributed message queue.
8. A system for processing traffic data, comprising:
the first distributed message queue is used for storing original service data from a data source;
and the stream processing system is configured to store the intermediate service data obtained by processing the original service data in second distributed message queues and process the intermediate service data in the second distributed message queues, where each of the second distributed message queues in the stream processing system is configured to store one type of intermediate service data.
9. A storage medium, characterized in that the storage medium comprises a stored program, wherein the program when executed performs the method of any of the preceding claims 1 to 5.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the method of any of the preceding claims 1 to 5 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010949620.5A CN112181678A (en) | 2020-09-10 | 2020-09-10 | Service data processing method, device and system, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010949620.5A CN112181678A (en) | 2020-09-10 | 2020-09-10 | Service data processing method, device and system, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112181678A true CN112181678A (en) | 2021-01-05 |
Family
ID=73921822
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010949620.5A Pending CN112181678A (en) | 2020-09-10 | 2020-09-10 | Service data processing method, device and system, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112181678A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860710A (en) * | 2021-03-18 | 2021-05-28 | 杭州云灵科技有限公司 | Data processing method, device and system and data query method and system |
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN116132540A (en) * | 2023-04-13 | 2023-05-16 | 北京东大正保科技有限公司 | Multi-service system data processing method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130339473A1 (en) * | 2012-06-15 | 2013-12-19 | Zynga Inc. | Real time analytics via stream processing |
| CN110784419A (en) * | 2019-10-22 | 2020-02-11 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional data of railway electric affairs |
| CN111241078A (en) * | 2020-01-07 | 2020-06-05 | 网易（杭州）网络有限公司 | Data analysis system, data analysis method and device |
| CN111339175A (en) * | 2020-02-28 | 2020-06-26 | 成都运力科技有限公司 | Data processing method and device, electronic equipment and readable storage medium |
-
2020
- 2020-09-10 CN CN202010949620.5A patent/CN112181678A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130339473A1 (en) * | 2012-06-15 | 2013-12-19 | Zynga Inc. | Real time analytics via stream processing |
| CN110784419A (en) * | 2019-10-22 | 2020-02-11 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Method and system for visualizing professional data of railway electric affairs |
| CN111241078A (en) * | 2020-01-07 | 2020-06-05 | 网易（杭州）网络有限公司 | Data analysis system, data analysis method and device |
| CN111339175A (en) * | 2020-02-28 | 2020-06-26 | 成都运力科技有限公司 | Data processing method and device, electronic equipment and readable storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860710A (en) * | 2021-03-18 | 2021-05-28 | 杭州云灵科技有限公司 | Data processing method, device and system and data query method and system |
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN113656370B (en) * | 2021-08-16 | 2024-04-30 | 南方电网数字电网集团有限公司 | Data processing method and device for electric power measurement system and computer equipment |
| CN116132540A (en) * | 2023-04-13 | 2023-05-16 | 北京东大正保科技有限公司 | Multi-service system data processing method and device |
| CN116132540B (en) * | 2023-04-13 | 2023-08-01 | 北京东大正保科技有限公司 | Multi-service system data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10447772B2 (en) | Managed function execution for processing data streams in real time | |
| US11392416B2 (en) | Automated reconfiguration of real time data stream processing | |
| CN107577805B (en) | Business service system for log big data analysis | |
| US10860618B2 (en) | Low-latency streaming analytics | |
| US11838351B1 (en) | Customizable load balancing in a user behavior analytics deployment | |
| CN111241078A (en) | Data analysis system, data analysis method and device | |
| WO2020258290A1 (en) | Log data collection method, log data collection apparatus, storage medium and log data collection system | |
| CN112181678A (en) | Service data processing method, device and system, storage medium and electronic device | |
| CN111090699A (en) | Service data synchronization method and device, storage medium and electronic device | |
| CN110909063A (en) | User behavior analysis method and device, application server and storage medium | |
| US11449371B1 (en) | Indexing data at a data intake and query system based on a node capacity threshold | |
| CN113010565B (en) | Server real-time data processing method and system based on server cluster | |
| US10182104B1 (en) | Automatic propagation of resource attributes in a provider network according to propagation criteria | |
| EP2862338A1 (en) | Method, server, and client for pushing and displaying splash screen | |
| CN112559301A (en) | Service processing method, storage medium, processor and electronic device | |
| CN110928681A (en) | Data processing method and device, storage medium and electronic device | |
| CN111367953A (en) | Streaming processing method and device for information data | |
| CN114090388A (en) | Information acquisition method, server cluster, server and equipment | |
| CN112732793A (en) | Service data processing method and device, storage medium and electronic device | |
| CN114971714A (en) | Accurate customer operation method based on big data label and computer equipment | |
| CN112506887B (en) | Vehicle terminal CAN bus data processing method and device | |
| US10733002B1 (en) | Virtual machine instance data aggregation | |
| CN112417018A (en) | Data sharing method and device | |
| CN111988368A (en) | Data interaction system and interaction method | |
| CN103856359A (en) | Method and system for obtaining information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |