CN114330273A

CN114330273A - Stream data processing method, device, server and storage medium

Info

Publication number: CN114330273A
Application number: CN202111639987.8A
Authority: CN
Inventors: 高飞; 郗宏伟; 李宁; 李旭堂; 刘津津
Original assignee: Agricultural Bank of China
Current assignee: Agricultural Bank of China
Priority date: 2021-12-29
Filing date: 2021-12-29
Publication date: 2022-04-12

Abstract

The invention discloses a stream data processing method, a stream data processing device, a server and a storage medium, wherein the method comprises the following steps: acquiring stream data of a preset theme; acquiring the subscription information of a consumer according to target data in the stream data; determining a data processing template according to the subscription information; processing the stream data according to the data processing template to obtain target data; the target data is sent to the consumer. The stream data processing method provided by the invention can acquire the pre-configured data processing template of the subscriber for the target data of the consumer with the subscription information after the stream data is pulled, and process the stream data according to the data processing template. The data processing template can be quickly applied to different users without complex manual code editing, so that the expandability of stream data processing can be improved.

Description

Stream data processing method, device, server and storage medium

Technical Field

The embodiment of the invention relates to a big data processing technology, in particular to a streaming data processing method, a streaming data processing device, a server and a storage medium.

Background

With the development of big data, the Flink real-time stream computing architecture is widely used for processing stream data. The server serves as a production party to provide data push service for consumers. Code editing of the stream data processing logic is typically required according to the needs of the consumer. When facing a plurality of consumers, code editing of data processing logic needs to be performed respectively, but the code editing needs manual editing, and the scalability is poor.

Disclosure of Invention

The invention provides a stream data processing method, a stream data processing device, a server and a storage medium, which aim to improve the expandability of stream data processing.

In a first aspect, an embodiment of the present invention provides a stream data processing method, including:

acquiring stream data of a preset theme;

acquiring signing information of a consumer according to target data in the streaming data;

determining a data processing template according to the subscription information;

processing the target data according to the data processing template to obtain result data;

and sending the result data to a consumer.

In a second aspect, an embodiment of the present invention further provides a stream data processing apparatus, including:

the stream data acquisition module is used for acquiring stream data of a preset theme;

the contract information acquisition module is used for acquiring contract information of a consumer according to target data in the streaming data;

the template determining module is used for determining a data processing template according to the subscription information;

the target data processing module is used for processing the target data according to the data processing template to obtain result data;

and the sending module is used for sending the result data to a consumer.

In a third aspect, an embodiment of the present invention further provides a streaming data processing server, where the server includes:

one or more processors;

a storage device for storing one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors implement the streaming data processing method according to the embodiment of the present application.

In a fourth aspect, the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform the streaming data processing method according to the embodiments of the present application.

The embodiment of the invention provides a stream data processing method, which comprises the steps of obtaining stream data of a preset theme; acquiring signing information of a consumer according to target data in the streaming data; determining a data processing template according to the subscription information; processing the target data according to the data processing template to obtain result data; and sending the result data to a consumer. Compared with the prior art that manual code editing development needs to be performed according to the requirements of consumers, the stream data processing method provided by the embodiment of the invention can acquire the pre-configured data processing template of the signatory for the target data of the consumers with the subscription information after the stream data is pulled, and process the stream data according to the data processing template. The data processing template can be quickly applied to different users without complex manual code editing, so that the expandability of stream data processing can be improved.

Drawings

Fig. 1 is a flowchart of a stream data processing method according to a first embodiment of the present invention;

fig. 2 is a schematic structural diagram of a stream data processing apparatus according to a second embodiment of the present invention;

fig. 3 is a schematic structural diagram of another stream data processing apparatus according to a second embodiment of the present invention;

fig. 4 is a schematic structural diagram of a stream data processing server in the third embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Example one

Fig. 1 is a flowchart of a streaming data processing method according to an embodiment of the present invention, where this embodiment is applicable to a situation where data is pushed to a consumer based on a Flink framework, and the method may be executed by a server, and specifically includes the following steps:

and step 110, acquiring stream data of a preset theme.

Kafka is a distributed publish-subscribe messaging system. When a certain type of data in the business system is changed, the data can be sent to Kafka, and the Kafka arranges the data into a topic list, wherein the topic list comprises a plurality of messages under a specific topic. The Flink engine is a distributed processing engine for performing stateful computations on unbounded and bounded data streams, performing the computations at memory speed. The method and the device can be applied to a server for executing the calculation of the flink engine. Topic (Topic) is an identification in Flink for marking a class of messages, the role of sending streaming data to a particular Topic is called producer, and the role of pulling data from a particular Topic is called consumer.

The specific theme in the application can be the change of the amount of the user account or the change of the user attribute information in the user account. The application takes the business system as a consumer to pull the business data containing the theme. And the target data obtained after the processing is sent to the user equipment as the consumer as the producer. Kafka acts as a producer output message queue. The server of the Flink engine is used as a consumer and acquires the streaming data from the kafka in a pull streaming mode.

Alternatively, the stream data may be acquired from kafka in a preset data amount. For example, 500MB of stream data is acquired at a time. And after the stream data pulled this time is processed, acquiring the next batch of stream data from kafka again through the offset.

And step 120, acquiring the subscription information of the consumer according to the target data in the streaming data.

In the application, the consumer can be a personal user or a gate-level user of a department. The signing information records whether the consumer has signed a contract or not and a data processing template used by the consumer. Information of different users under the above specific subject may be included in the streaming data. The signed users have subscription information, and the users who do not have the subscription lack the subscription information. Whether the target data is subjected to subsequent processing can be judged according to the subscription information.

Further, before obtaining the subscription information of the consumer according to the target data in the streaming data, the method further includes:

and filtering the target data according to a preset filtering rule.

And filtering the acquired streaming data. The stream data is composed of a plurality of subdata, and each subdata is a piece of information under the specific theme. The target data is any one of the subdata. The preset filtering rule may be to determine whether there is a defect of missing field or error in field type in the target data. The preset filter rules may include a field type of the criteria and a field instance of the criteria. Whether the target data has the defects can be determined by comparing the example with the fields contained in the target data. And if so, filtering out the target data with the defects.

By filtering the target data, the target data with wrong contents can be prevented from being processed, and the accuracy of data processing is improved.

and step 210, judging whether the target data is repeated data according to the historical data stored in the cache.

The data pulled from Kafka may have duplicates, and handling the duplicate target data may result in duplicate pushing. In order to solve the above problem, the processed history data is stored in a cache. And if the target data is the same as the historical data, determining that the target data is repeated data. And if not, determining that the target data is non-repeated data. If the data is duplicated, step 220 is performed. If not, go to step 230.

Further, before determining whether the target data is duplicate data according to the historical data stored in the cache in step 210, the method further includes:

and updating the historical data in the cache according to a preset time length.

A preset duration may be set, and when the preset duration is reached, the historical data stored in the cache is deleted. The preset time period may be 1 hour. By setting the preset time length, the repeated data can be prevented from being sent within the preset time length.

Step 220, if the target data is the repeated data, acquiring the next target data.

And if the data is repeated data, canceling the processing of the current target data and reading the next target data. And judging whether the next target data is the repeated data.

Step 230, if the target data is non-duplicate data, obtaining the subscription information of the consumer according to the target data in the stream data.

If the data is not duplicated, step 120 is executed.

By judging whether the target data is repeated data or not, repeated result data can be prevented from being sent to a consumer for multiple times within a certain time, and the pushing efficiency is improved.

signing with a consumer to obtain signing information, wherein the signing information is used for indicating whether target data are allowed to be pushed to the consumer; and storing the subscription information through a cache.

The server signs a contract with the consumer as a producer of the result data. The subscription information includes identification information of the consumer and information on whether to sign a subscription. And storing the subscription information through a cache.

It should be noted that the above cache may be implemented using redis.

The result data can be provided for the specific consumer through signing, and the accuracy and the safety of data pushing are improved.

And step 130, determining a data processing template according to the subscription information.

The signing information also comprises data processing template identification information which is pre-configured by the consumer, and the data processing template can be found according to the processing template identification information. The data processing template comprises. The data processing template can be an xml format document, wherein a plurality of ordered data processing modes are recorded.

Before determining the data processing template according to the subscription information, the method further comprises the following steps:

and determining a data processing template according to the requirements of the consumers, wherein the data processing template is used for recording a plurality of sequentially executed data processing modes.

Illustratively, the data processing method includes:

data transcoding, data encryption, data desensitization, data interception, numerical calculation, primary key alias mapping or push content selection.

And providing a plurality of selectable data processing modes for a user through a visual interface. The data processing mode is an edited code segment. The consumer can select a plurality of data processing modes in the visual interface according to the self requirement, and the execution sequence of the data processing modes is set. And obtaining a data processing template according to the execution sequence and the data processing mode set by the user, realizing the processing mode of rapidly configuring the target data according to the user requirements, and improving the usability.

And 140, processing the target data according to the data processing template to obtain result data.

And processing the target data according to the data processing mode recorded by the data processing template to obtain result data.

Step 150, the result data is sent to the consumer.

And after the result data is obtained, pushing the result information to a specific theme of the consumer.

Example two

Fig. 2 is a schematic structural diagram of a stream data processing apparatus according to a second embodiment of the present invention, where the present embodiment is applicable to a case of pushing data to a consumer, and the method may be executed by a server, and specifically includes: a stream data acquisition module 210, a subscription information acquisition module 220, a template determination module 230, a target data processing module 240, and a transmission module 250.

A stream data obtaining module 210, configured to obtain stream data of a preset topic;

the contract information acquisition module 220 is configured to acquire contract information of a consumer according to target data in the streaming data;

a template determining module 230, configured to determine a data processing template according to the subscription information;

a target data processing module 240, configured to process the target data according to the data processing template to obtain result data;

a sending module 250, configured to send the result data to the consumer.

On the basis of the above embodiment, as shown in fig. 3, a filtering module 310 is further included. The filtering module 310 is used for:

and filtering the target data according to a preset filtering rule.

On the basis of the above embodiment, the system further comprises a deduplication module 320. The deduplication module 320 is configured to:

judging whether the target data is repeated data or not according to historical data stored in the cache;

if the target data is repeated data, acquiring the next target data;

and if the target data is non-repeated data, acquiring the subscription information of the consumer according to the target data in the stream data.

On the basis of the above embodiment, the deduplication module 320 is further configured to:

On the basis of the above embodiment, the system further comprises a subscription module 330. The subscription module 330 is configured to:

signing with a consumer to obtain signing information, wherein the signing information is used for indicating whether target data are allowed to be pushed to the consumer;

and storing the subscription information through a cache.

On the basis of the above embodiment, the template editing module 340 is further included. The template editing module 340 is used for:

On the basis of the above embodiment, the data processing method includes:

In the stream data processing apparatus provided in the embodiment of the present invention, the stream data obtaining module 210 obtains stream data of a preset topic; the subscription information acquisition module 220 acquires subscription information of a consumer according to target data in the streaming data; the template determining module 230 determines a data processing template according to the subscription information; the target data processing module 240 processes the target data according to the data processing template to obtain result data; the sending module 250 sends the result data to the consumer. Compared with the current method that manual code editing and development are required to be carried out according to the requirements of consumers, the stream data processing device provided by the embodiment of the invention can acquire the pre-configured data processing template of the signatory for the target data of the consumers with the signing information after the stream data is pulled, and process the stream data according to the data processing template. The data processing template can be quickly applied to different users without complex manual code editing, so that the expandability of stream data processing can be improved.

The stream data processing device provided by the embodiment of the invention can execute the stream data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

EXAMPLE III

Fig. 4 is a schematic structural diagram of a stream data processing server according to a third embodiment of the present invention, and as shown in fig. 4, the stream data processing server includes a processor 40, a memory 41, an input device 42, and an output device 43; the number of the processors 40 in the stream data processing server may be one or more, and one processor 40 is taken as an example in fig. 4; the processor 40, the memory 41, the input device 42, and the output device 43 in the stream data processing server may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 4.

The memory 41, which is a computer-readable storage medium, may be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the streaming data processing method in the embodiment of the present invention (for example, the streaming data acquisition module 210, the subscription information acquisition module 220, the template determination module 230, the target data processing module 240, the transmission module 250, the filtering module 310, the deduplication module 320, the subscription module 330, and the template editing module 340 in the streaming data processing apparatus). The processor 40 executes various functional applications of the streaming data processing server and data processing by running software programs, instructions, and modules stored in the memory 41, that is, implements the streaming data processing method described above.

The memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 41 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 41 may further include a memory remotely located from the processor 40, and these remote memories may be connected to the streaming data processing server through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 42 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the stream data processing server. The output device 43 may include a display device such as a display screen.

Example four

An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a streaming data processing method, including:

acquiring stream data of a preset theme;

and sending the result data to a consumer.

On the basis of the above embodiment, before acquiring the subscription information of the consumer according to the target data in the streaming data, the method further includes:

and filtering the target data according to a preset filtering rule.

if the target data is repeated data, acquiring the next target data;

On the basis of the above embodiment, before determining whether the target data is duplicate data according to the history data stored in the cache, the method further includes:

and storing the subscription information through a cache.

On the basis of the above embodiment, before determining the data processing template according to the subscription information, the method further includes:

On the basis of the above embodiment, the data processing method includes:

Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the stream data processing method provided by any embodiment of the present invention.

From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.

It should be noted that, in the embodiment of the above search apparatus, each included unit and module are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A stream data processing method, characterized by comprising:

acquiring stream data of a preset theme;

and sending the result data to a consumer.

2. The method of claim 1, further comprising, prior to obtaining the subscription information of the consumer based on the target data in the streaming data:

and filtering the target data according to a preset filtering rule.

3. The method of claim 1, further comprising, prior to obtaining the subscription information of the consumer based on the target data in the streaming data:

if the target data is repeated data, acquiring the next target data;

4. The method according to claim 3, before determining whether the target data is duplicated data according to the history data stored in the cache, further comprising:

5. The method of claim 1, further comprising, prior to obtaining the subscription information of the consumer based on the target data in the streaming data:

and storing the subscription information through a cache.

6. The method of claim 1, wherein determining a data processing template based on the subscription information further comprises:

7. The method of claim 6, wherein the data processing means comprises:

8. A stream data processing apparatus characterized by comprising:

and the sending module is used for sending the result data to a consumer.

9. A streaming data processing server, characterized in that the server comprises:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the streaming data processing method of any one of claims 1-7.

10. A storage medium containing computer-executable instructions for performing the streaming data processing method of any one of claims 1-7 when executed by a computer processor.