CN114519517A

CN114519517A - Data processing method and device

Info

Publication number: CN114519517A
Application number: CN202210141064.8A
Authority: CN
Inventors: 胡建平
Original assignee: Beijing Jingdong Zhenshi Information Technology Co Ltd
Current assignee: Beijing Jingdong Zhenshi Information Technology Co Ltd
Priority date: 2022-02-16
Filing date: 2022-02-16
Publication date: 2022-05-20

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. One embodiment of the method comprises: in response to the acquired service data, registering a timing task corresponding to the service, and storing the service data in a first storage location; monitoring all timing tasks, and executing the timing tasks when the execution time of the timing tasks is reached; the timing task comprises the following steps: acquiring data corresponding to the timing task from the first storage position, and judging whether the service corresponding to the timing task is finished or not according to the data corresponding to the timing task; if so, deleting the service data in the first storage position, and storing the service data into a database; otherwise, deleting the service data in the first storage position, and storing the service data in the second storage position. The implementation mode can solve the problem of data delay and achieve the effect of real-time data processing.

Description

Data processing method and device

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for data processing.

Background

In the existing data processing, index detail data is generally processed by a big data processing frame and then written into a database, which relates to the judgment of the current time.

However, the above data processing method has at least the following drawbacks: due to time delay, a data processing system and a timing task system need to be developed, the number of data dependence systems is large, and the operation and maintenance difficulty is increased.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for data processing, which can reduce time delay, implement real-time data processing, and do not need to additionally develop a timing task system, thereby reducing difficulty and cost of system operation and maintenance, and improving efficiency.

To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data processing method including:

in response to the acquisition of service data, registering a timing task corresponding to the service, and storing the service data in a first storage location;

monitoring all timing tasks, and executing the timing tasks when the execution time of the timing tasks is reached; the timing task comprises the following steps: acquiring data corresponding to the timing task from the first storage position, and judging whether the service corresponding to the timing task is finished or not according to the data corresponding to the timing task; if so, deleting the data of the service in the first storage position, and storing the data of the service in a database; otherwise, deleting the data of the service in the first storage position, and storing the data of the service in a second storage position.

Optionally, the method further comprises: responding to the acquired service message of the service, and judging whether the service is finished according to the service message; if so, deleting the data of the service in the first storage position or the second storage position, and storing the data of the service in the database; otherwise, storing the service message into the first storage position.

Optionally, before registering the timing task corresponding to the service, the method further includes:

and determining the overtime time of the service according to the service data, and taking the overtime time as the execution time of the timing task.

Optionally, the method further comprises: under the condition that the service corresponding to the timing task is judged to be finished according to the data corresponding to the timing task, deleting the timing task;

and under the condition that the business corresponding to the timing task is judged to be not finished according to the data corresponding to the timing task, generating an overtime message corresponding to the business, and storing the overtime message into the database, wherein the overtime message indicates that the business is not finished due to overtime.

Optionally, after deleting the data of the service in the first storage location or the second storage location, the method further includes:

And under the condition that the service is judged to be completed according to the service message, deleting the timing task, generating a release message corresponding to the service, and storing the release message into the database, wherein the release message indicates that the service is completed.

Optionally, the service data is obtained by using a stream processing framework;

the flow processing frame is a Flink frame, and the timing task is registered based on a timer mechanism of the Flink frame; the first storage location and the second storage location are two state caches based on the stream processing framework declaration, respectively.

Optionally, after registering the timing task corresponding to the service, the method includes: and writing the timing tasks into the data stream which is the same as the service data according to the time sequence.

Another aspect of the embodiments of the present invention further provides a data processing apparatus, including:

the registration module is used for registering a timing task corresponding to the service in response to the acquired service data and storing the service data in a first storage position;

the monitoring module monitors all timing tasks and executes the timing tasks under the condition of reaching the execution time of the timing tasks;

The execution module is used for acquiring data corresponding to the timing task from the first storage position and judging whether the service corresponding to the timing task is finished or not according to the data corresponding to the timing task; if so, deleting the data of the service in the first storage position, and storing the data of the service in a database; otherwise, deleting the data of the service in the first storage position, and storing the data of the service in a second storage position.

According to another aspect of an embodiment of the present invention, there is provided an electronic apparatus including:

one or more processors;

a storage device for storing one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data processing method provided by the present invention.

According to a further aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing the method of data processing provided by the present invention.

One embodiment of the above invention has the following advantages or benefits: registering a timing task corresponding to a service when service data is acquired, monitoring all timing tasks, and executing the timing tasks when the execution time of the timing tasks is reached, wherein the timing tasks comprise: acquiring data corresponding to the timing task from the first storage location, judging whether the service corresponding to the timing task is completed or not, if so, deleting the data of the service in the first storage location, and storing the data of the service in a database; and if not, deleting the service data in the first storage position, and storing the service data in the second storage position. By registering the timing task when the service data is acquired and executing the timing task when the execution time is reached, the time delay of the time effect can be reduced, the real-time processing of the data is realized, the data processing and processing process and the timing task are integrated, the development workload is reduced, the operation and maintenance cost is reduced, and the efficiency is improved.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of a main flow of a method of data processing according to an embodiment of the invention;

FIG. 2 is a schematic diagram of a main flow of another method of data processing according to an embodiment of the invention;

FIG. 3 is a schematic diagram of the main flow of yet another method of data processing according to an embodiment of the invention;

FIG. 4 is a schematic diagram of a flow of a method of data processing according to an embodiment of the invention;

FIG. 5 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention;

FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic diagram of a main flow of a data processing method according to an embodiment of the present invention, as shown in fig. 1, the data processing method includes the following steps:

step S101: in response to the acquired service data, registering a timing task corresponding to the service, and storing the service data in a first storage location;

step S102: monitoring all timing tasks, judging whether the timing tasks reach the execution time, and if so, executing the step S103; if not, circularly executing the step S102;

step S103: executing the timing task, and acquiring data corresponding to the timing task from the first storage location;

step S104: judging whether the service corresponding to the timing task is finished according to the data corresponding to the timing task, if so, executing step S105, and if not, executing step S106;

step S105: deleting the data of the service in the first storage position, and storing the data of the service in a database;

step S106: and deleting the data of the service in the first storage position, and storing the data of the service in the second storage position.

In the embodiment of the present invention, the service may be a service with different service identifiers, for example, an order with different order numbers in an e-commerce system, or may also be a different node under the same service identifier, for example, a different node where the order with the same order number is located, such as a collecting node, a shipping node, and the like.

In the embodiment of the invention, the service data is processed by adopting a flow processing framework, the flow processing framework can be a Flink framework, and the Flink framework can be adopted to acquire data in real time and process a large data flow.

In the embodiment of the present invention, the service data can trigger registration of the timing task corresponding to the service, that is, the service data has a signal identifier for triggering registration of the timing task corresponding to the service, and when the service data is acquired, operation of registering the timing task corresponding to the service is triggered. Optionally, before registering the timing task corresponding to the service, determining timeout time of the service according to the service data, and taking the timeout time as execution time of the timing task. For example, the service data is order data, in response to acquiring the order data, a timing task corresponding to the order is registered, one or more timing tasks corresponding to the service may be registered, for example, after a user places an order, the order data is acquired, order placing time of the user may be acquired according to the order data, timeout time of one or more subsequent nodes is determined according to the order placing time and preset conditions, for example, timeout time of a node to be collected may be calculated according to the order placing time of the user and a preset time interval, the timing task corresponding to the order is registered, and execution time of the timing task is the timeout time of the node to be collected; the timeout time of the delivery node can be calculated, and the timed task with the execution time being the timeout time of the delivery node is registered.

In the embodiment of the invention, all timing tasks are monitored to judge whether the timing tasks reach the execution time, if the timing tasks do not reach the execution time, the process is circulated, and it can be understood that the service corresponding to the timing tasks is possibly finished before the execution time is reached; if the execution time is reached, the timing task is executed, data corresponding to the timing task is obtained from the first storage location, if the data of the service indicates that the service corresponding to the timing task is completed, the data of the service in the first storage location is directly deleted, meanwhile, the data of the service is stored in the database, if the data of the service is only that the service corresponding to the timing task is not completed, the data of the service in the first storage location is deleted, and the data of the service is stored in the second storage location, namely, the data of the unfinished service corresponding to the timing task reaching the execution time is stored in the second storage location.

In the embodiment of the present invention, step S101 and step S102 may be executed sequentially or asynchronously and concurrently. That is to say, the Flink framework acquires service data from the message middleware, registers a timing task corresponding to the service, stores the service data in a first storage location, and at the same time, asynchronously executes and monitors all timing tasks and judges whether the timing tasks reach execution time.

In the embodiment of the present invention, the first storage location and the second storage location may be two state (state) caches based on statements of a Flink stream processing framework, such as a Redis cache, an external storage, and the like, and may also be databases; optionally, if the first storage location is a first cache based on a Flink stream processing framework, and the second storage location is a second cache based on the Flink stream processing framework, the first cache and the second cache are respectively used for storing data of a service that is not completed before the execution of the timing task and data of a service that is not completed after the execution of the timing task, and by declaring the first cache and the second cache, the Flink can directly obtain the data corresponding to the timing task from the first cache without obtaining the data from the database, thereby reducing the access pressure of the database and improving the efficiency of data obtaining.

In the embodiment of the invention, the timing task is registered based on a timer mechanism of a Flink framework, so that the Flink executes the operation of acquiring the data corresponding to the timing task from the first storage position when the execution time of the timing task is reached. Moreover, the timing task is integrated into the Flink framework, a timing task system does not need to be additionally developed, the development complexity, the operation and maintenance difficulty and the cost are reduced, and the efficiency is improved. In addition, the timing task is hosted in the Flink framework, can be recovered when the Flink is restarted or is sent out, data recalculation cannot be caused, and partial data recalculation for ensuring data accuracy after the timing task is abnormal in the prior art is overcome.

Optionally, after registering the timing task corresponding to the service, the method includes: and writing the timing tasks into the data stream which is the same as the service data according to the time sequence. For example, when acquiring a plurality of data streams, namely data stream 1, data stream 2, data stream 3, … …, etc., consuming the plurality of data streams, acquiring service data in data stream 1, and then registering a timing task corresponding to the service according to the timeout time of the service, where the execution time of the timing task is the timeout time of the service, and writing the timing task into data stream 1 according to the execution time of the timing task in a time sequence, the Flink does not cause data coverage and read-write pressure of a database, and overcomes the data coverage and read-write pressure of the database when the conventional timing task calculates write-back.

As shown in fig. 2, in the embodiment of the present invention, the data processing method further includes:

step S201: responding to the acquired service message of the service, and judging whether the service is finished according to the service message; if yes, go to step S202; if not, executing step S203;

step S202: deleting the data of the service in the first storage position or the second storage position, and storing the data of the service in a database;

Step S203: the service message is stored in a first memory location.

In the embodiment of the invention, a service message is a message related to the service, after the service message of the service is acquired, whether the service is completed or not is judged according to the service message, if the service is completed, whether the data of the service is in a first storage position or a second storage position is judged, if the data of the service is in the first storage position, namely the service is completed before a timing task is executed, namely the service is not completed overtime, the data of the service in the first storage position is deleted, and the data of the service is stored in a database; and if the data of the service is in the second storage position, namely the service is finished after the execution time of the timing task, namely the service is finished after time out, deleting the data of the service in the second storage position, and storing the data of the service in the database. That is, after the service is completed, the first storage location and the second storage location do not store the data of the service any more.

In the embodiment of the present invention, when the flow processing framework acquires a message from the message middleware, it is first determined whether the acquired message is service data or a service message, and if the acquired message is service data, step S101 is executed, that is, a timing task corresponding to the service is registered, and the service data is stored in the first storage location; if the acquired message is a service message, step S201 is executed, that is, whether the service corresponding to the service message is completed is determined according to the service message. That is, step S101 and step S201 may be executed asynchronously. Optionally, step S201 and step S102 may be executed sequentially or asynchronously and concurrently.

As shown in fig. 3, in the embodiment of the present invention, the method further includes:

step S301: judging whether the service corresponding to the timing task is finished or not according to the data corresponding to the timing task; if yes, go to step S302; if not, executing step S303;

step S302: deleting the timed task;

step S303: generating a timeout message corresponding to the service, and storing the timeout message in a database, wherein the timeout message indicates that the service is not timed out.

That is, when the service corresponding to the timing task is judged to be completed according to the data corresponding to the timing task, the timing task is deleted, the data of the service is deleted from the first storage location, and the data of the service is stored in the database; if the business corresponding to the timing task is judged to be not completed, an overtime message corresponding to the business is generated, the overtime message is stored in a database, the data of the business is deleted from the first storage position, and the data of the business is stored in the second storage position.

In this embodiment of the present invention, when the service is determined to have been completed according to the service message, after deleting data of the service in the first storage location or the second storage location, the method further includes: and deleting the timing task, generating a release message corresponding to the service, and storing the release message into the database, wherein the release message indicates that the service is completed. And when the service message indicates that the service is finished, deleting the data of the service in the first storage position or the second storage position, deleting the timing task, releasing the timing task from the service, and issuing a release message to the database.

In the embodiment of the present invention, the database may be an elastic search (es), a clickhouse (ck), a hbase, or other databases used for data storage and big data real-time computation, such as an elastic search (a Lucene-based search server).

Fig. 4 is a flowchart illustrating a method for processing data according to an embodiment of the present invention, where a production table (e.g., ordering data, logistics distribution data, etc. in the e-commerce field) is sent to a message middleware, and a Flink framework consumes data of the production table from the message middleware, and then performs data processing in real time, where the method includes: the method comprises the steps of cleaning and converting data (removing unnecessary data and converting a data structure), conducting exception marking and data calculation logic on the data, packaging and compressing the data (compressing the data into a set format), and then storing the packaged and compressed data into a database (such as es, ck and the like), wherein two state caches, namely a first cache and a second cache, are declared according to a Redis cache and/or an external storage connected through an interface (such as an rpc protocol interface). Wherein, marking the data abnormally and calculating the logic of the data comprises: consuming a message of a data stream by a Flink frame, judging whether the message is service data or service message, registering a timing task corresponding to the service when the message is the service data, and storing the service data into a first cache (such as suspected abnormal cache); when the message is a service message of a service, judging whether the service is finished according to the service message, if so, deleting the timing task and deleting the data of the service stored in the first cache; if not, storing the service message into a database; when the timing task reaches the execution time, the timing task acquires data corresponding to the timing task from the first cache, judges whether a service corresponding to the timing task is finished, if so, deletes the data of the service in the first cache, and stores the data in a database; if not, deleting the data of the service in the first cache, storing the service data into a second cache (such as an abnormal cache), generating an overtime message corresponding to the service, and storing the overtime message into a database. And when the service message of the service is acquired and the service is judged to be completed according to the service message, deleting the data of the service in the second cache, generating a release message corresponding to the service, and storing the release message into a database.

According to the data processing method, when the service data is obtained, the timing task corresponding to the service is registered, all the timing tasks are monitored, and the timing task is executed when the execution time of the timing task is reached; and when the service message of the service is acquired, whether the service is finished is judged according to the service message, so that the data of the service is processed according to the finishing condition of the service. The embodiment of the invention registers the timing task based on a timer mechanism of a stream processing frame such as a Flink frame, executes the timing task when the execution time is reached, and solves the problem of data delay (the delay can be ignored), thereby achieving the effect of real-time processing; a timing task system does not need to be additionally developed, so that the development complexity is reduced, the operation and maintenance cost is reduced, and the working efficiency is improved; the problem of data coverage caused by the calculation and write-back of the timing task can be solved, and the reading and writing pressure of the database is reduced; in addition, the timing task can be recovered when a stream processing frame such as a Flink frame is restarted or sent, and the problem of data recalculation after the timing task is abnormal can be effectively solved.

As shown in fig. 5, another aspect of the present invention provides an apparatus 500 for data processing, including:

The registration module 501, in response to acquiring service data, registers a timing task corresponding to the service, and stores the service data in a first storage location;

a monitoring module 502 for monitoring all timing tasks and executing the timing tasks when the execution time of the timing tasks is reached;

the execution module 503 is configured to acquire data corresponding to the timing task from the first storage location, and determine whether a service corresponding to the timing task is completed according to the data corresponding to the timing task; if so, deleting the data of the service in the first storage position, and storing the data of the service in a database; otherwise, deleting the data of the service in the first storage position, and storing the data of the service in a second storage position.

In an embodiment of the present invention, the data processing apparatus 500 further includes: the acquisition module responds to the acquired service message of the service and judges whether the service is finished according to the service message; if so, deleting the data of the service in the first storage position or the second storage position, and storing the data of the service in the database; otherwise, storing the service message into the first storage position.

In this embodiment of the present invention, the registration module 501 is further configured to: and before registering the timing task corresponding to the service, determining the overtime of the service according to the service data, and taking the overtime as the execution time of the timing task.

In this embodiment of the present invention, the execution module 503 is further configured to: under the condition that the service corresponding to the timing task is judged to be finished according to the data corresponding to the timing task, deleting the timing task; and under the condition that the business corresponding to the timing task is judged to be not finished according to the data corresponding to the timing task, generating an overtime message corresponding to the business, and storing the overtime message into the database, wherein the overtime message indicates that the business is not finished due to overtime.

In this embodiment of the present invention, the execution module 503 is further configured to: after deleting the data of the service in the first storage position or the second storage position, under the condition that the service is judged to be completed according to the service message, deleting the timing task, generating a release message corresponding to the service, and storing the release message into the database, wherein the release message indicates that the service is completed.

In the embodiment of the invention, the service data is obtained by adopting a stream processing framework; the flow processing frame is a Flink frame, and the timing task is registered based on a timer mechanism of the Flink frame; the first storage location and the second storage location are two state caches based on the stream processing framework declaration, respectively.

In this embodiment of the present invention, the registration module 501 is further configured to: and after registering the timing task corresponding to the service, writing the timing task into the data stream which is the same as the service data according to the time sequence.

In another aspect, an embodiment of the present invention provides an electronic device, including: one or more processors; a storage device, configured to store one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method for data processing according to the embodiment of the present invention.

Yet another aspect of the embodiments of the present invention provides a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing a method of data processing of the embodiments of the present invention.

Fig. 6 shows an exemplary system architecture 600 of a data processing apparatus or a method of data processing to which embodiments of the invention may be applied.

As shown in fig. 6, the system architecture 600 may include

terminal devices

601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the

terminal devices

601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

601, 602, 603 to interact with the server 605 via the network 604 to receive or send messages or the like. The

terminal devices

601, 602, 603 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 605 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

601, 602, 603. The backend management server may analyze and process the received data such as the product information query request, and feed back a processing result (for example, target push information and product information — just an example) to the terminal device.

It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the data processing apparatus is generally disposed in the server 605.

It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for an implementation.

Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 701.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a registration module, a monitoring module, and an execution module. The names of these modules do not constitute a limitation to the module itself in some cases, and for example, the registration module may also be described as a "module that registers a timing task corresponding to a service and stores the service data in a first storage location in response to acquiring the service data".

As another aspect, the present invention also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not assembled into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: in response to the acquisition of the service data, registering a timing task corresponding to the service, and storing the service data in a first storage location; monitoring all timing tasks, and executing the timing tasks under the condition that the execution time of the timing tasks is reached; the timing task comprises the following steps: acquiring data corresponding to the timing task from the first storage position, and judging whether the service corresponding to the timing task is finished or not according to the data corresponding to the timing task; if so, deleting the service data in the first storage position, and storing the service data into a database; otherwise, deleting the service data in the first storage position, and storing the service data in the second storage position.

According to the technical scheme of the embodiment of the invention, the data processing method registers the timing task corresponding to the service when the service data is acquired, monitors all the timing tasks, and executes the timing task when the execution time of the timing task is reached; and when the service message of the service is acquired, whether the service is completed or not is judged according to the service message, so that the data of the service is processed according to the completion condition of the service. The embodiment of the invention registers the timing task based on a timer mechanism of a stream processing frame such as a Flink frame, executes the timing task when the execution time is reached, and solves the problem of data delay (the delay can be ignored), thereby achieving the effect of real-time processing; a timing task system does not need to be additionally developed, so that the development complexity is reduced, the operation and maintenance cost is reduced, and the working efficiency is improved; the problem of data coverage caused by the calculation and write-back of the timing task can be solved, and the reading and writing pressure of the database is reduced; in addition, the timing task can be recovered when a stream processing frame such as a Flink frame is restarted or sent, and the problem of data recalculation after the timing task is abnormal can be effectively solved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of data processing, comprising:

2. The method of claim 1, further comprising:

responding to the acquired service message of the service, and judging whether the service is finished according to the service message; if so, deleting the data of the service in the first storage position or the second storage position, and storing the data of the service in the database; otherwise, storing the service message into the first storage position.

3. The method of claim 1, further comprising, prior to registering a timing task corresponding to the service:

4. The method of claim 1, further comprising:

under the condition that the service corresponding to the timing task is judged to be finished according to the data corresponding to the timing task, deleting the timing task;

5. The method of claim 2, wherein after deleting the data of the service in the first storage location or the second storage location, further comprising:

6. The method of claim 1, wherein the service data is obtained using a stream processing framework;

7. The method of claim 6, wherein after registering the timing task corresponding to the service, comprising: and writing the timing tasks into the data stream which is the same as the service data according to the time sequence.

8. An apparatus for data processing, comprising:

The monitoring module monitors all timing tasks and executes the timing tasks under the condition that the execution time of the timing tasks is reached;

9. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.