CN112612621A - Data processing method and related equipment - Google Patents

Data processing method and related equipment Download PDF

Info

Publication number
CN112612621A
CN112612621A CN202011484880.6A CN202011484880A CN112612621A CN 112612621 A CN112612621 A CN 112612621A CN 202011484880 A CN202011484880 A CN 202011484880A CN 112612621 A CN112612621 A CN 112612621A
Authority
CN
China
Prior art keywords
streaming data
data
event
target
parameters
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011484880.6A
Other languages
Chinese (zh)
Inventor
付雷
郝云峰
彭展
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
JD Digital Technology Holdings Co Ltd
Original Assignee
JD Digital Technology Holdings Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by JD Digital Technology Holdings Co Ltd filed Critical JD Digital Technology Holdings Co Ltd
Priority to CN202011484880.6A priority Critical patent/CN112612621A/en
Publication of CN112612621A publication Critical patent/CN112612621A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/542—Event management; Broadcasting; Multicasting; Notifications
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The disclosure provides a data processing method, a system, message middleware and a non-transitory computer readable storage medium, and relates to the technical field of computers. The data processing method comprises the following steps: the message middleware receives target streaming data; the message middleware judges whether the target streaming data corresponds to an event in a preset event chain according to data parameters carried by the target streaming data; under the condition that the target streaming data corresponds to the event, if the target event corresponding to the target streaming data is the 1 st event in a preset event chain, the message middleware stores the target streaming data; if the target event is the ith event in the preset event chain, the message middleware judges whether historical streaming data corresponding to the (i-1) th event in the preset event chain is stored or not, if the historical streaming data is stored, the message middleware stores the target streaming data, and i is a positive integer greater than 1. The storage efficiency of the streaming data can be improved, and therefore timeliness of streaming calculation is improved.

Description

Data processing method and related equipment
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, a data processing system, a message middleware, and a non-transitory computer-readable storage medium.
Background
In the face of the risk control problem, it is usually necessary to adopt a streaming computing technique to count whether a certain event occurs within a certain time or to count whether other pre-events occur within a certain time before the certain event occurs in real time. For example, when a user account transfer event occurs, a streaming computing technology is needed to count whether an event of logging in an off-site APP, a password modification event and a transfer amount increase event occur within 1 hour before the event occurs. If the pre-events occur, the risk that the number of the user is stolen exists at the moment, and risk control measures need to be taken, such as intercepting a transfer request, re-authenticating the identity of the user and the like.
In the streaming computing technology, real-time complex computation and statistics on large-scale streaming data are required. In conventional streaming computing techniques, as events occur, all streaming data is stored in its entirety in a relational or non-relational database. When a new streaming data table name is received and a preset event occurs, a more complex structured query language is written in a database to query whether historical streaming data corresponding to a preposed event exists in the database. And if the historical streaming data corresponding to the preset event exists, representing the preset event which occurs before, and taking risk control measures.
Disclosure of Invention
One technical problem solved by the present disclosure is how to improve timeliness of streaming computation.
According to a first aspect of the present disclosure, there is provided a data processing method including: the message middleware receives target streaming data; the message middleware judges whether the target streaming data corresponds to an event in a preset event chain according to data parameters carried by the target streaming data; under the condition that the target streaming data corresponds to the event, if the target event corresponding to the target streaming data is the 1 st event in a preset event chain, the message middleware stores the target streaming data; if the target event is the ith event in the preset event chain, the message middleware judges whether historical streaming data corresponding to the (i-1) th event in the preset event chain is stored or not, if the historical streaming data is stored, the message middleware stores the target streaming data, and i is a positive integer greater than 1.
In some embodiments, the data processing method further comprises: and under the condition that the target streaming data corresponds to the event, if the target event is the ith event in the preset event chain and the historical streaming data is not stored, the message middleware discards the target streaming data.
In some embodiments, the data processing method further comprises: in the event that the target streaming data does not correspond to an event, the message middleware discards the target streaming data.
In some embodiments, the data parameters include behavioral parameters and user parameters; judging whether the target streaming data corresponds to an event in a preset event chain according to data parameters carried by the target streaming data comprises the following steps: judging whether the behavior parameters and the user parameters meet the parameter conditions of the event or not; if the behavior parameters and the user parameters meet the parameter conditions of the events, the target streaming data corresponds to the events in the preset event chain; and if the behavior parameters and the user parameters do not meet the parameter conditions of the events, the target streaming data does not correspond to the events in the preset event chain.
In some embodiments, the historical streaming data carries historical data parameters, the data parameters and the historical data parameters including a time parameter; the message middleware judges whether historical streaming data corresponding to the i-1 th event in a preset event chain is stored, and if the historical streaming data is stored, the message middleware stores target streaming data and comprises the following steps: the message middleware judges whether historical streaming data is stored or not; if the historical streaming data is stored, the message middleware judges whether the difference between the historical data parameters and the time parameters of the data parameters is smaller than a preset value; and if the difference of the time parameters is smaller than a preset value, the message middleware stores the target streaming data.
In some embodiments, the data processing method further comprises: after receiving the target streaming data, the message middleware judges whether the target streaming data carries data parameters; under the condition that the target streaming data does not carry data parameters, the message middleware discards the target streaming data; and under the condition that the target streaming data carries the data parameters, the message middleware judges whether the target streaming data corresponds to the events in the preset event chain according to the data parameters carried by the target streaming data.
In some embodiments, the data processing method further comprises: the message middleware receives an event query request, wherein the event query request carries query user parameters and a query time window; the message middleware queries streaming data associated with the query user parameter and the query time window from the stored streaming data, wherein the data parameters carried by the associated streaming data comprise the user parameter associated with the query user parameter and the time parameter positioned in the query time window; the message middleware sends the associated streaming data.
In some embodiments, the message middleware receives target streaming data sent by the message sending device; the data processing method further comprises: after discarding the target loss data or storing the target stream data, the message middleware sends a processing completion response to the message sending device; the message middleware receives the target streaming data from the message transmitting apparatus again in a case where the message transmitting apparatus does not receive the processing completion response.
In some embodiments, the data processing method further comprises: the message middleware receives a pull request of a message receiving device for the streaming data to be processed; and the message middleware receives the pull request of the message receiving device for the streaming data to be processed again after the message receiving device fails to process the streaming data to be processed.
In some embodiments, the data processing method further comprises: the message middleware records the repetition times of receiving the pulling request; and the message middleware stores the streaming data to be processed into the data processing exception queue under the condition that the repetition times are greater than a preset value.
According to a second aspect of the present disclosure, there is provided a message middleware comprising: a data receiving module configured to receive target streaming data; the event judging module is configured to judge whether the target streaming data corresponds to an event in a preset event chain according to data parameters carried by the target streaming data; a data processing module configured to: under the condition that the target streaming data corresponds to the event, if the target event corresponding to the target streaming data is the 1 st event in the preset event chain, storing the target streaming data; if the target event is the ith event in the preset event chain, judging whether historical streaming data corresponding to the (i-1) th event in the preset event chain is stored or not, and if the historical streaming data is stored, storing the target streaming data, wherein i is a positive integer greater than 1.
In some embodiments, the data processing module is further configured to: and under the condition that the target streaming data corresponds to the event, if the target event is the ith event in the preset event chain and the historical streaming data is not stored, discarding the target streaming data.
In some embodiments, the data processing module is further configured to: and in the case that the target streaming data does not correspond to the event, discarding the target streaming data.
In some embodiments, the data parameters include behavioral parameters and user parameters; the event determination module is configured to: judging whether the behavior parameters and the user parameters meet the parameter conditions of the event or not; if the behavior parameters and the user parameters meet the parameter conditions of the events, the target streaming data corresponds to the events in the preset event chain; and if the behavior parameters and the user parameters do not meet the parameter conditions of the events, the target streaming data does not correspond to the events in the preset event chain.
In some embodiments, the historical streaming data carries historical data parameters, the data parameters and the historical data parameters including a time parameter; the data processing module is configured to: judging whether historical streaming data corresponding to the (i-1) th event in a preset event chain is stored or not; if the historical streaming data is stored, judging whether the difference between the historical data parameters and the time parameters of the data parameters is smaller than a preset value; and if the difference of the time parameters is smaller than a preset value, storing the target streaming data.
In some embodiments, the message middleware further comprises a parameter determination module configured to: after receiving the target streaming data, judging whether the target streaming data carries data parameters; the data processing module is further configured to: under the condition that the target streaming data does not carry data parameters, discarding the target streaming data; the event determination module is configured to: and under the condition that the target streaming data carries data parameters, judging whether the target streaming data corresponds to an event in a preset event chain or not according to the data parameters carried by the target streaming data.
In some embodiments, the message middleware further comprises: the query request receiving module is configured to receive an event query request, and the event query request carries query user parameters and a query time window; the data query module is configured to query streaming data associated with query user parameters and a query time window from the stored streaming data, wherein the data parameters carried by the associated streaming data comprise the user parameters associated with the query user parameters and time parameters located in the query time window; a data transmission module configured to transmit the associated streaming data.
In some embodiments, the data receiving module is configured to receive target streaming data transmitted by the message transmitting apparatus; the message middleware also comprises a response sending module which is configured to send a processing completion response to the message sending device after the target loss data is discarded or the target streaming data is stored; the data receiving module is further configured to: in a case where the message transmitting apparatus does not receive the processing completion response, the target streaming data is received again from the message transmitting apparatus.
In some embodiments, the message middleware further comprises a pull request receiving module configured to: receiving a pull request of a message receiving device for the streaming data to be processed; and after the message receiving device fails to process the streaming data to be processed, re-receiving a pull request of the message receiving device for the streaming data to be processed.
In some embodiments, the message middleware further comprises: a number recording module configured to record a number of repetitions of receiving a pull request; and the exception handling module is configured to store the streaming data to be processed into the data processing exception queue under the condition that the repetition times are greater than a preset value.
According to a third aspect of the present disclosure, there is provided another message middleware, comprising: a memory; and a processor coupled to the memory, the processor configured to perform the aforementioned data processing method based on instructions stored in the memory.
According to a fourth aspect of the present disclosure, there is provided a data processing system comprising: the message middleware of (1); a message sending device configured to: sending the target streaming data to the message middleware; if the processing completion response sent by the message middleware is not received, the target streaming data is sent to the message middleware again; a message receiving device configured to: sending a pull request for the streaming data to be processed to message middleware; under the condition that the streaming data to be processed fails, sending a pull request for the streaming data to be processed to the message middleware again; an event querying device configured to: sending an event query request to the message middleware, wherein the event query request carries query user parameters and a query time window; and receiving streaming data sent by the message middleware.
According to a fifth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer instructions which, when executed by a processor, implement the aforementioned data processing method.
The storage efficiency of the streaming data can be improved, and therefore timeliness of streaming calculation is improved.
Other features of the present disclosure and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or technical solutions in the related art, the drawings required to be used in the description of the embodiments or the related art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings may be obtained according to the drawings without inventive exercise.
Fig. 1 shows a flow diagram of a data processing method of some embodiments of the present disclosure.
Fig. 2 shows a flow diagram of a data processing method according to further embodiments of the present disclosure.
Fig. 3 shows a flow diagram of a data processing method of further embodiments of the present disclosure.
Fig. 4 illustrates a structural schematic diagram of message middleware of some embodiments of the present disclosure.
Fig. 5 shows a schematic structural diagram of message middleware of further embodiments of the present disclosure.
FIG. 6 shows a block diagram of a data processing system according to some embodiments of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
The inventors have found that conventional streaming computing imposes the requirement to store all streaming data. With the increase of the system running time, when the query is performed in a large amount of streaming data, more query time is consumed, and the timeliness of streaming computation is reduced. In addition, in the process of storing all streaming data, the workload of the storage control is increased, and the storage efficiency is reduced. In view of this, the present disclosure provides a data processing method, which can improve storage efficiency of streaming data, thereby improving timeliness of streaming computation.
Some embodiments of the data processing method of the present disclosure are first described with reference to fig. 1 to introduce a storage process of streaming data.
Fig. 1 shows a flow diagram of a data processing method of some embodiments of the present disclosure. As shown in fig. 1, the method includes steps S101 to S106.
In step S101, the message middleware receives target streaming data.
For example, the message middleware receives the target streaming data transmitted by the message transmitting apparatus in real time. The target streaming data may carry data parameters.
In step S103, the message middleware determines whether the target streaming data corresponds to an event in a preset event chain according to the data parameter carried by the target streaming data.
The preset event chain comprises a plurality of events which are continuous front and back. For example, the preset event chain may be "event a (zhangsan login)" > "event B (zhangsan modified password)" > "event C (zhangsan initiated transfer)". The preset event chain can be configured in advance according to the occurrence sequence of the continuous events which need to be counted. Each event in the preset event chain includes certain parameter conditions, the parameter conditions may include, for example, behavior parameter conditions and user parameter conditions, and the parameter conditions of each event may also be configured in advance as needed.
The data parameters may include a behavior parameter and a user parameter, for example, behavior parameter action 1 indicates login behavior, and user parameter user 3 indicates user Zhang III. And the message middleware judges whether the behavior parameters and the user parameters meet the parameter conditions of the events in the preset event chain.
And if the behavior parameters and the user parameters meet the parameter conditions of the events, the target streaming data corresponds to the events in the preset event chain. For example, action 1 and user 3 satisfy the parameter condition of the event a in the preset event chain. In this case, step S104 to step S105 are executed;
and if the behavior parameters and the user parameters do not meet the parameter conditions of the events, the target streaming data does not correspond to the events in the preset event chain. For example, if action 1 and user 100 do not satisfy the parameter condition of any event in the preset event chain, the target streaming data does not correspond to the event in the preset event chain. In this case, step S106 is executed.
In step S104, it is determined that the target event corresponding to the target streaming data is an event sequence number in a preset event chain.
In step S105, the message middleware determines whether to discard the target data or to store the target data according to the event sequence number.
And if the target event corresponding to the stream marked data is the 1 st event in the preset event chain, the message middleware stores the target stream data. For example, the preset event chain includes A, B, C three events, and if the target event is an a event, the message middleware may directly store the target streaming data in the Redis cache. One skilled in the art will appreciate that the target streaming data may also be stored in a file system. However, storing the target streaming data in the cache can significantly reduce storage latency as compared to storing the target streaming data in the file system.
If the target event is the ith event (i is a positive integer greater than 1) in the preset event chain, the message middleware judges whether historical streaming data corresponding to the (i-1) th event in the preset event chain is stored. If the historical streaming data corresponding to the (i-1) th event is stored, the message middleware stores the target streaming data; and if the historical streaming data corresponding to the (i-1) th event is not stored, the message middleware discards the target streaming data. For example, A, B, C events are included in the preset event chain. If the target event is the event B, the message middleware inquires whether historical streaming data corresponding to the event A is stored in a Redis cache or not, and if the historical streaming data corresponding to the event A is stored, the message middleware continues to store the target streaming data corresponding to the event B; and if the historical streaming data corresponding to the event A is not stored, discarding the target streaming data corresponding to the event B. Similarly, if the target event is a C event, the message middleware queries whether historical streaming data corresponding to the event B is already stored in the Redis cache, and if the historical streaming data corresponding to the event B is already stored, the message middleware continues to store the target streaming data corresponding to the event C; and if the historical streaming data corresponding to the event B is not stored, discarding the target streaming data corresponding to the event A.
In step S106, the message middleware discards the target streaming data.
In the above embodiment, the full amount of streaming data is not stored in the streaming calculation process, but before the target streaming data is stored, whether the target streaming data corresponds to an event in the preset event chain is detected, whether a pre-event in the preset event chain occurs is further detected, and then whether the target streaming data is stored or discarded is determined according to the detection result. Therefore, the embodiment can avoid storing a large amount of useless data, save the data storage space, reduce the data storage load and improve the storage efficiency of the streaming data. Just because the data amount in the data storage space is reduced, the time required for query operation and statistical operation according to the stored streaming data is reduced, so the embodiment can improve the timeliness and the working efficiency of streaming calculation.
In some embodiments, the historical streaming data corresponding to the i-1 th event carries historical data parameters, and both the data parameters and the historical data parameters further include time parameters, and the time parameters represent events in which corresponding events actually occur. Accordingly, in step S105, the message middleware first determines whether the historical streaming data corresponding to the i-1 st event has been stored. If the historical streaming data corresponding to the i-1 th event is stored, the message middleware can further judge whether the difference between the historical data parameter and the time parameter of the data parameter is smaller than a preset value. If the difference of the time parameters is smaller than a preset value, the message middleware stores target streaming data; and if the difference of the time parameters is not less than the preset value, the message middleware discards the target streaming data.
It is still exemplified that the preset event chain includes A, B, C events. The time parameter time of the data parameters carried by the target streaming data being 10:00:00 indicates that the transfer action occurs at 10:00:00 of zhang san, the target streaming data corresponds to the event C, and the historical streaming data corresponding to the event B is stored. If the time parameter time in the historical data parameter carried in the historical streaming data corresponding to the event B is 9:30:00, indicating that the password modification behavior occurs in Zhang III at 9:30:00, and the difference between the time parameters is smaller than a preset value (1 hour), and storing the target streaming data; if the time parameter time in the historical data parameter carried in the historical streaming data corresponding to the event B is 8:30:00, indicating that the password modification behavior occurs in zhang san at 8:30:00, and the difference between the time parameters is not less than the preset value (1 hour), the target streaming data is discarded.
By adding the time parameter, the interval between the continuously occurring events can be considered, the target streaming data is stored only when the continuous events in the preset event chain occur in a short time, the data storage space can be further saved, and the data storage load can be reduced, so that the storage efficiency of the streaming data and the timeliness of streaming calculation are further improved.
In some embodiments, the data processing method further comprises step S102.
In step S102, after the message middleware receives the target streaming data, it is determined whether the target streaming data carries data parameters.
In case the target streaming data does not carry data parameters, step S106 is directly performed. In case the target streaming data carries data parameters, step S103 is performed. For example, if the target streaming data does not carry the behavior parameters and the user parameters, the target streaming data is directly discarded.
Those skilled in the art can also set the relevant condition for directly discarding the target streaming data according to the condition that the target streaming data carries other parameters, which is not listed in detail herein. Target streaming data which does not carry data parameters is discarded in advance, so that the data storage space can be further saved, the data storage load can be reduced, and the storage efficiency of the streaming data and the timeliness of streaming calculation can be further improved.
Further embodiments of the disclosed data processing method are described below in conjunction with fig. 2 to introduce a query process for streaming data.
Fig. 2 shows a flow diagram of a data processing method according to further embodiments of the present disclosure. As shown in fig. 2, the method includes steps S201 to S203.
In step S201, the message middleware receives an event query request, where the event query request carries a query user parameter and a query time window.
In step S202, the message middleware queries streaming data associated with the query user parameter and the query time window from the stored streaming data.
Wherein the data parameters carried by the associated streaming data include a user parameter associated with the query user parameter and a time parameter within the query time window. For example, the event query request carries a query user parameter of "zhang san" and a query time window of 9:00:00 to 10:00: 00. And the message middleware inquires the streaming data of the user Zhang III between 9:00:00 and 10:00:00 from the stored streaming data.
In step S203, the message middleware transmits the associated streaming data.
The embodiment realizes the purpose of inquiring whether the continuous events in the preset event chain occur in the time window. In the process of storing the streaming data, the streaming data is stored only under the condition that the continuous events in the preset event chain occur, so that in the process of inquiring the streaming data, the inquiring operation does not need to be carried out from the full amount of streaming data, and the inquiring speed and the inquiring efficiency are greatly improved.
The inventor further finds that the message middleware transmits the streaming data to the message middleware by the message transmitting device and transmits the streaming data to the message receiving device by the message middleware in a process of cooperating with the upstream and downstream devices of the streaming computation. In this procedure, there may be a case where the message middleware fails to receive the streaming data from the message transmission apparatus, a case where the message middleware fails to process the streaming data, or a case where the message reception apparatus fails to process the streaming data. Considering that the message middleware does not store the full amount of streaming data and needs to perform real-time calculation processing every time the streaming data is received, it is necessary to ensure that the streaming data is properly transmitted and processed.
Further embodiments of the disclosed data processing method are described below with reference to fig. 3 to introduce a process in which message middleware cooperates with upstream and downstream devices of streaming computation.
Fig. 3 shows a flow diagram of a data processing method of further embodiments of the present disclosure. As shown in fig. 3, the method includes steps S301 to S306.
(1) Collaboration of message middleware with message sending device
In step S301, after discarding the target churn data or storing the target streaming data, the message middleware sends a processing completion response to the message sending apparatus.
In step S302, the message middleware receives the target streaming data from the message transmission apparatus again in a case where the message transmission apparatus does not receive the processing completion response.
If the message transmission apparatus does not receive the processing completion response, the message middleware fails to normally store the target streaming data or to normally receive the target streaming data. At this time, the message transmission apparatus retransmits the target streaming data.
(2) Collaboration of message middleware with message receiving device
In step S303, the message middleware receives a pull request for the pending streaming data by the message receiving apparatus.
Those skilled in the art will appreciate that message receiving devices are continually pulling pending streaming data from the message middleware for data processing. If the message receiving device succeeds in processing the to-be-processed streaming data, the message middleware is informed that the to-be-processed streaming data is successfully consumed after the streaming data processing fails, and can mark the to-be-processed streaming data as consumed. If the message receiving device fails to process the pending streaming data, the message middleware is notified that the pending streaming data has not been successfully consumed after the failure to process the streaming data is successful, and the message middleware can mark the pending streaming data as not consumed.
In step S304, after the message receiving apparatus fails to process the to-be-processed streaming data, the message middleware receives a pull request of the message receiving apparatus for the to-be-processed streaming data again.
As will be understood by those skilled in the art, each time the message receiving apparatus redraws the pending streaming data, the number of repetitions of the pull request of the pending streaming data is incremented by one.
In step S305, the message middleware records the number of repetitions of receiving the pull request.
For example, the message middleware may record the number of repetitions of the pull request for each pending streaming data, respectively.
In step S306, the message middleware stores the streaming data to be processed into the data processing exception queue when the number of times of repetition is greater than the preset value.
The embodiment provides a guarantee mechanism in the cooperation process of the message middleware and the upstream and downstream devices of the streaming computation, which can reduce the abnormal situations that the message middleware does not successfully receive or process streaming data, reduce the abnormal situations that the message receiving device does not successfully process the streaming data, and identify the streaming data with the abnormal situations and store an abnormal queue for human intervention.
Some embodiments of the disclosure message middleware are described below in conjunction with fig. 4.
Fig. 4 illustrates a structural schematic diagram of message middleware of some embodiments of the present disclosure. As shown in fig. 4, the message middleware 40 includes: a data receiving module 401 configured to receive target streaming data; an event determining module 403, configured to determine whether the target streaming data corresponds to an event in a preset event chain according to a data parameter carried by the target streaming data; a data processing module 404 configured to: under the condition that the target streaming data corresponds to the event, if the target event corresponding to the target streaming data is the 1 st event in the preset event chain, storing the target streaming data; if the target event is the ith event in the preset event chain, judging whether historical streaming data corresponding to the (i-1) th event in the preset event chain is stored or not, and if the historical streaming data is stored, storing the target streaming data, wherein i is a positive integer greater than 1.
In some embodiments, the data processing module 404 is further configured to: and under the condition that the target streaming data corresponds to the event, if the target event is the ith event in the preset event chain and the historical streaming data is not stored, discarding the target streaming data.
In some embodiments, the data processing module 404 is further configured to: and in the case that the target streaming data does not correspond to the event, discarding the target streaming data.
In some embodiments, the data parameters include behavioral parameters and user parameters; the event determination module 403 is configured to: judging whether the behavior parameters and the user parameters meet the parameter conditions of the event or not; if the behavior parameters and the user parameters meet the parameter conditions of the events, the target streaming data corresponds to the events in the preset event chain; and if the behavior parameters and the user parameters do not meet the parameter conditions of the events, the target streaming data does not correspond to the events in the preset event chain.
In some embodiments, the historical streaming data carries historical data parameters, the data parameters and the historical data parameters including a time parameter; the data processing module 404 is configured to: judging whether historical streaming data corresponding to the (i-1) th event in a preset event chain is stored or not; if the historical streaming data is stored, judging whether the difference between the historical data parameters and the time parameters of the data parameters is smaller than a preset value; and if the difference of the time parameters is smaller than a preset value, storing the target streaming data.
In some embodiments, message middleware 40 further includes a parameter determination module 402 configured to: after receiving the target streaming data, judging whether the target streaming data carries data parameters; the data processing module 404 is further configured to: under the condition that the target streaming data does not carry data parameters, discarding the target streaming data; the event determination module 403 is configured to: and under the condition that the target streaming data carries data parameters, judging whether the target streaming data corresponds to an event in a preset event chain or not according to the data parameters carried by the target streaming data.
In some embodiments, message middleware 40 further includes: a query request receiving module 405 configured to receive an event query request, where the event query request carries a query user parameter and a query time window; a data query module 406 configured to query streaming data associated with the query user parameter and the query time window from the stored streaming data, wherein the data parameters carried by the associated streaming data include a user parameter associated with the query user parameter and a time parameter within the query time window; a data transmission module 407 configured to transmit the associated streaming data.
In some embodiments, the data receiving module 401 is configured to receive target streaming data sent by the message sending apparatus; the message middleware 40 further includes a response sending module 408 configured to send a processing completion response to the message sending apparatus after discarding the target attrition data or storing the target streaming data; the data receiving module 401 is further configured to: in a case where the message transmitting apparatus does not receive the processing completion response, the target streaming data is received again from the message transmitting apparatus.
In some embodiments, message middleware 40 further includes a pull request receiving module 409 configured to: receiving a pull request of a message receiving device for the streaming data to be processed; and after the message receiving device fails to process the streaming data to be processed, re-receiving a pull request of the message receiving device for the streaming data to be processed.
In some embodiments, message middleware 40 further includes: a number recording module 410 configured to record the number of repetitions of receiving the pull request; and the exception handling module 411 is configured to store the streaming data to be processed into the data processing exception queue when the number of times of repetition is greater than a preset value.
Further embodiments of the disclosure message middleware are described below in conjunction with fig. 5.
Fig. 5 shows a schematic structural diagram of message middleware of further embodiments of the present disclosure. As shown in fig. 5, the message middleware 50 includes: a memory 510 and a processor 520 coupled to the memory 510, the processor 520 being configured to perform the data processing method of any of the foregoing embodiments based on instructions stored in the memory 510.
Memory 510 may include, for example, system memory, fixed non-volatile storage media, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader (Boot Loader), and other programs.
Message middleware 50 may also include an input-output interface 530, a network interface 540, a storage interface 550, and the like. These interfaces 530, 540, 550 and the connections between the memory 510 and the processor 620 may be, for example, via a bus 560. The input/output interface 530 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 540 provides a connection interface for various networking devices. The storage interface 550 provides a connection interface for external storage devices such as an SD card and a usb disk.
Some embodiments of the disclosed data processing system are described below in conjunction with FIG. 6.
FIG. 6 shows a block diagram of a data processing system according to some embodiments of the present disclosure. As shown in fig. 6, the data processing system 60 includes:
message middleware 40 or 50;
a message sending apparatus 601 configured to: sending the target streaming data to the message middleware; if the processing completion response sent by the message middleware is not received, the target streaming data is sent to the message middleware again;
a message receiving apparatus 602 configured to: sending a pull request for the streaming data to be processed to message middleware; under the condition that the streaming data to be processed fails, sending a pull request for the streaming data to be processed to the message middleware again;
an event querying device 603 configured to: sending an event query request to the message middleware, wherein the event query request carries query user parameters and a query time window; and receiving streaming data sent by the message middleware.
The present disclosure also includes a non-transitory computer readable storage medium having stored thereon computer instructions that, when executed by a processor, implement a data processing method in any of the foregoing embodiments.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only exemplary of the present disclosure and is not intended to limit the present disclosure, so that any modification, equivalent replacement, or improvement made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.

Claims (23)

1. A method of data processing, comprising:
the message middleware receives target streaming data;
the message middleware judges whether the target streaming data corresponds to an event in a preset event chain according to data parameters carried by the target streaming data;
under the condition that the target streaming data corresponds to the event, if the target event corresponding to the target streaming data is the 1 st event in a preset event chain, the message middleware stores the target streaming data; if the target event is the ith event in the preset event chain, the message middleware judges whether historical streaming data corresponding to the (i-1) th event in the preset event chain is stored or not, if the historical streaming data is stored, the message middleware stores the target streaming data, and i is a positive integer greater than 1.
2. The data processing method of claim 1, further comprising:
and under the condition that the target streaming data corresponds to the event, if the target event is the ith event in a preset event chain and the historical streaming data is not stored, the message middleware discards the target streaming data.
3. The data processing method of claim 1, further comprising:
in the event that the target streaming data does not correspond to the event, the message middleware discards the target streaming data.
4. The data processing method of claim 1, wherein the data parameters include behavioral parameters and user parameters; the step of judging whether the target streaming data corresponds to the event in the preset event chain according to the data parameter carried by the target streaming data comprises the following steps:
judging whether the behavior parameters and the user parameters meet the parameter conditions of the event or not;
if the behavior parameters and the user parameters meet the parameter conditions of the events, the target streaming data corresponds to the events in a preset event chain;
and if the behavior parameters and the user parameters do not meet the parameter conditions of the events, the target streaming data does not correspond to the events in the preset event chain.
5. The data processing method of claim 4, wherein the historical streaming data carries historical data parameters, the data parameters and the historical data parameters including time parameters; the message middleware judges whether historical streaming data corresponding to the i-1 th event in a preset event chain is stored, and if the historical streaming data is stored, the message middleware stores target streaming data, wherein the message middleware stores the target streaming data and comprises:
the message middleware judges whether the historical streaming data is stored or not;
if the historical streaming data is stored, the message middleware judges whether the difference between the historical data parameters and the time parameters of the data parameters is smaller than a preset value;
and if the difference of the time parameters is smaller than a preset value, the message middleware stores the target streaming data.
6. The data processing method of claim 1, further comprising:
after receiving the target streaming data, the message middleware judges whether the target streaming data carries the data parameters;
under the condition that the target streaming data does not carry the data parameters, the message middleware discards the target streaming data;
and under the condition that the target streaming data carries the data parameters, the message middleware judges whether the target streaming data corresponds to the events in the preset event chain according to the data parameters carried by the target streaming data.
7. The data processing method of claim 5, further comprising:
the message middleware receives an event query request, wherein the event query request carries query user parameters and a query time window;
the message middleware queries streaming data associated with the query user parameter and the query time window from stored streaming data, wherein the data parameters carried by the associated streaming data comprise the user parameter associated with the query user parameter and a time parameter positioned in the query time window;
the message middleware sends the associated streaming data.
8. The data processing method of claim 1, wherein the message middleware receives target streaming data transmitted by the message transmitting apparatus;
the data processing method further comprises:
after discarding the target loss data or storing the target stream data, the message middleware sends a processing completion response to the message sending device;
the message middleware receives the target streaming data from the message transmitting apparatus again in a case where the message transmitting apparatus does not receive the processing completion response.
9. The data processing method of claim 1, further comprising:
the message middleware receives a pull request of a message receiving device for the streaming data to be processed;
and the message middleware receives the pull request of the message receiving device for the streaming data to be processed again after the message receiving device fails to process the streaming data to be processed.
10. The data processing method of claim 9, further comprising:
the message middleware records the repetition times of receiving the pulling request;
and the message middleware stores the streaming data to be processed into a data processing exception queue under the condition that the repetition times are greater than a preset value.
11. A message middleware, comprising:
a data receiving module configured to receive target streaming data;
the event judging module is configured to judge whether the target streaming data corresponds to an event in a preset event chain according to data parameters carried by the target streaming data;
a data processing module configured to: under the condition that the target streaming data corresponds to the event, if the target event corresponding to the target streaming data is the 1 st event in the preset event chain, storing the target streaming data; if the target event is the ith event in the preset event chain, judging whether historical streaming data corresponding to the (i-1) th event in the preset event chain is stored or not, if the historical streaming data is stored, storing the target streaming data, wherein i is a positive integer greater than 1.
12. The message middleware of claim 11, wherein the data processing module is further configured to:
and under the condition that the target streaming data corresponds to the event, if the target event is the ith event in a preset event chain and the historical streaming data is not stored, discarding the target streaming data.
13. The message middleware of claim 11, wherein the data processing module is further configured to:
in the event that the target streaming data does not correspond to the event, the target streaming data is discarded.
14. The message middleware of claim 11, wherein the data parameters include behavioral parameters and user parameters; the event determination module is configured to:
judging whether the behavior parameters and the user parameters meet the parameter conditions of the event or not;
if the behavior parameters and the user parameters meet the parameter conditions of the events, the target streaming data corresponds to the events in a preset event chain;
and if the behavior parameters and the user parameters do not meet the parameter conditions of the events, the target streaming data does not correspond to the events in the preset event chain.
15. The message middleware of claim 14, wherein the historical streaming data carries historical data parameters, the data parameters and the historical data parameters including a time parameter; the data processing module is configured to:
judging whether historical streaming data corresponding to the (i-1) th event in a preset event chain is stored or not;
if the historical streaming data is stored, judging whether the difference between the historical data parameters and the time parameters of the data parameters is smaller than a preset value;
and if the difference of the time parameters is smaller than a preset value, storing the target streaming data.
16. The message middleware of claim 11, further comprising a parameter determination module configured to: after receiving the target streaming data, judging whether the target streaming data carries the data parameters;
the data processing module is further configured to: under the condition that the target streaming data does not carry the data parameters, discarding the target streaming data;
the event determination module is configured to: and under the condition that the target streaming data carries the data parameters, judging whether the target streaming data corresponds to the events in the preset event chain or not according to the data parameters carried by the target streaming data.
17. The message middleware of claim 15, further comprising:
the query request receiving module is configured to receive an event query request, and the event query request carries query user parameters and a query time window;
a data query module configured to query streaming data associated with the query user parameter and the query time window from stored streaming data, wherein data parameters carried by the associated streaming data include a user parameter associated with the query user parameter and a time parameter located in the query time window;
a data transmission module configured to transmit the associated streaming data.
18. The message middleware of claim 11,
the data receiving module is configured to receive target streaming data sent by a message sending device;
the message middleware also comprises a response sending module which is configured to send a processing completion response to the message sending device after the target loss data is discarded or the target streaming data is stored;
the data receiving module is further configured to: in a case where the message transmitting apparatus does not receive the processing completion response, the target streaming data is received again from the message transmitting apparatus.
19. The message middleware of claim 11, further comprising a pull request receiving module configured to:
receiving a pull request of a message receiving device for the streaming data to be processed;
and after the message receiving device fails to process the streaming data to be processed, re-receiving a pull request of the message receiving device for the streaming data to be processed.
20. The message middleware of claim 19, further comprising:
a number recording module configured to record a number of repetitions of receiving the pull request;
and the exception handling module is configured to store the streaming data to be processed into a data processing exception queue under the condition that the repetition times are greater than a preset value.
21. A message middleware, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the data processing method of any of claims 1 to 10 based on instructions stored in the memory.
22. A data processing system comprising:
the message middleware of any one of claims 11 to 21;
a message sending device configured to: sending the target streaming data to the message middleware; if the processing completion response sent by the message middleware is not received, the target streaming data is sent to the message middleware again;
a message receiving device configured to: sending a pull request for the streaming data to be processed to message middleware; under the condition that the streaming data to be processed fails, sending a pull request for the streaming data to be processed to the message middleware again;
an event querying device configured to: sending an event query request to the message middleware, wherein the event query request carries query user parameters and a query time window; and receiving streaming data sent by the message middleware.
23. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions which, when executed by a processor, implement the data processing method of any one of claims 1 to 10.
CN202011484880.6A 2020-12-16 2020-12-16 Data processing method and related equipment Pending CN112612621A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011484880.6A CN112612621A (en) 2020-12-16 2020-12-16 Data processing method and related equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011484880.6A CN112612621A (en) 2020-12-16 2020-12-16 Data processing method and related equipment

Publications (1)

Publication Number Publication Date
CN112612621A true CN112612621A (en) 2021-04-06

Family

ID=75239689

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011484880.6A Pending CN112612621A (en) 2020-12-16 2020-12-16 Data processing method and related equipment

Country Status (1)

Country Link
CN (1) CN112612621A (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101069315B1 (en) * 2010-07-12 2011-10-06 최철기 Method of providing event history information for social network game, system and computer-readable recording medium with program thereof
CN102571635A (en) * 2012-01-18 2012-07-11 浪潮(北京)电子信息产业有限公司 Message transmission method and equipment
US8549540B1 (en) * 2012-06-28 2013-10-01 International Business Machines Corporation Message queue transaction tracking using application activity trace data
CN108198003A (en) * 2018-01-31 2018-06-22 湖南新泰邦电子商务有限公司 A kind of data processing method, device, equipment and system
CN109344170A (en) * 2018-09-04 2019-02-15 阿里巴巴集团控股有限公司 Stream data processing method, system, electronic equipment and readable storage medium storing program for executing
CN110019001A (en) * 2017-08-17 2019-07-16 中国电信股份有限公司 Promote method, system and the monitoring module of message-oriented middleware message trace ability
CN110609852A (en) * 2019-07-16 2019-12-24 招联消费金融有限公司 Streaming data processing method and device, computer equipment and storage medium
CN111309761A (en) * 2020-02-16 2020-06-19 苏州浪潮智能科技有限公司 Method, system, equipment and medium for optimizing database middleware query
CN111836119A (en) * 2020-07-22 2020-10-27 广州华多网络科技有限公司 Live broadcast wonderful video tracking method, device, equipment and storage medium
CN111953655A (en) * 2017-02-28 2020-11-17 华为技术有限公司 Method and equipment for server to respond to request message in communication system

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101069315B1 (en) * 2010-07-12 2011-10-06 최철기 Method of providing event history information for social network game, system and computer-readable recording medium with program thereof
CN102571635A (en) * 2012-01-18 2012-07-11 浪潮(北京)电子信息产业有限公司 Message transmission method and equipment
US8549540B1 (en) * 2012-06-28 2013-10-01 International Business Machines Corporation Message queue transaction tracking using application activity trace data
CN111953655A (en) * 2017-02-28 2020-11-17 华为技术有限公司 Method and equipment for server to respond to request message in communication system
CN110019001A (en) * 2017-08-17 2019-07-16 中国电信股份有限公司 Promote method, system and the monitoring module of message-oriented middleware message trace ability
CN108198003A (en) * 2018-01-31 2018-06-22 湖南新泰邦电子商务有限公司 A kind of data processing method, device, equipment and system
CN109344170A (en) * 2018-09-04 2019-02-15 阿里巴巴集团控股有限公司 Stream data processing method, system, electronic equipment and readable storage medium storing program for executing
CN110609852A (en) * 2019-07-16 2019-12-24 招联消费金融有限公司 Streaming data processing method and device, computer equipment and storage medium
CN111309761A (en) * 2020-02-16 2020-06-19 苏州浪潮智能科技有限公司 Method, system, equipment and medium for optimizing database middleware query
CN111836119A (en) * 2020-07-22 2020-10-27 广州华多网络科技有限公司 Live broadcast wonderful video tracking method, device, equipment and storage medium

Similar Documents

Publication Publication Date Title
EP3234776B1 (en) Data stream processing language for analyzing instrumented software
KR101888029B1 (en) Method and system for monitoring virtual machine cluster
US8805999B2 (en) Administering event reporting rules in a distributed processing system
US8447851B1 (en) System for monitoring elastic cloud-based computing systems as a service
US9384114B2 (en) Group server performance correction via actions to server subset
US9077610B2 (en) Performing call stack sampling
US9389943B2 (en) Determining a number of unique incidents in a plurality of incidents for incident processing in a distributed processing system
US20170126531A1 (en) Network Aware Distributed Business Transaction Anomaly Detection
KR101835458B1 (en) Method, system and computer-readable storage medium for restarting data processing systems
US9389936B2 (en) Monitoring the responsiveness of a user interface
US10452469B2 (en) Server performance correction using remote server actions
CN107204875B (en) Data reporting link monitoring method and device, electronic equipment and storage medium
JP4506520B2 (en) Management server, message extraction method, and program
CN108540533B (en) Request answering method and device
US9015731B2 (en) Event handling system and method
CN112988433B (en) Method, apparatus and computer program product for fault management
US20160124833A1 (en) Analyzing physical machine impact on business transaction performance
CN111813573A (en) Communication method of management platform and robot software and related equipment thereof
CN109766028B (en) Touch control sub-management system and method for infrared touch screen
CN112612621A (en) Data processing method and related equipment
JP7038629B2 (en) Equipment condition monitoring device and program
KR101326451B1 (en) Monitoring method for estimating system failure with multiple failure condition and monitoring server using the same
CN111291104B (en) Method and system for transmitting data based on asynchronous response
CN113850664A (en) Data anomaly detection method and data reporting service
CN113742004A (en) Data processing method and device based on flink framework

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176

Applicant after: Jingdong Technology Holding Co.,Ltd.

Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176

Applicant before: Jingdong Digital Technology Holding Co., Ltd

CB02 Change of applicant information