CN109582476B - Data processing method, device and system - Google Patents
Data processing method, device and system Download PDFInfo
- Publication number
- CN109582476B CN109582476B CN201811441490.3A CN201811441490A CN109582476B CN 109582476 B CN109582476 B CN 109582476B CN 201811441490 A CN201811441490 A CN 201811441490A CN 109582476 B CN109582476 B CN 109582476B
- Authority
- CN
- China
- Prior art keywords
- event
- information
- processing
- standardization
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 211
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000010606 normalization Methods 0.000 claims description 17
- 238000013507 mapping Methods 0.000 claims description 15
- 230000004044 response Effects 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
One or more embodiments of the present disclosure disclose a data processing method, apparatus, and system, so as to implement near real-time performance of data processing, thereby improving data processing efficiency and reducing system pressure. The method comprises the following steps: receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file; generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule; and carrying out corresponding processing on the first event according to the event standardization information.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method, device, and system.
Background
Mobility management is essentially a data-driven management method, for example, in which mobility risk conditions are calculated and analyzed from complex business data, banking flowing water, financial market data, and the like, so as to reflect mobility levels and risk conditions of financial institutions. Taking financial services as an example, in general, a complete financial software IT facility is made up of more than tens, and even thousands, of systems. Such a large number of systems can generate significant amounts of heterogeneous data, such as order images, transaction information, customer information, accounting information, etc., which are the data sources for mobility management, which can pose significant challenges for mobility management.
Most conventional mobility management decision systems use database synchronization, that is, data on each system T day is synchronized to a data warehouse environment for analysis on t+n (N > =1) days, as shown in fig. 1. The fluidity data analysis method shown in fig. 1 has the following problems: firstly, because of the isomerism of data, a fluidity model needs to work on ETL (Extract-Transform-Load) of the data before an analysis decision, and the cost is high; secondly, because the data synchronization is T day data acquired on T+N days, the timeliness of the data is poor, the data can only be used for post analysis, and the front-end service cannot be finely managed and controlled; again, the synchronization of bulk data can put significant stress On the database, and even may affect the ability of the database to be online OLTP (On-Line Transaction Processing, online transaction process) when the database resources are strained.
Disclosure of Invention
An object of one or more embodiments of the present disclosure is to provide a data processing method, apparatus, and system, so as to achieve near real-time performance of data processing, thereby improving data processing efficiency and reducing system pressure.
To solve the above technical problems, one or more embodiments of the present specification are implemented as follows:
In one aspect, one or more embodiments of the present disclosure provide a data processing method, applied to a data processing system, including:
receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
and carrying out corresponding processing on the first event according to the event standardization information.
In one embodiment, the generating the event standardization information of the first event according to the event occurrence message and a preset event standardization rule includes:
acquiring a source file of the first event according to the event occurrence message;
acquiring specified information in the first event from the source file;
and combining the specified information according to a preset combination rule to obtain event standardization information of the first event.
In one embodiment, the event normalization rule comprises:
determining information which is provided in each event processed by the data processing system and has the same structure after being combined by the preset combination rule; and taking the determined information as the appointed information of the event.
In one embodiment, the specified information includes at least one of an event name, an event code, an event state, an event attribute code, an event attribute name, an event attribute value type.
In one embodiment, when the event occurrence message includes identification information of the source file, the acquiring the source file of the first event according to the event occurrence message includes:
sending an acquisition request carrying the identification information for acquiring the source file to the first equipment;
and receiving a source file of the first event sent by the first device in response to the acquisition request.
In one embodiment, the processing the first event according to the event normalization information includes:
determining a first event processing model corresponding to the first event according to the event standardization information;
determining an event processing mode corresponding to the first event processing model;
and correspondingly processing the first event according to the event processing mode.
In one embodiment, each of the event processing models includes at least one of model code, model name, model processing mode information;
Correspondingly, the determining the first event processing model corresponding to the first event according to the event standardization information includes:
and determining the first event processing model corresponding to the event standardization information of each first event according to the mapping relation between the preset event standardization information and the event processing model.
In another aspect, one or more embodiments of the present specification provide a data processing apparatus, for use in a data processing system, comprising:
the receiving module is used for receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
the generation module is used for generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
and the processing module is used for correspondingly processing the first event according to the event standardization information.
In one embodiment, the generating module includes:
the first acquisition unit is used for acquiring a source file of the first event according to the event occurrence message;
a second obtaining unit, configured to obtain, from the source file, specified information in the first event;
And the combination unit is used for combining the specified information according to a preset combination rule to obtain event standardization information of the first event.
In one embodiment, the apparatus further comprises:
the determining module is used for determining information which is provided with all events processed by the data processing system and has the same structure after being combined by the preset combination rule; and taking the determined information as the appointed information of the event.
In one embodiment, the specified information includes at least one of an event name, an event code, an event state, an event attribute code, an event attribute name, an event attribute value type.
In one embodiment, the first acquisition unit is further configured to:
when the event occurrence message comprises the identification information of the source file, sending an acquisition request carrying the identification information for acquiring the source file to the first equipment;
and receiving a source file of the first event sent by the first device in response to the acquisition request.
In one embodiment, the processing module comprises:
the first determining unit is used for determining a first event processing model corresponding to the first event according to the event standardization information;
The second determining unit is used for determining an event processing mode corresponding to the first event processing model;
and the processing unit is used for correspondingly processing the first event according to the event processing mode.
In one embodiment, each of the event processing models includes at least one of model code, model name, model processing mode information;
the first determining unit is further configured to:
and determining the first event processing model corresponding to the event standardization information of each first event according to the mapping relation between the preset event standardization information and the event processing model.
In yet another aspect, one or more embodiments of the present specification provide a data processing system comprising:
the first device is used for sending an event occurrence message to the event standardization platform, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
the event standardization platform is used for receiving event occurrence information sent by the first equipment; generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule; sending the event standardization information to an event processing model platform;
The event processing model platform is used for receiving the event standardization information sent by the event standardization platform; and carrying out corresponding processing on the first event according to the event standardization information.
In yet another aspect, one or more embodiments of the present specification provide a data processing apparatus comprising:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
the event normalization platform is used for receiving event occurrence information sent by the first equipment, wherein the event occurrence information comprises a source file of a first event and/or identification information of the source file; generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule; sending the event standardization information to an event processing model platform;
the event processing model platform is used for receiving the event standardization information sent by the event standardization platform; and carrying out corresponding processing on the first event according to the event standardization information.
In yet another aspect, embodiments of the present application provide a storage medium storing computer-executable instructions that, when executed, implement the following:
The event normalization platform is used for receiving event occurrence information sent by the first equipment, wherein the event occurrence information comprises a source file of a first event and/or identification information of the source file; generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule; sending the event standardization information to an event processing model platform;
the event processing model platform is used for receiving the event standardization information sent by the event standardization platform; and carrying out corresponding processing on the first event according to the event standardization information.
By adopting the technical scheme of one or more embodiments of the present specification, event standardization information of a first event is generated by receiving an event occurrence message sent by a first device and according to the event occurrence message and a preset event standardization rule, and then the first event is correspondingly processed according to the event standardization information. Compared with the traditional batch data processing mode, the technical scheme can process data in quasi-real time, and the batch data is not required to be synchronized into the data processing system, but is processed in an event-driven mode, so that the pressure of the data processing system is greatly reduced, the data processing efficiency is improved, and the development of front-end business is better supported.
Drawings
In order to more clearly illustrate one or more embodiments of the present specification or the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described, it being apparent that the drawings in the following description are only some of the embodiments described in one or more embodiments of the present specification, and that other drawings may be obtained from these drawings without inventive faculty for a person of ordinary skill in the art.
FIG. 1 is a schematic block diagram of a prior art data processing system;
FIG. 2 is a schematic block diagram of a data processing system in accordance with an embodiment of the present specification;
FIG. 3 is a schematic flow chart diagram of a data processing method according to an embodiment of the present disclosure;
FIG. 4 is a schematic block diagram of a data processing apparatus according to an embodiment of the present description;
fig. 5 is a schematic block diagram of a data processing apparatus according to an embodiment of the present specification.
Detailed Description
One or more embodiments of the present disclosure provide a data processing method, apparatus, and system, so as to achieve near real-time performance of data processing, thereby improving data processing efficiency and reducing system pressure.
In order to enable a person skilled in the art to better understand the technical solutions in one or more embodiments of the present specification, the technical solutions in one or more embodiments of the present specification will be clearly and completely described below with reference to the drawings in one or more embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which may be made by one of ordinary skill in the art based on one or more embodiments of the present disclosure without departing from the scope of the invention as defined by the claims.
FIG. 2 is a schematic block diagram of a data processing system in accordance with an embodiment of the present specification. As shown in fig. 2, the data processing system includes a first device 210, an event normalization platform 220, and an event processing model platform 230; wherein:
the first device 210 is configured to send an event occurrence message to the event normalization platform 220, where the event occurrence message includes a source file of the first event that occurred and/or identification information of the source file.
An event standardization platform 210, configured to receive an event occurrence message sent by the first device 210; generating event standardization information of a first event according to the event occurrence message and a preset event standardization rule; event normalization information is sent to the event processing model platform 220.
An event processing model platform 220, configured to receive event normalization information sent by the event normalization platform 210; and correspondingly processing the first event according to the event standardization information.
The data processing methods provided in the following embodiments are all applied to the data processing system shown in fig. 2.
FIG. 3 is a schematic flow chart of a data processing method according to an embodiment of the present disclosure, as shown in FIG. 3, the method including:
s302, receiving an event occurrence message sent by the first device, wherein the event occurrence message comprises a source file of the first event and/or identification information of the source file.
The first device may be any one or more devices capable of interacting with the event normalization platform, for example, in the field of financial business, the first device may be a financial system, a core system, a business system, a transaction system, a product system, or the like. The identification information of the source file may be location information of the source file in the first device, or may be information capable of identifying specific contents of the source file, such as a title, a file name, a keyword, etc. of the source file. The first event is then an event associated with the first device, such as a financial event, a core event, a business event, a product event, a transaction event, and the like.
S304, generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule.
S306, carrying out corresponding processing on the first event according to the event standardization information.
By adopting the technical scheme of one or more embodiments of the present specification, event standardization information of a first event is generated by receiving an event occurrence message sent by a first device and according to the event occurrence message and a preset event standardization rule, and then the first event is correspondingly processed according to the event standardization information. Compared with the traditional batch data processing mode, the technical scheme can process data in quasi-real time, and the batch data is not required to be synchronized into the data processing system, but is processed in an event-driven mode, so that the pressure of the data processing system is greatly reduced, the data processing efficiency is improved, and the development of front-end business is better supported.
In one embodiment, the event normalization rule includes: and determining information which is provided in each event processed by the data processing system and has the same structure after being combined by a preset combination rule, and taking the determined information as the appointed information of the event.
Wherein the events processed by the data processing system include events that have been and/or are about to be processed by the data processing system. The preset combining rule is not limited in this embodiment, that is, the preset combining rule may be any one of the achievable combining rules, for example: a rule of sequentially combining the respective information in the order of acquisition, a rule of sequentially combining the respective information in the order of information size, and the like.
Based on the above-described setting of the event normalization rule, the designation information may include at least one of an event name, an event code, an event state, an event attribute code, an event attribute name, an event attribute value, and an event attribute value type.
Tables 1 and 2 schematically show pieces of specification information, wherein table 1 is an event architecture table including event codes, event names, and event states; table 2 is an event attribute table including event codes, event attribute names, event attribute values, and event attribute value types. In tables 1 and 2, the information in the first column is the name of the specified information, and the information in the second column is the meaning corresponding to the name of the specified information in the first column.
TABLE 1
| event_code | Event code |
| event_name | Event name |
| status | Event status |
TABLE 2
| event_code | Event code |
| event_property_code | Event attribute code |
| event_property_name | Event attribute name |
| event_property_value | Event attribute value |
| event_property_value_type | Event attribute value type |
As can be seen from tables 1 and 2, between event and event attribute is 1: n, i.e., an event may correspond to a plurality of event attributes. The event code, the event name, the event state, the event attribute code and the event attribute name can be obtained from a source file of the event, and the event attribute value can be a fixed value or a specific logic, for example, logic for obtaining specified information from the source file, and the specific logic can be expressed by using an Mvel expression. The event attribute value type indicates that the event attribute value is a fixed value or a specific piece of logic.
When the event attribute value is a piece of logic expressed using an Mvel expression, the manner in which the specified information of the event is acquired from the source file can be determined by the event attribute value, and the specified information of the event can be acquired by using the determined manner. The method of using the Mvel expression to represent a segment of logic is the prior art, and will not be described herein.
Based on the description of the event standardization rule and the specified information in the above embodiment, when generating the event standardization information of the first event according to the event occurrence message and the preset event standardization rule, the source file of the first event may be first obtained according to the event occurrence message, then the specified information in the first event is obtained from the source file, and the specified information is combined according to the preset combination rule, so as to obtain the event standardization information of the first event.
In this embodiment, if the event occurrence message includes the source file of the first event, the source file in the event occurrence message may be directly obtained; if the event occurrence message includes identification information of a source file of the first event (such as location information of the source file in the first device), an acquisition request carrying the identification information for acquiring the source file may be sent to the first device, and after the first device receives the acquisition request, the first device searches for the source file of the first event according to the identification information, and sends the source file to the data processing system, where the data processing system further receives the source file of the first event sent by the first device.
In one embodiment, when the first event is processed according to the event standardization information, a first event processing model corresponding to the first event may be determined according to the event standardization information, further an event processing manner corresponding to the first event processing model may be determined, and the first event may be processed according to the determined event processing manner. Each event processing model comprises information such as model codes, model names, model processing modes and the like.
Table 3 schematically shows the model code, model name, and parameters of the model processing manner in the event processing model, wherein the first column of table 3 is the name of the parameter, and the second column is the meaning corresponding to the name of the parameter.
TABLE 3 Table 3
| model_code | Model code |
| model_name | Model name |
| model_script | Model logic |
When determining the first event processing model corresponding to the first event according to the event standardization information, the first event processing model corresponding to the event standardization information of each first event can be determined according to the mapping relation between the preset event standardization information and the event processing model.
For example, a mapping relationship between the event code and the model code is preset, and then according to the information of the event code contained in the event standardization information, an event processing model corresponding to the event standardization information can be determined.
Table 4 schematically shows the mapping relationship between the event code and the model code.
TABLE 4 Table 4
| event_code | Event code |
| model_code | Model code |
The data processing method provided by the above embodiment is described below by way of a specific embodiment.
In one embodiment, LCR (Liquidity Coverage Ratio, flow coverage) is used as an example. The LCR is a mobility management indicator, and it is assumed that a first device in the data processing system monitors that the LCR indicator has changed, and the first device sends an event occurrence message about the change of the LCR indicator.
Then, the event normalization platform receives an event occurrence message about LCR index change sent by the first device, and obtains a source file about LCR index change (hereinafter abbreviated as LCR event) according to the event occurrence message, and further obtains specified information of the LCR event from the source file, as shown in table 5 and table 6 below, wherein table 5 shows event information of the LCR event, and table 6 shows event attribute information of the LCR event.
In this embodiment, the LCR event includes two kinds of attribute information, see the third and fourth columns in Table 6. As can be seen from the event attribute value type, the event attribute value may be a logical "com.alipay.LCRDTO.value" or a fixed value of "1".
TABLE 5
| event_code | Event code | LCR |
| event_name | Event name | LCR index change event |
| status | Event status | Take effect |
TABLE 6
After the specified information in the above table 5 and table 6 is obtained, combining the specified information according to a preset specified rule, and obtaining event standardization information corresponding to the LCR event.
And then determining an event processing model corresponding to the LCR event according to the event standardization information corresponding to the LCR event.
Specifically, an event processing model corresponding to the LCR event is determined according to the mapping relationship between the event standardization information and the event processing model, and the mapping relationship between the event code and the model code of the LCR event is assumed to be shown in table 7.
TABLE 7
| event_code | Event code | LCR |
| model_code | Model code | LCR_DECISION |
Then, according to the mapping relationship shown in table 7, it can be determined that the event processing model corresponding to the LCR event is shown in table 8 below. In table 8, since the specific model processing mode is determined according to the actual application scenario, the specific model processing mode is not shown and is replaced by an ellipsis.
TABLE 8
| model_code | Model code | LCR_DECISION |
| model_name | Model name | LCR change decision |
| model_script | Model processing mode | …… |
And finally, according to a model processing mode of an event processing model corresponding to the LCR event, carrying out corresponding processing on the LCR event.
According to the embodiment, in the processing process of the mobility data, the batch data are not synchronized into the data processing system, but are processed in an event-driven manner, so that the pressure of the data processing system is greatly reduced, the timeliness, flexibility and expandability of the mobility decision are improved, the management of the mobility data is more refined, and the mobility risk is more controllable, thereby better supporting the development of front-end business.
In summary, particular embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may be advantageous.
The data processing method provided for one or more embodiments of the present disclosure above also provides a data processing device based on the same concept.
Fig. 4 is a schematic block diagram of a data processing apparatus according to an embodiment of the present specification. As shown in fig. 4, the data processing apparatus 400 is applied to a data processing system, and includes:
a receiving module 410, configured to receive an event occurrence message sent by a first device, where the event occurrence message includes a source file of a first event that occurs and/or identification information of the source file;
the generating module 420 is configured to generate event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
and the processing module 430 is configured to perform corresponding processing on the first event according to the event normalization information.
In one embodiment, the generation module 420 includes:
the first acquisition unit is used for acquiring a source file of a first event according to the event occurrence message;
a second acquiring unit configured to acquire, from a source file, specified information in a first event;
and the combination unit is used for combining the specified information according to a preset combination rule to obtain event standardization information of the first event.
In one embodiment, the apparatus 400 further comprises:
the determining module is used for determining information which is provided with all events processed by the data processing system and has the same structure after being combined by preset combination rules; the determined information is used as the appointed information of the event.
In one embodiment, the specified information includes at least one of an event name, an event code, an event state, an event attribute code, an event attribute name, an event attribute value type.
In one embodiment, the first acquisition unit is further configured to:
when the event occurrence message comprises the identification information of the source file, sending an acquisition request carrying the identification information for acquiring the source file to the first device;
a source file of a first event sent by a first device in response to an acquisition request is received.
In one embodiment, the processing module 430 includes:
the first determining unit is used for determining a first event processing model corresponding to the first event according to the event standardization information;
the second determining unit is used for determining an event processing mode corresponding to the first event processing model;
and the processing unit is used for correspondingly processing the first event according to the event processing mode.
In one embodiment, each event processing model comprises at least one item of information of model code, model name and model processing mode;
the first determining unit is further configured to:
and determining a first event processing model corresponding to the event standardization information of each first event according to the mapping relation between the preset event standardization information and the event processing models.
By adopting the device of one or more embodiments of the present disclosure, event standardization information of a first event is generated by receiving an event occurrence message sent by a first device and according to the event occurrence message and a preset event standardization rule, and then corresponding processing is performed on the first event according to the event standardization information. Compared with the traditional batch data processing mode, the technical scheme can process data in quasi-real time, and the batch data is not required to be synchronized into the data processing system, but is processed in an event-driven mode, so that the pressure of the data processing system is greatly reduced, the data processing efficiency is improved, and the development of front-end business is better supported.
It should be understood by those skilled in the art that the above data processing apparatus can be used to implement the above data processing method, and the detailed description thereof should be similar to the detailed description of the above method, so as to avoid complexity and avoid redundancy.
Based on the same considerations, one or more embodiments of the present disclosure also provide a data processing apparatus, as shown in fig. 5. The data processing apparatus may vary considerably in configuration or performance and may include one or more processors 501 and memory 502, in which memory 502 may store one or more stored applications or data. Wherein the memory 502 may be transient storage or persistent storage. The application programs stored in memory 502 may include one or more modules (not shown) each of which may include a series of computer executable instructions for use in a data processing apparatus. Still further, the processor 501 may be arranged to communicate with the memory 502 and execute a series of computer executable instructions in the memory 502 on a data processing apparatus. The data processing device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input/output interfaces 505, and one or more keyboards 506.
In particular, in this embodiment, the data processing apparatus includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the data processing apparatus, and the one or more programs configured to be executed by the one or more processors comprise instructions for:
Receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
and carrying out corresponding processing on the first event according to the event standardization information.
Optionally, the computer executable instructions, when executed, may further cause the processor to:
acquiring a source file of the first event according to the event occurrence message;
acquiring specified information in the first event from the source file;
and combining the specified information according to a preset combination rule to obtain event standardization information of the first event.
Optionally, the computer executable instructions, when executed, may further cause the processor to:
determining information which is provided in each event processed by the data processing system and has the same structure after being combined by the preset combination rule; and taking the determined information as the appointed information of the event.
Optionally, the specified information includes at least one of an event name, an event code, an event state, an event attribute code, an event attribute name, an event attribute value, and an event attribute value type.
Optionally, the computer executable instructions, when executed, may further cause the processor to:
when the event occurrence message comprises the identification information of the source file, sending an acquisition request carrying the identification information for acquiring the source file to the first equipment;
and receiving a source file of the first event sent by the first device in response to the acquisition request.
Optionally, the computer executable instructions, when executed, may further cause the processor to:
determining a first event processing model corresponding to the first event according to the event standardization information;
determining an event processing mode corresponding to the first event processing model;
and correspondingly processing the first event according to the event processing mode.
Optionally, each event processing model includes at least one item of information of a model code, a model name and a model processing mode;
accordingly, the computer-executable instructions, when executed, may further cause the processor to:
and determining the first event processing model corresponding to the event standardization information of each first event according to the mapping relation between the preset event standardization information and the event processing model.
One or more embodiments of the present specification also provide a computer-readable storage medium storing one or more programs, the one or more programs including instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform the above-described data processing method, and in particular to perform:
receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
and carrying out corresponding processing on the first event according to the event standardization information.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing one or more embodiments of the present description.
One skilled in the art will appreciate that one or more embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
One or more embodiments of the present specification are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
One or more embodiments of the present specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is merely one or more embodiments of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to one or more embodiments of this description will be apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of one or more embodiments of the present disclosure, are intended to be included within the scope of the claims of one or more embodiments of the present disclosure.
Claims (13)
1. A data processing method applied to a data processing system, the method comprising:
receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
carrying out corresponding processing on the first event according to the event standardization information;
the processing the first event according to the event standardization information includes:
determining a first event processing model corresponding to the event standardization information of the first event according to a mapping relation between an event code contained in preset event standardization information and a model code in an event processing model; the event processing model also comprises a model processing mode which is determined according to the actual application scene;
determining an event processing mode corresponding to the first event processing model;
and according to the event processing mode, correspondingly processing the first event in an event-driven mode.
2. The method of claim 1, wherein generating the event normalization information of the first event according to the event occurrence message and a preset event normalization rule comprises:
acquiring a source file of the first event according to the event occurrence message;
acquiring specified information in the first event from the source file;
and combining the specified information according to a preset combination rule to obtain event standardization information of the first event.
3. The method of claim 2, the event normalization rule comprising:
determining information which is provided in each event processed by the data processing system and has the same structure after being combined by the preset combination rule; and taking the determined information as the appointed information of the event.
4. The method of claim 3, the specified information further comprising at least one of an event name, an event state, an event attribute code, an event attribute name, an event attribute value type.
5. The method of claim 2, when the event occurrence message includes identification information of the source file, the acquiring the source file of the first event according to the event occurrence message, comprising:
Sending an acquisition request carrying the identification information for acquiring the source file to the first equipment;
and receiving a source file of the first event sent by the first device in response to the acquisition request.
6. A data processing apparatus for use in a data processing system, the apparatus comprising:
the receiving module is used for receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
the generation module is used for generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
the processing module is used for correspondingly processing the first event according to the event standardization information;
the processing module comprises:
the first determining unit is used for determining a first event processing model corresponding to the event standardized information of the first event according to a mapping relation between an event code contained in preset event standardized information and a model code in an event processing model; the event processing model also comprises a model processing mode which is determined according to the actual application scene;
The second determining unit is used for determining an event processing mode corresponding to the first event processing model;
and the processing unit is used for correspondingly processing the first event in an event-driven mode according to the event processing mode.
7. The apparatus of claim 6, the generating module comprising:
the first acquisition unit is used for acquiring a source file of the first event according to the event occurrence message;
a second obtaining unit, configured to obtain, from the source file, specified information in the first event;
and the combination unit is used for combining the specified information according to a preset combination rule to obtain event standardization information of the first event.
8. The apparatus of claim 7, further comprising:
the determining module is used for determining information which is provided with all events processed by the data processing system and has the same structure after being combined by the preset combination rule; and taking the determined information as the appointed information of the event.
9. The apparatus of claim 8, the designation information further comprising at least one of an event name, an event state, an event attribute code, an event attribute name, an event attribute value type.
10. The apparatus of claim 7, the first acquisition unit further to:
when the event occurrence message comprises the identification information of the source file, sending an acquisition request carrying the identification information for acquiring the source file to the first equipment;
and receiving a source file of the first event sent by the first device in response to the acquisition request.
11. A data processing system, comprising:
the first device is used for sending an event occurrence message to the event standardization platform, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
the event standardization platform is used for receiving event occurrence information sent by the first equipment; generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule; sending the event standardization information to an event processing model platform;
the event processing model platform is used for receiving the event standardization information sent by the event standardization platform; carrying out corresponding processing on the first event according to the event standardization information;
The processing the first event according to the event standardization information includes:
determining a first event processing model corresponding to the event standardization information of the first event according to a mapping relation between an event code contained in preset event standardization information and a model code in an event processing model; the event processing model also comprises a model processing mode which is determined according to the actual application scene;
determining an event processing mode corresponding to the first event processing model;
and according to the event processing mode, correspondingly processing the first event in an event-driven mode.
12. A data processing apparatus comprising:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
Carrying out corresponding processing on the first event according to the event standardization information;
the processing the first event according to the event standardization information includes:
determining a first event processing model corresponding to the event standardization information of the first event according to a mapping relation between an event code contained in preset event standardization information and a model code in an event processing model; the event processing model also comprises a model processing mode which is determined according to the actual application scene;
determining an event processing mode corresponding to the first event processing model;
and according to the event processing mode, correspondingly processing the first event in an event-driven mode.
13. A storage medium storing computer-executable instructions that when executed implement the following:
receiving an event occurrence message sent by first equipment, wherein the event occurrence message comprises a source file of a first event and/or identification information of the source file;
generating event standardization information of the first event according to the event occurrence message and a preset event standardization rule;
Carrying out corresponding processing on the first event according to the event standardization information;
the processing the first event according to the event standardization information includes:
determining a first event processing model corresponding to the event standardization information of the first event according to a mapping relation between an event code contained in preset event standardization information and a model code in an event processing model; the event processing model also comprises a model processing mode which is determined according to the actual application scene;
determining an event processing mode corresponding to the first event processing model;
and according to the event processing mode, correspondingly processing the first event in an event-driven mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811441490.3A CN109582476B (en) | 2018-11-29 | 2018-11-29 | Data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811441490.3A CN109582476B (en) | 2018-11-29 | 2018-11-29 | Data processing method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109582476A CN109582476A (en) | 2019-04-05 |
| CN109582476B true CN109582476B (en) | 2024-01-16 |
Family
ID=65925509
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811441490.3A Active CN109582476B (en) | 2018-11-29 | 2018-11-29 | Data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109582476B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110321181A (en) * | 2019-06-28 | 2019-10-11 | 阿里巴巴集团控股有限公司 | Event production method and its device under SOA system |
| CN117726237B (en) * | 2024-02-07 | 2024-05-10 | 四川大学华西医院 | Instant evaluation method, instant evaluation device, computer equipment and readable storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20070115078A (en) * | 2006-05-30 | 2007-12-05 | 동국대학교 산학협력단 | Crm gateway system used for customer relationship management |
| EP1944676A1 (en) * | 2001-06-14 | 2008-07-16 | Cisco Systems, Inc. | Stateful reference monitor |
| CN101799684A (en) * | 2010-04-20 | 2010-08-11 | 杭州和利时自动化有限公司 | Event-handling method and system of distributed control system |
| US8087087B1 (en) * | 2002-06-06 | 2011-12-27 | International Business Machines Corporation | Management of computer security events across distributed systems |
| CN102340495A (en) * | 2010-07-26 | 2012-02-01 | 中国移动通信集团广东有限公司 | Event center supporting cross-system service linkage and event processing method of event center |
| CN102348185A (en) * | 2010-07-29 | 2012-02-08 | 埃森哲环球服务有限公司 | Computer-implemented method, system, and computer program product for telecommunication rating |
| CN103380603A (en) * | 2011-01-05 | 2013-10-30 | 皇家飞利浦电子股份有限公司 | System and method for distributing meaningful clinical alerts |
| CN103458033A (en) * | 2013-09-04 | 2013-12-18 | 北京邮电大学 | System for providing services of event-driven service-oriented internet of things and working method thereof |
| CN105843878A (en) * | 2016-03-17 | 2016-08-10 | 杭州优云软件有限公司 | IT system event standardization realization method |
| CN107645414A (en) * | 2017-09-16 | 2018-01-30 | 广西电网有限责任公司电力科学研究院 | A kind of power transmission and transformation equipment state alarming processing system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10528906B2 (en) * | 2011-04-22 | 2020-01-07 | Progress Software Corporation | System and method for responsive process management driven by business visibility and complex event processing |
| US10725979B2 (en) * | 2017-03-24 | 2020-07-28 | Microsoft Technology Licensing, Llc | Measuring usage of computing resources by storing usage events in a distributed file system |
-
2018
- 2018-11-29 CN CN201811441490.3A patent/CN109582476B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1944676A1 (en) * | 2001-06-14 | 2008-07-16 | Cisco Systems, Inc. | Stateful reference monitor |
| US8087087B1 (en) * | 2002-06-06 | 2011-12-27 | International Business Machines Corporation | Management of computer security events across distributed systems |
| KR20070115078A (en) * | 2006-05-30 | 2007-12-05 | 동국대학교 산학협력단 | Crm gateway system used for customer relationship management |
| CN101799684A (en) * | 2010-04-20 | 2010-08-11 | 杭州和利时自动化有限公司 | Event-handling method and system of distributed control system |
| CN102340495A (en) * | 2010-07-26 | 2012-02-01 | 中国移动通信集团广东有限公司 | Event center supporting cross-system service linkage and event processing method of event center |
| CN102348185A (en) * | 2010-07-29 | 2012-02-08 | 埃森哲环球服务有限公司 | Computer-implemented method, system, and computer program product for telecommunication rating |
| CN103380603A (en) * | 2011-01-05 | 2013-10-30 | 皇家飞利浦电子股份有限公司 | System and method for distributing meaningful clinical alerts |
| CN103458033A (en) * | 2013-09-04 | 2013-12-18 | 北京邮电大学 | System for providing services of event-driven service-oriented internet of things and working method thereof |
| CN105843878A (en) * | 2016-03-17 | 2016-08-10 | 杭州优云软件有限公司 | IT system event standardization realization method |
| CN107645414A (en) * | 2017-09-16 | 2018-01-30 | 广西电网有限责任公司电力科学研究院 | A kind of power transmission and transformation equipment state alarming processing system |
Non-Patent Citations (2)
| Title |
|---|
| Multimodal feature fusion for robust event detection in web videos;Pradeep Natarajan 等;《2012 IEEE Conference on Computer Vision and Pattern Recognition》;1298-1305 * |
| 智能电网监控中的分布式复杂事件处理技术研究;祖向荣;《中国博士学位论文全文数据库 工程科技II辑》;C042-163 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109582476A (en) | 2019-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108173706B (en) | Service marking method, device and equipment under multi-service system | |
| CN107229619B (en) | Method and device for counting and displaying calling condition of internet service link | |
| CN110008018A (en) | A kind of batch tasks processing method, device and equipment | |
| CN111899008B (en) | Resource transfer method, device, equipment and system | |
| CN107463422A (en) | Code dispositions method and device, electronic equipment | |
| CN112182004B (en) | Method, device, computer equipment and storage medium for checking data in real time | |
| CN111414369A (en) | Data processing method, device and equipment | |
| CN109582476B (en) | Data processing method, device and system | |
| CN103678591A (en) | Device and method for automatically executing multi-service receipt statistical treatment | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN116595588A (en) | Safety analysis method and device for railway signal system development process | |
| CN110489392A (en) | Data access method, device, system, storage medium and equipment between multi-tenant | |
| CN112491943A (en) | Data request method, device, storage medium and electronic equipment | |
| CN115660591A (en) | Process configuration method, system, electronic device and medium for business approval | |
| CN114253922A (en) | Resource directory management method, resource management method, device, equipment and medium | |
| CN112231377A (en) | Data mapping method, system, device, server and storage medium | |
| CN116993519A (en) | Bill generation method and device | |
| CN110674110A (en) | Bank distributed database construction method and device | |
| CN115484149B (en) | Network switching method, network switching device, electronic equipment and storage medium | |
| CN111414162B (en) | Data processing method, device and equipment thereof | |
| CN115455114A (en) | Office automation system data synchronization method and related equipment | |
| CN116467146A (en) | Log processing method and application server | |
| CN117216164A (en) | Financial data synchronous processing method, apparatus, device, medium and program product | |
| CN117331997A (en) | Recording data generation method, device, platform, equipment and readable storage medium | |
| CN116069784A (en) | Data processing method, device and medium for ERP system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200922 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200922 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |