CN115617849A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115617849A CN115617849A CN202211153733.XA CN202211153733A CN115617849A CN 115617849 A CN115617849 A CN 115617849A CN 202211153733 A CN202211153733 A CN 202211153733A CN 115617849 A CN115617849 A CN 115617849A
- Authority
- CN
- China
- Prior art keywords
- data
- event
- source
- listener
- data object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 29
- 238000000034 method Methods 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 33
- 230000004044 response Effects 0.000 claims abstract description 28
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012544 monitoring process Methods 0.000 claims description 8
- 238000013480 data collection Methods 0.000 claims description 4
- 230000001960 triggered effect Effects 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 4
- 238000012546 transfer Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
- G06F16/24565—Triggers; Constraints
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a data processing device, electronic equipment and a storage medium. Wherein, the method comprises the following steps: acquiring initial data of at least one type of data source, and converting the initial data into a data object according to a preset format; binding an event listener for the data object according to a predefined event model; storing the data object to a response data set upon triggering of the event listener. The embodiment of the invention solves the problems of single data source and low retrieval efficiency in the existing data retrieval method, can perform data retrieval on complex and huge multi-source heterogeneous data, and simultaneously adopts an event-driven mode to trigger data to perform active response, thereby improving the data retrieval efficiency.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
With the continuous improvement of automation, informatization, intellectualization and other technologies, the data form and data source on the network are also increasingly abundant. The multi-source heterogeneous data such as structured data, unstructured data and semi-structured data become main target objects of data processing at present, valuable information is extracted and derived from complex and huge multi-source heterogeneous data, data analysis can be facilitated for users, and important basis is provided for work decision making.
Most of the existing data processing and retrieval methods aim at structured data, and the query mode mainly solves the problem of frequent query of hot data blocks by actively traversing the data of a database or adding indexes. Therefore, the existing data processing and searching method has the problems of single processing data source and low searching efficiency.
Disclosure of Invention
The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, which are used for realizing data retrieval operation of multi-source heterogeneous data, and meanwhile, an event-driven mode is adopted to trigger data to actively respond, so that the data retrieval efficiency is improved.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
acquiring initial data of at least one type of data source, and converting the initial data into a data object according to a preset format;
binding an event monitor for the data object according to a predefined event model;
the data objects are stored to the response data set upon triggering of the event listener.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, including:
the data object module is used for acquiring initial data of at least one type of data source and converting the initial data into a data object according to a preset format;
the monitoring binding module is used for binding an event monitor for the data object according to the predefined event model;
a data collection module for storing the data object to the response data set upon triggering of the event listener.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data processing method according to any of the embodiments of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, where computer instructions are stored, and the computer instructions are configured to enable a processor to implement the data processing method according to any embodiment of the present invention when executed.
According to the technical scheme of the embodiment of the invention, the problems of single data source and low retrieval efficiency in the existing data retrieval method are solved by acquiring the initial data of at least one type of data source, converting the initial data into the data object according to the preset format, binding the event monitor for the data object according to the predefined event model, and storing the data object into the response data set when the event monitor is triggered, so that the data retrieval method can be used for retrieving the complex and huge multi-source heterogeneous data, and meanwhile, the data is triggered to actively respond in an event-driven mode, and the data retrieval efficiency is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention;
FIG. 3 is a flowchart of a data processing method according to a third embodiment of the present invention;
FIG. 4 is a diagram illustrating an exemplary architecture of a data processing system in which a third embodiment of the present invention is applicable;
fig. 5 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device implementing the data processing method according to the embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.
Example one
Fig. 1 is a flowchart of a data processing method, where the embodiment of the present invention is applicable to a case of performing data processing on multi-source heterogeneous data, and the method may be executed by a data processing apparatus, where the data processing apparatus may be implemented in a form of hardware and/or software. As shown in fig. 1, a data processing method provided in this embodiment specifically includes the following steps:
s110, acquiring initial data of at least one type of data source, and converting the initial data into a data object according to a preset format.
In the embodiment of the present invention, the initial data may be understood as multi-source heterogeneous data for processing, and exemplarily, the initial data may include unstructured data, structured data, binary stream data, semi-structured data, and the like. The preset format may be understood as a unified data format for performing a structuring and standardization process on the initial data of different data sources, and may include, for example and not limited to, the following elements: object tags, object attributes, attribute types, object values, and the like. The data object may be understood as data obtained by structuring initial data of different data sources, and may exemplarily include an object name, attribute information, a data value, and the like.
Specifically, the data processing apparatus may, according to the service requirement and based on the conditions of the data amount, the PULL frequency, the PULL method, and the like, adopt a manner of actively pushing from the data processing apparatus through the data source to the PUSH (PUSH), and/or actively obtaining initial data from different data sources from the PULL (PULL) of the data source, in a specific embodiment, if the data source has a plurality of types (for example, includes multiple types of unstructured data, structured data, binary stream data, semi-structured data, and the like), and the frequency of data generated by the data source side is relatively random, for example, the data source side generates data irregularly and the data amount is also not fixed, in order to enable the data processing apparatus to obtain the data generated each time completely in the first time, the data acquisition may be performed in a PUSH manner; in another embodiment, if the service requirement has to update data at a fixed frequency (e.g. every 10 minutes) or at a fixed time point (e.g. 8; meanwhile, the initial data can be acquired by multiple access modes such as batch, real-time or one-time load, and the specific data acquisition mode can be comprehensively determined according to the service requirement, the structural characteristics of a data source and the like, which is not limited in the embodiment of the invention; secondly, carrying out induction analysis according to the label attributes of the initial data of different data sources to obtain a uniform data standard format, namely a preset format; finally, the initial data is converted into a required data object through a preset format, where the data object may include some important parameters of the initial data, and for example, the data object may include an object name, attribute information, a data value, and the like. By converting initial data of different data sources into data objects with the same data format, the efficiency of later data retrieval can be improved conveniently.
And S120, binding an event listener for the data object according to the predefined event model.
In the embodiment of the present invention, the predefined event model may be understood as a specific event generated in advance, and may define a retrieval data event model, for example, an event model of "transferring a student object, an age attribute, and a data object with a data value less than 10 to a temporary data pool". In some embodiments, the predefined event model may also be understood as a copy of all event instructions. An event listener may be understood as an event trigger mechanism for receiving information and event instructions of a predefined event model, and may include event listeners such as, for example, scroll bar movement events, keyboard entries, mouse movements, text change events, and window events.
Specifically, the data processing apparatus may pre-define an event model according to a service requirement, where the predefined event model may include a data object and a trigger condition corresponding to an event, and the number of the predefined event models may be one or more; the binding between the data object and the event listener can then be achieved through a listening interface (event binding function) on the event listener, which can include, for example, a component event class interface, a window event interface, a click event interface, a text box event interface, a mouse button event interface, a keyboard event interface, a focus event interface, and the like. In some embodiments, each data object corresponds to an event listener. After the data object and the event monitor are bound, the event monitor can receive information and an event instruction of a predefined event model in real time, and trigger response is conveniently carried out according to the monitored external event instruction.
S130, storing the data object to the response data set when the event listener is triggered.
In the embodiment of the present invention, the response data set may be understood as a data set obtained after responding to the event instruction, and for example, the response data set may include a data table of data objects and event correspondences, or a data file stored in another form, and the like. Triggering can be understood as a condition that an event listener performs a corresponding operation after receiving a certain event instruction, and exemplary triggering can include clicking a mouse, pressing a keyboard, and the like by a user.
Specifically, one or more event listeners in the data processing apparatus may perform real-time monitoring on an externally input event instruction in a thread creating manner, and when the monitored event instruction is matched with a predefined event model, the event listener may intercept the event, execute a corresponding event instruction, and export and store a data object meeting an instruction condition into a response data set. In some embodiments, the event monitoring process may include that a user operation (e.g., clicking) causes an event trigger, an event object is generated after the event trigger, the event object is transmitted to the event monitor as a parameter, and then the event monitor calls a corresponding method to process the event object. In a specific embodiment, when the event instruction is "after clicking a search button with a mouse, transferring a data object with a student object, an age attribute and a data value smaller than 10 into a temporary data pool", and the event instruction is matched with a certain predefined event model, the event monitor performs trigger response, picks out the data object meeting the instruction condition, and transfers the acquired response data set into the temporary data pool for storage.
According to the technical scheme of the embodiment of the invention, the problems of single data source and low retrieval efficiency in the existing data retrieval method are solved by acquiring the initial data of at least one type of data source, converting the initial data into the data object according to the preset format, binding the event monitor for the data object according to the predefined event model, and storing the data object into the response data set when the event monitor is triggered, so that the data retrieval method can be used for retrieving the complex and huge multi-source heterogeneous data, and meanwhile, the data is triggered to actively respond in an event-driven mode, and the data retrieval efficiency is improved.
Example two
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention, which further details a process of converting initial data into a data object, a process of binding an event listener for the data object according to a predefined event model, and a process of storing the data object into a response data set when the event listener triggers according to the second embodiment of the present invention. As shown in fig. 2, the data processing method provided in the second embodiment specifically includes the following steps:
s210, acquiring initial data of at least one of an unstructured data source, a structured data source, a binary stream data source and a semi-structured data source.
Specifically, the data processing apparatus may access at least one of the initial data of the unstructured data source, the structured data source, the binary stream data source, and the semi-structured data source to the data lake for storage through a uniform access manner provided by the data repository, such as a uniform API (Application Program Interface) or an Interface. The unstructured data refers to data without fixed modes, such as e-mail, WORD, PPT, EXCEL, PDF documents, and the like; structured data refers to data of a relational model, such as a table in a relational database; binary stream data refers to data stored in binary, such as images, audio, and video; the semi-structured data refers to data which is not in a relational model and has a basic fixed structure mode, such as log files, XML documents, JSON documents and the like. Illustratively, the Data repository may be a Data Lake, where the Data Lake (Data Lake) is a centralized repository, which can store Data in the original format of the multi-source heterogeneous Data without structuring the Data in advance, and thus the Data Lake has greater flexibility.
And S220, determining the object name, the attribute information and the data value of the initial data.
Specifically, the initial data is processed according to a predefined conversion component, and the processed data includes an object name, attribute information, and a data value. The object name can be determined by the file name of the initial data, and can also be generated by self-definition according to the service requirement; the attribute information may include a nominal attribute, a binary attribute, an ordinal attribute, a numerical attribute, and the like of the initial data; the data value may be determined by a specific data value of the initial data itself, optionally, the length of the data value may be set in a user-defined manner according to a service requirement, for example, assuming that the length of the data value is 20, if the length of the data value of the initial data exceeds 20 bits, the first 20 bits may be intercepted as the final data value of the initial data, and if the length of the data value of the initial data does not exceed 20 bits, the last several bits of the data value may be subjected to zero padding and then taken as the final data value of the initial data. In practical operation, one or more conversion components may be used to process the initial data, that is, the initial data of different sources may be processed by different conversion components, and accordingly, different conversion components may correspond to different data conversion methods.
And S230, storing the object name, the attribute information and the data value as data objects in a data lake.
Specifically, after the initial data is subjected to the uniform format standard conversion processing, the obtained object name, attribute information, and data value are stored in the data lake as the corresponding data object. In some embodiments, data objects may be stored to the data lake using columnar storage and/or row-column hybrid data storage. The principle of column-wise storage and column-wise hybrid storage can be referred to the related description in the prior art, and will not be described herein.
And S240, extracting the object name of the data object.
Specifically, the data processing apparatus may extract a desired object name from the data object conforming to the standard data format. In some embodiments, the converted data object may be searched in the data lake, and then the corresponding object name may be extracted from the data object according to the rule for determining the object name. In other embodiments, the data processing apparatus may also adaptively select a manner of extracting the object name of the data object according to business requirements, for example, may select to continuously extract the object name of the data object, or extract the object name of the data object once every preset time.
And S250, searching a predefined event model matched with the data object according to the object name.
Specifically, all event instructions may be stored in one or more predefined event models in advance, and then according to the object name of the previously extracted data object, predefined event models matched with the data object may be searched in the data lake by using a sequential table lookup, an ordered table lookup (binary lookup, interpolation lookup, fibonacci lookup), a binary tree lookup (binary ordering tree, balanced binary tree, multipath lookup tree), or a hash table (hash table) lookup. In some embodiments, each predefined event model may include one or more data objects.
S260, mounting the data object to an event listener of a predefined event model.
In the embodiment of the present invention, the mounting may be understood as binding the data object and the event listener of the predefined event model, and exemplarily, the mounting may include binding the data object and the event listener of the predefined event model through a preset mounting instruction.
Specifically, each data object can realize the mount binding operation between the data object and the event listener of the predefined event model through the listening interface on the event listener. In some embodiments, one event listener can mount one or more data objects, the data objects and the event listener are in one-to-one correspondence, and the data objects need to be mounted on the event listener, so that when the event listener receives a predefined event model instruction, the event listener can perform trigger response, thereby implementing event driving.
And S270, controlling the event monitor to monitor an external event source.
In the embodiment of the present invention, an external event source may be understood as an object where an event occurs, and for example, an external event source may generally refer to a specific component, such as: a user clicks a certain button, and the button is an external event source; for another example: and if the window is closed, the window is the external event source.
In particular, the data processing apparatus may control the event listener to listen to the external event source in real time through one or more capturing components, which may illustratively include a component for capturing action information such as mouse actions (e.g., press, click, and release, etc.) and keyboard actions (e.g., press, release, and tap, etc.). In some embodiments, the event listener may be caused to perform a corresponding snoop operation by way of creating a thread.
S280, when the event characteristics of the external event source meet the event listener, storing the data object bound by the event listener into a response data set.
In the embodiment of the present invention, the event characteristics of the external event source may be understood as data objects and related attribute information describing an event, which are included in an event instruction triggered by the external event source, and for example, for an event instruction "transfer a student object, an age attribute, and a data object with a data value less than 10 to a temporary data pool", the corresponding event characteristics may include five parts, namely "student object", "age attribute", "data value less than 10", "data object", and "transfer to a temporary data pool".
In some embodiments, the event characteristics of the external event source satisfy an event listener, comprising: the event characteristics of the external event source are in accordance with the triggering conditions of the predefined event model corresponding to the event listener.
In the embodiment of the present invention, the triggering condition may be understood as that when a certain condition is satisfied, a specific event is triggered.
Specifically, after the external event source triggers the event instruction, if the event characteristics of the external event source are consistent with a certain predefined event model bound by the event listener, that is, the event characteristics of the external event source meet the triggering conditions of the predefined event model corresponding to the event listener, the data processing device may invoke the event listener to perform response, execute corresponding operations according to the event instruction, and further export and store the data object bound by the event listener into a response data set.
In some embodiments, the data processing method may further include:
and creating a predefined event model according to the business requirements, wherein the predefined event model at least comprises a triggering condition and a data object.
Specifically, the user may create one or more databases containing all events, i.e. predefined event models, in advance according to business requirements, such as data collection, sorting, storage, retrieval, etc., wherein the predefined event models include at least trigger conditions and data objects. Illustratively, when a certain predefined event model is 'mouse click on a commodity query, a commodity object, a price attribute and a data object with a price of 50 to 100 are transferred into a temporary data pool', wherein the triggering condition is 'mouse click', and the data object comprises 'commodity object', 'price attribute' and 'price data value'. In some embodiments, the predefined event model may also be understood as a copy of all event instructions.
In some embodiments, the data processing method may further include: the event monitor monitors information of a predefined event model and an event instruction through a database trigger.
In the embodiment of the present invention, the database trigger may be understood as a mechanism provided for executing the business rule and maintaining Data integrity, and may be automatically triggered before or after performing the operations of inserting, updating, deleting, and the like, and exemplarily, the database trigger may include a DML (Data management Language) trigger and a DDL (Data definition Language) trigger.
Specifically, the event listener may perform real-time monitoring on information of the predefined event model and the event instruction through one or more database triggers, where the information of the predefined event model may include a data object and a corresponding trigger condition; the event instruction may be input externally, and an example of the event instruction may be "transfer employee object, gender attribute, and gender male data object to the temporary data pool".
According to the technical scheme, initial data of at least one of an unstructured data source, a structured data source, a binary stream data source and a semi-structured data source are collected, an object name, attribute information and a data value of the initial data are determined, the object name, the attribute information and the data value are stored in a data lake as data objects, the object name of the data object is extracted, a predefined event model matched with the data object is searched according to the object name, the data object is mounted to an event monitor of the predefined event model, the event monitor is controlled to monitor an external event source, and further when the event characteristics of the external event source meet the event monitor, the data object bound by the event monitor is stored in a response data set.
EXAMPLE III
Fig. 3 is a flowchart of a data processing method according to a third embodiment of the present invention. On the basis of the foregoing embodiments, the present embodiment provides a preferred implementation of a data processing method, which can perform data retrieval on complex and huge multi-source heterogeneous data quickly. FIG. 4 is a diagram illustrating an exemplary structure of a data processing system to which a third embodiment of the present invention is applied.
As shown in fig. 3, a data processing method provided in the third embodiment of the present invention specifically includes the following steps:
s310, accessing the multi-source heterogeneous initial data into a data lake.
In some embodiments, the data lake has a standard interface widely accepted by the industry, such as HDFS (Hadoop Distributed File System), S3 (Amazon Simple Storage Service), ORC (Optimized Row column), queue, etc. at the Storage level.
And S320, converting the initial data into a data object and storing the data object in a data lake.
And S330, predefining an event model according to the service requirement.
And S340, binding an event listener to the data object.
And S350, controlling the event monitor to monitor an external event source.
And S360, when the external event source accords with the event model characteristics, the time monitor bound with the data object receives the instruction, responds to the data which accords with the instruction condition and displays the data.
According to the technical scheme, initial data of multi-source isomerism is accessed into a data lake, the data entering the data lake are converted into data objects and stored in the data lake, then an event model is defined in advance according to business requirements, an event monitor is bound to the data objects, the event monitor is controlled to monitor external event sources, when the external event sources meet the characteristics of the event model, the time monitor bound to the data objects receives instructions, the data meeting the conditions of the instructions are responded and displayed correspondingly, the problems that in an existing data retrieval method, the data sources are single, the retrieval efficiency is low are solved, data retrieval can be conducted on complex and huge multi-source isomerism data, meanwhile event models are used for conducting correlation on events and triggering data to conduct active response in an event driving mode, and the data retrieval efficiency is improved.
Example four
Fig. 5 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention. As shown in fig. 5, the apparatus includes:
the data object module 41 is configured to obtain initial data of at least one type of data source, and convert the initial data into a data object according to a preset format.
And a listener binding module 42 for binding an event listener for the data object according to the predefined event model.
A data collection module 43 for storing the data objects to the response data set upon triggering of the event listener.
According to the technical scheme, the initial data of at least one type of data source is obtained through the data object module, the initial data is converted into the data object according to the preset format, the monitoring binding module binds the event monitor for the data object according to the predefined event model, and the data object is stored into the response data set through the data set module when the event monitor is triggered.
Further, the data object module 41 includes:
the initial data unit is used for acquiring initial data of at least one of an unstructured data source, a structured data source, a binary stream data source and a semi-structured data source;
the data determination unit is used for determining the object name, the attribute information and the data value of the initial data;
and the data object storage unit is used for storing the object name, the attribute information and the data value as data objects to the data lake.
Further, the snooping binding module 42 includes:
an object name extraction unit for extracting an object name of the data object;
the data object matching unit is used for searching a predefined event model matched with the data object according to the object name;
and the data object mounting unit is used for mounting the data object to an event listener of a predefined event model.
Further, the data aggregation module 43 includes:
the monitor control unit is used for controlling the event monitor to monitor an external event source;
and the data response unit is used for storing the data object bound by the event listener to a response data set when the event characteristics of the external event source meet the event listener.
In some embodiments, the apparatus may further include:
and the event monitoring module is used for monitoring the information of the predefined event model and the event instruction by the event monitor through the database trigger.
In some embodiments, the apparatus may further include:
and the event characteristic triggering module is used for enabling the event characteristics of the external event source to accord with the triggering conditions of the predefined event model corresponding to the event monitor.
In some embodiments, the apparatus may further include:
and the event model creating module is used for creating a predefined event model according to the business requirements, wherein the predefined event model at least comprises a triggering condition and a data object.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
FIG. 6 illustrates a schematic diagram of an electronic device 50 that may be used to implement an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 6, the electronic device 50 includes at least one processor 51, and a memory communicatively connected to the at least one processor 51, such as a Read Only Memory (ROM) 52, a Random Access Memory (RAM) 53, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 51 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 52 or the computer program loaded from a storage unit 58 into the Random Access Memory (RAM) 53. In the RAM 53, various programs and data necessary for the operation of the electronic apparatus 50 can also be stored. The processor 51, the ROM 52, and the RAM 53 are connected to each other via a bus 54. An input/output (I/O) interface 55 is also connected to bus 54.
A plurality of components in the electronic apparatus 50 are connected to the I/O interface 55, including: an input unit 56 such as a keyboard, a mouse, or the like; an output unit 57 such as various types of displays, speakers, and the like; a storage unit 58 such as a magnetic disk, an optical disk, or the like; and a communication unit 59 such as a network card, modem, wireless communication transceiver, etc. The communication unit 59 allows the electronic device 50 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The processor 51 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processors 51 include, but are not limited to, central Processing Units (CPUs), graphics Processing Units (GPUs), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processors, controllers, microcontrollers, and the like. The processor 51 performs the various methods and processes described above, such as data processing methods.
In some embodiments, the data processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and/or installed onto electronic device 50 via ROM 52 and/or communications unit 59. When the computer program is loaded into the RAM 53 and executed by the processor 51, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 51 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring initial data of at least one type of data source, and converting the initial data into a data object according to a preset format;
binding an event listener for the data object according to a predefined event model;
storing the data object to a response data set upon triggering of the event listener.
2. The method of claim 1, wherein the obtaining initial data of at least one type of data source and converting the initial data into data objects according to a preset format comprises:
acquiring the initial data of at least one of an unstructured data source, a structured data source, a binary stream data source and a semi-structured data source;
determining an object name, attribute information and a data value of the initial data;
and storing the object name, the attribute information and the data value as the data object to a data lake.
3. The method according to claim 1, wherein said binding an event listener for the data object according to a predefined event model comprises:
extracting an object name of the data object;
searching the predefined event model matched with the data object according to the object name;
mounting the data object to the event listener of the predefined event model.
4. The method of claim 3, wherein the event listener listens for information of the predefined event model and event instructions through a database trigger.
5. The method of claim 1, wherein storing the data object to a response data set upon triggering of the event listener comprises:
controlling the event listener to monitor an external event source;
storing the data object bound by the event listener to the response data set when the event characteristics of the external event source satisfy the event listener.
6. The method according to claim 5, wherein the event characteristics of the external event source satisfy the event listener, comprising:
and the event characteristics of the external event source meet the triggering conditions of the predefined event model corresponding to the event listener.
7. The method of claims 1-6, further comprising:
and creating the predefined event model according to business requirements, wherein the predefined event model at least comprises a trigger condition and a data object.
8. A data processing apparatus, comprising:
the data object module is used for acquiring initial data of at least one type of data source and converting the initial data into a data object according to a preset format;
the monitoring binding module is used for binding an event monitor for the data object according to a predefined event model;
a data collection module to store the data object to a response data set when the event listener triggers.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data processing method of any one of claims 1-7.
10. A computer-readable storage medium, characterized in that it stores computer instructions for causing a processor to implement the data processing method of any of claims 1-7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211153733.XA CN115617849A (en) | 2022-09-21 | 2022-09-21 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211153733.XA CN115617849A (en) | 2022-09-21 | 2022-09-21 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115617849A true CN115617849A (en) | 2023-01-17 |
Family
ID=84858322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211153733.XA Pending CN115617849A (en) | 2022-09-21 | 2022-09-21 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115617849A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048426A (en) * | 2023-03-27 | 2023-05-02 | 南京芯驰半导体科技有限公司 | Signal processing method, device, electronic equipment and storage medium |
| CN116319653A (en) * | 2023-03-28 | 2023-06-23 | 中国工商银行股份有限公司 | E-mail data processing method and device |
-
2022
- 2022-09-21 CN CN202211153733.XA patent/CN115617849A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048426A (en) * | 2023-03-27 | 2023-05-02 | 南京芯驰半导体科技有限公司 | Signal processing method, device, electronic equipment and storage medium |
| CN116319653A (en) * | 2023-03-28 | 2023-06-23 | 中国工商银行股份有限公司 | E-mail data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109189835B (en) | Method and device for generating data wide table in real time | |
| US20210390428A1 (en) | Method, apparatus, device and storage medium for training model | |
| US11657612B2 (en) | Method and apparatus for identifying video | |
| CN115617849A (en) | Data processing method and device, electronic equipment and storage medium | |
| US20160292591A1 (en) | Streamlined analytic model training and scoring system | |
| EP3968185A1 (en) | Method and apparatus for pushing information, device and storage medium | |
| US20230134615A1 (en) | Method of processing task, electronic device, and storage medium | |
| US20220129418A1 (en) | Method for determining blood relationship of data, electronic device and storage medium | |
| CN113568938B (en) | Data stream processing method and device, electronic equipment and storage medium | |
| Ahsaan et al. | Big data analytics: challenges and technologies | |
| CN110795494A (en) | Automatic testing method and device for synchronous and asynchronous cache data | |
| CN117633116A (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN109947736B (en) | Method and system for real-time computing | |
| CN113220710B (en) | Data query method, device, electronic equipment and storage medium | |
| CN113722600A (en) | Data query method, device, equipment and product applied to big data | |
| CN113239054A (en) | Information generation method, related device and computer program product | |
| CN116955856A (en) | Information display method, device, electronic equipment and storage medium | |
| CN116594709A (en) | Method, apparatus and computer program product for acquiring data | |
| CN114491253B (en) | Method and device for processing observation information, electronic equipment and storage medium | |
| CN112817930A (en) | Data migration method and device | |
| CN113656689B (en) | Model generation method and network information pushing method | |
| CN115525659A (en) | Data query method and device, electronic equipment and storage medium | |
| CN111680508B (en) | Text processing method and device | |
| CN114547477A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114625763A (en) | Information analysis method and device for database, electronic equipment and readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |