CN112699183A - Data processing method, system, readable storage medium and computer equipment - Google Patents
Data processing method, system, readable storage medium and computer equipment Download PDFInfo
- Publication number
- CN112699183A CN112699183A CN202011637964.9A CN202011637964A CN112699183A CN 112699183 A CN112699183 A CN 112699183A CN 202011637964 A CN202011637964 A CN 202011637964A CN 112699183 A CN112699183 A CN 112699183A
- Authority
- CN
- China
- Prior art keywords
- data
- changed
- layer
- message middleware
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000013500 data storage Methods 0.000 claims abstract description 61
- 238000012545 processing Methods 0.000 claims description 48
- 230000008859 change Effects 0.000 claims description 19
- 238000000034 method Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 241001178520 Stomatepia mongo Species 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a system, a readable storage medium and computer equipment, wherein the data processing method comprises the steps of acquiring data generated by an application service layer, and storing the data into a data source of a corresponding type according to the data type of the data; when the data in the data source is detected to be changed, sending changed data to a corresponding data synchronization tool so as to send the changed data to message middleware through the data synchronization tool; and transmitting the changed data to a data storage layer by utilizing the message middleware, and storing the changed data in the data storage layer. The invention solves the problem of complex operation when data query in multiple data sources is carried out in the prior art.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method, a data processing system, a readable storage medium, and a computer device.
Background
In a distributed network system architecture, in order to cope with the massive increase of data and ensure the high availability of services, the adoption of multiple data sources for storing data in different databases is a common solution. However, when data is queried, data in a plurality of data source sub-libraries needs to be queried, and the operation is relatively complex and has certain limitations.
In the prior art, in order to solve the problem of limitation of data access of a multi-data source library, field redundancy is adopted, and system layer assembly is adopted.
However, data redundancy is a typical denormal design, has a limited application scenario, is suitable for a situation with few dependent fields, and is difficult to ensure data consistency. And system layer assembly, namely acquiring data and assembling fields by calling components or services of different modules on the system layer. In practical use, the problems of circularly calling service, circularly inquiring a database and the like are easily solved, and the method is troublesome to realize in the aspects of sequencing and filtering of multiple data sources.
Disclosure of Invention
In view of the above, the present invention provides a data processing method, a data processing system, a readable storage medium, and a computer device, and aims to solve the problem in the prior art that data query is complicated in a multi-data source library.
The embodiment of the invention is realized as follows: a data processing method, the data processing method comprising:
acquiring data generated by an application service layer, and storing the data into a data source of a corresponding type according to the data type of the data;
when the data in the data source is detected to be changed, sending changed data to a corresponding data synchronization tool so as to send the changed data to message middleware through the data synchronization tool;
and transmitting the changed data to a data storage layer by utilizing the message middleware, and storing the changed data in the data storage layer.
Further, in the data processing method, when it is detected that the data in the data source is changed, and when it is detected that the data in the data source is changed, the step of sending the changed data to a corresponding data synchronization tool so as to send the changed data to a message middleware through the data synchronization tool includes:
when the data in the data source is detected to be changed, the changed data is sent to the corresponding data synchronization tool in the form of log or trigger, and the changed data is sent to the message middleware through the synchronization tool.
Further, in the data processing method, the step of transferring the changed data to a data storage layer by using the message middleware and storing the changed data in the data storage layer includes:
and transmitting the changed data to a data processing layer by using the message middleware, filtering the changed data through the data processing layer according to preset information to obtain target data, and storing the target data into the data storage layer.
Further, in the data processing method, the step of transmitting the changed data to a data processing layer by using the message middleware, filtering the changed data by the data processing layer according to preset information to obtain target data, and then storing the target data in the data storage layer includes:
the preset information includes at least one of a database name, a data table name, a specific field name, and a change operation.
Further, the data processing method further includes:
when a search signal is acquired, analyzing search data according to the search signal, and searching the change data corresponding to the search data in the data storage layer.
It is another object of an embodiment of the present invention to provide a data processing system, including:
the acquisition module is used for acquiring data generated by an application service layer and storing the data into a data source of a corresponding type according to the data type of the data;
the detection module is used for sending changed data to a corresponding data synchronization tool when detecting that the data in the data source is changed, so that the changed data is sent to the message middleware through the data synchronization tool;
and the storage module is used for transmitting the changed data to a data storage layer by using the message middleware and storing the changed data in the data storage layer.
Further, in the data processing system, the detection module is specifically configured to, when detecting that the data in the data source is changed, send the changed data to the corresponding data synchronization tool in the form of a log or a trigger, and send the changed data to the message middleware through the synchronization tool.
Further, in the data processing system, the storage module is specifically configured to transmit the changed data to a data processing layer by using the message middleware, filter the changed data through the data processing layer according to preset information to obtain target data, and then store the target data in the data storage layer.
Further, in the data processing system, in the storage module, the preset information includes at least one of a database name, a data table name, a specific field name, and a change operation.
Further, the data processing system described above, wherein the data processing system further comprises:
and the searching module is used for analyzing searching data according to the searching signal when the searching signal is acquired, and searching the changed data corresponding to the searching data in the data storage layer.
It is a further object of embodiments of the invention to provide a readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the steps of the method as described above.
It is a further object of embodiments of the present invention to provide a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method as described above when executing the program.
According to the embodiment of the invention, different data sources are set, different data synchronization tools are used for different data sources to synchronize the changed data, then the changed data are sent to the message middleware through each data synchronization tool to be processed uniformly, the changed data are transmitted to the data storage layer by the message middleware to be stored, and the data in the multiple data sources are stored in the data storage layer in a centralized manner, so that the data to be queried only needs to be queried through the data storage layer when the query is performed, and the query steps are simplified.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is an architecture diagram of a data processing system according to the present invention;
FIG. 2 is a flow chart of a data processing method according to a first embodiment of the present invention;
FIG. 3 is a flow chart of a data processing method according to a second embodiment of the present invention;
FIG. 4 is a flow chart of a data processing method according to a third embodiment of the present invention;
FIG. 5 is a block diagram showing a configuration of a data processing system according to a fourth embodiment of the present invention;
fig. 6 is a block diagram of a computer apparatus according to the present invention.
Detailed Description
To facilitate an understanding of the invention, the invention will now be described more fully with reference to the accompanying drawings. Several embodiments of the invention are presented in the drawings. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
It will be understood that when an element is referred to as being "secured to" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. The terms "vertical," "horizontal," "left," "right," and the like as used herein are for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
In the prior art, in order to deal with the mass increase of data and ensure the high availability of service, the adoption of multiple data sources for storing data in different databases is a common solution. However, when data is queried, data in a plurality of data source sub-libraries needs to be queried, and the operation is relatively complex and has certain limitations.
The invention aims to provide a data processing method, a data processing system, a readable storage medium and computer equipment, and aims to solve the problem that in the prior art, data query is complicated in a multi-data source library.
Referring to FIG. 1, a block diagram of a data processing system according to the present invention is shown, and the following embodiments can be applied to the block diagram of the data processing system shown in FIG. 1.
Referring to fig. 2, a first embodiment of the present invention provides a data processing method, which includes steps S10-S12.
Step S10, acquiring data generated by the application service layer, and storing the data into a data source of a corresponding type according to the data type of the data.
The application service layer mainly comprises actual production and users of data, the data production party mainly comprises foreground businesses such as apps and applets, and the data is mainly generated by daily operations of a large number of C-end users.
In the embodiment, three data sources are mainly listed, namely a traditional relational database mysql, a non-relational database mongodb and a hBase respectively.
And step S11, when detecting that the data in the data source is changed, sending the changed data to a corresponding data synchronization tool, so as to send the changed data to a message middleware through the data synchronization tool.
Specifically, the data synchronization tool and the data source synchronization tool are closely related to the data source, and when data in the corresponding data source is changed, more specifically, when the data in the corresponding data source has changes such as addition, deletion, check, change and the like, the changed data is sent to the data synchronization tool. And then, the changed data is sent to the message middleware through each data synchronization tool for unified processing.
In the present embodiment, three data sources are shown, namely, a conventional relational database mysql, a non-relational database mongodb and a hbase.
In specific practice, wherein:
the MySQL database uses canal as a data synchronization tool, the canal is mainly used for providing incremental data subscription and consumption based on incremental log analysis of the MySQL database, simulates an interactive protocol of MySQL slave, pretends to be MySQL slave, sends a dump protocol to a MySQL master, the MySQL master receives the dump request, starts to push binary log to the slave (namely canal), analyzes binary log objects (originally, byte streams), and delivers change records to an MQ system.
The Mongodb database, uses Mongo-Shake as a tool for synchronizing data, which is a common service platform based on oplog of MongoDB. It retrieves oplog from the source mongo database and then rebroadcasts or sends it to other peers in different tunnels in the target mongo database, e.g., rpc, file, tcp, kafka, etc.
Hbase database, an Observer Coprocessor using Hbase, which is similar to a trigger in a conventional database, is called by the Server when some event occurs. The Observer processor is a hook scattered in the HBase Server end code and is called when a fixed event occurs. Such as: a hook function prePut is arranged before the put operation, and the function is called by a Region Server before the put operation is executed; after the put operation there is a postPut hook function. With this feature, an mq message can be sent to kafka after data change.
And step S12, transmitting the changed data to a data storage layer by using the message middleware, and storing the changed data in the data storage layer.
Specifically, the message middleware is mainly used for transmitting the changed data, transmitting the changed data to the data storage layer and storing the changed data in the data storage layer, and when inquiring, only the data needing to be inquired needs to be inquired through the data storage layer, so that the inquiring steps are simplified.
Furthermore, the message middleware can also decouple the data synchronization tool from the specific service, improve the expansibility and facilitate the increase of data sources. And the speed of changing data consumption can be flexibly controlled, and the system availability is improved, for example and not limitation, the message middleware used in the embodiment is kafkaMq, and the kafkaMq has the characteristics of high throughput, high availability and the like.
The data storage layer, where the final data falls to the ground, in this embodiment, where the changed data is finally stored, is an ElasticSearch used in this embodiment, which is not only capable of storing mass data, but also an advanced search engine.
In summary, in the data processing method in the above embodiment of the present invention, different data sources are set, different data synchronization tools are used for different data sources to synchronize changed data, then the changed data are sent to the message middleware through each data synchronization tool to be processed uniformly, the changed data are transmitted to the data storage layer by the message middleware to be stored, and in a manner of storing data in multiple data sources in a centralized manner in the data storage layer, when an inquiry is performed, only data to be inquired need to be inquired through the data storage layer, so that the inquiry step is simplified.
Referring to fig. 3, a data processing method according to a second embodiment of the present invention includes steps S20-S22.
Step S20, acquiring data generated by the application service layer, and storing the data into a data source of a corresponding type according to the data type of the data.
Step S21, when detecting that the data in the data source is changed, sending the changed data to the corresponding data synchronization tool in the form of log or trigger, and sending the changed data to the message middleware through the synchronization tool.
Specifically, when data in the corresponding data source is changed, more specifically, when data in the corresponding data source has changes such as addition, deletion, check, change and the like, the changed data is sent to the data synchronization tool in a log mode, a trigger mode or the like, and then the changed data is sent to the message middleware through each data synchronization tool for unified processing.
Step S22, transmitting the changed data to a data processing layer by using the message middleware, filtering the changed data by the data processing layer according to preset information to obtain target data, and storing the target data in the data storage layer.
The preset information comprises at least one of a database name, a data table name, a specific field name and a change operation.
Specifically, the data processing layer mainly filters changed data, after receiving the changed data, Mq consumer applications filter and combine the data according to self business requirements through conditions such as database names, data table names, specific field names, changing operations and the like, output target data to be stored, store the target data into the data processing layer, filter and process the changed data through the data processing layer, filter out data required by Mq consumers, store the data into the data storage layer, and further facilitate query.
Referring to fig. 4, a data processing method according to a third embodiment of the present invention includes steps S30 to S33:
step S30, acquiring data generated by the application service layer, and storing the data into a data source of a corresponding type according to the data type of the data.
And step S31, when detecting that the data in the data source is changed, sending the changed data to a corresponding data synchronization tool, so as to send the changed data to a message middleware through the data synchronization tool.
And step S32, transmitting the changed data to a data storage layer by using the message middleware, and storing the changed data in the data storage layer.
Step S33, when a search signal is acquired, parsing search data according to the search signal, and searching for the changed data corresponding to the search data in the data storage layer.
Specifically, data are directly searched and obtained from a data storage layer, a business database is not affected, system stability is improved, and more specifically, an open-source elastic search is adopted by a search engine, almost every system has a searching function in actual project development, and when the data volume is small, Mysql can be directly searched from a main database, for example, but when the searching is performed to a certain extent, for example, when the data volume of the system is 10 hundred million and 100 hundred million, the I/O performance and the statistical analysis performance of a traditional relational database cannot meet the requirements of users easily.
The ElasticSearch can rapidly store, search and analyze mass data, the Wikipedia, Stack Overflow and Github all adopt the ElasticSearch, the bottom layer of the ElasticSearch is an open source library Lucene which is the encapsulation of the Lucene, an operation interface of REST API is provided, and the ElasticSearch can be used after being opened, so that the ElasticSearch is convenient to use.
Referring to fig. 5, a data processing system according to a fourth embodiment of the present invention includes:
an obtaining module 100, configured to obtain data generated by an application service layer, and store the data into a data source of a corresponding type according to a data type of the data;
the detection module 200 is configured to, when it is detected that the data in the data source is changed, send the changed data to a corresponding data synchronization tool, so as to send the changed data to a message middleware through the data synchronization tool;
the storage module 300 is configured to transmit the change data to a data storage layer by using the message middleware, and store the change data in the data storage layer.
Further, the detection module 200 is specifically configured to, when detecting that the data in the data source is changed, send the changed data to the corresponding data synchronization tool in the form of a log or a trigger, and send the changed data to the message middleware through the synchronization tool.
Further, the storage module 300 is further specifically configured to transmit the changed data to a data processing layer by using the message middleware, filter the changed data through the data processing layer according to preset information to obtain target data, and then store the target data in the data storage layer. Wherein:
in the storage module 300, the preset information includes at least one of a database name, a data table name, a specific field name, and a change operation.
Further, the data processing system further includes:
the search module 400 is configured to, when a search signal is obtained, parse search data according to the search signal, and search the data storage layer for the changed data corresponding to the search data.
The functions or operation steps of the above modules when executed are substantially the same as those of the above method embodiments, and are not described herein again.
Another aspect of the present invention also provides a readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the steps of the method according to any one of embodiments 1 to 3 above.
The method comprises the following steps:
acquiring data generated by an application service layer, and storing the data into a data source of a corresponding type according to the data type of the data;
when detecting that the data in the data source is changed, sending the changed data to a corresponding data synchronization tool, and sending the changed data to a message middleware through the data synchronization tool;
and transmitting the changed data to a data storage layer by utilizing the message middleware, and storing the changed data in the data storage layer.
In some optional embodiments, when detecting that the data in the data source is changed, sending the changed data to the corresponding data synchronization tool in a form of log or trigger, and sending the changed data to message middleware through the synchronization tool.
In some optional embodiments, the message middleware is used to transmit the changed data to a data processing layer, and filter the changed data through the data processing layer according to preset information to obtain target data, and then store the target data in the data storage layer.
Specifically, the preset information includes at least one of a database name, a data table name, a specific field name, and a change operation.
In some optional embodiments, the method further comprises the step of:
when a search signal is acquired, analyzing search data according to the search signal, and searching the change data corresponding to the search data in the data storage layer.
In summary, the readable storage medium provided by the invention simplifies the query steps by setting different data sources, synchronizing the changed data with different data synchronization tools for different data sources, then sending the changed data to the message middleware through each data synchronization tool for uniform processing, transmitting the changed data to the data storage layer by using the message middleware for storage, and by storing the data in multiple data sources into the data storage layer in a centralized manner, when querying, only querying the data to be queried through the data storage layer.
In another aspect, the present invention further provides a computer device, as shown in fig. 6, the computer device includes a memory 600, a processor 500, and a computer program 700 stored in the memory 600 and executable on the processor 500, and the processor 500 executes the computer program to implement the steps of the method according to any one of the embodiments 1 to 3.
The method comprises the following steps:
acquiring data generated by an application service layer, and storing the data into a data source of a corresponding type according to the data type of the data;
when detecting that the data in the data source is changed, sending the changed data to a corresponding data synchronization tool, and sending the changed data to a message middleware through the data synchronization tool;
and transmitting the changed data to a data storage layer by utilizing the message middleware, and storing the changed data in the data storage layer.
In some optional embodiments, when detecting that the data in the data source is changed, sending the changed data to the corresponding data synchronization tool in a form of log or trigger, and sending the changed data to message middleware through the synchronization tool.
In some optional embodiments, the message middleware is used to transmit the changed data to a data processing layer, and filter the changed data through the data processing layer according to preset information to obtain target data, and then store the target data in the data storage layer.
Specifically, the preset information includes at least one of a database name, a data table name, a specific field name, and a change operation.
In some optional embodiments, the method further comprises the step of:
when a search signal is acquired, analyzing search data according to the search signal, and searching the change data corresponding to the search data in the data storage layer.
In summary, the computer device provided by the present invention sets different data sources, synchronizes changed data with different data sources using different data synchronization tools, then sends the changed data to the message middleware through each data synchronization tool for uniform processing, transmits the changed data to the data storage layer for storage by using the message middleware, and only needs to query the data to be queried through the data storage layer when querying by storing the data in multiple data sources in the data storage layer in a centralized manner, thereby simplifying the querying steps.
In summary, the present invention sets different data sources, synchronizes changed data for different data sources using different data synchronization tools, then sends the changed data to the message middleware through each data synchronization tool for uniform processing, transmits the changed data to the data storage layer for storage by using the message middleware, and queries the data to be queried through the data storage layer when querying by storing the data in multiple data sources in the data storage layer in a centralized manner, thereby simplifying the querying steps.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
Those of skill in the art will understand that the logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be viewed as implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A data processing method, characterized in that the data processing method comprises:
acquiring data generated by an application service layer, and storing the data into a data source of a corresponding type according to the data type of the data;
when the data in the data source is detected to be changed, sending changed data to a corresponding data synchronization tool so as to send the changed data to message middleware through the data synchronization tool;
and transmitting the changed data to a data storage layer by utilizing the message middleware, and storing the changed data in the data storage layer.
2. The data processing method according to claim 1, wherein the step of sending changed data to a corresponding data synchronization tool when detecting that the data in the data source is changed, so as to send the changed data to a message middleware through the synchronization tool comprises:
when the data in the data source is detected to be changed, the changed data is sent to the corresponding data synchronization tool in the form of log or trigger, and the changed data is sent to the message middleware through the synchronization tool.
3. The data processing method of claim 1, wherein the step of transferring the change data to a data store layer using the message middleware and storing the change data in the data store layer comprises:
and transmitting the changed data to a data processing layer by using the message middleware, filtering the changed data through the data processing layer according to preset information to obtain target data, and storing the target data into the data storage layer.
4. The data processing method according to claim 3, wherein the step of transmitting the changed data to a data processing layer by using the message middleware, filtering the changed data by the data processing layer according to preset information to obtain target data, and then storing the target data in the data storage layer comprises:
the preset information includes at least one of a database name, a data table name, a specific field name, and a change operation.
5. The data processing method of claim 1, further comprising:
when a search signal is acquired, analyzing search data according to the search signal, and searching the change data corresponding to the search data in the data storage layer.
6. A data processing system, characterized in that the data processing system comprises:
the acquisition module is used for acquiring data generated by an application service layer and storing the data into a data source of a corresponding type according to the data type of the data;
the detection module is used for sending changed data to a corresponding data synchronization tool when detecting that the data in the data source is changed, so that the changed data is sent to the message middleware through the data synchronization tool;
and the storage module is used for transmitting the changed data to a data storage layer by using the message middleware and storing the changed data in the data storage layer.
7. The data processing system of claim 6, wherein the storage module is specifically configured to transmit the modified data to a data processing layer by using the message middleware, filter the modified data by the data processing layer according to preset information to obtain target data, and store the target data in the data storage layer.
8. The data processing system of claim 6, further comprising:
and the searching module is used for analyzing searching data according to the searching signal when the searching signal is acquired, and searching the changed data corresponding to the searching data in the data storage layer.
9. A readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1-5.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1-5 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637964.9A CN112699183A (en) | 2020-12-31 | 2020-12-31 | Data processing method, system, readable storage medium and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637964.9A CN112699183A (en) | 2020-12-31 | 2020-12-31 | Data processing method, system, readable storage medium and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112699183A true CN112699183A (en) | 2021-04-23 |
Family
ID=75513943
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011637964.9A Pending CN112699183A (en) | 2020-12-31 | 2020-12-31 | Data processing method, system, readable storage medium and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112699183A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113297325A (en) * | 2021-05-04 | 2021-08-24 | 西安博达软件股份有限公司 | Unauthorized data pre-correction method and system in homepage platform data synchronization |
| CN113742096A (en) * | 2021-07-14 | 2021-12-03 | 广州市玄武无线科技股份有限公司 | Method and system for realizing event queue |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
| US20200304373A1 (en) * | 2019-03-19 | 2020-09-24 | Hitachi, Ltd. | Computer system and data management method |
-
2020
- 2020-12-31 CN CN202011637964.9A patent/CN112699183A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200304373A1 (en) * | 2019-03-19 | 2020-09-24 | Hitachi, Ltd. | Computer system and data management method |
| CN110704458A (en) * | 2019-08-15 | 2020-01-17 | 平安科技（深圳）有限公司 | Data synchronization method and device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 蔡立志，武星，刘振宇: "《大数据测评》", 31 January 2015, 上海科学技术出版社 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113297325A (en) * | 2021-05-04 | 2021-08-24 | 西安博达软件股份有限公司 | Unauthorized data pre-correction method and system in homepage platform data synchronization |
| CN113742096A (en) * | 2021-07-14 | 2021-12-03 | 广州市玄武无线科技股份有限公司 | Method and system for realizing event queue |
| CN113742096B (en) * | 2021-07-14 | 2022-04-22 | 广州市玄武无线科技股份有限公司 | Method and system for realizing event queue |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11768811B1 (en) | Managing user data in a multitenant deployment | |
| US20220300464A1 (en) | Log data extraction from data chunks of an isolated execution environment | |
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN111859132A (en) | Data processing method and device, intelligent equipment and storage medium | |
| CN110543512B (en) | Information synchronization method, device and system | |
| CN112699183A (en) | Data processing method, system, readable storage medium and computer equipment | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN111400393A (en) | Data processing method and device based on multi-application platform and storage medium | |
| CN106202848A (en) | The acquisition methods of medical image file, user terminal and master server | |
| CN113792008A (en) | Method and device for acquiring network topology structure, electronic equipment and storage medium | |
| CN113672692B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN114661823A (en) | Data synchronization method and device, electronic equipment and readable storage medium | |
| KR20170053013A (en) | Data Virtualization System for Bigdata Analysis | |
| CN114817389A (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN110704422A (en) | Data query method, device, system, computer equipment and storage medium | |
| CN114500676A (en) | Information interaction method and device among industrial internet devices and storage medium | |
| CN117874033A (en) | Database sub-table method and device, electronic equipment and storage medium | |
| CN110688355A (en) | Method and device for changing container state | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN112463527A (en) | Data processing method, device, equipment, system and storage medium | |
| CN111488386A (en) | Data query method and device | |
| CN107463618B (en) | Index creating method and device | |
| CN115905313A (en) | MySQL big table association query system and method | |
| CN111475505B (en) | Data acquisition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210423 |