CN114297211A - Data online analysis system, method, equipment and storage medium - Google Patents
Data online analysis system, method, equipment and storage medium Download PDFInfo
- Publication number
- CN114297211A CN114297211A CN202111629940.3A CN202111629940A CN114297211A CN 114297211 A CN114297211 A CN 114297211A CN 202111629940 A CN202111629940 A CN 202111629940A CN 114297211 A CN114297211 A CN 114297211A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- information
- target change
- change information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data online analysis system, a method, equipment and a storage medium. The system comprises: the system comprises an information acquisition unit, a data synchronization unit and an online analysis processing engine; the information acquisition unit is used for acquiring target change information of the database and synchronizing the target change information to the data synchronization unit; the data synchronization unit is used for updating database synchronization data stored in the online analysis processing engine according to the target change information; and the online analysis processing engine is used for receiving the database query statement through the online analysis interface and executing corresponding analysis operation on the updated database synchronous data stored in the online analysis processing engine according to the database query statement. By the technical scheme, the changed data of the database can be synchronized to the online analysis processing engine in real time, real-time multidimensional analysis is carried out through query statement operation on the online analysis processing engine, and the increasingly complex business query requirements are met.
Description
Technical Field
The embodiment of the invention relates to the technical field of databases, in particular to a data online analysis system, a data online analysis method, data online analysis equipment and a storage medium.
Background
With the continued development of business intelligence, online analytical processing engines have been introduced to address the needs of multidimensional analysis in the areas of sales business reporting, marketing, management reporting, business process management, budgeting and forecasting, financial reporting, and the like.
Generally, the back-end database of the business system of a company is a traditional relational database, such as MySQL, and such a database is used as a transaction processing system and is not suitable for various complex queries in an online analysis scenario, so that an offline ETL tool is usually used to import the business database data into an online analysis processing engine in batches. The problem with this approach is that it does not meet the near real-time multidimensional analysis requirements.
Disclosure of Invention
Embodiments of the present invention provide a data online analysis system, method, device, and storage medium, so as to implement real-time synchronization of changed data of a database to an online analysis processing engine, perform real-time multidimensional analysis through query statement operations on the online analysis processing engine, and meet increasingly complex business query requirements.
In a first aspect, an embodiment of the present invention provides an online data analysis system, including: the system comprises an information acquisition unit, a data synchronization unit and an online analysis processing engine;
the information acquisition unit is used for acquiring target change information of a database and synchronizing the target change information to the data synchronization unit;
the data synchronization unit is used for updating database synchronization data stored in the online analysis processing engine according to the target change information;
and the online analysis processing engine is used for receiving a database query statement through an online analysis interface and executing corresponding analysis operation on the updated database synchronous data stored in the online analysis processing engine according to the database query statement.
Optionally, the system further includes: a collected information caching unit for caching the collected information,
the acquired information caching unit is used for storing a first reading position, and the first reading position is a position for reading the log information corresponding to the target change information from the database.
Optionally, the collected information caching unit is further configured to store a second reading position when the information collecting unit synchronizes the target change information to the data synchronizing unit through a data queue, where the second reading position is a position where the target change information is read from the data queue.
Optionally, the system further includes: the synchronization task monitoring unit is used for monitoring the synchronization task,
the synchronous task monitoring unit is used for calculating data delay between any two of the database, the acquired information caching unit and the online analysis processing engine, and prompting if the data delay is larger than a preset threshold value.
In a second aspect, an embodiment of the present invention further provides a data online analysis method, where the method includes:
acquiring target change information of a database through an information acquisition unit, and synchronizing the target change information to a data synchronization unit;
updating database synchronization data stored in an online analysis processing engine according to the target change information through the data synchronization unit;
receiving a database query statement through an online analysis interface of an online analysis processing engine, and executing corresponding analysis operation on updated database synchronous data stored in the online analysis processing engine according to the database query statement.
Optionally, the acquiring, by the information acquisition unit, target change information of the database includes:
collecting log information of a database in real time, wherein the log information comprises data change information in the database;
and acquiring attribute information of the target change information, and extracting the target change information from the log information based on the attribute information.
Optionally, synchronizing the target change information to a data synchronization unit includes:
and synchronizing the target change information to the data synchronization unit through a data queue.
Optionally, after synchronizing the target change information to the data synchronization unit, the method further includes:
updating a first reading position in an acquired information cache unit, wherein the first reading position is a position for reading log information corresponding to the target change information from the database;
and updating a second reading position in the acquired information cache unit, wherein the second reading position is a position for reading the target change information from the data queue.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the online data analysis method according to any one of the embodiments of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the online data analysis method according to any one of the embodiments of the present invention.
The embodiment of the invention provides a data online analysis system, which comprises: the system comprises an information acquisition unit, a data synchronization unit and an online analysis processing engine; the information acquisition unit is used for acquiring target change information of the database and synchronizing the target change information to the data synchronization unit; the data synchronization unit is used for updating database synchronization data stored in the online analysis processing engine according to the target change information; the online analysis processing engine is used for receiving the database query statement through the online analysis interface, executing corresponding analysis operation on updated database synchronous data stored in the online analysis processing engine according to the database query statement, synchronizing the changed data to the online analysis processing engine in real time, and performing real-time multidimensional analysis through the query statement operation on the online analysis processing engine, so that the problem that the database is taken as a transaction processing system and is not suitable for various complex queries in an online analysis scene is solved, and the requirement of increasingly complex business query is met.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a block diagram of an online data analysis system according to a first embodiment of the present invention;
FIG. 2 is a block diagram of another online data analysis system according to a first embodiment of the present invention;
FIG. 3 is a flow chart of a data online analysis method according to a second embodiment of the present invention;
FIG. 4 is a flow chart of a data online analysis method according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer device in the fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Example one
Fig. 1 is a schematic structural diagram of a data online analysis system according to an embodiment of the present invention, which is applicable to a situation of performing a real-time query operation on a database. As shown in fig. 1, the data online analysis system 1 specifically includes: an information acquisition unit 110, a data synchronization unit 120 and an online analysis processing engine 130;
the information acquisition unit 110 is used for acquiring target change information of the database 140 and synchronizing the target change information to the data synchronization unit;
a data synchronization unit 120, configured to update database synchronization data stored in the online analysis processing engine according to the target change information;
the online analysis processing engine 130 is configured to receive the database query statement through the online analysis interface 150, and perform a corresponding analysis operation on the updated database synchronization data stored in the online analysis processing engine according to the database query statement.
The database in the embodiment of the invention can be a relational database, rapid submission of things is realized in the relational database through logs, and the operation process and the operation content of the things are recorded; the change information of the database can be captured through the analysis of the log of the relational database, for example, the Binlog log of MySQL, and the archive log file of Oracle. Other databases that can provide change information are also possible.
Specifically, the database records the change information of the data in the database in a log mode, and therefore the acquisition unit is used for acquiring the log of the database and determining the target change information from the log. The target change information may determine the required target change information, such as a designated library or table, according to a manner of writing a configuration file in the database, and the target change information acquired from the database according to the configuration file needs to be converted into a target format and then sent to the data synchronization unit, for example, the target change information may be converted into a json format.
For example, before acquiring the target change information from the database, the log acquisition unit needs to acquire a log acquisition permission from the database, and the manner of acquiring the log acquisition permission may be completed by setting a corresponding configuration item in the configuration file. The manner of synchronizing the target change information to the data synchronization unit may be a manner of a data queue or a manner of publish/subscribe.
And after receiving the target change information, the data synchronization unit updates the database synchronization data stored in the online analysis processing engine according to the target change information, so that the database synchronization data stored in the online analysis processing engine and the data in the database are updated synchronously.
The online analysis processing engine is provided with an online analysis interface for receiving a database query statement, and can perform corresponding analysis operations, such as query operations and logic judgment operations, on the updated database synchronization data stored in the online analysis processing engine according to the database query statement.
In addition, requirements such as user authority, consumed time and consumed resources of the query SQL statement, and a query specification format of the SQL statement may be set for the database query statement input in the online analysis interface of the online analysis processing engine, so as to protect the stability of the online analysis processing engine.
The data online analysis system provided by the technical scheme of the embodiment comprises: the system comprises an information acquisition unit, a data synchronization unit and an online analysis processing engine; the information acquisition unit is used for acquiring target change information of the database and synchronizing the target change information to the data synchronization unit; the data synchronization unit is used for updating database synchronization data stored in the online analysis processing engine according to the target change information; the online analysis processing engine is used for receiving the database query statement through the online analysis interface, executing corresponding analysis operation on updated database synchronous data stored in the online analysis processing engine according to the database query statement, synchronizing the changed data to the online analysis processing engine in real time, performing real-time multidimensional analysis through the query statement operation on the online analysis processing engine, and meeting the increasingly complex business query requirement.
Optionally, as shown in fig. 2, the online data analysis system further includes: and the collected information caching unit 160 is configured to store a first reading position, where the log information corresponding to the target change information is read from the database.
Specifically, the database generates log information when data change occurs, and the information acquisition unit needs to acquire the log information from the database in real time and extract target change information from each piece of log information. Therefore, after acquiring the log information corresponding to the target change information each time, the position of the log information corresponding to the target change information read from the database is recorded as a first read position and stored in the acquired information cache unit, so that when the log information is read next time, the first read position is acquired from the acquired information cache unit, and the log information is continuously acquired from the recorded first read position.
Optionally, the collected information caching unit is further configured to store a second reading position when the information collecting unit synchronizes the target change information to the data synchronizing unit through the data queue, where the second reading position is a position where the target change information is read from the data queue.
Specifically, when the target change information is synchronized to the data synchronization unit in a data queue manner, a position at which the data synchronization unit reads the target change information in the data queue each time, that is, a second reading position, needs to be recorded, and the second reading position is stored in the collected information cache unit. And the second reading position is recorded and used for continuously acquiring the target change information from the data queue.
Optionally, as shown in fig. 2, the online data analysis system further includes: and the synchronous task monitoring unit 170 is used for calculating data delay between any two of the database, the acquired information caching unit and the online analysis processing engine, and prompting if the data delay is greater than a preset threshold value.
Specifically, in order to ensure that the online analysis processing engine in the online analysis system can synchronize data changes in the database in real time, the data queried by the online analysis processing engine is real-time data of the data, and it is necessary to ensure that the data synchronization delay between the database and the online analysis processing engine is within a preset range. Therefore, the synchronous task monitoring unit is arranged to monitor the data delay between any two of the database, the collected information caching unit and the online analysis processing engine.
And sending a prompt instruction to the prompt to perform delayed prompt by calling a prompt unit corresponding to the online analysis processing engine. The delay prompting mode may be an interface display mode, and the embodiment of the present invention is not limited.
Illustratively, the latest position corresponding to the log information of the database is obtained, data corresponding to the first reading position in the information cache unit is collected, a difference value between the latest position and the first reading position is determined as a first data delay, and if the first data delay is larger than a first preset threshold value, delay prompt is performed. And acquiring data corresponding to a first reading position in the acquired information cache unit and the position of log information corresponding to database synchronous data stored in the online analysis processing engine, determining a difference value between the latest position and the first reading position as a second data delay, and performing delay prompt if the second data delay is greater than a second preset threshold.
Example two
Fig. 3 is a flowchart of a data online analysis method according to a second embodiment of the present invention, where this embodiment is applicable to a situation where a real-time query operation is performed on a database, and the method may be executed by a data online analysis system according to the first embodiment of the present invention, and the system may be implemented in a software and/or hardware manner, as shown in fig. 3, the method specifically includes the following steps:
and S210, acquiring target change information of the database through the information acquisition unit, and synchronizing the target change information to the data synchronization unit.
Specifically, the database records the change information of the data in the database in a log mode, and therefore the acquisition unit is used for acquiring the log of the database and determining the target change information from the log. The target change information may determine the required target change information, such as a designated library or table, according to a manner of writing a configuration file in the database, and the target change information acquired from the database according to the configuration file needs to be converted into a target format and then sent to the data synchronization unit, for example, the target change information may be converted into a json format.
For example, the manner of synchronizing the target change information to the data synchronization unit may be a manner of a data queue or a manner of publish/subscribe.
And S220, updating the database synchronization data stored in the online analysis processing engine according to the target change information through the data synchronization unit.
Specifically, after receiving the target change information, the data synchronization unit updates the database synchronization data stored in the online analysis processing engine according to the target change information, so that the database synchronization data stored in the online analysis processing engine and the data in the database are updated synchronously.
And S230, receiving a database query statement through an online analysis interface of the online analysis processing engine, and executing corresponding analysis operation on the updated database synchronous data stored in the online analysis processing engine according to the database query statement.
Specifically, the online analysis processing engine is provided with an online analysis interface for receiving a database query statement, and corresponding analysis operations, such as query operations and logic judgment operations, can be performed on the updated database synchronization data stored in the online analysis processing engine according to the database query statement.
According to the technical scheme of the embodiment, the target change information of the database is acquired through the information acquisition unit, and the target change information is synchronized to the data synchronization unit; updating database synchronous data stored in the online analysis processing engine according to the target change information through a data synchronization unit; the online analysis processing engine receives the database query statement through an online analysis interface of the online analysis processing engine, executes corresponding analysis operation on updated database synchronous data stored in the online analysis processing engine according to the database query statement, can synchronize the changed data of the database to the online analysis processing engine in real time, performs real-time multidimensional analysis through the query statement operation on the online analysis processing engine, and meets the increasingly complex business query requirement.
Optionally, the acquiring, by the information acquisition unit, target change information of the database includes:
collecting log information of a database in real time, wherein the log information comprises data change information in the database;
and acquiring attribute information of the target change information, and extracting the target change information from the log information based on the attribute information.
The log information is information generated when the data of the database is changed, and is automatically stored in a table object in the MySql database, and the log information comprises data change information. The target change information is change information required by the query service in the log information, and can be set according to the requirements of users.
For example, the information collection unit obtains log information from a database in real time, and obtains attribute information of the target change information, such as information of a library or a table where the target change information is located, by writing a configuration file in the information collection unit. And analyzing the log information, and extracting target change information in the log information based on the attribute information.
EXAMPLE III
Fig. 4 is a flowchart of a data online analysis method according to a third embodiment of the present invention, where the present embodiment is optimized based on the foregoing embodiment, and the synchronizing of the target change information to the data synchronization unit in the present embodiment includes: and synchronizing the target change information to the data synchronization unit through the data queue.
As shown in fig. 4, the method specifically includes the following steps:
and S310, acquiring target change information of the database through an information acquisition unit.
S320, synchronizing the target change information to the data synchronization unit through the data queue.
Wherein a data queue is a container that holds data during transmission of the data, such as kafka or redis.
Specifically, the log collection unit serves as a producer to send target change information to the data queue, and the change data synchronization unit serves as a consumer to obtain the target change information from the data queue. When the speed or stability of the log collection unit as a producer and the data synchronization unit as a consumer in the data online analysis system are inconsistent, the difference between the two parties can be closed by adopting the data queue. Using the data queue, the producer side can return a response without waiting for the processing result to place the target change information into the data queue. When the data synchronization unit needs the target change information, the data synchronization unit automatically acquires the target change information from the data queue, so that the response speed and the stability of the system can be improved.
Illustratively, the data queue has a publishing function and a subscribing function, the information acquisition unit publishes the acquired target change information to the data queue, and the data synchronization unit waiting for subscription reads the target change information from the data queue in sequence.
And S330, updating the database synchronization data stored in the online analysis processing engine according to the target change information through the data synchronization unit.
S340, receiving a database query statement through an online analysis interface of the online analysis processing engine, and executing corresponding analysis operation on the updated database synchronous data stored in the online analysis processing engine according to the database query statement.
According to the technical scheme of the embodiment, the target change information of the database is acquired through the information acquisition unit, and the target change information is synchronized to the data synchronization unit through the data queue; updating database synchronous data stored in the online analysis processing engine according to the target change information through a data synchronization unit; the online analysis processing engine receives the database query statement through an online analysis interface of the online analysis processing engine, executes corresponding analysis operation on updated database synchronous data stored in the online analysis processing engine according to the database query statement, can synchronize the changed data of the database to the online analysis processing engine in real time, performs real-time multidimensional analysis through the query statement operation on the online analysis processing engine, and meets the increasingly complex business query requirement.
Optionally, after synchronizing the target change information to the data synchronization unit, the method further includes:
updating a first reading position in the acquired information cache unit, wherein the first reading position is a position for reading log information corresponding to the target change information from the database;
and updating a second reading position in the acquired information cache unit, wherein the second reading position is a position for reading the target change information from the data queue.
Specifically, after the target change information is synchronized to the data synchronization unit, the position of the log information corresponding to the target change information currently read from the database is acquired, and the first read position in the collected information cache unit is updated. And acquiring the position of the current target change information read from the data queue, and updating the second read position in the acquired information cache unit.
Example four
Fig. 5 is a block diagram of a computer apparatus according to a fourth embodiment of the present invention, as shown in fig. 5, the computer apparatus includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of the processors 410 in the computer device may be one or more, and one processor 410 is taken as an example in fig. 5; the processor 410, the memory 420, the input device 430 and the output device 440 in the computer apparatus may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 5.
The memory 420 serves as a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data online analysis method in the embodiment of the present invention (for example, the information acquisition unit 110, the data synchronization unit 120, and the online analysis processing engine 130 in the data online analysis apparatus). The processor 410 executes various functional applications and data processing of the computer device by executing software programs, instructions and modules stored in the memory 420, that is, the above-described data online analysis method is implemented.
The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 420 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 420 may further include memory located remotely from processor 410, which may be connected to a computer device through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 430 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the computer apparatus. The output device 440 may include a display device such as a display screen.
EXAMPLE five
The fifth embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data online analysis method provided in all the embodiments of the present invention: acquiring target change information of a database through an information acquisition unit, and synchronizing the target change information to a data synchronization unit; updating database synchronization data stored in an online analysis processing engine according to the target change information through the data synchronization unit; receiving a database query statement through an online analysis interface of an online analysis processing engine, and executing corresponding analysis operation on updated database synchronous data stored in the online analysis processing engine according to the database query statement.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. An online data analysis system, comprising: the system comprises an information acquisition unit, a data synchronization unit and an online analysis processing engine;
the information acquisition unit is used for acquiring target change information of a database and synchronizing the target change information to the data synchronization unit;
the data synchronization unit is used for updating database synchronization data stored in the online analysis processing engine according to the target change information;
and the online analysis processing engine is used for receiving a database query statement through an online analysis interface and executing corresponding analysis operation on the updated database synchronous data stored in the online analysis processing engine according to the database query statement.
2. The system of claim 1, further comprising: a collected information caching unit for caching the collected information,
the acquired information caching unit is used for storing a first reading position, and the first reading position is a position for reading the log information corresponding to the target change information from the database.
3. The system of claim 2, wherein the collected information buffer unit is further configured to store a second reading position when the information collecting unit synchronizes the target change information to the data synchronizing unit through a data queue, and the second reading position is a position where the target change information is read from the data queue.
4. The system of claim 2, further comprising: the synchronization task monitoring unit is used for monitoring the synchronization task,
the synchronous task monitoring unit is used for calculating data delay between any two of the database, the acquired information caching unit and the online analysis processing engine, and prompting if the data delay is larger than a preset threshold value.
5. A method for online data analysis, applied to the system of claim 1, the method comprising:
acquiring target change information of a database through an information acquisition unit, and synchronizing the target change information to a data synchronization unit;
updating database synchronization data stored in an online analysis processing engine according to the target change information through the data synchronization unit;
receiving a database query statement through an online analysis interface of an online analysis processing engine, and executing corresponding analysis operation on updated database synchronous data stored in the online analysis processing engine according to the database query statement.
6. The method of claim 5, wherein collecting the target change information of the database by the information collecting unit comprises:
collecting log information of a database in real time, wherein the log information comprises data change information in the database;
and acquiring attribute information of the target change information, and extracting the target change information from the log information based on the attribute information.
7. The method of claim 5, wherein synchronizing the target change information to a data synchronization unit comprises:
and synchronizing the target change information to the data synchronization unit through a data queue.
8. The method of claim 7, further comprising, after synchronizing the target change information to a data synchronization unit:
updating a first reading position in an acquired information cache unit, wherein the first reading position is a position for reading log information corresponding to the target change information from the database;
and updating a second reading position in the acquired information cache unit, wherein the second reading position is a position for reading the target change information from the data queue.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 5-8 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 5-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111629940.3A CN114297211A (en) | 2021-12-28 | 2021-12-28 | Data online analysis system, method, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111629940.3A CN114297211A (en) | 2021-12-28 | 2021-12-28 | Data online analysis system, method, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114297211A true CN114297211A (en) | 2022-04-08 |
Family
ID=80972135
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111629940.3A Pending CN114297211A (en) | 2021-12-28 | 2021-12-28 | Data online analysis system, method, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114297211A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114647703A (en) * | 2022-05-23 | 2022-06-21 | 武汉中科通达高新技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2021
- 2021-12-28 CN CN202111629940.3A patent/CN114297211A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114647703A (en) * | 2022-05-23 | 2022-06-21 | 武汉中科通达高新技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114647703B (en) * | 2022-05-23 | 2022-08-30 | 武汉中科通达高新技术股份有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109034993B (en) | Account checking method, account checking equipment, account checking system and computer readable storage medium | |
| CN110147398B (en) | Data processing method, device, medium and electronic equipment | |
| CN108920698B (en) | Data synchronization method, device, system, medium and electronic equipment | |
| US9152691B2 (en) | System and method for performing set operations with defined sketch accuracy distribution | |
| CN111597257A (en) | Database synchronization method and device, storage medium and terminal | |
| CN110569214B (en) | Index construction method and device for log file and electronic equipment | |
| CN108958959B (en) | Method and device for detecting hive data table | |
| CN110505495B (en) | Multimedia resource frame extraction method, device, server and storage medium | |
| CN110928903B (en) | Data extraction method and device, equipment and storage medium | |
| CN111339171A (en) | Data query method, device and equipment | |
| CN112559475A (en) | Data real-time capturing and transmitting method and system | |
| CN113886485A (en) | Data processing method, device, electronic equipment, system and storage medium | |
| CN112948397A (en) | Data processing system, method, device and storage medium | |
| CN113282611A (en) | Method and device for synchronizing stream data, computer equipment and storage medium | |
| CN112416991A (en) | Data processing method and device and storage medium | |
| CN115757626A (en) | Data quality detection method and device, electronic equipment and storage medium | |
| CN114297211A (en) | Data online analysis system, method, equipment and storage medium | |
| CN117271584A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN112463527A (en) | Data processing method, device, equipment, system and storage medium | |
| CN114443599A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN111488386A (en) | Data query method and device | |
| CN112835863A (en) | Processing method and processing device of operation log | |
| EP4280565A1 (en) | Sample message processing method and apparatus | |
| CN116186053A (en) | Data processing method, device and storage medium | |
| CN111291127B (en) | Data synchronization method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |