CN115470288A - Data application method based on multi-source copying, storage medium and electronic equipment - Google Patents

Data application method based on multi-source copying, storage medium and electronic equipment Download PDF

Info

Publication number
CN115470288A
CN115470288A CN202110655303.7A CN202110655303A CN115470288A CN 115470288 A CN115470288 A CN 115470288A CN 202110655303 A CN202110655303 A CN 202110655303A CN 115470288 A CN115470288 A CN 115470288A
Authority
CN
China
Prior art keywords
library
data
slave
master
application method
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110655303.7A
Other languages
Chinese (zh)
Inventor
王君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Pateo Network Technology Service Co Ltd
Original Assignee
Shanghai Pateo Network Technology Service Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Pateo Network Technology Service Co Ltd filed Critical Shanghai Pateo Network Technology Service Co Ltd
Priority to CN202110655303.7A priority Critical patent/CN115470288A/en
Publication of CN115470288A publication Critical patent/CN115470288A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275Synchronous replication
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2457Query processing with adaptation to user needs

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a data application method based on multi-source copying, a storage medium and electronic equipment, wherein the data application method based on multi-source copying comprises the following steps: determining basic data in a master library; wherein the base data refers to data for multi-source replication; creating a slave library, constructing multi-source replication between the master library and the slave library, performing data synchronization of the basic data between the master library and the slave library, and generating a slave library data set; generating an intermediate data table from the slave library data set; generating an application data table by using the intermediate data table in combination with service requirements; and performing service analysis by using the intermediate data table and/or the application data table. The invention can improve the data real-time performance and the data query efficiency and reduce the load of the service library, thereby better ensuring the normal operation of the service library.

Description

Data application method based on multi-source copying, storage medium and electronic equipment
Technical Field
The invention belongs to the technical field of data application, relates to a data application method, and particularly relates to a data application method based on multi-source copying, a storage medium and an electronic device.
Background
In the big data era, how to correctly use and analyze data becomes especially important. In a data processing mode, a server is used for timing tasks, a base table is specified in MySQLdb, the whole data is imported from one instance to another instance, and then the whole data is inquired for a data table. MySQL is a small relational database management system, and is widely applied to small and medium-sized internet websites as a website database due to the characteristics of small volume, high speed, low overall ownership cost and particularly open source codes.
On the one hand, however, as the data volume increases, the data import time becomes longer, data query cannot be performed during the import process, data delay increases, data real-time is lower, and even reading and writing of other service libraries of the instance are affected. On the other hand, if the full data analysis is performed for each Query, as the data volume increases, the efficiency of Structured Query Language (SQL) becomes lower, and therefore, the user experience is affected. In the prior art, canal can be used to extract data from one instance to another instance and then perform operations such as data aggregation. The main purpose of canal is to provide incremental data subscription and consumption based on MySQL database incremental log parsing, and the defects of the method are as follows: canal is not a database-owned function, which increases maintenance cost, and data analysis is still performed in the business repository, which adds a certain load to the business repository.
Therefore, how to provide a data application method, a storage medium, and an electronic device based on multi-source replication to solve the defects that the prior art cannot improve the business data analysis efficiency and reduce the load of the business database, and the like, is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
In view of the above-mentioned shortcomings of the prior art, the present invention is directed to a data application method, a storage medium and an electronic device based on multi-source replication, which are advantageous in that it is possible to achieve reduction of load of a business database while improving business data analysis efficiency.
Another object of the present invention is to provide a data application method, a storage medium, and an electronic device based on multi-source replication, which have the advantages of improving data real-time performance and data query efficiency, and further better ensuring normal operation of a business database.
Another object of the present invention is to provide a data application method, a storage medium, and an electronic device based on multi-source replication, which have the advantages of performing data synchronization between a master library and a slave library through multi-source replication established between the master library and the slave library, and separating business analysis from a business database.
Another object of the present invention is to provide a data application method, a storage medium, and an electronic device based on multi-source replication, which are advantageous in that a user is helped to perform business analysis more flexibly and efficiently through the creation of an intermediate data table and an application data table.
Another object of the present invention is to provide a data application method, a storage medium and an electronic device based on multi-source replication, which have the advantage that effective data to be analyzed can be provided for users more conveniently for different data analysis services.
To achieve the above and other related objects, an aspect of the present invention provides a data application method based on multi-source replication, including the steps of: determining basic data in a main library; wherein the base data refers to data for multi-source replication; creating a slave library, constructing multi-source replication between the master library and the slave library, performing data synchronization of the basic data between the master library and the slave library, and generating a slave library data set; generating an intermediate data table from the slave library data set; generating an application data table by using the intermediate data table in combination with service requirements; and performing service analysis by using the intermediate data table and/or the application data table.
To achieve the above and other related objects, another aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the multi-source replication based data application method.
To achieve the above and other related objects, a last aspect of the present invention provides an electronic device, comprising: a processor and a memory; the memory is used for storing a computer program, and the processor is used for executing the computer program stored by the memory so as to enable the electronic equipment to execute the multi-source copy-based data application method.
Drawings
FIG. 1 is a schematic flow chart diagram illustrating a multi-source replication-based data application method according to an embodiment of the present invention.
Fig. 2 shows a flow chart of multi-source copy building of the multi-source copy-based data application method according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating data synchronization of a multi-source replication-based data application method according to an embodiment of the present invention.
FIG. 4 is a flowchart illustrating the generation of a data table according to an embodiment of the multi-source replication-based data application method of the present invention.
FIG. 5 is a flow chart illustrating the creation of a slave library according to an embodiment of the multi-source replication-based data application method of the present invention.
FIG. 6 is a flow chart illustrating a business analysis of a multi-source replication-based data application method according to an embodiment of the present invention.
Fig. 7 is a schematic structural connection diagram of an electronic device according to an embodiment of the invention.
Description of the element reference numerals
7. Electronic device
71. Processor with a memory for storing a plurality of data
72. Memory device
S11 to S15
S121 to S122
S122A to S122D
S141 to S146 steps
S61-S64 steps
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It should be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the drawings only show the components related to the present invention rather than being drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of each component in actual implementation may be changed arbitrarily, and the layout of the components may be more complicated.
The data application method, the storage medium and the electronic equipment based on multi-source copying can improve the real-time performance and the data query efficiency of data and reduce the load of a service library, thereby better ensuring the normal operation of the service library.
The principles and embodiments of a data application method, a storage medium and an electronic device based on multi-source copy according to the present embodiment will be described in detail below with reference to fig. 1 to 7, so that those skilled in the art can understand the data application method, the storage medium and the electronic device based on multi-source copy without creative work.
Referring to fig. 1, a schematic flow chart of a multi-source copy-based data application method according to an embodiment of the invention is shown.
As shown in fig. 1, the data application method based on multi-source replication specifically includes the following steps:
s11, determining basic data in a master library; wherein the base data refers to data for multi-source replication.
Specifically, a specific library is derived from the corebase and the base cluster, i.e., the corebase core library and the base core library, for example, dbdevice library in the specified corebase core library, msbluetoothth library in the base core library. The data in the dbdevice library and msblue library are the basic data in the determined master library.
It should be noted that, in addition to the corebase core library and the base core library, other databases such as tsp, cpsp, ivigeley and the like may be used in the present invention. The corebase and the base data have strong relevance and are all provided with relevant query tasks every day, so that the corebase core library and the base core library can be applied to data query service for illustration.
And S12, creating a slave library, building multi-source replication between the master library and the slave library, carrying out data synchronization of the basic data between the master library and the slave library, and generating a slave library data set.
Specifically, through MySQL multi-source replication, data in a specified library dbdevice library and a msbluetoothe library are imported into a slave library instance in real time, and therefore the problems that data cannot be accessed and data delay occurs during data import can be solved.
Further, when the service analysis is a data query service, performing data combined query on different data tables in different specified libraries through key fields such as a common device ID number, a user ID number, an order identification number and the like, so as to realize data combination and obtain the slave library data set.
Wherein the multi-source replication may use binary log-based replication or transaction-based replication. The two modes can ensure the data consistency on the aspect of data consistency. In the DB (Database) operation and maintenance aspect, when a master-slave DB model wants to create a slave library, which becomes a master-slave DB model, a copy of current data needs to be extracted. If the transaction-based replication is performed, data can be extracted from the original slave library, so that the write-in function of the master library is prevented from being influenced, and no pressure exists on the master library. Since each action in the master-slave library log will be given the same ID. If the method is based on the common binary replication, data extraction needs to be carried out on the master library, and certain influence is brought to the master library. In the log of the master library and the slave library, the ID corresponding to each operation behavior is different, and data must be extracted from the master library.
In an embodiment, after the step of creating the slave library and before the step of building the multi-source copy between the master library and the slave library, the multi-source copy-based data application method includes the following steps: and respectively configuring a master library and a slave library so as to build multi-source copy between the master library and the slave library after the master library and the slave library are configured. In another embodiment, the step of configuring the master library and the step of configuring the slave library may also be preset before the multi-source replication-based data application method is executed.
Specifically, the master library configuration includes the steps of:
(1) And configuring the node identification code of the master library.
In practical applications, the configuration code is: server-id =1. The requirements set by the master database node ID are as follows: and need to be unique within a cluster.
(2) And configuring the format, storage directory and name of the log in the master library.
In practical application, the log format configuration code is: binlog _ format = row; the log stores a directory and a name log _ bin =/data/mysql/mysql-bin. The log format is stored in a row format, and the log is stored in a data-mysql path and named mysql-bin.
(3) And configuring a storage mode of the master library and the slave library for copying information.
In practical application, the main library copy information configuration code is: master _ info _ repositor = TABLE; copying the information configuration code from the library as: relay _ log _ info _ repositor = TABLE.
Specifically, the slave library configuration comprises the following steps:
(1) And configuring the node identification code of the slave library.
In practical application, the configuration code of the node identification code is: server-ID =3, and the requirements set from the library database node ID are: and need to be unique within a cluster.
(2) And configuring the storage directory and the name of the relay log in the slave library.
In practical application, the storage directory of the relay log and the configuration code of the name of the relay log are as follows: relay _ log =/data/mysql/mysql-relay-bin. The relay log is stored under a data-mysql path and named mysql-relay-bin.
(3) And configuring the slave library to be in a read-only mode.
In practical applications, the configuration code of the read-only mode is: read _ only =1.
(4) And configuring a database name of data required to be synchronized with the master library for the slave library.
In practical applications, the configuration code of the database name is: replicate-do-db = xxx, where xxx represents the binlog log in the slave bank specifying which banks to synchronize, and if there are multiple banks of data to synchronize, multiple rows are configured.
Referring to fig. 2, a flow chart of multi-source copy building of an embodiment of the multi-source copy-based data application method of the present invention is shown. As shown in fig. 2, S12 includes the steps of:
and S121, the slave library is connected with the master library through a master-slave synchronous account and accesses the master library.
Specifically, the slave library accesses the master library through the master-slave synchronous account, and in the process of synchronizing data, the FOR CHANNEL parameter is used FOR distinguishing different master libraries, such as: FOR CHANNEL 'Master _1' is used to identify Master library 1.
And S122, detecting a master-slave state, and synchronizing the data of the basic data between the master library and the slave library when the master-slave state is normal.
In one embodiment, the master-slave status comprises normal operation of the slave library input/output and normal operation of the slave library database function.
Specifically, the code "show Slave status \ G" is used to confirm the master-Slave status, when the parameters in the program show Slave _ IO _ Running: YES and Slave _ SQL _ Running: YES, it indicates that the synchronization is normal, and data addition, deletion, and modification can be performed in the master library to see whether the Slave libraries are synchronized.
Referring to fig. 3, a data synchronization flow chart of a multi-source replication-based data application method according to an embodiment of the invention is shown. The multi-source copy can implement multi-master-slave copy, distinguish different master libraries according to the ID (Identity document) of the DB, and further implement data synchronization with the slave libraries, as shown in fig. 3, S122 includes the following steps:
S122A, recording the data change of the master library in a binary log.
Specifically, the Master library Master records data changes into a binary log (binary log), that is, a file specified by a configuration file log-bin, and the recorded information generated by the data changes is called a binary log event (binary log events).
S122B, the slave library reads the binary log events in the binary log through the input and output threads.
Specifically, the Slave library Slave reads the binary log event entries in the Master library Master through the I/O thread by using the Master-Slave synchronous account.
S122C, the slave library writes the binary log event into a relay log.
Specifically, the Slave library Slave writes the binary log events in the Master library into the relay log (relay log) of the Slave library.
And S122D, executing the event information in the relay log by the slave library, and generating a new slave library data set.
Specifically, the Slave library Slave redos the events in the relay log, executes the event information in the relay log one by one locally, and completes the local storage of the data, thereby realizing the reflection of the data change of the master library to the own data of the Slave library and realizing the data replay.
And S13, generating an intermediate data table according to the slave database data set.
Specifically, a timed task is completed through a timed execution task command (crontab command) in Linux or an event in MySQL, and original basic data are synthesized into a related intermediate result table or an application data table at a fixed time according to specific requirements such as timed time. The intermediate result table and the application data table are relative concepts, and when the intermediate result table is not further analyzed, the intermediate result table can be called as the application data table.
And S14, generating an application data table by using the intermediate data table in combination with the service requirement.
Specifically, in order to implement deep analysis of the service, aiming at the requirement that the real-time requirement is not very high, an intermediate result table or an application data table can be derived regularly, wherein the application data table is a final result table generated after statistical calculation is performed on the basis of the intermediate result table. Therefore, the problem of low SQL efficiency after the data volume is increased can be solved. For example, report type query is performed on a part of T +1 or in units of hours, where T represents a reference time point, and T +1 represents data corresponding to a field of time, and data in alternate days are queried, for example, only 5 months and 2 days, 5 days, 4 days, and 5 months and 6 days \8230, and corresponding report data.
Please refer to fig. 4, which illustrates a flowchart of generating a data table according to an embodiment of the data application method based on multi-source replication of the present invention. As shown in fig. 4, in the order query service, the intermediate data table is a first order data table, and the application data table is a second order data table. S14 includes the steps of:
and S141, presetting a query time period according to the query requirement.
Specifically, if it is determined that a query request is to query an order and detailed contents that have been completed in about 3 months, the query time period is set to 3 months.
S142, screening all orders in the query time period from the slave database set.
Specifically, all orders in the order sheet and order detail sheet will be presented, called out from the order sheet and order detail sheet in the library dataset. For example, the order form is a profile of order information and only includes information such as an order number and an order time, and the order detail form includes detailed order information such as an order purchase item name, a price, and a quantity in addition to field information such as an order number that can identify the order.
S143, removing unfinished orders and abandoned orders in the screened order data set.
Specifically, if the called order form or order detail form has information of transaction intermediate processes such as pending payment, pending delivery, pending receipt and the like, the order form is regarded as an unfinished order form and deleted; if the called order form or order detail form has abnormal conditions such as order cancel, unpaid order form, return goods, etc., the order form is regarded as a waste order form and deleted.
And S144, taking the cleared order data set as the first order data table.
Specifically, after the incomplete orders and the abandoned orders are cleared, all normally completed orders in the order table and all normally completed orders in the order detail table are combined together according to fields common to the two, and the formed data table is used as a first order data table. Each order record in the first order data table includes details of the order, including information such as the order number, the name of the item purchased by the order, the price, the quantity, etc.
S145, determining an analysis field; the analysis field includes at least one of an order amount and an order time.
Specifically, the analysis fields are determined as the order amount and the order time, and the order amount is summarized in time intervals according to the price recorded by each order.
S146, carrying out statistical calculation on related orders in the first order data table according to the analysis field to generate the second order data table.
Specifically, the order amount of 1-3 months is found and summarized to obtain the total order amount of the first quarter; summarizing the searched order sums of the months 4-6 to obtain the total order sum of the second quarter; the order amount of 7-9 months is found and summarized to obtain the total order amount of the third quarter; and summarizing the searched order sums of the months 10-12 to obtain the total order sum of the fourth quarter. Thus, the correspondence of each quarter to the total amount of the order forms a second order data table.
In practical application, the determined analysis field can be any field in the order record besides the order amount and the order time, and the first order data table is subjected to secondary processing and analysis through the attributes of the field and other fields, so that data information which cannot be visually presented by the first order data table is obtained through statistical analysis. For example, the analysis field may also be a user name, and the total amount of the corresponding order generated by different users in this year is analyzed as the second order data table.
It should be noted that, the order of the step of removing the incomplete order and the waste order and the step of combining the order table are not limited in the present invention, and the incomplete order and the waste order may be removed first to form a slave library data set, and then the order table may be combined to form a first order data table, or the order may be combined first to form a slave library data set, and then the incomplete order and the waste order may be removed to form the first order data table.
And S15, performing service analysis by using the intermediate data table and/or the application data table.
Referring to fig. 5, a flow chart of creating a slave library according to an embodiment of the multi-source copy-based data application method of the present invention is shown. As shown in fig. 5, a case is presented in which 1 slave library is created from 4 master libraries, and multi-source replication is performed. Respectively determining basic data in a master library master1, a master library master2, a master library master3 and a master library master 4; and creating a slave library slave, constructing multi-source replication among the master library master1, the master library master2, the master library master3, the master library master4 and the slave library slave, and synchronizing the data of the basic data among the master library master1, the master library master2, the master library master3, the master library master4 and the slave library slave.
Referring to fig. 6, a business analysis flowchart of a multi-source replication-based data application method according to an embodiment of the invention is shown. As shown in fig. 6, taking the master library and slave library architecture shown in fig. 5 as an example, the process of data query, i.e., business analysis, is described in detail, and specifically includes the following steps:
s61, backing up the data of the designated database from different master databases.
Specifically, the data of the designated library in the master library master1, master library master2, master library master3, master library master4 are backed up respectively.
S62, creating a slave library, recovering master data, and building copy synchronization.
Specifically, a slave library slave is created, data of a designated library in a master library master1, a master library master2, a master library master3 and a master library master4 are restored in the slave library slave, and copy synchronization is established among the slave library slave, the master library master1, the master library master2, the master library master3 and the master library master4 so as to change and synchronize the data of the master library into the slave library in real time.
And S63, according to the requirement, obtaining an intermediate data table or an application data table from the service raw data.
Specifically, an order form corresponding to a certain service is used as service raw data, and if a male user order is inquired from the service raw data according to the requirement, an intermediate data form is generated according to the inquired male user order; and if the demand is the total amount of the order form of each male user, generating an application data table according to the counted total amount information of the order form of each male user.
And S64, the data query directly passes through the intermediate data table or the application data table of the new slave instance, and the query efficiency is improved.
Specifically, the details of the personal order of the male user a can be queried in the intermediate data table, and the total amount of the order corresponding to the male user a can be queried in the application data table.
Compared with the prior art, the method can realize quasi-real-time data import of the master library into the slave library. Specifically, for example, data in the corebase and the base are imported into the slave library, in the existing data import process, the corebase data is periodically imported into the base library, and the process involves the following processes: firstly, exporting corebase data, and storing the corebase data into a server where a base library is located in a file form; then, removing the corebase data in the base library, wherein the process relates to table deletion operation; and finally, importing the new corebase data into the base library, wherein the process relates to table building and data importing operation. In the above steps, the corebase data in the base library is cleaned and re-imported, and cannot be accessed, and because the corebase data is imported by a timing task, certain data delay exists, for example, if the data delay is once in 5 minutes, the data delay is 5 minutes. In the invention, data in corebase and base are imported into a slave library in quasi-real time, and the delay is generally 1 to 2 seconds.
The protection scope of the multi-source copy-based data application method is not limited to the execution sequence of the steps listed in this embodiment, and all the schemes of adding, subtracting, and replacing the steps in the prior art according to the principles of the present invention are included in the protection scope of the present invention.
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the multi-source replication-based data application method.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the above method embodiments may be performed by hardware associated with a computer program. The aforementioned computer program may be stored in a computer readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned computer-readable storage medium comprises: various computer storage media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Please refer to fig. 7, which is a schematic structural connection diagram of an electronic device according to an embodiment of the present invention. As shown in fig. 7, the present embodiment provides an electronic device 7, which specifically includes: a processor 71 and a memory 72; the memory 72 is configured to store a computer program, and the processor 71 is configured to execute the computer program stored in the memory 72, so as to enable the electronic device 7 to execute the steps of the multi-source copy-based data application method.
The Processor 71 may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware component.
The Memory 72 may include a Random Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory.
In practical applications, the electronic device may be a computer that includes components such as memory, a memory controller, one or more processing units (CPU), peripheral interfaces, RF circuitry, audio circuitry, speakers, microphone, input/output (I/O) subsystems, a display screen, other output or control devices, and external ports; the computer includes, but is not limited to, a Personal computer such as a desktop computer, a notebook computer, a tablet computer, a smart phone, a smart television, a Personal Digital Assistant (PDA), and the like, and the electronic device may also be a vehicle end or smart glasses, a smart watch, or other wearable devices. In other embodiments, the electronic device may also be a server, where the server may be arranged on one or more entity servers according to various factors such as functions and loads, or may be a cloud server formed by a distributed or centralized server cluster, and this embodiment is not limited.
In summary, the data application method, the storage medium and the electronic device based on multi-source replication can improve the business data analysis efficiency and reduce the load of the business database. The data real-time performance and the data query efficiency can be improved, and the normal operation of the service library can be better guaranteed. And data synchronization is carried out between the master library and the slave library through multi-source replication built between the master library and the slave library, so that business analysis is separated from the business database. And the user is helped to more flexibly and effectively analyze the service by establishing the intermediate data table and the application data table. Aiming at different data analysis services, effective data to be analyzed can be provided for users more conveniently. The invention effectively overcomes various defects in the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.

Claims (10)

1. A data application method based on multi-source replication is characterized by comprising the following steps:
determining basic data in a master library; wherein the base data refers to data for multi-source replication;
creating a slave library, constructing multi-source replication between the master library and the slave library, performing data synchronization of the basic data between the master library and the slave library, and generating a slave library data set;
generating an intermediate data table from the slave library data set;
generating an application data table by using the intermediate data table in combination with service requirements;
and performing service analysis by using the intermediate data table and/or the application data table.
2. The multi-source replication-based data application method according to claim 1, after the step of creating a slave library and before the step of building a multi-source replication between the master library and the slave library, the multi-source replication-based data application method comprising the steps of:
and respectively configuring a master library and a slave library so as to build multi-source copy between the master library and the slave library after the master library and the slave library are configured.
3. The multi-source replication-based data application method of claim 2, the master library configuration comprising the steps of:
configuring a node identification code of the master library;
configuring the format, storage catalog and name of the log in the master library;
and configuring a storage mode of the master library and the slave library for copying information.
4. The multi-source replication-based data application method of claim 2, the slave library configuration comprising the steps of:
configuring a node identification code of the slave library;
configuring a storage directory and a name of the relay log in the slave library;
configuring the slave library to be in a read-only mode;
and configuring a database name of data needing to be synchronized with the master library for the slave library.
5. The multi-source replication-based data application method of claim 1, wherein the building of multi-source replication between the master library and the slave library comprises the following steps:
the slave library is connected with the master library through a master-slave synchronous account and accesses the master library;
and detecting a master-slave state, and carrying out data synchronization of the basic data between the master library and the slave library when the master-slave state is normal.
6. The multi-source replication-based data application method of claim 5, wherein the data synchronization of the base data between the master library and the slave library comprises the following steps:
recording data changes of the master library in a binary log;
the slave library reads binary log events in the binary log through an input and output thread;
the slave library writes the binary log event into a relay log;
and the slave library executes the event information in the relay log to generate a new slave library data set.
7. The multi-source replication-based data application method of claim 5, wherein the master-slave state is normal and comprises normal operation of input and output from a library and normal operation of database functions of the slave library.
8. The multi-source replication-based data application method of claim 1, wherein in an order query service, the intermediate data table is a first order data table, and the application data table is a second order data table; the method for generating the application data table by using the intermediate data table in combination with the service requirement comprises the following steps:
presetting a query time period according to query requirements;
screening out all orders in the query time period from the slave database set;
removing unfinished orders and abandoned orders in the screened order data set;
taking the cleared order data set as the first order data table;
determining an analysis field; the analysis field comprises at least one of order amount and order time;
and carrying out statistical calculation on related orders in the first order data table according to the analysis field to generate a second order data table.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the multi-source replication-based data application method of any one of claims 1 to 8.
10. An electronic device, comprising: a processor and a memory;
the memory is used for storing a computer program, and the processor is used for executing the computer program stored by the memory to enable the electronic equipment to execute the multi-source copy-based data application method of any one of claims 1 to 8.
CN202110655303.7A 2021-06-11 2021-06-11 Data application method based on multi-source copying, storage medium and electronic equipment Pending CN115470288A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110655303.7A CN115470288A (en) 2021-06-11 2021-06-11 Data application method based on multi-source copying, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110655303.7A CN115470288A (en) 2021-06-11 2021-06-11 Data application method based on multi-source copying, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN115470288A true CN115470288A (en) 2022-12-13

Family

ID=84364923

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110655303.7A Pending CN115470288A (en) 2021-06-11 2021-06-11 Data application method based on multi-source copying, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN115470288A (en)

Similar Documents

Publication Publication Date Title
CN107835983B (en) Backup and restore in distributed databases using consistent database snapshots
Plattner A common database approach for OLTP and OLAP using an in-memory column database
US8924365B2 (en) System and method for range search over distributive storage systems
CN107209704A (en) Detect the write-in lost
US10437853B2 (en) Tracking data replication and discrepancies in incremental data audits
CN111651519B (en) Data synchronization method, data synchronization device, electronic equipment and storage medium
Yang et al. F1 Lightning: HTAP as a Service
WO2019109854A1 (en) Data processing method and device for distributed database, storage medium, and electronic device
US11372569B2 (en) De-duplication in master data management
US20170270153A1 (en) Real-time incremental data audits
KR20190063835A (en) System for processing real-time data modification of in-memory database
CN114329096A (en) Method and system for processing native map database
US11500738B2 (en) Tagging application resources for snapshot capability-aware discovery
US20220138167A1 (en) View tables framework
US11436089B2 (en) Identifying database backup copy chaining
US20220121524A1 (en) Identifying database archive log dependency and backup copy recoverability
US20210334165A1 (en) Snapshot capability-aware discovery of tagged application resources
Pandagale et al. Hadoop-HBase for finding association rules using Apriori MapReduce algorithm
CN115470288A (en) Data application method based on multi-source copying, storage medium and electronic equipment
Haug Bad big data science
CN114490865A (en) Database synchronization method, device, equipment and computer storage medium
CN114661693A (en) Data auditing realization method, storage medium, electronic equipment and system
CN113886505A (en) Management system for realizing dynamic modeling based on search engine and relational database
JP6680897B2 (en) Computer system and analysis source data management method
CN115878563B (en) Method for realizing directory-level snapshot of distributed file system and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination