CN110019527B - Slave library reading method, related device and equipment - Google Patents
Slave library reading method, related device and equipment Download PDFInfo
- Publication number
- CN110019527B CN110019527B CN201711391348.8A CN201711391348A CN110019527B CN 110019527 B CN110019527 B CN 110019527B CN 201711391348 A CN201711391348 A CN 201711391348A CN 110019527 B CN110019527 B CN 110019527B
- Authority
- CN
- China
- Prior art keywords
- library
- slave
- read
- transaction
- reading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a slave library reading method, which comprises the following steps: starting a master library and a slave library to perform data synchronization; receiving a read request for a slave bank in a case where the slave bank is allowed to be read while data synchronization is performed by a master bank and the slave bank; and for the read request, reading the slave library according to the recorded state of the transaction in the underlying storage engine, and outputting a read result. The invention also discloses a slave library reading device and equipment, which solve the technical problems that the delay of the request is obviously increased, the QPS is reduced and a plurality of slow logs appear on the slave library because the strength of the global lock is too large.
Description
Technical Field
The present invention relates to the field of computers, and in particular, to a method for reading from a library, and related apparatus and devices.
Background
A Database (Database) is a repository that organizes, stores, and manages data according to a data structure; with the development of information technology and markets, particularly after the nineties of the twentieth century, data management is no longer just storing and managing data, but is turning into various data management ways required by users. Databases are of many types, ranging from the simplest of tables that store a variety of data to large database systems that are capable of mass data storage.
Due to the consideration of disaster tolerance, load balancing, data centralization and distribution, etc., the databases in the prior art generally establish master and slave databases, i.e., a master database and a slave database. Currently, in the process of data synchronization of the master library and the slave library, in order to avoid factors such as dirty data reading of the slave library, a global lock is often applied to the slave library to control the unreadable slave library in the period.
However, the strength of the global lock is too large, so that if a slave bank receives a read request during the data synchronization of the master bank and the slave bank, the read request may be blocked until the data synchronization is finished, so that the delay of the request is obviously increased, the Query Per Second (Query Per Second, QPS) is reduced, many slow logs may appear on the slave bank, and if serious, the normal service of the service may be affected.
Disclosure of Invention
The technical problem to be solved by the embodiments of the present invention is to provide a slave library reading method, a slave library reading apparatus, a slave library reading device, and a computer readable storage medium, which solve the technical problems that a request delay is significantly increased, a QPS is decreased, and a lot of slow logs appear on a slave library due to too large strength of a global lock.
In order to solve the above technical problem, a first aspect of an embodiment of the present invention discloses a method for reading from a library, including:
starting a master library and a slave library to perform data synchronization;
receiving a read request for a slave bank in a case where the slave bank is allowed to be read while data synchronization is performed by a master bank and the slave bank;
and for the read request, reading the slave library according to the recorded state of the transaction in the underlying storage engine, and outputting a read result.
A second aspect of embodiments of the present invention discloses a device for reading from a library, comprising means for performing the method of the first aspect as described above.
A third aspect of the embodiments of the present invention discloses a device for reading from a library, including a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, where the memory is used to store an application program code, and the processor is configured to call the program code to execute the method according to the first aspect.
A fourth aspect of embodiments of the present invention discloses a computer-readable storage medium storing a computer program comprising program instructions that, when executed by a processor, cause the processor to perform the method of the first aspect as described above.
By implementing the embodiment of the invention, when the master library and the slave libraries are started to carry out data synchronization, a reading request for the slave libraries is received under the condition that the slave libraries are allowed to be read when the master library and the slave libraries carry out data synchronization; aiming at the read request, the slave library is read according to the recorded state of the transaction in the bottom storage engine, and a read result is output, namely, the global lock is removed when the master library and the slave library carry out data synchronization, the technical problems that the request delay is obviously increased, the QPS is reduced and a plurality of slow logs can appear on the slave library due to too large strength of the global lock are solved, the slave library reading cannot be influenced under the condition that the data synchronization pressure of the master library and the slave library is large, the QPS is improved, the normal operation of services is ensured, and dirty data reading on the slave library is also avoided.
Drawings
In order to illustrate embodiments of the present invention or technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a schematic diagram of a framework for data synchronization of a master library and a slave library provided in an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a method for reading from a library according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a recording transaction provided by an embodiment of the invention;
FIG. 4 is a schematic flow chart diagram illustrating another embodiment of a read from library method provided by the present invention;
FIG. 5 is a schematic diagram of a configuration read from a library snapshot according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of test results from two reads from a library provided by an embodiment of the present invention;
FIG. 7 is a schematic diagram of a slave library reader according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of another embodiment of a slave library reading apparatus provided in the present invention;
fig. 9 is a schematic structural diagram of a slave library reading apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
In particular implementations, a terminal or device or database server described in embodiments of the present invention includes, but is not limited to, a device such as a desktop computer, a laptop computer, or a tablet computer.
In order to better understand the method for reading from a library, the apparatus for reading from a library, and the device for reading from a library provided by the embodiments of the present invention, a framework for reading from a library to which the embodiments of the present invention are applicable will be described below. Referring to fig. 1, fig. 1 is a schematic diagram of a framework for performing data synchronization on a master library and a slave library according to an embodiment of the present invention. As shown in fig. 1:
fig. 1 is an example of CMongo, namely Cloud MongoDB Service, which is a high-performance NoSQL database management platform manufactured based on MongoDB (database based on distributed file storage). The replay set (a copy group consisting of a series of mongod instances (processes)) of the mongod db may include a primary library (primary) and several secondary libraries (secondary), data is written through the primary, and data is synchronized between the primary and secondary through an operation log (Oplog), which may specifically refer to a set with a fixed size, storing modification operations on the databases for master-slave synchronization. After the write operation on primary is completed, an Oplog is written into the special local-Oplog-rs set, and secondary is responsible for pulling the Oplog from a Replication source (generally primary, but Mongo also supports chain Replication, i.e. secondary can also be used as a Replication source) and playing back the Oplog on secondary, so as to maintain the consistency of data between the master and the slave. In which fig. 1 is a typical Replica set, comprising 1 primary and 2 secondary.
The embodiment does not limit the database management platform based on the MongoDB, and as long as the data synchronization process of the master library and the slave library is involved, the slave library reading method of the embodiment of the invention can be applied to the database management platform for adding the global lock to the slave library.
The following flowchart of the method for reading from a library, which is provided in connection with the embodiment of the present invention shown in fig. 2, may include the following steps:
step S200: starting a master library and a slave library to perform data synchronization;
specifically, when the master library slave library needs to perform data synchronization, the master library slave library is started to perform data synchronization.
Step S202: receiving a read request for a slave bank in a case where the slave bank is allowed to be read while data synchronization is performed by a master bank and the slave bank;
specifically, after the master library and the slave libraries are started for data synchronization, whether to allow the slave libraries to be read when the master library and the slave libraries perform data synchronization can be judged. The judging mode can be to check the configuration information; the configuration information is used for indicating whether the slave library is allowed to be read when the master library performs data synchronization with the slave library; and under the condition that the configuration information is checked out to be the first identifier, reading the slave library when the master library carries out data synchronization with the slave library, and under the condition that the configuration information is checked out to be the second identifier, reading the slave library when the master library carries out data synchronization with the slave library is not allowed. For example, the first identifier may be 1, and the second identifier may be 0, so that when the viewing configuration information is 1, it indicates that the slave library is allowed to be read when the master library performs data synchronization with the slave library, and when the viewing configuration information is 0, it indicates that the slave library is not allowed to be read when the master library performs data synchronization with the slave library.
In one embodiment, the check configuration information may specifically be a state of querying a preset certain configuration item, for example, a configuration item of 1 indicates that reading of the slave library is allowed when data synchronization is performed between the master library and the slave library, a configuration item of 0 indicates that reading of the slave library is not allowed when data synchronization is performed between the master library and the slave library, when the preset configuration item of 1 is queried, it is determined that reading of the slave library is allowed when data synchronization is performed between the master library and the slave library, otherwise, it is determined that reading of the slave library is not allowed when data synchronization is performed between the master library and the slave library.
Then, in the case where the slave library is allowed to be read while the master library performs data synchronization with the slave library, step S204 may be performed for the received read request for the slave library.
It should be noted that, in one embodiment of the present invention, a technician may analyze the data read-write amount of the database in advance to set the value of the configuration item; for example, analysis predicts that traffic write pressure may be very large when the database is subject to master and slave database data synchronization, then a permission may be set to read the slave when the master and slave databases are subject to data synchronization.
Step S204: and for the read request, reading the slave library according to the recorded state of the transaction in the underlying storage engine, and outputting a read result.
Specifically, the embodiment of the present invention records the state of the transaction in the underlying storage engine in advance, for example, the schematic diagram of the principle of recording the transaction provided by the embodiment of the present invention shown in fig. 3, assuming that the time of recording is time T, the transaction intervals 0 to T1 in fig. 3 are transaction intervals that have been committed, the transaction intervals T1 to T5 are transaction intervals that are being executed, and the transaction intervals T5 are transaction intervals that will be executed later, then the state of the transaction in the underlying storage engine recorded at time T can only be the transaction state of the transaction interval before time T, for example, the state of the transaction that has already been committed, that is, the state of the transaction interval 0 to T1 that has already been committed in fig. 3, and the state of the transaction interval T2 that has already committed (committed) in the transaction intervals T1 to T5.
Further, in one embodiment, when the master library and the slave library are applied for data synchronization, the state of the transaction in the underlying storage engine at the current time may be recorded, so that it may be ensured that no new write is performed in the process of recording the state of the transaction. And under the condition that the recorded state of the transaction in the bottom storage engine exists before the current time, the recorded state of the transaction in the bottom storage engine at the current time is covered with the recorded state of the transaction in the bottom storage engine before the current time, so that the recorded state of the transaction is better managed.
The embodiment of the invention reads the slave library according to the recorded state of the transaction in the bottom storage engine, realizes that the slave library can be read when the master library and the slave library carry out data synchronization, solves the technical problems that the request delay is obviously increased, the QPS is reduced and a plurality of slow logs appear on the slave library because the global lock strength is too large, realizes that the slave library reading is not influenced under the condition that the data synchronization pressure of the master library and the slave library is very large, improves the QPS and ensures the normal operation of the service; because the states of the transactions in the bottom storage engine of the record are recorded according to the sequence, partial data can not be missed when the data are read from the library, and dirty reading when the data are read from the library is avoided.
Further, as illustrated in the following by a database management platform based on MongoDB, the flow diagram of another embodiment of the method for reading from a library provided by the present invention as shown in fig. 4 may include the following steps:
step S400: judging whether an Oplog is pulled from a library or not;
specifically, in the MongoDB, when it is determined that the slave library slave pulls the Oplog from the copy source, it indicates that the slave library is about to start parallel playback of the Oplog for data synchronization of the master library slave library, and then step S402 is performed. If it is determined that Oplog is not pulled from the library, step S410 is performed.
Step S402: judging whether the snapshot reading of the slave library is started or not;
specifically, when the technician establishes the database, the data reading and writing amount of the database can be analyzed in advance, and whether to start the snapshot reading from the database is set. For example, as shown in fig. 5, a schematic diagram of reading configuration from a library snapshot according to an embodiment of the present invention may be configured, for example, a configuration information (i.e., a configuration item) snapshot is newly added to a start configuration file of a montodb to indicate whether to start reading from the library snapshot, where a first flag true indicates start, and a second flag false indicates that the read is not started by default. Then, whether the snapshot reading from the library is started or not can be judged by reading and analyzing the content of the configuration item snapshot read, if so, step S404 is executed, and if not, step S406 is executed.
Wherein, the bottom storage engine in the embodiment of the present invention is an engine supporting a snapshot function, for example, the bottom storage engine in the MongoDB is WiredTiger (WT for short); the snapshot is the basis for WT to realize transaction, that is, before the transaction starts or operates, a snapshot is taken of the transaction being executed or about to be executed in the whole WT engine, the state of all the transactions in the whole engine at the time is saved, and it is determined which transactions are visible to the WT and which transactions are invisible to the WT. That is, a series of transaction ID intervals; a snapshot, based on the WT, may be understood as the state of a point in a database. For example, fig. 3 shows a schematic diagram of a principle of recording transactions, where snapshot is performed at time T, the states of the committed transaction intervals 0 to T1 in fig. 3 and the state of the committed transaction interval T2 in the transaction intervals T1 to T5 are visible, and other transaction intervals are not visible.
Step S404: playing back Oplog from the library and creating snapshot;
specifically, for an engine supporting a snapshot function, the embodiment of the present invention may create the snapshot when applying for oplog (applyoplog) from the library each time is finished, thereby ensuring that no new write is generated in the creation process.
Further, in the case of a lot of writes, the applyoplolog may be very frequent, and in one embodiment, it may be determined whether the recording time interval is greater than a threshold when the master library and the slave library are applied for data synchronization (i.e., for example, when the slave library finishes each applyoplolog), and when the state of the transaction in the underlying storage engine that has been recorded before the current time is found; wherein the logging time interval comprises a time interval from a logging time of a state of a transaction in an underlying storage engine to the current time before the current time;
when the judgment is larger than the threshold value, recording the state of the transaction in the bottom storage engine at the current moment; or, when the determination is less than the threshold, the reading the slave library according to the recorded state of the transaction in the underlying storage engine includes reading the slave library according to the recorded state of the transaction in the underlying storage engine before the current time.
That is, for example, if the threshold is 1 second, if the state of the transaction in the underlying storage engine recorded 0.2 seconds before the current time, that is, the time interval is 0.2 seconds and is less than the threshold 1 second, the state of the transaction in the underlying storage engine recorded 0.2 seconds before the current time can be used as the state of the transaction in the underlying storage engine at the current time, that is, the slave library can be read based on the state of the transaction in the underlying storage engine recorded 0.2 seconds before the current time. If the recorded state of the transaction in the underlying storage engine exists 1.2 seconds before the current time, that is, the interval is 1.2 seconds and is greater than the threshold value of 1 second, the state of the transaction in the underlying storage engine at the current time can be recorded, and the recorded state of the transaction in the underlying storage engine at the current time can be covered with the recorded state of the transaction in the underlying storage engine 1.2 seconds before the current time.
Step S406: adding a global lock to the slave library, and playing back an Oplog from the slave library;
specifically, when a slave library snapshot read is not turned on, then a global lock will be applied to the slave library to block all reads and writes during this period, and then Oplog is played back from the library until step S410.
Step S408: receiving a reading request of a slave library, reading the slave library according to the created snapshot, and outputting a reading result;
in particular, where the read request comprises a request for an external read, the slave library read may be modified to be a slave library snapshot read where the external read queries the request by way of a QUERY operation (OP _ QUERY) or a walk-through operation (OP _ get). Because of the particularity of the Mongo protocol, the OP _ QUERY is divided into a Command (Command) and a QUERY (QUERY) according to ns, and for the read request entries, the read from the library needs to be configured to be read from the library snapshot (i.e. read from the snapshot).
In the case where the read request includes a request for an internal read, if the read priority (readconcentrate) is set to read the majority (readmajor), the read from the library is modified to read from the library snapshot. In Mongo, if readConcern is set as readMaxauthority, Mongo can start a background thread and make a snapshot on Oplog synchronized to most nodes to realize readMaxauthority. Reading own Oplog from the library is not in a command form, but calls an internal interface, so that in order to ensure the consistency of data when the Oplog is read from the library, the Oplog is read from the library snapshot (namely, read from the snapshot) instead.
In the embodiment of the invention, the snapshot from the library is read according to the recorded states of all the transactions to a certain point of the database.
Step S410: playback of Oplog from the library ends.
The test is performed for the technical effect of the slave bank read method according to the embodiment of the present invention, and a 10G WT cache space (cacheSize) is taken as an example to test the delay of slave bank read and QPS. FIG. 6 is a schematic diagram of test results from two reads from a library provided by an embodiment of the present invention. Native mongo is a mode of reading from a library in the prior art, and a snapshot version is a mode of reading from a library by adopting the embodiment of the invention, wherein conditions of two tests are the same, including that writing pressure of a master library is the same, the pressure is large enough, writing pressure of the master library on an analog line is large, and thus writing played back from the library is also high. The written data for each test is 5 million pieces, and the data size is larger than the WT cacheSize. In addition, the cpu is not limited in the test. It can be seen from fig. 6 that in the case of 4 different single data sizes of the snapshot version, the latency of reading from the bank is significantly reduced, and the reduction in latency brings about an improvement in QPS. From the time delay data, it can be seen that there is a large boost in QPS assuming that the same cpu usage and the same write pressure are used.
The embodiment of the invention reads the slave library according to the recorded state of the transaction in the bottom storage engine, realizes that the slave library can be read when the master library and the slave library carry out data synchronization, solves the technical problems that the request delay is obviously increased, the QPS is reduced and a plurality of slow logs appear on the slave library because the global lock strength is too large, realizes that the slave library reading is not influenced under the condition that the data synchronization pressure of the master library and the slave library is very large, improves the QPS and ensures the normal operation of the service; because the states of the transactions in the bottom storage engine of the record are recorded according to the sequence, partial data can not be missed when the data are read from the library, and dirty reading when the data are read from the library is avoided.
In order to better implement the above solution of the embodiment of the present invention, the present invention further provides a slave library reading apparatus, which is described in detail below with reference to the accompanying drawings:
as shown in fig. 7, which is a schematic structural diagram of the slave library reading apparatus provided in the embodiment of the present invention, the slave library reading apparatus 70 may include: an activation unit 700, a receiving unit 702, a reading unit 704 and an output unit 706, wherein,
a starting unit 700, configured to start a master library and a slave library to perform data synchronization;
a receiving unit 702 configured to receive a read request to a slave bank in a case where the slave bank is allowed to be read at the time of data synchronization by a master bank and the slave bank;
a reading unit 704, configured to read the slave library according to the recorded status of the transaction in the underlying storage engine for the read request;
and an output unit 706, configured to output the read result.
Specifically, the starting unit 700 may specifically include a pull judgment unit for the output unit 706 and a start sub unit for starting the slave library playback Oplog if it is judged that Oplog is pulled from the library.
The bottom storage engine supports a snapshot function; as shown in fig. 8, the schematic structural diagram of another embodiment of the slave library reading apparatus provided by the present invention, the slave library reading apparatus 70 includes, in addition to the starting unit 700, the receiving unit 702, the reading unit 704, and the output unit 706, further includes: opening judgment unit 708, global lock unit 7010, recording unit 7012, recording judgment unit 7014, and recording override unit 7016, wherein,
the starting judgment unit 708 is configured to check the configuration information after the starting unit 700 starts the master library and the slave libraries to perform data synchronization; the configuration information is used for indicating whether the reading of the slave library is allowed when the data synchronization of the master library and the slave library is carried out; and under the condition that the configuration information is the first identifier, allowing the slave library to be read when the master library performs data synchronization on the slave library.
Specifically, the starting determining unit 708 may be specifically configured to determine whether a snapshot reading of the slave library is started after the starting unit 700 starts the master library and the slave library to perform data synchronization; and when the slave library snapshot reading is judged to be opened, allowing the slave library to be read when the master library performs data synchronization on the slave library.
A global lock unit 7010, configured to add a global lock to the slave library when the opening determining unit 708 checks that the configuration information is the second identifier, that is, when it is determined that the slave library snapshot is not opened, and the master library performs data synchronization on the slave library.
The reading unit 704 is specifically configured to modify the slave library read into the slave library snapshot read when the read request includes a request of an external read and when the external read queries the request in an OP _ QUERY or OP _ get manner; and in the case that the read request comprises a request of an internal read, modifying the read from the library to the read from the library snapshot in the case that the readConcern is set to the readMajauthority.
A recording unit 7012, configured to record the state of the transaction in the underlying storage engine at the current time when the master library and the slave library are applied for data synchronization;
a record determining unit 7014, configured to determine, when the master library and the slave library are applied for data synchronization, and under a condition that a state of a transaction in the bottom storage engine that has a record before the current time is found, whether a record time interval is greater than a threshold; wherein the recording time interval comprises a time interval from a recording time of the state of the transaction in the underlying storage engine to the current time before the current time; when the judgment is larger than the threshold value, recording the state of the transaction in the bottom storage engine at the current moment; or, when the determination is less than the threshold, the reading the slave library according to the recorded state of the transaction in the underlying storage engine includes reading the slave library according to the recorded state of the transaction in the underlying storage engine before the current time.
The record covering unit 7016 is configured to, when the state of the transaction in the underlying storage engine recorded before the current time is found, cover the state of the transaction in the underlying storage engine recorded at the current time with the state of the transaction in the underlying storage engine recorded before the current time.
It should be noted that, in the embodiment of the present invention, the slave library reading apparatus 70 is a terminal or a device or a database server in the embodiments of fig. 1 to fig. 6, and the functions of each module in the slave library reading apparatus 70 may refer to the specific implementation manner of the embodiments of fig. 1 to fig. 6 in the embodiments of the methods, which is not described herein again.
In order to better implement the above solution of the embodiments of the present invention, the present invention further provides a slave library reading apparatus, which is described in detail below with reference to the accompanying drawings:
as shown in fig. 9, which is a schematic structural diagram of the slave library reading apparatus provided in the embodiment of the present invention, the slave library reading apparatus 90 may include a processor 901, an input unit 902, an output unit 903, a memory 904, and a communication unit 905, and the processor 901, the input unit 902, the output unit 903, the memory 904, and the communication unit 905 may be connected to each other through a bus 906. The memory 904 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory, and the memory 704 includes a flash memory according to an embodiment of the present invention. The memory 904 may optionally be at least one memory system located remotely from the processor 901. The memory 904 is used for storing application program codes, and may include an operating system, a network communication module, a user interface module, and a program read from a library, and the communication unit 905 is used for information interaction with an external unit; the processor 901 is configured to call the program code, and perform the following steps:
starting a master library and a slave library to perform data synchronization;
in the case where the slave library is allowed to be read at the time of data synchronization by the master library and the slave library, a read request for the slave library is received through the input unit 902;
for the read request, the slave library is read according to the recorded state of the transaction in the underlying storage engine, and the read result is output through the output unit 903.
Specifically, the starting of the master library and the slave library by the processor 901 for data synchronization may include:
judging whether an Oplog is pulled from a library or not;
in the event that it is determined that Oplog is pulled from the library, then the slave library playback Oplog is initiated.
Specifically, in the case where the slave library is allowed to be read when the master library performs data synchronization with the slave library, the processor 901 receives a read request for the slave library through the input unit 902, and may include:
viewing the configuration information; the configuration information is used for indicating whether the slave library is allowed to be read when the master library performs data synchronization with the slave library;
and under the condition that the configuration information is the first identifier, allowing the slave library to be read when the master library performs data synchronization on the slave library.
Specifically, after the processor 901 looks at the configuration information, it may further perform:
and under the condition that the configuration information is checked out to be the second identifier, adding a global lock to the slave library when the data synchronization is carried out on the master library and the slave library.
Specifically, the underlying storage engine supports snapshot functionality; in the case that the read request includes an external read request, the processor 901 reads the slave library according to the recorded status of the transaction in the underlying storage engine, which may include:
in the case where an external read queries a request by way of OP _ QUERY or OP _ GETMER, the read from the library is modified to be a read from the library snapshot.
Specifically, in the case that the read request includes a request for internal reading, the processor 901 reads the slave library according to the recorded state of the transaction in the underlying storage engine, which may include:
in the case that readConcern is set to readMaxauthority, the slave library read is modified to be a slave library snapshot read.
Specifically, before the processor 901 starts the master library and the slave libraries to perform data synchronization, the following steps may be further performed:
and when the master library and the slave library are applied for data synchronization, recording the state of the transaction in the bottom-layer storage engine at the current time.
Specifically, before the processor 901 starts the master library and the slave libraries to perform data synchronization, the following steps may be further performed:
when the slave library of the application master library finishes data synchronization and under the condition of finding out the state of the transaction in the bottom storage engine which has records before the current moment, judging whether the recording time interval is greater than a threshold value; wherein the recording time interval comprises a time interval from a recording time of the state of the transaction in the underlying storage engine to the current time before the current time;
when the judgment is larger than the threshold value, recording the state of the transaction in the bottom storage engine at the current moment; or, when the determination is less than the threshold, the reading the slave library according to the recorded state of the transaction in the underlying storage engine includes reading the slave library according to the recorded state of the transaction in the underlying storage engine before the current time.
Specifically, after the processor 901 records the state of the transaction in the underlying storage engine at the current time, it may further perform:
and under the condition that the recorded state of the transaction in the bottom storage engine exists before the current moment, covering the recorded state of the transaction in the bottom storage engine at the current moment with the recorded state of the transaction in the bottom storage engine before the current moment.
It should be noted that, in the embodiment of the present invention, the slave library reading device 90 is a terminal or a device or a database server in the embodiments of fig. 1 to fig. 6, and specific reference may be made to specific implementation manners of the embodiments of fig. 1 to fig. 6 in the above method embodiments, which are not described herein again. The above-described structure of the slave library reading apparatus 90 is merely one example provided by the embodiment of the present invention, and the slave library reading apparatus 90 may have more or less components than those shown, may combine two or more components, or may have a different configuration implementation of the components.
The embodiment of the invention reads the slave library according to the recorded state of the transaction in the bottom storage engine, realizes that the slave library can be read when the master library and the slave library carry out data synchronization, solves the technical problems that the request delay is obviously increased, the QPS is reduced and a plurality of slow logs appear on the slave library because the global lock strength is too large, realizes that the slave library reading is not influenced under the condition that the data synchronization pressure of the master library and the slave library is very large, improves the QPS and ensures the normal operation of the service; because the states of the transactions in the bottom storage engine of the record are recorded according to the sequence, partial data can not be missed when the data are read from the library, and dirty reading when the data are read from the library is avoided.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by a computer program, which may be stored in a computer readable storage medium and executed by a computer to implement the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.
Claims (10)
1. A method of reading from a library, comprising:
starting a master library and a slave library to perform data synchronization;
receiving a read request for a slave bank in a case where the slave bank is allowed to be read while data synchronization is performed by a master bank and the slave bank;
for the reading request, reading the slave library according to the recorded state of the transaction in the bottom storage engine, and outputting a reading result; the underlying storage engine supports a snapshot function;
the reading the slave library according to the recorded state of the transaction in the underlying storage engine comprises:
under the condition that the read request comprises an external read request, under the condition that the external read queries the request in a query operation or follow-up operation mode, modifying the slave library read into the slave library snapshot read; or alternatively
In the case where the read request includes a request for an internal read, the read from the bank is modified to the read from the bank snapshot in the case where the read priority is set to a majority of reads.
2. The method of claim 1, wherein said initiating a master library slave library for data synchronization comprises:
judging whether the slave library pulls the operation log or not;
and starting the slave library playback operation log when the operation log is judged to be pulled from the library.
3. The method of claim 1, wherein receiving a read request to a slave library in the event that the slave library is allowed to be read while data synchronization is being performed at a master library and the slave library comprises:
viewing the configuration information; the configuration information is used for indicating whether the slave library is allowed to be read when the master library performs data synchronization with the slave library;
and under the condition that the configuration information is checked out to be the first identifier, allowing the slave library to be read when the master library carries out data synchronization with the slave library.
4. The method of claim 3, wherein said viewing configuration information is further followed by:
and under the condition that the configuration information is checked out to be the second identifier, adding a global lock to the slave library when the data synchronization is carried out on the master library and the slave library.
5. The method of claim 1, wherein prior to initiating data synchronization by the master library slave library, further comprising:
and recording the state of the transaction in the bottom layer storage engine at the current time when the master library and the slave library are applied for data synchronization.
6. The method of claim 1, wherein prior to initiating data synchronization by the master library slave library, further comprising:
when the master library and the slave libraries are applied for data synchronization, and under the condition that the state of the transaction in the bottom storage engine recorded before the current moment is found, judging whether the recording time interval is greater than a threshold value; wherein the logging time interval comprises a time interval from a logging time of a state of a transaction in an underlying storage engine to the current time before the current time;
when the judgment is larger than the threshold value, recording the state of the transaction in the bottom storage engine at the current moment; or, when the determination is smaller than the threshold, the reading the slave library according to the recorded state of the transaction in the underlying storage engine includes reading the slave library according to the recorded state of the transaction in the underlying storage engine before the current time.
7. The method of claim 5 or 6, wherein after recording the state of the transaction in the underlying storage engine at the current time, further comprising:
and under the condition that the recorded state of the transaction in the bottom storage engine exists before the current time, covering the recorded state of the transaction in the bottom storage engine at the current time with the recorded state of the transaction in the bottom storage engine before the current time.
8. A device reading from a library, comprising means for performing the method of any of claims 1-7.
9. A device for reading from a library, comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being interconnected, wherein the memory is configured to store application program code and the processor is configured to invoke the program code to perform the method of any of claims 1-7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711391348.8A CN110019527B (en) | 2017-12-20 | 2017-12-20 | Slave library reading method, related device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711391348.8A CN110019527B (en) | 2017-12-20 | 2017-12-20 | Slave library reading method, related device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019527A CN110019527A (en) | 2019-07-16 |
| CN110019527B true CN110019527B (en) | 2022-09-27 |
Family
ID=67187032
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711391348.8A Active CN110019527B (en) | 2017-12-20 | 2017-12-20 | Slave library reading method, related device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019527B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113760860B (en) * | 2020-11-12 | 2023-09-22 | 北京京东振世信息技术有限公司 | Data reading method and device |
| CN112445780A (en) * | 2020-11-27 | 2021-03-05 | 上海上讯信息技术股份有限公司 | Master-slave architecture deployment method and device based on snapshot |
| CN113485869A (en) * | 2021-06-08 | 2021-10-08 | 上海众言网络科技有限公司 | Database upgrading method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9600500B1 (en) * | 2013-06-21 | 2017-03-21 | Amazon Technologies, Inc. | Single phase transaction commits for distributed database transactions |
| CN106777225A (en) * | 2016-12-26 | 2017-05-31 | 腾讯科技（深圳）有限公司 | The moving method and system of a kind of data |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11615115B2 (en) * | 2010-12-23 | 2023-03-28 | Mongodb, Inc. | Systems and methods for managing distributed database deployments |
| US9298752B2 (en) * | 2013-08-26 | 2016-03-29 | Dropbox, Inc. | Facilitating data migration between database clusters while the database continues operating |
-
2017
- 2017-12-20 CN CN201711391348.8A patent/CN110019527B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9600500B1 (en) * | 2013-06-21 | 2017-03-21 | Amazon Technologies, Inc. | Single phase transaction commits for distributed database transactions |
| CN106777225A (en) * | 2016-12-26 | 2017-05-31 | 腾讯科技（深圳）有限公司 | The moving method and system of a kind of data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019527A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021180025A1 (en) | Message processing method and apparatus, electronic device and medium | |
| KR102119258B1 (en) | Technique for implementing change data capture in database management system | |
| US20110218964A1 (en) | Automatic synchronization conflict resolution | |
| CN110865888A (en) | Resource loading method and device, server and storage medium | |
| CN110019527B (en) | Slave library reading method, related device and equipment | |
| CN111797172B (en) | Data migration method, device, equipment, distributed system and storage medium | |
| CN111414403A (en) | Data access method and device and data storage method and device | |
| CN101763433B (en) | Data storage system and method | |
| CN102867035A (en) | High-availability method and device of distributed document system cluster | |
| CN108427728A (en) | Management method, equipment and the computer-readable medium of metadata | |
| US20090132534A1 (en) | Remote replication synchronizing/accessing system and method thereof | |
| CN112867999B (en) | Version-based table locking | |
| US11853284B2 (en) | In-place updates with concurrent reads in a decomposed state | |
| CN111399753B (en) | Method and device for writing pictures | |
| CN103617039B (en) | Method and device for accessing user space file system | |
| CN111309799A (en) | Method, device and system for realizing data merging and storage medium | |
| US9009430B2 (en) | Restoration of data from a backup storage volume | |
| US11481142B2 (en) | Method and device for downloading resources | |
| WO2020107352A1 (en) | Log sequence number generation method and apparatus and readable storage medium | |
| CN116225630A (en) | Transaction processing method, apparatus, electronic device, storage medium, and program product | |
| US11726978B2 (en) | Computer program for providing efficient change data capture in a database system | |
| CN113687921A (en) | Transaction processing method and device, distributed database system and electronic equipment | |
| US8977814B1 (en) | Information lifecycle management for binding content | |
| CN112699129A (en) | Data processing system, method and device | |
| CN111625302A (en) | Function time consumption tracking method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230914 Address after: 518057 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 floors Patentee after: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. Patentee after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd. Address before: 518057 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 floors Patentee before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. |