CN117555884B - Method, device and equipment for reading data page and readable storage medium - Google Patents
Method, device and equipment for reading data page and readable storage medium Download PDFInfo
- Publication number
- CN117555884B CN117555884B CN202410049489.5A CN202410049489A CN117555884B CN 117555884 B CN117555884 B CN 117555884B CN 202410049489 A CN202410049489 A CN 202410049489A CN 117555884 B CN117555884 B CN 117555884B
- Authority
- CN
- China
- Prior art keywords
- data page
- log
- playback
- log information
- instance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 230000004048 modification Effects 0.000 claims description 64
- 238000012986 modification Methods 0.000 claims description 64
- 230000015654 memory Effects 0.000 claims description 27
- 238000004590 computer program Methods 0.000 claims description 24
- 238000012545 processing Methods 0.000 claims description 18
- 238000010586 diagram Methods 0.000 description 18
- 238000004364 calculation method Methods 0.000 description 17
- 238000005192 partition Methods 0.000 description 17
- 230000002093 peripheral effect Effects 0.000 description 10
- 230000001133 acceleration Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 239000002699 waste material Substances 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 230000002045 lasting effect Effects 0.000 description 4
- 230000008447 perception Effects 0.000 description 4
- 230000002085 persistent effect Effects 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 239000000919 ceramic Substances 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001680 brushing effect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 239000010409 thin film Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a method, a device and equipment for reading a data page and a readable storage medium, and belongs to the technical field of databases. The method comprises the following steps: acquiring a first reading request sent by a first slave instance, wherein the first slave instance plays back log information positioned at a first site and before the first site in a target log sequence to obtain a first data page, and the first reading request is used for requesting to read the first data page released by the first slave instance; playback is carried out on a reference log sequence to obtain a first data page, wherein the reference log sequence comprises log information positioned behind a reference site in a target log sequence, and the reference site is smaller than the first site; a first page of data is sent to a first slave instance. According to the method and the device, the first slave instance plays back the log information positioned behind the first site in the target log sequence based on the first data page, repeated playback is not needed, and the computing resource of the first slave instance is saved.
Description
Technical Field
The embodiment of the application relates to the technical field of databases, in particular to a method, a device and equipment for reading a data page and a readable storage medium.
Background
In the technical field of databases, a master instance can modify a data page for multiple times, and each pair of data pages is modified once, the master instance generates log information for recording modification operations, and the log information forms a log sequence according to the sequence of generation. The slave instance reads the data page from the master instance and synchronizes the log sequence, and the data page is played back according to the sequence of the log information, so that the read data page is modified for multiple times, and the data synchronization of the master instance and the slave instance is realized. The playback site may be used to characterize the location of the last log information in the log sequence that is played back, for example, site 500, and to characterize the 500 th log information in the log sequence that is played back.
In the related art, the slave instances may be plural, and there is a difference in playback points of the different slave instances. The master instance may obtain playback points for each slave instance, and determine a slowest playback point from the playback points for each slave instance. The data page is obtained by playing back the log sequence to the slowest playback site and stored in the shared memory. When playback of the log sequence from the instance needs to continue, the data page may be read from the shared storage and playback of the log sequence may be resumed from the slowest playback site.
For the slave instance with the playback locus larger than the slowest playback locus, the log sequence is replayed from the slowest playback locus, so that a great number of repeated calculations are carried out on the slave instance, and the waste of calculation resources is caused.
Disclosure of Invention
The application provides a method, a device, equipment and a readable storage medium for reading a data page, which can enable a slave instance to continue to play back at the last play back position, and save the computing resource of the slave instance.
In one aspect, a method for reading a data page is provided, the method comprising: acquiring a first reading request sent by a first slave instance, wherein the first slave instance is used for playing back log information positioned at a first site and before the first site in a target log sequence, the first reading request is used for requesting to read a first data page released by the first slave instance, and the first data page is a data page obtained after playing back the log information positioned at the first site; the method comprises the steps that a reference log sequence is played back based on a reference data page, the first data page is obtained, the reference data page is obtained after log information located at a reference position in the target log sequence is played back, the reference position is smaller than the first position, and the reference log sequence comprises log information located behind the reference position in the target log sequence; the first data page is sent to the first slave instance for playback of log information in the target log sequence that is located after the first site based on the first data page by the first slave instance.
In another aspect, there is provided a reading apparatus for a data page, the apparatus comprising: the system comprises an acquisition module, a first reading module and a second reading module, wherein the acquisition module is used for acquiring a first reading request sent by a first slave instance, the first slave instance is used for playing back log information positioned at a first site and before the first site in a target log sequence, the first reading request is used for requesting to read a first data page released by the first slave instance, and the first data page is a data page obtained after playing back the log information positioned at the first site; the playback module is used for playing back a reference log sequence based on a reference data page to obtain the first data page, wherein the reference data page is a data page obtained after playing back log information positioned at a reference position in the target log sequence, the reference position is smaller than the first position, and the reference log sequence comprises log information positioned behind the reference position in the target log sequence; and the sending module is used for sending the first data page to the first slave instance, so that the first slave instance plays back log information positioned behind the first locus in the target log sequence based on the first data page.
In one possible implementation manner, the obtaining module is further configured to obtain a reference position point sent by the master instance, where the reference position point is not greater than a slowest playback position point of playback positions of a plurality of slave instances, where the playback position point of the slave instance is used to characterize that log information of a last playback completion of the slave instance when the slave instance plays back the target log sequence is at a position point of the target log sequence, and the plurality of slave instances includes the first slave instance; and the playback module is further used for playing back the log information positioned at the reference position and before the reference position in the target log sequence to obtain the reference data page.
In a possible implementation manner, the obtaining module is further configured to obtain the target log sequence sent by the master instance, where the target log sequence is used to record a modification operation that the master instance makes multiple modifications to an initial data page; the apparatus further comprises: the storage module is used for storing the target log sequence in a log storage area; and the playback module is used for reading the target log sequence from the log storage area, and playing back log information positioned in the target log sequence and before the reference position to obtain the reference data page.
In one possible implementation, the apparatus further includes: and the release module is used for releasing the log information positioned in the target log sequence and before the reference position from the log storage area to obtain the reference log sequence stored in the log storage area.
In one possible implementation, the apparatus further includes: a storage module for storing the reference data page in a data page storage area; and the playback module is used for reading the reference data page from the data page storage area, and playing back the reference log sequence based on the reference data page to obtain the first data page.
In a possible implementation manner, the playback module is configured to modify, for a first log information in the reference log sequence, the reference data page based on a modification operation recorded by the first log information, to obtain a playback data page of the first log information, where when the first log information is the log information of the first location, the playback data page of the first log information is the first data page; and if the first log information is not the log information of the first site, modifying a playback data page of the last log information based on the modification operation recorded by the non-first log information for the non-first log information in the reference log sequence to obtain the playback data page of the non-first log information, wherein when the non-first log information is the log information of the first site, the playback data page of the non-first log information is the first data page.
In one possible implementation, the apparatus further includes: the storage module is used for storing the first reading request in a request queue, the request queue is used for storing a second reading request sent by a second slave instance, the second slave instance is used for playing back log information positioned at a second position and before the second position in a target log sequence, the second reading request is used for requesting to read a second data page released by the second slave instance, and the second data page is a data page obtained after playing back the log information positioned at the second position; and the playback module is used for playing back the reference log sequence based on the reference data page to obtain the first data page and the second data page if the request queue meets the execution condition.
In one possible implementation, the request queue meeting an execution condition includes at least one of: the sum of the number of the first read requests and the second read requests is not less than a number threshold; the waiting time of the first read request in the request queue exceeds a time threshold.
In one possible implementation, the sending module is further configured to send the second data page to the second slave instance, for the second slave instance to play back log information located after the second location in the target log sequence based on the second data page.
In one possible implementation, the apparatus is located on a storage cluster, the storage cluster including a first storage node and a plurality of second storage nodes, different second storage nodes being used to manage data pages of different numbered intervals; the first storage node is configured to obtain the first read request, where the first read request carries a number of the first data page; the first storage node is further configured to send the first read request to a target storage node in the plurality of second storage nodes, where a number interval corresponding to the target storage node includes a number of the first data page; and the target storage node is used for playing back the reference log sequence based on the reference data page, obtaining the first data page and sending the first data page to the first slave instance.
In another aspect, an electronic device is provided, where the electronic device includes a processor and a memory, where at least one computer program is stored in the memory, where the at least one computer program is loaded and executed by the processor, so that the electronic device implements a method for reading a data page as described in any one of the foregoing.
In another aspect, there is also provided a computer readable storage medium having stored therein at least one computer program loaded and executed by a processor to cause an electronic device to implement a method of reading a data page as described in any of the above.
In another aspect, there is also provided a computer program, where the computer program is at least one, and at least one computer program is loaded and executed by a processor, so as to enable an electronic device to implement a method for reading any one of the data pages.
In another aspect, there is also provided a computer program product having at least one computer program stored therein, the at least one computer program being loaded and executed by a processor to cause an electronic device to implement a method of reading any one of the data pages described above.
The technical scheme provided by the application has at least the following beneficial effects:
According to the technical scheme provided by the application, the first slave instance plays back the log information positioned at the first site and before the first site in the target log sequence to obtain the first data page. When the first slave instance releases the first page of data, the first slave instance may send a first read request requesting to read the first page of data. The first data page is obtained by playing back the reference log sequence based on the reference data page, and the first data page is sent to the first slave instance, so that the first slave instance can play back log information positioned behind the first site in the target log sequence based on the first data page, the first slave instance can play back continuously at the last playback position without repeated playback, and the computing resource of the first slave instance is saved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram of an implementation environment of a data page reading method according to an embodiment of the present application.
Fig. 2 is a flowchart of a method for reading a data page according to an embodiment of the present application.
Fig. 3 is a schematic playback diagram of a log sequence according to an embodiment of the present application.
Fig. 4 is a schematic diagram of a data page for brushing down according to an embodiment of the present application.
FIG. 5 is a schematic diagram of a persistent log and a swipe data page provided by an embodiment of the application.
FIG. 6 is a schematic diagram of a persistent log provided by an embodiment of the present application.
Fig. 7 is a schematic diagram of a playback log in a shared storage according to an embodiment of the present application.
Fig. 8 is a schematic diagram of log playback from an instance provided in an embodiment of the present application.
FIG. 9 is a diagram of a multi-request reply according to an embodiment of the present application.
FIG. 10 is a schematic diagram of a partition according to an embodiment of the present application.
Fig. 11 is a schematic diagram of reading a data page according to an embodiment of the present application.
Fig. 12 is a schematic diagram of log playback according to an embodiment of the present application.
Fig. 13 is a schematic structural diagram of a data page reading device according to an embodiment of the present application.
Fig. 14 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Fig. 15 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.
In the technical field of databases, a master instance can generate a log sequence, and a modification operation of the master instance for modifying a data page for a plurality of times is recorded through the log sequence. The slave instance can copy the log sequence, and the data page is modified for a plurality of times through replaying the log sequence, so that the data synchronization of the master instance and the slave instance is realized.
However, the speed at which the log sequence is played back from the instance differs from time to time due to network latency, hardware performance, etc. In the related art, the master instance may collect playback points of the respective slave instances, from which the slowest playback point is determined. The data page is obtained by playing back the log sequence to the slowest playback site and stored in the shared memory. When playback of the log sequence from the instance needs to continue, the data page may be read from the shared storage and playback of the log sequence may be resumed from the slowest playback site.
For the slave instance with the playback locus larger than the slowest playback locus, the log sequence is replayed from the slowest playback locus, so that a great number of repeated calculations are carried out on the slave instance, and the waste of calculation resources is caused. Based on the above, the embodiment of the application provides a method for reading a data page, which can support reading the first data page from an instance and continue playing back the log sequence on the basis of the first data page under the condition that the first data page is obtained by playing back the log sequence from the instance and the first data page is released from the instance, thereby avoiding repeated playback of the log sequence and saving computing resources of the slave instance.
As shown in fig. 1, fig. 1 is a schematic diagram of an implementation environment of a data page reading method according to an embodiment of the present application, where the implementation environment includes a master instance 101, a slave instance 102, and a storage node 103.
The master instance 101 is also called a read-write database instance, and is used for performing read-write operations on a database. Slave instance 102, also referred to as a read-only database instance, is used to perform read-only operations on a database. Database instances are independent database entities created in a database. Each database instance has its own data set, configuration, and operating environment. Any one database instance may operate and manage data by connecting to other database instances. That is, the master instance 101 and any one of the slave instances 102 are communicatively connected, and any two of the slave instances 102 are communicatively connected.
Alternatively, the database instance may be one service process, and a single physical device may run at least one service process, such that there may be multiple database instances on a single physical device. That is, the master instance 101 may be deployed on a physical device independently or with at least one slave instance 102. Any slave instance 102 may be deployed independently on a physical device, or may be deployed on a physical device with the master instance 101, or with at least one slave instance 102, or with the master instance 101, or with the slave instance 102.
It will be appreciated that the number of master instances 101 is at least one, as is the number of slave instances 102. For example, one possible cloud-native relational database that utilizes distributed storage to achieve separation of computation and storage, and load balancing by horizontally expanding multiple database instances. In the database, there are one read-write database instance and at most 15 read-only database instances, that is, the number of the master instances 101 is 1, and the number of the slave instances 102 is not more than 15, so that the read-write separation of the database is realized.
Storage node 103 is a single physical device that makes up the storage system, storage node 103 being used to provide storage capacity and data processing capabilities for the entire storage system. Storage node 103 may be a server, a dedicated storage device, or a virtual machine running on a shared infrastructure. It will be appreciated that the number of storage nodes 103 is at least one, any one storage node 103 being communicatively coupled to the master instance 101, any one slave instance 102. Alternatively, a plurality of storage nodes 103 may constitute a storage cluster.
The physical device in the embodiment of the application can be a terminal device or a server. The terminal device may be a smart phone, a game console, a desktop computer, a tablet computer, a laptop, a smart television, a smart car device, a smart voice interaction device, a smart home appliance, etc. The server may be a server, or a server cluster formed by a plurality of servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiment of the present application. The server may be communicatively connected to the terminal device via a wired network or a wireless network. The server may have functions of data processing, data storage, data transceiving, etc., and is not limited in the embodiment of the present application. The number of terminal devices and servers is not limited and may be one or more.
As shown in fig. 2, fig. 2 is a flowchart of a method for reading a data page according to an embodiment of the present application, and the method can be applied to the above-mentioned implementation environment. The reading of the data page in the embodiment of the present application is performed by a storage node or a storage cluster, and for convenience of description, the storage node is taken as an example. As shown in fig. 2, the method includes the following steps.
In step 201, a first read request sent by a first slave instance is obtained, where the first slave instance is used for playing back log information located at a first location and before the first location in a target log sequence, and the first read request is used for requesting to read a first data page released by the first slave instance, where the first data page is a data page obtained after playing back the log information located at the first location.
In the embodiment of the application, in the process of generating the initial data page, the master instance records the generation operation of the initial data page through log information. In addition, the main instance can also make multiple modifications to the initial data page, and each modified modification operation can be recorded through log information, that is, each modification operation corresponds to one log information. The log information forms a target log sequence according to the generated sequence. That is, the target log sequence includes a plurality of log information for recording a modification operation for performing a plurality of modifications to the initial data page.
Optionally, the first log information of the target log sequence is used to record a generation operation when the initial data page is generated. Or the first log information of the target log sequence is used for recording a modification operation for modifying the blank data page.
For example, the target log sequence includes 1 st to 2000 th log information, the 1 st log information is used for recording a generating operation of an initial data page, the 2 nd log information is used for recording a modifying operation of performing a first modification on the initial data page, the 3 rd log information is used for recording a modifying operation of performing a second modification on the initial data page, the 4 th log information is used for recording a modifying operation of performing a third modification on the initial data page, and so on, and will not be described herein.
The master instance may play back the target log sequence, and after the target log sequence is obtained from the instance, the master instance may play back the target log sequence. The master instance or the slave instance plays back each log information in turn according to the sequence of each log information in the target log sequence. And playing back one piece of log information, namely modifying the data page obtained by playing back the last piece of log information according to the modification operation recorded by the log information to obtain the data page obtained by playing back the log information. The embodiment of the present application does not limit the modification operation, and illustratively, the modification operation includes at least one of adding data, deleting data, modifying data, and the like.
As shown in fig. 3, a certain data page (i.e., an initial data page) is sequentially modified 500 times to obtain logs 1 to 500, and the logs 1 to 500 are arranged according to a generating order to form a target log sequence for recording a modification operation of each modification, for example, inserting a line of data in the data page, deleting a line of data in the data page, and the like. The target log sequence is located in a redo log (redo log) queue and is ordered in the order of generation of logs 1 through 500. And when the target log sequence is played back, sequentially playing back the target log sequence according to the generation sequence of the logs, and updating the data page once every time one log is played back. That is, the data page 1 is generated according to the generation operation described in the log 1, the data page 1 is modified according to the modification operation described in the log 2 to obtain the data page 2, the data page 2 is modified according to the modification operation described in the log 3 to obtain the data page 3, and so on.
Because the target log sequence is played back according to the sequence of the log information, in the process of playing back the target log sequence, part of the log information is already played back, and the other part of the log information is not played back. The position of the last log information in the target log sequence after playback is completed is called a playback site, and is called a site for short. As in fig. 3, if the playback of log 3 has been completed and the playback of log 4 has not been started, since log 3 is the third log in the target log sequence, the playback site is 3. Log 3 has completed playback, at which point data page 3 is obtained, that is, the playback site is the position in the target log sequence of the log information of the current data page obtained by playback.
It will be appreciated that both the master instance and the slave instance may perform playback of the target log sequence, and that the slave instance needs to copy the target log sequence from the master instance. The playback speeds of the master instance and the different slave instances are different due to the influence of the delay of the replication network, the load difference of the master instance and the different slave instances, and the like. That is, the playback sites of the master instance and the different slave instances are different.
In the embodiment of the present application, the playback site of the first slave instance is the first site. That is, the first slave instance has completed playback of log information in the target log sequence that is located before and at the first site. For example, the first site is 500, and the first slave instance has completed playback of log information 1 to 500 in the target log sequence. As another example, the first site is 1000, and then the first slave instance has completed playback of log information 1 through 1000 in the target log sequence. For another example, the first location is 1500 and the first slave instance has completed playback of log information 1 through 1500 in the target log sequence.
After the first slave instance plays back the log information at the first location, a first data page may be obtained, where the first data page is typically stored in the memory of the first slave instance. It will be appreciated that the first slave instance also needs to perform other service processing, for example, the first slave instance needs to process the read request of the client in real time, and the service processing data needs to be stored in the memory. Because of the internal limitations, the first slave instance needs to release memory sporadically to cope with various situations. If the first data page in the memory of the first slave instance is released and the first slave instance needs to continue playback of the target log sequence, i.e. the first slave instance needs to play back log information in the target log sequence that is located after the first location, the first slave instance needs to read the first data page from the storage node. In this case, the first slave instance sends a first read request to the storage node to request reading of the first page of data from the storage node via the first read request.
Step 202, playing back a reference log sequence based on a reference data page to obtain a first data page, wherein the reference data page is a data page obtained after playing back log information located at a reference position in a target log sequence, the reference position is smaller than the first position, and the reference log sequence comprises log information located behind the reference position in the target log sequence.
In the embodiment of the application, the storage node stores a reference data page and a reference log sequence. The manner in which the storage node obtains the reference data page and the reference log sequence is not limited herein.
In one possible implementation, step 202 is preceded by steps A1 to A2 (not shown in the figure).
And A1, acquiring a reference locus sent by a master instance, wherein the reference locus is not more than the slowest playback locus in the playback loci of a plurality of slave instances, the playback locus of the slave instance is used for representing the locus of the target log sequence of log information of which the last playback is completed when the slave instance plays back the target log sequence, and the plurality of slave instances comprise a first slave instance.
In the embodiment of the application, the master instance is in communication connection with any slave instance. The master instance may periodically send a site acquisition request to the slave instance requesting to acquire the playback site of the target log sequence number, the slave instance sending the playback site to the master instance in response to the site acquisition request. Or the slave instance may periodically actively send playback sites to the master instance. In this way, the master instance is enabled to collect playback sites for individual slave instances.
It will be appreciated that playback sites of the various slave instances differ due to the effects of delays in the replication network, different loads of the different slave instances, and the like. For example, at the same time, playback of the log information 500 in the target log sequence is completed from example 1, and the log information 501 does not start playback, then the playback site from example 1 is 500. Playback of log information 1000 in the target log sequence is completed from example 2, and log information 1001 does not start playback, then the playback site from example 2 is 1000. Playback of log information 1500 in the target log sequence is completed from example 3, and log information 1501 does not start playback, and the playback site from example 3 is 1500.
The master instance may determine a slowest playback locus from among the playback loci of the respective slave instances, determine a reference locus based on the slowest playback locus, and send the reference locus to the storage node. Optionally, the main instance determines the slowest playback position as the reference position, or the main instance subtracts the setting position from the slowest playback position to obtain the reference position, wherein the setting position is positive and smaller than the slowest playback position. For example, from example 1 at playback locus 500, from example 2 at playback locus 1000, from example 3 at playback locus 1500, the reference locus is the slowest playback locus 500, or the reference locus is 500-5=495 if locus 5 is set.
It can be understood that, in the general scenario, after the master instance obtains the slowest playback site, the log information located before the slowest playback site and the slowest playback site in the target log sequence needs to be played back, so as to obtain a playback data page of the log information located at the slowest playback site, which is abbreviated as a data page of the slowest playback site. When log information located after the slowest playback site is included in the target log sequence, the data page of the slowest playback site is a dirty data page because the log information is used to record a modification operation for modifying the data page of the slowest playback site. That is, a dirty data page is a modified data page. The master instance may send the data page of the slowest playback site to the storage node, enabling the data page of the slowest playback site to be stored in the shared storage cluster, so that any slave instance reads the data page of the slowest playback site from the shared storage cluster.
As shown in fig. 4, in the general scenario, the master instance, from instance 1 to 3, is located at the computing layer, and the shared storage cluster is located at the storage layer. The master instance is used to generate a target log sequence that includes 2000 redo logs, i.e., the latest site of the master instance is 2000. The redo log is copied from each of examples 1 to 3 and played back. Wherein playback of the 500 th redo log is completed from example 1 and the 501 th redo log is not played back, i.e., the playback site from example 1 is 500. Similarly, the playback point from example 2 is 1000 and the playback point from example 3 is 1500.
The master instance may collect the playback points from instances 1 to 3, resulting in the slowest playback point, i.e., playback point 500. The master instance plays back the 1 st to 500 th redo logs in the target log sequence, obtains a data page with playback site of 500, and brushes down the data page into the shared storage cluster.
In the embodiment of the application, after the main instance obtains the slowest playback site, the target log sequence is not required to be played back, but the reference site determined based on the slowest playback site is directly brushed down to the storage node, so that the reference site is brushed down to the shared storage cluster. By saving the computing resources and storage resources required to playback the target log sequence, the load pressure of the master instance is reduced.
And step A2, playing back the log information positioned at the reference position and before the reference position in the target log sequence to obtain the reference data page.
In the embodiment of the present application, the storage node stores the target log sequence, and the mode of the storage node obtaining the target log sequence is not limited herein.
In an exemplary embodiment, step A2 is preceded by: acquiring a target log sequence sent by a main instance, wherein the target log sequence is used for recording the modification operation of the main instance for modifying an initial data page for a plurality of times; the target log sequence is stored in a log storage area. The step A2 comprises the following steps: and reading the target log sequence from the log storage area, and playing back the log information positioned in the target log sequence and before the reference position to obtain the reference data page.
In the embodiment of the application, the storage node can periodically send the acquisition request of the target log sequence to the main instance. Or the storage node sends a request for acquiring the target log sequence to the master instance when receiving a read request of any data page related to the target log sequence, for example, when the storage node receives the first read request. The master instance sends a target log sequence to the storage node in response to the acquisition request. Or the master instance may actively send the target log sequence to the storage node on a timed basis.
As mentioned above, the master instance may send a sequence of target logs to multiple slave instances. Optionally, the master instance may also send an instruction to any slave instance, the instruction being for instructing the slave instance to send the target log sequence to the storage node. After receiving the instruction and the target log sequence from the instance, the target log sequence is sent to the storage node, and then the target log sequence is played back.
The storage node has a log storage area. After the storage node acquires the target log sequence, the target log sequence is stored in the log storage area to be stored in a lasting mode.
After the storage node acquires the reference location, the target log sequence may be played back based on the reference location. The storage node plays back the log information positioned at the reference position and before the reference position in the target log sequence to obtain the reference data page.
For the first log information in the target log sequence, generating an initial data page based on the generation operation recorded by the first log information, wherein the initial data page is the playback data page of the first log information. Or modifying the blank data page based on the modification operation recorded by the first log information to obtain a playback data page of the first log information. It will be appreciated that the blank data page does not include data, and thus, modifications made to the blank data page include only the operation of inserting data. That is, the first log information in the target log sequence may record an insert operation of inserting data in a blank data page.
And modifying the playback data page of the last log information based on the modification operation recorded by the non-first log information for the non-first log information positioned at or before the reference position in the target log sequence to obtain the playback data page of the non-first log information. It is understood that the playback data page of the last log information includes data, and thus, the modification operation recorded by the non-first log information includes an insert operation when inserting data, a delete operation when deleting data, a modify operation when modifying data, and the like. The playback data page of the log information at the reference position is the reference data page.
In an exemplary embodiment, step A2 further comprises, after: the reference data page is stored in the data page storage area. Step 202 comprises: and reading the reference data page from the data page storage area, and playing back the reference log sequence based on the reference data page to obtain a first data page.
In the embodiment of the application, the storage node has a data page storage area. When the storage node plays back the obtained reference data page, the reference data page can be stored in the data page storage area so as to realize the persistent storage of the reference data page.
It is understood that in a general scenario, shared storage is a generic storage, where shared storage refers to storage nodes or storage clusters that support a master instance and individual slave instances to access, and generic storage refers to storage nodes or storage clusters that are not traffic aware. In general, the generic storage is only responsible for providing a basic storage interface to the master and slave instances for ease of service usage. For example, the generic storage provides a file system interface that facilitates reading files by the master instance and slave instances, and for example, the generic storage provides a block storage interface that facilitates writing data by the master instance and slave instances to storage blocks. That is, the generic store is the master instance, what data is stored for what data from the instance; data is returned if the primary and secondary instances are present in the generic store, and an error hint is returned if the primary and secondary instances are absent in the generic store.
In the general scenario, the master instance generates a target log sequence, and sends the target log sequence to the general storage to persist the target log sequence. And the slave instance obtains a playback data page by copying the target log sequence and playing back the target log sequence. Because the playback progress of each slave instance is different, the master instance collects the playback sites of each slave instance, determines the slowest playback site, plays back the target log sequence to obtain the data page of the slowest playback site, and swipes the data page of the slowest playback site down in the universal storage. Since only the data page of the slowest playback site exists in the universal storage and there is no traffic perception, only the data page of the slowest playback site can be read from the universal storage from the instance.
As shown in fig. 5, the shared storage is a general-purpose storage including a log file and a data page file. The master instance persists the log to a log file. The slave instance copies the log of the master instance and obtains the data page through log playback. The master instance collects playback points of the slave instances and determines the slowest playback point. The master instance obtains the data page of the slowest playback site through log playback and swipes the data page of the slowest playback site down to the data page file so that the slave instance can only read the data page of the slowest playback site from the universal storage.
It has been mentioned above that the master instance and the slave instance are at the computation level. In a general scenario, the playback tasks of the target log sequence are completed only at the computation layer. However, shared storage is a storage node or storage cluster comprising a plurality of storage nodes, any of which may be a physical server or a cluster of a plurality of physical servers, having sufficient computing resources. Under the general scene, the shared storage has no service perception, and data calculation is not needed, so that a great amount of resource waste is caused.
In the embodiment of the application, the main instance generates a target log sequence, and sends the target log sequence to the shared storage to store the target log sequence in a lasting mode. And the slave instance obtains a playback data page by copying the target log sequence and playing back the target log sequence. The master instance collects playback sites for each slave instance, determines a reference site, and swipes the reference site down in the shared store. The shared storage performs log playback based on the reference location, obtains a reference data page, and persistently stores the reference data page. Subsequently, the shared storage may perform playback of the target log sequence based on the reference data page, resulting in a playback data page of log information at any location after the reference location.
It will be appreciated that the target log sequence is used to record a plurality of modification operations to the original data page, and that different modified data pages may be considered as different versions of the data page. Based on this, the playback data page of the log information at an arbitrary position after the reference position can be referred to as a specified version of the data page.
Unlike the general scenario, the shared storage in the embodiment of the present application may perform log playback based on the reference location, i.e., the shared storage in the embodiment of the present application may perceive the service. Such shared storage capable of awareness traffic is also referred to as awareness storage. The sensing service in the embodiment of the application comprises a calculation task of log playback by a storage node.
As shown in fig. 6, the shared storage is a perception-type storage including a log storage area and a data page storage area. The master instance persists the log to the log storage area. The slave instance copies the log of the master instance and obtains the data page through log playback. The master instance collects playback points of the slave instances and determines a reference point. The perception type storage obtains a reference data page through log playback, and stores the reference data page into a data page storage area. Subsequently, when the slave instance needs to read the specified version data page, the shared storage can perform log playback based on the reference data page to obtain the specified version data page, and send the specified version data page to the slave instance, so that the specified version data page is read from the perception type storage by the slave instance.
In the general scenario, the master instance is used to play back the log and swipe the data page down to the general storage, while the embodiment of the application plays back the log by the storage node. By transferring the calculation task of log playback from the calculation layer to the storage layer, the calculation resource of the storage node is fully utilized, the resource waste of the storage node is avoided, the calculation resource of the main instance is saved, and the load pressure of the main instance is reduced.
It will be appreciated that the transfer of the computing tasks of log playback from the computing layer to the storage layer is not a complete transfer, but a partial transfer. That is, the slave instance located in the compute layer requires the computation task of log playback, while the master instance located in the compute layer does not require the computation task of log playback, and the storage node located in the storage layer requires the computation task of log playback.
As shown in fig. 7, in one aspect, a master instance generates a target log sequence that includes logs 1 through 2000. The master instance synchronizes the target log sequence to the shared store, the target log sequence is located in the redo log queue and is arranged according to logs 1 through 2000.
On the other hand, the target log sequence is synchronized from the main instance from the instances 1 to 3, and the target log sequence is played back. At some point, the playback locus from example 1 is 500, the playback locus from example 2 is 1000, and the playback locus from example 3 is 1500. The master instance gathers the playback loci of the individual slave instances, resulting in the slowest playback locus, here 500. When the reference site is the slowest playback site, the master instance synchronizes the slowest playback site to the shared storage.
The background thread sharing the storage will log the target log sequence in the redo log queue based on the slowest playback site 500. The playback area of the log is log 1 to log 500, that is, the target log sequence plays back log 1 to log 500 in the redo log queue in sequence, and the data page obtained after the log 500 is played back is stored in the disk in a lasting manner.
In an exemplary embodiment, step A2 further comprises, after: and releasing the log information positioned in the target log sequence and before the reference position from the log storage area to obtain the reference log sequence stored in the log storage area.
In the embodiment of the application, the storage node has completed playback of the log information located at and before the reference site in the target log sequence, so that the log information located at and before the reference site in the target log sequence belongs to the used information. The storage node saves the storage space of the storage node by releasing the used information.
Optionally, since the target log sequence is stored in the log storage area, releasing the log information in the target log sequence is equivalent to releasing the log information from the log storage area, thereby saving the storage space of the log storage area.
It should be noted that the above-mentioned manner of the storage node acquiring the reference data page and the reference log sequence is merely exemplary, and other implementations are also possible in practical applications. For example, after the master instance obtains the reference location, the master instance plays back the log information located at and before the reference location in the target log sequence to obtain the reference data page, and uses the log information located at and after the reference location in the target log sequence as the reference log sequence, and sends the reference data page and the reference log sequence to the storage node.
In the embodiment of the application, the storage node stores a reference data page and a reference log sequence. After the storage node receives the first read request sent by the first slave instance, the first location may be extracted from the first read request. If the first location is smaller than the reference location, the reference data page is a data page obtained by modifying the first data page at least once. If the storage node does not store the first data page and the storage node does not store log information in the target log sequence that precedes the first location, the storage node cannot obtain the first data page. In this case, the storage node sends an error hint to the first slave instance, informing the first slave instance that the storage node cannot provide the first page of data by means of the error hint. If the first location is equal to the reference location, the reference data page is the first data page. The storage node may send the first page of data directly to the first slave instance. If the first location is greater than the reference location, the first data page is a data page that results from at least one modification to the reference data page, and the reference log sequence records the modifications to the reference data page. In this case, the storage node may play back the reference log sequence based on the reference data page, resulting in the first data page.
In one possible implementation, step 202 includes steps 2021 to 2022 (not shown in the figures).
In step 2021, for the first log information in the reference log sequence, the reference data page is modified based on the modification operation recorded by the first log information, so as to obtain a playback data page of the first log information, where when the first log information is the log information of the first location, the playback data page of the first log information is the first data page.
In the embodiment of the application, the first log information in the reference log sequence is used for recording the modification operation for modifying the reference data page. For example, the first log information records the addition of one line of data in the reference data page. The storage node performs modification corresponding to the modification operation on the reference data page based on the modification operation recorded by the first log information by playing back the first log information, for example, adding one line of data in the reference data page to obtain a playback data page of the first log information.
It is understood that the modification operation performed on the reference data page includes an operation of inserting data, an operation of deleting data, an operation of modifying data, and the like. If the first location is the reference location plus 1, the playback data page of the first log information is the first data page.
In step 2022, if the first log information is not the first location log information, for the non-first log information in the reference log sequence, modifying the playback data page of the previous log information based on the modification operation recorded by the non-first log information to obtain a playback data page of the non-first log information, where the playback data page of the non-first log information is the first data page when the non-first log information is the first location log information.
In the embodiment of the application, the non-first log information in the reference log sequence is used for recording the modification operation for modifying the playback data page of the last log information. For example, the 750 th log information records a column of data in the playback data page from which the 749 th log information is deleted. The storage node performs modification corresponding to the modification operation on the playback data page of the previous log information based on the modification operation recorded by the non-first log information by playing back the non-first log information, for example, deleting a column of data in the playback data page of the 749 th log information, to obtain the playback data page of the non-first log information.
It is understood that the modification operation performed on the playback data page of the last log information includes an operation of inserting data, an operation of deleting data, an operation of modifying data, and the like. If the non-first log information is the log information located at the first location, the playback data page of the non-first log information is the first data page.
Step 203, a first data page is sent to the first slave instance for the first slave instance to play back log information located after the first site in the target log sequence based on the first data page.
In the embodiment of the application, the storage node responds to the first read request sent by the first slave instance, and sends the first data page to the first slave instance after the first data page is obtained through playback. After the first slave instance receives the first data page, the log information located after the first site in the target log sequence is played back based on the first data page.
Alternatively, log information located after the first site in the target log sequence is referred to as a first log sequence. The first log information in the first log sequence is used for recording a modification operation for modifying the first data page, and the first slave instance can modify the first data page corresponding to the modification operation based on the modification operation recorded by the first log information to obtain a playback data page of the first log information. For the non-first log information in the first log sequence, the non-first log information is used for recording a modification operation for modifying the playback data page of the last log information, and the first slave instance can modify the playback data page of the last log information corresponding to the modification operation based on the modification operation recorded by the non-first log information to obtain the playback data page of the non-first log information.
It will be appreciated that in a general scenario, the shared storage cluster stores only the playback data pages of the slowest playback site, while the playback site from the instance may be the slowest playback site, possibly greater than the slowest playback site. Since the slave instance can only read the playback data page of the slowest playback site from the shared storage, the slave instance can only continue to play back the target log sequence from the slowest playback site based on the playback data page of the slowest playback site, so that the slave instance with the playback site larger than the slowest playback site has the phenomenon of repeated playback, a great deal of repeated calculation is caused, the calculation resources are wasted, and the played back log information cannot be released, so that the storage resources are wasted.
As shown in fig. 8, in the general scenario, the last playback site from example 1 is log 500, the last playback site from example 2 is log 1000, and the last playback site from example 3 is log 1500. The master instance collects the last playback points from examples 1 to 3, resulting in the slowest playback point being log 500. The master instance plays back the target log sequence to the log 500 to obtain a playback data page of the log 500, and downstreams the playback data page of the log 500 in the shared storage cluster.
If the buffer of the slave instance reaches a threshold value, the slave instance needs to release the buffer, which may lead to the playback data page of the last playback site being obsolete, in which case the slave instance can only read the playback data page from the shared storage cluster. Since only the playback data page of the log 500 exists in the shared storage cluster, only the playback data page of the log 500 can be read from each of examples 1 to 3.
For the slave example 1, since the last playback site of the slave example 1 is the log 500, the current playback site and the last playback site of the slave example 1 coincide, and the slave example 1 can continue the last playback based on the playback data page of the log 500, that is, the slave example 1 continues the playback of the target log sequence from the log 501 based on the playback data page of the log 500.
For the slave example 2, since the last playback site from example 2 is the log 1000, and only the playback data page of the log 500 can be acquired from example 2, the current playback site and the last playback site from example 2 do not coincide. From example 2, a playback target log sequence is required based on the playback data page of log 500, starting from log 501. This results in repeated playback of log 501 to log 1000 from instance 2, wasting computing resources, and failure to release log 501 to log 1000 from instance 2, wasting storage resources.
Similarly, for slave example 3, slave example 3 requires playback of the data page based on log 500, starting with log 501 with the playback target log sequence. This results in repeated playback of log 501 to log 1500 from instance 3, wasting computing resources, and failure to release log 501 to log 1500 from instance 3, wasting storage resources.
In the embodiment of the application, the first slave instance plays back the log information positioned at the first site and before the first site in the target log sequence to obtain the first data page. When the first slave instance releases the first data page, the storage node obtains the first data page by playing back the reference log sequence and sends the first data page to the first slave instance, so that the first data page can continue playing back the target log sequence based on the first data page. That is, the first slave instance does not need to repeatedly play back, saving computing resources, and the first slave instance can release the played back log information, saving storage resources.
In a possible implementation, the method according to an embodiment of the present application further includes step 204 (not shown in the figure). Wherein step 204 is performed prior to step 202.
In step 204, the first read request is stored in a request queue, where the request queue is used to store a second read request sent by a second slave instance, where the second slave instance is used to play back log information located at a second location and before the second location in the target log sequence, and the second read request is used to request to read a second data page released by the second slave instance, where the second data page is a data page obtained after playing back the log information located at the second location.
In the embodiment of the present application, the playback site of the second slave instance is the second site. That is, the second slave instance has completed playback of log information located before and at the second point in the target log sequence. For example, the second site is 1000, and then the second slave instance has completed playback of log information 1 to 1000 in the target log sequence.
After the second slave instance plays back the log information at the second location, a second data page may be obtained, and typically, the second data page is stored in the memory of the second slave instance. It will be appreciated that the second slave instance also needs to perform other service processing, for example, the second slave instance needs to process the read request of the client in real time, and the service processing data needs to be stored in the memory. Because of the internal limitations, the second slave instance requires an untimely release of memory to cope with various situations. If the second data page in the memory of the second slave instance is released and the second slave instance needs to continue playback of the target log sequence, i.e. the second slave instance needs to play back log information in the target log sequence that is located after the second location, the second slave instance needs to read the second data page from the storage node. In this case, the second slave instance sends a second read request to the storage node to request a read of the second page of data from the storage node via the second read request.
When the storage node receives the second read request, the second read request is deposited in a request queue. In addition, when the storage node receives the first read request, the first read request is also deposited in the request queue.
It should be noted that at least one second read request is provided, and the second data pages requested by each two second read requests may be the same or different data pages. That is, the second sites corresponding to each two second read requests may be the same or different. Any of the second sites may be greater than or equal to or less than the first sites. The order in which the storage node receives any one of the second read requests and the first read requests is not limited, for example, the storage node may receive the second read request first, then the first read request, and then another second read request.
Step 202 comprises: and if the request queue meets the execution condition, playing back the reference log sequence based on the reference data page to obtain a first data page and a second data page.
The embodiment of the application does not limit the execution condition satisfied by the request queue. In an exemplary embodiment, the request queue satisfying the execution condition includes at least one of a condition B1 and a condition B2.
The sum of the numbers of the first read request and the second read request is not less than the number threshold value, condition B1.
And (2) the waiting time of the first read request in the request queue exceeds a time threshold value.
The number threshold may be determined by a method not limited herein, and the data threshold may be a value determined randomly, or the number threshold may be a value specified by a user, for example, the number threshold is 5. Similarly, the manner in which the duration threshold is determined is not limited herein, and the duration threshold may be a randomly determined value, or the duration threshold may be a user-specified value, for example, the user-specified duration threshold is 30 seconds, for example.
The above conditions B1 and B2 are merely exemplary, and other conditions are also possible in practical applications. For example, the request queue satisfying the execution condition includes at least one of the conditions B1 to B3, the condition B3 being: the set time is reached and there is a read request in the request queue.
If the request queue meets the execution condition, the storage node plays back the reference log sequence based on the reference data page to obtain the first data page and the second data page, and the playback mode is described above and will not be described herein.
In an exemplary embodiment, after the playback of the reference log sequence based on the reference data page to obtain the first data page and the second data page, the method further includes: and sending the second data page to the second slave instance for playback of log information located after the second location in the target log sequence based on the second data page by the second slave instance.
In the embodiment of the application, the storage node responds to the second read request sent by the second slave instance, and sends the second data page to the second slave instance after playing back the second data page. After the second slave instance receives the second data page, the log information located after the second position point in the target log sequence is played back based on the second data page.
As shown in fig. 9, the playback site from example 1 is a log 500, and a request for reading a playback data page of the log 500 is sent from example 1 to the shared storage site 500. Similarly, from example 2, the playback site is log 1000, and from example 2, a site 1000 request is sent to the shared storage for requesting a read of a playback data page of log 1000. From example 3, the playback site is log 1500, and from example 3, a request is sent to the shared storage site 1500 for requesting a read of a playback data page of log 1500. The shared store places the site 500 request, the site 1000 request, and the site 1500 request in a data page request queue. The location 500 request, the location 1000 request, and the location 1500 request correspond to the first read request or the second read request, and the data page request queue is the request queue mentioned above.
It has been mentioned above that if the reference location is the log 500, the storage node plays back the target log sequence and releases the played back log information, resulting in a reference data page and a reference log sequence. Wherein the reference data page is a playback data page of the log 500, and the reference log sequence includes log information following the log 500 in the target log sequence. As shown in fig. 9, the log reservation area includes a log 500 and a reference log sequence following the log 500.
When a request in the data page request queue reaches a quantity threshold, the shared storage triggers a dying log playback. Since the playback data page of log 500 has been deposited in the shared store, the shared store triggers a reply to the request for site 500, i.e., the shared store sends the playback data page of log 500 to slave instance 1 so that playback of the target log sequence can continue from site 500 from instance 1.
Next, the shared storage plays back the log information in the log-reserved area from the log 500. When the shared storage finishes playing back the log 1000, the shared storage obtains a playback data page of the log 1000, and at this time, the shared storage triggers a reply to the request of the location 1000, that is, the shared storage sends the playback data page of the log 1000 to the slave instance 2, so that the slave instance 2 can continue the playback of the target log sequence from the location 1000.
Thereafter, the shared storage continues playback of the log information in the log retention area from the log 1000. When the shared storage has completed playing back log 1500, the shared storage obtains the playback data page of log 1500, at which point the shared storage triggers a reply to the request from site 1500, i.e., the shared storage sends the playback data page of log 1500 to slave instance 3 so that playback of the target log sequence can continue from site 1500 from instance 3.
By the method, playback of the temporary log at one time is realized, a plurality of playback data pages requested by the request are obtained, playback times are saved, and calculation pressure of the storage node is reduced.
In one possible implementation, the method of an embodiment of the present application is performed by a storage cluster comprising a first storage node and a plurality of second storage nodes, different second storage nodes being used for managing data pages of different numbered intervals.
The first storage node is used for acquiring a first read request, and the first read request carries the number of the first data page. That is, the first storage node is used to perform step 201.
The first storage node is further configured to send a first read request to a target storage node in the plurality of second storage nodes, where a number interval corresponding to the target storage node includes a number of the first data page.
And the target storage node is used for playing back the reference log sequence based on the reference data page, obtaining a first data page and sending the first data page to the first slave instance. That is, the target storage node is configured to perform steps 202 and 203.
In the embodiment of the application, each data page belongs to a table space. Tablespace is a logical concept for unified management of data pages in space. It will be appreciated that a storage cluster is used to manage a database, which may correspond to multiple tablespaces, but that a tablespace may correspond to only one database. In short, the database corresponds to a room enclosure, the table space corresponds to a space inside the room, and the data page corresponds to a box in which the object is placed.
Each data page in the tablespace corresponds to a data page number that may reflect the offset of the data page relative to the tablespace. In the embodiment of the application, any table space can be partitioned to obtain a plurality of logical partitions, so that the table space comprises all data pages and is partitioned into different logical partitions. The partition mode is not limited herein. Illustratively, the partitions may be randomly or uniformly partitioned, e.g., 2 gigabytes (Gigabyte, GB) per logical partition, or may be performed according to a user-specified partitioning scheme.
Different logical partitions may be assigned to different second storage nodes according to a data distribution algorithm. The embodiment of the application does not limit the data distribution algorithm, and the data distribution algorithm can be a polling algorithm or a consistent hash algorithm, for example. Since one logical partition corresponds to a plurality of data pages in the table space and each data page corresponds to a respective number, a different second storage node is used to manage the data pages of different numbered intervals.
The first read request carries the number of the first data page. The first storage node may determine a number section in which the number of the first data page is located based on the number of the first data page and the number section managed by each second storage node, and send a first read request to the target storage node by using the second storage node corresponding to the number section in which the number of the first data page is located as the target storage node, so that the target storage node executes step 202 and step 203.
It will be appreciated that since the target log sequence is used to record modification operations that modify the original data page a number of times, each playback data page derived based on the target log sequence is a different version of the same data page. That is, the playback data pages correspond to the numbers of the same data page. The data pages in different numbered intervals are managed through different second storage nodes, so that log sequences of different data page numbers are managed through different second storage nodes, log playback of the data pages in different partitions can be executed through different second storage nodes, multi-node computing resources of a storage cluster are fully utilized, and the efficiency of reading the data pages from an instance can be accelerated through concurrent log playback.
As shown in fig. 10, partition 1, partition 2, partition 3, and so on are obtained by partitioning the table space. Shared storage is a storage cluster comprising storage node 1, storage node 2, storage node 3, and so on. Storage node 1 manages partition 1, storage node 2 manages partition 2, and storage node 3 manages partition 3.
The master instance generates a target log sequence and sends the target log sequence to the shared store. If the target log sequence is used to record a modification operation for modifying the original data page belonging to partition 1 a plurality of times, the target log sequence is deposited in storage node 1 to enable storage node 1 to play back the target log sequence. Similarly, if the target log sequence is used to record a modification operation that modifies an initial data page belonging to partition 2a plurality of times, the target log sequence is stored in storage node 2 so that storage node 2 performs playback of the target log sequence.
Taking storage node 1 as an example, as shown in fig. 10, the target log sequence is located in the redo log queue, including logs 1 through 2000. When the reference site is the slowest playback site, the master instance synchronizes the slowest playback site to storage node 1. The background thread of the storage node 1 performs log playback on the target log sequence in the redo log queue based on the slowest playback site. The playback area of the log is log 1 to log 500, that is, the target log sequence plays back log 1 to log 500 in the redo log queue in sequence, and the data page obtained after the log 500 is played back is stored in the disk in a lasting manner.
It will be appreciated that other nodes, such as the storage node 2 and the storage node 3, may also perform the operations of the storage node 1 described above, which will not be described herein.
It should be noted that, the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals related to the present application are all authorized by the user or are fully authorized by the parties, and the collection, use and processing of the related data need to comply with the relevant laws and regulations and standards of the relevant region. For example, the data pages, log sequences, etc. referred to in this application are all acquired with sufficient authorization.
In the method, the first slave instance plays back the log information located at the first site and before the first site in the target log sequence to obtain the first data page. When the first slave instance releases the first page of data, the first slave instance may send a first read request requesting to read the first page of data. The first data page is obtained by playing back the reference log sequence based on the reference data page, and the first data page is sent to the first slave instance, so that the first slave instance can play back log information positioned behind the first site in the target log sequence based on the first data page, the first slave instance can play back continuously at the last playback position without repeated playback, and the computing resource of the first slave instance is saved.
The foregoing describes the method for reading a data page according to the embodiment of the present application from the perspective of method steps, and is described in more detail below from the perspective of a data page reading system. Wherein the data page reading system comprises a master instance, a storage node and at least one slave instance. For ease of description, the following description will be given by taking two examples as examples. As shown in fig. 11, the data page reading system includes a master instance, a storage node, a slave instance 1, and a slave instance 2.
In the embodiment of the application, the master instance is used for generating the target log sequence and sending the target log sequence to the storage node, the slave instance 1 and the slave instance 2. The storage node is used for storing the target log sequence in the log storage area. From example 1 for playback of the target log sequence, a first location and a first data page are obtained, and from example 2 for playback of the target log sequence, a second location and a second data page are obtained.
The first site is sent from instance 1 to the master instance and the second site is sent from instance 2 to the master instance. The master instance determines a slowest playback site based on the first site and the second site and determines a reference site based on the slowest playback site. The reference site is less than or equal to the slowest playback site. The master instance sends a reference location to the storage node.
The storage node plays back the target log sequence based on the reference node and releases the played back log information to obtain the reference data page and the reference log sequence. The reference data page is stored in the data page storage area, and the reference log sequence is stored in the log storage area.
Optionally, the first data page is released from instance 1. In this case, a data page read request carrying the first location is sent from instance 1 to the storage node. Similarly, the second page of data is released from example 2. In this case, a data page read request carrying the second location is sent from instance 2 to the storage node.
Optionally, the reference site is smaller than the first site, and the first site is smaller than the second site. In this case, the storage node plays back the reference log sequence based on the reference data page, resulting in a first location and a first data page, which the storage node sends to from example 1. Playback of the target log sequence from example 1 continues from the first location based on the first page of data. Further, the storage node continues to play back the reference log sequence from the first location based on the first data page, resulting in a second location and a second data page, which the storage node sends to slave instance 2. Playback of the target log sequence from the second location is continued from example 2 based on the second data page.
In summary, as shown in FIG. 12, the master instance in embodiments of the present application no longer swipes the persisted dirty data page down the shared store, but persists the log. The shared storage deposits the log in the log storage area, and plays back the log to obtain the data page using sufficient computing resources, and persists the data page to the data page storage area. And sending a data page reading request carrying the specified site from the instance to the shared storage, and obtaining the playback data page of the specified site by the storage node through the playback log and sending the playback data page to the slave instance, so that the slave instance can continue to play back based on the playback data page of the specified site without repeatedly carrying out log playback calculation, thereby avoiding the waste of calculation resources.
For example, in fig. 12, a data page read request carrying the site 500 is transmitted from the example 1 to the shared storage, and the storage node obtains the playback data page of the site 500 through the playback log and transmits to the example 1 so that the playback can be continued based on the playback data page of the site 500 from the example 1. Similarly, a data page read request carrying site 1000 is sent from instance 2 to the shared storage, and the storage node sends the playback data page of site 1000 to the slave instance 2 so that playback can continue from instance 2 based on the playback data page of site 1000. Sending a data page read request carrying site 1500 from instance 3 to the shared storage, the storage node sends the playback data page of site 1500 from instance 3 so that playback can continue from instance 3 based on the playback data page of site 1500.
By the method, the calculation task of log playback is transferred from the master node to the storage node with sufficient calculation resources, and the storage node allows the appointed position when the data page is read from the instance, so that the reading of the multi-version data page is realized. Meanwhile, the storage nodes combine the read requests of the multi-version data pages, and each request is replied through one-time log sequence playback, so that the waste of the computing resources of the storage nodes is avoided, the computing resources of the slave examples are saved, and the use experience of a user on the read-write separation database is improved.
Fig. 13 is a schematic structural diagram of a data page reading device according to an embodiment of the present application, and as shown in fig. 13, the device includes the following modules.
The obtaining module 1301 is configured to obtain a first read request sent by a first slave instance, where the first slave instance is configured to play back log information located at a first location and before the first location in the target log sequence, and the first read request is configured to request to read a first data page released by the first slave instance, where the first data page is a data page obtained after playing back the log information located at the first location.
The playback module 1302 is configured to play back a reference log sequence based on a reference data page to obtain a first data page, where the reference data page is a data page obtained after playing back log information located at a reference location in the target log sequence, and the reference location is smaller than the first location, and the reference log sequence includes log information located after the reference location in the target log sequence.
The sending module 1303 is configured to send a first data page to the first slave instance, for the first slave instance to play back log information located after the first site in the target log sequence based on the first data page.
In one possible implementation, the obtaining module 1301 is further configured to obtain a reference location sent by the master instance, where the reference location is not greater than a slowest playback location among playback locations of the plurality of slave instances, where the playback location of the slave instance is used to characterize that log information of a last playback completion of the slave instance when playing back the target log sequence is at a location of the target log sequence, and the plurality of slave instances includes the first slave instance.
The playback module 1302 is further configured to play back the log information located before the reference location and the reference location in the target log sequence, to obtain a reference data page.
In a possible implementation manner, the obtaining module 1301 is further configured to obtain a target log sequence sent by the master instance, where the target log sequence is used to record a modification operation that the master instance makes multiple modifications to the initial data page.
The apparatus also includes the following modules.
And the storage module is used for storing the target log sequence in the log storage area.
And the playback module 1302 is configured to read the target log sequence from the log storage area, and play back the log information located before the reference location and the reference location in the target log sequence, so as to obtain the reference data page.
In one possible implementation, the apparatus further includes the following modules.
And the release module is used for releasing the log information positioned in the target log sequence and before the reference position from the log storage area to obtain the reference log sequence stored in the log storage area.
In one possible implementation, the apparatus further includes the following modules.
And the storage module is used for storing the reference data page in the data page storage area.
The playback module 1302 is configured to read a reference data page from the data page storage area, and play back the reference log sequence based on the reference data page, to obtain a first data page.
In a possible implementation manner, the playback module 1302 is configured to modify, for a first log information in the reference log sequence, the reference data page based on a modification operation recorded by the first log information, to obtain a playback data page of the first log information, where the playback data page of the first log information is the first data page when the first log information is the log information of the first location; and if the first log information is not the log information of the first site, modifying the playback data page of the last log information based on the modification operation recorded by the non-first log information for the non-first log information in the reference log sequence to obtain the playback data page of the non-first log information, wherein when the non-first log information is the log information of the first site, the playback data page of the non-first log information is the first data page.
In one possible implementation, the apparatus further includes the following modules.
The storage module is used for storing the first read request in a request queue, the request queue is used for storing a second read request sent by a second slave example, the second slave example is used for playing back log information positioned at a second site and before the second site in the target log sequence, the second read request is used for requesting to read a second data page released by the second slave example, and the second data page is a data page obtained after the log information positioned at the second site is played back.
And the playback module 1302 is configured to play back the reference log sequence based on the reference data page if the request queue meets the execution condition, so as to obtain a first data page and a second data page.
In one possible implementation, the request queue meeting the execution condition includes at least one of: the sum of the number of first read requests and the number of second read requests is not less than a number threshold; the waiting time of the first read request in the request queue exceeds a time threshold.
In one possible implementation, the sending module 1303 is further configured to send a second data page to the second slave instance, for the second slave instance to play back log information located after the second location in the target log sequence based on the second data page.
In one possible implementation, the apparatus is located on a storage cluster comprising a first storage node and a plurality of second storage nodes, different second storage nodes being used for managing data pages of different numbered intervals.
The first storage node is used for acquiring a first read request, and the first read request carries the number of the first data page.
The first storage node is further configured to send a first read request to a target storage node in the plurality of second storage nodes, where a number interval corresponding to the target storage node includes a number of the first data page.
And the target storage node is used for playing back the reference log sequence based on the reference data page, obtaining a first data page and sending the first data page to the first slave instance.
In the device, the first slave instance plays back the log information positioned at the first site and before the first site in the target log sequence to obtain the first data page. When the first slave instance releases the first page of data, the first slave instance may send a first read request requesting to read the first page of data. The first data page is obtained by playing back the reference log sequence based on the reference data page, and the first data page is sent to the first slave instance, so that the first slave instance can play back log information positioned behind the first site in the target log sequence based on the first data page, the first slave instance can play back continuously at the last playback position without repeated playback, and the computing resource of the first slave instance is saved.
It should be understood that, in implementing the functions of the apparatus provided in fig. 13, only the division of the functional modules is illustrated, and in practical application, the functional modules may be allocated to different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and the method embodiments provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the apparatus and the method embodiments are detailed in the method embodiments and are not repeated herein.
Fig. 14 shows a block diagram of a terminal device 1400 provided by an exemplary embodiment of the present application. The terminal device 1400 includes: a processor 1401 and a memory 1402.
Processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1401 may be implemented in at least one hardware form of DSP (DIGITAL SIGNAL Processing), FPGA (Field-Programmable gate array), PLA (Programmable Logic Array ). Processor 1401 may also include a main processor, which is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit, central processor), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1401 may be integrated with a GPU (Graphics Processing Unit, image processor) for rendering and rendering of content that is required to be displayed by the display screen. In some embodiments, the processor 1401 may also include an AI (ARTIFICIAL INTELLIGENCE ) processor for processing computing operations related to machine learning.
Memory 1402 may include one or more computer-readable storage media, which may be non-transitory. Memory 1402 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1402 is used to store at least one computer program for execution by processor 1401 to implement a method of reading a data page provided by an embodiment of a method in the present application.
In some embodiments, the terminal device 1400 may further optionally include: a peripheral interface 1403 and at least one peripheral. The processor 1401, memory 1402, and peripheral interface 1403 may be connected by a bus or signal lines. The individual peripheral devices may be connected to the peripheral device interface 1403 via buses, signal lines or a circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 1404, a display screen 1405, a camera assembly 1406, audio circuitry 1407, and a power source 1408.
Peripheral interface 1403 may be used to connect at least one Input/Output (I/O) related peripheral to processor 1401 and memory 1402. In some embodiments, processor 1401, memory 1402, and peripheral interface 1403 are integrated on the same chip or circuit board; in some other embodiments, either or both of processor 1401, memory 1402, and peripheral interface 1403 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.
The Radio Frequency circuit 1404 is configured to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1404 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 1404 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1404 includes: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and so forth. The radio frequency circuit 1404 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the world wide web, metropolitan area networks, intranets, generation mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (WIRELESS FIDELITY ) networks. In some embodiments, the radio frequency circuit 1404 may also include NFC (NEAR FIELD Communication) related circuits, which are not limited by the present application.
The display screen 1405 is used to display UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 1405 is a touch display screen, the display screen 1405 also has the ability to collect touch signals at or above the surface of the display screen 1405. The touch signal may be input to the processor 1401 as a control signal for processing. At this time, the display 1405 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 1405 may be one, disposed on the front panel of the terminal device 1400; in other embodiments, the display 1405 may be at least two, respectively disposed on different surfaces of the terminal device 1400 or in a folded design; in other embodiments, the display 1405 may be a flexible display disposed on a curved surface or a folded surface of the terminal device 1400. Even more, the display 1405 may be arranged in a non-rectangular irregular pattern, i.e. a shaped screen. The display 1405 may be made of LCD (Liquid CRYSTAL DISPLAY), OLED (Organic Light-Emitting Diode), or other materials.
The camera component 1406 is used to capture images or video. Optionally, camera assembly 1406 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal and the rear camera is disposed on the rear surface of the terminal. In some embodiments, the at least two rear cameras are any one of a main camera, a depth camera, a wide-angle camera and a tele camera, so as to realize that the main camera and the depth camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize a panoramic shooting and Virtual Reality (VR) shooting function or other fusion shooting functions. In some embodiments, camera assembly 1406 may also include a flash. The flash lamp can be a single-color temperature flash lamp or a double-color temperature flash lamp. The dual-color temperature flash lamp refers to a combination of a warm light flash lamp and a cold light flash lamp, and can be used for light compensation under different color temperatures.
The audio circuitry 1407 may include a microphone and a speaker. The microphone is used for collecting sound waves of users and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 1401 for processing, or inputting the electric signals to the radio frequency circuit 1404 for voice communication. For purposes of stereo acquisition or noise reduction, a plurality of microphones may be respectively disposed at different portions of the terminal device 1400. The microphone may also be an array microphone or an omni-directional pickup microphone. The speaker is used to convert electrical signals from the processor 1401 or the radio frequency circuit 1404 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only the electric signal can be converted into a sound wave audible to humans, but also the electric signal can be converted into a sound wave inaudible to humans for ranging and other purposes. In some embodiments, audio circuitry 1407 may also include a headphone jack.
A power supply 1408 is used to power the various components in terminal device 1400. The power supply 1408 may be alternating current, direct current, disposable battery, or rechargeable battery. When the power supply 1408 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.
In some embodiments, terminal device 1400 also includes one or more sensors 1409. The one or more sensors 1409 include, but are not limited to: acceleration sensor 1411, gyroscope sensor 1412, pressure sensor 1413, optical sensor 1414, and proximity sensor 1415.
The acceleration sensor 1411 can detect the magnitudes of accelerations on three coordinate axes of the coordinate system established with the terminal apparatus 1400. For example, the acceleration sensor 1411 may be used to detect components of gravitational acceleration in three coordinate axes. The processor 1401 may control the display screen 1405 to display a user interface in a landscape view or a portrait view according to the gravitational acceleration signal acquired by the acceleration sensor 1411. The acceleration sensor 1411 may also be used for the acquisition of motion data of a game or a user.
The gyro sensor 1412 may detect the body direction and the rotation angle of the terminal device 1400, and the gyro sensor 1412 may collect the 3D motion of the user to the terminal device 1400 in cooperation with the acceleration sensor 1411. The processor 1401 may implement the following functions based on the data collected by the gyro sensor 1412: motion sensing (e.g., changing UI according to a tilting operation by a user), image stabilization at shooting, game control, and inertial navigation.
The pressure sensor 1413 may be disposed at a side frame of the terminal device 1400 and/or at an underside of the display 1405. When the pressure sensor 1413 is provided at a side frame of the terminal device 1400, a grip signal of the terminal device 1400 by a user can be detected, and the processor 1401 performs left-right hand recognition or quick operation according to the grip signal collected by the pressure sensor 1413. When the pressure sensor 1413 is disposed at the lower layer of the display screen 1405, the processor 1401 realizes control of the operability control on the UI interface according to the pressure operation of the user on the display screen 1405. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
The optical sensor 1414 is used to collect the ambient light intensity. In one embodiment, processor 1401 may control the display brightness of display screen 1405 based on the intensity of ambient light collected by optical sensor 1414. Specifically, when the intensity of the ambient light is high, the display luminance of the display screen 1405 is turned high; when the ambient light intensity is low, the display luminance of the display screen 1405 is turned down. In another embodiment, the processor 1401 may also dynamically adjust the shooting parameters of the camera assembly 1406 based on the intensity of ambient light collected by the optical sensor 1414.
A proximity sensor 1415, also referred to as a distance sensor, is typically provided on the front panel of the terminal device 1400. The proximity sensor 1415 is used to collect the distance between the user and the front of the terminal device 1400. In one embodiment, when the proximity sensor 1415 detects a gradual decrease in the distance between the user and the front face of the terminal device 1400, the processor 1401 controls the display 1405 to switch from the bright screen state to the off screen state; when the proximity sensor 1415 detects that the distance between the user and the front surface of the terminal apparatus 1400 gradually increases, the display screen 1405 is controlled by the processor 1401 to switch from the off-screen state to the on-screen state.
It will be appreciated by those skilled in the art that the structure shown in fig. 14 is not limiting and that terminal device 1400 may include more or less components than those shown, or may combine certain components, or employ a different arrangement of components.
Fig. 15 is a schematic structural diagram of a server according to an embodiment of the present application, where the server 1500 may have a relatively large difference due to different configurations or performances, and may include one or more processors 1501 and one or more memories 1502, where the one or more memories 1502 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1501 to implement the method for reading a data page according to the foregoing method embodiments, and the processor 1501 is a CPU. Of course, the server 1500 may also have a wired or wireless network interface, a keyboard, an input/output interface, etc. for performing input/output, and the server 1500 may also include other components for implementing device functions, which are not described herein.
In an exemplary embodiment, there is also provided a computer-readable storage medium having stored therein at least one computer program loaded and executed by a processor to cause an electronic device to implement a method of reading any one of the data pages described above.
Alternatively, the above-mentioned computer readable storage medium may be a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Read-Only optical disk (Compact Disc Read-Only Memory, CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, there is also provided a computer program, which is at least one computer program, loaded and executed by a processor, to cause an electronic device to implement a method of reading any one of the data pages described above.
In an exemplary embodiment, there is also provided a computer program product in which at least one computer program is stored, the at least one computer program being loaded and executed by a processor to cause an electronic device to implement a method of reading any one of the data pages described above.
It should be understood that references herein to "a plurality" are to two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the present application, any modifications, equivalent substitutions, improvements, etc. that fall within the principles of the present application should be included in the scope of the present application.
Claims (21)
1. A method of reading a page of data, the method comprising:
Obtaining a reference locus sent by a master instance, wherein the reference locus is not more than the slowest playback locus in the playback loci of a plurality of slave instances, the playback locus of the slave instance is used for representing the locus of a target log sequence in which log information of the last playback completion when the slave instance plays back the target log sequence, and the plurality of slave instances comprise a first slave instance;
the log information positioned in the target log sequence and before the reference position is played back to obtain a reference data page;
Acquiring a first read request sent by the first slave instance, wherein the first slave instance is used for playing back log information positioned at a first site and before the first site in the target log sequence, the first read request is used for requesting to read a first data page released by the first slave instance, and the first data page is a data page obtained after playing back the log information positioned at the first site;
Playback is carried out on a reference log sequence based on the reference data page, the first data page is obtained, the reference site is smaller than the first site, and the reference log sequence comprises log information located behind the reference site in the target log sequence;
The first data page is sent to the first slave instance for playback of log information in the target log sequence that is located after the first site based on the first data page by the first slave instance.
2. The method according to claim 1, wherein the method further comprises:
acquiring the target log sequence sent by a main instance, wherein the target log sequence is used for recording the modification operation of the main instance for modifying an initial data page for a plurality of times;
storing the target log sequence in a log storage area;
The step of playing back the log information located at the reference position and before the reference position in the target log sequence to obtain a reference data page includes:
And reading the target log sequence from the log storage area, and playing back log information positioned in the target log sequence and before the reference position to obtain the reference data page.
3. The method of claim 2, wherein the playing back the log information in the target log sequence before the reference location and the reference location, after obtaining the reference data page, further comprises:
and releasing the log information positioned in the target log sequence and before the reference position from the log storage area to obtain the reference log sequence stored in the log storage area.
4. The method of claim 1, wherein the playing back the log information in the target log sequence before the reference position and the reference position, after obtaining the reference data page, further comprises:
storing the reference data page in a data page storage area;
the playback of the reference log sequence based on the reference data page, to obtain the first data page, includes:
and reading the reference data page from the data page storage area, and playing back the reference log sequence based on the reference data page to obtain the first data page.
5. The method of claim 1, wherein the playing back the reference log sequence based on the reference data page results in the first data page, comprising:
Modifying the reference data page based on the modification operation recorded by the first log information to obtain a playback data page of the first log information, wherein when the first log information is the log information of the first site, the playback data page of the first log information is the first data page;
And if the first log information is not the log information of the first site, modifying a playback data page of the last log information based on the modification operation recorded by the non-first log information for the non-first log information in the reference log sequence to obtain the playback data page of the non-first log information, wherein when the non-first log information is the log information of the first site, the playback data page of the non-first log information is the first data page.
6. The method according to claim 1, wherein the method further comprises:
storing the first read request in a request queue, wherein the request queue is used for storing a second read request sent by a second slave example, the second slave example is used for playing back log information positioned at a second position and before the second position in a target log sequence, the second read request is used for requesting to read a second data page released by the second slave example, and the second data page is a data page obtained after playing back the log information positioned at the second position;
the playback of the reference log sequence based on the reference data page, to obtain the first data page, includes:
and if the request queue meets the execution condition, playing back the reference log sequence based on the reference data page to obtain the first data page and the second data page.
7. The method of claim 6, wherein the request queue meeting an execution condition comprises at least one of:
the sum of the number of the first read requests and the second read requests is not less than a number threshold;
The waiting time of the first read request in the request queue exceeds a time threshold.
8. The method of claim 6, wherein the playback of the reference log sequence based on the reference data page, after obtaining the first data page and the second data page, further comprises:
And sending the second data page to the second slave instance for playback of log information located after the second position point in the target log sequence based on the second data page by the second slave instance.
9. The method according to any of claims 1-8, wherein the method is performed by a storage cluster comprising a first storage node and a plurality of second storage nodes, different second storage nodes being used for managing data pages of different numbered intervals;
the first storage node is configured to obtain the first read request, where the first read request carries a number of the first data page;
the first storage node is further configured to send the first read request to a target storage node in the plurality of second storage nodes, where a number interval corresponding to the target storage node includes a number of the first data page;
And the target storage node is used for playing back the reference log sequence based on the reference data page, obtaining the first data page and sending the first data page to the first slave instance.
10. A device for reading a page of data, said device comprising:
The system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a reference locus sent by a master instance, the reference locus is not more than the slowest playback locus in playback loci of a plurality of slave instances, the playback locus of the slave instances is used for representing the locus of a target log sequence of log information of which the last playback is completed when the slave instances play back the target log sequence, and the plurality of slave instances comprise a first slave instance;
the playback module is used for playing back the log information positioned at the reference position and before the reference position in the target log sequence to obtain a reference data page;
The acquisition module is further configured to acquire a first read request sent by the first slave instance, where the first slave instance is configured to play back log information located at a first location and before the first location in the target log sequence, and the first read request is used to request to read a first data page released by the first slave instance, where the first data page is a data page obtained after playing back the log information located at the first location;
the playback module is further configured to playback a reference log sequence based on the reference data page to obtain the first data page, where the reference location is smaller than the first location, and the reference log sequence includes log information located after the reference location in the target log sequence;
And the sending module is used for sending the first data page to the first slave instance, so that the first slave instance plays back log information positioned behind the first locus in the target log sequence based on the first data page.
11. The apparatus of claim 10, wherein the obtaining module is further configured to obtain the target log sequence sent by a master instance, where the target log sequence is used to record a modification operation performed by the master instance on an initial data page for multiple modifications;
The apparatus further comprises:
the storage module is used for storing the target log sequence in a log storage area;
And the playback module is used for reading the target log sequence from the log storage area, and playing back log information positioned in the target log sequence and before the reference position to obtain the reference data page.
12. The apparatus of claim 11, wherein the apparatus further comprises:
And the release module is used for releasing the log information positioned in the target log sequence and before the reference position from the log storage area to obtain the reference log sequence stored in the log storage area.
13. The apparatus of claim 10, wherein the apparatus further comprises:
A storage module for storing the reference data page in a data page storage area;
and the playback module is used for reading the reference data page from the data page storage area, and playing back the reference log sequence based on the reference data page to obtain the first data page.
14. The apparatus of claim 10, wherein the playback module is configured to, for a first log information in the reference log sequence, modify the reference data page based on a modification operation recorded by the first log information to obtain a playback data page of the first log information, and when the first log information is the log information of the first location, the playback data page of the first log information is the first data page; and if the first log information is not the log information of the first site, modifying a playback data page of the last log information based on the modification operation recorded by the non-first log information for the non-first log information in the reference log sequence to obtain the playback data page of the non-first log information, wherein when the non-first log information is the log information of the first site, the playback data page of the non-first log information is the first data page.
15. The apparatus of claim 10, wherein the apparatus further comprises:
The storage module is used for storing the first reading request in a request queue, the request queue is used for storing a second reading request sent by a second slave instance, the second slave instance is used for playing back log information positioned at a second position and before the second position in a target log sequence, the second reading request is used for requesting to read a second data page released by the second slave instance, and the second data page is a data page obtained after playing back the log information positioned at the second position;
And the playback module is used for playing back the reference log sequence based on the reference data page to obtain the first data page and the second data page if the request queue meets the execution condition.
16. The apparatus of claim 15, wherein the request queue meeting an execution condition comprises at least one of: the sum of the number of the first read requests and the second read requests is not less than a number threshold; the waiting time of the first read request in the request queue exceeds a time threshold.
17. The apparatus of claim 15, wherein the means for transmitting is further for transmitting the second data page to the second slave instance for playback of log information in the target log sequence that follows the second location based on the second data page by the second slave instance.
18. The apparatus according to any of claims 10 to 17, wherein the apparatus is located on a storage cluster comprising a first storage node and a plurality of second storage nodes, different second storage nodes being used for managing data pages of different numbered intervals;
the first storage node is configured to obtain the first read request, where the first read request carries a number of the first data page;
the first storage node is further configured to send the first read request to a target storage node in the plurality of second storage nodes, where a number interval corresponding to the target storage node includes a number of the first data page;
And the target storage node is used for playing back the reference log sequence based on the reference data page, obtaining the first data page and sending the first data page to the first slave instance.
19. An electronic device comprising a processor and a memory, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to cause the electronic device to implement a method of reading a data page according to any one of claims 1 to 9.
20. A computer readable storage medium, characterized in that at least one computer program is stored in the computer readable storage medium, which is loaded and executed by a processor to cause an electronic device to implement a method of reading a data page according to any of claims 1 to 9.
21. A computer program product having at least one computer program stored therein, the at least one computer program being loaded and executed by a processor to cause an electronic device to implement a method of reading a data page according to any one of claims 1 to 9.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202410049489.5A CN117555884B (en) | 2024-01-12 | 2024-01-12 | Method, device and equipment for reading data page and readable storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202410049489.5A CN117555884B (en) | 2024-01-12 | 2024-01-12 | Method, device and equipment for reading data page and readable storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN117555884A CN117555884A (en) | 2024-02-13 |
CN117555884B true CN117555884B (en) | 2024-04-26 |
Family
ID=89817126
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202410049489.5A Active CN117555884B (en) | 2024-01-12 | 2024-01-12 | Method, device and equipment for reading data page and readable storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117555884B (en) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9460008B1 (en) * | 2013-09-20 | 2016-10-04 | Amazon Technologies, Inc. | Efficient garbage collection for a log-structured data store |
CN106855822A (en) * | 2015-12-08 | 2017-06-16 | 阿里巴巴集团控股有限公司 | For the method and apparatus of distributing real time system |
CN111190884A (en) * | 2019-12-17 | 2020-05-22 | 腾讯科技(深圳)有限公司 | Data verification method, device and computer readable storage medium |
CN111324665A (en) * | 2020-01-23 | 2020-06-23 | 阿里巴巴集团控股有限公司 | Log playback method and device |
CN113590596A (en) * | 2021-07-02 | 2021-11-02 | 阿里巴巴新加坡控股有限公司 | Data processing method, system, device, computer program product and storage medium |
CN115114370A (en) * | 2022-01-20 | 2022-09-27 | 腾讯科技(深圳)有限公司 | Synchronization method and device for master database and slave database, electronic equipment and storage medium |
CN115114258A (en) * | 2021-03-19 | 2022-09-27 | 腾讯科技(深圳)有限公司 | Data copying method and device, electronic equipment and computer storage medium |
CN115292407A (en) * | 2022-07-15 | 2022-11-04 | 阿里巴巴(中国)有限公司 | Synchronization method, apparatus and storage medium |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200050692A1 (en) * | 2018-08-10 | 2020-02-13 | Microsoft Technology Licensing, Llc | Consistent read queries from a secondary compute node |
-
2024
- 2024-01-12 CN CN202410049489.5A patent/CN117555884B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9460008B1 (en) * | 2013-09-20 | 2016-10-04 | Amazon Technologies, Inc. | Efficient garbage collection for a log-structured data store |
CN106855822A (en) * | 2015-12-08 | 2017-06-16 | 阿里巴巴集团控股有限公司 | For the method and apparatus of distributing real time system |
CN111190884A (en) * | 2019-12-17 | 2020-05-22 | 腾讯科技(深圳)有限公司 | Data verification method, device and computer readable storage medium |
CN111324665A (en) * | 2020-01-23 | 2020-06-23 | 阿里巴巴集团控股有限公司 | Log playback method and device |
CN115114258A (en) * | 2021-03-19 | 2022-09-27 | 腾讯科技(深圳)有限公司 | Data copying method and device, electronic equipment and computer storage medium |
CN113590596A (en) * | 2021-07-02 | 2021-11-02 | 阿里巴巴新加坡控股有限公司 | Data processing method, system, device, computer program product and storage medium |
CN115114370A (en) * | 2022-01-20 | 2022-09-27 | 腾讯科技(深圳)有限公司 | Synchronization method and device for master database and slave database, electronic equipment and storage medium |
CN115292407A (en) * | 2022-07-15 | 2022-11-04 | 阿里巴巴(中国)有限公司 | Synchronization method, apparatus and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN117555884A (en) | 2024-02-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112035410B (en) | Log storage method, device, node equipment and storage medium | |
CN111190748B (en) | Data sharing method, device, equipment and storage medium | |
CN110147380B (en) | Data updating method, device, server and storage medium | |
CN111694834A (en) | Method, device and equipment for putting picture data into storage and readable storage medium | |
CN111352863B (en) | Memory management method, device, equipment and storage medium | |
CN113626522A (en) | Data synchronization method, device, medium and equipment | |
CN115811520B (en) | Method and device for electing master node in distributed system and electronic equipment | |
WO2020088542A1 (en) | Data management method, apparatus and system, and server, terminal and storage medium | |
CN110636144A (en) | Data downloading method and device | |
CN109783504A (en) | Method of data synchronization, device, computer equipment and storage medium | |
CN113766043B (en) | Method, device, computer equipment and storage medium for configuring target container | |
CN117555884B (en) | Method, device and equipment for reading data page and readable storage medium | |
CN112084157A (en) | File recovery method and device, computer equipment and storage medium | |
CN112711636B (en) | Data synchronization method, device, equipment and medium | |
CN110597923B (en) | Block chain resource management method, device and storage medium | |
CN112783860B (en) | Method, device, storage medium and computer equipment for constructing mirror image database | |
CN115657943A (en) | Data processing method, device and equipment for thin volume of storage system and storage medium | |
CN112669884B (en) | Audio data processing method, device, equipment and storage medium | |
CN111190751B (en) | Task processing method and device based on song list, computer equipment and storage medium | |
CN112698787A (en) | Data writing method and device and computer storage medium | |
CN114299201A (en) | Animation frame display method, device, equipment and storage medium | |
CN117155759B (en) | Data processing method, device, computer equipment and storage medium | |
CN116700997B (en) | Capacity expansion method, device and equipment of edge cloud equipment and readable storage medium | |
CN115113994B (en) | Request processing method, device, computing equipment and storage medium | |
CN111368103B (en) | Multimedia data playing method, device, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |