WO2020108544A1 - 同步缓存数据的方法、装置和系统 - Google Patents
同步缓存数据的方法、装置和系统 Download PDFInfo
- Publication number
- WO2020108544A1 WO2020108544A1 PCT/CN2019/121356 CN2019121356W WO2020108544A1 WO 2020108544 A1 WO2020108544 A1 WO 2020108544A1 CN 2019121356 W CN2019121356 W CN 2019121356W WO 2020108544 A1 WO2020108544 A1 WO 2020108544A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- fragment
- name
- target
- segment
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 93
- 239000012634 fragment Substances 0.000 claims abstract description 472
- 230000001360 synchronised effect Effects 0.000 claims description 40
- 230000004048 modification Effects 0.000 claims description 36
- 238000012986 modification Methods 0.000 claims description 36
- 230000015654 memory Effects 0.000 claims description 26
- 230000008569 process Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 abstract description 22
- 238000013500 data storage Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000004044 response Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 239000002699 waste material Substances 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
Definitions
- the present application relates to the field of data storage, in particular to a method, device and system for synchronizing cache data.
- the interaction between devices becomes particularly important. For example, the interaction between the terminal and the server, the interaction between the server and the server, etc.
- the data stored on the devices to be interacted is generally synchronized.
- the specific processing is that each time the data on a device is modified, it will be synchronized to the device interacting with the device, so that the interacting device updates the cached data in a timely manner.
- the embodiments of the present disclosure provide a method, device and system for synchronizing cache data.
- the technical solution is as follows:
- a method for synchronizing cached data includes:
- the first device receives a data synchronization request sent by the second device, where the data synchronization request carries the name of the fragment cached by the second device;
- the first device determines a time interval between the current time the first device receives the data synchronization request and the last time it receives the data synchronization request sent by the second device;
- the first device determines, according to the time interval, the name of the segment cached by the second device, and the duration of the cache emptying corresponding to the second device, the name of the segment cached by the second device determines The name of the shard that is emptied, the name of the shard to be updated, and the name of the shard that has not been updated;
- the first device sends to the second device the name of the fragment to be cleared, the name of the non-updated fragment and the name of the fragment to be updated, and each of the fragments to be updated Updated data.
- the data synchronization request also carries a time stamp of the last time the first device received the data synchronization request sent by the second device;
- the first device determining the time interval between the first device receiving the data synchronization request this time and the last time receiving the data synchronization request sent by the second device includes:
- the first device takes the difference between the time stamp of the data synchronization request received by the first device this time and the time stamp carried in the data synchronization request to obtain the data received by the first device this time
- the time interval between the synchronization request and the last time the data synchronization request sent by the second device is received.
- the name of the fragment cached by the second device determines the name of the fragment to be cleared, the name of the fragment to be updated and the name of the fragment without update, including:
- the first device determines the fragment emptying duration corresponding to the fragments according to the names of the fragments cached by the second device ;
- the first device determines the name of the fragment to be cleared and the to-be-updated in the name of the fragment cached by the second device according to the time interval and the fragment clear duration corresponding to each fragment The name of the shard and the name of the shard without update.
- the method further includes:
- the first device sends a cache emptying notification to the second device to enable the second device to store the currently stored fragment The data is cleared.
- the first device determining, according to the name of each segment cached by the second device, the segment clearing duration corresponding to each segment includes:
- the first device determines, according to the name of each fragment cached by the second device, the correspondence between the name of the stored fragment and the fragment type, the fragment type to which each fragment cached by the second device belongs;
- the first device determines the fragment clearing duration corresponding to the fragment type to which each fragment belongs according to the stored correspondence between the fragment type and the fragment clearing duration.
- the first device determines the name of the segment to be emptied and the pending segment in the segments cached by the second device according to the time interval and the duration of the segment corresponding to the segments.
- the name of the updated slice and the name of the non-updated slice include:
- the first device determines that the name of the fragment is the fragment to be cleared If the time interval is less than the fragment clearing duration of the fragment, the first device determines the target synchronization duration and the number of target synchronization records corresponding to the fragment.
- the target synchronization duration is used to indicate The longest interval at which the data of the fragment is synchronized, the target synchronization record number is used to indicate the maximum number of times the fragment is synchronized;
- the first device determines that the name of the fragment is the name of the fragment to be cleared;
- the first device determines that the name of the fragment is the name of the fragment without update
- the first device determines that the name of the fragment is the name of the fragment to be updated;
- the first device determines that the name of the slice is the name of the slice to be updated.
- the first device determining the target synchronization duration and the number of target synchronization records corresponding to the fragments includes:
- the first device determines the segment type to which the segment belongs according to the correspondence between the stored segment name and the segment type
- the first device determines the target synchronization duration and the number of target synchronization records corresponding to the segment type to which the segment belongs according to the stored correspondence between the segment type, the target synchronization duration, and the number of target synchronization records.
- the method further includes:
- the first device obtains the number of synchronized records of the fragments from the target cache area.
- the method further includes:
- the first device receives a data modification request, where the data modification request carries the modified data, the identifier of the data before modification, and the name of the segment to which the modified data belongs;
- the first device modifies the data corresponding to the identification of the data before modification into modified data in the database, and records the modified data in the target cache area Correspondence between the name of the shard to which it belongs, the target shard type, the timestamp storing the modified data and the modified data, and the target shard type is the shard to which the modified data belongs The shard type corresponding to the name of.
- the first device receiving the data synchronization request sent by the second device includes:
- the first device When the connection between the first device and the second device is a short connection, the first device receives a data synchronization request sent by the second device every target period.
- the method further includes:
- connection between the first device and the second device of the first device is a long connection
- the segment to which the target data belongs is the second device
- the data synchronization notification is sent to the second device, where the data synchronization notification carries the modified target data and the name of the segment to which the modified target data belongs.
- the data synchronization request carries the name of the segment whose number of requests exceeds the target number of the segments cached by the second device, where the requested number of times refers to the target duration closest to the current time point The number of times requested.
- a method for synchronizing cached data includes:
- the second device sends a data synchronization request to the first device, where the data synchronization request carries the name of the fragment cached by the second device;
- the second device receives the name of the fragment to be cleared, the name of the fragment without update, the name of the fragment to be updated, and the data to be updated in each fragment to be updated sent by the first device;
- the second device performs an emptying process on each segment corresponding to the name of the segment to be cleared, and updates data to be updated in each segment corresponding to the name of the segment to be updated deal with.
- the second device sending a data synchronization request to the first device includes:
- the second device When the connection between the second device and the first device is a short connection, the second device sends a data synchronization request to the first device every target period.
- the method further includes:
- the second device When the connection between the second device and the first device is a long connection, the second device receives a data synchronization notification sent by the first device, where the data synchronization notification carries a modified target Data, and the name of the segment to which the modified target data belongs;
- the target data is cached in the fragment to which the modified target data belongs in the second device, the target data is modified to the modified target data, if the second device If the target data is not cached in the segment to which the modified target data belongs, the modified target data is deleted.
- the method further includes:
- the second device When receiving the data query request, the second device creates a new segment if the data corresponding to the data query request does not belong to the segment currently cached by the second device, and sends the creation to the first device A message of a new segment; when receiving the corresponding data based on the data query request from the server, the second device stores the corresponding data in the new segment and feeds back the data query request Corresponding data;
- the segment to which the data corresponding to the data query request belongs is the target segment currently cached by the second device, feedback is performed based on the data corresponding to the data query request in the target segment.
- a device for synchronizing cached data includes:
- a receiving module configured to receive a data synchronization request sent by a second device, where the data synchronization request carries the name of the fragment cached by the second device;
- a determining module configured to determine a time interval between the first device receiving the data synchronization request this time and the last time receiving the data synchronization request sent by the second device; according to the time interval, the fragments cached by the second device And the cache emptying duration corresponding to the second device, among the names of the fragments cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated and the fragment without update The name;
- a sending module configured to send to the second device the name of the fragment to be cleared, the name of the non-updated fragment, the name of the fragment to be updated, and each of the fragments to be updated Updated data.
- the data synchronization request also carries a time stamp of the last time the first device received the data synchronization request sent by the second device;
- the determination module is used to:
- the determination module is used to:
- the time interval and the fragment clearing duration corresponding to each fragment in the name of the fragment cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated and No updated shard name.
- the sending module is also used to:
- a cache clearing notification is sent to the second device to enable the second device to clear the data in the currently stored fragment .
- the determination module is used to:
- the fragment clear time corresponding to the fragment type to which each fragment belongs is determined.
- the determination module is used to:
- the name of the fragment is determined to be the name of the fragment to be cleared. If the time interval is less than the fragment clearing duration of the fragment, the target synchronization duration and the number of target synchronization records corresponding to the fragment are determined, and the target synchronization duration is used to indicate the longest synchronization of the data of the fragment At a time interval, the number of target synchronization records is used to indicate the maximum number of times the fragments are synchronized;
- the time interval is greater than or equal to the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronized records of the fragment on the first device is greater than or equal to the target corresponding to the fragment.
- the number of simultaneous records it is determined that the name of the shard is the name of the shard to be cleared;
- the time interval is less than the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is less than the number of target synchronization records corresponding to the fragment, It is determined that the name of the shard is the name of the shard without update;
- the time interval is greater than or equal to the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is less than the target synchronization records corresponding to the fragment Number, it is determined that the name of the fragment is the name of the fragment to be updated;
- the time interval is less than the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is greater than or equal to the target synchronization record corresponding to the fragment Number, it is determined that the name of the slice is the name of the slice to be updated.
- the determination module is used to:
- the target synchronization duration and the number of target synchronization records corresponding to the segment type to which the segment belongs are determined according to the correspondence between the stored segment types, the target synchronization duration, and the number of target synchronization records.
- the determination module is also used to:
- the receiving module is also used to:
- the data modification request carries the modified data, the identifier of the data before modification, and the name of the segment to which the modified data belongs;
- the device also includes:
- the synchronization module is used to modify the data corresponding to the identifier of the data before modification into the modified data in the database according to the data modification request, and record the modified data in the target cache area Correspondence between the name of the shard to which it belongs, the target shard type, the timestamp storing the modified data and the modified data, the target shard type is the shard to which the modified data belongs The shard type corresponding to the name of.
- the receiving module is used to:
- connection between the first device and the second device is a short connection
- the sending module is also used to:
- connection between the first device and the second device is a long connection
- the segment to which the target data belongs is the segment of the second device
- the data synchronization request carries the name of the segment whose number of requests exceeds the target number of the segments cached by the second device, where the requested number of times refers to the target duration closest to the current time point The number of times requested.
- a device for synchronizing cache data includes:
- a sending module configured to send a data synchronization request to the first device, where the data synchronization request carries the name of the fragment cached by the second device;
- a receiving module configured to receive the name of the fragment to be cleared, the name of the fragment without update, the name of the fragment to be updated, and the data to be updated in each fragment to be updated sent by the first device;
- the synchronization module is used for clearing each segment corresponding to the name of the segment to be cleared, and updating the data to be updated in each segment corresponding to the name of the segment to be updated deal with.
- the sending module is used to:
- a data synchronization request is sent to the first device every target period.
- the receiving module is also used to:
- the connection between the second device and the first device is a long connection
- receive a data synchronization notification sent by the first device where the data synchronization notification carries the modified target data, and the The name of the shard to which the modified target data belongs;
- the synchronization module is also used for:
- the target data is cached in the fragment to which the modified target data belongs in the second device, the target data is modified to the modified target data, if the second device If the target data is not cached in the segment to which the modified target data belongs, the modified target data is deleted.
- the sending module is also used to:
- the synchronization module is further configured to: when the server receives the corresponding data based on the data query request, store the corresponding data in the new segment;
- the sending module is also used to feed back data corresponding to the data query request;
- the sending module is further configured to, if the segment to which the data corresponding to the data query request belongs belongs to the target segment currently cached by the second device, based on the data corresponding to the data query request in the target segment Give feedback.
- a system for synchronizing cached data includes a first device and a second device, where:
- the first device is the device mentioned in the third aspect above.
- the second device is the device mentioned in the fourth aspect above.
- a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the method steps of the first aspect described above are implemented.
- a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the method steps of the second aspect described above are implemented.
- a device for synchronizing cached data including a processor and a memory, wherein the memory is used to store a computer program; the processor is used to execute a program stored on the memory to implement The method steps of the first aspect above.
- a device for synchronizing cache data including a processor and a memory, wherein the memory is used to store a computer program; and the processor is used to execute a program stored on the memory to implement The method steps of the second aspect above.
- the second device may send a data synchronization request to the first device and carry the name of the fragment cached by the second device.
- the first device may determine this time The time interval between receiving the data synchronization request and the last time to receive the data synchronization request sent by the second device, and then the first device may use the time interval, the name of the fragment cached by the second device, and the cache emptying duration corresponding to the second device , Among the names of the fragments cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated and the name of the fragment without update, and then notify the second device that the second device can proceed The shards to be emptied are cleared, and the shards to be updated are updated. In this way, targeted cache processing is performed for different fragments, so a method for synchronizing cached data is provided.
- FIG. 1 is a system diagram of synchronous cache data provided by an embodiment of the present disclosure
- FIG. 2 is a system diagram of synchronously caching data provided by an embodiment of the present disclosure
- FIG. 3 is a flowchart of a method for synchronizing cache data provided by an embodiment of the present disclosure
- FIG. 4 is a schematic structural diagram of a device for synchronizing cache data provided by an embodiment of the present disclosure
- FIG. 5 is a schematic structural diagram of a device for synchronizing cache data provided by an embodiment of the present disclosure
- FIG. 6 is a schematic structural diagram of a device for synchronizing cache data provided by an embodiment of the present disclosure
- FIG. 7 is a schematic structural diagram of a server provided by an embodiment of the present disclosure.
- An embodiment of the present disclosure provides a method for synchronizing cache data, and the execution subject of the method may be a first device and a second device.
- the method of synchronously caching data can be applied to multiple application scenarios.
- the types of the first device and the second device are different. The following gives two possible application scenarios:
- the first device is a third-party server
- the second device is a background server of an application program.
- the back-end server interacts with the third-party server to obtain authorization from the third-party server.
- the first device may be provided with a processor, a memory, a transceiver, and so on.
- the processor can be used to process the process of synchronizing cache data;
- the memory can be used to store the data required during the process of synchronizing cache data and the generated data;
- the transceiver can be used to receive and send data.
- the second device may be provided with a processor, a memory, a transceiver, and so on.
- the processor can be used to process the process of synchronizing cache data.
- the memory can be used to store data required during the process of synchronizing cache data and generated data.
- the transceiver can be used to receive and send data.
- the first device may be a background server of an application program, and the second device is a terminal installed with the application program.
- the terminal can interact with the background server, and some data is cached in the terminal.
- the user can obtain some data from the terminal's cache without having to obtain it from the background server.
- the first device may be a server
- the second device may be a terminal
- the terminal may be a mobile phone, a computer, or the like.
- the first device may be provided with a processor, a memory, a transceiver, and so on.
- the processor can be used to process the process of synchronizing cache data; the memory can be used to store the data required during the process of synchronizing cache data and the generated data; the transceiver can be used to receive and send data.
- the second device may be provided with a processor, a memory, a transceiver, and so on.
- the processor can be used to process the process of synchronizing cache data; the memory can be used to store the data required during the process of synchronizing cache data and the generated data; the transceiver can be used to receive and send data.
- the second device may also be provided with an input and output device such as a screen, the screen may be used to display an operation interface, etc., and the screen may be a touch screen.
- the scheme is described in detail by taking the case that the first device and the second device are both servers (that is, the first application scenario), and other situations are similar, and the embodiments of the present disclosure will not be described in detail.
- An embodiment of the present disclosure provides a method for synchronizing cache data. As shown in FIG. 3, the execution flow of the method may be as follows:
- Step 301 The second device sends a data synchronization request to the first device.
- the second device can obtain the names of all the shards stored by itself (divide a large piece of data into smaller and more manageable parts, each part is a shard) . Then the second device may generate a data synchronization request, and add the names of all fragments to the data synchronization request. The second device sends a data synchronization request to the first device.
- the name of the fragment carried in the data synchronization request is the name of the fragment whose number of requests exceeds the target number among the fragments cached by the second device, and the number of requested times refers to the target length that is closest to the current time point.
- the target duration can be preset, such as two days.
- the second device when the first device and the second device are in a short connection, the second device sends a data synchronization request to the first device, and the corresponding processing may be as follows:
- the second device When the connection between the second device and the first device is a short connection, every second target cycle, the second device sends a data synchronization request to the first device.
- the short connection is relative to the long connection, which means that in the data transmission process, a connection is established only when data needs to be sent, and the connection is disconnected after the data is sent. Only one service is sent per connection.
- the target period can be preset and stored in the second device, such as 0.5 hours.
- the second device can be operated to select a short connection or a long connection. If it chooses not to receive the push, it is determined that the connection between the second device and the first device is a short connection. If you choose to receive the push, it is determined that the connection between the second device and the first device is a long connection.
- the second device determines that the connection with the first device is a short connection, it establishes a short connection with the first device every target period and sends a data synchronization request to the first device.
- the above short connection may be a TCP (Transmission Control Protocol) short connection (also may be HTTP (Hyper Text Transfer Protocol) short connection, socket (socket) short connection, etc. ).
- TCP Transmission Control Protocol
- HTTP Hyper Text Transfer Protocol
- socket socket
- Step 302 The first device receives the data synchronization request sent by the second device.
- the data synchronization request is received, and the corresponding processing may be as follows:
- the first device When the connection between the first device and the second device is a short connection, the first device receives the data synchronization request sent by the second device every target period.
- step 303 the first device determines the time interval between the first device receiving the data synchronization request this time and the last time receiving the data synchronization request sent by the second device.
- the first device may determine the time point of receiving the data synchronization request this time. Then obtain the time point when the data synchronization request sent by the second device was last received (optionally, after each time the first device receives the data synchronization request sent by the second device, the first device will record that the data synchronization request is received this time Time point, delete the last recorded time point to save storage space, which is equivalent to recording only one time point), and then the first device takes the difference between this time point and the last time point to get the first The time interval between the device receiving the data synchronization request this time and the last time receiving the data synchronization request sent by the second device.
- the data synchronization request also carries the timestamp information of the last time the first device received the data synchronization request sent by the second device.
- the way to determine the time interval may be as follows:
- the first device takes the difference between the time stamp of the data synchronization request received by the first device and the time stamp carried in the data synchronization request to obtain the data synchronization request received by the first device this time and the data received by the second device last time. Time interval for synchronization request.
- each time the first device receives the data synchronization request sent by the second device it will record the timestamp of receiving the data synchronization request, and then send the timestamp to the second device, and the second device will record it.
- the time stamp may be added to the data synchronization request and sent to the first device.
- the first device After receiving the data synchronization request sent by the second device, the first device can parse the time stamp from it. And record the reception timestamp of the data synchronization request received by the first device this time, and then the first device subtracts the timestamp in the data synchronization request from the current timestamp to obtain the data synchronization request received by the first device and The interval at which the data synchronization request sent by the second device was last received.
- Step 304 The first device determines the name of the fragment to be cleared from the name of the fragment cached by the second device according to the time interval, the name of the fragment cached by the second device, and the length of the cache emptying corresponding to the second device.
- the buffer emptying duration can be preset and stored in the first device.
- the cache emptying duration is the shortest cache emptying duration, which means that the data storage duration will be deleted after the minimum cache emptying duration.
- the duration of emptying the cache corresponding to each device may be the same or different.
- the shard to be cleared is the shard to be deleted; the shard to be updated is the shard to be updated; the shard without update is the shard that does not need to be deleted and updated (that is, the cache duration has not reached the shortest cache Clear the duration, and there is no shard to update).
- the first device may obtain the cache emptying duration stored in advance corresponding to the second device. Then the first device uses the time interval, cache emptying duration, and the name of the fragment cached by the second device, and among the names of the fragments cached by the second device, determines the name of the fragment to be cleared, and the name of the fragment to be updated The name and the name of the shard without update.
- the first device may use the cache emptying duration to determine the name of the fragment to be cleared, the name of the fragment to be updated, and the name of the non-updated fragment in the name of the fragment cached by the second device, corresponding
- the processing of step 304 can be as follows:
- the first device determines the fragment emptying duration corresponding to each segment according to the name of each segment cached by the second device.
- the first device determines the name of the shard to be emptied, the name of the shard to be updated, and the non-updated shard in the name of the shard cached by the second device according to the time interval and the shard empty duration corresponding to each shard.
- the name of the shard is the name of the shard to be emptied, the name of the shard to be updated, and the non-updated shard in the name of the shard cached by the second device according to the time interval and the shard empty duration corresponding to each shard. The name of the shard.
- the first device may determine the time interval in step 303 corresponding to the length of the buffer empty duration corresponding to the second device. If the time interval is less than the cache emptying duration corresponding to the second device, the first device may use the name of each fragment cached by the second device to determine the fragment clearing duration corresponding to each fragment. Then the first device may use the time interval in step 303 and the fragment clear duration corresponding to each fragment to determine the name of the fragment to be cleared and the fragment to be updated in the name of the fragment cached by the second device And the name of the shard without update.
- the first device may send a cache emptying notification to the second device.
- the second device may delete all the data of each fragment stored by the second device. In this way, when the time interval between receiving two data synchronization requests is relatively large, most of the fragmented data stored in the second device needs to be updated. If the first device sends all the updated data to the second device, it will The second device receives more data in a short time, resulting in a poorer response of the second device. However, in this disclosure, the second device is notified to clear all data, which will not cause a poor response of the second device.
- the first device uses the name of the shard, and the method for obtaining the shard clearing duration may be as follows:
- the first device determines the segment type to which each segment cached by the second device belongs according to the correspondence between the stored segment name and the segment type.
- the first device determines the fragment clearing duration corresponding to the fragment type to which each fragment belongs according to the stored correspondence between the fragment type and the fragment clearing duration.
- the fragment clearing duration is used to indicate that when the storage duration of a fragment reaches the fragment clearing duration corresponding to the fragment, the fragment will be determined as the fragment to be cleared.
- the correspondence between the name of the slice and the type of slice is stored in the first device.
- Different slice types are used to indicate that the data of the slice is of different types, and that different types of data have different importance.
- the background server of a shopping application can be divided into permission data, product information, historical browsing data, and personal information.
- permission data and personal information can be one type, important data; commodity information can be another type, less important data; historical browsing data can be another type, non-important data, etc.
- the importance of the data of each shard under the same shard type is the same, so the shard emptying duration is also the same.
- the first device also stores the correspondence between the name of the fragment and the fragment type.
- the first device can obtain the correspondence between the name of the fragment and the fragment type stored in advance, and in the correspondence, determine the second device The type of shard to which each cached shard belongs.
- the correspondence between the fragment type and the fragment clear time is also stored.
- the first device may obtain the pre-stored correspondence between the fragment type and the fragment emptying duration, and determine the fragment clearing duration corresponding to the fragment type to which each fragment cached by the second device belongs to the corresponding relationship. In this way, the first device can determine the fragment clearing duration corresponding to each fragment cached by the second device.
- the first device may use the fragment clearing duration to determine the name of the fragment to be cleared, the name of the fragment to be updated, and the name of the fragment without update.
- the corresponding processing may be as follows:
- the first device determines that the name of the fragment is the name of the fragment to be cleared, if the time interval is less than the When the fragment clearing duration of the fragment, the first device determines the target synchronization duration and the number of target synchronization records corresponding to the fragment. If the time interval is greater than or equal to the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is greater than or equal to the number of target synchronization records corresponding to the fragment, then the first device determines The name of the shard is the name of the shard to be cleared.
- the first device determines the fragment's The name is the name of the shard without update. If the time interval is greater than or equal to the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is less than the number of target synchronization records corresponding to the fragment, the first device determines the fragment The name of the slice is the name of the slice to be updated.
- the first The device determines that the name of the fragment is the name of the fragment to be updated.
- the target synchronization duration is set corresponding to the shard.
- the target synchronization duration is the longest interval at which the data of the shard is synchronized.
- the synchronization time of the target should be synchronized in time in order to make the data in the fragment relatively new.
- the number of target synchronization records is set corresponding to the shard.
- the number of target synchronous records is the maximum number of times the data of the shard is synchronized. The number of times the slice is synchronized exceeds the number of target synchronization records. In order to make the data in the slice relatively new, it is necessary to synchronize in time.
- the first device may determine the size of the time interval determined in step 303 and the fragment clearing duration of the fragment. If the time interval is greater than or equal to the fragment clearing duration of the fragment, the first device may determine that the fragment is a fragment to be cleared, and the name of the fragment is the name of the fragment to be cleared. If the time interval determined in step 303 is less than the fragment clearing duration of the fragment, the first device may determine the target synchronization duration and the number of target synchronization records corresponding to the fragment, and then determine the size of the above time interval and the target synchronization duration.
- the first device obtains the number of times that the fragment on the first device has been modified during this time interval (that is, the time interval determined in step 303), that is, the number of synchronized records, and then compares the number of synchronized records with the number of target synchronized records the size of.
- the first device may determine the fragment's
- the name is the name of the shard to be cleared. For example, the number of target synchronization records is 100, the number of synchronization records of the shard is 105, the above time interval is 5 minutes, and the target synchronization duration is 2 minutes. The first device may determine that the name of the shard is the shard to be cleared name.
- the first device may determine that the name of the fragment is a non-updated fragment The name. In this way, the above time interval is less than the target synchronization duration corresponding to the fragment, and the number of synchronization records for the fragment on the first device does not exceed the number of target synchronization records, indicating that the fragment does not need to be synchronized, and the first device can notify the second The device does not need to be synchronized.
- the first device may determine that the name of the fragment is to be updated The name of the shard. In this way, the above time interval is greater than or equal to the target synchronization duration corresponding to the shard.
- the number of synchronization records for the shard on the first device does not exceed the number of target synchronization records for the shard, since the above time interval has reached the target synchronization Duration, in order to keep the data in the shard on the second device up to date, it should be synchronized in time.
- the first device may determine the name of the fragment to be updated The name of the shard. In this way, although the above time interval is less than the target synchronization duration corresponding to the fragment, the number of synchronization records for the fragment on the first device exceeds the number of target synchronization records corresponding to the fragment. The data in is the latest, so it should be synchronized in time.
- an embodiment of the present disclosure also provides a method for determining the target synchronization duration and the number of target synchronization records of the fragments.
- the processing may be as follows:
- the first device determines the segment type to which the segment belongs according to the correspondence between the stored segment name and the segment type.
- the first device determines the target synchronization duration and the number of target synchronization records corresponding to the segment type to which the segment belongs according to the correspondence between the stored segment types, the target synchronization duration, and the number of target synchronization records.
- the first device pre-stores the correspondence between the name of the shard and the type of shard.
- the first device may use the name of the shard. In the correspondence, determine the shard corresponding to the name of the shard Type to get the shard type to which the shard belongs. Then, the first device may obtain the correspondence relationship between the pre-stored fragment type and the target synchronization duration and the number of target synchronization records, and then determine the target synchronization duration and the number of target synchronization records corresponding to the fragment type to which the fragment belongs from the corresponding relationship . Under the same fragment type, the target synchronization duration of all fragments is the same, and the number of target synchronization records of all fragments is the same. In this way, since two correspondences are stored, it is easier to find the target synchronization duration and the number of target synchronization records of the fragments.
- the first device may also obtain the number of synchronized records of fragments from the cache area of the first device, and the corresponding processing may be as follows:
- the first device obtains the number of fragmented synchronized records from the target cache area.
- the target cache area is any block cache area set in the first device.
- a target cache area is set in the first device, and each time the second device requests to modify the data on the first device, the first device can update the modified data to the database, and can The modified data is stored in the cache area, and the modified data will correspond to the timestamp of the modified data stored by the first device, and indicate the segment to which the modified data belongs and the segment to which the segment belongs Shard type.
- the first storage method is to store timestamps. The later the timestamp of the modified data, the more it will be at the forefront, as shown in Table 1.
- Modified data Timestamp Shard type The name of the shard Data1 T1 A 121 Data2 T2 B 122 Data3 T3 A 126 ... ... ... ...
- T1 is later than T2
- T2 is later than T3.
- the second storage method is to store in the manner of shard type.
- the same shard type is stored together, and the later the time stamp of the modified data of the same shard type, the higher it will be in front, as shown in Table 2. .
- Modified data Timestamp Shard type The name of the shard Data1 T1 A 121 Data3 T3 A 126 Data2 T2 B 122 ... ... ... ... ...
- T1 is later than T3.
- the first device can use the time interval in step 303 to determine the timestamp range corresponding to the time interval (such as the time interval is July 1, 2019 Until July 15, 2019, the timestamp range is from July 1, 2019 to July 15, 2019), and then determine the number of synchronized records of the shard within this timestamp range. In this way, since the number of synchronized records can be obtained from the target cache area without accessing the database, processing time can be saved.
- the embodiments of the present disclosure also provide a process of modifying data by other devices (other devices refer to devices that have a service association with the first device), and the corresponding processing may be as follows:
- the first device receives a data modification request, where the data modification request carries the modified data, the identifier of the pre-modified data, and the name of the segment to which the modified data belongs.
- the first device modifies the data corresponding to the identifier of the modified data into the modified data in the database, and records the name of the shard to which the modified data belongs and the target shard in the target cache area The corresponding relationship between the type, the timestamp storing the modified data, and the modified data.
- the target shard type is the shard type corresponding to the name of the shard to which the modified data belongs.
- a device when a device (which may include a second device) that has a service association with the first device wants to modify data, it may send a data modification request to the first device.
- the data modification request carries the modified data, the identifier of the data before modification and the name of the segment to which the modified data belongs.
- the first device After receiving the data modification request, the first device can parse out the modified data and the name of the slice to which the modified data belongs. The first device can then use the name of the shard in the database to find the corresponding shard. Then the first device uses the identification of the data before modification to find the data before modification, and replaces the data before modification with the modified data. And the first device may record the name of the segment to which the modified data belongs, the target segment type, the time stamp of the modified data, and the correspondence between the modified data in the target cache area.
- Step 305 The first device sends the name of the fragment to be cleared, the name of the fragment without update, the name of the fragment to be updated, and the data to be updated in each fragment to be updated to the second device.
- the first device may send the name of the fragment to be cleared and the name of the fragment to be updated to the second device And the name of the shard without update.
- Step 306 The second device receives the name of the fragment to be cleared, the name of the fragment without update, the name of the fragment to be updated, and the data to be updated in each fragment to be updated sent by the first device.
- Step 307 The second device performs an emptying process for each segment corresponding to the name of the segment to be cleared, and updates the data to be updated in each segment corresponding to the name of the segment to be updated.
- the second device receives the name of the fragment to be cleared, the name of the fragment to be updated without the name of the fragment sent from the first device, and the data to be updated in each fragment to be updated . You can use the name of the shard to be emptied to determine the shard to be emptied. Then the second device deletes the data in the fragment to be emptied, and can use the name of the fragment to be updated to determine the fragment to be updated.
- the fragment to be updated determine the data to be updated belonging to the second device, update the data to be updated belonging to the second device, and delete the other data to be updated that does not belong to the second device (this is Since the data to be updated sent by the first device may not all be data of the second device, the data to be updated of the second device must be determined in it first).
- the second device will not perform update processing on the fragments without update.
- an embodiment of the present disclosure also provides a process of processing a long connection between the second device and the first device:
- the connection between the first device and the second device is a long connection by the first device
- the target data modification is detected, if the segment to which the target data belongs is the segment of the second device, it is sent to the second device Data synchronization notification.
- the data synchronization notification carries the modified target data and the name of the shard to which the modified target data belongs.
- the second device receives the data synchronization notification sent by the first device. If the target data is cached in the segment to which the modified target data belongs in the second device, the target data is modified to the modified target data, if the segment to which the modified target data belongs in the second device is not cached Target data, then delete the modified target data.
- a long connection is a relatively short connection. After completing a data transmission, the long connection is not closed. Whenever there is data to update, it is pushed.
- the second device may send the name of the cached fragment to the first device, the first device After receiving, it can be stored, so that the first device stores the name of the fragment cached on the second device.
- the first device determines whether the fragment to which the data (which may be referred to as target data) belongs is a fragment cached by the second device. If the fragment is cached by the second device, the first device may send a data synchronization notification to the second device, and carry the modified target data and the name of the fragment to which the modified target data belongs.
- the second device After receiving the data synchronization notification sent by the first device, the second device can parse out the modified target data and the name of the segment to which the modified target data belongs. Then the second device can use the name of the slice to find the corresponding slice, and determine whether there is target data in the slice. If the target data exists, the target data is modified to the modified target data. If the target data does not exist, the modified target data can be deleted. This is because a piece of data may store data of other devices. If the data of other devices is modified, the second device does not need the modified data, so it can be deleted.
- the above-mentioned long connection may be a TCP long connection (also may be an HTTP long connection or a socket long connection).
- the second device achieves strong consistency with the first device, so each time the data of the first device is modified, the second device is notified.
- the second device achieves the final consistency, that is, the data synchronization allows a certain delay, but the final data is the same as the data on the first device.
- the second device is The data of the first device can be synchronized periodically.
- processing for receiving a data query request by the second device is also provided, and the corresponding processing may be as follows:
- the second device When the second device receives the data query request, if the segment to which the data corresponding to the data query request belongs does not belong to the segment currently cached by the second device, it creates a new segment and sends the first device to create the new segment Message and data query request; when receiving the corresponding data based on the data query request from the server, the second device stores the corresponding data in the new segment and feeds back the data corresponding to the data query request. If the segment to which the data corresponding to the data query request belongs is the target segment currently cached by the second device, feedback is performed based on the data corresponding to the data query request in the target segment.
- the user when the user wants to query data, the user can send a data query request to the second device through the terminal installed in the application, and the second device can determine whether the data to be requested by the data query request belongs to the currently cached shard. If it does not belong to the currently cached shard, you can create a new shard, and then send a message to create a new shard and a data query request to the first device. After receiving the message to create a new segment and the data query request, the first device may return data of any segment to the second device. After receiving the second device, the second device may write the received data into the new shard. At the same time, the first device queries the corresponding data from the database based on the data query request and returns it to the second device.
- the second device may update the data corresponding to the received data query request to the new segment.
- the data corresponding to the data query request is fed back to the terminal installed by the application.
- the feedback data is first written to the fragment of the second device without waiting.
- the first device may query the corresponding data from the database based on the data query request and return it to the second device.
- the second device may update the data corresponding to the received data query request to the new segment.
- the data corresponding to the data query request is fed back to the terminal installed by the application.
- the data corresponding to the data query request belongs to the target fragment currently cached by the second device, the data corresponding to the data query request is obtained from the target fragment and fed back to the terminal installed by the application.
- the second device can record the last time point of use of each piece of data from the current The length of time can be deleted when the preset deletion time is exceeded. In addition, if a shard does not include any data, the shard is deleted.
- the difference between the second device and the related art second device may be: a second client is added to the second device of this application, and the client has a cache area, which is different from the first device Synchronization, so in any system that synchronizes data, as long as a client provides a cache area.
- the data in different fragments is processed differently based on certain rules, which can make the processing reasonable, and there are multiple fragments stored on the second device.
- the feedback can be performed without obtaining from the first device, which not only improves the response efficiency, but also reduces the pressure on the first device.
- the second device may send a data synchronization request to the first device and carry the name of the fragment cached by the second device.
- the first device may determine this time The time interval between receiving the data synchronization request and the last time to receive the data synchronization request sent by the second device, and then the first device may use the time interval, the name of the fragment cached by the second device, and the cache emptying duration corresponding to the second device , Among the names of the fragments cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated and the name of the fragment without update, and then notify the second device that the second device can proceed The shards to be emptied are cleared, and the shards to be updated are updated. In this way, targeted cache processing is performed for different fragments, so a method for synchronizing cached data is provided.
- an embodiment of the present disclosure also provides a device for synchronizing cache data (ie, the first device). As shown in FIG. 4, the device includes:
- the receiving module 410 is configured to receive a data synchronization request sent by a second device, where the data synchronization request carries the name of the fragment cached by the second device;
- the determining module 420 is used to determine the time interval between the first device receiving the data synchronization request this time and the last time receiving the data synchronization request sent by the second device; according to the time interval, the points cached by the second device The name of the slice and the duration of the cache emptying corresponding to the second device, and among the names of the fragments cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated, and the non-updated fragment The name of the film;
- the sending module 430 is configured to send the name of the fragment to be cleared, the name of the non-updated fragment, the name of the fragment to be updated and each fragment to be updated to the second device The data to be updated.
- the data synchronization request also carries a time stamp of the last time the first device received the data synchronization request sent by the second device;
- the determining module 420 is used to:
- the determination module 420 is used to:
- the time interval and the fragment clearing duration corresponding to each fragment in the name of the fragment cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated and No updated shard name.
- the sending module 430 is also used to:
- a cache clearing notification is sent to the second device to enable the second device to clear the data in the currently stored fragment .
- the determination module 420 is used to:
- the fragment clear time corresponding to the fragment type to which each fragment belongs is determined.
- the determination module 420 is used to:
- the name of the fragment is determined to be the name of the fragment to be cleared. If the time interval is less than the fragment clearing duration of the fragment, the target synchronization duration and the number of target synchronization records corresponding to the fragment are determined, and the target synchronization duration is used to indicate the longest synchronization of the data of the fragment At a time interval, the number of target synchronization records is used to indicate the maximum number of times the fragments are synchronized;
- the time interval is greater than or equal to the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronized records of the fragment on the first device is greater than or equal to the target corresponding to the fragment.
- the number of simultaneous records it is determined that the name of the shard is the name of the shard to be cleared;
- the time interval is less than the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is less than the number of target synchronization records corresponding to the fragment, It is determined that the name of the shard is the name of the shard without update;
- the time interval is greater than or equal to the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is less than the target synchronization records corresponding to the fragment Number, it is determined that the name of the fragment is the name of the fragment to be updated;
- the time interval is less than the target synchronization duration corresponding to the fragment, and within the time interval, the number of synchronization records of the fragment on the first device is greater than or equal to the target synchronization record corresponding to the fragment Number, it is determined that the name of the slice is the name of the slice to be updated.
- the determination module 420 is used to:
- the target synchronization duration and the number of target synchronization records corresponding to the segment type to which the segment belongs are determined according to the correspondence between the stored segment types, the target synchronization duration, and the number of target synchronization records.
- the determination module 420 is also used to:
- the receiving module 410 is also used to:
- the data modification request carries the modified data, the identifier of the data before modification, and the name of the segment to which the modified data belongs;
- the device further includes:
- the synchronization module 440 is configured to modify the data corresponding to the identifier of the data before modification into modified data in the database according to the data modification request, and record the modified data in the target cache area
- the name of the segment to which the data belongs, the target segment type, the correspondence between the time stamp storing the modified data and the modified data, and the target segment type is the segment to which the modified data belongs
- the receiving module 410 is configured to:
- connection between the first device and the second device is a short connection
- the sending module 430 is also used to:
- connection between the first device and the second device is a long connection
- the segment to which the target data belongs is the segment of the second device
- the data synchronization request carries the name of the segment whose number of requests exceeds the target number of the segments cached by the second device, where the requested number of times refers to the target duration closest to the current time point The number of times requested.
- the second device may send a data synchronization request to the first device and carry the name of the fragment cached by the second device.
- the first device may determine this time The time interval between receiving the data synchronization request and the last time to receive the data synchronization request sent by the second device, and then the first device may use the time interval, the name of the fragment cached by the second device, and the cache emptying duration corresponding to the second device , Among the names of the fragments cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated and the name of the fragment without update, and then notify the second device that the second device can proceed The shards to be emptied are cleared, and the shards to be updated are updated. In this way, targeted cache processing is performed for different fragments, so a method for synchronizing cached data is provided.
- the device for synchronizing cached data provided in the above embodiments only uses the division of the above functional modules as an example for synchronizing cached data.
- the above functions can be allocated by different functional modules according to needs Completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
- the device for synchronizing cached data provided in the above embodiments and the method embodiment for synchronizing cached data belong to the same concept. For the specific implementation process, see the method embodiment, and details are not described here.
- an embodiment of the present disclosure also provides a device for synchronizing cache data (ie, a second device). As shown in FIG. 6, the device includes:
- the sending module 610 is configured to send a data synchronization request to the first device, where the data synchronization request carries the name of the fragment cached by the second device;
- the receiving module 620 is configured to receive the name of the fragment to be cleared, the name of the fragment without update and the name of the fragment to be updated sent by the first device, and the data to be updated in each fragment to be updated;
- the synchronization module 630 is configured to perform an emptying process on each segment corresponding to the name of the segment to be cleared, and perform data to be updated in each segment corresponding to the name of the segment to be updated Update processing.
- the sending module 610 is configured to:
- a data synchronization request is sent to the first device every target period.
- the receiving module 620 is also used to:
- the connection between the second device and the first device is a long connection
- receive a data synchronization notification sent by the first device where the data synchronization notification carries the modified target data, and the The name of the shard to which the modified target data belongs;
- the synchronization module 630 is also used to:
- the target data is cached in the fragment to which the modified target data belongs in the second device, the target data is modified to the modified target data, if the second device If the target data is not cached in the segment to which the modified target data belongs, the modified target data is deleted.
- the sending module 610 is also used to:
- the synchronization module 630 is further configured to: when receiving the corresponding data obtained by the server based on the data query request, store the corresponding data in the new shard;
- the sending module 610 is also used to feed back data corresponding to the data query request;
- the sending module 610 is further configured to, if the fragment to which the data corresponding to the data query request belongs belongs to the target fragment currently cached by the second device, based on the data corresponding to the data query request in the target fragment Data feedback.
- the second device may send a data synchronization request to the first device and carry the name of the fragment cached by the second device.
- the first device may determine this time The time interval between receiving the data synchronization request and the last time to receive the data synchronization request sent by the second device, and then the first device may use the time interval, the name of the fragment cached by the second device, and the cache emptying duration corresponding to the second device , Among the names of the fragments cached by the second device, determine the name of the fragment to be cleared, the name of the fragment to be updated and the name of the fragment without update, and then notify the second device that the second device can proceed The shards to be emptied are cleared, and the shards to be updated are updated. In this way, targeted cache processing is performed for different fragments, so a method for synchronizing cached data is provided.
- the device for synchronizing cached data provided in the above embodiments only uses the division of the above functional modules as an example for synchronizing cached data.
- the above functions can be allocated by different functional modules according to needs Completed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
- the device for synchronizing cached data provided in the above embodiments and the method embodiment for synchronizing cached data belong to the same concept. For the specific implementation process, see the method embodiment, and details are not described here.
- the device 700 may have a relatively large difference due to different configurations or performances, and may include one or more processors (central processing units, CPU ) 701 and one or more memories 702, wherein at least one instruction is stored in the memory 702, and the at least one instruction is loaded and executed by the processor 701 to implement the steps of the above method for synchronizing cached data.
- processors central processing units, CPU
- memories 702 wherein at least one instruction is stored in the memory 702, and the at least one instruction is loaded and executed by the processor 701 to implement the steps of the above method for synchronizing cached data.
- the present application also provides a computer-readable storage medium in which a computer program is stored.
- the computer program is executed by a processor, the method steps of synchronizing cached data described above are implemented.
- the present application also provides a device for synchronizing cache data, including a processor and a memory, wherein the memory is used to store a computer program; the processor is used to execute a program stored on the memory to implement the above Method steps for synchronizing cached data.
- the program may be stored in a computer-readable storage medium.
- the mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
修改后的数据 | 时间戳 | 分片类型 | 分片的名称 |
Data1 | T1 | A | 121 |
Data2 | T2 | B | 122 |
Data3 | T3 | A | 126 |
… | … | … | … |
修改后的数据 | 时间戳 | 分片类型 | 分片的名称 |
Data1 | T1 | A | 121 |
Data3 | T3 | A | 126 |
Data2 | T2 | B | 122 |
… | … | … | … |
Claims (23)
- 一种同步缓存数据的方法,其特征在于,所述方法包括:第一设备接收第二设备发送的数据同步请求,其中,所述数据同步请求中携带有所述第二设备缓存的分片的名称;所述第一设备确定所述第一设备本次接收所述数据同步请求和上次接收所述第二设备发送的数据同步请求的时间间隔;所述第一设备根据所述时间间隔、所述第二设备缓存的分片的名称和所述第二设备对应的缓存清空时长,在所述第二设备缓存的分片的名称中,确定待清空的分片的名称、待更新的分片的名称和无更新的分片的名称;所述第一设备向所述第二设备发送所述待清空的分片的名称、所述无更新的分片的名称和所述待更新的分片的名称以及各待更新的分片中待更新的数据。
- 根据权利要求1所述的方法,其特征在于,所述数据同步请求中还携带有所述第一设备上次接收到所述第二设备发送的数据同步请求的时间戳;所述第一设备确定所述第一设备本次接收所述数据同步请求和上次接收所述第二设备发送的数据同步请求的时间间隔,包括:所述第一设备将所述第一设备本次接收到所述数据同步请求的时间戳与所述数据同步请求中携带的时间戳取差值,获得所述第一设备本次接收所述数据同步请求和上次接收所述第二设备发送的数据同步请求的时间间隔。
- 根据权利要求1所述的方法,其特征在于,所述第一设备根据所述时间间隔、所述第二设备缓存的分片的名称和所述第二设备对应的缓存清空时长,在所述第二设备缓存的分片的名称中,确定待清空的分片的名称、待更新的分片的名称和无更新的分片的名称,包括:如果所述时间间隔小于所述第二设备对应的缓存清空时长,则所述第一设备根据所述第二设备缓存的各分片的名称,确定所述各分片分别对应的分片清空时长;所述第一设备根据所述时间间隔和所述各分片分别对应的分片清空时长,在所述第二设备缓存的分片的名称中,确定待清空的分片的名称、待更新的分片的名称和无更新的分片的名称。
- 根据权利要求3所述的方法,其特征在于,所述方法还包括:如果所述时间间隔大于或等于所述第二设备对应的缓存清空时长,则所述第一设备向所述第二设备发送缓存清空通知,以使所述第二设备对当前存储的分片中的数据进行清空处理。
- 根据权利要求3所述的方法,其特征在于,所述第一设备根据所述第二设备缓存的各分片的名称,确定所述各分片分别对应的分片清空时长,包括:所述第一设备根据所述第二设备缓存的各分片的名称、存储的分片的名称与分片类型的对应关系,确定所述第二设备缓存的各分片所属的分片类型;所述第一设备根据存储的分片类型与分片清空时长的对应关系,确定所述各分片所属的分片类型所对应的分片清空时长。
- 根据权利要求3至5任一所述的方法,其特征在于,所述第一设备根据所述时间间隔和所述各分片分别对应的分片清空时长,在所述第二设备缓存的分片中,确定待清空的分片的名称、待更新的分片的名称和无更新的分片的名称,包括:对于所述第二设备缓存的每个分片,如果所述时间间隔大于或等于所述分片的分片清空时长,则所述第一设备确定所述分片的名称为待清空的分片的名称,如果所述时间间隔小于所述分片的分片清空时长,则所述第一设备确定所述分片对应的目标同步时长和目标同步记录数目,所述目标同步时长用于指示所述分片的数据被同步的最长时间间隔,所述目标同步记录数目用于指示所述分片被同步的最多次数;如果所述时间间隔大于或等于所述分片对应的目标同步时长,且在所述时间间隔内,所述第一设备上所述分片的同步记录数目大于或等于所述分片对应的目标同步记录数目,则所述第一设备确定所述分片的名称为待清空的分片的名称;如果所述时间间隔小于所述分片对应的目标同步时长,且在所述时间间隔内,所述第一设备上所述分片的同步记录数目小于所述分片对应的目标同步记录数目,则所述第一设备确定所述分片的名称为无更新的分片的名称;如果所述时间间隔大于或等于所述分片对应的目标同步时长,且在所述时间间隔内,所述第一设备上所述分片的同步记录数目小于所述分片对应的目标同步记录数目,则所述第一设备确定所述分片的名称为待更新的分片的名称;如果所述时间间隔小于所述分片对应的目标同步时长,且在所述时间间隔 内,所述第一设备上所述分片的同步记录数目大于或等于所述分片对应的目标同步记录数目,则所述第一设备确定所述分片的名称为待更新的分片的名称。
- 根据权利要求6所述的方法,其特征在于,所述第一设备确定所述分片对应的目标同步时长和目标同步记录数目,包括:所述第一设备根据存储的分片的名称与分片类型的对应关系,确定所述分片所属的分片类型;所述第一设备根据存储的分片类型与目标同步时长、目标同步记录数目的对应关系,确定所述分片所属的分片类型对应的目标同步时长和目标同步记录数目。
- 根据权利要求6所述的方法,其特征在于,所述方法还包括:所述第一设备从目标缓存区域获取所述分片的同步记录数目。
- 根据权利要求8所述的方法,其特征在于,所述方法还包括:所述第一设备接收数据修改请求,其中,所述数据修改请求中携带有修改后的数据、修改前的数据的标识和所述修改后的数据所属的分片的名称;所述第一设备根据所述数据修改请求,在数据库中,将所述修改前的数据的标识对应的数据修改为所述修改后的数据,并且在所述目标缓存区域,记录所述修改后的数据所属的分片的名称、目标分片类型、存储所述修改后的数据的时间戳和所述修改后的数据的对应关系,所述目标分片类型为所述修改后的数据所属的分片的名称对应的分片类型。
- 根据权利要求1所述的方法,其特征在于,所述第一设备接收第二设备发送的数据同步请求,包括:在所述第一设备与所述第二设备之间的连接为短连接时,所述第一设备接收所述第二设备每隔目标周期发送的数据同步请求。
- 根据权利要求10所述的方法,其特征在于,所述方法还包括:所述第一设备在所述第一设备与所述第二设备之间的连接为长连接时,每当检测到目标数据修改时,如果所述目标数据所属的分片为所述第二设备的分片,则向所述第二设备发送数据同步通知,其中,所述数据同步通知中携带有修改后的目标数据、以及所述修改后的目标数据所属的分片的名称。
- 根据权利要求1所述的方法,其特征在于,所述数据同步请求中携带有所述第二设备缓存的分片中被请求次数超过目标次数的分片的名称,其中, 所述被请求次数指距离当前时间点最近的目标时长内被请求次数。
- 一种同步缓存数据的方法,其特征在于,所述方法包括:第二设备向第一设备发送数据同步请求,其中,所述数据同步请求中携带有所述第二设备缓存的分片的名称;所述第二设备接收所述第一设备发送的待清空的分片的名称、无更新的分片的名称和待更新的分片的名称以及各待更新的分片中待更新的数据;所述第二设备对所述待清空的分片的名称对应的每个分片进行清空处理,并且对所述待更新的分片的名称对应的每个分片中的待更新的数据进行更新处理。
- 根据权利要求13所述的方法,其特征在于,所述第二设备向第一设备发送数据同步请求,包括:在第二设备与第一设备的连接为短连接时,每隔目标周期,所述第二设备向第一设备发送数据同步请求。
- 根据权利要求14所述的方法,其特征在于,所述方法还包括:在所述第二设备与所述第一设备的连接为长连接时,所述第二设备接收所述第一设备发送的数据同步通知,其中,所述数据同步通知中携带有修改后的目标数据、以及所述修改后的目标数据所属的分片的名称;如果所述第二设备中所述修改后的目标数据所属的分片中缓存有所述目标数据,则将所述目标数据修改为所述修改后的目标数据,如果所述第二设备中所述修改后的目标数据所属的分片中未缓存有所述目标数据,则删除所述修改后的目标数据。
- 根据权利要求13至15任一所述的方法,其特征在于,所述方法还包括:所述第二设备当接收到数据查询请求时,如果所述数据查询请求对应的数据不属于所述第二设备当前缓存的分片,则创建新分片,并向所述第一设备发送创建新分片的消息;所述第二设备当接收到所述服务器基于所述数据查询请求获取到相应数据时,将所述相应数据存储在所述新分片中,并反馈所述数据查询请求对应的数据;如果所述数据查询请求对应的数据所属的分片为所述第二设备当前缓存的 目标分片,则基于所述目标分片中所述数据查询请求对应的数据进行反馈。
- 一种同步缓存数据的设备,其特征在于,所述设备包括:接收模块,用于接收第二设备发送的数据同步请求,其中,所述数据同步请求中携带有所述第二设备缓存的分片的名称;确定模块,用于确定第一设备本次接收所述数据同步请求和上次接收所述第二设备发送的数据同步请求的时间间隔;根据所述时间间隔、所述第二设备缓存的分片的名称和所述第二设备对应的缓存清空时长,在所述第二设备缓存的分片的名称中,确定待清空的分片的名称、待更新的分片的名称和无更新的分片的名称;发送模块,用于向所述第二设备发送所述待清空的分片的名称、所述无更新的分片的名称和所述待更新的分片的名称以及各待更新的分片中待更新的数据。
- 一种同步缓存数据的设备,其特征在于,所述设备包括:发送模块,用于向第一设备发送数据同步请求,其中,所述数据同步请求中携带有第二设备缓存的分片的名称;接收模块,用于接收所述第一设备发送的待清空的分片的名称、无更新的分片的名称和待更新的分片的名称以及各待更新的分片中待更新的数据;同步模块,用于对所述待清空的分片的名称对应的每个分片进行清空处理,并且对所述待更新的分片的名称对应的每个分片中的待更新的数据进行更新处理。
- 一种同步缓存数据的系统,其特征在于,所述系统包括第一设备和第二设备,其中:所述第一设备,如所述权利要求17所述的同步缓存数据的设备;所述第二设备,如所述权利要求18所述的同步缓存数据的设备。
- 一种计算机可读存储介质,其特征在于,所述存储介质内存储有计算机程序,所述计算机程序被处理器执行时实现权利要求1-12任一所述的方法步骤。
- 一种计算机可读存储介质,其特征在于,所述存储介质内存储有计算机程序,所述计算机程序被处理器执行时实现权利要求13至16任一所述的方法步骤。
- 一种同步缓存数据的设备,其特征在于,包括处理器和存储器,其中,所述存储器,用于存放计算机程序;所述处理器,用于执行所述存储器上所存放的程序,实现权利要求1-12任一所述的方法步骤。
- 一种同步缓存数据的设备,其特征在于,包括处理器和存储器,其中,所述存储器,用于存放计算机程序;所述处理器,用于执行所述存储器上所存放的程序,实现权利要求13至16任一所述的方法步骤。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811427828.XA CN111221469B (zh) | 2018-11-27 | 2018-11-27 | 同步缓存数据的方法、装置和系统 |
CN201811427828.X | 2018-11-27 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020108544A1 true WO2020108544A1 (zh) | 2020-06-04 |
Family
ID=70827903
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/121356 WO2020108544A1 (zh) | 2018-11-27 | 2019-11-27 | 同步缓存数据的方法、装置和系统 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111221469B (zh) |
WO (1) | WO2020108544A1 (zh) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113765886A (zh) * | 2021-08-04 | 2021-12-07 | 深圳Tcl新技术有限公司 | 多媒体播放方法、装置、存储介质及电子设备 |
CN115250293A (zh) * | 2022-06-30 | 2022-10-28 | 深圳水趣智能零售系统有限公司 | 数据上传方法、装置及计算机可读存储介质 |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102253362B1 (ko) | 2020-09-22 | 2021-05-20 | 쿠팡 주식회사 | 전자 장치 및 이를 이용한 정보 제공 방법 |
CN112445864B (zh) * | 2020-12-11 | 2024-03-26 | 北京北信源软件股份有限公司 | 组织人员信息同步方法、装置、电子设备及存储介质 |
CN113364830B (zh) * | 2021-04-26 | 2023-07-07 | 福建天泉教育科技有限公司 | 一种长链接的缓存优化方法及系统 |
CN114691269B (zh) * | 2022-03-29 | 2024-05-14 | 广东万和新电气股份有限公司 | 页面更新方法、装置、系统和存储介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120066180A1 (en) * | 2010-09-10 | 2012-03-15 | Microsoft Corporation | File-backed in-memory structured storage for service synchronization |
CN106469158A (zh) * | 2015-08-17 | 2017-03-01 | 杭州海康威视系统技术有限公司 | 数据同步方法和装置 |
CN106612308A (zh) * | 2015-10-22 | 2017-05-03 | 阿里巴巴集团控股有限公司 | 数据传输方法及装置 |
CN107018170A (zh) * | 2016-10-27 | 2017-08-04 | 阿里巴巴集团控股有限公司 | 一种数据同步方法和设备 |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8533410B1 (en) * | 2007-03-29 | 2013-09-10 | Netapp, Inc. | Maintaining snapshot and active file system metadata in an on-disk structure of a file system |
CN106951179B (zh) * | 2016-01-07 | 2020-03-27 | 杭州海康威视数字技术股份有限公司 | 一种数据迁移方法及装置 |
-
2018
- 2018-11-27 CN CN201811427828.XA patent/CN111221469B/zh active Active
-
2019
- 2019-11-27 WO PCT/CN2019/121356 patent/WO2020108544A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120066180A1 (en) * | 2010-09-10 | 2012-03-15 | Microsoft Corporation | File-backed in-memory structured storage for service synchronization |
CN106469158A (zh) * | 2015-08-17 | 2017-03-01 | 杭州海康威视系统技术有限公司 | 数据同步方法和装置 |
CN106612308A (zh) * | 2015-10-22 | 2017-05-03 | 阿里巴巴集团控股有限公司 | 数据传输方法及装置 |
CN107018170A (zh) * | 2016-10-27 | 2017-08-04 | 阿里巴巴集团控股有限公司 | 一种数据同步方法和设备 |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113765886A (zh) * | 2021-08-04 | 2021-12-07 | 深圳Tcl新技术有限公司 | 多媒体播放方法、装置、存储介质及电子设备 |
CN113765886B (zh) * | 2021-08-04 | 2023-07-25 | 深圳Tcl新技术有限公司 | 多媒体播放方法、装置、存储介质及电子设备 |
CN115250293A (zh) * | 2022-06-30 | 2022-10-28 | 深圳水趣智能零售系统有限公司 | 数据上传方法、装置及计算机可读存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN111221469A (zh) | 2020-06-02 |
CN111221469B (zh) | 2021-05-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020108544A1 (zh) | 同步缓存数据的方法、装置和系统 | |
EP3508985B1 (en) | Scalable synchronization with cache and index management | |
US10534776B2 (en) | Proximity grids for an in-memory data grid | |
WO2017097059A1 (zh) | 分布式数据库系统及其自适应方法 | |
JP2023518374A (ja) | データベースのトランザクション処理方法、データベースのトランザクション処理装置、サーバ、及びコンピュータプログラム | |
US11755580B2 (en) | Method and system for defining an object-agnostic offlinable synchronization model | |
JP2017084334A (ja) | 仮想マシンイメージファイルを抽出する方法および装置 | |
WO2023093245A1 (zh) | 基于分布式文件系统的元数据查询方法、设备和存储介质 | |
KR20130107280A (ko) | 디렉토리 리싱 | |
WO2021120995A1 (zh) | 数据库的数据同步方法、装置以及存储介质 | |
US20150142845A1 (en) | Smart database caching | |
CN107818111B (zh) | 一种缓存文件数据的方法、服务器及终端 | |
CN113094430B (zh) | 一种数据处理方法、装置、设备以及存储介质 | |
CN117616411A (zh) | 用于处理分布式在线事务处理(oltp)数据库中数据库事务的方法和系统 | |
US10579604B2 (en) | Database system, information processing device, method and medium | |
CN115309819A (zh) | 数据处理方法、装置、系统、计算机设备和存储介质 | |
CN113703917B (zh) | 一种多集群资源数据处理系统与方法、非暂态存储介质 | |
CN106603610B (zh) | 一种数据获取以及同步方法、装置 | |
JP7341330B2 (ja) | 日付切替データオフロード方法、装置、設備及び媒体 | |
US10572503B2 (en) | Database system, information processing device, method and medium | |
CN110334076B (zh) | 数据处理方法、系统、服务器和装置 | |
US9749426B2 (en) | Method, system, and apparatus for agent-based architecture for integrated mobile applications | |
CN112749172A (zh) | 一种缓存与数据库之间的数据同步方法及系统 | |
WO2013177375A1 (en) | Composite graph cache management | |
WO2022135471A1 (zh) | 多版本并发控制和日志清除方法、节点、设备和介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19889857 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19889857 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19889857 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 02/12/2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19889857 Country of ref document: EP Kind code of ref document: A1 |