CN110196680B - Data processing method, device and storage medium - Google Patents
Data processing method, device and storage medium Download PDFInfo
- Publication number
- CN110196680B CN110196680B CN201810257102.XA CN201810257102A CN110196680B CN 110196680 B CN110196680 B CN 110196680B CN 201810257102 A CN201810257102 A CN 201810257102A CN 110196680 B CN110196680 B CN 110196680B
- Authority
- CN
- China
- Prior art keywords
- service data
- application service
- data
- storage node
- application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data processing method, a data processing device and a storage medium. The method comprises the following steps: receiving a read data request from an application server; determining a second storage node set storing the application service data according to the user identification and the identification of the application service data carried in the data reading request; reading multiple application service data from each storage node in the second storage node set according to the identifier of the application service data; when the contents of the read multiple application service data are inconsistent, determining the confidence coefficient characteristics of the multiple application service data; determining a target application service data from the plurality of application service data according to the determined confidence characteristic; and sending the target application service data to an application server.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a data processing method, apparatus, and storage medium.
Background
With the development of information technology, global data is growing explosively, and the storage of the data puts higher demands on data storage technology. Data storage technology is mainly used for meeting two requirements of simplicity and reliability. In general, if the data storage system is relatively easy to implement, the reliability of reading data from the data storage system is difficult to guarantee; if the reliability of reading data from the data storage system is high, the data storage system is relatively complex to implement. Therefore, how to implement a data storage system with both simplicity and reliability is a problem to be solved by those skilled in the art.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing device and a storage medium, so that the complexity of reading data in a distributed storage system is reduced, and the accuracy of the data can be ensured.
The application example provides a data processing method, which comprises the following steps: receiving a data reading request from an application server, wherein the data reading request carries a user identifier and an identifier of application service data; determining a first storage node set storing the application service data according to the user identification and the identification of the application service data; reading multiple application service data from each storage node in the first storage node set according to the identifier of the application service data; when the read contents of the multiple application service data are inconsistent, determining the confidence coefficient characteristics of the multiple application service data; determining a target application service data from the multiple application service data according to the determined confidence characteristic; and sending the target application service data to the application server.
The example of the present application also provides a data processing apparatus, the apparatus includes: the receiving module is used for receiving a data reading request from the application server, wherein the data reading request carries a user identifier and an identifier of application service data; the reading module is used for determining a first storage node set in which the application service data is stored according to the user identification and the identification of the application service data; reading multiple application service data from each storage node in the first storage node set according to the identifier of the application service data; the determining module is used for determining the confidence coefficient characteristics of the multiple copies of application service data when the read contents of the multiple copies of application service data are inconsistent; determining a target application service data from the multiple application service data according to the determined confidence characteristic; and the sending module is used for sending the target application service data to the application server.
The present application also provides a storage medium storing computer-readable instructions that can cause at least one processor to perform the above-described method.
By adopting the technical scheme, the complexity of reading data in the distributed storage system can be reduced, and the accuracy of reading data is improved, so that the system performance is further improved.
Drawings
In order to more clearly illustrate the technical solutions in the examples of the present application, the drawings needed to be used in the description of the examples are briefly introduced below, and it is obvious that the drawings in the following description are only some examples of the present application, and it is obvious for a person skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic diagram of a system architecture to which the present application relates;
FIG. 2 is a flow chart of a method according to an example of the present application;
FIG. 3 is a distributed process to which the present application relates;
FIG. 4 is a message interaction diagram of an example of media data processing according to the present application;
FIG. 5 is a diagram of an apparatus according to an embodiment of the present application; and
fig. 6 is a diagram of an apparatus according to an example of the present application.
Detailed Description
The technical solutions in the examples of the present application will be clearly and completely described below with reference to the drawings in the examples of the present application, and it is obvious that the described examples are only a part of the examples of the present application, and not all examples. All other examples, which can be obtained by a person skilled in the art without making any inventive step based on the examples in this application, are within the scope of protection of this application.
In some examples, a centralized storage server is used to store all data, i.e., all data in the application server is written into the centralized storage server, and all the read data services of the application server are read from the centralized storage server. The method has very high simplicity, does not make any backup storage, but under the condition of network interruption, the stored data is completely inaccessible. At this time, the centralized storage server becomes a bottleneck of system performance, and cannot meet the requirement of large-scale storage application.
In some examples, a distributed storage system is used to store data in a distributed manner on multiple independent storage servers. The system adopts an expandable system structure, and utilizes a plurality of storage servers to share the storage load, namely, when the data is written for service, the same data can be backed up in multiple copies. The data reading service can read a copy nearby according to the regional distribution of the application server and the data storage server. The method not only improves the reliability, the availability and the access efficiency of the system, but also is easy to expand. However, due to the existence of failure, parallel storage and the like, inconsistency may exist among multiple copies of the same read data, and the inconsistency problem may be tolerated sometimes and is difficult to tolerate sometimes. In general, when writing data, consistency algorithms such as PAXOS, zokeeper, RAFT, and video summary reporting may be used to ensure consistency of the multiple backups, and this strong consistency processing is very complex to implement, the process of confirming consistency is also very long, and the processing delay of writing data is also relatively large.
Based on the above technical problem, the present application provides a data processing method, an apparatus and a storage medium. The above method can be applied to the system architecture shown in fig. 1. As shown in fig. 1, the system architecture includes: an application client 101, an application server 102 and a distributed storage system 103, which communicate via the internet 104, wherein the distributed storage system 103 comprises a plurality of node servers, such as a master node server, a node server 1, and a node server 2 … …, a node server n, as shown in fig. 1.
The user accesses the application server 102 using the application client 101, such as: browsing web pages or watching online videos, etc., application server 102 may be a web server providing various internet services, such as: a web portal server, a server providing an online video/audio playing service, a server of a social platform, and the like, wherein the application server 102 may provide various services to the application client 101. The user may generate various data to be stored in the distributed storage system 103 during various operations performed by the application client 101, such as: the user registration information is generated after the user completes the registration operation, the member information is generated after the user completes the member purchase operation, and the commodity order information is generated after the user completes the commodity purchase operation. The application service data such as the user registration information, the member information, and the commodity order information are stored in the distributed storage system 103 by the application server 102. Specifically, the application server 102 sends a data writing request to the distributed storage system 103, where the data writing request carries a user identifier and application service data to be stored associated with the user identifier; the master node server in the distributed storage system 103 stores the application service data to each storage node (i.e. node server) in a first storage node set respectively in response to the write data request, and associates the user identifier and the identifier of the application service data with the first storage node set.
When data is to be read, the application server 102 sends a data reading request to the master node server in the distributed storage system 103, where the data reading request carries the user identifier and the identifier of the application service data. The main node server receives the data reading request, and determines a second storage node set storing the application service data according to a user identifier carried by the request and the identifier of the application service data; and reading a plurality of copies of the application service data from each storage node in the second storage node set according to the identifier of the application service data. When the contents of the read multiple copies of the application service data are inconsistent, determining the confidence coefficient characteristics of the multiple copies of the application service data; determining a target application service data from the plurality of application service data according to the determined confidence characteristic; and sending the target application service data to the application server 102, so that the application server 102 pushes corresponding media content to the application client 101 according to the target application service data.
In some examples, the data processing method proposed in the present application may be applied to a master node server in the distributed storage system 103. As shown in fig. 2, the method includes two main processes: firstly, data writing processing (comprising steps 201 and 202) and secondly, data reading processing (comprising steps 203-208). In some examples, the method includes read data processing (including steps 203-208), in which case the primary node server may be responsible only for processing read data requests. In other examples, the method includes write data processing and read data processing (including steps 201-208), in which case the home node server is responsible for both write data request processing and read data request processing. The specific treatment of each step is as follows:
step 201: and receiving a data writing request from an application server, wherein the data writing request carries a user identifier and application service data to be stored associated with the user identifier.
Here, a write data request from the application server is received to write the application service data, so that the application server reads the application service data (such as member information, authority information, etc. of the application service) from the distributed storage system, and provides an online service according to the application service data, such as pushing corresponding media content to the application client (such as a video server pushing an advertisement or a video program to a video client according to video member information, authority information, etc.).
Step 202: and respectively storing the application service data to each storage node in a first storage node set, and associating the user identification and the identification of the application service data with the first storage node set.
In some examples, before the application service data is respectively stored to each storage node in the first storage node set, the method further includes: storing the application service data to the cache server and obtaining a storage address; wherein the storing the application service data to each storage node in a first storage node set respectively includes: and respectively sending a data writing request carrying the storage address to each storage node in the first storage node set so that each storage node acquires the application service data from the cache server according to the storage address.
After the cache server is configured, after the master node server in the distributed storage system responds to the write data request, the application service data may be stored in the cache server and a storage address is obtained, and the write operation completion information of the application service data in the cache server is sent to the application server; and then, asynchronously sending a data writing request carrying the storage address to each storage node in the first storage node set, so that each storage node acquires the application service data from the cache server according to the storage address to store the application service data and acquire multiple backup data of the application service data. By configuring the cache server, the write-in operation completion information of the application service data in the cache server is sent to the application server to inform the application server that the application service data is stored, so that the problem of large delay of data write-in operation can be further solved, the write-in operation of data is accelerated, and the running performance of the system is improved.
As mentioned above, the distributed storage system includes at least one node server, and here, after the master node server in the distributed storage system receives a write data request, the application service data of the application server may be stored in any multiple backup, depending on the requirement for data reliability and the performance of the distributed storage system itself. For example, the application service data may be stored in each storage server in the distributed storage system, or the application service data may be stored in a part of the storage servers in the distributed storage system.
In some examples, the storing the application service data to each storage node in a first storage node set in response to the write data request includes: sending the write data request to each storage node in the first storage node set; receiving response messages sent by one or more storage nodes in the first storage node set; and respectively storing the application service data to one or more storage nodes which send the response messages. Here, the distributed storage system does not use a consistency algorithm to ensure consistency of the multiple backups stored on the multiple storage nodes when writing data, and thus, the efficiency of writing data can be improved. Such as: when a response message sent by any storage node is received, the application service data can be stored in the storage node.
Specifically, the application server sends an operation (write data operation for short) of updating data (i.e., updating data a to data B) to the distributed storage system, and the master node server receives the write data operation and initiates an operation of updating data to the servers of the storage nodes in the distributed storage system. The server of any storage node may send a response message to the master node server to indicate that the server of the storage node may perform an operation to update data. In the case of using a consistency algorithm such as a PAXOS algorithm, the master node server and the servers of each storage node transmit a write data request and a response message based on a message transmission manner, and when more than half of the servers of the storage nodes send a response message to the master node server, it indicates that the data a can be updated to the data B, and the data B is written into each storage node. Such a process becomes more complex as the number of storage nodes increases, thereby causing communication delay and further causing data write delay. In contrast, in the embodiment of the present application, a consistency algorithm is not used when data is written, so as to ensure consistency of multiple backups, thereby significantly reducing time delay of writing data, greatly increasing the maximum writing rate that the server of each storage node can bear, and improving system performance.
Here, the data writing operation does not need to consider the data consistency and synchronization problems, and greatly simplifies the deployment and implementation of remote disaster recovery of the distributed storage system, thereby further improving the reliability of the application service data.
Step 203: and receiving a data reading request from the application server, wherein the data reading request carries a user identifier and an identifier of the application service data.
As described above, the application server stores application service data such as user registration information, member information, commodity order information, and the like in the distributed storage system, and when a user sends a media content access request to the application server through the application client, the application server needs to read the application service data of the user from the distributed storage system in order to determine media content pushed to the user, so as to determine information of the user. And the application server sends a data reading request to the distributed storage system, wherein the data reading request carries the user identification and the identification of the application service data.
For example, when a user logs in an application client, in order to determine home page information pushed to the application client, the application server needs to determine information of the user, such as whether the user is a member, and the user uses browsing records of the application client, so as to directionally and accurately push corresponding content to the user. Or the application server needs to send a read data request to the distributed storage system to read the information of the user from the distributed storage system in order to determine whether to push the pre-tile advertisement to the application client before pushing the corresponding video to the application client, and if the user is a member and can be free of advertisements, the application server only pushes the video to the application client and does not push the pre-tile advertisement.
Step 204: and determining a second storage node set storing the application service data according to the user identification and the identification of the application service data.
Here, in relation to a configuration change of the distributed storage nodes by the system, the first storage node set and the second storage node set may be the same or different, and the first storage node set and the second storage node set may or may not have an intersection. Such as: when the application server 1 writes data, according to the configuration at that time, the master node server may write the application service data from the application server 1 into its corresponding storage node set (for example, including nodes 1 and 2), and the subsequent system may change the storage node set corresponding to the application server 1, for example: instead, a set of storage nodes comprising nodes 1 and 3, and possibly nodes 3 and 4, may be included. For another example: due to factors such as that the application service data is stored in each storage node in the first storage node set for a long time, which may cause a storage node in the first storage node set to fail (that is, the storage node no longer stores the application service data), the non-failed storage node in the first storage node set belongs to the second storage node set. Thus, when the application server 1 reads data, the storage node storing the application service data may change, and the second storage node set and the first storage node set may be different. When the distributed storage system responds to the data reading request, according to the user identifier and the identifier of the application service data, all non-failed storage nodes storing the application service data, namely a second storage node set, need to be determined, so as to read multiple copies of the application data from each storage node in the second storage node set. It should be noted that, the first storage node set and the second storage node set are not named in a fixed manner, but rather are used to refer to an order of occurrence in one example for clarity, where "first" and "second" are used to represent different storage node sets, and in the same example, one storage node set that occurs first is referred to as a first storage node set, and another storage node set that occurs later is referred to as a second storage node set.
Step 205: and reading a plurality of copies of the application service data from each storage node in the second storage node set according to the identifier of the application service data.
As mentioned above, the distributed storage system stores multiple backups of the application service data, and when reading data, the distributed storage system reads all backups of the application service data, so as to avoid reading inaccuracy (for example, the application service data read from a node server is not up-to-date at this time) caused by inaccurate writing of the application service data in the node server, so as to further improve availability and accuracy of the application service data.
Step 206: and when the read contents of the multiple copies of the application service data are inconsistent, determining the confidence coefficient characteristics of the multiple copies of the application service data.
Here, the determining confidence characteristics of the multiple copies of the application service data may compare the multiple copies of the application service data to determine multiple different sets of application data; or analyzing the application service data to obtain a characteristic field in the application service data, wherein the characteristic field is used for expressing the characteristic attribute of the user; or determining a writing time stamp of the application service data so as to determine a target application service data from the multiple copies of the application service data according to the confidence characteristics of the multiple copies of the application service data.
Step 207: and determining a target application service data from the multiple application service data according to the determined confidence characteristic.
In some examples, the confidence characteristics of the multiple copies of the application service data include: the number of application service data having the same content; wherein the determining confidence characteristics of the multiple copies of the application service data includes: dividing the multiple copies of the application service data into a plurality of groups, wherein the application service data belonging to the same group have the same content, and different groups correspond to different contents; determining the number of application service data contained in each group; wherein the determining a target application service data from the plurality of application service data comprises: determining a group containing the maximum number of application service data; and using the determined application service data in the group as the target application service data.
Here, when the read multiple pieces of application service data are inconsistent, the master node server in the distributed storage system may arbitrate the multiple pieces of application service data by using a majority logic algorithm to determine the target application service data. For example, if the primary node server reads 3 backup data of the application service data from the distributed storage system, where 2 backup data are consistent, and the other backup data is inconsistent with the 2 backup data, then one of the 2 backup data is used as the target application service data according to a majority logic algorithm (i.e., a minority obeys majority).
In some examples, the confidence characteristics of the multiple copies of the application service data include: a confidence weight value of application service data having the same content; the method further comprises: configuring a weight coefficient for each storage node in advance; wherein the determining confidence characteristics of the multiple copies of the application service data includes: dividing the multiple copies of the application service data into a plurality of groups, wherein the application service data belonging to the same group have the same content, and different groups correspond to different contents; for each group, determining the confidence score of the group of application service data according to the weight coefficient of the storage node from which each part of application service data contained in the group comes; wherein the determining a target application service data from the plurality of application service data comprises: determining the group with the maximum confidence score; and using the determined application service data in the group as the target application service data.
Here, the distributed storage system may further add a weighting factor to the majority logic algorithm to assist arbitration, for example, a weighting factor may be configured in advance for each node server in the distributed storage system, for example, a higher weighting factor may be configured for a node server with better performance, the master node server reads 6 backup data of the application service data from the distributed storage system, and the backup data 1 and 2 read from the node servers 1 and 2 in the distributed storage system are the same, and the node servers 1 and 2 have better performance, so that the node servers 1 and 2 are respectively configured with weighting factors 3 in advance, the backup data 3 to 6 read from the node servers 3 to 6 in the distributed storage system are the same, and the node servers 3 to 6 have general performance, so that the node servers 3 to 6 are configured with weighting factors 1, and therefore, the backup data 1, 6, and 6 are configured with the weighting factors, The total credibility of 2 is 6 (namely the weight of the node server 1 + the weight of the node server 2), and the total credibility of the backup data 3-6 is 4 (namely the weight of the node server 3 + the weight of the node server 4 + the weight of the node server 5 + the weight of the node server 6), and one of the backup data 1 and 2 should be used as the target application service data according to the minority majority rule of the majority logic algorithm.
In some examples, the confidence characteristics of the multiple pieces of application service data include: a write timestamp for each of the application service data; wherein, the determining the confidence characteristics of the multiple pieces of application service data includes: determining a write timestamp of each piece of the application service data; wherein the determining a target application service data from the multiple application service data includes: determining application service data with the earliest writing timestamp; and using the determined application service data as the target application service data.
Here, when the read multiple pieces of application service data are inconsistent, the master node server in the distributed storage system may further arbitrate the multiple pieces of application service data by using a principle that a write timestamp is newer and more preferential, so as to determine the target application service data. Because the written time stamp is attached when the data is written, when the plurality of backup data read by the application server from the distributed storage system are inconsistent, the backup data with the latest written time stamp is used as the target application service data according to the principle that the newer the written time stamp is, the more the priority is, the more the written time stamp is.
In some examples, the confidence characteristics of the multiple pieces of application service data include: the value of a characteristic field related to confidence degree contained in the application service data; the determining the confidence characteristics of the multiple pieces of application service data includes: analyzing each application service data to obtain the characteristic fields contained in the application service data; determining the value of the characteristic field contained in the application service data; wherein the determining a target application service data from the multiple application service data includes: determining a value of the feature field representing a highest confidence; and using an application service data containing the value of the characteristic field representing the highest confidence as the target application service data.
Here, when the read multiple pieces of application service data are inconsistent, the master node server in the distributed storage system may further arbitrate the multiple pieces of application service data by using a custom ticket method, determine the target application service data, that is, analyze the application service data to obtain a feature field included in the application service data, such as a term field such as a member validity period, a coupon validity period, and the like, and a feature field such as the number of movie viewing tickets, and use the feature field as a ticket to determine the target application service data of the user according to the ticket.
For example, the application server requests to read the member information of the user from the distributed storage system, the master node server in the distributed storage system analyzes a plurality of pieces of member information read from each storage node to obtain one validity field in the member information, the validity field is used as an arbitration field, when a plurality of different validity fields occur, the value of the maximum validity field is determined, one piece of member information with the maximum value of the validity field is determined as target member information, and the member information with the longest validity period or the member information which expires at the latest is considered to have the highest confidence and is used as target application service data. For another example, the master node server may further analyze multiple pieces of member information read from each storage node to obtain one field of number of viewing tickets in the member information, and use the field as an arbitration field, when multiple different fields of number of viewing tickets occur, determine a value of the field of the largest number of viewing tickets, and determine a piece of member information with the largest value of the field of number of viewing tickets as target member information, that is, consider that the confidence coefficient of the member information representing the largest number of viewing tickets is highest, and use the target member information as target application service data.
In some examples, the confidence characteristics of the multiple pieces of application traffic data include: the method comprises the steps that the number of application service data with the same content, writing time stamps contained in the application service data and values of characteristic fields contained in the application service data and related to confidence degrees are the same, wherein the number of the application service data corresponds to a first weight coefficient, the writing time stamps correspond to a second weight coefficient, and the values of the characteristic fields correspond to a third weight coefficient; wherein, the determining the confidence characteristics of the multiple pieces of application service data includes: dividing the multiple application service data into multiple sets, wherein the application service data belonging to the same set have the same content, and different sets correspond to different contents; determining the number of application service data contained in each set; determining a write timestamp of each piece of application service data; analyzing each part of the application service data to obtain the characteristic field contained in the application service data, and determining the value of the characteristic field contained in the part of the application service data; wherein the determining a target application service data from the multiple application service data includes: determining a first group containing the maximum number of application service data; determining a second group containing application service data with the earliest writing time stamp; determining a value of the feature field representing the highest confidence level and a third set of application traffic data comprising the value of the feature field representing the highest confidence level; configuring the first weight coefficient for each application service data in the first group, configuring the second weight coefficient for each application service data in the second group, and configuring the third weight coefficient for each application service data in the third group; calculating confidence scores of the application service data in the first group, the second group and the third group according to the first weight coefficient, the second weight coefficient and the third weight coefficient; and taking the application service data corresponding to the maximum confidence score as the target application service data.
In some examples, before configuring the first weighting factor for each application traffic data in the first set, the second weighting factor for each application traffic data in the second set, and the third weighting factor for each application traffic data in the third set, the method further comprises: and when determining that one part of application service data belongs to the first group, the second group and the third group, taking the part of application service data as the target application service data.
Here, the master node server in the distributed storage system may also use any combination of a majority logic algorithm, a write time stamp priority principle, and a custom ticketing method to determine the target application service data. For example, a majority logic algorithm, a write time stamp priority rule and a custom ticket method are simultaneously adopted to determine target application service data, a first group containing most application service data (for example, the first group contains 5 pieces of application service data) is determined by using the majority logic algorithm, a second group containing application service data with the earliest write time stamp (for example, the second group contains 2 pieces of application service data) is determined by using the write time stamp priority rule, a third group containing application service data with the earliest member validity period (for example, the third group contains 2 pieces of application service data) is determined by using the custom ticket method, configuring the first weight coefficient (e.g. 2) for each application traffic data in the first group, configuring the second weight coefficient (e.g. 2) for each application traffic data in the second set, and configuring the third weight coefficient (such as 3) for each application service data in the third group; and if one application service data is in the first group and the second group, or the second group and the third group, or the first group, the second group and the third group, and so on, the confidence score of the application service number is the sum of the weighting coefficients of the groups, and the application service data corresponding to the maximum confidence score is taken as the target application service data. And if one part of application service data belongs to the first group, the second group and the third group at the same time, the confidence score of the part of application service data is the highest, and the part of application service data is used as target application service data.
Step 208: and sending the target application service data to the application server.
Here, the distributed storage system transmits the target application service data to the application server, so that the application server transmits corresponding content to the application client according to the target application service data.
For example, when a user sends a media content access request to an application server through an application client, a Software Development Kit (SDK) for pushing promotion content (such as an advertisement) in the application client sends a promotion content push request (such as an advertisement exposure request) to the application server, for example, an advertisement SDK, when the target application service data indicates that the user is an authorized user and the authority limit of the user is still within the valid period, the application server does not respond to the promotion content push request, that is, does not push the promotion content to the application client, only responds to the media content access request, sends the media content to the application client, and the application client displays the media content and does not display the promotion content.
By the technical scheme, the complexity of data reading can be reduced, the accuracy of the data reading is improved, the deployment and implementation of remote disaster recovery of the distributed storage system are greatly simplified, and the reliability of the data is further improved. For example, by adopting the technical scheme, for the data reading service of the member information, when the data reading request rate is high, the accuracy of the read data reaches 99.99999%, and when the data reading request rate is low, the accuracy of the read data even reaches 99.9999999%.
In some examples, the distributed storage system includes a plurality of master node servers, e.g., as shown in fig. 3, a master node server 1, a master node server 2, and a master node server 3, which may be located in different domains (i.e., different domain names for each master node server). A plurality of application servers may initiate a plurality of write data requests (i.e. write service 1, write service 2, and write service 3 as shown in fig. 3) and a plurality of read data requests to the distributed storage system, when one application server initiates a write data request or a read data request to the distributed storage system, a master node server having the same network operator as the application server and closer to the area of the application server may be selected, for example, when the application server 1 initiates write service 1, the area where the application server 1 is located is some city in Jiangsu province, and the bandwidth used by the application server 1 is a broadband for internet access, the application server 1 may select a master node server for internet access in the east area, for example, the master node server 1, to store the application service data in the write data request to each storage node in the first storage node set, as shown in fig. 3, node server 1, node server 2 and node server 3.
When an application server initiates a data reading request to a distributed storage system, a method for selecting a master node server is the same as the method for selecting a master node server during writing service, that is, a master node server having the same network operator as the application server is selected nearby, for example, the application server 1 selects the master node server 1 nearby, and sends the data reading request to the master node server 1, and the master node server 1 determines a second storage node set storing the application service data according to the user identifier and the identifier of the application service data in response to the data reading request, where the second storage node set is the same as the node servers included in the first storage node set and is all the node server 1, the node server 2, and the node server 3; the master node server 1 reads multiple application service data from each storage node in the second storage node set according to the identifier of the application service data; when the contents of the read multiple application service data are inconsistent, the master node server 1 determines the confidence characteristics of the multiple application service data according to the arbitration algorithm, determines a target application service data from the multiple application service data according to the determined confidence characteristics, and sends the target application service data to the application server 1.
Fig. 4 shows an interaction diagram of a data processing method proposed in the present application. In this example, the method is applied to a distributed storage system, and the interaction process involves four node servers in the distributed storage system, namely, a master node server, a node server 1, a node server 2, and a node server 3. Taking the promotion content as an advertisement as an example, the application client is a video APP, and the application server is a video server, wherein the video server can push a video and an advertisement to the video APP, as shown in the figure, the method includes the following steps:
and (3) a data writing process: step 401 to step 405
Step 401: the user completes the registration operation on the video APP and completes the member purchasing operation.
Step 402: and the video APP sends a data writing request to the video server to request to store the application service data generated in the process, namely user registration information and member information.
Step 403: the video server establishes user registration information and member information of the user according to the data storage request, such as member names, member validity periods and the like; if the user already has member information on the video server, the video server rewrites the validity period of the member.
Step 404: the video server sends a data writing request to a main node server in the distributed storage system, wherein the data writing request carries a user identifier and application service data to be stored related to the user identifier.
Step 405: and the main node server responds to the data writing request, stores the application service data to each storage node in a first storage node set respectively, and associates the user identification and the identification of the application service data with the first storage node set. Here, each storage node in the first set of storage nodes is a node server 1-3. In some examples, the distributed storage system may be implemented based on redis 2.6.
And (3) reading data: step 406 to step 411
Step 406: the video app sends a video content acquisition request to the video server, and meanwhile, an advertisement SDK in the video app sends an advertisement pushing request to the video server so as to acquire the front-mounted advertisement.
Step 407: and the video server generates a read data request according to the advertisement push request, wherein the read data request carries a user identifier and an identifier of the application service data, and sends the read data request to the main node server.
Step 408: and responding to the read data request by a master node server in the distributed storage system, and determining a second storage node set storing the application service data according to the user identifier and the identifier of the application service data, wherein the second storage node set may be different from the first storage node set, and in this example, the first storage node set and the second storage node set are the same and are both node servers 1-3.
Step 409: and reading the application service data of the user from each storage node by each node server storing the application service data according to the identification of the application service data.
Step 410: and when the contents of the read multiple application service data are inconsistent, the main node server determines target application service data according to an arbitration algorithm and sends the determined target application service data to the video server. As mentioned above, the arbitration algorithm may be a majority logic algorithm, a write timestamp newer and more preferred rule, a custom ticketing method, or any combination thereof.
Step 411: when the target application service data indicates that the user is a member and the member deadline of the user is still within the valid period, the video server does not respond to the advertisement pushing request, namely does not push the advertisement to the client, and only sends the video content to the client.
Step 412: the video APP shows the video content.
Based on the method provided by the above example, the present application also provides a data processing apparatus 500. As shown in fig. 5, the apparatus 500 includes:
the receiving module 501 receives a read data request from an application server, where the read data request carries a user identifier and an identifier of application service data.
The reading module 503, in response to the read data request, determines a first storage node set in which the application service data is stored according to the user identifier and the identifier of the application service data; and reading a plurality of application service data from each storage node in the first storage node set according to the identification of the application service data.
A determining module 504, configured to determine confidence characteristics of the multiple copies of application service data when the read contents of the multiple copies of application service data are inconsistent; and determining a target application service data from the multiple application service data according to the determined confidence characteristic.
And a sending module 505, configured to send the target application service data to the application server.
In some examples, the apparatus 500 further comprises: a storage module 502. The receiving module 501 further receives a data writing request from the application server, where the data writing request carries a user identifier and application service data to be stored associated with the user identifier; and the storage module 502 responds to the write data request, stores the application service data to each storage node in a second storage node set, and associates the user identifier and the identifier of the application service data carried in the write data request with the second storage node set.
In some examples, the apparatus 500 further comprises: the cache module 506 stores the application service data to the cache server and obtains a storage address; the storing module 502 stores the application service data to each storage node in a second storage node set, respectively, including: and sending a data writing request carrying the storage address to each storage node in the second storage node set respectively, so that each storage node acquires the application service data from the cache server according to the storage address.
In some examples, the storing module 502, in response to the write data request, stores the application service data to each storage node in a second storage node set respectively, including: sending the write data request to each storage node in the first storage node set; the receiving module 501 further receives a response message sent by one or more storage nodes in the second storage node set; the storage module 502 further stores the application service data to the one or more storage nodes sending the response messages respectively.
In some examples, the confidence characteristics of the multiple pieces of application service data include: the number of application service data having the same content; the determining module 504 determines the confidence characteristics of the multiple pieces of application service data, including: dividing the multiple application service data into multiple groups, wherein the application service data belonging to the same group have the same content, and different groups correspond to different contents; determining the number of application service data contained in each group; the determining module 504 determines a target application service data from the multiple application service data sets, including: determining a group containing the maximum number of application service data; and using the determined application service data in the group as the target application service data.
In some examples, the confidence characteristics of the multiple pieces of application service data include: a confidence weight value of application service data having the same content; the apparatus 500 further comprises: a configuration module 507, which configures a weight coefficient for each storage node in advance; the determining module determines confidence characteristics of the multiple pieces of application service data, and the determining module includes: dividing the multiple application service data into multiple groups, wherein the application service data belonging to the same group have the same content, and different groups correspond to different contents; for each group, determining the confidence score of the group of application service data according to the weight coefficient of the storage node from which each part of application service data contained in the group comes; wherein the determining module determines a target application service data from the multiple application service data, and includes: determining the group with the maximum confidence score; and using the determined application service data in the group as the target application service data.
In some examples, the confidence characteristics of the multiple pieces of application service data include: a write timestamp for each of the application service data; the determining module 504 determines the confidence characteristics of the multiple pieces of application service data, including: determining a write timestamp of each piece of the application service data; the determining module 504 determines a target application service data from the multiple application service data sets, including: determining application service data with the earliest writing timestamp; and using the determined application service data as the target application service data.
In some examples, the confidence characteristics of the multiple pieces of application service data include: the value of a characteristic field related to confidence degree contained in the application service data; the determining module 504 determines the confidence characteristics of the multiple pieces of application service data, including: analyzing each application service data to obtain the characteristic fields contained in the application service data; determining the value of the characteristic field contained in the application service data; the determining module 504 determines a target application service data from the multiple application service data sets, including: determining a value of the feature field representing a highest confidence; and using an application service data containing the value of the characteristic field representing the highest confidence as the target application service data.
In some examples, the confidence characteristics of the multiple pieces of application traffic data include: the method comprises the steps that the number of application service data with the same content, writing time stamps contained in the application service data and values of characteristic fields contained in the application service data and related to confidence degrees are the same, wherein the number of the application service data corresponds to a first weight coefficient, the writing time stamps correspond to a second weight coefficient, and the values of the characteristic fields correspond to a third weight coefficient; the determining module 504 determines the confidence characteristics of the multiple pieces of application service data, including: dividing the multiple application service data into multiple sets, wherein the application service data belonging to the same set have the same content, and different sets correspond to different contents; determining the number of application service data contained in each set; determining a write timestamp of each piece of application service data; analyzing each part of the application service data to obtain the characteristic field contained in the application service data, and determining the value of the characteristic field contained in the part of the application service data; the determining module 504 determines a target application service data from the multiple application service data sets, including: determining a first group containing the maximum number of application service data; determining a second group containing application service data with the earliest writing time stamp; determining a value of the feature field representing the highest confidence level and a third set of application traffic data comprising the value of the feature field representing the highest confidence level; the configuration module 507 configures the first weight coefficient for each application service data in the first group, configures the second weight coefficient for each application service data in the second group, and configures the third weight coefficient for each application service data in the third group; the determining module 504 further calculates confidence scores of the application service data in the first group, the second group and the third group according to the first weight coefficient, the second weight coefficient and the third weight coefficient; and taking the application service data corresponding to the maximum confidence score as the target application service data.
In some examples, before the configuring module 507 configures the first weighting factor for each application service data in the first group, configures the second weighting factor for each application service data in the second group, and configures the third weighting factor for each application service data in the third group, the determining module 504 is further configured to: and when determining that one part of application service data belongs to the first group, the second group and the third group, taking the part of application service data as the target application service data.
The specific implementation principle of the functions of the above modules has been described in the foregoing, and is not described herein again.
In addition, the data processing method and the data processing apparatus in each example of the present application and each module therein may be integrated into one processing unit, or each module may exist alone physically, or two or more devices or modules may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
Fig. 6 shows a block diagram of the components of a computing device 600 in which the data processing apparatus 500 is located. As shown in fig. 6, the computing device includes one or more processors (CPUs) 602, memory 604, user interface 606, and communication bus 608 for interconnecting these components.
The user interface 606 includes one or more output devices 612, including one or more speakers and/or one or more visual displays. The user interface 610 also includes one or more input devices 614, including, for example, a keyboard, a mouse, a voice command input unit or microphone, a touch screen display, a touch sensitive tablet, a gesture capture camera or other input buttons or controls, and the like.
The memory 604 stores a set of instructions executable by the processor 602, including:
an operating system 616, including programs for handling various basic system services and for performing hardware related tasks;
the application 618 includes various application programs for data processing, which can implement the processing flow in the above examples, and may include some or all of the units in the data processing apparatus 500 shown in fig. 5, for example. Each of the modules 501-507 may store machine executable instructions. The processor 602 can further implement the functions of the modules 501 and 507 by executing the machine executable instructions of the modules 501 and 507 in the memory 606.
It should be noted that not all steps and modules in the above flows and structures are necessary, and some steps or modules may be omitted according to actual needs. The execution order of the steps is not fixed and can be adjusted as required. The division of each module is only for convenience of describing adopted functional division, and in actual implementation, one module may be divided into multiple modules, and the functions of multiple modules may also be implemented by the same module, and these modules may be located in the same device or in different devices.
The hardware modules in the examples may be implemented in hardware or a hardware platform plus software. The software includes machine-readable instructions stored on a non-volatile storage medium. Thus, the examples may also be embodied as software products. For example, corresponding to the above data processing method and apparatus, the examples of the present application also provide a computer readable storage medium having stored thereon computer instructions, wherein the computer instructions, when executed by a processor, implement the steps of the above data processing method.
In various examples, the hardware may be implemented by specialized hardware or hardware executing machine-readable instructions. For example, the hardware may be specially designed permanent circuits or logic devices (e.g., special purpose processors, such as FPGAs or ASICs) for performing the specified operations. Hardware may also include programmable logic devices or circuits temporarily configured by software (e.g., including a general purpose processor or other programmable processor) to perform certain operations.
In addition, each example of the present application can be realized by a data processing program executed by a data processing apparatus such as a computer. It is clear that a data processing program constitutes the present application. Further, a data processing program, which is generally stored in one storage medium, is executed by directly reading the program out of the storage medium or by installing or copying the program into a storage device (such as a hard disk and/or a memory) of the data processing device. Such a storage medium therefore also constitutes the present application, which also provides a non-volatile storage medium in which a data processing program is stored, which data processing program can be used to carry out any one of the above-mentioned method examples of the present application.
The machine-readable instructions corresponding to the modules in fig. 5 may cause an operating system or the like operating on the computer to perform some or all of the operations described herein. The nonvolatile computer-readable storage medium may be a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer. A CPU or the like mounted on the expansion board or the expansion unit may perform part or all of the actual operations according to the instructions.
In addition, the devices and modules in the examples of the present application may be integrated into one processing unit, or each module may exist alone physically, or two or more devices or modules may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The above description is only a preferred example of the present application and should not be taken as limiting the present application, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present application should be included in the scope of the present application.
Claims (15)
1. A method of data processing, the method comprising:
receiving a data reading request from an application server, wherein the data reading request carries a user identifier and an identifier of application service data;
determining a first storage node set storing the application service data according to the user identification and the identification of the application service data; reading multiple application service data from each storage node in the first storage node set according to the identifier of the application service data;
when the read contents of the multiple application service data are inconsistent, determining the confidence coefficient characteristics of the multiple application service data; determining target application service data from the multiple copies of application service data according to the confidence characteristic; sending the target application service data to the application server;
the confidence characteristics of the multiple pieces of application service data comprise: the method comprises the steps that the number of application service data with the same content, writing time stamps contained in the application service data and values of characteristic fields contained in the application service data and related to confidence degrees are the same, wherein the number of the application service data corresponds to a first weight coefficient, the writing time stamps correspond to a second weight coefficient, and the values of the characteristic fields correspond to a third weight coefficient;
wherein, the determining the target application service data from the multiple copies of application service data according to the confidence characteristic comprises:
determining a first group containing the maximum number of application service data; determining a second group containing application service data with the earliest writing time stamp; determining a value of the characteristic field representing the highest confidence level and a third group of application traffic data containing the value of the characteristic field representing the highest confidence level;
configuring the first weight coefficient for each application service data in the first group, the second weight coefficient for each application service data in the second group, and the third weight coefficient for each application service data in the third group;
calculating confidence scores of the application service data in the first group, the second group and the third group according to the first weight coefficient, the second weight coefficient and the third weight coefficient; and taking the application service data corresponding to the maximum confidence score as the target application service data.
2. The method of claim 1, further comprising:
receiving a data writing request from the application server, wherein the data writing request carries the user identifier and the associated application service data to be stored;
and respectively storing the application service data to be stored to each storage node in a second storage node set, and associating the user identifier and the identifier of the application service data to be stored with the second storage node set.
3. The method according to claim 2, wherein before storing the application service data to be stored to each storage node in the second storage node set, the method further comprises:
storing the application service data to be stored to a cache server and obtaining a storage address;
wherein, the storing the application service data to be stored to each storage node in a second storage node set respectively includes:
and respectively sending a data writing request carrying the storage address to each storage node in the second storage node set, so that each storage node acquires the application service data to be stored from the cache server according to the storage address.
4. The method according to claim 2, wherein when the application service data to be stored is stored to each storage node in the second storage node set, a consistency algorithm is not adopted to ensure consistency among the multiple pieces of application service data stored in each storage node.
5. The method of claim 1, wherein applying traffic data comprises: member information and/or authority information of the application service.
6. The method of claim 2, wherein the write data request includes user registration information and/or member information.
7. The method of claim 1, wherein the method is applied to a master node server in a distributed storage system.
8. The method of claim 1, wherein the characteristic field refers to a validity period field in the member information.
9. The method of claim 1, wherein the determining confidence characteristics for the multiple applications traffic data comprises:
dividing the multiple application service data into multiple sets, wherein the application service data belonging to the same set have the same content, and different sets correspond to different contents;
determining the number of application service data contained in each set;
determining a writing time stamp of each application service data; and
and analyzing each application service data to obtain the characteristic field contained in the application service data, and determining the value of the characteristic field contained in the application service data.
10. The method of claim 1, wherein prior to configuring the first weight coefficient for each application traffic data in the first group, the second weight coefficient for each application traffic data in the second group, and the third weight coefficient for each application traffic data in the third group, the method further comprises:
and when determining that one part of application service data belongs to the first group, the second group and the third group, taking the part of application service data as the target application service data.
11. A data processing apparatus, characterized in that the apparatus comprises:
the receiving module is used for receiving a data reading request from the application server, wherein the data reading request carries a user identifier and an identifier of application service data;
the reading module is used for determining a first storage node set storing the application service data according to the user identification and the identification of the application service data; reading multiple application service data from each storage node in the first storage node set according to the identifier of the application service data;
the determining module is used for determining the confidence coefficient characteristics of the multiple copies of application service data when the read contents of the multiple copies of application service data are inconsistent; determining target application service data from the multiple copies of application service data according to the confidence characteristic;
the sending module is used for sending the target application service data to the application server;
the confidence characteristics of the multiple pieces of application service data comprise: the method comprises the steps that the number of application service data with the same content, writing time stamps contained in the application service data and values of characteristic fields contained in the application service data and related to confidence degrees are the same, wherein the number of the application service data corresponds to a first weight coefficient, the writing time stamps correspond to a second weight coefficient, and the values of the characteristic fields correspond to a third weight coefficient;
the determining module is used for determining a first group containing the maximum number of application service data; determining a second group containing application service data with the earliest writing time stamp; determining a value of the characteristic field representing the highest confidence level and a third group of application traffic data containing the value of the characteristic field representing the highest confidence level; configuring the first weight coefficient for each application service data in the first group, the second weight coefficient for each application service data in the second group, and the third weight coefficient for each application service data in the third group; calculating confidence scores of the application service data in the first group, the second group and the third group according to the first weight coefficient, the second weight coefficient and the third weight coefficient; and taking the application service data corresponding to the maximum confidence score as the target application service data.
12. The apparatus of claim 11, further comprising a storage module;
the receiving module is further configured to receive a data writing request from the application server, where the data writing request carries the user identifier and the application service data to be stored associated with the user identifier; and
the storage module is configured to store the application service data to be stored to each storage node in a second storage node set, and associate the user identifier and the identifier of the application service data to be stored with the second storage node set.
13. The apparatus of claim 12, wherein the apparatus further comprises:
the cache module is used for storing the application service data to be stored to a cache server and obtaining a storage address;
the storage module is configured to send a write data request carrying the storage address to each storage node in the second storage node set, so that each storage node obtains the application service data to be stored from the cache server according to the storage address.
14. A computer-readable storage medium storing computer-readable instructions for causing at least one processor to perform the method of any one of claims 1 to 10.
15. A computing device comprising a memory and a processor, the memory having stored therein computer-readable instructions that, when executed by the processor, implement the method of any of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810257102.XA CN110196680B (en) | 2018-03-27 | 2018-03-27 | Data processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810257102.XA CN110196680B (en) | 2018-03-27 | 2018-03-27 | Data processing method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110196680A CN110196680A (en) | 2019-09-03 |
| CN110196680B true CN110196680B (en) | 2021-10-26 |
Family
ID=67751054
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810257102.XA Active CN110196680B (en) | 2018-03-27 | 2018-03-27 | Data processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110196680B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104456A (en) * | 2019-10-12 | 2020-05-05 | 深圳壹账通智能科技有限公司 | Data persistence storage method and device, computer equipment and storage medium |
| CN111212111B (en) * | 2019-12-17 | 2022-05-13 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
| CN111541762B (en) * | 2020-04-20 | 2023-08-01 | 广州酷狗计算机科技有限公司 | Data processing method, management server, device and storage medium |
| CN113625936B (en) * | 2020-05-09 | 2024-07-12 | 腾讯科技（深圳）有限公司 | Data processing method, device, storage medium and equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521084A (en) * | 2011-12-20 | 2012-06-27 | 惠州市亿能电子有限公司 | Data storage and reading method |
| CN102970350A (en) * | 2012-11-05 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Method for maintaining replica consistency in storage system |
| EP2732375A4 (en) * | 2011-07-12 | 2015-01-28 | Violin Memory Inc | Raided memory system |
| CN104935654A (en) * | 2015-06-10 | 2015-09-23 | 华为技术有限公司 | Caching method, write point client and read client in server cluster system |
| CN105516230A (en) * | 2014-10-15 | 2016-04-20 | 深圳市腾讯计算机系统有限公司 | Data processing method and device |
| CN105608143A (en) * | 2015-12-17 | 2016-05-25 | 北京奇虎科技有限公司 | Detection method and device of consistency of multi-copy data |
| CN105892954A (en) * | 2016-04-25 | 2016-08-24 | 乐视控股（北京）有限公司 | Data storage method and device based on multiple copies |
| CN106325773A (en) * | 2016-08-23 | 2017-01-11 | 浪潮(北京)电子信息产业有限公司 | Data consistency guaranteeing method and system of storage system and cache apparatus |
| CN106375416A (en) * | 2016-08-30 | 2017-02-01 | 北京航空航天大学 | Consistency dynamic adjustment method and device in distributed data storage system |
| CN106484311A (en) * | 2015-08-31 | 2017-03-08 | 华为数字技术（成都）有限公司 | A kind of data processing method and device |
-
2018
- 2018-03-27 CN CN201810257102.XA patent/CN110196680B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2732375A4 (en) * | 2011-07-12 | 2015-01-28 | Violin Memory Inc | Raided memory system |
| CN102521084A (en) * | 2011-12-20 | 2012-06-27 | 惠州市亿能电子有限公司 | Data storage and reading method |
| CN102970350A (en) * | 2012-11-05 | 2013-03-13 | 曙光信息产业(北京)有限公司 | Method for maintaining replica consistency in storage system |
| CN105516230A (en) * | 2014-10-15 | 2016-04-20 | 深圳市腾讯计算机系统有限公司 | Data processing method and device |
| CN104935654A (en) * | 2015-06-10 | 2015-09-23 | 华为技术有限公司 | Caching method, write point client and read client in server cluster system |
| CN106484311A (en) * | 2015-08-31 | 2017-03-08 | 华为数字技术（成都）有限公司 | A kind of data processing method and device |
| CN105608143A (en) * | 2015-12-17 | 2016-05-25 | 北京奇虎科技有限公司 | Detection method and device of consistency of multi-copy data |
| CN105892954A (en) * | 2016-04-25 | 2016-08-24 | 乐视控股（北京）有限公司 | Data storage method and device based on multiple copies |
| CN106325773A (en) * | 2016-08-23 | 2017-01-11 | 浪潮(北京)电子信息产业有限公司 | Data consistency guaranteeing method and system of storage system and cache apparatus |
| CN106375416A (en) * | 2016-08-30 | 2017-02-01 | 北京航空航天大学 | Consistency dynamic adjustment method and device in distributed data storage system |
Non-Patent Citations (1)
| Title |
|---|
| 分布式存储系统中数据副本管理机制;徐小龙 等;《计算机技术与发展》;20130228;第245-249页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110196680A (en) | 2019-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11507480B2 (en) | Locality based quorums | |
| CN110196680B (en) | Data processing method, device and storage medium | |
| US8301600B1 (en) | Failover recovery in a distributed data store | |
| US9141685B2 (en) | Front end and backend replicated storage | |
| TWI509423B (en) | Synchronizing updates across cluster filesystems | |
| CN106815218B (en) | Database access method and device and database system | |
| US10747776B2 (en) | Replication control using eventually consistent meta-data | |
| US11226745B2 (en) | Optimizing data writes in a distributed computing system | |
| US10489378B2 (en) | Detection and resolution of conflicts in data synchronization | |
| US20130166505A1 (en) | Monitoring replication lag between geographically dispersed sites | |
| US9229970B2 (en) | Methods to minimize communication in a cluster database system | |
| CN112148798A (en) | Data processing method and device applied to distributed system | |
| CN113010549A (en) | Data processing method based on remote multi-active system, related equipment and storage medium | |
| CN114077518B (en) | Data snapshot method, device, equipment and storage medium | |
| US10599629B2 (en) | Reduced locking in a distributed data store | |
| WO2019109256A1 (en) | Log management method, server and database system | |
| CN107025257B (en) | Transaction processing method and device | |
| CN114168636A (en) | Data processing method, device and equipment based on cache consistency and storage medium | |
| CN117435569A (en) | Dynamic capacity expansion method, device, equipment, medium and program product for cache system | |
| CN112417043A (en) | Data processing system and method | |
| WO2021189283A1 (en) | Method and device for data processing, electronic device, and storage medium | |
| CN113253924B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN109976944B (en) | Data processing method and system, storage medium and electronic device | |
| CN111459931A (en) | Data duplication checking method and data duplication checking device | |
| US12032555B2 (en) | Techniques for synchronous access to database replicas |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |