CN110196680A

CN110196680A - Data processing method, device and storage medium

Info

Publication number: CN110196680A
Application number: CN201810257102.XA
Authority: CN
Inventors: 朱元; 王金华; 李大江
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2018-03-27
Filing date: 2018-03-27
Publication date: 2019-09-03
Anticipated expiration: 2038-03-27
Also published as: CN110196680B

Abstract

This application discloses a kind of data processing method, device and storage mediums.This method comprises: receiving the read data request from application server；According to the mark of the user identifier and applied business data that carry in read data request, the second memory node set for being stored with applied business data is determined；More parts of applied business data are read from each memory node in the second memory node set according to the mark of applied business data；When the content of more parts of applied business data of reading is inconsistent, the confidence characteristic of more parts of applied business data is determined；According to identified confidence characteristic, portion target application business datum is determined from more parts of applied business data；Target application business datum is sent to application server.

Description

Data processing method, device and storage medium

Technical field

This application involves Internet technical field more particularly to a kind of data processing methods, device and storage medium.

Background technique

With the development of information technology, global metadata is in explosive growth, and the storage of these data is to data storage technology More stringent requirements are proposed.Data storage technology requires mainly for meeting simplification and two class of reliability.Under normal conditions, if The realization of data-storage system is simpler, then being difficult to guarantee from the reliability for wherein reading data；If being stored from data The reliability that data are read in system is higher, then the realization of the data-storage system is more complicated.Therefore, a kind of letter how is realized Data-storage system both with easy property and reliability is those skilled in the art's problem to be solved.

Summary of the invention

Present application example proposes a kind of data processing method, device and storage medium, to reduce distributed memory system The middle complexity for reading data and the accuracy that can guarantee data.

Present application example provides a kind of data processing method, which comprises receives the reading from application server Request of data, the read data request carry the mark of user identifier and applied business data；According to the user identifier and institute The mark of applied business data is stated, determines the first memory node set for being stored with the applied business data；It is answered according to described More parts of applied business data are read from each memory node in the first memory node set with the mark of business datum；When When the content of the more parts of applied business data read is inconsistent, determine that the confidence level of the more parts of applied business data is special Sign；According to the identified confidence characteristic, portion target application business number is determined from the more parts of applied business data According to；The target application business datum is sent to the application server.

Present application example additionally provides a kind of data processing equipment, and described device includes: receiving module, receives and carrys out self-application The read data request of server, the read data request carry the mark of user identifier and applied business data；Read module, root According to the mark of the user identifier and the applied business data, the first storage section for being stored with the applied business data is determined Point set；It is more from each memory node reading in the first memory node set according to the mark of the applied business data Part applied business data；Determining module determines described more when the content of the more parts of applied business data of reading is inconsistent The confidence characteristic of part applied business data；According to the identified confidence characteristic, from the more parts of applied business data Middle determining a target application business datum；The target application business datum is sent to the application service by sending module Device.

Present invention also provides a kind of storage mediums, are stored with computer-readable instruction, can make at least one processor Execute the above method.

By adopting the above technical scheme, it is possible to reduce read the complexity of data in distributed memory system, improve and read data Accuracy, to further improve system performance.

Detailed description of the invention

It, below will be to needed in example description in order to illustrate more clearly of the technical solution in present application example Attached drawing is briefly described, it should be apparent that, the accompanying drawings in the following description is only some examples of the application, for this field For those of ordinary skill, without any creative labor, it can also be obtained according to these attached drawings other attached Figure.

Fig. 1 be this application involves a kind of system architecture schematic diagram；

Fig. 2 is the method flow diagram of one example of the application；

Fig. 3 be this application involves distributed treatment；

Fig. 4 is the interacting message figure that one media data of the application handles example；

Fig. 5 is the structure drawing of device of one example of the application；And

Fig. 6 is the equipment structure chart of one example of the application.

Specific embodiment

Below in conjunction with the attached drawing in present application example, the technical solution in present application example is carried out clearly and completely Description, it is clear that described example is only a part of example of the application, rather than whole examples.Based on the reality in the application Example, every other example obtained by those of ordinary skill in the art without making creative efforts belong to this Apply for the range of protection.

In some instances, all data are stored using the storage server of concentration, i.e., all numbers in application server According to being written in centrally stored server, the reading data service of application server is also all read from the centrally stored server It takes.This method has very high simplification, does not do any backup storage, but in the event of a network outage, the number of storage According to just completely inaccessible.At this point, centrally stored server just becomes the bottleneck of system performance, it is not able to satisfy Mass storage The needs of application.

In some instances, using distributed memory system, data dispersion is stored in more independent storage servers On.Using expansible system structure, storage load is shared using more storage servers, that is, same when data write service Part data can do more parts of backups.Reading data service can be according to the Regional Distribution of application server and data storage server, just It is close to read portion.This method not only increases the reliability, availability and access efficiency of system, is also easy to extend.But due to The presence of situations such as failure and parallel memorizing, there may be inconsistent feelings between multiple backups of the same data of reading Condition, this kind of inconsistence problems can be tolerated sometimes, then be difficult to tolerate sometimes.Under normal conditions, it when writing data, can use The consistency algorithms such as PAXOS, ZOOKEEPER, RAFT, VIEWSTAMPED REPLICATION guarantee above-mentioned multiple backups Consistency, the processing of this strong consistency implement sufficiently complex, confirm that the process of consistency is also very long, caused by write data Processing delay is also larger.

Based on above-mentioned technical problem, the application proposes a kind of data processing method, device and storage medium.The above method can Applied in system architecture shown in FIG. 1.As shown in Figure 1, the system architecture includes: applications client 101, application server 102 and distributed memory system 103, these entities are communicated by internet 104, wherein distributed memory system 103 wraps Multiple node servers are included, as shown in Figure 1, host node server, node server 1, node server 2 ... node server n。

User accesses application server 102 using applications client 101, such as: browsing webpage or viewing Online Video Deng, application server 102 can be to provide the Website server of various Internet services, such as: portal site server, offer Online Video/audio plays the server of service, server of social platform etc., wherein application server 102 can Xiang Yingyong Client 101 provides various services.User can generate various needs during completing various operations using applications client 101 The data of distributed memory system 103 are stored, such as: user can generate user's registration information, user after completing registration operation Membership information can be generated after completing membership buying operation, user can generate goods orders information etc. after completing commodity purchasing operation Deng.The applied business data such as these user's registration informations, membership information, goods orders information can be stored by application server 102 Into distributed memory system 103.Specifically, application server 102 sends write data requests to distributed memory system 103, Write data request carries user identifier and associated applied business data to be stored；In distributed memory system 103 Host node server is requested in response to write data, and the applied business data are stored respectively to the first memory node set In each memory node (i.e. node server), and by the mark of the user identifier and the applied business data with it is described First memory node set is associated.

When data to be read, read data request is sent to the main section in distributed memory system 103 by application server 102 Point server, the read data request carry the mark of the user identifier and the applied business data.The host node clothes The device reception read data request of being engaged in is determined according to the mark of the user identifier of request carrying and the applied business data It is stored with the second memory node set of the applied business data；According to the mark of the applied business data from described second Each memory node in memory node set reads the more parts of applied business data.When the more parts of applications of reading When the content of business datum is inconsistent, the confidence characteristic of the described more parts applied business data is determined；According to identified The confidence characteristic determines portion target application business datum from the described more parts applied business data；By the mesh Mark applied business data are sent to application server 102, so that application server 102 is according to the target application business datum Corresponding media content is pushed to applications client 101.

In some instances, the data processing method that the application proposes can be applied to the master in distributed memory system 103 Node server.As shown in Fig. 2, this method include mainly include two parts handle: one, write data processing (including step 201 and 202), two, reading data processing (including step 203~208).In some instances, this method includes reading data processing (including step Rapid 203~208), in such cases, host node server can be merely responsible for the processing of read data request.It, should in other examples Method includes writing data processing and reading data processing (including step 201~208), and in such cases, host node server was both born The processing of read data request is also responsible in the processing for blaming write data requests.The specific processing of each step is as follows:

Step 201: receive the write data requests from application server, write data request carry user identifier and its Associated applied business data to be stored.

Here, the write data requests from application server are received, applied business data are written, so as to application server The applied business data (such as membership information, authority information of applied business etc.) are read from distributed memory system, according to The applied business data provide online service, for example push corresponding media content (such as video server root to applications client According to video membership information or authority information etc. to videoconference client advertisement or video program etc.).

Step 202: the applied business data are stored into each memory node into the first memory node set respectively, And it is the mark of the user identifier and the applied business data is associated with the first memory node set.

In some instances, wherein stored respectively by the applied business data into the first memory node set Before each memory node, the method further includes: the applied business data are stored to the cache server simultaneously Obtain storage address；Wherein, described each to be deposited what the applied business data stored respectively into the first memory node set Store up node, comprising: each memory node in Xiang Suoshu the first memory node set is sent respectively carries the storage address Write data requests, so that each memory node obtains the applied business number from the cache server according to the storage address According to.

Here, after configuring the cache server, the host node server in distributed memory system is in response to described After write data requests, first above-mentioned applied business data can be stored to cache server and obtain storage address, answered by described in Information, which is completed, with write operation of the business datum in the cache server is sent to application server；Again asynchronously to described Each memory node in first memory node set sends the write data requests for carrying the storage address respectively, so that each Memory node obtains the applied business data from the cache server according to the storage address, to store the applied business Data obtain more parts of Backup Datas of the applied business data.By configuring the cache server, by the applied business number Information is completed according to the write operation in the cache server and is sent to application server, to inform the application server application Business datum has stored, and it is larger can further to solve the problems, such as that data write operation is delayed, thus accelerate the write operation of data, The runnability of lifting system.

As previously mentioned, distributed memory system includes at least one node server, here, in distributed memory system It, can be standby by any more parts of the storage of the applied business data of the application server after host node server receives write data requests Part, this depends on the performance of the demand and distributed memory system itself to data reliability.For example, industry can be applied by described Business data store to each of distributed memory system storage server, the applied business data can also be stored to Part storage server in distributed memory system.

In some instances, described to be requested in response to write data, the applied business data are stored respectively to the Each memory node in one memory node set, comprising: write data request is sent to the first memory node collection Each memory node in conjunction；The response that one or more memory nodes are sent in the first memory node set is received to disappear Breath；The applied business data are stored respectively to the one or more memory nodes for sending response message.Here, described Distributed memory system does not use consistency algorithm when writing data to guarantee the multiple backups stored on multiple memory nodes Consistency, the efficiency for writing data can be improved.Such as: when receiving the response message that any memory node is sent, then can incite somebody to action The applied business data storage is into the memory node.

(data A is updated to specifically, application server sends a more new data to the distributed memory system Data B) operation (abbreviation data writing operation), then host node server receives the data writing operation, and to distributed storage system The server of each memory node in system initiates the operation of more new data.The server of any memory node can be to host node service Device sends response message, to indicate that the server of the memory node can be updated the operation of data.Using such as PAXOS In the case where the consistency algorithms such as algorithm, the server of host node server and each memory node is passed based on the mode of message transmission Write data requests and response message are passed, is responded when the server for the memory node for being more than half is sent to host node server Message, then it represents that data A can be updated to data B, then data B can be written into each memory node.This process can be with Memory node quantity increases the more complicated of change, to cause communication delay, further results in data and writes delay.In contrast, Consistency algorithm is not used in present application example when writing data, to guarantee the consistency of multiple backups, so as to substantially reduce The time delay for writing data greatly promotes the maximum writing speed that the server of each memory node can bear, improves system performance.

Here, the data write operation enormously simplifies distribution and deposits without considering data consistency and stationary problem The deployment and realization of storage system long-distance disaster, to further increase the reliability of the applied business data.

Step 203: receiving the read data request from the application server, wherein the read data request, which carries, to be used The mark of family mark and the applied business data.

As previously mentioned, application server registers customers as the applied business data such as information, membership information, goods orders information It stores in distributed memory system, when user sends media content access request to application server by applications client When, application server is pushed to the media content of the user in order to determine, need to read the user's from distributed memory system Applied business data, to determine the information of the user.Application server then sends read data request, institute to distributed memory system State the mark that read data request carries user identifier and the applied business data.

For example, application server is pushed to the homepage letter of applications client in order to determine when user logs in applications client Breath, need to determine the information of the user, if whether the user is member, which is recorded using the browsing of applications client etc., with Orientation, accurately push corresponding contents give the user.Or application server is answered in the corresponding video of push to this to determine Before client, if bumper advertisements before pushing to the applications client need to send to distributed memory system and read data Request, can be in order to avoid advertisement, the then application if the user is member to read the information of the user from distributed memory system Server does not push preceding bumper advertisements only to applications client pushing video.

Step 204: according to the mark of the user identifier and the applied business data, determination is stored with described using industry Second memory node set of data of being engaged in.

Here, related with configuration variation of the system to distributed storage node, the first memory node set and the second storage Node set can be the same or different, and the first memory node set and the second memory node set may have intersection may also There is no intersection.Such as: when application server 1 writes data, according to configuration at that time, the possible self-application in future of host node server The applied business data of server 1 are written in its corresponding memory node set (such as comprising node 1 and 2), and subsequent system System may change the corresponding memory node set of application server 1, such as: it is changed to the memory node collection comprising node 1 and 3 It closes, it is also possible to be changed to the memory node set comprising node 3 and 4.For another example: because the applied business data are stored in for a long time The factors such as each memory node in the first memory node set may cause depositing in the first memory node set It stores up node failure (i.e. the memory node no longer stores the applied business data), then in the first memory node set not The memory node of failure belongs to the second memory node set.In this way, storing its application when application server 1 reads data The memory node of business datum may change, and the second above-mentioned memory node set and the first memory node set may It is different.When distributed memory system is in response to the read data request, according to the user identifier and the applied business data Mark, need to determine all memory nodes not failed for being stored with the applied business data, i.e. the second memory node set, With from each memory node in the second memory node set read more parts it is described using data.It should be noted that this Text the first memory node set and the second memory node set are not a kind of name of fixation, but in order to refer to clear use " first " and " second " represents the order of different memory node set occurred in an example, first in the same example A kind of memory node set of secondary appearance is referred to as the first memory node set, another memory node set quilt occurred later Referred to as the second memory node set.

Step 205: according to the mark of the applied business data from each storage in the second memory node set Node reads the more parts of applied business data.

As previously mentioned, the applied business data are stored more parts of backups by distributed memory system, and when reading data, distribution Formula storage system reads out the backup of all applied business data, to avoid because the applied business data are in some section Point server is not written accurately, and leads to read inaccuracy (for example, the applied business number read at this time from the node server According to not being newest), to further increase the availability and accuracy of the applied business data.

Step 206: when the content of the more parts of applied business data of reading is inconsistent, determine described more parts described in answer With the confidence characteristic of business datum.

Here, the confidence characteristic of the described more parts applied business data of the determination, can be to described in described more parts Applied business data are compared, to determine the different application data of multiple groups；Or the applied business data are solved Analysis, obtains the feature field in the applied business data, and this feature field is used to indicate the characteristic attribute of user；Or determine institute That states applied business data writes timestamp, with according to the confidence characteristic of the described more parts applied business data, from described more A target application business datum is determined in part applied business data.

Step 207: according to the identified confidence characteristic, portion mesh is determined from the more parts of applied business data Mark applied business data.

In some instances, the confidence characteristic of the described more parts applied business data, comprising: with identical content The number of applied business data；Wherein, the confidence characteristic of the described more parts applied business data of the determination, comprising: will The more parts of applied business data are divided into multiple groups, wherein the content for belonging to same group of applied business data is identical, no With the corresponding different content of group；And determine the number of each group of applied business data for including；Wherein, described from the more parts of institutes It states and determines portion target application business datum in applied business data, comprising: determine comprising most number applied business data Group；And using a applied business data in the identified group as the target application business datum.

Here, the main section when the more parts of applied business data of reading occur inconsistent, in distributed memory system Point server can arbitrate the more parts of applied business data using majority logic algorithm, determine target application business Data.For example, host node server has read 3 backup numbers of the applied business data from the distributed memory system According to wherein 2 Backup Datas are consistent, 2 Backup Datas of another Backup Data and this are inconsistent, then according to majority logic algorithm (i.e. the minority is subordinate to the majority), then using a copy of it of above-mentioned 2 Backup Datas as target application business datum.

In some instances, the confidence characteristic of the described more parts applied business data, comprising: with identical content The confidence weight value of applied business data；The method further includes: weight coefficient is configured to each memory node in advance； Wherein, the confidence characteristic of the described more parts applied business data of the determination, comprising: by the more parts of applied business Data are divided into multiple groups, wherein the content for belonging to same group of applied business data is identical, the corresponding different content of difference group； And for each group, each part applied business data for including according to the group from memory node weight coefficient, determining should The confidence score of group applied business data；Wherein, described that portion target is determined from the described more parts applied business data Applied business data, comprising: determine maximum group of confidence score；And by a applied business data in the identified group As the target application business datum.

Here, distributed memory system can also assist arbitrating to majority logic algorithm increase weight, example Such as, weight coefficient can be configured to each node server in distributed memory system in advance, such as the node serve of better performances Device configures higher weight coefficient, and host node server has read the applied business data from the distributed memory system 6 Backup Datas, and from distributed memory system node server 1,2 read Backup Data 1,2 it is identical, node clothes Business device 1,2 better performances, so that weight coefficient 3 is respectively configured to node server 1,2 in advance, from distributed memory system The Backup Data 3~6 that node server 3~6 is read is identical, and 3~6 performance of node server is general, therefore gives node server 3 ~6 configure weight coefficient 1, and therefore, total confidence level of Backup Data 1,2 is 6 (i.e. weight+node serves of node server 1 The weight of device 2), and total confidence level of Backup Data 3~6 be 4 (i.e. the weight of weight+node server 4 of node server 3+ The weight of weight+node server 6 of node server 5), according to the majority rule of majority logic algorithm, it should Using a copy of it in Backup Data 1,2 as target application business datum.

In some instances, the confidence characteristic of the more parts of applied business data, comprising: every part of applied business number According to write timestamp；Wherein, the confidence characteristic of the determination more parts of applied business data, comprising: determine every part described The applied business data write timestamp；Wherein, described that portion target application is determined from the more parts of applied business data Business datum, comprising: the earliest applied business data of timestamp are write in determination；And by the identified applied business data as institute State target application business datum.

Here, the main section when the more parts of applied business data of reading occur inconsistent, in distributed memory system Point server can also the timestamp the new more priority principle arbitrates the more parts of applied business data using writing, with determination Target application business datum out.Because data are attached with the timestamp of write-in when being written, when the application server is from distribution When the multiple Backup Datas read in storage system are inconsistent, the timestamp the new more priority principle according to writing, and will write timestamp Newest Backup Data is as target application business datum.

In some instances, the confidence characteristic of the more parts of applied business data, comprising: the applied business data packet The value of the feature field related with confidence level contained；The confidence characteristic of the determination more parts of applied business data, comprising: Applied business data described in every part are parsed, obtain it includes the feature field；Determine this part of applied business data The value for the feature field for including；Wherein, described that portion target application business is determined from the more parts of applied business data Data, comprising: determine the value for representing the feature field of highest confidence level；And it will be represented described in highest confidence level comprising this A applied business data of the value of feature field are as the target application business datum.

Here, the main section when the more parts of applied business data of reading occur inconsistent, in distributed memory system Point server can also arbitrate the more parts of applied business data using customized tickets method, determine target application Business datum, i.e., to parsing using business datum, obtain it includes a feature field, such as member's validity period, preferential The feature fields such as the number of the deadline fields such as the validity period of certificate and viewing certificate, and using the field as bill, it is true according to the bill The target application business datum of the fixed user.

For example, the membership information of the user, distributed storage are read in application server request from distributed memory system Host node server in system parses the more parts of membership informations read from each memory node, obtains membership information In a validity period field, and as arbitration field, when there are multiple and different validity period fields, determine it is maximum effectively The maximum a membership information of the value of validity period field is determined as target members information by the value of phase field, that is, thinking will be effective Phase longest overdue membership information confidence level highest and as target application business datum the latest in other words.For another example, Above-mentioned host node server can also parse the more parts of membership informations read from each memory node, obtain member's letter A viewing certificate number field in breath, and as arbitration field, when there are multiple and different viewing certificate number fields, determine The maximum a membership information of the value of viewing certificate number field is determined as target members by the value of maximum viewing certificate number field Information thinks to indicate the membership information confidence level highest of most viewing certificates and as target application business datum.

In some instances, the confidence characteristic of the more parts of applied business data includes: the application with identical content What the number of business datum, the applied business data included write timestamp and the applied business data include and confidence Spend the value of related feature field, wherein corresponding first weight coefficient of the number, it is described to write the corresponding second weight system of timestamp Number, the value of the feature field correspond to third weight coefficient；Wherein, the confidence level of the determination more parts of applied business data Feature, comprising: the more parts of applied business data are divided into multiple set, wherein belong to the applied business data of identity set Content it is identical, different sets correspond to different contents；Determine each number for gathering the applied business data for including；It determines every Part applied business data write timestamp；And applied business data described in every part are parsed, obtain it includes institute Feature field is stated, determines the value for the feature field that this part of applied business data include；Wherein, described to be applied from described more parts Portion target application business datum is determined in business datum, comprising: determine first group comprising most number applied business data； Determine to include second group for writing the earliest applied business data of timestamp；Determine the feature field for representing highest confidence level The third group of the applied business data of value and the value comprising the feature field for representing highest confidence level；To described first First weight coefficient described in every part of applied business data configuration in group, to every part of applied business data configuration in this second group Second weight coefficient, and the third weight coefficient described in every part of applied business data configuration in the third group；According to institute State the first weight coefficient, second weight coefficient and the third weight coefficient, calculate described first group, it is described second group and The confidence score of each part applied business data in the third group；And maximum confidence is scored corresponding applied business data As the target application business datum.

In some instances, to the first weight coefficient described in every part of applied business data configuration in described first group, The second weight coefficient described in every part of applied business data configuration in this second group, and industry is applied to every part in the third group It is engaged in front of third weight coefficient described in data configuration, the method further includes: when determining a applied business data belong to In described first group, it is second group and when the third group described, using this part of applied business data as the target application business Data.

Here, the host node server in distributed memory system can also using majority logic algorithm, to write timestamp excellent Any combination of first principle and customized tickets method determines the target application business datum.For example, simultaneously using big number Logical algorithm writes timestamp priority principle and customized tickets method determines target application business datum, is calculated using majority logic Method determines first group (such as first group includes 5 parts of applied business data) comprising most number applied business data, when using writing Between stamp priority principle determine that (such as second group comprising 2 parts using industry comprising write the earliest applied business data of timestamp second group Business data), use third group (such as third of the determining applied business data earliest comprising member's validity period of customized tickets method Group includes 2 parts of applied business data), to the first weight coefficient described in every part of applied business data configuration in described first group (such as 2), the second weight coefficient (such as 2) described in every part of applied business data configuration in this second group, and in the third group Every part of applied business data configuration described in third weight coefficient (such as 3)；Then every part of applied business data in first and second group Confidence score is 2, and the confidence score of every part of applied business data in third group is 3, if a applied business data are same When in first group and second group or second group and third group or first group, second group and third group etc., then this applies industry The confidence score of business number is the sum of each group weight coefficient, and maximum confidence is scored corresponding applied business data as described in Target application business datum.If a applied business data belong to first group, second group and third group simultaneously, which applies industry The confidence score highest for data of being engaged in, using this part of applied business data as target application business datum.

Step 208: target application business datum is sent to the application server.

Here, target application business datum is sent to application server by distributed memory system, so that application server Corresponding contents are sent to applications client according to target application business datum.

For example, when user sends media content access request to application server by applications client, applications client In for realizing promotional content (such as advertisement) push software development kit (SDK) such as advertisement SDK to application server transmission push away Wide content push request (such as advertisement exposure request), when the target application business datum indicates that the user is permission user, and The permission time limit of the user, then application server was not responding to the promotional content push request, i.e., not by institute also before the deadline It states promotional content and pushes to applications client, only in response to the media content access request, the media content is sent to and is answered With client, applications client shows the media content, and does not show the promotional content.

Through the above technical solutions, the complexity for reading data can be reduced, the accuracy for reading data is improved, is enormously simplified The deployment and realization of distributed memory system long-distance disaster, to further increase the reliability of data.For example, by using upper Technical solution is stated, for the reading data service of membership information, when read data request rate is higher, the accuracy of the data of reading reaches To 99.99999%, when read data request rate is lower, the accuracy of the data of reading is even up to 99.9999999%.

In some instances, the distributed memory system includes multiple host node servers, for example, as shown in figure 3, main Node server 1, host node server 2 and host node server 3, above-mentioned multiple host node servers can be located at different domains In the domain name of i.e. each host node server (different).Multiple application servers can write number to distributed memory system initiation is multiple According to request (i.e. as shown in Figure 3 writes service 1, writes service 2 and write service 3) and multiple read data requests, when an application service When device initiates write data requests or read data request to distributed memory system, it can choose with the application server with identical Network operator and from the closer host node server in the region of the application server, for example, service is write in the initiation of application server 1 When 1, the region where application server 1 is city of Jiangsu province, and the broadband that application server 1 uses is Netcom broadband, then answers It can choose Netcom's host node server of East China, for example, host node server 1 with server 1, data will be write and asked The applied business data asked store each memory node into the first memory node set respectively, as shown in figure 3, node takes Business device 1, node server 2 and node server 3.

Application server to distributed memory system initiate read data request when, select host node server method with it is upper State selects the method for host node server identical when writing service, i.e. selection nearby has identical network operation with the application server The host node server of quotient sends to host node server 1 and reads for example, application server 1 selects host node server 1 nearby Request of data, host node server 1 is in response to the read data request, according to the user identifier and the applied business data Mark, determine and be stored with the second memory node set of the applied business data, here, the second memory node set It is identical as the node server that the first memory node set includes, it is node server 1, node server 2 and node Server 3；Host node server 1 is according to the marks of the applied business data from each in the second memory node set Memory node reads more parts of applied business data；When the content of the more parts of applied business data of reading is inconsistent, main section Point server 1 determines the confidence characteristic of the more parts of applied business data according to its arbitration algorithm, and according to identified institute Confidence characteristic is stated, portion target application business datum is determined from the more parts of applied business data, by the target application Business datum is sent to the application server 1.

Fig. 4 shows a kind of interaction figure of data processing method of the application proposition.In this example, this method is applied to In distributed memory system, and the interactive process is related to four node servers in distributed memory system, respectively based on save Point server, node server 1, node server 2 and node server 3.By taking promotional content is advertisement as an example, applications client For video APP, application server is video server, wherein video server can be to video APP pushing video and advertisement, such as Shown in figure, this method comprises the following steps:

Data write process: step 401~step 405

Step 401: user completes registration on video APP and operates and complete membership buying operation.

Step 402: video APP sends write data requests to video server, and what is generated in the request storage above process answers With business datum, i.e. user's registration information and membership information.

Step 403: video server establishes the user's registration information and member's letter of user according to the data storage request Breath, such as Member Name, member's validity period；If user has had membership information on video server, video server changes Write the validity period of the member.

Step 404: host node server of the video server into distributed memory system sends write data requests, described Write data requests carry user identifier and associated applied business data to be stored.

Step 405: host node server in response to write data request, by the applied business data store respectively to Each memory node in first memory node set, and by the mark and institute of the user identifier and the applied business data It is associated to state the first memory node set.Here, each memory node in the first memory node set be node server 1~ 3.In some instances, the distributed memory system can be realized based on redis 2.6.

Reading data course: step 406~step 411

Step 406: video app sends video content acquisition request, while the advertisement in video app to video server SDK sends advertisement pushing request to video server, to obtain preceding bumper advertisements.

Step 407: video server requests generation to read data and takes request according to above-mentioned advertisement pushing, wherein the reading The mark of user identifier and the applied business data is carried according to request, and the read data request is sent to host node service Device.

Step 408: the host node server in distributed memory system is in response to the read data request, according to the use The mark of family mark and the applied business data, determines the second memory node set for being stored with the applied business data, Wherein, the second memory node set may be different from the first memory node set, and in this example, described first deposits It is identical with the second memory node set to store up node set, is node server 1~3.

Step 409: store each node servers of the applied business data according to the marks of the applied business data from Each memory node reads the applied business data of the user.

Step 410: when the content of more parts of applied business data of reading is inconsistent, host node server is calculated according to arbitration Method determines target application business datum, and determining target application business datum is sent to video server.As previously mentioned, institute Stating arbitration algorithm can be majority logic algorithm, write more new timestamp more priority principle and customized tickets method and its any group It closes.

Step 411: when the target application business datum indicates that the user is member, and the member of the user also exists in the time limit In validity period, then video server is not responding to the advertisement pushing request, i.e., not to the client push advertisement, only to described Client sends video content.

Step 412: video APP shows the video content.

Based on the method that examples detailed above provides, the application also proposed a kind of data processing equipment 500.As shown in figure 5, institute Stating device 500 includes:

Receiving module 501, receives the read data request from application server, and the read data request carries user identifier With the mark of applied business data.

Read module 503, in response to the read data request, according to the user identifier and the applied business data Mark determines the first memory node set for being stored with the applied business data；According to the mark of the applied business data More parts of applied business data are read from each memory node in the first memory node set.

Determining module 504 determines that described more parts are answered when the content of the more parts of applied business data of reading is inconsistent With the confidence characteristic of business datum；According to the identified confidence characteristic, from the more parts of applied business data really Fixed portion target application business datum.

The target application business datum is sent to the application server by sending module 505.

In some instances, described device 500 further include: memory module 502.Wherein, the receiving module 501 is further The write data requests from the application server are received, write data request carries user identifier and associated wait store Applied business data；And the memory module 502 is requested in response to write data, and the applied business data are deposited respectively Store up each memory node into the second memory node set, and the user identifier that write data request is carried and institute The mark for stating applied business data is associated with the second memory node set.

In some instances, described device 500 further include: cache module 506 stores the applied business data to institute It states cache server and obtains storage address；Wherein, the memory module 502 stores the applied business data respectively to Each memory node in two memory node set, comprising: each memory node point in Xiang Suoshu the second memory node set The write data requests of the storage address Fa Song not be carried, so that each memory node is according to the storage address from the caching Server obtains the applied business data.

In some instances, the memory module 502 is requested in response to write data, by the applied business data point Each memory node into the second memory node set is not stored, comprising: write data request is sent to described first Each memory node in memory node set；The receiving module 501 further receives in the second memory node set The response message that one or more memory nodes are sent；The memory module 502 further distinguishes the applied business data It stores to the one or more memory nodes for sending response message.

In some instances, the confidence characteristic of the more parts of applied business data, comprising: the application with identical content The number of business datum；Wherein, the determining module 504 determines the confidence characteristic of the more parts of applied business data, comprising: The more parts of applied business data are divided into multiple groups, wherein the content for belonging to same group of applied business data is identical, different The corresponding different content of group；And determine the number of each group of applied business data for including；Wherein, the determining module 504 from Portion target application business datum is determined in the more parts of applied business data, comprising: determine to include most number applied business The group of data；And using a applied business data in the identified group as the target application business datum.

In some instances, the confidence characteristic of the more parts of applied business data, comprising: the application with identical content The confidence weight value of business datum；Described device 500 further include: configuration module 507 configures each memory node weigh in advance Weight coefficient；Wherein, the determining module determines the confidence characteristic of the more parts of applied business data, comprising: by described more parts Applied business data are divided into multiple groups, wherein the content for belonging to same group of applied business data is identical, and difference group is corresponding different Content；And for each group, each part applied business data for including according to the group from memory node weight coefficient, Determine the confidence score of this group of applied business data；Wherein, the determining module is true from the more parts of applied business data Fixed portion target application business datum, comprising: determine maximum group of confidence score；And the portion in the identified group is answered Use business datum as the target application business datum.

In some instances, the confidence characteristic of the more parts of applied business data, comprising: every part of applied business number According to write timestamp；Wherein, the determining module 504 determines the confidence characteristic of the more parts of applied business data, comprising: really Fixed every part of applied business data write timestamp；Wherein, the determining module 504 is from the more parts of applied business numbers According to middle determining a target application business datum, comprising: the earliest applied business data of timestamp are write in determination；And it will be identified The applied business data are as the target application business datum.

In some instances, the confidence characteristic of the more parts of applied business data, comprising: the applied business data packet The value of the feature field related with confidence level contained；Wherein, the determining module 504 determines the more parts of applied business data Confidence characteristic, comprising: applied business data described in every part are parsed, obtain it includes the feature field；It determines The value for the feature field that this part of applied business data include；Wherein, the determining module 504 is from the more parts of applied business Portion target application business datum is determined in data, comprising: determine the value for representing the feature field of highest confidence level；And it will A applied business data of value comprising the feature field for representing highest confidence level are as the target application business Data.

In some instances, the confidence characteristic of the more parts of applied business data includes: the application with identical content What the number of business datum, the applied business data included write timestamp and the applied business data include and confidence Spend the value of related feature field, wherein corresponding first weight coefficient of the number, it is described to write the corresponding second weight system of timestamp Number, the value of the feature field correspond to third weight coefficient；Wherein, the determining module 504 determines the more parts of applied business The confidence characteristic of data, comprising: the more parts of applied business data are divided into multiple set, wherein belong to identity set The content of applied business data is identical, and different sets correspond to different contents；Determine the applied business data that each set includes Number；Determine every part of applied business data writes timestamp；And applied business data described in every part are parsed, it obtains To it includes the feature field, determine the value for the feature field that this part of applied business data include；Wherein, described true Cover half block 504 determines portion target application business datum from the more parts of applied business data, comprising: determines to include most parts First group of number applied business data；Determine to include second group for writing the earliest applied business data of timestamp；It determines and represents most The application industry of the value of the feature field of high confidence level and the value comprising the feature field for representing highest confidence level The third group for data of being engaged in；The configuration module 507 is to the first power described in every part of applied business data configuration in described first group Weight coefficient, the second weight coefficient described in every part of applied business data configuration in this second group, and to every in the third group Third weight coefficient described in part applied business data configuration；The determining module 504 further according to first weight coefficient, Second weight coefficient and the third weight coefficient, calculate described first group, it is each in second group and the third group The confidence score of part applied business data；And maximum confidence is scored into corresponding applied business data as the target and is answered Use business datum.

In some instances, in the configuration module 507 to every part of applied business data configuration institute in described first group The first weight coefficient is stated, the second weight coefficient described in every part of applied business data configuration in this second group, and to the third Before third weight coefficient described in every part of applied business data configuration in group, the determining module 504 is further used for: when true Fixed portion applied business data belong to described first group, it is second group and when the third group described, by this part of applied business number According to as the target application business datum.

The specific implementation principle of above-mentioned each functions of modules has been described above, and which is not described herein again.

In addition, the data processing method and data processing equipment and each module therein in each example of the application can To be integrated in a processing unit, it is also possible to modules and physically exists alone, it can also be with two or more devices Or module is integrated in one unit.Above-mentioned integrated unit both can take the form of hardware realization, can also use software The form of functional unit is realized.

Fig. 6 shows the composite structural diagram of the calculating equipment 600 where data processing equipment 500.As shown in fig. 6, the meter Calculating equipment includes one or more processor (CPU) 602, memory 604, user interface 606, and for interconnecting these groups The communication bus 608 of part.

User interface 606 includes one or more output equipments 612 comprising one or more speakers and/or one Or multiple visual displays.User interface 610 also includes one or more input equipments 614 comprising such as, keyboard, mouse Mark, voice command input unit or loudspeaker, touch screen displays, touch sensitive tablet, posture capture camera or other inputs are pressed Button or control etc..

Memory 604 can be high-speed random access memory, such as DRAM, SRAM, DDR RAM or other deposit at random Take solid storage device；Or nonvolatile memory, such as one or more disk storage equipments, optical disc memory apparatus, sudden strain of a muscle Deposit equipment or other non-volatile solid-state memory devices.

The executable instruction set of 604 storage processor 602 of memory, comprising:

Operating system 616, including the program for handling various basic system services and for executing hardware dependent tasks；

Using 618, including the various application programs for data processing, this application program can be realized above-mentioned each example In process flow, for example may include unit some or all of in data processing equipment 500 shown in fig. 5.Each module 501-507 can store machine-executable instruction.The machine that processor 602 passes through each module 501-507 in execution memory 606 Device executable instruction, and then can be realized the function of above-mentioned each module 501-507.

It should be noted that step and module not all in above-mentioned each process and each structure chart be all it is necessary, can To ignore certain steps or module according to the actual needs.Each step execution sequence be not it is fixed, can according to need into Row adjustment.The division of each module is intended merely to facilitate the division functionally that description uses, and in actual implementation, a module can It is realized with point by multiple modules, the function of multiple modules can also be realized by the same module, these modules can be located at same In a equipment, it can also be located in different equipment.

Hardware module in each example can in hardware or hardware platform adds the mode of software to realize.Above-mentioned software package Machine readable instructions are included, are stored in non-volatile memory medium.Therefore, each example can also be presented as software product.For example, Corresponding above-mentioned data processing method and device, the example of the application additionally provide a kind of computer readable storage medium, deposit thereon Contain computer instruction, wherein the step of computer instruction realizes above-mentioned data processing method when being executed by processor.

In each example, hardware can be by special hardware or the hardware realization of execution machine readable instructions.For example, hardware can Think the permanent circuit specially designed or logical device (such as application specific processor, such as FPGA or ASIC) for completing specifically to grasp Make.Hardware also may include programmable logic device or circuit by software provisional configuration (as included general processor or other Programmable processor) for executing specific operation.

In addition, each example of the application can pass through the data processor by data processing equipment such as computer execution To realize.Obviously, data processor constitutes the application.In addition, being commonly stored data processing in one storage medium Program is by directly reading out storage medium or the storage by program being installed or being copied to data processing equipment for program It is executed in equipment (such as hard disk and/or memory).Therefore, such storage medium also constitutes the application, present invention also provides A kind of non-volatile memory medium, wherein being stored with data processor, this data processor can be used for executing the application Any one of above method example example.

The corresponding machine readable instructions of module in Fig. 5 can be such that operating system operated on computer etc. completes here The some or all of operation of description.Non-volatile computer readable storage medium storing program for executing can be in the expansion board in insertion computer In set memory or write the memory being arranged in the expanding element being connected to a computer.Be mounted on expansion board or CPU on person's expanding element etc. can be according to instruction execution part and whole practical operations.

It, can also be in addition, the device and each module in each example of the application can integrate in one processing unit It is that modules physically exist alone, can also be integrated in one unit with two or more devices or module.Above-mentioned collection At unit both can take the form of hardware realization, can also realize in the form of software functional units.

The foregoing is merely the preferred embodiments of the application, all in spirit herein not to limit the application Within principle, any modification, equivalent substitution, improvement and etc. done be should be included within the scope of the application protection.

Claims

1. a kind of data processing method, which is characterized in that the described method includes:

Read data request of the reception from application server, the read data request carrying user identifier and applied business data Mark；

According to the mark of the user identifier and the applied business data, determination is stored with the first of the applied business data Memory node set；

More parts are read from each memory node in the first memory node set according to the mark of the applied business data Applied business data；

When the content of the more parts of applied business data of reading is inconsistent, the confidence of the more parts of applied business data is determined Spend feature；

According to the identified confidence characteristic, portion target application business number is determined from the more parts of applied business data According to；

The target application business datum is sent to the application server.

2. according to the method described in claim 1, further comprising:

The write data requests from application server are received, write data request carries user identifier and associated wait store Applied business data；

The applied business data are stored into each memory node into the second memory node set respectively, and by the user The mark of mark and the applied business data is associated with the second memory node set.

3. according to the method described in claim 2, wherein, being stored respectively by the applied business data to the second memory node Before each memory node in set, the method further includes:

The applied business data are stored to cache server and obtain storage address；

Wherein, described that the applied business data are stored to each memory node into the second memory node set, packet respectively It includes:

Each memory node into the second memory node set is sent respectively to be carried the data of writing of the storage address and asks It asks, so that each memory node obtains the applied business data from the cache server according to the storage address.

4. according to the method described in claim 2, wherein, being stored respectively when by the applied business data to the second memory node When each memory node in set, consistency algorithm is not used to guarantee the more parts of applications stored in each memory node Consistency between business datum.

5. according to the method described in claim 1, wherein, the confidence characteristic of the more parts of applied business data, comprising: have The number of the applied business data of identical content；

Wherein, the confidence characteristic of the determination more parts of applied business data, comprising:

The more parts of applied business data are divided into multiple groups, wherein the content for belonging to same group of applied business data is identical, The corresponding different content of difference group；And

Determine the number for the applied business data that each group includes；

Wherein, the confidence characteristic described according to determined by determines portion target from the more parts of applied business data Applied business data, comprising:

Determine the group comprising most number applied business data；And

Using a applied business data in the identified group as the target application business datum.

6. according to the method described in claim 1, wherein, the confidence characteristic of the more parts of applied business data, comprising: have The confidence weight value of the applied business data of identical content；

The method further includes:

Weight coefficient is configured to each memory node in advance；

For each group, each part applied business data for including according to the group from memory node weight coefficient, determine The confidence score of this group of applied business data；

Determine maximum group of confidence score；And

7. according to the method described in claim 1, wherein, the confidence characteristic of the more parts of applied business data, comprising: every part Applied business data write timestamp；

Determine every part of applied business data writes timestamp；

The earliest applied business data of timestamp are write in determination；And

By the identified applied business data as the target application business datum.

8. according to the method described in claim 1, wherein, the confidence characteristic of the more parts of applied business data, comprising: described The value for the feature field related with confidence level that applied business data include；

Every part of applied business data are parsed, obtain it includes the feature field；And

Determine the value for the feature field that this part of applied business data include；

Determine the value for representing the feature field of highest confidence level；And

A applied business data for the value that the feature field of highest confidence level is represented comprising this are answered as the target Use business datum.

9. according to the method described in claim 1, wherein, the confidence characteristic of the more parts of applied business data includes: to have What the number of the applied business data of identical content, the applied business data included writes timestamp and the applied business number According to comprising feature field related with confidence level value, wherein corresponding first weight coefficient of the number, it is described to write timestamp Corresponding second weight coefficient, the value of the feature field correspond to third weight coefficient；

The more parts of applied business data are divided into multiple set, wherein belong to the content of the applied business data of identity set Identical, different sets correspond to different contents；

Determine each number for gathering the applied business data for including；

Determine every part of applied business data writes timestamp；And

Applied business data described in every part are parsed, obtain it includes the feature field, determine this part of applied business The value for the feature field that data include；

Determine first group comprising most number applied business data；

Determine to include second group for writing the earliest applied business data of timestamp；

It determines the value for the feature field for representing highest confidence level and represents the tagged word of highest confidence level comprising this The third group of the applied business data of the value of section；

To the first weight coefficient described in every part of applied business data configuration in described first group, every part in this second group is answered Second weight coefficient is configured with business datum, and the third described in every part of applied business data configuration in the third group is weighed Weight coefficient；

According to first weight coefficient, second weight coefficient and the third weight coefficient, first group, the institute are calculated State the confidence score of each part applied business data in second group and the third group；And

Maximum confidence is scored into corresponding applied business data as the target application business datum.

10. according to the method described in claim 9, wherein, to every part of applied business data configuration institute in described first group The first weight coefficient is stated, the second weight coefficient described in every part of applied business data configuration in this second group, and to the third Before third weight coefficient described in every part of applied business data configuration in group, the method further includes:

When determine a applied business data belong to described first group, it is second group and when the third group described, which is answered Use business datum as the target application business datum.

11. a kind of data processing equipment, which is characterized in that described device includes:

Receiving module, receives the read data request from application server, and the read data request carries user identifier and application The mark of business datum；

Read module, according to the mark of the user identifier and the applied business data, determination is stored with the applied business First memory node set of data；According to the mark of the applied business data from each in the first memory node set A memory node reads more parts of applied business data；

Determining module determines the more parts of applied business when the content of the more parts of applied business data of reading is inconsistent The confidence characteristic of data；According to the identified confidence characteristic, determined from the more parts of applied business data a Target application business datum；

The target application business datum is sent to the application server by sending module.

12. device according to claim 11 further comprises memory module；

Wherein, the receiving module further receives the write data requests from the application server, write data request Carry user identifier and associated applied business data to be stored；And

The applied business data are stored each storage section into the second memory node set by the memory module respectively Point, and the mark for the user identifier and the applied business data that write data request is carried and second storage Node set is associated.

13. device according to claim 12, wherein described device further include:

The applied business data are stored to the cache server and obtain storage address by cache module；

Wherein, the applied business data are stored each storage into the second memory node set by the memory module respectively Node, comprising:

14. a kind of storage medium, is stored with computer-readable instruction, at least one processor can be made to execute such as claim 1 To 10 described in any item methods.