CN106980669B

CN106980669B - A kind of storage of data, acquisition methods and device

Info

Publication number: CN106980669B
Application number: CN201710179427.6A
Authority: CN
Inventors: 牟桂贤; 蔡伟星; 黄卫基; 甘俊源
Original assignee: Gree Electric Appliances Inc of Zhuhai
Current assignee: Gree Electric Appliances Inc of Zhuhai
Priority date: 2017-03-23
Filing date: 2017-03-23
Publication date: 2019-07-02
Anticipated expiration: 2037-03-23
Also published as: CN106980669A

Abstract

The invention discloses the storage acquisition methods and device of a kind of data, the storage method of the data includes: to receive data information to be stored, wherein including the identification information of data in the data information；According to the storage algorithm pre-saved in the identification information and data platform of the data, the corresponding destination server of the data information described in data platform is determined；By the data information memory into the destination server.Due in date storage method provided by the invention, it stores data into corresponding destination server, rather than it is stored in each server in data platform simultaneously, avoid the waste of server resource, and it is directed to a variety of different data informations, the destination server of storage may be different, and the pressure for avoiding individual server to a certain extent in this way is excessive, improves data access efficiency.

Description

A kind of storage of data, acquisition methods and device

Technical field

The present invention relates to field of computer technology more particularly to a kind of storages of data, acquisition methods and device.

Background technique

The computer technology industry of rapid development is in addition to changing the every aspect of people's life, promoting social rapid advances Outside, also for enterprise, this main body of the society brings more intuitive and effective influence.As the product business of enterprise is opened up Exhibition, builds a data platform for multiple product business scenario and has a very important significance, also receive enterprise especially The attention of enterprise in computer industry, and the high availability of data platform, service stability, Information Security, management are just Victory is the center of people's growing interest.For the data platform of enterprise-level, by the multiple product business of enterprise's extension How data are stored, and are faster, preferably enterprise product so that the data of multiple product business are managed and be analyzed The user of business provides service, the even more most important thing.

It is usually in the prior art for the two-server in data platform, by number to be stored about the storage of data According to being stored in the two-server simultaneously, wherein a server provides the data writing operation for increasing, deleting, modifying, separately One server provides the reading data manipulation of inquiry, that is, guarantees the read and write abruption of realization data after data storage.For enterprise For the data platform of industry grade, (Peta Byte claps word to usually TB (Trillion Byte, terabyte) grade even PB / thousand terabytes of section) it is superfluous to will result in data using this date storage method since data volume is huge the big data quantity of grade Remaining, server resource waste, furthermore after data storage, a large amount of concurrent operations read data or write data be will cause single The problem of pressure of server is excessive, encounters performance bottleneck reduces the efficiency of data access.

Summary of the invention

The present invention provides storage, acquisition methods and the device of a kind of data, to solve service existing in the prior art The problem of data consistency difference in the device wasting of resources, the low efficiency of data access and multiple servers.

In order to solve the problem above-mentioned, the present invention provides a kind of storage methods of data, this method comprises:

Data information to be stored is received, wherein including the identification information of data information in the data information；

According to the storage algorithm pre-saved in the identification information and data platform of the data information, determine in data The corresponding destination server of data information described in platform；

By the data information memory into the destination server.

Further, the identification information of the data information is Universally Unique Identifier UUID, then described according to the number It is believed that the storage algorithm pre-saved in the identification information and data platform of breath, determines the data information described in data platform Corresponding destination server includes:

The number of servers in the data platform is obtained, modulo operation is carried out to the UUID and number of servers；

According to the modulo operation as a result, and the modulo operation result that pre-saves and every server corresponding relationship, Determine the corresponding destination server of the data information.

Further, described to receive after the data information stored, the method also includes:

Judge whether the corresponding attribute of the data information is master data；

If not, carrying out subsequent operation.

Further, if the corresponding attribute of the data information is master data, by the data information memory to data In every server in platform.

Further, described by institute after the corresponding destination server of determination data information described in data platform Before data information memory is stated into the destination server, the method also includes:

Whether the data type for judging the data information is real time data；

If so, according to the type of theme carried in the data information, the type of theme that pre-saves and its right The fractionation rule for each type of data answered, the data information is split as to correspond to the subdata information of each keyword；

According to the corresponding relationship of each keyword pre-saved and tables of data, determine that each subdata information is corresponding Each tables of data；

It is described to include: into the destination server by the data information memory

By each subdata information of the data information, it each of is respectively stored under the destination server corresponding number According in table.

Further, if it is determined that destination server be at least two, each keyword that the basis pre-saves With the corresponding relationship of tables of data, determine that the corresponding each tables of data of each subdata information includes:

According to the first corresponding relationship of each keyword and leader Leader server that pre-save, determine described each Each Leader server in the corresponding destination server of subdata information；

According to the second corresponding relationship of tables of data in each keyword for pre-saving and Leader server, determine described in Each subdata information corresponding tables of data under each Leader server；

By the storage of each subdata information into corresponding tables of data under its corresponding each Leader server；

According to the data synchronization rules pre-saved, will be stored in corresponding tables of data under each Leader server Subdata information, be synchronized under remaining described destination server in each corresponding tables of data.

The present invention provides a kind of acquisition methods of data, this method comprises:

Data acquisition instruction is received, wherein including the mark letter of data information to be obtained in data acquisition instruction Breath；

According to the acquisition algorithm pre-saved in the identification information and data platform of the data information, determine in data The corresponding destination server of data information described in platform；

The data information is obtained in the destination server.

Further, the identification information of the data information is Universally Unique Identifier UUID, then described according to the number It is believed that the acquisition algorithm pre-saved in the identification information and data platform of breath, determines the data information described in data platform Corresponding destination server includes:

Further, after the reception data acquisition instruction, the method also includes:

If not, carrying out subsequent operation.

Further, any one in data platform if the corresponding data information of the data information is master data The data information is obtained in platform server.

Further, described in institute after the corresponding destination server of determination data information described in data platform It states before obtaining the data information in destination server, the method also includes:

Judge whether the corresponding data type of the identification information of the data information is real time data；

If so, described the data information obtained in the destination server include:

In each tables of data under the destination server, the identification information for obtaining the data information is corresponding each Subdata information；

Determine that the corresponding collection of each subdata information is combined into the data information.

The present invention provides a kind of storage device of data, which includes:

Receiving module, for receiving data information to be stored, wherein the mark in the data information including data is believed Breath；

Determining module, the storage algorithm for being pre-saved in the identification information and data platform according to the data, really It is scheduled on the corresponding destination server of data information described in data platform；

Memory module is used for the data information memory into the destination server.

Further, the determining module, specifically for obtaining the number of servers in the data platform, to data Identification information Universally Unique Identifier UUID and number of servers carry out modulo operation；It is according to the modulo operation as a result, and pre- The corresponding relationship of the modulo operation result and every server that first save, determines the corresponding destination server of the data information.

Further, described device further include:

Judgment module, for judging whether the corresponding attribute of the data information is master data；Mould is determined if not, triggering Block.

Further, the memory module, if being also used to the judgment module judges the corresponding category of the data information Property is master data, by the data information memory into every server in data platform.

Further, described device further include:

Determining module is judged, for judging whether the data type of the data information is real time data；If so, according to The type of theme carried in the data information, the type of theme pre-saved and its corresponding each type of data Rule is split, the data information is split as to correspond to the subdata information of each keyword；According to each pass pre-saved The corresponding relationship of keyword and tables of data determines the corresponding each tables of data of each subdata information；

The memory module, specifically for being respectively stored into the mesh for each subdata information of the data information It marks in each corresponding data table under server.

Further, the judgement determining module, being also used to determining destination server is at least two according to guarantor in advance First corresponding relationship of each keyword and leader Leader server deposited, determines the corresponding institute of each subdata information State each Leader server in destination server；According to data in each keyword and Leader server pre-saved Second corresponding relationship of table determines each subdata information corresponding tables of data under each Leader server；

The memory module, being specifically used for will be under the storage to its corresponding each Leader server of each subdata information In corresponding tables of data；According to the data synchronization rules pre-saved, by corresponding data under each Leader server The subdata information stored in table is synchronized under remaining described destination server in each corresponding tables of data.

The present invention provides a kind of acquisition device of data, which includes:

Receiving module, acquisition instruction for receiving data, wherein including data to be obtained in data acquisition instruction The identification information of information；

Determining module, the acquisition for pre-saving in the identification information and data platform according to the data information are calculated Method determines the corresponding destination server of the data information described in data platform；

Module is obtained, for obtaining the data information in the destination server.

Further, the determining module, specifically for obtaining the number of servers in the data platform, to the number It is believed that the identification information Universally Unique Identifier UUID and number of servers of breath carry out modulo operation；According to the modulo operation knot Fruit, and the modulo operation result and the corresponding relationship of every server that pre-save, determine the corresponding target of the data information Server.

Further, described device further include:

First judgment module, for judging whether the corresponding attribute of the data information is master data；If not, triggering is true Cover half block.

Further, the acquisition module, if being also used to the first judgment module judges that the data information is corresponding Attribute be master data, obtain the data information in any one server in data platform.

Further, described device further include:

Second judgment module, whether the corresponding data type of identification information for judging the data information is real-time number According to；If so, triggering obtains module；

The acquisition module is believed specifically in each tables of data under the destination server, obtaining the data The corresponding each subdata information of the identification information of breath；Determine that the corresponding collection of each subdata information is combined into the data letter Breath.

The present invention provides a kind of storage of data, acquisition methods and device, the storage method of the data includes: to receive Data information to be stored, wherein including the identification information of data in the data information；According to the identification information of the data, And the storage algorithm pre-saved in data platform, determine the corresponding destination server of the data information described in data platform； By the data information memory into the destination server.Due in date storage method provided by the invention, for wait deposit The storage algorithm pre-saved in the corresponding identification information of the data of storage and data platform determines the corresponding target of the data Server, then by data storage into corresponding destination server, rather than be stored in simultaneously each in data platform In platform server, the waste of server resource is avoided, and is directed to a variety of different data informations, the destination server of storage can Can be different, the pressure for avoiding individual server to a certain extent in this way is excessive, improves data access efficiency.

Detailed description of the invention

In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with It obtains other drawings based on these drawings.

Fig. 1 is a kind of schematic diagram of the storing process for data that the embodiment of the present invention 1 provides；

Fig. 2 is a kind of schematic diagram of the acquisition process for data that the embodiment of the present invention 5 provides；

Fig. 3 is a kind of software frame schematic diagram for big data platform that the embodiment of the present invention 9 provides；

Fig. 4 is a kind of structural schematic diagram for big data platform that the embodiment of the present invention 9 provides；

Fig. 5 is a kind of memory device structure schematic diagram of data provided in an embodiment of the present invention；

Fig. 6 is a kind of acquisition device structural schematic diagram of data provided in an embodiment of the present invention.

Specific embodiment

In order to avoid the waste of server resource, and data access efficiency is improved, the embodiment of the invention provides a kind of numbers According to storage, acquisition methods and device.

To make the objectives, technical solutions, and advantages of the present invention clearer, make below in conjunction with the attached drawing present invention into one Step ground detailed description, it is clear that described embodiment is only a part of the embodiments of the present invention, rather than whole implementation Example.Based on the embodiments of the present invention, obtained by those of ordinary skill in the art without making creative efforts Every other embodiment, shall fall within the protection scope of the present invention.

Embodiment 1:

Fig. 1 be a kind of schematic diagram of the storing process of data provided in an embodiment of the present invention, the process the following steps are included:

S101: receiving data information to be stored, wherein including the identification information of data information in the data information.

The storage method of data provided in an embodiment of the present invention is applied to management server, installs in the management server There is the management software for data platform.

It include multiple servers in the data platform, every server may serve to carry out the storage of data, management Member can will be under the data information memory be stored to data platform by the management software installed in management server In server, wherein the configuration of every server can with it is identical can be different, that is to say, that stored in every server Data information can part it is identical, can also be different.

The management server can be used to receive data information to be stored, and include that the data are believed in the data information Corresponding identification information is ceased, in addition, further including the corresponding content information of data information in the data information.The wherein mark Information can be UUID (Universally Unique Identifier, Universally Unique Identifier), be also possible to for the number It is believed that the unique number etc. of breath distribution, as long as data information described in unique identification can be used to.

S102: it according to the storage algorithm pre-saved in the identification information and data platform of the data information, determines The corresponding destination server of data information described in data platform.

Server in the data platform is used to store the data information, in embodiments of the present invention the data letter Which platform destination server breath is especially stored under, is determined by the identification information of the data information.

The storage algorithm pre-saved in the data platform such as can be modulo operation, take quotient's operation at the storage algorithm.Root According to every clothes in the operation result and the operation result and the data platform of the identification information and the storage algorithm The corresponding relationship of business device, can determine the corresponding destination server of the identification information, so that it is determined that the data information Corresponding destination server.

S103: by the data information memory into the destination server.

It is described to can be process of the data information memory into destination server, the data information memory is arrived In tables of data in destination server.

The data information memory can be the content letter by data information into which tables of data in destination server What the keyword in breath determined, it is also possible to the identification information according to the data information, is determined by the algorithm pre-saved 's.

It is described to can be information storage into the destination server, it is communicated by using unified data interaction Come what is completed, can be in the unified data interaction communication modes includes that application layer protocol is for mode and unified data-interface HTTP (HyperText Transfer Protocol, hypertext transfer protocol) or HTTPS (Hyper Text Transfer Secure Socket Layer is added under hypertext transfer protocol in Protocol over Secure Socket Layer), transport layer protocol For TCP/IP (Transmission Control Protocol/Internet Protocol, transmission control protocol/internet Interconnection protocol) or/and data API (Application Programming Interface, application programming interface) etc. Data interaction communication modes, can be in the unified data-interface mode includes unified technical protocol JDBC (Java The connection of Data Base Connectivity, Java database)/WebService, unified data protocol JSON (JavaScript Object Notation, JavaScript object representation)/XML (Extensible Markup Language, extensible markup language) etc. data-interface mode.

For the data platform of enterprise-level, due to usually requiring TB grades of storage even in the data platform of enterprise-level PB grades of big data quantity, data volume is huge, using the storage method of data provided in an embodiment of the present invention, for avoiding server The waste and raising data access efficiency of resource, better effect.

Due in date storage method provided by the invention, for the corresponding identification information of data information to be stored, and The storage algorithm pre-saved in data platform determines the corresponding destination server of the data information, then the data is believed Breath storage is into destination server, according to the identification information of data to be stored information, by data information memory to destination server In, rather than in each server being stored in data platform simultaneously, the waste of server resource is avoided, and for more The destination server of the different data information of kind, storage may be different, avoid individual server to a certain extent in this way Pressure is excessive, improves data access efficiency.

Embodiment 2:

In order to further improve the access efficiency of data, on the basis of the above embodiments, in the embodiment of the present invention, institute The identification information for stating data information is Universally Unique Identifier UUID, then the identification information according to the data information, and The storage algorithm pre-saved in data platform determines the corresponding destination server packet of the data information described in data platform It includes:

Since UUID is as the unique ID of Data for Global, there is global uniqueness, so applying in embodiments of the present invention UUID can after destination server of the data information memory into data platform as the identification information of the data information To guarantee to use the identification information can be with the unique identification data information.

And since UUID is generated by software for the data information, so can be determined by UUID with phase The data information of congenerous demand realizes the logic to data information using functional requirement as the fragment logic of data information Fragment, i.e., by the data information memory with identical function demand into corresponding destination server.During data access By the access pressure dissipation of data into different servers, the pressure for avoiding individual server is excessive, can also be in data When obtaining data information in access process, data information is obtained in the corresponding server of data information to be obtained, is avoided It is obtained in each server in data platform, further improves access efficiency.

In embodiments of the present invention, management server is modulus according to the storage algorithm pre-saved in the data platform Operation, the algorithm pre-saved in the identification information and data platform according to the data information determine destination server Process are as follows: according to the identification information of the data information, number of servers and modulo operation in data platform, to UUID And number of servers carries out modulo operation, according to modulo operation as a result, and the modulo operation result that is pre-saved in data platform With the corresponding relationship of every server, the corresponding destination server of the data information is determined.

The destination server can be one or more, when the destination server is more, can incite somebody to action A wherein at least destination server is as reading destination server, remaining server in addition to this at least a destination server As destination server is write, realizes read and write abruption, further improve the efficiency of data access.

The data platform is distributed data platform, and the build process of the data platform can be by developer Deployment includes the cluster of one of more Relational DBMSs (MySQL) server, using in distributed data base Between part (Mycat) tool build distributed data platform framework.

The embodiment of the present invention is illustrated with a specific embodiment below, that installs in management server is used to take The management software for building distributed data platform framework is Mycat, and the identification information of the data information is UUID, the modulus fortune Calculating is M%N=P, M=u1+u2+u3+ ...+u32, wherein first character, second that described u1, u2, u3 ... and u32 are UUID The ASCII character value of position character, third position character ... and the 32nd character, N are the number of servers in data platform, and P is Modulo operation as a result, preserve the corresponding relationship of modulo operation result Yu every server, according to modulus in the management server Operation result P, the corresponding relationship of the modulo operation result and every server determine the corresponding target clothes of the identification information Business device, by the corresponding destination server of data information described in the destination server.

Match specifically, adding sharding-by-prefixpattern rule in the existing rule.xml file of MyCat It sets, rule configuration is as follows:

<rule>

<algorithm>sharding-by-prefixpattern</algorithm>

</rule>

</tableRule>

< function name=" sharding-by-prefixpattern "

Class=" org.opencloudb.route.function.PartitionByPrefixPattern " >

<property name="mapFile">partition-prefixpattern.txt</property>

</function>

The title name in<tableRule>label including rule is sharding-by-prefixpattern, specifically Regular rule includes that columns is used to identify the literary name section id for fragment, and algorithm is used to identify fragment function i.e. modulus Operation.Title name in<function>label including the function is sharding-by-prefixpattern, affiliated class Class is org.opencloudb.route.function.PartitionByPrefixPattern, and patternValue is used The radix of modulo operation is identified, the radix of the modulo operation is the server in the data platform in embodiments of the present invention Quantity, e.g. 8, prefixLength is used to identify the digit 32 of the ASCII for calculating here, and mapFile is used to identify The path of configuration file is partition-prefixpattern.txt, i.e. configuration file partition- The corresponding relationship of modulo operation result Yu every server is preserved in prefixpattern.txt.

Due in date storage method provided by the invention, for the corresponding identification information of data information to be stored, and The modulo operation pre-saved in data platform determines the corresponding destination server of the data information, then the data is believed Breath storage avoids server into destination server, rather than in each server being stored in data platform simultaneously The waste of resource, and a variety of different data informations are directed to, the destination server of storage may be different, so to a certain extent The pressure for avoiding individual server is excessive, improves data access efficiency.

Embodiment 3:

It is described in the embodiment of the present invention on the basis of the various embodiments described above in order to further increase data access efficiency It receives after the data information stored, the method also includes:

If not, carrying out subsequent operation.

In data platform in embodiments of the present invention, according to the corresponding attribute of data information, data information is divided into master Data and business datum, master data refer in data platform between server for sharing data, business datum be for Indicate the corresponding data of business service of enterprise.Master data is primarily referred to as user information, authority information in embodiments of the present invention Deng other data informations that, business datum is in addition to attribute is master data.For data information to be stored, the number is judged It is believed that ceasing whether corresponding attribute is master data, wherein the corresponding attribute of the data information can carry the mark in data information In information, it is also possible to data information itself and carries its corresponding attribute.

Since the identification information of data information is by Software Create, it is possible to the data information described in Software Create Using the byte of specific length particular number of bits come the corresponding attribute of marking data information when identification information.Such as the data information Identification information be UUID, length is 128 (binary system), when the identification information of the data information described in Software Create, with the 10 are carried out the corresponding attribute of marking data information, for example 1 is master data, and 0 is business datum.

If the corresponding attribute of the data information is master data, and the data information memory is every into data platform In platform server.

By primary data stores such as user information, authority informations into every server in data platform, make host data There is complete backup on every server, realizes that master data is shared between the server in data platform, by business Data are stored into its corresponding destination server, for using when data acquisition.

Management server is receiving after the data information stored, judge the corresponding attribute of the data information whether based on Data, if so, specifically, can be the data information memory at every into every server in data platform Master data sheet is preserved in server, then by the data information memory into the master data sheet in every server.

If it is determined that the corresponding attribute of the data information is not master data, then the data information is business datum, according to this The identification information of data information, and the storage algorithm pre-saved, determine the corresponding destination server of the data information, will be described Data information memory is into corresponding destination server, wherein the storage algorithm can be modulo operation.

It is different furthermore according to the performance of every server in data platform, it can be by a server in data platform As primary server, which is only used to store master data, by remaining service in data platform in addition to primary server Device is used as from server, it is described from server be used to store master data and with this from the corresponding business datum of server.

A server by data platform is divided into primary server, remaining server is divided into from server Process, which can be, to be realized by developer using haproxy software, and developer can to data by haproxy software Every server in platform is managed.

Since by primary data store into every server in data platform, master data will be removed in the embodiment of the present invention Remaining outer business datum is stored into its corresponding destination server, during accessing data, when obtaining to data information The pressure for reducing other servers decreases the time-consuming for accessing other servers, improves access efficiency.

Embodiment 4:

In order to facilitate the extension of data information field, the extended capability of data platform is improved, in the base of the various embodiments described above It is described to incite somebody to action after the corresponding destination server of determination data information described in data platform in the embodiment of the present invention on plinth Before the data information memory is into the destination server, the method also includes:

Whether the data type for judging the data information is real time data；

For data information to be stored, judge whether the data type of the data information is real time data, be can be Judged according to the identification information of the data information, is also possible to carry its corresponding data in the data information Type is judged according to the data type carried in the data information.

Likewise, since the identification information of data information is by Software Create, it is possible to the number described in Software Create It is believed that using the byte of specific length particular number of bits come the corresponding data type of marking data information when the identification information of breath.Such as The identification information of the data information is UUID, and length is 128 (binary system), the mark of the data information described in Software Create When information, with the 20th come the corresponding data type of marking data information, for example 1 is real time data, and 0 is non-real-time data.

For data to be stored, determine that the corresponding type of theme of the data information is taken according in the data information Its corresponding type of theme of band determines.

Rule is split according to each type of data in the corresponding type of theme of data information and the type of theme, it will Data information splits into the subdata information of corresponding each keyword, by each subdata information preservation to corresponding each data In table.When if there is the demand for being directed to the data information new field, it is only necessary to it is corresponding to add new field in the server Tables of data, facilitate the extension of data information field, improve the extended capability of data platform.

Specifically, the format of data information to be stored can be each keyword and its corresponding value, or press According to the sequence of each keyword, by the fixed subdata information of length be ranked up as a result, data information for example to be stored Format be each keyword and its corresponding value when, for data information to be stored be normal user data information, then should Each keyword includes name on account, gender, age in normal user data information, and keyword is that the corresponding value of name on account is " small red ", it be " female " and keyword be age corresponding value is 23 that keyword, which is the corresponding value of gender,；Data information to be stored Format be sequence according to each keyword, when the result that the fixed subdata information of length is ranked up, for wait deposit The data information of storage is normal user data information, then the normal user data information is " small red 000 female 23 ", wherein described the 1 to the 5th is the corresponding subdata information of name on account keyword " small red ", and the 6th is the corresponding subnumber of gender keyword It is believed that breath is " female ", the 7th and the 8th is that the corresponding subdata information of age keyword is " 23 ".

The corresponding data type of data information includes real time data and non-real-time data, and management server can be according to described The identification information of data information, or according to its corresponding data type carried in the data information, judge the data letter Cease whether corresponding data type is real time data.

When the corresponding data type of data information is real time data, the type of theme of the data information is determined, wherein The type of theme carries in the data information, and table 1 includes that the type of theme Topic of data information and type of theme correspond to Each type, i.e. data classification pre-saves type of theme and its corresponding data of each type in data platform Split rule.

In data platform in embodiments of the present invention, the type of theme is included at least: operation data RunningData Type of theme, log Log type of theme and report Report type of theme.Wherein, data access frequency be 1000 times/s and with On data information correspond to RunningData type of theme, the corresponding each type of RunningData type of theme includes unit Data, status data, service request, system response etc.；Data access frequency is that the data information of 10 times/s or more corresponds to Log Type of theme, the corresponding each type of Log type of theme include user journal, Request Log, processing log etc.；Data access frequency Rate is that the data information of 1 time/s or more corresponds to Report type of theme, and the corresponding each type of Report type of theme includes Data sheet, Data View, analysis report etc..Each type of theme and its every kind corresponding has been pre-saved in management server The fractionation rule of the data of type.The corresponding keyword of each type of data information is different, so it is also just different to split rule.

Table 1:

Management server is according to the corresponding type of theme of the data information, the type of theme locally pre-saved and its right The each type of data answered fractionation rule, such as data information to be stored type of theme be RunningData Type of theme, according to the fractionation of the type of theme and its corresponding each type of data locally pre-saved rule, determine to The data information of storage is corresponding to split the corresponding fractionation rule of data that rule is service request type, the service request class The corresponding fractionation rule of the data of type is according to the pass such as user of service request mode, service request source and initiating business request Keyword is either split according to the corresponding regular length of each subdata information, the data information is split into corresponding every The subdata information of a keyword；According to pair of each subdata information and each keyword and tables of data that are obtained after fractionation It should be related to, determine the corresponding each tables of data of each subdata information, it will be under the storage to destination server of each subdata information Each of in corresponding tables of data.

Method provided in an embodiment of the present invention is after the data type for judging the data information is real time data, Ke Yitong The Kafka in data platform is crossed by the storage of each subdata information to each of under destination server in corresponding tables of data.

Further, in order to guarantee the consistency of the data information in server, on the basis of the various embodiments described above, really Fixed destination server can be one, or at least two.

If it is determined that destination server be one, data information to be stored only needs to be stored in this described target In server, after the data information memory is into this destination server, for the data information reading data manipulation and Data writing operation is all provided by this server, therefore can guarantee the consistency of the data information in server.

If it is determined that destination server be at least two, each keyword that the basis pre-saves and tables of data Corresponding relationship determines that the corresponding each tables of data of each subdata information includes:

In the data handling procedure of the data platform, real time data link, including real time data are constructed using Kafka It acquires and carries out data storage.It is the data information of real time data for data type, Kafka determines that the data information is corresponding Type of theme Topic, Kafka according to the type of theme Topic and type of theme Topic and its corresponding each type Data fractionation rule, the data information is split as to the subdata information of corresponding each keyword, i.e., by the number It is believed that breath is split as multiple subregion Partition.

If the corresponding destination server of the data information is at least two, Kafka is at least two destination servers Middle determination one is Leader server, specifically, is namely directed to each Partition for each keyword, it is described extremely There is the Leader server of corresponding each keyword in few two destination servers.

According to the first corresponding relationship of each keyword and Leader server that pre-save, i.e., each Partition With the first corresponding relationship of Leader server, it is possible to determine the corresponding Leader server of each subdata information, and And the second corresponding relationship of tables of data in each keyword and Leader server is pre-saved in management server, so can With each subdata information of determination under corresponding Leader server corresponding tables of data, i.e., each Partition with Second corresponding relationship of each tables of data under Leader server.

Each subdata information of the data information is respectively stored into its corresponding each Leader by management server Under server in corresponding tables of data, according to the Kafka data synchronization rules pre-saved, by each Leader server Under the subdata information that stores in corresponding tables of data, be synchronized to remaining described target in addition to each Leader server Under server in each corresponding tables of data, the consistency of the data information in data server ensure that.

And in the data handling procedure in the data platform, when using Kafka, write for data information The process of operation includes: that management server receives write operation requests, wherein at least carrying in the write operation requests wait grasp The identification information of the data information of work and content information to be operated.For data information to be operated, management server judgement The data type of the data information to be operated is real time data, determines the type of theme of the data information to be operated, According to the fractionation of the type of theme and its corresponding each type of data pre-saved rule, by the number to be operated It is believed that breath is split as corresponding to the subdata information of each keyword；It is taken according to each keyword and leader Leader that pre-save It is engaged in the first corresponding relationship of device, the corresponding each Leader server of each subdata information is determined, according to pre-saving Each keyword and Leader server in tables of data the second corresponding relationship, determine that the content information to be operated is corresponding Each subdata information under each Leader server corresponding tables of data；By corresponding number under each Leader server Write operation is carried out according to the data information stored in table；It, will be described each according to the Kafka data synchronization rules pre-saved Subdata information after the write operation stored in corresponding tables of data under Leader server is synchronized to remaining described destination service Under device in each corresponding tables of data.

For some or all of data information delete operation, management server receives removal request, wherein described delete Except the content information for the identification information and deletion at least carrying data information to be deleted in request.Management server receives After removal request, the treatment process for data information to be deleted and the write operation for data information to be operated are processed Cheng Xiangtong is not repeated them here in embodiments of the present invention.

Due to applying Kafka data synchronization rules in the data platform in the embodiment of the present invention, each Leader is serviced Data information after the modified data information stored in corresponding tables of data under device or deletion, is synchronized to remaining described target Under server in each corresponding tables of data, the consistency of the data information in data server ensure that.

When the corresponding data type of data information is non-real-time data, the storage for data information can pass through Hadoop is realized.Management server belongs to the prior art by the process of Hadoop storing data information, implements in the present invention The process is not repeated them here in example.

When the corresponding data type of data information is non-real-time data, modification and deletion for data information can lead to Hadoop is crossed to realize.The process that management server was modified or deleted data information by Hadoop belongs to the prior art, at this The process is not repeated them here in inventive embodiments.

Due in the embodiment of the present invention management server according to the corresponding type of theme of data information and the type of theme pair The each type of data answered split rule, and data information is split into the subdata information of corresponding each keyword, will be each Subdata information preservation facilitates the extension of data information field, improves data platform into corresponding each tables of data Extended capability.

Embodiment 5:

Fig. 2 be a kind of schematic diagram of the acquisition process of data provided in an embodiment of the present invention, the process the following steps are included:

S201: receiving data acquisition instruction, wherein including the mark of data information to be obtained in data acquisition instruction Know information.

The acquisition methods of data provided in an embodiment of the present invention are applied to management server, install in the management server There is the management software for data platform.

It include multiple servers in the data platform, every server may serve to carry out the storage of data, management Member can be by obtaining data information in the server under the management software data platform installed in management server.

Management server can receive data acquisition instruction, and data letter to be obtained is carried in the data acquisition instruction The identification information of breath, management server are instructed according to the data acquisition, can determine the identification information of data information to be obtained. Wherein the identification information can be UUID, be also possible to the unique number etc. distributed for the data information, as long as can use Come data information described in unique identification and with corresponding identification information in the storage method of data.

S202: it according to the acquisition algorithm pre-saved in the identification information and data platform of the data information, determines The corresponding destination server of data information described in data platform.

The data information is stored in server in the data platform, the data information is needed in the data It obtaining in server in platform, the data information specifically obtains under which platform destination server in embodiments of the present invention, It is to be determined by the identification information of the data information.

The acquisition methods pre-saved in the data platform can be modulo operation, take the acquisition methods such as quotient's operation, only If acquisition methods corresponding with the storage method in the storage method of data.Management server is according to the identification information Corresponding with every server in the operation result of the acquisition algorithm and the operation result and the data platform is closed System, can determine the corresponding destination server of the identification information, so that it is determined that the destination service of the data information memory Device.

S203: the data information is obtained in the destination server.

It is described obtained in the destination server data information can be it is logical by using unified data interaction Come what is completed, can be in the unified data interaction communication modes includes application layer protocol for news mode and unified data-interface For the data interaction communication modes of HTTP or HTTPS, transport layer protocol TCP/IP or/and data API etc., this is unified Can be in data-interface mode includes unified technical protocol JDBC (Java Data Base Connectivity, Java number Connected according to library)/WebService, unified data protocol JSON (JavaScript Object Notation, JavaScript Object representation)/XML (Extensible Markup Language, extensible markup language) etc. data-interface mode.

Due in data capture method provided by the invention, for the corresponding identification information of data information to be obtained, and The acquisition algorithm pre-saved in data platform determines the corresponding destination server of the data information, then in destination server It is middle to obtain the data information, since data information to be obtained only stores in destination server, rather than it is stored in simultaneously In each server in data platform, the waste of server resource is avoided, and is directed to a variety of different data informations, is deposited The destination server of storage may be different, avoid individual server to a certain extent during obtaining data information in this way Pressure is excessive, improves data access efficiency.

Embodiment 6:

In order to further improve the access efficiency of data, on the basis of the various embodiments described above, in the embodiment of the present invention, The identification information of the data information is UUID, then pre- in the identification information and data platform according to the data information The acquisition algorithm first saved determines that the corresponding destination server of the data information described in data platform includes:

Since UUID is as the unique ID of Data for Global, there is global uniqueness, thus in embodiments of the present invention using with The identical UUID of the storage method of data as the data information identification information, when data information memory to data platform In destination server after, it is ensured that the identification information can with the unique identification data information, thus guarantee obtain number It is believed that breath can uniquely determine.

And it since UUID is generated by software for the data information, can be determined by UUID with identical function The data information of energy demand realizes the logic fragment to data information using functional requirement as the fragment logic of data information, There will be the data information memory of identical function demand into corresponding destination server, obtain number during data access It is believed that by the access pressure dissipation of data into different servers, the pressure for avoiding individual server is excessive, may be used also when breath When obtaining data information during data access, data letter is obtained in the corresponding server of data information to be obtained Breath, avoids and obtains in each server in data platform, further improve access efficiency.

In embodiments of the present invention, management server is according to the acquisition algorithm and data pre-saved in the data platform The storage method of information is identical, is modulo operation, in the identification information and data platform according to the data information in advance The algorithm of preservation determines the process of destination server are as follows: the service according to the identification information of the data information, in data platform Device quantity and modulo operation carry out modulo operation to UUID and number of servers, according to modulo operation as a result, and data platform In the modulo operation result that pre-saves and every server corresponding relationship, determine the corresponding destination service of the data information Device.

The destination server can be for one or multiple, when the destination server is multiple, can incite somebody to action Wherein at least one destination server is as reading destination server, remaining server in addition at least one destination server As destination server is write, realizes read and write abruption, further improve the efficiency of data access.

The data platform is distributed data platform, and the build process of the data platform can be by developer Deployment includes the cluster of more MySQL servers, and distributed data platform framework is built using Mycat tool.

<rule>

<algorithm>sharding-by-prefixpattern</algorithm>

</rule>

</tableRule>

< function name=" sharding-by-prefixpattern "

Class=" org.opencloudb.route.function.PartitionByPrefixPattern " >

<property name="mapFile">partition-prefixpattern.txt</property>

</function>

Due in data acquisition method for storing provided by the invention, for the corresponding identification information of data information to be obtained, And the modulo operation pre-saved in data platform, determine the corresponding destination server of the data information, then in destination service The data information is obtained in device, since data information is not in each server being stored in data platform simultaneously, and It is stored in corresponding destination server, avoids the waste of server resource, and be directed to a variety of different data informations, mesh Marking server may be different, and the pressure for avoiding individual server to a certain extent in this way is excessive, improves data access effect Rate.

Embodiment 7:

It is described in the embodiment of the present invention on the basis of the various embodiments described above in order to further increase data access efficiency After receiving data acquisition instruction, the method also includes:

If not, carrying out subsequent operation.

If the corresponding data information of the data information is master data, in any one server in data platform Obtain the data information.

In data platform in embodiments of the present invention, according to the corresponding attribute of data information, data information is divided into master Data and business datum, master data refer in data platform between server for sharing data, business datum be for Indicate the corresponding data of business service of enterprise.Master data is primarily referred to as user information, authority information in embodiments of the present invention Deng other data informations that, business datum is in addition to attribute is master data.For data information to be obtained, the number is judged It is believed that ceasing whether corresponding attribute is master data, wherein the corresponding attribute of the data information can carry the mark in data information In information, it is also possible to described data information itself and carries its corresponding attribute.

Due in the storage method of data, by primary data stores such as user information, authority informations into data platform In every server, host data is made to have complete backup on every server, realizes master data in data platform It is shared between server, by business datum storage into its corresponding destination server, for being used when data acquisition, if It is business datum that user, which needs to obtain data information, needs to verify user's letter in the corresponding destination server of the data information After breath, authority information, the data information is obtained, does not need to call the data information in other servers, reduces other services The pressure of device decreases the time-consuming for accessing other servers, improves access efficiency.

Whether management server judges the corresponding attribute of data information to be obtained after receiving data acquisition instruction For master data, if so, obtaining the data information in any one server in data platform.Specifically, if In the master data sheet for saving primary data store in every server in the storage method of data, then in any one server In master data sheet in obtain the data information.

If it is determined that the corresponding attribute of the data information is not master data, then the data information is business datum, according to this The identification information of data information, and storage algorithm, determine the corresponding destination server of the data information, obtain in destination server Take the data information.

It is different furthermore according to the performance of every server in data platform, it can be by a server in data platform As primary server, which is only used to store master data, by remaining service in data platform in addition to primary server Device is used as from server, described to be used to store its corresponding business datum and master data from server.If data to be obtained When the corresponding attribute of information is master data, it is only necessary to obtain the data information in primary server, reduce other servers Pressure, further improve access efficiency.

Due to since primary data store is in every server in data platform, removing master data in the embodiment of the present invention Remaining outer business datum is stored in its corresponding destination server, during accessing data, when obtaining to data information The pressure for reducing other servers decreases the time-consuming for accessing other servers, improves access efficiency.

Embodiment 8:

In order to facilitate the extension of data information field, the extended capability of data platform is improved, in the base of the various embodiments described above On plinth, in the embodiment of the present invention, after the corresponding destination server of determination data information described in data platform, it is described Before obtaining the data information in the destination server, the method also includes:

For data to be stored, judge whether the data type of the data information is real time data, can be basis What the identification information of the data information was judged.

Due to only carrying the identification information of data information to be obtained in the data acquisition instruction that receives, without taking The corresponding subdata information of keyword with data information to be obtained, so in the target determined by the identification information It is obtained in each tables of data under server, can just obtain whole subdata information of the data information, and the whole The corresponding set of subdata information is determined as the data information.

The management server by whole subdata information it is corresponding gather be determined as the process of the data information can be with Are as follows: for whole subdata information of the data information got, by the whole subdata information entire set to together, Set is determined as the data information.

The corresponding data type of data information includes real time data and non-real-time data, and management server can be according to described The identification information of data information judges whether the corresponding data type of the data information is real time data.

When the corresponding data type of data information to be obtained is real time data, each of under the destination server In tables of data, the corresponding each subdata information of identification information of the data information is obtained, by the collection of each subdata information Conjunction is determined as the data information.

Method provided in an embodiment of the present invention can be realized by the Kafka in data platform.If the data are flat Real time data link is constructed using Kafka in platform, the real time data link includes the acquisition of real time data, due to Kafka In preserve the corresponding message logging of identification information of data information to be obtained, so can be according to the mark using Kafka Know information and directly determine the corresponding tables of data of the data information, therefore can directly go in corresponding tables of data to obtain the data letter Corresponding subdata information is ceased, data access efficiency is further improved.

When the corresponding data type of data information is non-real-time data, the acquisition for data information can pass through Hadoop is realized.Management server belongs to the prior art by the process that Hadoop obtains data information, implements in the present invention It is not repeated them here in example.

Embodiment 9:

The various embodiments described above are illustrated with a specific embodiment below, Fig. 3 is provided in an embodiment of the present invention A kind of software frame schematic diagram of big data platform, storage and acquisition for data information can be based on the big data platforms Software frame is realized, as shown in figure 3, the source that the data source of the big data platform is data information to be stored includes APP (Application, computer applied algorithm), third-party platform, unit and data platform etc., specifically, the source APP can The software on smart machine to be mounted in enterprise's Internet of Things, third-party platform, which can be, provides number for the business in enterprise According to the platform etc. for supporting service, unit can be the hardware device such as air-conditioning etc. for the business service in enterprise, data platform It can be the platform etc. of the business service information for enterprise.Due to the diversity of data source, so the big data platform can With the application of integrating enterprise, after sale, data, the various data informations such as service, realize the integrated management of the business service of enterprise.

The logical of the uniform data interaction of standardization is used during big data platform storage or acquisition data information News mode realizes the quick substitution of applied business, further improves user in the efficiency of data access, also reduces more Exploitation complexity of the corresponding system of kind data source when integrated, as shown in figure 3, the communication side of big data platform standardization Formula may include application layer protocol for HTTP or HTTPS including using, transport layer protocol TCP/IP and data API etc., this Publish/subscribe message outside due to using Kafka frame in the big data platform, in the Kafka frame that also standardized.

Data type in the big data platform includes about request/response data, file data information, real-time streaming data Information etc. further includes Message data information additionally due to using Kafka frame in the big data platform.

Language in the big data platform data interactive mode including data interaction is Java and C++, acts on behalf of Proxy, HDFS (Hadoop Distributed File System, Hadoop distributed file system), furthermore in the big data platform Data interaction is realized using Kafka.

The efficiency that data access is further increased in the big data platform carries out the data type of data information real When data and non-real-time data differentiation, handled using Kafka frame for real time data, also using Hadoop frame It is the distributed management that HDFS carries out the non-real-time datas such as log, operation, data acquisition, by real time data and non real-time number According to management, to realize data summarization and the unified management of the big data platform.

The big data platform uses Kafka, realizes the real-time release of data information, constructs Real-time Data Service link, comes Realize acquisition, storage and acquisition, the publication of real time data.Kafka passes through simulated production person in acquisition, the storage of real time data Producer determines the corresponding type of theme Topic of the data information, and Kafka is according to type of theme Topic and the master The data information is split as corresponding each by the fractionation rule for inscribing type Topic and its corresponding each type of data The data information is split as multiple subregion Partition by the subdata information of keyword, according to each of pre-saving First corresponding relationship of the Leader server in keyword and at least two destination servers, i.e., each Partition With the first corresponding relationship of Leader server, it is possible to determine the corresponding Leader server of each subdata information, and And the second corresponding relationship of tables of data in each keyword and Leader server is pre-saved in management server, so can With each subdata information of determination under each Leader server corresponding each tables of data, i.e., each Partition with Second corresponding relationship of each tables of data under Leader server, by each subdata information of the data information, respectively It stores under its corresponding each Leader server in corresponding tables of data, according to the synchronous rule of the Kafka data pre-saved Then, the subdata information that will be stored in corresponding tables of data under each Leader server, it is synchronous except described each Under remaining described destination server outside Leader server in each corresponding tables of data.

Kafka in the acquisition of real time data, publication by phantom shoppers Consumer, complete data information acquisition and Publication carries out data processing to data information, to realize the displaying of data information.In the big data platform data analysis mode A kind of language C# etc. including Object Relation Mapping frame Hibernate, Redis (high performance database) and data parsing, Furthermore Kafka is used in the big data platform realizes data parsing.

Thread Task, trigger Trigger, storing process are used in the big data platform data processing method Procedure, MySQL Cluster (MySQL cluster) furthermore use haproxy and MyCat and manage in the big data platform Tool.

The big data platform applies MyCat management tool in build process, and MyCat completes distributed MySQL collection The management of group, has built the most basic distributed data frame of the bottom of the big data platform, has realized the distribution of metadata Storage.MyCat disposes more MySQL Cluster i.e. MySQL server to carry out the storage of data.MyCat is according to business number It is believed that the functional requirement of breath, carries out fragment to data information, if data information is the master datas such as user information, permission letter Breath, then all stored in every MySQL Cluster, realize the full backup of master data, if data information is remaining Business datum realizes the logic fragment to data information then according to functional requirement and the corresponding fragment logic of Various types of data information, It is saved in it in corresponding MySQL Cluster, data access pressure is distributed in different MySQL Cluster, with This pressure and performance bottleneck to evade individual node, improves the efficiency of data access.MyCat is corresponding by configuring Rule.xml file, specifically to realize logical partition.

The big data platform in build process using haproxy tool to the MySQL server being applied in MyCat into Row management is divided in MySQL Cluster according to the performance of MySQL server and divides primary server and from server, main number According to being stored in primary server and each from server, it is corresponding from server that remaining business datum is stored in its.Exploit person Member, can also be according to performance of server etc. come in primary server and from server during building the big data platform On the basis of realize the higher level structure such as node dual-active.

The big data platform uses unified data-interface to provide the storage of data and obtain and service, and wherein data connect It include unified technical protocol JDBC/WebService, unified data protocol JSON/XML in mouthful.

The bottom of WebService is write using Java language in the big data platform, passes through invocation framenort spring (Java Development Framework) and cxf (a kind of Open Framework) realize the exploitation and deployment of the WebService data-interface, simultaneously The access link for encapsulating redis and MyCat realizes data external access consistency and Information Security, and further Improve the efficiency of data access.

The application of the big data platform includes the application on display platform, service platform and other PC, and is applied flat Platform management function includes: system monitoring, rights management, metadata management, task management and safety management etc..

Application in the big data platform can introduce the API and data service of third-party platform, to improve in metadata Interaction and business service information, may be implemented business service equipment life cycle data management, bordereau, system stablize Property monitoring etc. platform management functions.

After the completion of the big data platform is built, the data being stored in the existing system of enterprise can be stored big to this In server under data platform, the modularization access of system is realized, for system to be accessed, developer or user are prior The access section of data and the distribution processor end of data are set, i.e., shows number using which kind of data source and using which kind of application According to, it will be able to realize that data information, into the server of the big data platform, realizes the quick of system by the storage of existing system Access, reduces the exploitation complexity of developer.For example, data source is " unit ", for different unit equipments, exploitation Personnel or the different unit equipment of user setting and the big data platform, the data type of corresponding data information to be stored and After the completion of the API of communication modes, which uses the API and unified Http, TCP, since the unit is corresponding Data source is non-real-time data, by the data information memory into HDFS, in storing process using JDBC or WebService completes the storage of data information.In the storing process of data, developer or user only need that data are arranged Data type and communication modes API, remaining process standardized in advance according to the big data platform completes, realizes different Unit equipment accesses the big data platform in a modular manner.

After the data information memory of the different unit equipment is to the big data platform, in the displaying to data information When, using unified data-interface complete call, the exhibiting data information in the demonstration tool of application.

Fig. 4 is a kind of structural schematic diagram of big data platform provided in an embodiment of the present invention, as shown in figure 4, the big data The data of platform are mainly provided API by api interface service and third data platform and support to support with data, and application is obtained Service is supported with data, provides basic data foundation for the big data platform.

It is mainly realized by service system, organization and administration, resource management and basic platform inside the big data platform The united analysis and management of user and business datum, wherein including: message server, file service, report in the service system Table service and device service include: general utility tool, computing engines, third-party platform, the resource management in the organization and administration Include: metadata, Service Source and device resource, includes: management platform and monitor supervision platform in the basic platform.

The big data platform is supported by Internet of Things service, channel application and light application etc. to provide unified hardware and software platform.

Due in date storage method provided by the invention, for the corresponding identification information of data information to be stored, and The storage algorithm pre-saved in data platform determines the corresponding destination server of the data information, then the data is believed Breath storage is into destination server, according to the identification information of data to be stored information, by data information memory to destination server In, rather than in each server being stored in data platform simultaneously, the waste of server resource is avoided, is avoided simultaneously The problem of keeping data consistency is needed in every server, and is directed to a variety of different data informations, and the target of storage takes Being engaged in device may be different, and the pressure for avoiding individual server to a certain extent in this way is excessive, improves data access efficiency.

Fig. 5 is a kind of memory device structure schematic diagram of data provided in an embodiment of the present invention, which includes:

Receiving module 51, for receiving data information to be stored, wherein including the mark of data in the data information Information；

Determining module 52, the storage algorithm for being pre-saved in the identification information and data platform according to the data, Determine the corresponding destination server of the data information described in data platform；

Memory module 53 is used for the data information memory into the destination server.

The determining module 52, the identification information specifically for the data are Universally Unique Identifier UUID, obtain institute The number of servers in data platform is stated, modulo operation is carried out to the UUID and number of servers；According to the modulo operation As a result, and the modulo operation result that pre-saves and every server corresponding relationship, determine the corresponding mesh of the data information Mark server.

Described device further include:

Judgment module 54, for judging whether the corresponding attribute of the data information is master data；If not, triggering determines Module 52.

The memory module 53 is also used to work as the judgment module and judges number based on the corresponding attribute of the data information According to by the data information memory into every server in data platform.

Described device further include:

Determining module 55 is judged, for judging whether the data type of the data information is real time data；If so, root The type of theme pre-saved according to the type of theme carried in the data information and its corresponding each type of data Fractionation rule, the data information is split as to correspond to the subdata information of each keyword；According to each of pre-saving The corresponding relationship of keyword and tables of data determines the corresponding each tables of data of each subdata information；

The memory module 53, it is described specifically for being respectively stored into each subdata information of the data information In each corresponding data table under destination server.

The judgement determining module 55, being also used to determining destination server is that at least two bases each of pre-save First corresponding relationship of keyword and leader Leader server determines the corresponding target clothes of each subdata information The each Leader server being engaged in device；According to second of tables of data in each keyword and Leader server pre-saved Corresponding relationship determines each subdata information corresponding tables of data under each Leader server；

The memory module 53 is specifically used for each subdata information storage arriving its corresponding each Leader server Under in corresponding tables of data；According to the data synchronization rules pre-saved, by corresponding number under each Leader server According to the subdata information stored in table, it is synchronized under remaining described destination server in each corresponding tables of data.

Fig. 6 is a kind of acquisition device structural schematic diagram of data provided in an embodiment of the present invention, which includes:

Receiving module 61, acquisition instruction for receiving data, wherein including number to be obtained in data acquisition instruction It is believed that the identification information of breath；

Determining module 62, the acquisition for being pre-saved in the identification information and data platform according to the data information Algorithm determines the corresponding destination server of the data information described in data platform；

Module 63 is obtained, for obtaining the data information in the destination server.

The determining module 62, the identification information specifically for the data information is Universally Unique Identifier UUID, is obtained The number of servers in the data platform is taken, modulo operation is carried out to the UUID and number of servers；According to the modulus Operation result, and the modulo operation result and the corresponding relationship of every server that pre-save, determine that the data information is corresponding Destination server.

Described device further include:

First judgment module 64, for judging whether the corresponding attribute of the data information is master data；If not, triggering Determining module 62.

The acquisition module 63 is also used to judge based on the corresponding attribute of the data information when the first judgment module Data obtain the data information in any one server in data platform.

Described device further include:

Second judgment module 65, whether the corresponding data type of identification information for judging the data information is real-time Data；If so, triggering obtains module；

The acquisition module 63, specifically for obtaining the data in each tables of data under the destination server The corresponding each subdata information of the identification information of information；Determine that the corresponding collection of each subdata information is combined into the data Information.

The present invention provides the storage acquisition methods and device of a kind of data, the storage method of the data includes: to receive Data information to be stored, wherein including the identification information of data in the data information；According to the identification information of the data, And the storage algorithm pre-saved in data platform, determine the corresponding destination server of the data information described in data platform； By the data information memory into the destination server.Due in date storage method provided by the invention, for wait deposit The storage algorithm pre-saved in the corresponding identification information of the data of storage and data platform determines the corresponding target of the data Server, then by data storage into corresponding destination server, rather than be stored in simultaneously each in data platform In platform server, the waste of server resource is avoided, and is directed to a variety of different data informations, the destination server of storage can Can be different, the pressure for avoiding individual server to a certain extent in this way is excessive, improves data access efficiency.

For systems/devices embodiment, since it is substantially similar to the method embodiment, so the comparison of description is simple Single, the relevent part can refer to the partial explaination of embodiments of method.

It should be understood by those skilled in the art that, embodiments herein can provide as method, system or computer program Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the application Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the application, which can be used in one or more, The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces The form of product.

The application is referring to method, the process of equipment (system) and computer program product according to the embodiment of the present application Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates, Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or The function of being specified in multiple boxes.

These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one The step of function of being specified in a box or multiple boxes.

Although the preferred embodiment of the application has been described, it is created once a person skilled in the art knows basic Property concept, then additional changes and modifications can be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as It selects embodiment and falls into all change and modification of the application range.

Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies Within, then the present invention is also intended to include these modifications and variations.

Claims

1. a kind of storage method of data, which is characterized in that this method comprises:

According to the storage algorithm pre-saved in the identification information and data platform of the data information, determine in data platform Described in the corresponding destination server of data information；

By the data information memory into the destination server；

It is described by the data information memory after the corresponding destination server of determination data information described in data platform Before into the destination server, the method also includes:

Whether the data type for judging the data information is real time data；

If so, according to the type of theme carried in the data information, the type of theme that pre-saves and its corresponding The fractionation rule of each type of data, the data information is split as to correspond to the subdata information of each keyword；

According to the corresponding relationship of each keyword pre-saved and tables of data, determine that each subdata information is corresponding every A tables of data；

By each subdata information of the data information, each corresponding data table for being respectively stored under the destination server In.

2. the method as described in claim 1, which is characterized in that the identification information of the data information is Universally Unique Identifier UUID, then the storage algorithm pre-saved in the identification information and data platform according to the data information are determined in number Include: according to the corresponding destination server of data information described in platform

3. the method as described in claim 1, which is characterized in that described to receive after the data information stored, the method is also Include:

If not, carrying out subsequent operation.

4. method as claimed in claim 3, which is characterized in that, will if the corresponding attribute of the data information is master data The data information memory is into every server in data platform.

5. the method as described in claim 1, which is characterized in that if it is determined that destination server be at least two, described According to the corresponding relationship of each keyword pre-saved and tables of data, the corresponding each data of each subdata information are determined Table includes:

According to the first corresponding relationship of each keyword and leader Leader server that pre-save, each subnumber is determined It is believed that each Leader server in the corresponding destination server of breath；

According to the second corresponding relationship of tables of data in each keyword and Leader server pre-saved, determine described each Subdata information corresponding tables of data under each Leader server；

According to the data synchronization rules pre-saved, the son that will be stored in corresponding tables of data under each Leader server Data information is synchronized under remaining described destination server in each corresponding tables of data.

6. a kind of acquisition methods of data, which is characterized in that this method comprises:

Data acquisition instruction is received, wherein including the identification information of data information to be obtained in data acquisition instruction；

According to the acquisition algorithm pre-saved in the identification information and data platform of the data information, determine in data platform Described in the corresponding destination server of data information；

The data information is obtained in the destination server；

It is described in the destination server after the corresponding destination server of determination data information described in data platform Before obtaining the data information, the method also includes:

In each tables of data under the destination server, the corresponding each subnumber of identification information of the data information is obtained It is believed that breath；

7. method as claimed in claim 6, which is characterized in that the identification information of the data information is Universally Unique Identifier UUID, the then acquisition algorithm pre-saved in the identification information and data platform according to the data information are determined in number Include: according to the corresponding destination server of data information described in platform

8. method as claimed in claim 6, which is characterized in that after the reception data acquisition instruction, the method also includes:

If not, carrying out subsequent operation.

9. method according to claim 8, which is characterized in that if number based on the corresponding data information of the data information According to obtaining the data information in any one server in data platform.

10. a kind of storage device of data, which is characterized in that the device includes:

Receiving module, for receiving data information to be stored, wherein including the identification information of data in the data information；

Determining module, the storage algorithm for pre-saving in the identification information and data platform according to the data determine The corresponding destination server of data information described in data platform；

Memory module is used for the data information memory into the destination server；

Described device further include:

Determining module is judged, for judging whether the data type of the data information is real time data；If so, according to described The type of theme carried in data information, the fractionation of the type of theme pre-saved and its corresponding each type of data The data information is split as corresponding to the subdata information of each keyword by rule；According to each keyword pre-saved With the corresponding relationship of tables of data, the corresponding each tables of data of each subdata information is determined；

The memory module, specifically for being respectively stored into the target clothes for each subdata information of the data information It is engaged in each corresponding data table under device.

11. device as claimed in claim 10, which is characterized in that it is flat to be specifically used for the acquisition data for the determining module Number of servers in platform carries out modulo operation to the identification information Universally Unique Identifier UUID and number of servers of data； According to the modulo operation as a result, and the modulo operation result that pre-saves and every server corresponding relationship, determine described in The corresponding destination server of data information.

12. device as claimed in claim 10, which is characterized in that described device further include:

Judgment module, for judging whether the corresponding attribute of the data information is master data；If not, triggering determining module.

13. device as claimed in claim 12, which is characterized in that the memory module, if being also used to the judgment module The corresponding attribute of the data information is judged for master data, by the data information memory to every server in data platform In.

14. device as claimed in claim 10, which is characterized in that the judgement determining module is also used to determining target clothes Business device is the first corresponding relationship of each keyword and leader Leader server that at least two bases pre-save, determines institute State each Leader server in the corresponding destination server of each subdata information；According to each pass pre-saved The second corresponding relationship of tables of data, determines that each subdata information takes in each Leader in keyword and Leader server Corresponding tables of data under business device；

The memory module, being specifically used for will be corresponding under the storage to its corresponding each Leader server of each subdata information Tables of data in；It, will be under each Leader server in corresponding tables of data according to the data synchronization rules pre-saved The subdata information of storage is synchronized under remaining described destination server in each corresponding tables of data.

15. a kind of acquisition device of data, which is characterized in that the device includes:

Receiving module, acquisition instruction for receiving data, wherein including data information to be obtained in data acquisition instruction Identification information；

Determining module, the acquisition algorithm for being pre-saved in the identification information and data platform according to the data information, really It is scheduled on the corresponding destination server of data information described in data platform；

Module is obtained, for obtaining the data information in the destination server；

Described device further include:

Second judgment module, whether the corresponding data type of identification information for judging the data information is real time data； If so, triggering obtains module；

The acquisition module, specifically for obtaining the data information in each tables of data under the destination server The corresponding each subdata information of identification information；Determine that the corresponding collection of each subdata information is combined into the data information.

16. device as claimed in claim 15, which is characterized in that it is flat to be specifically used for the acquisition data for the determining module Number of servers in platform takes the identification information Universally Unique Identifier UUID and number of servers of the data information Modular arithmetic；According to the modulo operation as a result, and the modulo operation result that pre-saves and every server corresponding relationship, really Determine the corresponding destination server of the data information.

17. device as claimed in claim 15, which is characterized in that described device further include:

First judgment module, for judging whether the corresponding attribute of the data information is master data；Mould is determined if not, triggering Block.

18. device as claimed in claim 17, which is characterized in that the acquisition module, if being also used to first judgement Module judges the corresponding attribute of the data information for master data, in any one server in data platform described in acquisition Data information.