CN106980669A

CN106980669A - A kind of storage of data, acquisition methods and device

Info

Publication number: CN106980669A
Application number: CN201710179427.6A
Authority: CN
Inventors: 牟桂贤; 蔡伟星; 黄卫基; 甘俊源
Original assignee: Gree Electric Appliances Inc of Zhuhai
Current assignee: Gree Electric Appliances Inc of Zhuhai
Priority date: 2017-03-23
Filing date: 2017-03-23
Publication date: 2017-07-25
Anticipated expiration: 2037-03-23
Also published as: CN106980669B

Abstract

The invention discloses the storage acquisition methods and device of a kind of data, the storage method of the data includes：Data message to be stored is received, wherein the data message includes the identification information of data；According to the identification information of the data, and the storage algorithm pre-saved in data platform, it is determined that the corresponding destination server of the data message described in data platform；By the data information memory into the destination server.In the date storage method provided due to the present invention, store data into corresponding destination server, rather than be stored in simultaneously in each server in data platform, avoid the waste of server resource, and for a variety of different data messages, the destination server of storage may be different, and the pressure that individual server is so avoided to a certain extent is excessive, improves data access efficiency.

Description

A kind of storage of data, acquisition methods and device

Technical field

The present invention relates to field of computer technology, more particularly to a kind of storage of data, acquisition methods and device.

Background technology

The computer technology industry developed rapidly is except changing the every aspect of people's life, promoting social rapid advances Outside, also for enterprise, this main body of the society brings more directly perceived and effective influence.Opened up with the product business of enterprise Exhibition, builds a data platform tool for multiple product business scenario and is of great significance, also receive enterprise especially The attention of enterprise in computer industry, and the high availability of data platform, service stability, Information Security, manage just Victory is the center of people's growing interest.For the data platform of enterprise-level, the multiple product business that enterprise is extended How data are stored, and are faster, preferably enterprise product so that the data of multiple product business are managed and analyzed The user of business provides service, the even more most important thing.

The storage on data is typically the two-server being directed in data platform in the prior art, by number to be stored According to being stored in the two-server simultaneously, wherein a server provides increase, the data writing operation deleted, changed, separately One server provides the reading data manipulation of inquiry, that is, ensures to realize the read and write abruption of data after data storage.For enterprise For the data platform of industry level, (Peta Byte clap word to typically TB (Trillion Byte, terabyte) levels even PB Save/thousand terabytes) level big data quantity, because data volume is huge, will result in data using this date storage method superfluous Remaining, server resource is wasted, in addition after data storage, the substantial amounts of concurrent operations read data or write data, can be caused single The pressure of server is excessive, the problem of running into performance bottleneck, reduces the efficiency of data access.

The content of the invention

The present invention provides a kind of storage of data, acquisition methods and device, to solve to service present in prior art The device wasting of resources, data access efficiency is low and multiple servers in data consistency it is poor the problem of.

The problem of in order to solve above-mentioned, the invention provides a kind of storage method of data, this method includes：

Data message to be stored is received, wherein the data message includes the identification information of data message；

According to the identification information of the data message, and the storage algorithm pre-saved in data platform, it is determined that in data The corresponding destination server of data message described in platform；

By the data information memory into the destination server.

Further, the identification information of the data message is general unique identifier UUID, then described according to the number It is believed that the identification information of breath, and the storage algorithm pre-saved in data platform, it is determined that the data message described in data platform Corresponding destination server includes：

The number of servers in the data platform is obtained, modulo operation is carried out to the UUID and number of servers；

According to the modulo operation result, and the modulo operation result pre-saved and the corresponding relation of every server, Determine the corresponding destination server of the data message.

Further, described to receive after data message to be stored, methods described also includes：

Whether judge the corresponding attribute of the data message is master data；

If not, carrying out subsequent operation.

Further, if the corresponding attribute of the data message is master data, by the data information memory to data In every server in platform.

Further, it is described by institute after the corresponding destination server of determination data message described in data platform Before data information memory is stated into the destination server, methods described also includes：

Whether the data type for judging the data message is real time data；

If it is, according to the type of theme carried in the data message, the type of theme pre-saved and its right The fractionation rule for each type of data answered, the data message is split as the subdata information of each keyword of correspondence；

According to each keyword and the corresponding relation of tables of data pre-saved, each subdata information correspondence is determined Each tables of data；

It is described to include the data information memory to the destination server：

By each subdata information of the data message, each correspondence number under the destination server is respectively stored into According in table.

Further, if it is determined that destination server be at least two, each keyword that the basis is pre-saved With the corresponding relation of tables of data, determine that the corresponding each tables of data of each subdata information includes：

According to the first corresponding relation of each keyword pre-saved and leader's Leader servers, determine described each Each Leader servers in the corresponding destination server of subdata information；

According to each keyword pre-saved and the second corresponding relation of tables of data in Leader servers, it is determined that described Each subdata information corresponding tables of data under each Leader servers；

By the storage of each subdata information into corresponding tables of data under its corresponding each Leader server；

According to the data syn-chronization rule pre-saved, it will be stored in corresponding tables of data under each Leader servers Subdata information, be synchronized under remaining described destination server in each corresponding tables of data.

The invention provides a kind of acquisition methods of data, this method includes：

Data acquisition instruction is received, wherein data acquisition instruction includes the mark letter of data message to be obtained Breath；

According to the identification information of the data message, and the acquisition algorithm pre-saved in data platform, it is determined that in data The corresponding destination server of data message described in platform；

The data message is obtained in the destination server.

Further, the identification information of the data message is general unique identifier UUID, then described according to the number It is believed that the identification information of breath, and the acquisition algorithm pre-saved in data platform, it is determined that the data message described in data platform Corresponding destination server includes：

Further, after the reception data acquisition instruction, methods described also includes：

Whether judge the corresponding attribute of the data message is master data；

If not, carrying out subsequent operation.

Further, it is any one in data platform if the corresponding data message of the data message is master data The data message is obtained in platform server.

Further, it is described in institute after the corresponding destination server of determination data message described in data platform State before the data message is obtained in destination server, methods described also includes：

Whether judge data type corresponding to the identification information of the data message is real time data；

The data message is obtained in the destination server include if it is, described：

In each tables of data under the destination server, corresponding each of identification information of the data message is obtained Subdata information；

Determine that the corresponding collection of each subdata information is combined into the data message.

The invention provides a kind of storage device of data, the device includes：

Receiving module, the data message to be stored for receiving, wherein the data message includes the mark letter of data Breath；

Determining module, for the identification information according to the data, and the storage algorithm pre-saved in data platform, really It is scheduled on the corresponding destination server of data message described in data platform；

Memory module, for by the data information memory into the destination server.

Further, the determining module, specifically for obtaining the number of servers in the data platform, to data The general unique identifier UUID of identification information and number of servers carry out modulo operation；According to the modulo operation result, and in advance The modulo operation result first preserved and the corresponding relation of every server, determine the corresponding destination server of the data message.

Further, described device also includes：

Judge module, for judging whether the corresponding attribute of the data message is master data；If not, triggering determines mould Block.

Further, the memory module, if being additionally operable to the judge module judges the corresponding category of the data message Property is master data, by the data information memory into every server in data platform.

Further, described device also includes：

Determining module is judged, for judging whether the data type of the data message is real time data；If it is, according to The type of theme carried in the data message, the type of theme pre-saved and its corresponding each type of data Rule is split, the data message is split as to the subdata information of each keyword of correspondence；According to each pass pre-saved The corresponding relation of keyword and tables of data, determines the corresponding each tables of data of each subdata information；

The memory module, specifically for by each subdata information of the data message, being respectively stored into the mesh Mark in each corresponding data table under server.

Further, the judgement determining module, the destination server for being additionally operable to determine is at least two bases guarantor in advance First corresponding relation of each keyword deposited and leader's Leader servers, determines the corresponding institute of each subdata information State each Leader servers in destination server；According to each keyword pre-saved and data in Leader servers Second corresponding relation of table, determines each subdata information corresponding tables of data under each Leader servers；

The memory module, specifically for the storage of each subdata information is arrived under its corresponding each Leader server In corresponding tables of data；According to the data syn-chronization rule pre-saved, by corresponding data under each Leader servers The subdata information stored in table, is synchronized under remaining described destination server in each corresponding tables of data.

The invention provides a kind of acquisition device of data, the device includes：

Receiving module, for receiving data acquisition instruction, wherein data acquisition instruction includes data to be obtained The identification information of information；

Determining module, for the identification information according to the data message, and the acquisition pre-saved in data platform is calculated Method, it is determined that the corresponding destination server of the data message described in data platform；

Acquisition module, for obtaining the data message in the destination server.

Further, the determining module, specifically for obtaining the number of servers in the data platform, to the number It is believed that the general unique identifier UUID of identification information and number of servers of breath carry out modulo operation；According to the modulo operation knot Really, and the modulo operation result that pre-saves and the corresponding relation of every server, the corresponding target of the data message is determined Server.

Further, described device also includes：

First judge module, for judging whether the corresponding attribute of the data message is master data；If not, triggering is true Cover half block.

Further, the acquisition module, if being additionally operable to first judge module judges the data message correspondence Attribute be master data, obtain the data message in any one server in data platform.

Further, described device also includes：

Second judge module, for judging whether data type corresponding to the identification information of the data message is real-time number According to；If it is, triggering acquisition module；

The acquisition module, specifically in each tables of data under the destination server, obtaining the data letter Each subdata information corresponding to the identification information of breath；Determine that the corresponding collection of each subdata information is combined into the data letter Breath.

The invention provides a kind of storage of data, acquisition methods and device, the storage method of the data includes：Receive Data message to be stored, wherein the data message includes the identification information of data；According to the identification information of the data, And the storage algorithm pre-saved in data platform, it is determined that the corresponding destination server of the data message described in data platform； By the data information memory into the destination server.In the date storage method provided due to the present invention, for waiting to deposit The corresponding identification information of data of storage, and the storage algorithm pre-saved in data platform, determine the corresponding target of the data Server, then by the corresponding destination servers of the data Cun Chudao, rather than be stored in simultaneously each in data platform In platform server, it is to avoid the waste of server resource, and for a variety of different data messages, the destination server of storage can Can be different, the pressure that individual server is so avoided to a certain extent is excessive, improves data access efficiency.

Brief description of the drawings

In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with Other accompanying drawings are obtained according to these accompanying drawings.

Fig. 1 is a kind of schematic diagram of the storing process for data that the embodiment of the present invention 1 is provided；

Fig. 2 is a kind of schematic diagram of the acquisition process for data that the embodiment of the present invention 5 is provided；

Fig. 3 is a kind of software frame schematic diagram for big data platform that the embodiment of the present invention 9 is provided；

Fig. 4 is a kind of structural representation for big data platform that the embodiment of the present invention 9 is provided；

Fig. 5 is a kind of memory device structure schematic diagram of data provided in an embodiment of the present invention；

Fig. 6 is a kind of acquisition device structural representation of data provided in an embodiment of the present invention.

Embodiment

In order to avoid the waste of server resource, and data access efficiency is improved, the embodiments of the invention provide a kind of number According to storage, acquisition methods and device.

In order that the object, technical solutions and advantages of the present invention are clearer, make below in conjunction with the accompanying drawing present invention into one It is described in detail on step ground, it is clear that described embodiment is only a part of embodiment of the present invention, rather than whole implementation Example.Based on the embodiment in the present invention, what those of ordinary skill in the art were obtained under the premise of creative work is not made Every other embodiment, belongs to the scope of protection of the invention.

Embodiment 1：

Fig. 1 is a kind of schematic diagram of the storing process of data provided in an embodiment of the present invention, and the process comprises the following steps：

S101：Data message to be stored is received, wherein the data message includes the identification information of data message.

The storage method of data provided in an embodiment of the present invention is applied to install in management server, the management server There is the management software for data platform.

The data platform includes multiple servers, and every server may serve to carry out the storage of data, management Member can be by the management software installed in management server by under the data information memory to be stored to data platform In server, wherein the configuration of every server can with it is identical can be different, that is to say, that stored in every server Data message can part it is identical, can also be different.

The management server can be for receiving data message to be stored, and the data message includes data letter Corresponding identification information is ceased, in addition, also including the corresponding content information of data message in the data message.Wherein described mark Information can be UUID (Universally Unique Identifier, general unique identifier) or be the number It is believed that unique number of breath distribution etc., as long as data message described in unique mark can be used for.

S102：According to the identification information of the data message, and the storage algorithm pre-saved in data platform, it is determined that The corresponding destination server of data message described in data platform.

Server in the data platform is used for storing the data message, in embodiments of the present invention the data letter Which platform destination server breath is especially stored under, is determined by the identification information of the data message.

The storage algorithm pre-saved in the data platform can be modulo operation, take business's computing etc. to store algorithm.Root According to the operation result of the identification information and the storage algorithm, and the operation result and every clothes in the data platform The corresponding relation of business device, it may be determined that go out destination server corresponding to the identification information, so that it is determined that the data message Corresponding destination server.

S103：By the data information memory into the destination server.

The process by the data information memory into destination server can arrive the data information memory In tables of data in destination server.

The data information memory can be the content letter by data message into which of destination server tables of data Keyword in breath the is determined or identification information according to the data message, is determined by the algorithm pre-saved 's.

Described by described information storage is communicated by using unified data interaction into the destination server Mode and unified data-interface, come what is completed, can be to include application layer protocol to be in the unified data interaction communication modes HTTP (HyperText Transfer Protocol, HTTP) or HTTPS (Hyper Text Transfer SSL is added under Protocol over Secure Socket Layer, HTTP), transport layer protocol For TCP/IP (Transmission Control Protocol/Internet Protocol, transmission control protocol/internet Interconnection protocol), or/and data API (Application Programming Interface, application programming interface) etc. Data interaction communication modes, can be to include unified technical protocol JDBC (Java in the unified data-interface mode Data Base Connectivity, Java database are connected)/WebService, unified data protocol JSON (JavaScript Object Notation, JavaScript object representation)/XML (Extensible Markup Language, extensible markup language) etc. data-interface mode.

For the data platform of enterprise-level, due to usually requiring TB grades of storage even in the data platform of enterprise-level PB grades of big data quantity, data volume is huge, using the storage method of data provided in an embodiment of the present invention, for avoiding server The waste of resource and raising data access efficiency, it is better.

In the date storage method provided due to the present invention, for the corresponding identification information of data message to be stored, and The storage algorithm pre-saved in data platform, determines the corresponding destination server of the data message, then the data are believed Breath storage is into destination server, according to the identification information of data message to be stored, by data information memory to destination server In, rather than in each server being stored in data platform simultaneously, it is to avoid the waste of server resource, and for many Different data messages are planted, the destination server of storage may be different, and individual server is so avoided to a certain extent Pressure is excessive, improves data access efficiency.

Embodiment 2：

In order to further improve the access efficiency of data, on the basis of above-described embodiment, in the embodiment of the present invention, institute The identification information for stating data message is general unique identifier UUID, then the identification information according to the data message, and The storage algorithm pre-saved in data platform, it is determined that the corresponding destination server bag of the data message described in data platform Include：

Because UUID is as the unique ID of Data for Global, with global uniqueness, so applying in embodiments of the present invention UUID is used as the identification information of the data message, can after destination server of the data information memory into data platform To ensure to use the identification information can be with the unique mark data message.

And because UUID is generated by software for the data message, so can be determined with phase by UUID The data message of congenerous demand, using functional requirement as the burst logic of data message, realizes the logic to data message Burst, will have the data information memory of identical function demand into corresponding destination server.During data access By the access pressure dissipation of data into different servers, it is to avoid the pressure of individual server is excessive, can also be in data When obtaining data message in access process, data message is obtained in the corresponding server of data message to be obtained, it is to avoid Obtained in each server in data platform, further increase access efficiency.

In embodiments of the present invention, management server is modulus according to the storage algorithm pre-saved in the data platform Computing, the identification information according to the data message, and the algorithm pre-saved in data platform determine destination server Process be：According to the identification information of the data message, the number of servers in data platform, and modulo operation, to UUID And number of servers carries out modulo operation, according to modulo operation result, and the modulo operation result pre-saved in data platform With the corresponding relation of every server, the corresponding destination server of the data message is determined.

It can also be many that the destination server, which can be one,, can be by when the destination server is many One destination server of wherein at least is as reading destination server, remaining server in addition at least one destination server As destination server is write, read and write abruption is realized, the efficiency of data access is further improved.

The data platform is distributed data platform, and the build process of the data platform can be by developer Deployment includes the cluster of one of many Relational DBMSs (MySQL) server, using in distributed data base Between part (Mycat) instrument build distributed data platform framework.

The embodiment of the present invention is illustrated with a specific embodiment below, installed in management server be used for take The management software for building distributed data platform framework is Mycat, and the identification information of the data message is UUID, the modulus fortune Calculate as M%N=P, M=u1+u2+u3+ ...+u32, wherein described u1, u2, u3 ... and first character, second that u32 is UUID The ASCII character value of position character, the 3rd character ... and the 32nd character, N is the number of servers in data platform, and P is Modulo operation result, preserves modulo operation result and the corresponding relation of every server, according to modulus in management server Operation result P, the modulo operation result and the corresponding relation of every server, determine target clothes corresponding to the identification information Business device, by the corresponding destination server of data message described in the destination server.

Match somebody with somebody specifically, adding sharding-by-prefixpattern rules in the existing rule.xml files of MyCat Put, rule configuration is as follows：

<rule>

<algorithm>sharding-by-prefixpattern</algorithm>

</rule>

</tableRule>

<Function name=" sharding-by-prefixpattern "

Class=" org.opencloudb.route.function.PartitionByPrefixPattern ">

<Property name=" mapFile ">partition-prefixpattern.txt</property>

</function>

<tableRule>The title name that label includes rule is sharding-by-prefixpattern, specifically Regular rule includes columns and is used for identifying the literary name section id for burst, and algorithm is used for identifying burst function i.e. modulus Computing.<function>The title name that label includes the function is sharding-by-prefixpattern, affiliated class Class is org.opencloudb.route.function.PartitionByPrefixPattern, and patternValue is used To identify the radix of modulo operation, the radix of the modulo operation is the server in the data platform in embodiments of the present invention Quantity, e.g. 8, prefixLength is for identifying the digit 32 of the ASCII for calculating here, and mapFile is used for identifying The path of configuration file is partition-prefixpattern.txt, i.e. configuration file partition- Modulo operation result and the corresponding relation of every server are preserved in prefixpattern.txt.

In the date storage method provided due to the present invention, for the corresponding identification information of data message to be stored, and The modulo operation pre-saved in data platform, determines the corresponding destination server of the data message, then the data are believed Breath storage is into destination server, rather than in each server being stored in data platform simultaneously, it is to avoid server The waste of resource, and for a variety of different data messages, the destination server of storage may be different, so to a certain extent The pressure for avoiding individual server is excessive, improves data access efficiency.

Embodiment 3：

It is described in the embodiment of the present invention on the basis of the various embodiments described above in order to further improve data access efficiency Receive after data message to be stored, methods described also includes：

Whether judge the corresponding attribute of the data message is master data；

If not, carrying out subsequent operation.

In data platform in embodiments of the present invention, according to the corresponding attribute of data message, data message is divided into master Data and business datum, master data refer in data platform the data for being used to share between server, business datum be for Represent the corresponding data of business service of enterprise.Master data is primarily referred to as user profile, authority information in embodiments of the present invention It is other data messages in addition to attribute is master data Deng, business datum.For data message to be stored, the number is judged It is believed that ceasing whether corresponding attribute is master data, the wherein corresponding attribute of the data message can carry the mark in data message In information or data message carries its corresponding attribute in itself.

Because the identification information of data message is by Software Create, it is possible in data message described in Software Create Using the byte of length-specific particular number of bits come the corresponding attribute of marking data information during identification information.Such as described data message Identification information be UUID, length is 128 (binary system), in the identification information of data message described in Software Create, with the 10 are carried out the corresponding attribute of marking data information, such as 1 is master data, and 0 is business datum.

If the corresponding attribute of the data message is master data, and the data information memory is every into data platform In platform server.

By primary data stores such as user profile, authority informations into every server in data platform so that master data Have complete backup on every server, realize it is shared between server of the master data in data platform, by business In its corresponding destination server of data Cun Chudao, used during for data acquisition.

Management server after data message to be stored is received, judge the corresponding attribute of the data message whether based on Data, if it is, can be at every specifically by the data information memory into every server in data platform Master data sheet is preserved in server, then by the data information memory into the master data sheet in every server.

If it is determined that the corresponding attribute of the data message is not master data, then the data message is business datum, according to this The identification information of data message, and the storage algorithm pre-saved, determine the corresponding destination server of the data message, will be described Data information memory is into corresponding destination server, wherein the storage algorithm can be modulo operation.

Performance further according to every server in data platform is different, can be by a server in data platform As master server, the master server is only used for storing master data, by remaining service in data platform in addition to master server Device as from server, it is described from server be used for storing master data and with this from the corresponding business datum of server.

A server by data platform is divided into master server, and remaining server is divided into from server Process can be realized that developer can be to data by haproxy softwares by developer using haproxy softwares Every server in platform is managed.

Due in the embodiment of the present invention, by primary data store into every server in data platform, master data will be removed Remaining outer business datum is stored into its corresponding destination server, in data procedures are accessed, when being obtained to data message The pressure of other servers is reduced, decreases and accesses the time-consuming of other servers, improve access efficiency.

Embodiment 4：

The extension of data information field for convenience, improves the extended capability of data platform, in the base of the various embodiments described above It is described to determine after the corresponding destination server of the data message described in data platform in the embodiment of the present invention on plinth, it is described to incite somebody to action Before the data information memory is into the destination server, methods described also includes：

Whether the data type for judging the data message is real time data；

For data message to be stored, whether the data type for judging the data message is real time data, Ke Yishi Its corresponding data is carried in data message judged according to the identification information of the data message or described Type, is judged according to the data type carried in the data message.

Likewise, because the identification information of data message is by Software Create, it is possible in number described in Software Create It is believed that using the byte of length-specific particular number of bits come the corresponding data type of marking data information during the identification information of breath.For example The identification information of the data message is UUID, and length is 128 (binary system), in the mark of data message described in Software Create During information, with the 20th come the corresponding data type of marking data information, such as 1 is real time data, and 0 is non-real-time data.

For data to be stored, determining the corresponding type of theme of the data message is taken according in the data message Its corresponding type of theme of band is determined.

According to the corresponding type of theme of data message, and each type of data fractionation rule in the type of theme, will Data message splits into the subdata information of each keyword of correspondence, and each subdata information is saved in into corresponding each data In table.During if there is demand for the data message new field, it is only necessary to add new field correspondence in the server Tables of data, facilitate the extension of data information field, improve the extended capability of data platform.

Specifically, the form of data message to be stored can be each keyword and its corresponding value, or press According to the order of each keyword, the result that the subdata information that length is fixed is ranked up, such as data message to be stored Form be each keyword and its corresponding value when, for data message to be stored be normal user data information, then should Each keyword includes name on account, sex, age in normal user data information, and keyword is that the corresponding value of name on account is " small red ", keyword is that the corresponding value of sex is " female ", and keyword is that age corresponding value is 23；Data message to be stored Form be according to each keyword order, during the result that the subdata information that length is fixed is ranked up, for waiting to deposit The data message of storage is normal user data information, then the normal user data information is " small red 000 female 23 ", wherein described the 1 to the 5th is the corresponding subdata information of name on account keyword " small red ", and the 6th is the corresponding subnumber of sex keyword It is believed that breath is " female ", the 7th and the 8th is that the corresponding subdata information of age keyword is " 23 ".

The corresponding data type of data message includes real time data and non-real-time data, and management server can be according to described The identification information of data message, or according to its corresponding data type carried in the data message, judge the data letter Whether cease corresponding data type is real time data.

When the corresponding data type of data message is real time data, the type of theme of the data message is determined, wherein The type of theme is carried in the data message, and table 1 includes the type of theme Topic and type of theme correspondence of data message Each type, i.e. data classify, and type of theme and its corresponding data of each type are pre-saved in data platform Split rule.

In data platform in embodiments of the present invention, the type of theme at least includes：Service data RunningData Type of theme, daily record Log type of theme and form Report type of theme.Wherein, data access frequency be 1000 times/s and with On data message correspondence RunningData type of theme, the corresponding each type of RunningData type of theme includes unit Data, status data, service request, system response etc.；Data access frequency is 10 times/s and the data message correspondence Log of the above Type of theme, the corresponding each type of Log type of theme includes user journal, Request Log, processing daily record etc.；Data access frequency Rate is 1 time/s and the data message correspondence Report type of theme of the above, and the corresponding each type of Report type of theme includes Data sheet, Data View, analysis report etc..Each type of theme is pre-saved in management server and its corresponding every kind of The fractionation rule of the data of type.The corresponding keyword of each type of data message is different, so it is also just different to split rule.

Table 1：

Management server is according to the corresponding type of theme of the data message, the type of theme locally pre-saved and its right The fractionation rule for each type of data answered, such as type of theme for data message to be stored are RunningData Type of theme, according to the fractionation of the type of theme and its corresponding each type of data locally pre-saved rule, it is determined that treating The corresponding corresponding fractionation rule of data for splitting rule for service request type of the data message of storage, the service request class The corresponding rule that splits of the data of type is to be closed according to user of service request mode, service request source and initiating business request etc. Keyword is either split according to the corresponding regular length of each subdata information, and it is every that the data message is split into correspondence The subdata information of individual keyword；According to each subdata information obtained after fractionation, and each keyword and pair of tables of data It should be related to, it is determined that the corresponding each tables of data of each subdata information, by under each subdata information storage to destination server Each corresponding tables of data in.

Method provided in an embodiment of the present invention is after judging the data type of the data message for real time data, Ke Yitong The Kafka in data platform is crossed to store each subdata information into each corresponding tables of data under destination server.

Further, in order to ensure the uniformity of the data message in server, on the basis of the various embodiments described above, really Fixed destination server can be one, or at least two.

If it is determined that destination server be one, data message to be stored only needs to be stored in this target In server, after the data information memory is into this destination server, for the data message reading data manipulation and Data writing operation is all provided by this server, therefore can ensure the uniformity of the data message in server.

If it is determined that destination server be at least two, each keyword that the basis is pre-saved and tables of data Corresponding relation, determines that the corresponding each tables of data of each subdata information includes：

In the data handling procedure of the data platform, real time data link, including real time data are built using Kafka Gather and carry out data storage.For the data message that data type is real time data, Kafka determines the data message correspondence Type of theme Topic, Kafka according to type of theme Topic, and the type of theme Topic and its corresponding each type Data fractionation rule, the data message is split as to the subdata information of corresponding each keyword, will the number It is believed that breath is split as multiple subregion Partition.

If the corresponding destination server of the data message is at least two, Kafka is at least two destination servers It is middle determine one be Leader servers, specifically, for each keyword namely be directed to each Partition, it is described extremely There are the Leader servers of each keyword of correspondence in few two destination servers.

According to the first corresponding relation of each keyword pre-saved and Leader servers, i.e., each Partition With the first corresponding relation of Leader servers, it is possible to it is determined that the corresponding Leader servers of each subdata information, and And each keyword and the second corresponding relation of tables of data in Leader servers are pre-saved in management server, so can To determine each subdata information corresponding tables of data under corresponding Leader servers, i.e., each Partition with Second corresponding relation of each tables of data under Leader servers.

Each subdata information of the data message is respectively stored into its corresponding each Leader by management server Under server in corresponding tables of data, according to the Kafka data syn-chronizations rule pre-saved, by each Leader servers Under the subdata information that stores in corresponding tables of data, be synchronized to remaining described target in addition to each Leader servers Under server in each corresponding tables of data, it is ensured that the uniformity of the data message in data server.

And in the data handling procedure in the data platform, during using Kafka, for entering row write to data message The process of operation includes：Management server receives write operation requests, needs to be grasped wherein at least carrying in the write operation requests The identification information of the data message of work and content information to be operated.For data message to be operated, management server judges The data type of the data message to be operated is real time data, determines the type of theme of the data message to be operated, According to the fractionation of the type of theme and its corresponding each type of data pre-saved rule, by the number to be operated It is believed that breath is split as the subdata information of each keyword of correspondence；Taken according to each keyword pre-saved with leader Leader It is engaged in the first corresponding relation of device, the corresponding each Leader servers of each subdata information is determined, according to pre-saving Each keyword and Leader servers in tables of data the second corresponding relation, determine the content information correspondence to be operated Each subdata information under each Leader servers corresponding tables of data；By corresponding number under each Leader servers Write operation is carried out according to the data message stored in table；, will be described each according to the Kafka data syn-chronizations rule pre-saved Subdata information after the write operation stored under Leader servers in corresponding tables of data, is synchronized to remaining described destination service Under device in each corresponding tables of data.

For the part or all of deletion action of data message, management server receives removal request, wherein described delete Except the identification information that data message to be deleted is at least carried in request and the content information deleted.Management server is received After removal request, the processing procedure for data message to be deleted is treated with the write operation for data message to be operated Cheng Xiangtong, is not repeated in embodiments of the present invention.

Due to, using Kafka data syn-chronizations rule, each Leader being serviced in the data platform in the embodiment of the present invention Data message after the amended data message stored under device in corresponding tables of data or deletion, is synchronized to remaining described target Under server in each corresponding tables of data, it is ensured that the uniformity of the data message in data server.

When the corresponding data type of data message is non-real-time data, the storage for data message can pass through Hadoop is realized.Management server belongs to prior art by the process of Hadoop data storage information, in present invention implementation The process is not repeated in example.

When the corresponding data type of data message is non-real-time data, modification and deletion for data message can lead to Hadoop is crossed to realize.The process that data message was changed or deleted to management server by Hadoop belongs to prior art, at this The process is not repeated in inventive embodiments.

Due in the embodiment of the present invention management server according to the corresponding type of theme of data message, and the type of theme pair The each type of data answered split rule, and data message is split into the subdata information of each keyword of correspondence, will be each Subdata information is saved in corresponding each tables of data, is facilitated the extension of data information field, is improved data platform Extended capability.

Embodiment 5：

Fig. 2 is a kind of schematic diagram of the acquisition process of data provided in an embodiment of the present invention, and the process comprises the following steps：

S201：Data acquisition instruction is received, wherein data acquisition instruction includes the mark of data message to be obtained Know information.

The acquisition methods of data provided in an embodiment of the present invention are applied to install in management server, the management server There is the management software for data platform.

The data platform includes multiple servers, and every server may serve to carry out the storage of data, management Member can be by obtaining data message in the server under the management software data platform installed in management server.

Management server can receive data acquisition instruction, and data letter to be obtained is carried in the data acquisition instruction The identification information of breath, management server is instructed according to the data acquisition, it may be determined that the identification information of data message to be obtained. Wherein described identification information can be UUID or the unique number distributed for the data message etc., as long as can use Come data message described in unique mark and identification information corresponding with the storage method of data.

S202：According to the identification information of the data message, and the acquisition algorithm pre-saved in data platform, it is determined that The corresponding destination server of data message described in data platform.

Be stored with the data message in server in the data platform, and the data message is needed in the data Obtained in server in platform, the data message is specific in embodiments of the present invention obtains under which platform destination server, It is to be determined by the identification information of the data message.

The acquisition methods pre-saved in the data platform can be modulo operations, take the acquisition methods such as business's computing, only If acquisition methods corresponding with the storage method in the storage method of data.Management server is according to the identification information With the operation result of the acquisition algorithm, and operation result pass corresponding with every server in the data platform System, it may be determined that go out destination server corresponding to the identification information, so that it is determined that the destination service of the data information memory Device.

S203：The data message is obtained in the destination server.

It is described that the data message is obtained in the destination server can be logical by using unified data interaction News mode and unified data-interface, come what is completed, can be to include application layer protocol in the unified data interaction communication modes For HTTP or HTTPS, transport layer protocol is TCP/IP, or/and data API etc. data interaction communication modes, this is unified Can be to include unified technical protocol JDBC (Java Data Base Connectivity, Java numbers in data-interface mode Connected according to storehouse)/WebService, unified data protocol JSON (JavaScript Object Notation, JavaScript Object representation)/XML (Extensible Markup Language, extensible markup language) etc. data-interface mode.

In the data capture method provided due to the present invention, for the corresponding identification information of data message to be obtained, and The acquisition algorithm pre-saved in data platform, determines the corresponding destination server of the data message, then in destination server It is middle to obtain the data message, because data message to be obtained is only stored in destination server, rather than it is stored in simultaneously In each server in data platform, it is to avoid the waste of server resource, and for a variety of different data messages, deposit The destination server of storage may be different, so avoid individual server to a certain extent during data message is obtained Pressure is excessive, improves data access efficiency.

Embodiment 6：

In order to further improve the access efficiency of data, on the basis of the various embodiments described above, in the embodiment of the present invention, The identification information of the data message is UUID, then the identification information according to the data message, and in data platform in advance The acquisition algorithm first preserved, it is determined that the corresponding destination server of the data message described in data platform includes：

Because UUID is as the unique ID of Data for Global, with global uniqueness, so in embodiments of the present invention apply with The storage method identical UUID of data is used as the identification information of the data message, when data information memory to data platform In destination server after, it is ensured that the identification information can with the unique mark data message so that ensure obtain number It is believed that breath can be uniquely determined.

And because UUID is generated by software for the data message, it can be determined with identical work(by UUID The data message of energy demand, using functional requirement as the burst logic of data message, realizes the logic burst to data message, The data information memory that will have identical function demand obtains number into corresponding destination server during data access It is believed that during breath, by the access pressure dissipation of data into different servers, it is to avoid the pressure of individual server is excessive, may be used also So that when obtaining data message during data access, data letter is obtained in the corresponding server of data message to be obtained Breath, it is to avoid obtained in each server in data platform, further increase access efficiency.

In embodiments of the present invention, management server is according to the acquisition algorithm and data pre-saved in the data platform The storage method of information is identical, is modulo operation, the identification information according to the data message, and in data platform in advance The algorithm of preservation determines that the process of destination server is：According to the identification information of the data message, the service in data platform Device quantity, and modulo operation, carry out modulo operation, according to modulo operation result, and data platform to UUID and number of servers In the modulo operation result that pre-saves and the corresponding relation of every server, determine the corresponding destination service of the data message Device.

The destination server can be one can also be it is multiple, when the destination server for it is multiple when, can be by Wherein at least one destination server is as reading destination server, remaining server in addition at least one destination server As destination server is write, read and write abruption is realized, the efficiency of data access is further improved.

The data platform is distributed data platform, and the build process of the data platform can be by developer Deployment includes the cluster of many MySQL servers, and distributed data platform framework is built using Mycat instruments.

<rule>

<algorithm>sharding-by-prefixpattern</algorithm>

</rule>

</tableRule>

<Function name=" sharding-by-prefixpattern "

Class=" org.opencloudb.route.function.PartitionByPrefixPattern ">

<Property name=" mapFile ">partition-prefixpattern.txt</property>

</function>

In the data acquisition method for storing provided due to the present invention, for the corresponding identification information of data message to be obtained, And the modulo operation pre-saved in data platform, the corresponding destination server of the data message is determined, then in destination service The data message is obtained in device, due to data message be not while in each server being stored in data platform, and It is stored in corresponding destination server, it is to avoid the waste of server resource, and for a variety of different data messages, mesh Marking server may be different, and the pressure that individual server is so avoided to a certain extent is excessive, improves data access effect Rate.

Embodiment 7：

It is described in the embodiment of the present invention on the basis of the various embodiments described above in order to further improve data access efficiency Receive after data acquisition instruction, methods described also includes：

Whether judge the corresponding attribute of the data message is master data；

If not, carrying out subsequent operation.

If during the corresponding data message of the data message is master data, any one server in data platform Obtain the data message.

In data platform in embodiments of the present invention, according to the corresponding attribute of data message, data message is divided into master Data and business datum, master data refer in data platform the data for being used to share between server, business datum be for Represent the corresponding data of business service of enterprise.Master data is primarily referred to as user profile, authority information in embodiments of the present invention It is other data messages in addition to attribute is master data Deng, business datum.For data message to be obtained, the number is judged It is believed that ceasing whether corresponding attribute is master data, the wherein corresponding attribute of the data message can carry the mark in data message In information or described data message carries its corresponding attribute in itself.

Due in the storage method of data, by primary data stores such as user profile, authority informations into data platform In every server so that master data has complete backup on every server, realize master data in data platform It is shared between server, by business datum storage into its corresponding destination server, used during for data acquisition, if User needs to obtain data message for business datum, it is necessary to verify user's letter in the corresponding destination server of the data message After breath, authority information, the data message is obtained, it is not necessary to call the data message in other servers, reduce other services The pressure of device, decreases and accesses the time-consuming of other servers, improve access efficiency.

Whether management server judges the corresponding attribute of data message to be obtained after data acquisition instruction is received For master data, if it is, obtaining the data message in any one server in data platform.Specifically, if In the master data sheet for preserving primary data store in every server in the storage method of data, then in any one server In master data sheet in obtain the data message.

If it is determined that the corresponding attribute of the data message is not master data, then the data message is business datum, according to this The identification information of data message, and storage algorithm, determine the corresponding destination server of the data message, are obtained in destination server Take the data message.

Performance further according to every server in data platform is different, can be by a server in data platform As master server, the master server is only used for storing master data, by remaining service in data platform in addition to master server Device is described to be used for storing its corresponding business datum and master data from server as from server.If data to be obtained When the corresponding attribute of information is master data, it is only necessary to the data message is obtained in master server, other servers are reduced Pressure, further increase access efficiency.

Due in the embodiment of the present invention, due in every server of the primary data store in data platform, except master data Remaining outer business datum is stored in its corresponding destination server, in data procedures are accessed, when being obtained to data message The pressure of other servers is reduced, decreases and accesses the time-consuming of other servers, improve access efficiency.

Embodiment 8：

The extension of data information field for convenience, improves the extended capability of data platform, in the base of the various embodiments described above On plinth, in the embodiment of the present invention, after the corresponding destination server of determination data message described in data platform, it is described Obtained in the destination server before the data message, methods described also includes：

For data to be stored, whether the data type for judging the data message is real time data, can be basis What the identification information of the data message was judged.

The identification information of data message to be obtained is only carried in being instructed due to the data acquisition received, without taking The corresponding subdata information of keyword with data message to be obtained, so in the target determined by the identification information Obtained in each tables of data under server, can just obtain whole subdata information of the data message, and the whole The corresponding set of subdata information is defined as the data message.

The management server by whole subdata information it is corresponding gather be defined as the process of the data message can be with For：For whole subdata information of the data message got, by whole subdata information entire sets to together, Set is defined as the data message.

The corresponding data type of data message includes real time data and non-real-time data, and management server can be according to described Whether the identification information of data message, it is real time data to judge the corresponding data type of the data message.

It is each under the destination server when the corresponding data type of data message to be obtained is real time data In tables of data, each subdata information corresponding to the identification information of the data message is obtained, by the collection of each subdata information Conjunction is defined as the data message.

Method provided in an embodiment of the present invention can be realized by the Kafka in data platform.If the data are put down Real time data link is built using Kafka in platform, the real time data link includes the acquisition of real time data, due to Kafka In preserve message logging corresponding to identification information of data message to be obtained, so can be according to the mark using Kafka Know information and directly determine the corresponding tables of data of the data message, therefore can directly go in corresponding tables of data to obtain the data letter Corresponding subdata information is ceased, data access efficiency is further increased.

When the corresponding data type of data message is non-real-time data, the acquisition for data message can pass through Hadoop is realized.Management server belongs to prior art by the Hadoop processes for obtaining data message, in present invention implementation Do not repeated in example.

Embodiment 9：

The various embodiments described above are illustrated with a specific embodiment below, Fig. 3 is provided in an embodiment of the present invention A kind of software frame schematic diagram of big data platform, storage and acquisition for data message can be based on the big data platforms Software frame is realized, as shown in figure 3, the source that the data source of the big data platform is data message to be stored includes APP (Application, computer applied algorithm), third-party platform, unit and data platform etc., specifically, APP sources can With the software being mounted on the smart machine of enterprise's Internet of Things, third-party platform can be that the business being directed in enterprise provides number According to platform for supporting service etc., unit can be for hardware device such as air-conditioning of business service in enterprise etc., data platform Can be for platform of business service information of enterprise etc..Due to the diversity of data source, so the big data platform can With the application of integrating enterprise, after sale, data, the various data messages such as service, realize the integrated management of the business service of enterprise.

The big data platform stores or employed during obtaining data message the logical of the uniform data interaction of standardization News mode, realizes the quick substitution of applied business, further increases efficiency of the user in data access, also reduces many Exploitation complexity of the corresponding system of kind of data source when integrated, as shown in figure 3, the communication side of the big data platform specification Formula, which includes using, can include application layer protocol for HTTP or HTTPS, and transport layer protocol is TCP/IP, and data API etc., this Outside due to employing Kafka frameworks in the big data platform, the publish/subscribe message in the Kafka frameworks that also standardized.

Data type in the big data platform is included on request/response data, file data information, real-time streaming data Information etc., additionally due to Kafka frameworks are employed in the big data platform, in addition to Message data messages.

The language that the big data platform data interactive mode includes data interaction is Java and C++, acts on behalf of Proxy, HDFS (Hadoop Distributed File System, Hadoop distributed file system), in addition in the big data platform Employ Kafka and realize data interaction.

In order to further improve the efficiency of data access in the big data platform, the data type to data message is carried out in fact When data and non-real-time data differentiation, handled using Kafka frameworks for real time data, using Hadoop frameworks also It is the distributed management that HDFS carries out the non-real-time datas such as daily record, operation, data acquisition, by real time data and non real-time number According to management, to realize data summarization and the unified management of the big data platform.

The big data platform uses Kafka, realizes the real-time release of data message, builds Real-time Data Service link, comes Realize collection, storage and acquisition, the issue of real time data.Kafka passes through simulated production person in collection, the storage of real time data Producer, determines the corresponding type of theme Topic of the data message, Kafka is according to type of theme Topic, and the master The fractionation rule of type Topic and its corresponding each type of data is inscribed, the data message is split as corresponding each The subdata information of keyword, will the data message be split as multiple subregion Partition, it is each according to what is pre-saved First corresponding relation of keyword and the Leader servers at least two destination servers, i.e., each Partition With the first corresponding relation of Leader servers, it is possible to it is determined that the corresponding Leader servers of each subdata information, and And each keyword and the second corresponding relation of tables of data in Leader servers are pre-saved in management server, so can To determine each subdata information corresponding each tables of data under each Leader servers, i.e., each Partition with Second corresponding relation of each tables of data under Leader servers, by each subdata information of the data message, difference Store under its corresponding each Leader server in corresponding tables of data, advised according to the Kafka data syn-chronizations pre-saved Then, the subdata information that will be stored in corresponding tables of data under each Leader servers, it is synchronous except described each Under remaining described destination server outside Leader servers in each corresponding tables of data.

Kafka in the acquisition of real time data, issue by phantom shoppers Consumer, complete data message acquisition and Issue, carries out data processing, so as to realize the displaying of data message to data message.In the big data platform data analysis mode Including Object Relation Mapping framework Hibernate, Redis (a kind of high performance database), and data parsing language C# etc., In addition Kafka is employed in the big data platform and realizes that data are parsed.

Thread Task, trigger Trigger, storing process are employed in the big data platform data processing mode Procedure, MySQL Cluster (MySQL clusters), employ haproxy and MyCat management in addition in the big data platform Instrument.

The big data platform applies MyCat management tools in build process, and MyCat completes distributed MySQL collection The management of group, has built the most basic distributed data framework of the bottom of the big data platform, has realized the distribution of metadata Storage.MyCat disposes many MySQL Cluster i.e. MySQL server to carry out the storage of data.MyCat is according to business number It is believed that the functional requirement of breath, burst is carried out to data message, if data message is the master datas such as user profile, authority letter Breath, then all stored in every MySQL Cluster, realize the full backup of master data, if data message is remaining Business datum, then according to functional requirement and the corresponding burst logic of Various types of data information, realize the logic burst to data message, It is saved in corresponding MySQL Cluster, data access pressure is distributed in different MySQL Cluster, with This evades the pressure and performance bottleneck of individual node, improves the efficiency of data access.MyCat is corresponding by configuring Rule.xml files, specifically to realize logical partition.

The big data platform is entered in build process using haproxy instruments to the MySQL server being applied in MyCat Row management, according to the performance of MySQL server, is divided in MySQL Cluster and divides master server and from server, main number According to master server is stored in each from server, it is corresponding from server that remaining business datum is stored in its.Exploit person Member, can also be according to performance of server etc. come in master server and from server during the big data platform is built On the basis of realize the higher level structure such as node dual-active.

The storage that the big data platform employs unified data-interface to provide data is serviced with obtaining, and wherein data connect Mouth includes unified technical protocol JDBC/WebService, unified data protocol JSON/XML.

WebService bottom is write using Java language in the big data platform, passes through invocation framenort spring (Java Development Frameworks) and cxf (a kind of Open Framework) realize the exploitation and deployment of the WebService data-interfaces, simultaneously Redis and MyCat access link is encapsulated, the access uniformity and Information Security of data externally is realized, and further Improve the efficiency of data access.

The application of the big data platform includes the application on display platform, service platform, and other PC, and apply it is flat Platform management function includes：System monitoring, rights management, metadata management, task management and safety management etc..

Application in the big data platform can introduce the API and data, services of third-party platform, to improve in metadata Interaction and business service information, it is possible to achieve the life cycle data management of business service equipment, bordereau, system are stable Property monitoring etc. platform management functions.

, can this be big by the data Cun Chudao being stored in the existing system of enterprise after the completion of the big data platform building In server under data platform, the modularization access of system is realized, for system to be accessed, developer or user are prior The access section of data and the distribution processor end of data are set, i.e., using which kind of data source and using which kind of application displaying number According to, it becomes possible to realize that data message is stored into the server of the big data platform by existing system, realize the quick of system Access, reduces the exploitation complexity of developer.For example, data source is " unit ", for different unit equipments, exploitation Personnel or user set different unit equipments and the big data platform, the data type of corresponding data message to be stored and After the completion of the API of communication modes, the big data platform uses the API and unified Http, TCP, because the unit is corresponding Data source is non-real-time data, by the data information memory into HDFS, in storing process using JDBC or WebService, completes the storage of data message.In the storing process of data, developer or user only need to set data Data type and communication modes API, remaining process standardized in advance according to the big data platform completes, realizes different Unit equipment accesses the big data platform in a modular manner.

After the data information memory of the different unit equipment is to the big data platform, in the displaying to data message When, using unified data-interface complete call, the exhibiting data information in the demonstration tool of application.

Fig. 4 is a kind of structural representation of big data platform provided in an embodiment of the present invention, as shown in figure 4, the big data The data of platform are mainly provided API by api interface service and the 3rd data platform and supported and data support, obtain application Service is supported with data, and basic data foundation is provided for the big data platform.

The big data platform interior is mainly realized by service system, organization and administration, resource management and basic platform The united analysis and management of user and business datum, wherein the service system includes：Message server, file service, report Table is serviced and device service, and the organization and administration include：General utility tool, computing engines, third-party platform, the resource management Including：Metadata, Service Source and device resource, the basic platform include：Management platform and monitor supervision platform.

The big data platform is supported by Internet of Things service, channel application and light application etc. to provide unified hardware and software platform.

In the date storage method provided due to the present invention, for the corresponding identification information of data message to be stored, and The storage algorithm pre-saved in data platform, determines the corresponding destination server of the data message, then the data are believed Breath storage is into destination server, according to the identification information of data message to be stored, by data information memory to destination server In, rather than in each server being stored in data platform simultaneously, it is to avoid the waste of server resource, avoid simultaneously The problem of needing to keep data consistency in every server, and for a variety of different data messages, the target of storage takes Being engaged in device may be different, and the pressure that individual server is so avoided to a certain extent is excessive, improves data access efficiency.

Fig. 5 is a kind of memory device structure schematic diagram of data provided in an embodiment of the present invention, and the device includes：

Receiving module 51, the data message to be stored for receiving, wherein the data message includes the mark of data Information；

Determining module 52, for the identification information according to the data, and the storage algorithm pre-saved in data platform, It is determined that the corresponding destination server of the data message described in data platform；

Memory module 53, for by the data information memory into the destination server.

The determining module 52, the identification information specifically for the data is general unique identifier UUID, obtains institute The number of servers in data platform is stated, modulo operation is carried out to the UUID and number of servers；According to the modulo operation As a result, and the modulo operation result that pre-saves and the corresponding relation of every server, the corresponding mesh of the data message is determined Mark server.

Described device also includes：

Judge module 54, for judging whether the corresponding attribute of the data message is master data；If not, triggering is determined Module 52.

The memory module 53, is additionally operable to when the judge module judges number based on the corresponding attribute of the data message According to by the data information memory into every server in data platform.

Described device also includes：

Determining module 55 is judged, for judging whether the data type of the data message is real time data；If it is, root According to the type of theme carried in the data message, the type of theme pre-saved and its corresponding each type of data Fractionation rule, the data message is split as to the subdata information of each keyword of correspondence；It is each according to what is pre-saved The corresponding relation of keyword and tables of data, determines the corresponding each tables of data of each subdata information；

The memory module 53, it is described specifically for by each subdata information of the data message, being respectively stored into In each corresponding data table under destination server.

It is described judgement determining module 55, be additionally operable to determine destination server be at least two bases pre-save it is each First corresponding relation of keyword and leader's Leader servers, determines the corresponding target clothes of each subdata information The each Leader servers being engaged in device；According to each keyword pre-saved and second of tables of data in Leader servers Corresponding relation, determines each subdata information corresponding tables of data under each Leader servers；

The memory module 53, specifically for the storage of each subdata information is arrived into its corresponding each Leader server Under in corresponding tables of data；According to the data syn-chronization rule pre-saved, by corresponding number under each Leader servers According to the subdata information stored in table, it is synchronized under remaining described destination server in each corresponding tables of data.

Fig. 6 is a kind of acquisition device structural representation of data provided in an embodiment of the present invention, and the device includes：

Receiving module 61, for receiving data acquisition instruction, wherein data acquisition instruction includes number to be obtained It is believed that the identification information of breath；

Determining module 62, for the identification information according to the data message, and the acquisition pre-saved in data platform Algorithm, it is determined that the corresponding destination server of the data message described in data platform；

Acquisition module 63, for obtaining the data message in the destination server.

The determining module 62, the identification information specifically for the data message is general unique identifier UUID, is obtained The number of servers in the data platform is taken, modulo operation is carried out to the UUID and number of servers；According to the modulus Operation result, and the modulo operation result pre-saved and the corresponding relation of every server, determine the data message correspondence Destination server.

Described device also includes：

First judge module 64, for judging whether the corresponding attribute of the data message is master data；If not, triggering Determining module 62.

The acquisition module 63, is additionally operable to when first judge module is judged based on the corresponding attribute of the data message The data message is obtained in data, any one server in data platform.

Described device also includes：

Second judge module 65, for judging whether data type corresponding to the identification information of the data message is real-time Data；If it is, triggering acquisition module；

The acquisition module 63, specifically in each tables of data under the destination server, obtaining the data Each subdata information corresponding to the identification information of information；Determine that the corresponding collection of each subdata information is combined into the data Information.

The invention provides the storage acquisition methods and device of a kind of data, the storage method of the data includes：Receive Data message to be stored, wherein the data message includes the identification information of data；According to the identification information of the data, And the storage algorithm pre-saved in data platform, it is determined that the corresponding destination server of the data message described in data platform； By the data information memory into the destination server.In the date storage method provided due to the present invention, for waiting to deposit The corresponding identification information of data of storage, and the storage algorithm pre-saved in data platform, determine the corresponding target of the data Server, then by the corresponding destination servers of the data Cun Chudao, rather than be stored in simultaneously each in data platform In platform server, it is to avoid the waste of server resource, and for a variety of different data messages, the destination server of storage can Can be different, the pressure that individual server is so avoided to a certain extent is excessive, improves data access efficiency.

For systems/devices embodiment, because it is substantially similar to embodiment of the method, so the comparison of description is simple Single, the relevent part can refer to the partial explaination of embodiments of method.

It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program Product.Therefore, the application can be using the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware Apply the form of example.Moreover, the application can be used in one or more computers for wherein including computer usable program code The computer program production that usable storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.) The form of product.

The application is the flow with reference to method, equipment (system) and computer program product according to the embodiment of the present application Figure and/or block diagram are described.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided The processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.

These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which is produced, to be included referring to Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or The function of being specified in multiple square frames.

These computer program instructions can be also loaded into computer or other programmable data processing devices so that in meter Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one The step of function of being specified in individual square frame or multiple square frames.

Although having been described for the preferred embodiment of the application, those skilled in the art once know basic creation Property concept, then can make other change and modification to these embodiments.So, appended claims are intended to be construed to include excellent Select embodiment and fall into having altered and changing for the application scope.

Obviously, those skilled in the art can carry out the essence of various changes and modification without departing from the present invention to the present invention God and scope.So, if these modifications and variations of the present invention belong to the scope of the claims in the present invention and its equivalent technologies Within, then the present invention is also intended to comprising including these changes and modification.

Claims

1. a kind of storage method of data, it is characterised in that this method includes：

According to the identification information of the data message, and the storage algorithm pre-saved in data platform, it is determined that in data platform Described in the corresponding destination server of data message；

By the data information memory into the destination server.

2. the method as described in claim 1, it is characterised in that the identification information of the data message is general unique identifier UUID, the then identification information according to the data message, and the storage algorithm pre-saved in data platform, it is determined that in number Include according to the corresponding destination server of data message described in platform：

According to the modulo operation result, and the modulo operation result pre-saved and the corresponding relation of every server, it is determined that The corresponding destination server of the data message.

3. the method as described in claim 1, it is characterised in that after reception data message to be stored, methods described is also Including：

Whether judge the corresponding attribute of the data message is master data；

If not, carrying out subsequent operation.

4. method as claimed in claim 3, it is characterised in that if the corresponding attribute of the data message is master data, will The data information memory is into every server in data platform.

5. the method as described in claim 1, it is characterised in that determination data message described in data platform is corresponding After destination server, it is described by the data information memory into the destination server before, methods described also includes：

Whether the data type for judging the data message is real time data；

If it is, according to the type of theme carried in the data message, the type of theme pre-saved and its corresponding The fractionation rule of each type of data, the data message is split as the subdata information of each keyword of correspondence；

According to each keyword and the corresponding relation of tables of data pre-saved, determine that each subdata information is corresponding every Individual tables of data；

By each subdata information of the data message, each corresponding data table under the destination server is respectively stored into In.

6. method as claimed in claim 5, it is characterised in that if it is determined that destination server be at least two, described According to each keyword and the corresponding relation of tables of data pre-saved, the corresponding each data of each subdata information are determined Table includes：

According to the first corresponding relation of each keyword pre-saved and leader's Leader servers, each subnumber is determined It is believed that each Leader servers in the corresponding destination server of breath；

According to each keyword pre-saved and the second corresponding relation of tables of data in Leader servers, determine described each Subdata information corresponding tables of data under each Leader servers；

According to the data syn-chronization rule pre-saved, the son that will be stored in corresponding tables of data under each Leader servers Data message, is synchronized under remaining described destination server in each corresponding tables of data.

7. a kind of acquisition methods of data, it is characterised in that this method includes：

Data acquisition instruction is received, wherein data acquisition instruction includes the identification information of data message to be obtained；

According to the identification information of the data message, and the acquisition algorithm pre-saved in data platform, it is determined that in data platform Described in the corresponding destination server of data message；

The data message is obtained in the destination server.

8. method as claimed in claim 7, it is characterised in that the identification information of the data message is general unique identifier UUID, the then identification information according to the data message, and the acquisition algorithm pre-saved in data platform, it is determined that in number Include according to the corresponding destination server of data message described in platform：

9. method as claimed in claim 7, it is characterised in that after the reception data acquisition instruction, methods described also includes：

Whether judge the corresponding attribute of the data message is master data；

If not, carrying out subsequent operation.

10. method as claimed in claim 9, it is characterised in that if number based on the corresponding data message of the data message According to obtaining the data message in any one server in data platform.

11. method as claimed in claim 7, it is characterised in that determination data message correspondence described in data platform Destination server after, it is described obtain the data message in the destination server before, methods described also includes：

In each tables of data under the destination server, each subnumber corresponding to the identification information of the data message is obtained It is believed that breath；

12. a kind of storage device of data, it is characterised in that the device includes：

Receiving module, the data message to be stored for receiving, wherein the data message includes the identification information of data；

Determining module, for the identification information according to the data, and the storage algorithm pre-saved in data platform, it is determined that The corresponding destination server of data message described in data platform；

Memory module, for by the data information memory into the destination server.

13. device as claimed in claim 12, it is characterised in that the determining module, puts down specifically for obtaining the data Number of servers in platform, modulo operation is carried out to the general unique identifier UUID of identification information and number of servers of data； According to the modulo operation result, and the modulo operation result pre-saved and the corresponding relation of every server, it is determined that described The corresponding destination server of data message.

14. device as claimed in claim 12, it is characterised in that described device also includes：

Judge module, for judging whether the corresponding attribute of the data message is master data；If not, triggering determining module.

15. device as claimed in claim 14, it is characterised in that the memory module, if being additionally operable to the judge module The corresponding attribute of the data message is judged for master data, by the data information memory to every server in data platform In.

16. device as claimed in claim 12, it is characterised in that described device also includes：

Determining module is judged, for judging whether the data type of the data message is real time data；If it is, according to described The type of theme carried in data message, the fractionation of the type of theme pre-saved and its corresponding each type of data Rule, the data message is split as the subdata information of each keyword of correspondence；According to each keyword pre-saved With the corresponding relation of tables of data, the corresponding each tables of data of each subdata information is determined；

The memory module, takes specifically for by each subdata information of the data message, being respectively stored into the target It is engaged in each corresponding data table under device.

17. device as claimed in claim 16, it is characterised in that the judgement determining module, is additionally operable to the target clothes determined Business device is the first corresponding relation of each keyword and leader's Leader servers that at least two bases are pre-saved, determines institute State each Leader servers in the corresponding destination server of each subdata information；According to each pass pre-saved Keyword and the second corresponding relation of tables of data in Leader servers, determine that each subdata information takes in each Leader Corresponding tables of data under business device；

The memory module, specifically for the storage of each subdata information is arrived into correspondence under its corresponding each Leader server Tables of data in；According to the data syn-chronization rule pre-saved, by corresponding tables of data under each Leader servers The subdata information of storage, is synchronized under remaining described destination server in each corresponding tables of data.

18. a kind of acquisition device of data, it is characterised in that the device includes：

Receiving module, for receiving data acquisition instruction, wherein data acquisition instruction includes data message to be obtained Identification information；

Determining module, for the identification information according to the data message, and the acquisition algorithm pre-saved in data platform, really It is scheduled on the corresponding destination server of data message described in data platform；

Acquisition module, for obtaining the data message in the destination server.

19. device as claimed in claim 18, it is characterised in that the determining module, puts down specifically for obtaining the data Number of servers in platform, takes to the general unique identifier UUID of identification information and number of servers of the data message Modular arithmetic；According to the modulo operation result, and the modulo operation result pre-saved and the corresponding relation of every server, really Determine the corresponding destination server of the data message.

20. device as claimed in claim 18, it is characterised in that described device also includes：

First judge module, for judging whether the corresponding attribute of the data message is master data；If not, triggering determines mould Block.

21. device as claimed in claim 20, it is characterised in that the acquisition module, judges if being additionally operable to described first Module judges the corresponding attribute of the data message for master data, obtains described in any one server in data platform Data message.

22. device as claimed in claim 18, it is characterised in that described device also includes：

Second judge module, for judging whether data type corresponding to the identification information of the data message is real time data； If it is, triggering acquisition module；

The acquisition module, specifically in each tables of data under the destination server, obtaining the data message Each subdata information corresponding to identification information；Determine that the corresponding collection of each subdata information is combined into the data message.