Summary of the invention
In view of this, the embodiment of the invention provides a kind of data storing method and system, to realize convenient access data efficiently.
To achieve these goals, the embodiment of the invention provides following technical scheme:
A kind of data storing method comprises:
After storage server receives data, according to the storage directory of the temporal information specified data of data class and data; With the form storage of data, include temporal information in the filename of described data file with data file;
After storage server receives query requests, be put in the listed files according to the data class of request inquiry and temporal information data query file and with the data file that inquires;
Come from tabulation, to extract data file, the data in the read data files successively according to resource or a plurality of threads that read of configuration establishment.
Preferably, in the embodiment of the invention, also comprise:
The storage server that is used for storing data comprises the disk array of being made up of a plurality of disc drivers, in advance described disk array is carried out logical partition;
After storage server receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of described data start a process; According to the logical partition of the described storage data of correspondence, the temporal information of data and the storage directory of data class specified data; With the form storage of data, include temporal information in the filename of described data file with data file.
Preferably, in the embodiment of the invention, also comprise:
Before the storage data, the data that strategic server will need to store are given storage server by setting strategy distribution;
Before the storage server data query, load equalizer receives query requests, is identified for carrying out the interface server of query task according to the loading condition of each interface server;
Carry out the interface server of query task and on strategic server, inquire about the storage server of having preserved these kind data, and send query requests to described storage server according to the kind of query requests data.
Preferably, in the embodiment of the invention, also comprise:
After carrying out logical partition, in each logical partition, set up multistage file directory by setting rule;
Set up the division rule of data file, the size of data file that makes storage is in the scope of setting;
During the storage data, before the storage server storage data, analyze data earlier and determine storage directory according to the foundation rule of multistage catalogue;
For the storing data files through setting-up time is set up index;
The storage server data query adopts segmentation multi-thread concurrent technology to come the path of scanning index and data file.
Preferably, in the embodiment of the invention, the described index of setting up is for setting up multiple index.
Preferably, in the embodiment of the invention, also comprise step: after index is set up, regularly upgrade index.
Preferably, in the embodiment of the invention, describedly set up multistage file directory and be by setting rule:
Set up first class catalogue, comprise two file directorys of normal data and delayed data in the first class catalogue;
The normal data catalogue is used to be stored in the data that can arrive storage system in the setting-up time scope; The delayed data catalogue is used to store the data of the system that arrives above setting-up time;
In each first class catalogue, set up identical second-level directory; Second-level directory is the file directory of miscellaneous service, and data are by the class of service classification, and the data of every kind of business are stored under the corresponding with it second-level directory;
Set up three grades of identical catalogues in each second-level directory; Three grades of catalogues are by being divided into 12 file directorys month, and three grades of catalogues are used for the month storage data of data according to storage;
During the storage data, described foundation rule according to multistage catalogue determines that storage directory is:
Follow the current time relatively, determining to deposit in the normal data catalogue still is the delayed data catalogue; Judge that according to class of business needs deposit the sort of operation list in;
According to judging that needs deposit the catalogue of which month in current month;
Divide definite filename according to the division document convention of setting up in advance, data are write file;
Described establishment index is specially:
Index comprises the index corresponding with data file, gathers index and multiple index to what a plurality of index files gathered.
Preferably, in the embodiment of the invention, also comprise step:
The time that need keep according to the memory requirement of data definition miscellaneous service is the data that the unit deletion surpasses retention time with the data file when disk space is lower than the set point of configuration.
Preferably, in the embodiment of the invention, also comprise step: when existing disk space can not satisfy storage demand, increase disk partition.
A kind of system that stores data comprises,
On logical construction, be provided with:
The server that is used for the storage service storage;
Be used for giving the strategic server of storage server by setting strategy distribution with data;
Be used to carry out the interface server of query task;
The load equalizer that is used for the equally loaded of docking port server;
Carry out logical partition on a plurality of disc drivers in described storage server as required;
Storage server is that the business datum of every kind of needs storage starts a process, can create some threads according to the disk partition quantity that book server is used to store data during process initiation; Each thread is according to disk partition and professional definite storage directory of correspondence, and each thread is evenly shared the processing of this business datum by the data distribution technology;
When load equalizer is received query requests, determine interface server; Query requests is sent query requests to described interface server;
Interface server inquire about the storage server of having preserved data that should business according to the business of request on strategic server, and to described storage server transmission query requests;
Storage server is received after the query requests inquiry file and is put it in the listed files, and creates a plurality of threads that read according to resource or configuration, and each reads thread extraction document from formation successively, reads the data in the file.
In embodiments of the present invention,, and use multithreading respectively different subregions to be read and write, thereby improved utilization the IO bandwidth by a plurality of logical partitions of division in disk array.Because the bottleneck place when IO normally stores, so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.And in the embodiment of the invention, with the form storage and the taking-up data of data file, what data adopted when storing with transmission is identical form, does not need format conversion, so saved the step of data transaction, has improved the access efficiency of data.And, in the embodiment of the invention,, to carry out query task by storage server and obtain the desired data file by querying condition is issued to storage server, network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the invention, the technical scheme in the embodiment of the invention is clearly and completely described, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, those of ordinary skills belong to the scope of protection of the invention not making the every other embodiment that is obtained under the creative work prerequisite.
The invention discloses a kind of data storing method, to realize storing efficiently data and data query efficiently, as shown in Figure 1, concrete steps are as follows:
S11, on a plurality of disc drivers of the storage server that is used for storing data, carry out logical partition as required;
In actual applications, generally the disposal ability of data quantity stored and server as required disposes a plurality of disc drivers, by forming disk array and carrying out logical partition to improve the read or write speed to data on described disk array.And, carry out the capacity limit that logical partition can not be subjected to single disk on the disk array that uses a plurality of disc drivers to form, can improve the data capacity of storage server greatly.
When S12, storage, after storage server receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of described data start a process; Each thread is according to the logical partition of the described storage data of correspondence and the storage directory of data class specified data; With the form storage of data, include temporal information in the filename of described data file with data file;
In embodiments of the present invention, by dividing a plurality of logical partitions, and use multithreading respectively different logical partitions to be read and write, thereby improved utilization the IO bandwidth.Because the bottleneck place when IO normally stores, so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.Memory technology in the prior art, database for example, the different data format (coding) that uses in storage and transmission need be done Data Format Transform receiving storage like this, also needs Data Format Transform when returning inquiring.In embodiments of the present invention, adopt the mode of file storage, what data adopted when storing with transmission is identical form, does not need format conversion, so saved the step of data transaction, has improved the access efficiency of data.
In addition, make when the data query to be querying condition with time by scheme, the convenient desired data of inquiring about efficiently with the name nominating data file that includes temporal information.
When S13, data query, after storage server is received query requests, the temporal information data query file that includes in the filename according to data file also is put into the data file that inquires in the listed files, and create a plurality of threads that read according to resource or configuration and come from tabulation, to extract data file, the data in the read data files successively.
In embodiments of the present invention, by querying condition is issued to storage server, storage server is carried out query task and is obtained the desired data file, and network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
In another embodiment, as shown in Figure 2, can also include following steps:
Carry out logical partition as required on S21, a plurality of disc drivers for the storage server that is used for storing data;
In actual applications, generally the disposal ability of data quantity stored and server as required disposes a plurality of disc drivers, by forming disk array and carrying out logical partition to improve the read or write speed to data on described disk array.And, carry out the capacity limit that logical partition can not be subjected to single disk on the disk array that uses a plurality of disc drivers to form, can improve the data capacity of storage server greatly.
Before S22, the storage, the data that the usage policy server will need to store are given storage server by setting strategy distribution;
In embodiments of the present invention, with storage before storage server, first usage policy server will need the data of storing to give storage server by setting strategy distribution; Described setting strategy is normally pressed the storage and the disposal ability pro-rata of server, can guarantee that like this load of the storage server of storage data in all is kept in balance.
When S23, storage, after storage server receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of described data start a process; Each thread is according to the logical partition of the described storage data of correspondence and the storage directory of data class specified data; With the form storage of data, include temporal information in the filename of described data file with data file;
In embodiments of the present invention, by dividing a plurality of logical partitions, and use multithreading respectively different logical partitions to be read and write, thereby improved utilization the IO bandwidth.Because the bottleneck place when IO normally stores, so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.
Before S24, the data query, first working load equalizer receives query requests, is identified for carrying out the interface server of query task according to the loading condition of each interface server, thereby has played the effect of balanced each interface server load.
The interface server of S25, execution query task is inquired about the storage server of having preserved these kind data according to the kind of query requests data on strategic server, and sends query requests to described storage server.
When S26, data query, after storage server is received query requests, the temporal information data query file that includes in the filename according to data file also is put into the data file that inquires in the listed files, and create a plurality of threads that read according to resource or configuration and come from tabulation, to extract data file, the data in the read data files successively.In embodiments of the present invention, by querying condition is issued to storage server, is carried out query task and obtained the desired data file by storage server, network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
In addition,, improved the reading speed of storage server, improved the operating efficiency of storage server data by adopting the multithreading reading of data.
In another inventive embodiments, as shown in Figure 3, can also may further comprise the steps:
Carry out logical partition as required on S31, a plurality of disc drivers for the storage server that is used for storing data;
In actual applications, generally the disposal ability of data quantity stored and server as required disposes a plurality of disc drivers, by forming disk array and carrying out logical partition to improve the read or write speed to data on described disk array.And, carry out the capacity limit that logical partition can not be subjected to single disk on the disk array that uses a plurality of disc drivers to form, can improve the data capacity of storage server greatly.
S32, in each logical partition, set up multistage file directory by setting rule;
Describedly set up multistage file directory and be by setting rule:
Set up first class catalogue, comprise two file directorys of normal data and delayed data in the first class catalogue;
The normal data catalogue is used to be stored in the data that can arrive storage system in the setting-up time scope; The delayed data catalogue is used to store the data of the system that arrives above setting-up time;
In each first class catalogue, set up identical second-level directory; Second-level directory is the file directory of miscellaneous service, and data are by the class of service classification, and the data of every kind of business are stored under the corresponding with it second-level directory;
Set up three grades of identical catalogues in each second-level directory; Three grades of catalogues are by being divided into 12 file directorys month, and three grades of catalogues are used for the month storage data of data according to storage.
By setting up catalogue, the storage data can be stored by setting rule classification, thereby can set up respective index, and then made things convenient for the inquiry of data for the storage data by the rule of setting.In embodiments of the present invention, the storage The data multistage catalogue, thereby guaranteed that the file under each catalogue can be not too many, guaranteed the efficient of retrieving files.Simultaneously also for moving or Backup Data provides convenience, when we need certain data, only need to copy a catalogue rather than in a lot of files, going to search.
S33, set up the division rule of data file, the size that makes data file is in the scope of setting; According to data volume can have by the hour, by the sky, by the week, monthly wait dividing mode, guarantee that a data file can be not excessive.Can not cause a large amount of small documents, too many small documents can influence search efficiency.
Memory technology in the prior art, database for example, the different data format (coding) that uses in storage and transmission need be done Data Format Transform receiving storage like this, also needs Data Format Transform when returning inquiring.In embodiments of the present invention, adopt the mode of file storage, what data adopted when storing with transmission is identical form, does not need format conversion, so saved the step of data transaction, has improved the access efficiency of data.
Before S34, the storage, the data that the usage policy server will need to store are given storage server by setting strategy distribution;
In embodiments of the present invention, with storage before storage server, first usage policy server will need the data of storing to give storage server by setting strategy distribution; Described setting strategy is normally pressed the storage and the disposal ability pro-rata of server, can guarantee that like this load of the storage server of storage data in all is kept in balance.
S35, storage be during data, receive the data that need storage at storage server after, be that every kind of described data start a process, be that each logical partition that is used to store data in the book server is created thread during process initiation; Thread is determined storage directory according to the foundation rule of multistage catalogue; With the form storage of data, include temporal information in the filename of described data file with data file;
Described foundation rule according to multistage catalogue determines that storage directory is:
Follow the current time relatively, determining to deposit in the normal data catalogue still is the delayed data catalogue; Judge that according to class of business needs deposit the sort of operation list in;
According to judging that needs deposit the catalogue of which month in current month;
Divide definite filename according to the division document convention of setting up in advance, data are write file.
Store data according to the directory rules of setting, thereby make things convenient for the inquiry of data.In embodiments of the present invention, the storage The data multistage catalogue, thereby guaranteed that the file under each catalogue can be not too many, guaranteed the efficient of retrieving files.Simultaneously also for moving or Backup Data provides convenience, when we need certain data, only need to copy a catalogue rather than in a lot of files, going to search.
S36, through behind the setting-up time, for data file is set up index file; Index file comprises the index corresponding with data file, gathers index and multiple index to what a plurality of index files gathered.
The described index of setting up is for setting up multiple index.After index file is set up, regularly upgrade index file.
By setting up index, make when the data query file, reduced the number of scanning document, improved the performance of inquiry file, and the multi-stage data index has further reduced the number of scanning document, has reduced the number of times of opening close file, has improved the efficient of data query.
Before S37, the data query, first working load equalizer receives query requests, is identified for carrying out the interface server of query task according to the loading condition of each interface server;
The interface server of S38, execution query task is inquired about the storage server of having preserved these kind data according to the kind of query requests data on strategic server, and sends query requests to described storage server.
When S39, described storage server data query, adopt segmentation multi-thread concurrent technology to come the path of scanning index and data file earlier.If there is not index file, the temporal information data query file that then includes in the filename according to data file.
In embodiments of the present invention, the meeting scanning index if created index file, and has relevant querying condition then in the query script of data in query requests, then can utilize index file.It is multistage to check that index is divided into again, checks and gathers index, if there be preferential use of the index of gathering to gather index, if index or data file are old when gathering the index score, then need not gather index, but inquire about by scan data file.
S40, storage server will inquire data file and put it in the listed files, and come to extract data file, the data in the read data files successively from tabulation according to resource or a plurality of threads that read of configuration establishment.In embodiments of the present invention, by querying condition is issued to storage server, storage server is carried out query task, obtain the desired data file after, network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
In addition,, improved the reading speed of storage server, improved the operating efficiency of storage server data by adopting the multithreading reading of data.
In another embodiment, among above-mentioned all embodiment, also comprise step: according to the memory requirement of data, time that the definition miscellaneous service need keep and the freespace threshold in the logical partition, when the free space in the logical partition was lower than the threshold values of setting, deletion surpassed the data of retention time;
Because in embodiments of the present invention, data are the unit storage with the file, so during deletion, be that unit deletes data and gets final product with the file.
In another embodiment, among above-mentioned all embodiment, also comprise:
When can not satisfying storage demand, existing disk space increases logical partition.
In embodiments of the present invention, when existing space is not enough to satisfy storage demand on the storage server, adopt the method that increases logical partition to realize the expansion of memory space, new logical partition adds fashionable, because the data volume that each logical partition receives is balanced, newly-increased logical partition data volume can be less, the data volume of old logical partition can be bigger, but as long as the logical partition increase is timely, the newly-increased data volume of old logical partition is shared, and the data volume that newly writes is suitable with the data volume of safeguarding deletion, loses the problem of data with regard to not causing old logical partition insufficient space, As time goes on, each logical partition data volume reaches balanced consistent.
In addition, as shown in Figure 4, a kind of storage data system also is provided in the embodiment of the invention, comprise, be used to store data storage server 1, be used for by the data of setting strategy and will needing to store be distributed to storage server strategic server 2, be used to carry out the interface server 3 of query task and be used for the load equalizer 4 that loading condition according to each interface server is identified for carrying out the interface server of query task;
At first, on a plurality of disc drivers of the storage server 1 that is used for storing data, carry out logical partition as required.
The embodiment of the invention passes through to divide a plurality of logical partitions, and uses multithreading respectively different logical partitions to be read and write, thereby has improved the utilization to the IO bandwidth.Because the bottleneck place when IO normally stores, so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.
During storage, the data that first usage policy server 2 will need to store are given storage server 1 by setting strategy distribution; After storage server 1 receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of described data start a process; Each thread is according to the storage directory of the logical partition of the described storage data of correspondence and data class specified data and store data, and each thread is evenly shared the processing of this business datum by the data distribution technology;
During data query, first working load equalizer 4 receives query requests, is identified for carrying out the interface server 3 of query task according to the loading condition of each interface server 3;
Carry out interface service 3 devices of query task and on strategic server 2, inquire about the storage server 1 of having preserved these kind data, and send query requests to described storage server 1 according to the kind of query requests data.
Storage server 1 is received after the query requests data query file and is put it in the listed files, and creates a plurality of threads that read according to resource or configuration and come to extract data file, the data in the read data files successively from tabulation.
The segmentation multi-thread concurrent technology that scan data file adopts when the data query file in the embodiment of the invention, this multi-thread concurrent technology can improve search efficiency greatly.And the data that read from file in the embodiment of the invention can be filtered at book server according to querying condition, return to interface server after the filtration again, have significantly reduced transmission volume.
Each embodiment adopts the mode of going forward one by one to describe in this specification, and what each embodiment stressed all is and the difference of other embodiment that identical similar part is mutually referring to getting final product between each embodiment.
To the above-mentioned explanation of the disclosed embodiments, make this area professional and technical personnel can realize or use the present invention.Multiple modification to these embodiment will be conspicuous concerning those skilled in the art, and defined herein General Principle can realize under the situation that does not break away from the spirit or scope of the present invention in other embodiments.Therefore, the present invention will can not be restricted to these embodiment shown in this article, but will meet and principle disclosed herein and features of novelty the wideest corresponding to scope.