CN101795211B - Data storage method and system - Google Patents
Data storage method and system Download PDFInfo
- Publication number
- CN101795211B CN101795211B CN2010100340698A CN201010034069A CN101795211B CN 101795211 B CN101795211 B CN 101795211B CN 2010100340698 A CN2010100340698 A CN 2010100340698A CN 201010034069 A CN201010034069 A CN 201010034069A CN 101795211 B CN101795211 B CN 101795211B
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- server
- file
- storage server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a stored data method and a system. In the invention, a disk array is divided into a plurality of logic subareas, and multiple threads are used for respectively reading and writing different subareas, thereby enhancing the utilization of IO bandwidth. Since IO is usually the bottleneck of storage, so the enhancement of the utilization of IO bandwidth enhances the data throughput of the storage server. In addition, in the embodiment of the invention, data is stored and retrieved in the form of data files; and both the storage and the transmission of the data use the same format and do not need format conversion, thereby saving the step of data conversion and enhancing the data access efficiency. Furthermore, in the embodiment of the invention, by transmitting the query conditions to a storage server, the storage server executes the query task and acquires the required data, and the network only transmits the required data, thereby reducing the data transmission quantity and enhancing the efficiency.
Description
Technical field
The present invention relates to data processing field, more particularly, relate to a kind of method and system of storage.
Background technology
Present communication network monitoring system, what face is the network signal data of magnanimity, these the business datum every days that need store are up to the TB magnitude.These business datums are used usually and are increased, look into, delete three kinds of visits in increasing, delete, change, looking in the preservation process, almost do not revise the demand of data, and deletion also only need be deleted outdated data forward by the time.That is to say that for business datum, most being operating as increases and inquire about two kinds.In addition, as time passes, the continuous development of telecommunication service, the continuous growth of traffic carrying capacity has also caused data volume constantly to increase, storage system also must be can expand thereupon,
In the prior art; For such mass data; Storage needs distributed, generally is to use database to pass through Distributed Storage (cloud computing) with disperseing to be stored on the network by certain algorithm behind the data fragmentation, to satisfy its demand on space and performance.The characteristics of inquiry are then normally by professional or further be accurate to certain certain professional keyword.
But, because general Distributed Storage (cloud computing) is generally with disperseing to be stored on the network by certain algorithm behind the data fragmentation, so for storage, the retrieval of mass data with move, can take a large amount of network bandwidths.Will be very high to the requirement of network service, be difficult to efficient location and extraction.And then make that technical scheme storage speed of the prior art is slow, and be difficult to writing fast of satisfying magnanimity data, also need high acquisition expenses and maintenance cost simultaneously.
Summary of the invention
In view of this, the embodiment of the invention provides a kind of data storing method and system, to realize convenient access data efficiently.
To achieve these goals, the embodiment of the invention provides following technical scheme:
A kind of data storing method comprises:
After storage server receives data, according to the storage directory of the temporal information specified data of data class and data; With the stored in form of data, include temporal information in the filename of said data file with data file;
After storage server receives query requests, be put in the listed files according to the data class of requesting query and temporal information data query file and with the data file that inquires;
Come from tabulation, to extract data file, the data in the read data files successively based on resource or a plurality of threads that read of configuration establishment.
Preferably, in the embodiment of the invention, also comprise:
The storage server that is used for storing data comprises the disk array of being made up of a plurality of disc drivers, in advance said disk array is carried out logical partition;
After storage server receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of said data start a process; According to the logical partition of the said storage data of correspondence, the temporal information of data and the storage directory of data class specified data; With the stored in form of data, include temporal information in the filename of said data file with data file.
Preferably, in the embodiment of the invention, also comprise:
Before the storage data, the data that strategic server will be stored are given storage server by setting strategy distribution;
Before the storage server data query, load equalizer receives query requests, confirms to be used to carry out the interface server of query task according to the loading condition of each interface server;
Carry out the interface server of query task and on strategic server, inquire about the storage server of having preserved these kind data, and send query requests to said storage server according to the kind of query requests data.
Preferably, in the embodiment of the invention, also comprise:
After carrying out logical partition, in each logical partition, set up multistage file directory by setting rule;
Set up the division rule of data file, the size of data file that makes storage is in the scope of setting;
During the storage data, before the storage server storage data, analyze data earlier and confirm storage directory according to the foundation rule of multistage catalogue;
For the storing data files through setting-up time is set up index;
The storage server data query adopts segmentation multi-thread concurrent technology to come the path of scanning index and data file.
Preferably, in the embodiment of the invention, the said index of setting up is for setting up multiple index.
Preferably, in the embodiment of the invention, also comprise step: after index is set up, regularly upgrade index.
Preferably, in the embodiment of the invention, saidly set up multistage file directory and be by setting rule:
Set up first class catalogue, comprise two file directorys of normal data and delayed data in the first class catalogue;
The normal data catalogue is used to be stored in the data that can arrive storage system in the setting-up time scope; The delayed data catalogue is used to store the data of the system that arrives above setting-up time;
In each first class catalogue, set up identical second-level directory; Second-level directory is the file directory of miscellaneous service, and data are by the class of service classification, and the data of every kind of business are stored under the corresponding with it second-level directory;
Set up three grades of identical catalogues in each second-level directory; Three grades of catalogues are by being divided into 12 file directorys month, and three grades of catalogues are used for the month storage data of data according to storage;
During the storage data, said foundation rule according to multistage catalogue confirms that storage directory is:
Follow the current time relatively, confirming to deposit in the normal data catalogue still is the delayed data catalogue; Deposit the sort of operation list in according to class of business judgement needs;
Deposit the catalogue of which month in according to current month judgement needs;
Divide definite filename based on the division document convention of setting up in advance, data are write file;
Said establishment index is specially:
Index comprises the index corresponding with data file, gathers index and multiple index to what a plurality of index files gathered.
Preferably, in the embodiment of the invention, also comprise step:
The time that need keep according to the memory requirement of data definition miscellaneous service is the data that the unit deletion surpasses retention time with the data file when disk space is lower than the set point of configuration.
Preferably, in the embodiment of the invention, also comprise step: when existing disk space can not satisfy storage demand, increase disk partition.
A kind of system that stores data comprises,
On logical construction, be provided with:
The server that is used for the storage service storage;
Be used for giving the strategic server of storage server by setting strategy distribution with data;
Be used to carry out the interface server of query task;
The load equalizer that is used for the equally loaded of docking port server;
Carry out logical partition on a plurality of disc drivers in said storage server as required;
Storage server is that the business datum of every kind of needs storage starts a process, can create some threads according to the disk partition quantity that book server is used to store data during process initiation; Each thread is according to the disk partition and professional definite storage directory of correspondence, and each thread is evenly shared the processing of this business datum through the data distribution technology;
When load equalizer is received query requests, confirm interface server; Query requests is sent query requests to said interface server;
Interface server inquire about the storage server of having preserved data that should business based on the business of request on strategic server, and to said storage server transmission query requests;
Storage server is received after the query requests inquiry file and is put it in the listed files, and creates a plurality of threads that read according to resource or configuration, and each reads thread extraction document from formation successively, reads the data in the file.
In embodiments of the present invention, through a plurality of logical partitions of division in disk array, and use multithreading respectively different partition areas to be read and write, thereby improved utilization the IO bandwidth.Because the place of the bottleneck when IO normally stores is so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.And in the embodiment of the invention, with the stored in form and the taking-up data of data file, what data adopted when storing with transmission is identical form, does not need format conversion, so saved the step of data transaction, has improved the access efficiency of data.And, in the embodiment of the invention,, to carry out query task by storage server and obtain the desired data file through querying condition is issued to storage server, network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art; To do to introduce simply to the accompanying drawing of required use in embodiment or the description of the Prior Art below; Obviously, the accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills; Under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the flow chart of steps of method described in the embodiment of the invention;
Fig. 2 is the flow chart of steps of method described in further embodiment of this invention;
Fig. 3 is the flow chart of steps of method described in further embodiment of this invention;
Fig. 4 is the composition sketch map of system described in the embodiment of the invention.
Embodiment
To combine the accompanying drawing in the embodiment of the invention below, the technical scheme in the embodiment of the invention is carried out clear, intactly description, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, those of ordinary skills are not making the every other embodiment that is obtained under the creative work prerequisite, all belong to the scope of the present invention's protection.
The invention discloses a kind of data storing method, to realize storing efficiently data and data query efficiently, as shown in Figure 1, concrete steps are following:
S11, on a plurality of disc drivers of the storage server that is used for storing data, carry out logical partition as required;
In practical application, generally the disposal ability of data quantity stored and server as required disposes a plurality of disc drivers, through forming disk array and on said disk array, carrying out logical partition to improve the read or write speed to data.And, carry out the capacity limit that logical partition can not receive single disk on the disk array that uses a plurality of disc drivers to form, can improve the data capacity of storage server greatly.
When S12, storage, after storage server receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of said data start a process; Each thread is according to the logical partition of the said storage data of correspondence and the storage directory of data class specified data; With the stored in form of data, include temporal information in the filename of said data file with data file;
In embodiments of the present invention, through dividing a plurality of logical partitions, and use multithreading respectively the different logical subregion to be read and write, thereby improved utilization the IO bandwidth.Because the place of the bottleneck when IO normally stores is so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.Memory technology in the prior art, database for example, the different data format (coding) that in storage and transmission, uses need be done Data Format Transform receiving storage like this, also needs Data Format Transform when returning inquiring.In embodiments of the present invention, adopt the mode of file storage, what data adopted when storing with transmission is identical form, does not need format conversion, so saved the step of data transaction, has improved the access efficiency of data.
In addition, make when the data query to be querying condition with time through scheme, the convenient desired data of inquiring about efficiently with the name nominating data file that includes temporal information.
When S13, data query; After storage server is received query requests; The temporal information data query file that includes in the filename according to data file also is put into the data file that inquires in the listed files; And create a plurality of threads that read according to resource or configuration and come from tabulation, to extract data file, the data in the read data files successively.
In embodiments of the present invention, through querying condition is issued to storage server, storage server is carried out query task and is obtained the desired data file, and network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
In another embodiment, as shown in Figure 2, can also include following steps:
Carry out logical partition as required on S21, a plurality of disc drivers for the storage server that is used for storing data;
In practical application, generally the disposal ability of data quantity stored and server as required disposes a plurality of disc drivers, through forming disk array and on said disk array, carrying out logical partition to improve the read or write speed to data.And, carry out the capacity limit that logical partition can not receive single disk on the disk array that uses a plurality of disc drivers to form, can improve the data capacity of storage server greatly.
Before S22, the storage, the data that the usage policy server will be stored are given storage server by setting strategy distribution;
In embodiments of the present invention, with storage before storage server, the data that first usage policy server will be stored are given storage server by setting strategy distribution; Said setting strategy is normally pressed the storage and the disposal ability pro-rata of server, can guarantee that like this load of the storage server of storage data in all is kept in balance.
When S23, storage, after storage server receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of said data start a process; Each thread is according to the logical partition of the said storage data of correspondence and the storage directory of data class specified data; With the stored in form of data, include temporal information in the filename of said data file with data file;
In embodiments of the present invention, through dividing a plurality of logical partitions, and use multithreading respectively the different logical subregion to be read and write, thereby improved utilization the IO bandwidth.Because the place of the bottleneck when IO normally stores is so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.
Before S24, the data query, first working load equalizer receives query requests, confirms to be used to carry out the interface server of query task according to the loading condition of each interface server, thereby has played the effect of balanced each interface server load.
The interface server of S25, execution query task is inquired about the storage server of having preserved these kind data according to the kind of query requests data on strategic server, and sends query requests to said storage server.
When S26, data query; After storage server is received query requests; The temporal information data query file that includes in the filename according to data file also is put into the data file that inquires in the listed files; And create a plurality of threads that read according to resource or configuration and come from tabulation, to extract data file, the data in the read data files successively.In embodiments of the present invention, through querying condition is issued to storage server, is carried out query task and obtained the desired data file by storage server, network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
In addition,, improved the reading speed of storage server, improved the operating efficiency of storage server data through adopting the multithreading reading of data.
In another inventive embodiments, as shown in Figure 3, can also may further comprise the steps:
Carry out logical partition as required on S31, a plurality of disc drivers for the storage server that is used for storing data;
In practical application, generally the disposal ability of data quantity stored and server as required disposes a plurality of disc drivers, through forming disk array and on said disk array, carrying out logical partition to improve the read or write speed to data.And, carry out the capacity limit that logical partition can not receive single disk on the disk array that uses a plurality of disc drivers to form, can improve the data capacity of storage server greatly.
S32, in each logical partition, set up multistage file directory by setting rule;
Saidly set up multistage file directory and be by setting rule:
Set up first class catalogue, comprise two file directorys of normal data and delayed data in the first class catalogue;
The normal data catalogue is used to be stored in the data that can arrive storage system in the setting-up time scope; The delayed data catalogue is used to store the data of the system that arrives above setting-up time;
In each first class catalogue, set up identical second-level directory; Second-level directory is the file directory of miscellaneous service, and data are by the class of service classification, and the data of every kind of business are stored under the corresponding with it second-level directory;
Set up three grades of identical catalogues in each second-level directory; Three grades of catalogues are by being divided into 12 file directorys month, and three grades of catalogues are used for the month storage data of data according to storage.
Through setting up catalogue, the storage data can be stored by setting rule classification, thereby can set up respective index, and then made things convenient for the inquiry of data for the storage data by the rule of setting.In embodiments of the present invention, the storage The data multistage catalogue, thereby guaranteed that the file under each catalogue can be not too many, guaranteed the efficient of retrieving files.Simultaneously also for moving or Backup Data provides convenience, when we need certain data, only need copy a catalogue rather than in a lot of files, going to search.
S33, set up the division rule of data file, the size that makes data file is in the scope of setting; According to data volume can have by the hour, by the sky, by the week, monthly wait dividing mode, guarantee that a data files can be not excessive.Can not cause a large amount of small documents, too many small documents can influence search efficiency.
Memory technology in the prior art, database for example, the different data format (coding) that in storage and transmission, uses need be done Data Format Transform receiving storage like this, also needs Data Format Transform when returning inquiring.In embodiments of the present invention, adopt the mode of file storage, what data adopted when storing with transmission is identical form, does not need format conversion, so saved the step of data transaction, has improved the access efficiency of data.
Before S34, the storage, the data that the usage policy server will be stored are given storage server by setting strategy distribution;
In embodiments of the present invention, with storage before storage server, the data that first usage policy server will be stored are given storage server by setting strategy distribution; Said setting strategy is normally pressed the storage and the disposal ability pro-rata of server, can guarantee that like this load of the storage server of storage data in all is kept in balance.
S35, storage be during data, receive the data that need storage at storage server after, be that every kind of said data start a process, be that each logical partition that is used to store data in the book server is created thread during process initiation; Thread is confirmed storage directory according to the foundation rule of multistage catalogue; With the stored in form of data, include temporal information in the filename of said data file with data file;
Said foundation rule based on multistage catalogue confirms that storage directory is:
Follow the current time relatively, confirming to deposit in the normal data catalogue still is the delayed data catalogue; Deposit the sort of operation list in according to class of business judgement needs;
Deposit the catalogue of which month in according to current month judgement needs;
Divide definite filename based on the division document convention of setting up in advance, data are write file.
Store data according to the directory rules of setting, thereby make things convenient for the inquiry of data.In embodiments of the present invention, the storage The data multistage catalogue, thereby guaranteed that the file under each catalogue can be not too many, guaranteed the efficient of retrieving files.Simultaneously also for moving or Backup Data provides convenience, when we need certain data, only need copy a catalogue rather than in a lot of files, going to search.
S36, through behind the setting-up time, for data file is set up index file; Index file comprises the index corresponding with data file, gathers index and multiple index to what a plurality of index files gathered.
The said index of setting up is for setting up multiple index.After index file is set up, regularly upgrade index file.
Through setting up index, make when the data query file, reduced the number of scanning document; Improved the performance of inquiry file; And the multi-stage data index has further reduced the number of scanning document, has reduced the number of times of opening close file, has improved the efficient of data query.
Before S37, the data query, first working load equalizer receives query requests, confirms to be used to carry out the interface server of query task according to the loading condition of each interface server;
The interface server of S38, execution query task is inquired about the storage server of having preserved these kind data according to the kind of query requests data on strategic server, and sends query requests to said storage server.
When S39, said storage server data query, adopt segmentation multi-thread concurrent technology to come the path of scanning index and data file earlier.If there is not index file, the temporal information data query file that then includes in the filename according to data file.
In embodiments of the present invention, the meeting scanning index if created index file, and has relevant querying condition then in the query script of data in query requests, then can utilize index file.The inspection index is divided into multistage again, and inspection gathers index, if there be preferential use of the index of gathering to gather index, if index or data file are old when gathering the index score, then need not gather index, but inquire about through scan data file.
S40, storage server will inquire data file and put it in the listed files, and come from tabulation, to extract data file, the data in the read data files successively based on resource or a plurality of threads that read of configuration establishment.In embodiments of the present invention, through querying condition is issued to storage server, storage server is carried out query task, obtain the desired data file after, network only transmits the desired data file, thereby reduces volume of transmitted data, raises the efficiency.Avoided in the prior art, read total data earlier when using cloud computing system and then through the query filter scheme, thus the network congestion problem that needs transferring large number of data to bring that causes.
In addition,, improved the reading speed of storage server, improved the operating efficiency of storage server data through adopting the multithreading reading of data.
In another embodiment; Among above-mentioned all embodiment, also comprise step: based on the memory requirement of data, time that the definition miscellaneous service need keep and the freespace threshold in the logical partition; When the free space in the logical partition was lower than the threshold values of setting, deletion surpassed the data of retention time;
Because in embodiments of the present invention, data are the unit storage with the file, so during deletion, be that unit deletes data and gets final product with the file.
In another embodiment, among above-mentioned all embodiment, also comprise:
, existing disk space increases logical partition when can not satisfying storage demand.
In embodiments of the present invention, when existing space is not enough to satisfy storage demand on the storage server, adopt the method that increases logical partition to realize the expansion of memory space; New logical partition adds fashionable, because the data volume that each logical partition receives is balanced, newly-increased logical partition data volume can be less; The data volume of old logical partition can be bigger, but need only timely that logical partition increases, and the newly-increased data volume of old logical partition is shared; The data volume that newly writes is suitable with the data volume of safeguarding deletion; Lose the problem of data with regard to not causing old logical partition insufficient space, As time goes on, it is balanced consistent that each logical partition data volume reaches.
In addition; As shown in Figure 4; A kind of storage data system also is provided in the embodiment of the invention; Comprise, be used to store data storage server 1, be used for by setting strategic server 2 that data that strategy will store are distributed to storage server, be used to carry out the interface server 3 of query task and be used for confirming to be used to carry out the load equalizer 4 of the interface server of query task according to the loading condition of each interface server;
At first, on a plurality of disc drivers of the storage server 1 that is used for storing data, carry out logical partition as required.
The embodiment of the invention passes through to divide a plurality of logical partitions, and uses multithreading respectively the different logical subregion to be read and write, thereby has improved the utilization to the IO bandwidth.Because the place of the bottleneck when IO normally stores is so improved the data throughput capabilities that the utilization of IO bandwidth has also just been improved storage server.
During storage, the data that first usage policy server 2 will be stored are given storage server 1 by setting strategy distribution; After storage server 1 receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of said data start a process; Each thread is according to the storage directory of the logical partition of the said storage data of correspondence and data class specified data and store data, and each thread is evenly shared the processing of this business datum through the data distribution technology;
During data query, first working load equalizer 4 receives query requests, confirms to be used to carry out the interface server 3 of query task according to the loading condition of each interface server 3;
Carry out interface service 3 devices of query task and on strategic server 2, inquire about the storage server 1 of having preserved these kind data, and send query requests to said storage server 1 according to the kind of query requests data.
The segmentation multi-thread concurrent technology that scan data file adopts when the data query file in the embodiment of the invention, this multi-thread concurrent technology can improve search efficiency greatly.And the data that from file, read in the embodiment of the invention can be filtered at book server according to querying condition, return to interface server after the filtration again, have significantly reduced transmission volume.
Each embodiment adopts the mode of going forward one by one to describe in this specification, and what each embodiment stressed all is and the difference of other embodiment that identical similar part is mutually referring to getting final product between each embodiment.
To the above-mentioned explanation of the disclosed embodiments, make this area professional and technical personnel can realize or use the present invention.Multiple modification to these embodiment will be conspicuous concerning those skilled in the art, and defined General Principle can realize under the situation that does not break away from the spirit or scope of the present invention in other embodiments among this paper.Therefore, the present invention will can not be restricted to these embodiment shown in this paper, but will meet and principle disclosed herein and features of novelty the wideest corresponding to scope.
Claims (10)
1. a data storing method is characterized in that, comprising:
After storage server receives data, according to the storage directory of the temporal information specified data of data class and data; With the stored in form of data, include temporal information in the filename of said data file with data file;
After storage server receives query requests, be put in the listed files according to the data class of requesting query and temporal information data query file and with the data file that inquires;
Come from tabulation, to extract data file, the data in the read data files successively based on resource or a plurality of threads that read of configuration establishment.
2. according to the said method of claim 1, it is characterized in that, comprising:
The storage server that is used for storing data comprises the disk array of being made up of a plurality of disc drivers, in advance said disk array is carried out logical partition;
After storage server receives data,, during process initiation each logical partition establishment thread that is used to store data in the book server for every kind of said data start a process; According to the logical partition of the said storage data of correspondence, the temporal information of data and the storage directory of data class specified data; With the stored in form of data, include temporal information in the filename of said data file with data file.
3. according to claim 1 or 2 said methods, it is characterized in that, also comprise:
Before the storage data, the data that strategic server will be stored are given storage server by setting strategy distribution;
Before the storage server data query, load equalizer receives query requests, confirms to be used to carry out the interface server of query task according to the loading condition of each interface server;
Carry out the interface server of query task and on strategic server, inquire about the storage server of having preserved these kind data, and send query requests to said storage server according to the kind of query requests data.
4. according to the said method of claim 3, it is characterized in that, also comprise:
After carrying out logical partition, in each logical partition, set up multistage file directory by setting rule;
Set up the division rule of data file, the size of data file that makes storage is in the scope of setting;
During the storage data, before the storage server storage data, analyze data earlier and confirm storage directory according to the foundation rule of multistage catalogue;
For the storing data files through setting-up time is set up index;
The storage server data query adopts segmentation multi-thread concurrent technology to come the path of scanning index and data file.
5. according to the said method of claim 4, it is characterized in that the said index of setting up is for setting up multiple index.
6. according to the said method of claim 5, it is characterized in that, also comprise step: after index is set up, regularly upgrade index.
7. according to the said method of claim 6, it is characterized in that, saidly set up multistage file directory and be by setting rule:
Set up first class catalogue, comprise two file directorys of normal data and delayed data in the first class catalogue;
The normal data catalogue is used to be stored in the data that can arrive storage system in the setting-up time scope; The delayed data catalogue is used to store the data of the system that arrives above setting-up time;
In each first class catalogue, set up identical second-level directory; Second-level directory is the file directory of miscellaneous service, and data are by the class of service classification, and the data of every kind of business are stored under the corresponding with it second-level directory;
Set up three grades of identical catalogues in each second-level directory; Three grades of catalogues are by being divided into 12 file directorys month, and three grades of catalogues are used for the month storage data of data according to storage;
During the storage data, said foundation rule according to multistage catalogue confirms that storage directory is:
Follow the current time relatively, confirming to deposit in the normal data catalogue still is the delayed data catalogue;
Deposit the sort of operation list in according to class of business judgement needs;
Deposit the catalogue of which month in according to current month judgement needs;
Divide definite filename based on the division document convention of setting up in advance, data are write file;
Said establishment index is specially:
Index comprises the index corresponding with data file, gathers index and multiple index to what a plurality of index files gathered.
8. according to the said method of claim 7, it is characterized in that, also comprise step:
The time that need keep according to the memory requirement of data definition miscellaneous service is the data that the unit deletion surpasses retention time with the data file when disk space is lower than the set point of configuration.
9. said according to Claim 8 method is characterized in that, also comprises step: when existing disk space can not satisfy storage demand, increase disk partition.
10. a system that stores data is characterized in that, comprising:
On logical construction, be provided with:
The storage server that is used for the storage service data;
Be used for giving the strategic server of storage server by setting strategy distribution with data;
Be used to carry out the interface server of query task;
The load equalizer that is used for the equally loaded of docking port server;
Carry out logical partition on a plurality of disc drivers in said storage server as required;
Storage server is that the business datum of every kind of needs storage starts a process, can create some threads according to the disk partition quantity that book server is used to store data during process initiation; Each thread is according to the disk partition and professional definite storage directory of correspondence, and each thread is evenly shared the processing of this business datum through the data distribution technology;
When load equalizer is received query requests, confirm interface server; Query requests is sent query requests to said interface server;
Interface server inquire about the storage server of having preserved data that should business based on the business of request on strategic server, and to said storage server transmission query requests;
Storage server is received after the query requests inquiry file and is put it in the listed files, and creates a plurality of threads that read according to resource or configuration, and each reads thread extraction document from formation successively, reads the data in the file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010100340698A CN101795211B (en) | 2010-01-13 | 2010-01-13 | Data storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010100340698A CN101795211B (en) | 2010-01-13 | 2010-01-13 | Data storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101795211A CN101795211A (en) | 2010-08-04 |
| CN101795211B true CN101795211B (en) | 2012-02-22 |
Family
ID=42587643
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010100340698A Expired - Fee Related CN101795211B (en) | 2010-01-13 | 2010-01-13 | Data storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101795211B (en) |
Families Citing this family (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102184190A (en) * | 2011-04-19 | 2011-09-14 | 北京神州数码思特奇信息技术股份有限公司 | Data comparison method |
| CN102664789B (en) * | 2012-04-09 | 2016-08-17 | 北京百度网讯科技有限公司 | The processing method of a kind of large-scale data and system |
| CN102752387B (en) * | 2012-06-29 | 2015-12-02 | 用友软件股份有限公司 | Data storage processing system and data storage handling method |
| CN103577330B (en) * | 2012-07-27 | 2018-09-11 | 中兴通讯股份有限公司 | A kind of content storage method and device |
| CN103023970B (en) * | 2012-11-15 | 2015-07-22 | 中国科学院计算机网络信息中心 | Method and system for storing mass data of Internet of Things (IoT) |
| CN103353892A (en) * | 2013-07-05 | 2013-10-16 | 北京东方网信科技股份有限公司 | Method and system for data cleaning suitable for mass storage |
| CN104572648B (en) * | 2013-10-11 | 2018-01-16 | 中国石油化工股份有限公司 | A kind of storage statistical system and method based on high-performance calculation |
| CN103810256B (en) * | 2014-01-24 | 2017-09-26 | 武汉虹翼信息有限公司 | Method based on partitioning technique quick distribution data in big data network optimization platform |
| CN104049917A (en) * | 2014-06-25 | 2014-09-17 | 北京思特奇信息技术股份有限公司 | Data processing method and system |
| CN105868197B (en) * | 2015-01-20 | 2019-09-10 | 中移信息技术有限公司 | A kind of statistical method and statistic device of call bill data |
| CN104699839B (en) * | 2015-03-31 | 2021-03-02 | 北京奇艺世纪科技有限公司 | File searching method and device |
| CN106326280B (en) * | 2015-06-30 | 2021-06-29 | 中兴通讯股份有限公司 | Data processing method, device and system |
| CN105138571B (en) * | 2015-07-24 | 2019-12-24 | 四川长虹电器股份有限公司 | Distributed file system and method for storing massive small files |
| US10706368B2 (en) * | 2015-12-30 | 2020-07-07 | Veritas Technologies Llc | Systems and methods for efficiently classifying data objects |
| CN105743772A (en) * | 2016-01-26 | 2016-07-06 | 深圳宸睿科技有限公司 | Message processing method and system |
| CN106844721A (en) * | 2017-02-09 | 2017-06-13 | 济南浪潮高新科技投资发展有限公司 | Date storage method, device, system, computer-readable recording medium and storage control |
| CN107451301B (en) * | 2017-09-12 | 2021-01-08 | 彩讯科技股份有限公司 | Processing method, device, equipment and storage medium for real-time delivery bill mail |
| CN108984659A (en) * | 2018-06-28 | 2018-12-11 | 山东浪潮商用系统有限公司 | A kind of file equalization methods for IDFS |
| CN109271361B (en) * | 2018-08-13 | 2020-07-24 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage method and system for massive small files |
| CN109522152B (en) * | 2018-09-06 | 2021-05-14 | 清华大学 | Disaster recovery method, storage medium and device for electron microscope data |
| CN109491797A (en) * | 2018-11-26 | 2019-03-19 | 深圳市斯博科技有限公司 | Date storage method, system and computer readable storage medium |
| CN110955704A (en) * | 2019-12-03 | 2020-04-03 | 北京锐安科技有限公司 | Data management method, device, equipment and storage medium |
| CN112866386A (en) * | 2021-01-19 | 2021-05-28 | 青岛越超传媒有限公司 | Data storage data construction method based on cloud computing |
| CN113515520A (en) * | 2021-03-26 | 2021-10-19 | 北京达佳互联信息技术有限公司 | Data management method, device, server and storage medium |
| CN113761003A (en) * | 2021-04-15 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | User portrait data processing method and device, electronic equipment and readable storage medium |
| CN113625967B (en) * | 2021-07-26 | 2024-05-03 | 深圳市汉云科技有限公司 | Data storage method, data query method and server |
| CN115964754B (en) * | 2023-03-16 | 2023-05-23 | 长城数字能源(西安)科技有限公司 | Big data secure storage method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101184132A (en) * | 2007-12-19 | 2008-05-21 | 中国移动通信集团黑龙江有限公司 | Memory file and real-time synchronization technique based fast fixed charges computing system |
| CN101436207A (en) * | 2008-12-16 | 2009-05-20 | 浪潮通信信息系统有限公司 | Data restoring and synchronizing method based on log snapshot |
-
2010
- 2010-01-13 CN CN2010100340698A patent/CN101795211B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101184132A (en) * | 2007-12-19 | 2008-05-21 | 中国移动通信集团黑龙江有限公司 | Memory file and real-time synchronization technique based fast fixed charges computing system |
| CN101436207A (en) * | 2008-12-16 | 2009-05-20 | 浪潮通信信息系统有限公司 | Data restoring and synchronizing method based on log snapshot |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101795211A (en) | 2010-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101795211B (en) | Data storage method and system | |
| CN105868286B (en) | The parallel method of adding and system merged based on distributed file system small documents | |
| CN101996217B (en) | Method for storing data and memory device thereof | |
| CN104317742B (en) | Automatic thin-provisioning method for optimizing space management | |
| CN100543702C (en) | File recording device and control method thereof and manner of execution | |
| US9003135B2 (en) | Efficient allocation and reclamation of thin-provisioned storage | |
| CN102436420A (en) | Low RAM space, high-throughput persistent key-value store using secondary memory | |
| CN102985909B (en) | Object for good lattice provides the method and apparatus of the high scalability network storage | |
| CN110109886B (en) | File storage method of distributed file system and distributed file system | |
| CN106201355A (en) | Data processing method and device and storage system | |
| CN103455577A (en) | Multi-backup nearby storage and reading method and system of cloud host mirror image file | |
| CN109240607B (en) | File reading method and device | |
| CN103139300A (en) | Virtual machine image management optimization method based on data de-duplication | |
| CN102984244A (en) | Uploading system and uploading method of bill data | |
| CN102662988A (en) | Method for filtering redundant data of RFID middleware | |
| CN102063270A (en) | Write operation method and device | |
| CN107026876A (en) | A kind of file data accesses system and method | |
| CN103473298A (en) | Data archiving method and device and storage system | |
| CN108089825A (en) | A kind of storage system based on distributed type assemblies | |
| CN101539884B (en) | Method and device for data storage | |
| CN103905517A (en) | Data storage method and equipment | |
| CN107506146A (en) | A kind of data-storage system | |
| CN1996305A (en) | Data saving and reading method and device therefore, and data transmission system | |
| CN103176750A (en) | Mobile internet data storage system based on interlaced time partition and method thereof | |
| CN104021088A (en) | Log storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent for invention or patent application | ||
| CB03 | Change of inventor or designer information |
Inventor after: Yu Hengxin Inventor after: Zhang Xiao Inventor after: Ma Libin Inventor after: Yang Hongtao Inventor after: Lu Kaizhi Inventor before: Jia Lin Inventor before: Li Yongli Inventor before: Ma Libin Inventor before: Zhang Changhua Inventor before: Lu Kaizhi Inventor before: Yang Hongtao |
|
| COR | Change of bibliographic data |
Free format text: CORRECT: INVENTOR; FROM: JIA LIN LI YONGLI MA LIBIN ZHANG ZHANGHUA LU KAIZHI YANG HONGYAO TO: YU HENGXIN ZHANG XIAO MA LIBIN YANG HONGYAO LU KAIZHI |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20120222 Termination date: 20170113 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |