CN105528425A - Method of implementing asynchronous data storage based on files in cloud computing environment - Google Patents
Method of implementing asynchronous data storage based on files in cloud computing environment Download PDFInfo
- Publication number
- CN105528425A CN105528425A CN201510900287.8A CN201510900287A CN105528425A CN 105528425 A CN105528425 A CN 105528425A CN 201510900287 A CN201510900287 A CN 201510900287A CN 105528425 A CN105528425 A CN 105528425A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- cloud computing
- storage
- asynchronous
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a method for implementing asynchronous data storage based on files in a cloud computing environment, comprising the steps of defining file formats of storage data; saving the storage data into the files according to preset requirements and formats; and storing the data in the files into a relational database through an asynchronous thread. By adoption of the method for implementing asynchronous data storage based on files in the cloud computing environment, the data storage process is more visual, storage efficiency is improved, the method is quick and convenient, system operation efficiency is relatively high, performance is stable and reliable, and costs of development and maintenance are reduced; system portability and extendibility are enhanced, so that users can concentrate on business logical processing; and further, the method can also be quickly adapted to changes of business requirements and technical changes, and can be used for supporting and establishing a software system that is highly modular and highly integrated, the application range is wider, and solid foundation is provided for rapid and stable storage of data for enterprise application.
Description
Technical field
The present invention relates to Computer Applied Technology field, particularly relate to computing machine enterprise applied technical field under cloud computing environment, specifically refer to and realize based on file the method that asynchronous data stores under a kind of cloud computing environment.
Background technology
At present, based in the application and development of cloud computing, also more and more higher to the processing requirements of data, every day produces several GB easily, the data of several PB, and these data are that follow-up data analysis lays the foundation.Along with the increase of data volume, also more and more urgent to the requirement of the quick storage of data, integrality.
For convenience of the analysis of data, usually adopt the deposit data of Water demand in relevant database, so that the follow-up analysis more easily data being carried out to various dimensions; For the storage of big data quantity every day, the IO pressure of database will be very large, and database becomes the bottleneck place that the business of impact is normally run.In order to alleviate the pressure of data loading, the current conventional practice has database point storehouse submeter and stores data queue's buffer memory two kinds.Database point storehouse submeter is exactly being originally stored in the data block storage in a storehouse on multiple storehouse, being originally stored in the data block storage of a table on multiple table, also the difficulty of database maintenance and application and development is brought while this mode brings performance boost, simultaneously after execution point storehouse submeter, because data have been stored on different storehouses, store data base transaction also can be more difficult.If the distributed transaction management function of dependency database itself goes to perform affairs, high performance cost will be paid; If gone to assist to control by application program, form the affairs on programmed logic, the burden of programming aspect can be caused again.The pattern storing data queue's buffer memory is first put into data in the buffer memory based on internal memory, then write data in database by the mode of asynchronous write, this pattern can alleviate the pressure of database, but when big data quantity, data cached accumulation can be caused, bring larger memory consumption, instability is brought to system, if system needs to restart due to some cause specifics or system is delayed machine, data then in buffer memory will be lost, and bring irremediable loss to system.
Based on current conventional data storage scheme, when the big data quantity based on cloud computing environment, larger challenge is brought to the integrality of data base administration, data, add a large amount of workloads to database operation maintenance personnel, system developer simultaneously, great waste is caused for cost; If the integrality of data can not get ensureing, later data results may differ greatly with actual, and then affects the formulation of last decision-making.
Summary of the invention
The object of the invention is the shortcoming overcoming above-mentioned prior art, provide a kind of can realize by the mode of file, ensure that integrality that data store, system reboot after data can not lose, simultaneously the storage speed of file compare under the cloud computing environment that relevant database can promote greatly at aspect of performance and realize based on file the method that asynchronous data stores.
To achieve these goals, under cloud computing environment of the present invention based on file realize asynchronous data store method there is following formation:
Realize the method for asynchronous data storage under this cloud computing environment based on file, its principal feature is, the document format data that described method comprises storing data defines; The data of storage are saved in data file by the requirement preset and form; By asynchronous thread, the data in described data file are stored in relevant database.
Preferably, described defines the document format data storing data, comprises the following steps:
(1) definition stores the supplemental characteristic of the data file of data, comprises file store path, file size, file number and name form;
(2) definition stores the storage format of the data file of data.
More preferably, described definition stores the storage format of the data file of data, is specially:
The storage format of the data file of configuration store data is also arranged with binary storage data.
More preferably, described defines the document format data storing data, further comprising the steps of:
(3) define the storage format of data directory file, described data directory file is in order to record initial, the end position of data;
(4) define the storage format of data processing pointer file, described data processing pointer file is in order to record the process progress of current data.
Preferably, described is saved in data file by the data of storage by the requirement preset and form, comprises the following steps:
(5) definition will store the form of data;
(6) data of assembling storage;
(7) data to data file is preserved;
(8) data directory is preserved to data directory file;
(9) file size exceedes to pre-set after value and automatically generates new data file.
More preferably, described definition will store the form of data, comprises the following steps:
(5-1) storage format of every bar data is configured;
(5-2) the coded format definition of data is stored.
More preferably, the data that described assembling stores, comprise the following steps:
(6-1) data stored store by specified format;
(6-2) data are encoded by the coded format of specifying.
More preferably, described preservation data to data file, comprises the following steps:
(7-1) the interface IDataStore of calling data storage;
(7-2) by the data file naming rule arranged, current data file is found;
(7-3) to move to file last for data file pointers;
(7-4) the data write data file after coding.
More preferably, described preservation data directory, to data directory file, comprises the following steps:
(8-1) according to the title of data file, current data directory file is found;
If (8-2) data file is newly-built, then newdata index file, data file is identical with data directory file name;
(8-3) to move to data file last for data directory file pointer;
(8-4) by the form of setting, the data of correspondence initial, end position are in the data file written to data directory file.
More preferably, described file size exceedes and arranges the new data file of rear automatic generation, comprises the following steps:
(9-1) judge the size of current data file and the size of file is set, whether newdata file, if so, then continuing step (9-2), otherwise continuing step (7);
(9-2) data file has reached the maximal value of setting, then by the regular newdata file of specifying and data directory file.
Preferably, described is stored in relevant database by asynchronous thread by the data in described data file, comprises the following steps:
(10) daemon thread is started;
(11) content in data processing pointer file is read;
(12) data are read in data file;
(13) data are split;
(14) data are preserved to relevant database;
(15) data documents disposal is complete, and the next data file that automatically switches carries out read operation.
More preferably, described startup daemon thread, comprises the following steps:
(10-1) newly-built daemon thread;
(10-2) daemon thread is started;
(10-3) for every bar data calling data processing logic;
Complete if (10-4) data file is processed, then thread waits.
More preferably, the content in described reading data processing pointer file, comprises the following steps:
(11-1) judge the whether existence of data processing pointer file, if there is no then after newdata process pointer file, continue step (11-2), otherwise directly continue step (11-2);
(11-2) content in the data processing pointer file described in reading, if be empty, be then the All Files under process assigned catalogue, if be not empty, then continue step (11-3);
(11-3) by specified format, the current data directory document location processed and the data file name processed in data processing pointer file is read.
More preferably, read data in described data file, comprise the following steps:
(12-1) according to the fixed index file of data directory document and position, initial, the end position of corresponding data in index file is read;
(12-2) read pointer of the data file of specifying moves to starting position;
(12-3) data of corresponding length are read from starting position.
More preferably, described splits data, comprises the following steps:
(13-1) data read are decoded by the coded system of specifying;
(13-2) data line is split by specified format, form data array.
More preferably, described preservation data, to relevant database, comprise the following steps:
(14-1) DataBase combining is obtained;
(14-2) data split write relational database by specified rule.
More preferably, described data documents disposal is complete, and the next data file that automatically switches carries out read operation, comprises the following steps:
(15-1) initial in data directory file and end position is judged;
(15-2) judge whether end position exceedes the file size of setting, if so, then by data file naming rule, select next data file, then continue step (15-3), otherwise continue step (11);
(15-3) judge whether next data file exists, if not, then thread waits, if existed, then continues step (11).
Have employed under the cloud computing environment in this invention and realize based on file the method that asynchronous data stores, there is following beneficial effect:
(1) method adopting the asynchronous data based on file to store, the data that will store first are saved in file, file stores by the size of setting, if data volume exceedes maximal value, then regenerate file by certain rule, by the mode of file, ensure that the integrality that data store, after system reboot, data can not be lost, and simultaneously the storage speed of file is compared relevant database and can greatly be promoted at aspect of performance.
(2) when system starts, a daemon thread can be started, dump is carried out to the data in file, in daemon thread, according to the writing speed of the pressure size adjustment data of database, the reliable and stable of system can be ensured, for the position of the data and digital independent that complete write, carry out record by pointer file, ensure that data can not be read repeatedly.For the file that unloading has been stored up, then corresponding data file can be deleted, ensure that the data file retained is only the need file to be processed be stored.
(3) by the enforcement of the method, make the storage work of data more simple and efficient, by the asynchronous process pattern of data, effectively can alleviate the pressure of database, by adjusting the writing speed of data, guarantee system reliable and stable, adopts file to store as intermediate medium simultaneously, effectively can ensure the integrality of data.
(4) adopt method of the present invention, the process stored data is more directly perceived, and improve the efficiency of storage, efficient and convenient, running efficiency of system is higher, stable and reliable for performance, reduces the cost of development and maintenance; And enhance system portability and extensibility, make user be absorbed in the process of service logic; Moreover, the method can also adapt to change and the technique variation of business demand fast, can support the software systems setting up high modularization and high integration again, the scope of application is comparatively extensive, for laying a solid foundation to the fast and stable storage of data in enterprise's application.
Accompanying drawing explanation
Fig. 1 is the storage format of data file in the method stored based on the asynchronous data of file under cloud computing environment of the present invention.
Fig. 2 is the storage format of data directory file in the method stored based on the asynchronous data of file under cloud computing environment of the present invention.
Fig. 3 is based on the data processing pointer file storage format that the asynchronous data of file stores under cloud computing environment of the present invention.
Fig. 4 is that in the method stored based on the asynchronous data of file under cloud computing environment of the present invention, data store IDataStore associated class figure.
Fig. 5 is that in the method stored based on the asynchronous data of file under cloud computing environment of the present invention, data store call flow chart.
Fig. 6 is graph of a relation between data file in the method stored based on the asynchronous data of file under cloud computing environment of the present invention.
Embodiment
In order to more clearly describe technology contents of the present invention, conduct further description below in conjunction with specific embodiment.
The definition of storage file form comprises data storage file, data directory file, the form definition of data processing pointer file three kinds of files and the naming method definition of file; The data stored are saved in file by certain requirement and form, comprise automatically generated data file and exceed setting file size after to automatically switch file; Asynchronous thread stores up relevant database to the data conversion storage in data file, processing, carrying out record simultaneously to the data cases of process by starting asynchronous thread to data.
Wherein, the operation of described storage file form definition, comprises the following steps:
(1) define the supplemental characteristic of data storage file, comprise file store path, file size, file number, name form;
(2) storage format of data storage file is defined;
(3) storage format of data directory file is defined;
(4) storage format of data processing pointer file is defined.
The data of described storage are saved in the operation in file by certain requirement and form, comprise the following steps:
(5) definition will store the form of data;
(6) data of assembling storage;
(7) data to data file is preserved;
(8) data directory is preserved to data directory file;
(9) file size exceedes and arranges the new data file of rear automatic generation.
Described asynchronous thread stores up the operation of relevant database to the data conversion storage in data file, comprises the following steps:
(10) daemon thread is started;
(11) content in data processing pointer file is read;
(12) data are read in data file;
(13) data are split;
(14) data are preserved to relevant database;
(15) data documents disposal is complete, and the next data file that automatically switches carries out read operation.
In the method stored based on the asynchronous data of file under this cloud computing environment, described step (1) is specially: the supplemental characteristic of definition data storage file, comprise file store path, file size, file number, name form, it comprises the following steps:
(1-1) the configuration file file_store_config.xml of newdata unloading;
(1-2) store path of configuration data file;
(1-3) size of each data file is configured;
(1-4) number of configuration data file generation;
(1-5) naming method of configuration data file associated documents, comprises data storage file, data traction file, data processing pointer file.
Described step (2) is specially: the storage format of definition data storage file, and it comprises the following steps:
(2-1) storage format of configuration data storage file;
(2-2) in data file data with binary storage.
Described step (3) is specially: the storage format of definition data directory file, and it comprises the following steps:
(3-1) storage format of configuration data index file;
(3-2) initial, the end position of data is recorded in data directory file.
Described step (4) is specially: the storage format of definition data processing pointer file, and it comprises the following steps:
(4-1) storage format of configuration data process pointer file;
(4-2) the process progress of current data is recorded in data processing pointer file.
Described step (5) is specially: definition will store the form of data, and it comprises the following steps:
(5-1) storage format of every bar data is configured;
(5-2) the coded format definition of data is stored.
Described step (6) is specially: the data that assembling stores, and it comprises the following steps:
(6-1) data stored store by specified format;
(6-2) data are encoded by the coded format of specifying.
Described step (7) is specially: preserve data to data file, it comprises the following steps:
(7-1) the interface IDataStore of calling data storage;
(7-2) by the data file naming rule arranged, current data file is found;
(7-3) to move to file last for data file pointers;
(7-4) the data write data file after coding.
Described step (8) is specially: preserve data directory to data directory file, it comprises the following steps:
(8-1) according to the title of data file, current data directory file is found;
(8-2) data file is newly-built, then newdata index file, and data file is identical with index file title, but file extension is different;
(8-3) to move to file last for data directory file pointer;
(8-4) by the form of setting, the data of correspondence initial, end position are in the data file written to data directory file.
Described step (9) is specially: file size exceedes and arranges the new data file of rear automatic generation, and it comprises the following steps:
(9-1) judge the size of current data file and the size of file is set, whether newdata file;
(9-2) data file has reached the maximal value of setting, then by the regular newdata file of specifying and data directory file.
Described step (10) is specially: start daemon thread, it comprises the following steps:
(10-1) newly-built daemon thread;
(10-2) daemon thread is started;
(10-3) for every bar data calling data processing logic;
Complete if (10-4) data file is processed, then thread waits.
Described step (11) is specially: read the content in data processing pointer file, it comprises the following steps:
(11-1) judge the existence of data processing pointer file, do not exist then newly-built;
(11-2) reading content wherein, if be empty, is then the All Files under process assigned catalogue;
(11-3) by specified format, the current data directory document location processed and the data file name processed in data processing pointer file is read.
Described step (12) is specially: read data in data file, it comprises the following steps:
(12-1) according to the fixed index file of data directory document and position, initial, the end position of corresponding data in index file is read;
(12-2) read pointer of the data file of specifying moves to starting position;
(12-3) data of corresponding length are read from starting position.
Described step (13) is specially: split data, it comprises the following steps:
(13-1) data read are decoded by the coded system of specifying;
(13-2) data line is split by specified format, form data array.
Described step (14) is specially: preserve data to relevant database, it comprises the following steps:
(14-1) DataBase combining is obtained;
(14-2) data split write relational database by specified rule.
Described step (15) is specially: data documents disposal is complete, and the next data file that automatically switches carries out read operation, and it comprises the following steps:
(15-1) initial in data directory file and end position is judged;
If (15-2) end position exceedes the file size of setting, then by data file naming rule, select next data file;
(15-3) data file does not exist, then wait for, otherwise continues process.
The method stored based on the asynchronous data of file under cloud computing environment of the present invention is applied in the specific embodiment based on the asynchronous storage of big data quantity: business function is described as the module realizing concrete service logic, show as the resource file of a system, comprise class file, configuration file, data file etc.
Based on the method that the asynchronous data of file stores under this cloud computing environment, comprise the definition of storage file form; The data stored are saved in file by certain requirement and form; Asynchronous thread stores up relevant database three part to the data conversion storage in data file.Storage file form defining operation comprises the supplemental characteristic of definition data storage file, as file store path, file size, file number, name form, comprise the storage format of definition data storage file, comprise the storage format of definition data directory file, comprise the storage format of definition data processing pointer file; The data stored are saved in operation in file by certain requirement and form and comprise the form that definition will store data, comprise the data that assembling stores, comprise and preserve data to data file, comprise and preserve data directory to data directory file, comprise file size and exceed the new data file of rear automatic generation is set; Asynchronous thread stores up operation in relevant database to the data conversion storage in data file and comprises startup daemon thread, comprise the content read in data processing pointer file, comprise in data file and read data, comprise and data are split, comprise and preserve data to relevant database, comprise data documents disposal complete, the next data file that automatically switches carries out read operation.
The operation of the storage file form definition in this embodiment comprises the following steps:
(1) define the supplemental characteristic of data storage file, comprise file store path, file size, file number, name form, comprise the following steps:
A) the configuration file file_store_config.xml of newdata unloading, the form of data is as follows:
Wherein element datafile_path represents the store path of data file; Datafile_size represents the size of data file, then can convert file automatically after super large; Datafile_length represents the setting of Generating Data File number, originally increases progressively by from 0, if reach maximum number, then again from 1; Datafile_prefix represents Data Filename prefix; Dataindex_prefix represents data directory filename prefix; Datanaming_name represents data processing pointer file title.
Relation between each file as shown in Figure 6.
B) store path of configuration data file
The value of element datafile_path in corresponding configuration file file_store_config.xml.
C) size of each data file is configured
The value of element datafile_size in corresponding configuration file file_store_config.xml.
D) number of configuration data file generation
The value of element datafile_length in corresponding configuration file file_store_config.xml.
E) naming method of configuration data file associated documents, comprises data storage file, data traction file, data processing pointer file
The value of element datafile_prefix, dataindex_prefix, datanaming_name in the corresponding configuration file file_store_config.xml of difference.
(2) define the storage format of data storage file, comprise the following steps:
A) storage format of configuration data storage file, its file memory format as shown in Figure 1.
In configuration file file_store_config.xml, increase the storage format of data file, the form of data is as follows:
Wherein element datatype shows the storage format of data, supports binary storage mode; Element charactor shows the code/decode format of data, as GBK, UTF8 etc.
B) in data file data with binary storage
The value arranging element datatype is byte, shows that the preservation form of data is that binary mode stores.
(3) define the storage format of data directory file, comprise the following steps:
A) storage format of configuration data index file, its file memory format as shown in Figure 2.
In configuration file file_store_config.xml, increase the storage format of data directory file, the form of data is as follows:
Wherein element length shows to be configured to the byte length that data directory position is stored in file and takies 8, to support the file size of long shaping.
B) initial, the end position of data is recorded in data directory file
When data are preserved, record the position at data place in the index file of corresponding data, each position takies the length word joint number of setting.
(4) define the storage format of data processing pointer file, comprise the following steps:
A) storage format of configuration data process pointer file, its file memory format as shown in Figure 3.
In configuration file file_store_config.xml, increase the storage format of data processing pointer file, the form of data is as follows:
Wherein element length shows the index position at current data process place, takies length consistent with the byte length shared by Data Position in index file, is configured to 8, support the file size of long shaping.
B) the process progress of current data is recorded in data processing pointer file
When digital independent process, in the data processing pointer file of corresponding data, record is when the index file position at the data place of pre-treatment, and data are the index file title processed thereafter.
The operation that the data of the storage in this embodiment are saved in file by certain requirement and form comprises the following steps:
(5) definition will store the form of data, comprises the following steps:
A) storage format of every bar data is configured
The value of the properties datatype of amendment configuration file file_store_config.xml interior joint filestore_datafile, default system supports that binary mode stores, and can be undertaken for specific format by extended mode.
B) the coded format definition of data is stored
The value of the properties charactor of amendment configuration file file_store_config.xml interior joint filestore_datafile, default system supports GBK coded format.
(6) data of assembling storage, comprise the following steps:
A) data stored store by specified format
The false code that object is converted to scale-of-two byte arrays is as follows:
SerializabledataObject＝……
ByteArrayOutputStreamelementOutstream＝newByteArrayOutputStream()；
ObjectOutputStreamelementObjstr＝newObjectOutputStream(elementOutstream)；
elementObjstr.writeObject(dataObject)；
elementObjstr.close()；
byte[]elementBuffer＝elementOutstream.toByteArray()；
elementOutstream.close()；
B) data are encoded by the coded format of specifying
Be character string for conservation object, if be converted to byte arrays, need the coding by specifying to encode, such reduction just carrying out original character string by the coded format of specifying, the false code of coding is as follows:
(7) preserve data to data file, comprise the following steps:
A) the interface IDataStore of calling data storage, interface class figure as shown in Figure 4
IDataStoredataStore＝DataStoreFactory.createDataStore("datastorename")；
B) by the data file naming rule arranged, current data file is found
Deposit path from the data file of specifying and sort to all data files, find the file name that current sequence number is maximum, if sequence number reaches the maximum storage number of file, then the continuous maximum sequence number from 1 is the file name when required preservation.The false code of file path sequence is as follows:
C) to move to file last for data file pointers
According to the file that will write, open file, then file pointer is moved to file last, write data, the mode of operation support " read-write " of file.
RandomAccessFiledataRandAccessFile＝newRandomAccessFile(dataFile,"rw")；
dataRandAccessFile.seek(dataLength)；
D) the data write data file after coding
Data object, after binary coding, forms byte arrays, writes data in data file.
dataRandAccessFile.write(elementBuffer)；
(8) preserve data directory to data directory file, comprise the following steps:
A) according to the title of data file, current data directory file is found
When data write data file, according to the file sequence number found, by the naming rule of file, find corresponding data directory file.
B) data file is newly-built, then newdata index file, and data file is identical with index file title, but file extension is different
C) to move to file last for data directory file pointer
According to the index file that will write, open file, then file pointer is moved to file last, write data, the mode of operation support " read-write " of file.
RandomAccessFiledataRandIndexAccessFile＝newRandomAccessFile(dataIndexFile,"rw")；
dataRandIndexAccessFile.seek(dataLength)；
D) by the form of setting, the data of correspondence initial, end position are in the data file written to data directory file, in indexed file, reference position is the position of current data in data file, and end position is reference position and write data length sum.The false code that shape data is converted to octet is as follows:
(9) file size exceedes and arranges the new data file of rear automatic generation, comprises the following steps:
A) judge the size of current data file and the size of file is set, whether newdata file.Obtain the current size needing to write data file, compare with the file maximal value arranged
B) data file has reached the maximal value of setting, then by the regular newdata file of specifying and data directory file, the sequence number of new filename is increased progressively successively.
Asynchronous thread in this embodiment stores up the operation of relevant database to the data conversion storage in data file, and as shown in Figure 5, this operation comprises the following steps its operating process run:
(10) start daemon thread, comprise the following steps:
A) newly-built daemon thread, carries out background process to data file
B) daemon thread is started
C) for every bar data calling data processing logic, the unloading of data is performed
Complete if d) data file is processed, then thread waits, the execution otherwise data process method circulates successively.
(11) read the content in data processing pointer file, comprise the following steps:
A) judge the existence of data processing pointer file, do not exist then newly-built.This operation is only newly-built to process pointer file when using for first time, and subsequent operation uses this file to record file when pre-treatment and position always.
B) read content wherein, if be empty, be then the All Files under process assigned catalogue, by the naming rule of file, listed files sorted, realize the principle of first in first out.
C) by specified format, read the current data directory document location processed and the data file name processed in data processing pointer file, false code is as follows:
(12) read data in data file, comprise the following steps:
A) according to the fixed index file of data directory document and position, initial, the end position of corresponding data in index file is read
The read pointer of the data file of b) specifying moves to starting position
C) data of corresponding length are read from starting position
The process of // index file
RandomAccessFileindexRandAccessFile.seek(currentReadIndex)；
byte[]start＝newbyte[8]；
indexRandAccessFile.read(start)；
byte[]end＝newbyte[8]；
indexRandAccessFile.read(end)；
// data documents disposal
dataRandAccessFile.seek(startIndex)；
byte[]data＝newbyte[(int)length]；
dataRandAccessFile.read(data)；
ByteArrayInputStreaminstr＝newByteArrayInputStream(data)；
ObjectInputStreamobjstr＝newObjectInputStream(instr)；
result＝(Serializable)objstr.readObject()；
The process of // index file
currentReadIndex＝currentReadIndex+16；
byte[]ri＝newbyte[8]；
Bits.putLong(ri,currentReadIndex)；
namingRandAccessFile.seek(0)；
namingRandAccessFile.write(ri)；
(13) data are split, comprise the following steps:
A) data read are decoded by the coded system of specifying, form manageable object
B) data object read is split by specified format, form data array, the storage operation of database can be carried out.
(14) preserve data to relevant database, comprise the following steps:
A) obtain DataBase combining, open the passage of database
B) data split write relational database by specified rule, by the update of performing database, write data in database.
(15) data documents disposal is complete, and the next data file that automatically switches carries out read operation, comprises the following steps:
A) judge initial in data directory file and end position, if having follow-up data in index file, then the data needing to process are described, continue to perform data process method, weight update is carried out to data.
If b) end position exceedes the file size of setting, then illustrate that notebook data file is disposed, then again by data file naming rule, select next data file, proceed read operation, preserve the operation of data to data storehouse.
If follow-up data file does not exist, or index file reaches end of file, then illustrate and do not need data to be processed at present, digital independent thread is then waited for, otherwise continues process
Have employed under the cloud computing environment in this invention and realize based on file the method that asynchronous data stores, there is following beneficial effect:
(1) method adopting the asynchronous data based on file to store, the data that will store first are saved in file, file stores by the size of setting, if data volume exceedes maximal value, then regenerate file by certain rule, by the mode of file, ensure that the integrality that data store, after system reboot, data can not be lost, and simultaneously the storage speed of file is compared relevant database and can greatly be promoted at aspect of performance.
(2) when system starts, a daemon thread can be started, dump is carried out to the data in file, in daemon thread, according to the writing speed of the pressure size adjustment data of database, the reliable and stable of system can be ensured, for the position of the data and digital independent that complete write, carry out record by pointer file, ensure that data can not be read repeatedly.For the file that unloading has been stored up, then corresponding data file can be deleted, ensure that the data file retained is only the need file to be processed be stored.
(3) by the enforcement of the method, make the storage work of data more simple and efficient, by the asynchronous process pattern of data, effectively can alleviate the pressure of database, by adjusting the writing speed of data, guarantee system reliable and stable, adopts file to store as intermediate medium simultaneously, effectively can ensure the integrality of data.
(4) adopt method of the present invention, the process stored data is more directly perceived, and improve the efficiency of storage, efficient and convenient, running efficiency of system is higher, stable and reliable for performance, reduces the cost of development and maintenance; And enhance system portability and extensibility, make user be absorbed in the process of service logic; Moreover, the method can also adapt to change and the technique variation of business demand fast, can support the software systems setting up high modularization and high integration again, the scope of application is comparatively extensive, for laying a solid foundation to the fast and stable storage of data in enterprise's application.
In this description, the present invention is described with reference to its specific embodiment.But, still can make various amendment and conversion obviously and not deviate from the spirit and scope of the present invention.Therefore, instructions and accompanying drawing are regarded in an illustrative, rather than a restrictive.
Claims (17)
1. realize a method for asynchronous data storage under cloud computing environment based on file, it is characterized in that, the document format data that described method comprises storing data defines; The data of storage are saved in data file by the requirement preset and form; By asynchronous thread, the data in described data file are stored in relevant database.
2. realize the method for asynchronous data storage under cloud computing environment according to claim 1 based on file, it is characterized in that, described defines the document format data storing data, comprises the following steps:
(1) definition stores the supplemental characteristic of the data file of data, comprises file store path, file size, file number and name form;
(2) definition stores the storage format of the data file of data.
3. realize the method for asynchronous data storage under cloud computing environment according to claim 2 based on file, it is characterized in that, described definition stores the storage format of the data file of data, is specially:
The storage format of the data file of configuration store data is also arranged with binary storage data.
4. realize the method for asynchronous data storage under cloud computing environment according to claim 2 based on file, it is characterized in that, described defines the document format data storing data, further comprising the steps of:
(3) define the storage format of data directory file, described data directory file is in order to record initial, the end position of data;
(4) define the storage format of data processing pointer file, described data processing pointer file is in order to record the process progress of current data.
5. realize the method for asynchronous data storage under cloud computing environment according to claim 1 based on file, it is characterized in that, described is saved in data file by the data of storage by the requirement preset and form, comprises the following steps:
(5) definition will store the form of data;
(6) data of assembling storage;
(7) data to data file is preserved;
(8) data directory is preserved to data directory file;
(9) file size exceedes to pre-set after value and automatically generates new data file.
6. realize the method for asynchronous data storage under cloud computing environment according to claim 5 based on file, it is characterized in that, described definition will store the form of data, comprises the following steps:
(5-1) storage format of every bar data is configured;
(5-2) the coded format definition of data is stored.
7. realize the method for asynchronous data storage under cloud computing environment according to claim 5 based on file, it is characterized in that, the data that described assembling stores, comprise the following steps:
(6-1) data stored store by specified format;
(6-2) data are encoded by the coded format of specifying.
8. realize the method for asynchronous data storage under cloud computing environment according to claim 5 based on file, it is characterized in that, described preservation data to data file, comprises the following steps:
(7-1) the interface IDataStore of calling data storage;
(7-2) by the data file naming rule arranged, current data file is found;
(7-3) to move to file last for data file pointers;
(7-4) the data write data file after coding.
9. realize the method for asynchronous data storage under cloud computing environment according to claim 5 based on file, it is characterized in that, described preservation data directory, to data directory file, comprises the following steps:
(8-1) according to the title of data file, current data directory file is found;
If (8-2) data file is newly-built, then newdata index file, data file is identical with data directory file name;
(8-3) to move to data file last for data directory file pointer;
(8-4) by the form of setting, the data of correspondence initial, end position are in the data file written to data directory file.
10. realize based on file the method that asynchronous data stores under cloud computing environment according to claim 5, it is characterized in that, described file size exceedes and arranges the new data file of rear automatic generation, comprises the following steps:
(9-1) judge the size of current data file and the size of file is set, whether newdata file, if so, then continuing step (9-2), otherwise continuing step (7);
(9-2) data file has reached the maximal value of setting, then by the regular newdata file of specifying and data directory file.
Realize the method for asynchronous data storage under 11. cloud computing environments according to claim 1 based on file, it is characterized in that, described is stored in relevant database by asynchronous thread by the data in described data file, comprises the following steps:
(10) daemon thread is started;
(11) content in data processing pointer file is read;
(12) data are read in data file;
(13) data are split;
(14) data are preserved to relevant database;
(15) data documents disposal is complete, and the next data file that automatically switches carries out read operation.
Realize the method for asynchronous data storage under 12. cloud computing environments according to claim 11 based on file, it is characterized in that, described startup daemon thread, comprises the following steps:
(10-1) newly-built daemon thread;
(10-2) daemon thread is started;
(10-3) for every bar data calling data processing logic;
Complete if (10-4) data file is processed, then thread waits.
Realize the method for asynchronous data storage under 13. cloud computing environments according to claim 11 based on file, it is characterized in that, the content in described reading data processing pointer file, comprises the following steps:
(11-1) judge the whether existence of data processing pointer file, if there is no then after newdata process pointer file, continue step (11-2), otherwise directly continue step (11-2);
(11-2) content in the data processing pointer file described in reading, if be empty, be then the All Files under process assigned catalogue, if be not empty, then continue step (11-3);
(11-3) by specified format, the current data directory document location processed and the data file name processed in data processing pointer file is read.
Realize the method for asynchronous data storage under 14. cloud computing environments according to claim 11 based on file, it is characterized in that, read data in described data file, comprise the following steps:
(12-1) according to the fixed index file of data directory document and position, initial, the end position of corresponding data in index file is read;
(12-2) read pointer of the data file of specifying moves to starting position;
(12-3) data of corresponding length are read from starting position.
Realize the method for asynchronous data storage under 15. cloud computing environments according to claim 11 based on file, it is characterized in that, described splits data, comprises the following steps:
(13-1) data read are decoded by the coded system of specifying;
(13-2) data line is split by specified format, form data array.
Realize the method for asynchronous data storage under 16. cloud computing environments according to claim 11 based on file, it is characterized in that, described preservation data, to relevant database, comprise the following steps:
(14-1) DataBase combining is obtained;
(14-2) data split write relational database by specified rule.
Realize the method for asynchronous data storage under 17. cloud computing environments according to claim 11 based on file, it is characterized in that, described data documents disposal is complete, and the next data file that automatically switches carries out read operation, comprises the following steps:
(15-1) initial in data directory file and end position is judged;
(15-2) judge whether end position exceedes the file size of setting, if so, then by data file naming rule, select next data file, then continue step (15-3), otherwise continue step (11);
(15-3) judge whether next data file exists, if not, then thread waits, if existed, then continues step (11).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510900287.8A CN105528425A (en) | 2015-12-08 | 2015-12-08 | Method of implementing asynchronous data storage based on files in cloud computing environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510900287.8A CN105528425A (en) | 2015-12-08 | 2015-12-08 | Method of implementing asynchronous data storage based on files in cloud computing environment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105528425A true CN105528425A (en) | 2016-04-27 |
Family
ID=55770648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510900287.8A Pending CN105528425A (en) | 2015-12-08 | 2015-12-08 | Method of implementing asynchronous data storage based on files in cloud computing environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105528425A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107451301A (en) * | 2017-09-12 | 2017-12-08 | 彩讯科技股份有限公司 | Processing method, device, equipment and the storage medium of bill mail are delivered in real time |
| CN110248205A (en) * | 2018-09-03 | 2019-09-17 | 浙江大华技术股份有限公司 | A kind of method and device that video record data are packaged |
| WO2019184618A1 (en) * | 2018-03-28 | 2019-10-03 | 腾讯科技（深圳）有限公司 | Method and device for storing data, server, and storage medium |
| CN112069270A (en) * | 2020-09-07 | 2020-12-11 | 青岛海信传媒网络技术有限公司 | File-based multi-dimensional data reading and writing method and display device |
| CN113204546A (en) * | 2021-03-15 | 2021-08-03 | 中国再保险(集团)股份有限公司 | Method for rapidly calculating large number of earthquake event influence fields |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102841860A (en) * | 2012-08-17 | 2012-12-26 | 珠海世纪鼎利通信科技股份有限公司 | Large data volume information storage and access method |
| US8832045B2 (en) * | 2006-04-07 | 2014-09-09 | Data Storage Group, Inc. | Data compression and storage techniques |
| CN104331460A (en) * | 2014-10-31 | 2015-02-04 | 北京思特奇信息技术股份有限公司 | Hbase-based data read-write operation method and system |
| CN104516988A (en) * | 2015-01-21 | 2015-04-15 | 天津书生云科技有限公司 | File writing method and device |
| CN104615736A (en) * | 2015-02-10 | 2015-05-13 | 上海创景计算机系统有限公司 | Quick analysis and storage method of big data based on database |
-
2015
- 2015-12-08 CN CN201510900287.8A patent/CN105528425A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8832045B2 (en) * | 2006-04-07 | 2014-09-09 | Data Storage Group, Inc. | Data compression and storage techniques |
| CN102841860A (en) * | 2012-08-17 | 2012-12-26 | 珠海世纪鼎利通信科技股份有限公司 | Large data volume information storage and access method |
| CN104331460A (en) * | 2014-10-31 | 2015-02-04 | 北京思特奇信息技术股份有限公司 | Hbase-based data read-write operation method and system |
| CN104516988A (en) * | 2015-01-21 | 2015-04-15 | 天津书生云科技有限公司 | File writing method and device |
| CN104615736A (en) * | 2015-02-10 | 2015-05-13 | 上海创景计算机系统有限公司 | Quick analysis and storage method of big data based on database |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107451301A (en) * | 2017-09-12 | 2017-12-08 | 彩讯科技股份有限公司 | Processing method, device, equipment and the storage medium of bill mail are delivered in real time |
| CN107451301B (en) * | 2017-09-12 | 2021-01-08 | 彩讯科技股份有限公司 | Processing method, device, equipment and storage medium for real-time delivery bill mail |
| WO2019184618A1 (en) * | 2018-03-28 | 2019-10-03 | 腾讯科技（深圳）有限公司 | Method and device for storing data, server, and storage medium |
| US11275759B2 (en) | 2018-03-28 | 2022-03-15 | Tencent Technology (Shenzhen) Company Limited | Data storage method and apparatus, server, and storage medium |
| CN110248205A (en) * | 2018-09-03 | 2019-09-17 | 浙江大华技术股份有限公司 | A kind of method and device that video record data are packaged |
| CN112069270A (en) * | 2020-09-07 | 2020-12-11 | 青岛海信传媒网络技术有限公司 | File-based multi-dimensional data reading and writing method and display device |
| CN113204546A (en) * | 2021-03-15 | 2021-08-03 | 中国再保险(集团)股份有限公司 | Method for rapidly calculating large number of earthquake event influence fields |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105528425A (en) | Method of implementing asynchronous data storage based on files in cloud computing environment | |
| US7676481B2 (en) | Serialization of file system item(s) and associated entity(ies) | |
| Floratou et al. | Column-oriented storage techniques for MapReduce | |
| US5740405A (en) | Method and system for providing data compatibility between different versions of a software program | |
| CN103595797B (en) | Caching method for distributed storage system | |
| CN100498781C (en) | Method for storing metadata of logic document system by adhesion property | |
| US20100082537A1 (en) | File system for storage device which uses different cluster sizes | |
| CN101650660A (en) | Booting a computer system from central storage | |
| US8051054B2 (en) | Method and system for data processing with database update for the same | |
| US11429494B2 (en) | File backup based on file type | |
| CN107391544A (en) | Processing method, device, equipment and the computer storage media of column data storage | |
| US11249968B2 (en) | Large object containers with size criteria for storing mid-sized large objects | |
| US7725507B1 (en) | Dynamic directories | |
| CN116578234B (en) | Flash memory access system and method | |
| US20080033948A1 (en) | Storage apparatus and method therefor | |
| US8423585B2 (en) | Variants of files in a file system | |
| US20170286442A1 (en) | File system support for file-level ghosting | |
| CN116226497A (en) | Retrieval method, medium, device and computing equipment | |
| US20130282756A1 (en) | Method of processing files in storage system and data server using the method | |
| CN100444166C (en) | Position-based interface access method and device for FAT file system | |
| US8990265B1 (en) | Context-aware durability of file variants | |
| US8495279B2 (en) | Flash file system and driving method thereof | |
| US10795875B2 (en) | Data storing method using multi-version based data structure | |
| US11914587B2 (en) | Systems and methods for key-based indexing in storage devices | |
| CN116185891B (en) | Descriptor management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160427 |
|
| RJ01 | Rejection of invention patent application after publication |