CN101510838B - Method for implementing perdurable data queue - Google Patents
Method for implementing perdurable data queue Download PDFInfo
- Publication number
- CN101510838B CN101510838B CN2009100783551A CN200910078355A CN101510838B CN 101510838 B CN101510838 B CN 101510838B CN 2009100783551 A CN2009100783551 A CN 2009100783551A CN 200910078355 A CN200910078355 A CN 200910078355A CN 101510838 B CN101510838 B CN 101510838B
- Authority
- CN
- China
- Prior art keywords
- data
- file
- control documents
- queue
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Abstract
The invention provides a method for realizing persistent data array. The method comprises the steps of obtaining information of a control file; loading the data array according to the information of the control file; and managing a file in a disc route through a control module. The method adopts the light-weight disc-file-based first-in-first-out end-to-end data array to realize the persistent data array, and is applicable to information transmission of a communication system or directly to the data retransmission and flow control which both require persistent information.
Description
Technical field
The present invention relates to the method for perdurable data queue, refer in particular to a kind of disk file that in communication system information transmits, adopts and realize the method for perdurable data queue.
Background technology
In communication system; Having needs message transmitted in a large number; And these information of being transmitted are to need lasting preservation to get off a bit, and message durability should be a kind of reasonable method for the reliable news transmission, and the message persistence has been arranged; Even sender and recipient are not that online or message center has shut down after the sender sends message simultaneously,, message center still can message be sent after restarting.The persistence message queue adopts data base persistence message mostly at present; The message queue of just desire being preserved is stored in database or the file, and the principle of this message durability is very simple, exactly after the sender sends message; Message center at first arrives local data file, memory database or remote data base etc. with message stores; Attempt then message is sent to the recipient, send successfully and then message is deleted from storage, failure then continues to attempt.Message center at first will be checked the memory location of formulation after starting, if do not send message of successful, then need send message.
Concerning the database persistence, need build database, and the database of building itself needs expense; Application for a lightweight; Build database for the message persistence specially, the expense of paying is too big, in addition; Data database storing itself is not possess the first in first out characteristics, therefore when operation, can cause efficient lower.Therefore adopt the mode of existing persistence message queue, although function is fairly perfect, treat a large amount of message data, the expense that needs is very big, and efficient is very low, in the application that processing speed is had relatively high expectations, then can be difficult to satisfy this demands of applications.
Summary of the invention
The object of the invention is to propose to realize the method for perdurable data queue, refer in particular to a kind of adopt lightweight, based on disk file, first in first out, the method that realizes of data queue end to end.
Method of the present invention can be used for the message transmission of communication system, also needing can directly be used for the data forwarding and the flow control of persistence message.
Method of the present invention is to provide a kind of method that realizes perdurable data queue, may further comprise the steps:
If in the disk path of appointment control documents is arranged, directly reads control documents information;
If when not having control documents in the disk path of said appointment, create said control documents information according to the form of setting, and said control documents information write in the control documents of new establishment;
Be written into first in first out data queue according to said control documents information;
Through the file in the control module hyperdisk path;
Wherein being written into said first in first out data queue according to said control documents information is, according to the formation of describing in said control documents information information end to end, is written into said first in first out data queue.
The data file of preferably, managing in the described disk path through said control module is specially:
Said control documents reads or revises or delete the data file in the said disk through said control module; And
Said control documents writes data file in the said disk through said control module;
Wherein said control module comprises the method for GetData, PutData, IsFull, IsEmpty, GetCount.
Preferably, the said control documents data that will desire persistence write disk and may further comprise the steps:
Adopt whether IsFull mode judgment data file is the full writing position of confirming these data.
Preferably, adopt GetData to be used for obtaining data from queue heads when adopting the IsFull mode, PutData is used for data are write rear of queue.
Preferably, in the mode that adopts IsEmpty whether dummy status of formation is provided, GetCount is used for obtaining formation current data unit number.
Preferably, the read-write operation of said data queue at first carries out in internal memory, when PutData operates, the data of newly adding is write said disk.
The present invention adopts file that persistence is carried out in the data formation; In the scheme of the present invention in the process of carrying out data persistence; Based on disk file be divided into two types: control documents and data file; Said control documents is used for to the controlling end to end of formation, and data file is used for preserving data cell.
Method of the present invention is a perdurable data queue implementation method based on the lightweight of disk file, does not compared with prior art have too much management function, only realizes the point-to-point persistence formation of a first in first out; Do not need the support of additional apparatus and software, can directly apply in the application that needs the persistence fifo queue, structure of the present invention and algorithm are simple; It is little to take resource, and speed is fast, can directly be embedded in the various application; The transmission data of different types in in-process formation is used, adopts the solution of this lightweight; Little than the prior art expense, the speed of service is fast, and efficient is high.
Description of drawings
Fig. 1 is the data structure show of control documents of the present invention;
Fig. 2 is the data structure show of data file of the present invention;
Fig. 3 is file description plot structure of the present invention explanation;
Fig. 4 is that data cell of the present invention is described the plot structure explanation;
Fig. 5 is data cell storage area structure of the present invention explanation;
Fig. 6 is the flow chart that is obtained data in the method for the present invention by queue heads;
Fig. 7 is the flow chart that in the method for the present invention data is write rear of queue;
Fig. 8 is that whether the specified data formation is the flow chart of empty queue in the method for the present invention;
Fig. 9 is that whether the specified data formation is the flow chart of full queue in the method for the present invention;
Figure 10 is the flow chart that obtains formation current data unit number in the method for the present invention;
Figure 11 is a method flow diagram of the present invention.
Embodiment
The present invention proposes a kind of method that realizes perdurable data queue, with reference to shown in Figure 11, may further comprise the steps: at first, obtain control documents information; Be written into data queue according to control documents information then; Said control documents is managed the data file in the described disk path.Wherein said data queue can be made up of control documents and data file, and control documents and data file are kept under the same disk path, that is to say, the disk path to an appointment is hinted obliquely in a data formation.
In scheme of the present invention, obtaining control documents information is through judging that the step that whether is provided with control documents under the specified path obtains, said control documents is even arranged in said specified path, directly reading the information in the control documents; If when not having control documents in the said specified path, then create control information, and control information write in the control documents of new establishment according to the form of setting.After obtaining control documents information,, be written into data queue according to the formation of describing in control documents information end to end.After said data queue is written into successfully; Can adopt the described method in back that read-write operation is carried out in the data formation; One or more reception message threads and one or more transmission message threads are all arranged in the general application of reality; Receiving thread writes formation with the message that receives, and sends thread and from formation, reads message.The queue message transmission sequence is first in first out.
Control documents described in the embodiments of the invention can be managed 65,536 data files at most through control module, wherein in the data file by said management; The data cell that each data file can be deposited is 10,000 at most, and capacity of queue is 10; 000 * 65; 536=655,360,000.Referring to figs. 1 to Fig. 5; Explained in the scheme of the present invention can reference control documents and the data structure of data file; Said control documents length is 8 bytes (with reference to figure 1), and this control documents can be divided into 2 territories: queue heads information field and rear of queue information field, and wherein there are 2 fields in each territory: the data unit number in document number and the file; Length is respectively 2 bytes, and type is the unsigned int number.The data cell that described queue heads information field has been described queue heads through document number and data unit number is in which the data unit in which file; Through queue heads information; Can directly have access to the descriptor of queue heads data cell in the file; Can directly have access to concrete data through this descriptor again, detailed process can be referring to the explanation of read data from data queue in the present embodiment.The data cell that said queue heads information field has been described queue heads through document number and data unit number is in which the data unit in which file; Through queue heads information; The descriptor of queue heads data cell in the file can be directly had access to, concrete data can be directly had access to through this descriptor again.
Data file described in the present embodiment is divided into 3 territories: file description district, data cell are described the district, (referring to Fig. 2) got in the data cell storage.Wherein said file description district is used for describing the essential information of notebook data file, and data cell is described the essential information that the district is used for describing each data cell in the notebook data file, and the data cell memory block is used for storing data.File description district and data cell are described the district and are fixed-length field, can realize the fast access of file data unit through calculating the particular location that side-play amount directly navigates to each data cell of data cell memory block like this.It is 12 bytes that said data file is described section length, by the quantity of data cell in file identifier, document time stamp, the file totally 3 fields form, as shown in Figure 3.Wherein file identifier is used for representing that presents is specific data queue's data file; Value is 0x00003c8e; Document time stamp is represented the time that the notebook data file generates, and gets the utc of the system time, and the file data element number has been described the data cell number of notebook data document storage.It is 10*10 that data cell is described section length, 000=100, and 000 byte is made up of 10,000 data unit descriptors, and each data cell descriptor length is 10 bytes, and is as shown in Figure 4.Each data cell descriptor by data cell side-play amount, data unit length, data cell attribute totally 3 fields form.Data cell side-play amount length is 4 bytes; Type is a unsigned int; Described residing absolute drift position in data file, notebook data unit, the side-play amount of an active data unit should be greater than 100,012; Can directly file pointer be navigated to data cell memory location hereof through the data cell side-play amount, realize the fast access of data cell.Data unit length is 4 bytes, and type is a unsigned int, has described data cell and has taken storage size hereof, has defined the scope that data cell is operated through data cell, realizes the asynchronous storage and the visit of data, reaches the effect of saving the space.The data cell attribute length is 2 bytes; Type is a unsigned int; Its value is self-defined by the application of using data queue, and a typical application is that the data cell attribute is used for representing the data cell type, is implemented in the purpose of the polytype data of storage in the data queue.
Control module in the scheme that the persistence of the present invention of forming the publicly-owned method that data queue in the present embodiment adopts existing 5 formations to use: GetData, PutData, IsFull, IsEmpty, GetCount realizes; Said control documents is managed the data file through the control module that these methods constitute; Wherein GetData is used for obtaining data (with reference to figure 6) from queue heads; PutData is used for data are write rear of queue (with reference to figure 7); IsFull provides formation whether to expire state (with reference to figure 9), and IsEmpty provides whether dummy status (with reference to figure 8) of formation, and GetCount is used for obtaining formation current data unit number (with reference to Figure 10).The data write operation is at first carried out in internal memory, then if the PutData operation then writes disk with the data cell of newly adding.
The process that the data of desiring persistence is write disk file is referring to following steps:
1, obtain control documents information, the acquisition of this control documents information can be repeated no more at this with reference to foregoing content;
2, because the data in the formation of control documents control data; Adopt the IsFull method to judge whether formation is full by said control documents; If for full then can not in data file, write data, directly return and write failure, carry out writing data manipulation if formation is discontented.It is following to write the data manipulation logic: judge according to the rear of queue information in the control documents whether mantissa writes fullly according to file, judgment rule is: if mantissa according to unit number less than 9999, then mantissa according to file less than, otherwise then mantissa has write full according to file.If mantissa according to file less than; Then with mantissa according to unit number+1, full if mantissa has been write according to file, then switch to next data file; Switching law is: new mantissa is according to document number=(document number+1 in the current tail information) %65536, and new mantissa is according to unit number=0.The mantissa that will write that obtains according to aforementioned logic opens the corresponding data file according to document number, creates this document if this data file does not exist, and the file operation pointer is navigated to EOF, writes data in the file.If write operation success; Then the file operation pointer is navigated to the data cell descriptor in the data file; Localization method is: calculate side-play amount according to the data unit number in the rear of queue information, computing formula is: 12+ (data unit number * 10) navigates to the side-play amount that previous calculations obtains with the file operation pointer then; After accomplish the location; Write the data cell descriptor of 10 bytes according to the structure (Fig. 4) of data cell descriptor, if all successes of the operation of front then write new tail information in the control documents.
In the present embodiment; Whether adopt the IsEmpty algorithm to be used for the judgment data formation is empty; This algorithm is obtained by queue heads information in the control documents and rear of queue information calculations; Concrete formula is: if the queue heads document number equal the rear of queue document number simultaneously the queue heads number occupy that the unit sky equals the rear of queue data unit number formation for empty, otherwise formation non-NULL then.Whether can adopt the IsFull algorithm to be used for the judgment data formation in the present embodiment is full; This algorithm is obtained by queue heads information in the control documents and rear of queue information calculations; Concrete logic is: calculate next data file number earlier, computing formula is: (rear of queue document number+1) %65536, if next data file number equals the queue heads document number; Then data queue is full, on the contrary then number to occupy formation discontented.
Present embodiment adopts the GetCount algorithm to be used for calculating the record number in the current queue; Algorithm is: tail document number-header file number at first; Obtain that number of files is poor end to end; If this difference, is calculated as number of files is poor end to end less than 0 then with this difference+65536 at last: number of files difference * 10000+ (mantissa is according to a unit number-data unit number) end to end obtains the record number in the current queue.
If more than be the process that down data is write disk file by control documents control has been described.
Down in the face of how from data queue read data be elaborated, idiographic flow is (with reference to figure 6) as follows: call above-mentioned IsEmpty method and judge whether formation is sky, if be sky in the data queue free of data readable; Directly return and read failure, if data are arranged in the formation then navigate to the data cell descriptor in the data file according to the queue heads information in the control documents, localization method is: at first find the corresponding data file according to the document number in the queue heads information and open; Calculate side-play amount according to the data unit number in the queue heads information again, computing formula is: 12+ (data unit number * 10) navigates to the side-play amount that previous calculations obtains with the file operation pointer at last; After accomplish the location, read the data cell descriptor of 10 bytes, resolve the file absolute offset values that obtains the queue heads data cell according to the structure (Fig. 4) of data cell descriptor; The file operation pointer is navigated on this side-play amount; Describe the data of length in the reading of data unit descriptor, queue heads is pointed to next data cell, specific algorithm is: with data unit number+1; If less than 10000; Then directly upgrade queue heads information in control documents,, show that then next data cell is in next data file if more than or equal to 10000; With queue heads document number+1; It is 0 that the queue heads data unit number is set, and deletes a last data files, returns success.
In scheme of the present invention, said control documents is also operated the data queue that writes disk data is revised or deletion, and its detailed process is no longer given unnecessary details at this.
In the present embodiment, described control documents is necessary, when creating data queue, produces, and revises during the operating data formation again; Data file not necessarily dynamically generates, revises and deletes when the operating data formation.
In sum, the algorithm that adopts in the inventive method and the method is simple, can directly be embedded in the various application, in in-process formation is used, adopts the solution of this lightweight, and little than the prior art expense, the speed of service is fast, and efficient is high.
Though below invention has been described in conjunction with the preferred embodiments; But it should be appreciated by those skilled in the art; Method of the present invention is not limited to the embodiment described in the embodiment; Under the situation that does not deviate from the spirit and scope of the invention that limits appended claims, can make various modifications, increase and replacement to the present invention.
Claims (6)
1. method that realizes perdurable data queue may further comprise the steps:
If in the disk path of appointment control documents is arranged, directly reads control documents information;
If when not having control documents in the disk path of said appointment, create said control documents information according to the form of setting, and said control documents information write in the control documents of new establishment;
Be written into first in first out data queue according to said control documents information;
Through the file in the control module hyperdisk path;
Wherein being written into said first in first out data queue according to said control documents information is, according to the formation of describing in said control documents information information end to end, is written into said first in first out data queue.
2. the method for realization perdurable data queue according to claim 1 is characterized in that, the data file of managing in the described disk path through said control module is specially:
Said control documents reads or revises or delete the data file in the said disk through said control module; And
Said control documents writes data file in the said disk through said control module;
Wherein said control module comprises the method for GetData, PutData, IsFull, IsEmpty, GetCount.
3. the method for realization perdurable data queue according to claim 2 is characterized in that,
The data that said control documents will be desired persistence write disk and may further comprise the steps:
Adopt whether IsFull mode judgment data file is the full writing position of confirming these data.
4. the method for realization perdurable data queue according to claim 3 is characterized in that,
Adopt GetData to be used for obtaining data from queue heads when adopting the IsFull mode, PutData is used for data are write rear of queue.
5. the method for realization perdurable data queue according to claim 3 is characterized in that, in the mode that adopts IsEmpty whether dummy status of formation is provided, and GetCount is used for obtaining formation current data unit number.
6. the method for realization perdurable data queue according to claim 2 is characterized in that, the read-write operation of said data queue at first carries out in internal memory, when PutData operates, the data of newly adding is write said disk.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2009100783551A CN101510838B (en) | 2009-02-26 | 2009-02-26 | Method for implementing perdurable data queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2009100783551A CN101510838B (en) | 2009-02-26 | 2009-02-26 | Method for implementing perdurable data queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101510838A CN101510838A (en) | 2009-08-19 |
| CN101510838B true CN101510838B (en) | 2012-07-04 |
Family
ID=41003105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2009100783551A Expired - Fee Related CN101510838B (en) | 2009-02-26 | 2009-02-26 | Method for implementing perdurable data queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101510838B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103809915B (en) * | 2012-11-05 | 2016-12-21 | 阿里巴巴集团控股有限公司 | The reading/writing method of a kind of disk file and device |
| CN104424186B (en) * | 2013-08-19 | 2018-04-03 | 阿里巴巴集团控股有限公司 | The method and device of persistence is realized in a kind of stream calculation application |
| CN103473329A (en) * | 2013-09-17 | 2013-12-25 | 广州市信息安全测评中心 | Persistent data queue as well as loading method and loading device of persistent data queue |
| CN106598473B (en) * | 2015-10-15 | 2020-09-04 | 南京中兴新软件有限责任公司 | Message persistence method and device |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN107391269B (en) * | 2016-03-28 | 2021-04-20 | 阿里巴巴集团控股有限公司 | Method and equipment for processing message through persistent queue |
| CN106095596A (en) * | 2016-05-27 | 2016-11-09 | 努比亚技术有限公司 | Message dilivery method and device |
| US11023275B2 (en) * | 2017-02-09 | 2021-06-01 | Intel Corporation | Technologies for queue management by a host fabric interface |
| CN107589907B (en) * | 2017-08-10 | 2019-12-13 | 深圳壹账通智能科技有限公司 | Data processing method, electronic device and computer readable storage medium |
| CN111866148B (en) * | 2020-07-23 | 2022-05-31 | 浪潮云信息技术股份公司 | Message queue flow control system |
| CN112087338B (en) * | 2020-09-14 | 2022-09-27 | 武汉虹信科技发展有限责任公司 | Control message transmission method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1905518A (en) * | 2005-07-29 | 2007-01-31 | 北京航空航天大学 | Method for ensuring reliable transmission of data exhange |
| CN101184056A (en) * | 2007-12-04 | 2008-05-21 | 浙江大学 | Distributed enterprise service bus message switching method |
-
2009
- 2009-02-26 CN CN2009100783551A patent/CN101510838B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1905518A (en) * | 2005-07-29 | 2007-01-31 | 北京航空航天大学 | Method for ensuring reliable transmission of data exhange |
| CN101184056A (en) * | 2007-12-04 | 2008-05-21 | 浙江大学 | Distributed enterprise service bus message switching method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101510838A (en) | 2009-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101510838B (en) | Method for implementing perdurable data queue | |
| KR102240557B1 (en) | Method, device and system for storing data | |
| US8347004B2 (en) | Systems and methods for compression of data for block mode access storage | |
| CA2564844C (en) | Method and system for compression of files for storage and operation on compressed files | |
| US8473652B2 (en) | Systems and methods for compression of data for block mode access storage | |
| EP2871809A1 (en) | Message processing method, device and system for internet of things | |
| CN105808151B (en) | Solid state hard disk stores the data access method of equipment and solid state hard disk storage equipment | |
| CN103095769B (en) | Across method of data synchronization and the system of machine room | |
| CN100481028C (en) | Method and device for implementing data storage using cache | |
| CN101707633B (en) | Message-oriented middleware persistent message storing method based on file system | |
| US20060190643A1 (en) | Method and system for compression of data for block mode access storage | |
| US20120284369A1 (en) | System, method and computer program product for managing a remote storage | |
| CN102130959A (en) | System and method for scheduling cloud storage resource | |
| CN103838830A (en) | Data management method and system of HBase database | |
| CN107786638A (en) | A kind of data processing method, apparatus and system | |
| US20080320062A1 (en) | Method of transferring file system, file system transference program, and file system transference device | |
| CN103036948A (en) | Network file processing method and execution node and software as a service (SaaS) platform | |
| US20080222255A1 (en) | Systems and methods for providing high performance and scalable messaging | |
| CN106020713A (en) | File storage method based on buffer area | |
| CN102520879B (en) | Priority-based file information storage method, device and system | |
| CN108647278A (en) | A kind of file management method and system | |
| CN101404797A (en) | Storage method, storage management apparatus and storage system for long message | |
| CN102902748A (en) | Establishing method and managing method for file systems and random access memory (RAM) and communication chip of file systems | |
| CN103488768A (en) | File management method and file management system based on cloud computing | |
| CN104333803B (en) | Method, equipment and the system of frame losing when preventing video editing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20120704 Termination date: 20210226 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |