CN110231997A - A kind of method and system of quick creation message queue file - Google Patents
A kind of method and system of quick creation message queue file Download PDFInfo
- Publication number
- CN110231997A CN110231997A CN201910501803.8A CN201910501803A CN110231997A CN 110231997 A CN110231997 A CN 110231997A CN 201910501803 A CN201910501803 A CN 201910501803A CN 110231997 A CN110231997 A CN 110231997A
- Authority
- CN
- China
- Prior art keywords
- message
- message queue
- file
- creation
- queue file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 230000002085 persistent effect Effects 0.000 description 3
- 239000000945 filler Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method and system of quickly creation message queue file, belong to the basic software technical field of message-oriented middleware.For the method for quick creation message queue file of the invention in the available message queue file of creation message-oriented middleware, only message queue file header and tail of file filling data occupies disk space, has a cavity in centre.The method of the quick creation message queue file of the invention can facilitate related application, improve user experience, have good application value in second grade creation tens or even a available message queue files of message-oriented middleware up to a hundred.
Description
Technical field
The present invention relates to the basic software technical fields of message-oriented middleware, specifically provide a kind of quickly creation message queue text
The method and system of part.
Background technique
Message-oriented middleware (Message Oriented Middleware) refer to support and ensure distributed application program it
Between synchronous/asynchronous messaging middleware.Message-oriented middleware is a kind of independent system software or service routine, it is being calculated
It is a critical software in machine system, between user's application and operation system and network software, is provided for application public
Means of communication, and independently of network and operating system.Message-oriented middleware provides for developer is common used to all environment
Application programming interfaces, when being embedded in its function call in application program, it can utilize the specific operation system and network of its operation
The function of environment is application execution communication function.
The message storage of message-oriented middleware is divided into non-persistentization storage and persistent storage, and non-persistentization storage will generally disappear
Breath directly exists in physical memory, after power down or system closing, information drop-out；Persistent storage generally stores message to disk
In file, some is directly stored in file, some and utilizes NoSql database using relational database, some.
Theme (Topic) is a key concept in message-oriented middleware, and in persistent storage, a general theme is needed
Want a message queue file corresponding, cloud messaging service platform CMSP is exactly this message-oriented middleware, and CMSP's is every
A node can support thousands of Topic, CMSP one Topic of every creation that it is opposite to require a message queue file
It answers and (is created before reusable).In the earlier version of CMSP, there is so a kind of situation, user will create into a short time sometimes
Hundred thousands of Topic, the amount of capacity of Topic differ, and the Topic creation demand for a 100MB size is exactly to create one
The message queue file of a 100MB takes around 1 second time under the conditions of common mechanical disk, and for one
The Topic of 1GB size then needs 10 seconds or so time, 100 this Topic just may require that 15 minutes or more when
Between, the time waited from the user's point of view will be long, and experience is not very well, to influence some applications based on it yet.
Slow reason is disk bottleneck, and the write performance of general mechanical disk is about in 100MB/S or so.
A kind of batch, which is disclosed, application No. is the patent of invention of CN201410826066 quickly creates file system metadata
It with the method for data, is copied specific to the working set of precognition, such as catalogue, compressed file decompression operations etc., is guaranteeing file
Under the premise of system reliability, first to the superblock of metadata area, block group descriptor, data bitmap, Inode node bitmap
Three kinds of metadata are modified.Then Inode nodal information and data are successively respectively written into metadata according to the sequence of operation
Region and data area.According to the method for the present invention, firstly, make system avoid the metadata page write back introducing it is frequent with
The small letter problem of machine；Secondly, increase the time of write delay, metadata and data area merger to a greater degree portion are allowed
Divide I/O request, the multiple update operation for causing same metadata object to be issued to disk is merged into a disk update；Again, exist
In metadata and data area, sequence is created with realizing Inode node and data block, reduces time of magnetic head tracking and positioning
Number.But this patent not can solve the problem that message queue document creation is slow in message-oriented middleware, reason is mechanical disk
Write performance does not catch up with the write request of message queue file mass data.
Summary of the invention
Technical assignment of the invention is that in view of the above problems, tens can be created even in second grade by providing one kind
The available message queue files of a message-oriented middleware up to a hundred, facilitate related application, and the quick creation for improving user experience disappears
Cease the method for queue file.
The system that the further technical assignment of the present invention is to provide a kind of quickly creation message queue file.
To achieve the above object, the present invention provides the following technical scheme that
A kind of method of quick creation message queue file, in the available message queue file of creation message-oriented middleware,
Only message queue file header and tail of file filling data occupies disk space, has a cavity in centre.Therefore
The magnetic disc i/o very little actually occurred, avoids disk performance bottleneck problem.The present invention can be among second grade quickly creation message
The available large number of message queue file of part, the big message queue file including GB grades of sizes.
The method of the quick creation message queue file cannot be only used for cloud messaging service platform CMSP, it can also be used to other
Message-oriented middleware.
Preferably, this method detailed process is under an operating system, to be first turned on an empty file and write-in one disappears
Queue file's head is ceased, secondly file pointer is displaced to the tail portion of predefined size file, then is written with the empty data of 0 filling, finally
Close file.
Preferably, the operating system is (SuSE) Linux OS.
Preferably, opening an empty file and the size of a message queue file header is written is tens KB.
Preferably, file pointer is displaced to the tail portion of predefined size file, then 4KB is written with the empty data of 0 filling.
A cavity is left between message queue file header and tail of file, cavity does not generate actual I/O, because
This can also complete the super big message queue file of 1 1TB second grade, and the disk space actually occupied only has tens KB, this
Tens spaces KB are mainly what message queue file header occupied, but in the message between part apparently, which is
1TB, the subsequent disk space that just really occupies when cavity position has real data write-in are concurrently given birth to magnetic disc i/o, are thus solved
Message queue file quickly creates problem.
Preferably, the message queue file header includes but is not limited to message queue file name, message queue file
Size, first message offset amount, the last item message offset amount, adds up to write message bar number, adds up to read to disappear at message queue version number
Breath item number does not consume message bar number currently, adds up to write message size, accumulative reading message size, message queue backup information, message
The last read time of queue, the message queue last write operation time.
A kind of system of quick creation message queue file, including message queue file header creation module and filling data wound
Modeling block: message queue file header creation module is for creating the available message queue file header of message-oriented middleware；Fill data
Creation module is for creating tail of file filling data；Wherein in the message-oriented middleware of message queue file header creation module creation
There is a cavity among available message queue file header and the tail of file filling data of filling data creation module creation.
Preferably, it includes but is not limited to message that the message queue file header, which includes but is not limited to message queue file header,
Queue file's name, message queue file size, message queue version number, first message offset amount, the last item message offset
It measures, adds up to write message bar number, accumulative reading message bar number, do not consume message bar number currently, add up to write message size, add up to read message
Size, message queue backup information, the last read time of message queue, message queue last write operation time.
Compared with prior art, the method for quick creation message queue file of the invention has beneficial effect following prominent
Fruit: the method for the quick creation message queue file ensures all size message queue file under (SuSE) Linux OS
Quickly creation can create the available message queue file of dozens to hundreds of a message-oriented middleware in 1 second, substantially improve wound
The experience for building message queue file, facilitates related application, has good application value.
Detailed description of the invention
Fig. 1 is the message-oriented middleware message queue of the method initial creation of quick creation message queue file of the present invention
File format schematic diagram.
Specific embodiment
Below in conjunction with drawings and examples, to the method and system of quick creation message queue file of the invention make into
One step is described in detail.
Embodiment
The method of quick creation message queue file of the invention, in the creation available message queue file of message-oriented middleware
When, only message queue file header and tail of file filling data occupies disk space, has a cavity in centre.Cause
This magnetic disc i/o very little actually occurred avoids disk performance bottleneck problem.The present invention can be in second grade quickly creation message
Between the available large number of message queue file of part, the big message queue file including GB grades of sizes.
The method of the quick creation message queue file is realized by quickly creating the system of message queue file.Quickly
The system of creation message queue file includes message queue file header creation module and filling data creation module: message queue text
Part head creation module is for creating the available message queue file header of message-oriented middleware；Filling data creation module is for creating text
Fill data in part tail portion；Wherein in the available message queue file of message-oriented middleware of message queue file header creation module creation
There is a cavity among head and the tail of file filling data of filling data creation module creation.
The method of the quick creation message queue file cannot be only used for cloud messaging service platform CMSP, it can also be used to other
Message-oriented middleware.
This method detailed process is under a linux operating system, to be first turned on an empty file and tens KB are written
Message queue file header, secondly file pointer be displaced to the tail portion of predefined size file, then be written 4KB with 0 filling empty number
According to finally closing file.
A cavity is left between message queue file header and tail of file, cavity does not generate actual I/O, because
This can also complete the super big message queue file of 1 1TB second grade, and the disk space actually occupied only has tens KB, this
Tens spaces KB are mainly what message queue file header occupied, but in the message between part apparently, which is
1TB, the subsequent disk space that just really occupies when cavity position has real data write-in are concurrently given birth to magnetic disc i/o, are thus solved
Message queue file quickly creates problem.
Message queue file header in the present invention includes but is not limited to that message queue file name, message queue file are big
Small, message queue version number, the last item message offset amount, adds up to write message bar number, accumulative reading message at first message offset amount
Item number does not consume message bar number currently, adds up to write message size, accumulative reading message size, message queue backup information, message team
Arrange last read time, the message queue last write operation time.
The system of quick creation message queue file of the invention, including message queue file header creation module and filler
According to creation module: message queue file header creation module is for creating the available message queue file header of message-oriented middleware；Filling
Data creation module is for creating tail of file filling data；Wherein in the message of message queue file header creation module creation
Between the available message queue file header of part and filling data creation module creation tail of file filling data among have a sky
Hole.
It includes but is not limited to message queue file that wherein message queue file header, which includes but is not limited to message queue file header,
Name, message queue version number, first message offset amount, the last item message offset amount, adds up to write at message queue file size
Message bar number, accumulative reading message bar number do not consume message bar number currently, add up to write message size, add up to read message size, message
The last read time of queue backup information, message queue, the message queue last write operation time.
The system of the quick creation message queue file can be realized quickly creation message queue file, detailed process are as follows:
When creating the available message queue file of message-oriented middleware, only message queue file header and tail of file filler
According to disk space is occupied, there is a cavity in centre.Therefore the magnetic disc i/o very little actually occurred, avoids disk performance bottleneck
Problem.The present invention can quickly create the available large number of message queue file of message-oriented middleware, including GB grades in second grade
The big message queue file of size.
The specific message team under a linux operating system, to be first turned on an empty file and tens KB being written
Column file header, secondly file pointer is displaced to the tail portion of predefined size file, then 4KB is written with the empty data of 0 filling, finally closes
Closed file.
Embodiment described above, the only present invention more preferably specific embodiment, those skilled in the art is at this
The usual variations and alternatives carried out within the scope of inventive technique scheme should be all included within the scope of the present invention.
Claims (8)
1. a kind of method of quickly creation message queue file, it is characterised in that: in the creation available message team of message-oriented middleware
When column file, only message queue file header and tail of file filling data occupies disk space, has one in centre
Cavity.
2. the method for quick creation message queue file according to claim 1, it is characterised in that: this method detailed process
Under an operating system, to be first turned on an empty file and a message queue file header being written, secondly file pointer is displaced to
The tail portion of predefined size file, then be written with the empty data of 0 filling, finally close file.
3. the method for quick creation message queue file according to claim 2, it is characterised in that: the operating system is
(SuSE) Linux OS.
4. the method for quick creation message queue file according to claim 3, it is characterised in that: open an empty file
And the size that a message queue file header is written is tens KB.
5. the method for quick creation message queue file according to claim 4, it is characterised in that: file pointer is displaced to
The tail portion of predefined size file, then 4KB is written with the empty data of 0 filling.
6. the method for quick creation message queue file according to claim 5, it is characterised in that: the message queue text
Part head includes but is not limited to message queue file name, message queue file size, message queue version number, first message offset
Amount, the last item message offset amount add up to write message bar number, accumulative reading message bar number, do not consume message bar number currently, add up to write
Message size, accumulative reading message size, message queue backup information, the last read time of message queue, message queue are last
The write operation time.
7. a kind of system of quickly creation message queue file, it is characterised in that: including message queue file header creation module and
Filling data creation module: message queue file header creation module is for creating the available message queue file of message-oriented middleware
Head；Filling data creation module is for creating tail of file filling data；Wherein created in message queue file header creation module
The available message queue file header of message-oriented middleware and filling data creation module creation tail of file filling data among
There is a cavity.
8. the system of quick creation message queue file according to claim 7, it is characterised in that: the message queue text
Part head include but is not limited to message queue file header include but is not limited to message queue file name, message queue file size,
Message queue version number, the last item message offset amount, adds up to write message bar number, accumulative reading message bar at first message offset amount
Number does not consume message bar number currently, adds up to write message size, accumulative reading message size, message queue backup information, message queue
Last read time, message queue last write operation time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910501803.8A CN110231997A (en) | 2019-06-11 | 2019-06-11 | A kind of method and system of quick creation message queue file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910501803.8A CN110231997A (en) | 2019-06-11 | 2019-06-11 | A kind of method and system of quick creation message queue file |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110231997A true CN110231997A (en) | 2019-09-13 |
Family
ID=67858851
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910501803.8A Pending CN110231997A (en) | 2019-06-11 | 2019-06-11 | A kind of method and system of quick creation message queue file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110231997A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020138582A1 (en) * | 2000-09-05 | 2002-09-26 | Mala Chandra | Methods and apparatus providing electronic messages that are linked and aggregated |
| US20110040811A1 (en) * | 2009-08-17 | 2011-02-17 | International Business Machines Corporation | Distributed file system logging |
| CN104424224A (en) * | 2013-08-26 | 2015-03-18 | 深圳市腾讯计算机系统有限公司 | File index storage method and device |
| CN105068769A (en) * | 2015-08-28 | 2015-11-18 | 浪潮软件股份有限公司 | Message middleware based message management method and apparatus |
| CN108370346A (en) * | 2015-10-09 | 2018-08-03 | 萨托里环球有限责任公司 | System and method for storing and transmitting message data |
-
2019
- 2019-06-11 CN CN201910501803.8A patent/CN110231997A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020138582A1 (en) * | 2000-09-05 | 2002-09-26 | Mala Chandra | Methods and apparatus providing electronic messages that are linked and aggregated |
| US20110040811A1 (en) * | 2009-08-17 | 2011-02-17 | International Business Machines Corporation | Distributed file system logging |
| CN104424224A (en) * | 2013-08-26 | 2015-03-18 | 深圳市腾讯计算机系统有限公司 | File index storage method and device |
| CN105068769A (en) * | 2015-08-28 | 2015-11-18 | 浪潮软件股份有限公司 | Message middleware based message management method and apparatus |
| CN108370346A (en) * | 2015-10-09 | 2018-08-03 | 萨托里环球有限责任公司 | System and method for storing and transmitting message data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10884633B2 (en) | System and method for optimized signature comparisons and data replication | |
| US20230325287A1 (en) | Data backup method and apparatus, data recovery method and apparatus, and storage medium | |
| US20170068469A1 (en) | Remote Shared Virtual Disk Snapshot Creation | |
| US8051054B2 (en) | Method and system for data processing with database update for the same | |
| CN109710185A (en) | Data processing method and device | |
| CN101640587A (en) | Data synchronization method and data synchronization device | |
| WO2024113717A1 (en) | Zns ssd-based file system control method, apparatus and device | |
| CN110502523A (en) | Business datum storage method, device, server and computer readable storage medium | |
| CN100504800C (en) | Method for snapshot of magnetic disc | |
| US20080320062A1 (en) | Method of transferring file system, file system transference program, and file system transference device | |
| CN114138200A (en) | Pre-writing log method and system based on rocksDB | |
| CN102833273A (en) | Data restoring method when meeting temporary fault and distributed caching system | |
| CN102073554A (en) | Method and device for recovering files closed abnormally | |
| US20120209891A1 (en) | Database management method, database management system and database management program | |
| US11580110B2 (en) | Methods and apparatuses for generating redo records for cloud-based database | |
| CN109983452A (en) | System and method for continuously available Network File System (NFS) status data | |
| CN105677579A (en) | Data access method and system in cache system | |
| CN110231997A (en) | A kind of method and system of quick creation message queue file | |
| CN114138424B (en) | Virtual machine memory snapshot generation method and device and electronic equipment | |
| CN112882658B (en) | Expansion method and device for data cache, computer equipment and storage medium | |
| CN109669878B (en) | Simplest cache data operation method | |
| CN111026764B (en) | Data storage method and device, electronic product and storage medium | |
| CN109960460B (en) | Distributed storage system | |
| JPS58223857A (en) | Operation of calculator | |
| CN101799758B (en) | Specification management device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190913 |
|
| RJ01 | Rejection of invention patent application after publication |