CN104102552A - Message processing method and device - Google Patents
Message processing method and device Download PDFInfo
- Publication number
- CN104102552A CN104102552A CN201310130431.5A CN201310130431A CN104102552A CN 104102552 A CN104102552 A CN 104102552A CN 201310130431 A CN201310130431 A CN 201310130431A CN 104102552 A CN104102552 A CN 104102552A
- Authority
- CN
- China
- Prior art keywords
- data
- data file
- file
- message
- written
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 23
- 238000013507 mapping Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 13
- 238000004064 recycling Methods 0.000 claims description 2
- 230000005856 abnormality Effects 0.000 abstract 1
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000003139 buffering effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a message processing method. The method comprises the following steps of creating a data file and an index file corresponding to the data file, and mapping the data file and the index file corresponding to the data file to a memory; writing message data-to-be-written into the data file, and filling the index file corresponding to the data file according to the information of the data file of the message data-to-be-written; reading the message data in the data file corresponding to the index file according to the index file. The invention also discloses a message processing device. By adopting the message processing method and device, the message data is protected against loss when the quit is caused by the abnormality of the process while the high reading-writing efficiency of the message is guaranteed.
Description
Technical Field
The invention relates to the internet communication technology, in particular to a message processing method and a message processing device.
Background
Currently, there are several schemes for processing a message queue:
the first scheme is as follows: buffering the message by a process-level memory, and asynchronously storing the message to a database;
scheme II: buffering the message by a process-level memory, and asynchronously storing the message to a data file;
the third scheme is as follows: the messages are written to the binary file sequentially and the binary file is asynchronously saved to the data file.
Aiming at the scheme I, the memory buffer of the process level can realize the rapid receiving of the message, but if the speed of storing the message into the database is slow, the receiving quantity of the buffer message of the memory buffer of the process level can be influenced, because the buffer is limited, and when the process is abnormal to cause exit, the buffer message is lost completely; the message throughput of the scheme is generally 5000-8000 messages per second.
For the second scheme, in order to solve the problem of the speed of saving the first scheme to the database, a mode of directly saving the first scheme to the data file is adopted, and the data file is a binary file, so that the problem of low storage speed caused by Structured Query Language (SQL) analysis and transaction control of the database is solved, the throughput of messages can be improved by 20-30%, but the scheme also avoids the problem that buffered messages are lost when the process is abnormal and the buffered messages are quitted.
Aiming at the third scheme, in order to solve the problem that buffered messages are lost when a process exception occurs in a memory buffer to cause exit, the scheme does not do the process buffer, but directly and sequentially writes a binary file, although the method greatly reduces the time of hard disk seek, the method is slower than the memory buffer of the process, therefore, the message throughput of the scheme is not high, and the buffered messages are not lost when the process exception occurs in the memory buffer to cause exit.
Disclosure of Invention
In view of the above, the main objective of the present invention is to provide a method and an apparatus for processing a message, which can ensure high read-write efficiency of the message and solve the problem of message loss when the process is exited due to an exception.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
the invention provides a message processing method, which comprises the following steps:
creating a data file and an index file corresponding to the data file, and mapping the data file and the index file corresponding to the data file to a memory;
writing the message data to be written into a data file, and filling an index file corresponding to the data file according to the information of the data file written into the message data to be written; and reading the message data in the data file corresponding to the index file according to the index file.
In the above solution, the creating a data file includes:
more than one data file with the same size is created, and the more than one data files with the same size form a data file queue.
In the above scheme, the filling the index file corresponding to the data file according to the information of the data file written with the message data to be written includes:
and acquiring the sequence number of the data file in the data file queue, the position information of the data file and the total number of the currently written message data in the data file, and writing the sequence number, the position information and the total number into an index file corresponding to the data file.
In the above scheme, the writing the message data to be written into the data file includes:
and determining the length information of the message data to be written, representing the message data to be written as binary data, and recording the length information and the binary data of the message data to be written in the data file.
In the above scheme, the method further comprises:
and reading a data file, deleting the mapping of the data file, and recycling the memory mapping area of the data file.
The invention also provides a message processing device, which comprises a creating unit, a control unit, a memory, a data reading unit and a data writing unit; wherein,
the creating unit is used for creating a data file and an index file corresponding to the data file;
the control unit is used for mapping the data file and the index file corresponding to the data file to the memory;
the data reading unit is used for reading the message data in the data file corresponding to the index file according to the index file;
and the data writing unit is used for writing the message data to be written into a data file and filling an index file corresponding to the data file according to the information of the data file written with the message data to be written.
In the above scheme, the creating unit is further configured to create more than one data file with the same size, so that the more than one data files with the same size form a data file queue.
In the above scheme, the control unit is further configured to obtain a sequence number of the data file in the data file queue, location information of the data file, and a total number of message data currently written in the data file;
the data writing unit is further configured to write the sequence number of the data file in the data file queue, the location information of the data file, and the total number of currently written message data in the data file, which are acquired by the control unit, into an index file corresponding to the data file.
In the above scheme, the control unit is further configured to determine length information of message data to be written;
the data writing unit is further configured to write the length information of the message data to be written, which is determined by the control unit, and the message data represented by the binary data into a data file.
In the foregoing solution, the control unit is further configured to delete the mapping of the data file and recycle the memory mapping area of the data file after reading the data file.
The message processing method and the device provided by the invention are used for creating a data file and an index file corresponding to the data file, and mapping the data file and the index file corresponding to the data file to a memory; writing the message data to be written into a data file, and filling an index file corresponding to the data file according to the information of the data file written into the message data to be written; reading message data in the data file corresponding to the index file according to the index file; therefore, high reading and writing efficiency of the message is guaranteed, and the message data cannot be lost when the process is abnormal and quit is caused.
Drawings
FIG. 1 is a flow chart illustrating a message processing method according to the present invention;
FIG. 2 is a schematic diagram of a data file according to the present invention;
fig. 3 is a schematic diagram of the composition structure of the message processing apparatus according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Fig. 1 is a schematic flow chart of a message processing method of the present invention, as shown in fig. 1, including the following steps:
step 101: creating a data file and an index file corresponding to the data file, and mapping the data file and the index file corresponding to the data file to a memory;
further, the creating the data file includes:
more than one data file with the same size is created, and the more than one data files with the same size form a data file queue.
Here, the data file is a data file created according to a preset size, and a plurality of data files form a data file queue; as shown in fig. 2;
wherein, the data file records the length of the message data and the message data expressed by binary data;
further, when reading a data file, an index file is created from the data file, for example: acquiring a sequence number of a data file in a data file queue, and assuming that the sequence number is 1, the location information XXX of the data file is acquired, and the total number of currently written message data in the data file is YYY, the format of the index file is as follows: 1/XXX/YYY, so that the data file can be read quickly according to the index file.
Furthermore, when the data file is written in, the created data file and the corresponding index file are both empty files, and the format of the index file and the format of the data file are reserved;
the size of the data file can be fixed to a certain value, for example, 10 megabits (10M), which can greatly reduce the required memory space.
Step 102: writing the message data to be written into a data file, and filling an index file corresponding to the data file according to the information of the data file written into the message data to be written;
further, the filling of the index file corresponding to the data file according to the information written in the data file of the message data to be written includes:
and acquiring the sequence number of the data file in the data file queue, the position information of the data file and the total number of the currently written message data in the data file, and writing the sequence number, the position information and the total number into an index file corresponding to the data file.
Here, the sequence number of the data file in the data file queue, the location information of the data file, and the total number of message data currently written in the data file may be acquired according to an interface provided by the control unit.
Further, the writing the message data to be written into the data file includes:
and determining the length information of the message data to be written, representing the message data to be written as binary data, and recording the length information and the binary data of the message data to be written in the data file.
Step 103: reading message data in a data file corresponding to the index file according to the index file;
further, the method further comprises: and after reading a data file, deleting the mapping of the data file, recovering the memory mapping area of the data file, and mapping the next data file in the data queue to the memory for continuously reading data.
Here, since the control unit is provided with a read-write interface, on the one hand for reading and writing data and on the other hand for controlling the rate of reading and writing data, and for obtaining the total number of message data written, the total number of message data read, the total number of remaining message data in the data file, etc.
In order to implement the above method, the present invention further provides a message processing apparatus, as shown in fig. 3, the apparatus includes a creating unit 35, a control unit 31, a memory 32, a data reading unit 33, and a data writing unit 34; wherein,
the creating unit 35 is configured to create a data file and an index file corresponding to the data file;
the control unit 31 is configured to map the data file and the index file corresponding to the data file to the memory 32;
the data reading unit 33 is configured to read message data in a data file corresponding to the index file according to the index file;
the data writing unit 34 is configured to write the message data to be written into a data file, and fill an index file corresponding to the data file according to information of the data file written with the message data to be written.
Further, the creating unit 35 is specifically configured to create more than one data file with the same size, so that the more than one data files with the same size form a data file queue.
Further, the control unit 31 is further configured to obtain a sequence number of the data file in the data file queue, location information of the data file, and a total number of message data currently written in the data file;
the data writing unit 34 is further configured to write the sequence number of the data file in the data file queue, the location information of the data file, and the total number of currently written message data in the data file, which are acquired by the control unit 31, into an index file corresponding to the data file.
Here, the control unit is provided with a read-write interface for reading and writing data on the one hand and for controlling the rate of reading and writing data on the other hand, and for obtaining the total number of message data written, the total number of message data read, the total number of remaining message data in the data file, etc.
Further, the control unit 31 is further configured to determine length information of message data to be written;
the data writing unit 34 is further configured to write the length information of the message data to be written, which is determined by the control unit 31, and the message data represented by the binary data into a data file.
Further, the control unit 31 is further configured to delete the mapping of the data file and recycle the memory mapping area of the data file when a data file is read.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present invention are included in the protection scope of the present invention.
Claims (10)
1. A method for message processing, the method comprising:
creating a data file and an index file corresponding to the data file, and mapping the data file and the index file corresponding to the data file to a memory;
writing the message data to be written into a data file, and filling an index file corresponding to the data file according to the information of the data file written into the message data to be written; and reading the message data in the data file corresponding to the index file according to the index file.
2. The message processing method of claim 1, wherein the creating the data file comprises:
more than one data file with the same size is created, and the more than one data files with the same size form a data file queue.
3. The message processing method according to claim 2, wherein the filling the index file corresponding to the data file according to the information of the data file written in the message data to be written comprises:
and acquiring the sequence number of the data file in the data file queue, the position information of the data file and the total number of the currently written message data in the data file, and writing the sequence number, the position information and the total number into an index file corresponding to the data file.
4. The message processing method according to claim 3, wherein the writing the message data to be written into the data file comprises:
and determining the length information of the message data to be written, representing the message data to be written as binary data, and recording the length information and the binary data of the message data to be written in the data file.
5. The message processing method of claim 2, wherein the method further comprises:
and reading a data file, deleting the mapping of the data file, and recycling the memory mapping area of the data file.
6. A message processing device is characterized by comprising a creating unit, a control unit, a memory, a data reading unit and a data writing unit; wherein,
the creating unit is used for creating a data file and an index file corresponding to the data file;
the control unit is used for mapping the data file and the index file corresponding to the data file to the memory;
the data reading unit is used for reading the message data in the data file corresponding to the index file according to the index file;
and the data writing unit is used for writing the message data to be written into a data file and filling an index file corresponding to the data file according to the information of the data file written with the message data to be written.
7. The message processing apparatus according to claim 6, wherein the creating unit is further configured to create more than one data file with the same size, and make the more than one data files with the same size form a data file queue.
8. The message processing apparatus according to claim 7,
the control unit is also used for acquiring the serial number of the data file in the data file queue, the position information of the data file and the total number of the message data currently written in the data file;
the data writing unit is further configured to write the sequence number of the data file in the data file queue, the location information of the data file, and the total number of currently written message data in the data file, which are acquired by the control unit, into an index file corresponding to the data file.
9. The message processing apparatus according to claim 7,
the control unit is also used for determining the length information of the message data to be written;
the data writing unit is further configured to write the length information of the message data to be written, which is determined by the control unit, and the message data represented by the binary data into a data file.
10. The message processing apparatus according to claim 7, wherein the control unit is further configured to delete the mapping of the data file and recycle the memory mapping area of the data file when reading a data file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310130431.5A CN104102552A (en) | 2013-04-15 | 2013-04-15 | Message processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310130431.5A CN104102552A (en) | 2013-04-15 | 2013-04-15 | Message processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104102552A true CN104102552A (en) | 2014-10-15 |
Family
ID=51670723
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310130431.5A Pending CN104102552A (en) | 2013-04-15 | 2013-04-15 | Message processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104102552A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138410A (en) * | 2015-08-31 | 2015-12-09 | 北京锐安科技有限公司 | Message queue achievement method and device based on disk buffer |
| CN107133111A (en) * | 2017-04-27 | 2017-09-05 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN108132845A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | Message storage, delivering method and device and electronic equipment |
| CN111541667A (en) * | 2020-04-16 | 2020-08-14 | 广东好太太科技集团股份有限公司 | Method, equipment and storage medium for intersystem message communication |
| CN112486913A (en) * | 2020-11-12 | 2021-03-12 | 思创数码科技股份有限公司 | Log asynchronous storage method and device based on cluster environment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1517906A (en) * | 2003-01-14 | 2004-08-04 | 联想(北京)有限公司 | Document system and document management method |
| CN101533408A (en) * | 2009-04-21 | 2009-09-16 | 北京四维图新科技股份有限公司 | Processing method and processing device of mass data |
| CN101783740A (en) * | 2009-01-21 | 2010-07-21 | 大唐移动通信设备有限公司 | Method and device for managing message file |
-
2013
- 2013-04-15 CN CN201310130431.5A patent/CN104102552A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1517906A (en) * | 2003-01-14 | 2004-08-04 | 联想(北京)有限公司 | Document system and document management method |
| CN101783740A (en) * | 2009-01-21 | 2010-07-21 | 大唐移动通信设备有限公司 | Method and device for managing message file |
| CN101533408A (en) * | 2009-04-21 | 2009-09-16 | 北京四维图新科技股份有限公司 | Processing method and processing device of mass data |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138410A (en) * | 2015-08-31 | 2015-12-09 | 北京锐安科技有限公司 | Message queue achievement method and device based on disk buffer |
| CN108132845A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | Message storage, delivering method and device and electronic equipment |
| CN107133111A (en) * | 2017-04-27 | 2017-09-05 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN111541667A (en) * | 2020-04-16 | 2020-08-14 | 广东好太太科技集团股份有限公司 | Method, equipment and storage medium for intersystem message communication |
| CN112486913A (en) * | 2020-11-12 | 2021-03-12 | 思创数码科技股份有限公司 | Log asynchronous storage method and device based on cluster environment |
| CN112486913B (en) * | 2020-11-12 | 2023-09-22 | 思创数码科技股份有限公司 | Log asynchronous storage method and device based on cluster environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9990276B2 (en) | Read-write control method for memory, and corresponding memory and server | |
| CN102158349B (en) | A kind of log management apparatus and method | |
| CN104268099B (en) | A kind of method and device for managing reading and writing data | |
| CN103246696A (en) | High-concurrency database access method and method applied to multi-server system | |
| CN104102552A (en) | Message processing method and device | |
| CN108628542B (en) | File merging method and controller | |
| CN103985393B (en) | A kind of multiple optical disk data parallel management method and device | |
| CN102479249A (en) | Method for eliminating cache data of memory of embedded browser | |
| CN104268088A (en) | Vehicle DVR (Digital Video Recorder) hard disk data storage method | |
| CN102521349A (en) | Pre-reading method of files | |
| CN107608625B (en) | Method for improving reading performance of fixed storage equipment | |
| CN104657366A (en) | Method and device for writing mass logs in database and log disaster-tolerant system | |
| CN110515550B (en) | Method and device for separating cold data and hot data of SATA solid state disk | |
| CN112346659B (en) | Storage method, equipment and storage medium for distributed object storage metadata | |
| CN103856516A (en) | Data storage and reading method and data storage and reading device | |
| CN102999441B (en) | Fine granularity memory access method | |
| CN103729149A (en) | Data storage method | |
| CN103136215A (en) | Data read-write method and device of storage system | |
| CN107729504A (en) | A kind of method and system for handling large data objectses | |
| CN111291022B (en) | Data storage system based on block chain | |
| CN105260139A (en) | Magnetic disk management method and system | |
| CN108647278B (en) | File management method and system | |
| CN108491166B (en) | Read data cache management method for optical disk library | |
| CN107229580A (en) | Sequential stream detection method and apparatus | |
| CN115840654B (en) | Message processing method, system, computing device and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20141015 |
|
| RJ01 | Rejection of invention patent application after publication |