CN106681658A - Method for achieving high-speed transfer of mass data of data recorder on basis of multithreading - Google Patents
Method for achieving high-speed transfer of mass data of data recorder on basis of multithreading Download PDFInfo
- Publication number
- CN106681658A CN106681658A CN201611049332.4A CN201611049332A CN106681658A CN 106681658 A CN106681658 A CN 106681658A CN 201611049332 A CN201611049332 A CN 201611049332A CN 106681658 A CN106681658 A CN 106681658A
- Authority
- CN
- China
- Prior art keywords
- data
- read
- thread
- write
- shared buffer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
Abstract
The invention belongs to the field of software algorithm designing, and particularly relates to a method for achieving high-speed transfer of mass data of a data recorder on the basis of multithreading. Compared with the prior art, the method has the following advantages that by means of a concurrent algorithm designing and multithreading programming implementing tool, the high-performance storage system hardware capacity is exerted to the maximum extent, and the resource utilization rate is increased; the data unloading speed of the data recorder can be significantly increased, the working efficiency of the data recorder is improved, and the practical utilization rate of the data recorder is increased.
Description
Technical field
The invention belongs to software algorithm design field, and in particular to one kind realizes data logger magnanimity number based on multithreading
According to the method for high speed unloading, its with the computer system of multi-core CPU as platform, for mass data in different memory space unloadings
Application demand, realize that the read-write of data conversion storage process is concurrent using multithreading instrument, reach the mesh for lifting data conversion storage speed
's.
Background technology
In numerous armies such as military flight test, unmanned plane investigation, filming image, monitoring is shot with video-corder, field is surveyed and drawn, track traffic
With and civil area, the various data and information in being required for jumbo exclusive data monitor for writing task.With
Logger task becomes increasingly complex, and the information category and quantity of required record is more and more, and the storage of various data loggers is held
Amount also becomes more huge.
Although memory capacity is a most important index of data logger, simultaneously for data logger for, only
There are the data that will be stored in data logger and information to dump to the calculation server in ground environment, or the letter of networking
In breath system (hereinafter referred data are imperial to be carried), the parsing and process of specialty in detail could be carried out deeply by people, so as to obtain
Analysis result.So, the data of data logger are imperial to carry speed, is the another item key index of data logger.With nothing
As a example by man-machine application, the memory capacity of the Airborne Data Recorder of many military, civilian unmanned planes is all up to several TB, by meter
Calculation machine interface (USB, SATA) or network interface, complete load shedding and are required for several hours to more than ten hour, and this is direct
The work efficiency of data analysiss and the utilization rate of data logger are affected, and then affects the work frequency of unmanned plane and using effect
Energy.The load shedding speed for how improving data logger is in the increasingly huger situation of current data monitor memory capacity
The problem that must be faced down and solve.
To solve this problem, many High Performance Cache and Memory System hardware structures are devised in the industry, main thought is to lift number
According to monitor and the concurrent reading and writing speed of ground-based server memory space, while the parallel high-speed data unloading interface of design,
Prop up the high speed to mass data to unload.One representational industrial data monitor ground data unloading hardware structure is for example attached
Shown in Fig. 1.Under the hardware structure shown in accompanying drawing 1, traditional load shedding tool work flow process of ground-based server is run on
As shown in Figure 2, it is described in detail below：
1) reading specifies the data block of size in internal memory from data logger；
2) data block in internal memory is stored in server storage；
3) repeat the operation of aforementioned two step in monitor all data conversion storages in server storage.
Under above-mentioned traditional load shedding tool flow, the order situation of the read-write of data logger data conversion storage is visible
Accompanying drawing 3.Analysis understands that uninstall is directed to each data block, devises reading and writing serial operation, i.e., each data block is complete
Two steps of reading and writing will be sequentially completed into the unloading from data logger to ground-based server, and be read from data logger
The speed of data block is N byte/second, and earthward the speed of server write is P byte per seconds.This uniprocessor in low performance
The server epoch of core are helpless methods, and at present, ground-based server all adopts already high-performance multicore architecture, completely can be with
Read-write operation is got up parallel, hardware resource is made full use of, load shedding speed is lifted.
The content of the invention
(1) technical problem to be solved
The technical problem to be solved in the present invention is：Such as how the high-performance server system of polycaryon processor is platform, is changed
Enter the algorithm and flow process of traditional load shedding instrument, the reading and writing serial operation of data conversion storage in load shedding work is revised as
Parallel work-flow, improves the work efficiency of load shedding work, the speed of data conversion storage is significantly put forward, so as to reduce data record
Time required for instrument load shedding.
(2) technical scheme
To solve above-mentioned technical problem, the present invention provides one kind and realizes data logger mass data at a high speed based on multithreading
The method of unloading, the method comprises the steps：
Step 1：Load shedding instrument hews out first two shared buffers in server memory, each relief area foot
It is enough to meet one unloading data block of preservation；Two shared buffers divide into the first shared buffer and the second shared buffer；
Step 2：Two shared buffers are set to be accessed by multiple threads, are the same of data in guarantee relief area
Step and concordance, are that each relief area is equipped with a signal lighties, respectively the first signal lighties and secondary signal lamp, for controlling
Read-write to corresponding buffering area；Thread only could be written and read behaviour when obtaining to signal lighties to specified relief area
Make；
Step 3：Two threads of initialization, respectively read thread and write thread, and the access of two shared buffers is believed
Signal lamp is initialized to read thread control；
Step 4：Load shedding instrument is while two threads of startup optimization；The follow procedures 5- steps in thread running is read
Rapid 7 flow process；The flow process of follow procedures 8- steps 10 in thread running is write；
Step 5：Read the first signal lighties of the shared buffer of thread request first, if failure the first signal lighties waited,
Read block in the first shared buffer, discharges the first signal lighties from data logger if success；
Step 6：Read thread to be further continued for asking the secondary signal lamp of the second shared buffer, wait second to believe if failure
Signal lamp；Read block in the second shared buffer, discharges secondary signal lamp from data logger if success；
Step 7：Read thread repeat step 5- step 6 to operate until the internal all data read-outs of monitor storage；
Step 8：The first signal lighties of the shared buffer of thread request first are write, if failure the first signal lighties are waited,
Write in home server memory space from read block in the first shared buffer if success, discharge the first signal
Lamp；
Step 9：Write thread to be further continued for asking the secondary signal lamp of the second shared buffer, wait second to believe if failure
Signal lamp；Write in home server memory space from read block in the second shared buffer if success, release second
Signal lighties；
Step 10：Write thread repeat step 8- step 9 and operate unloading until completing all data.
(3) beneficial effect
Compared with prior art, the present invention possesses following beneficial effect：
(1) instrument realized using Concurrent Algorithms design and multi-thread programming, has farthest played high-performance storage
System hardware ability, improves resource utilization.
(2) invention can significantly lift data logger load shedding speed, improve the work effect of data logger
Rate and put into practice utilization rate.
Description of the drawings
Fig. 1 is representative mass data record instrument ground data unloading hardware structure.
Fig. 2 is operate in the workflow diagram of the traditional data tool of unloading of ground-based server, is instrument under dotted line in figure
Workflow, is actual data stream on dotted line.
Fig. 3 is the fetch-store execution ordering analysis chart of data conversion storage in traditional load shedding instrument.
Fig. 4 is the new data tool of unloading workflow diagram designed in the present invention, is tool work flow process under dotted line in figure,
It is actual data stream on dotted line.
Fig. 5 is data conversion storage fetch-store execution ordering figure in the new data tool of unloading designed in the present invention.
Specific embodiment
To make the purpose of the present invention, content and advantage clearer, with reference to the accompanying drawings and examples, to the present invention's
Specific embodiment is described in further detail.
For the ground-based server of polycaryon processor system, under the support of operating system, the same time can be run many
Individual thread.The present invention utilizes operating system multithreading instrument DLL, carries out the algorithm and flow process of new load shedding instrument
Design, by starting two concurrent threads in the tool, i.e., one reading thread and one write thread, by the reading and writing of data conversion storage
Operation is communicated between two threads respectively by two thread complete independentlies by special signal lighties and shared drive, real
Now to reading and writing the control of flow process and the exchange of data.
Specifically, it is to solve problem of the prior art, the present invention provides one kind and realizes data logger based on multithreading
The method of mass data high speed unloading, as shown in fig. 4-5, the method comprises the steps：
Step 1：Load shedding instrument hews out first two shared buffers in server memory, each relief area foot
It is enough to meet one unloading data block of preservation；Two shared buffers divide into the first shared buffer and the second shared buffer；
Step 2：Two shared buffers are set to be accessed by multiple threads, are the same of data in guarantee relief area
Step and concordance, are that each relief area is equipped with a signal lighties, respectively the first signal lighties and secondary signal lamp, for controlling
Read-write to corresponding buffering area；Thread only could be written and read behaviour when obtaining to signal lighties to specified relief area
Make；
Step 3：Two threads of initialization, respectively read thread and write thread, and the access of two shared buffers is believed
Signal lamp is initialized to read thread control；
Step 4：Load shedding instrument is while two threads of startup optimization；The follow procedures 5- steps in thread running is read
Rapid 7 flow process；The flow process of follow procedures 8- steps 10 in thread running is write；
Step 5：Read the first signal lighties of the shared buffer of thread request first, if failure the first signal lighties waited,
Read block in the first shared buffer, discharges the first signal lighties from data logger if success；
Step 6：Read thread to be further continued for asking the secondary signal lamp of the second shared buffer, wait second to believe if failure
Signal lamp；Read block in the second shared buffer, discharges secondary signal lamp from data logger if success；
Step 7：Read thread repeat step 5- step 6 to operate until the internal all data read-outs of monitor storage；
Step 8：The first signal lighties of the shared buffer of thread request first are write, if failure the first signal lighties are waited,
Write in home server memory space from read block in the first shared buffer if success, discharge the first signal
Lamp；
Step 9：Write thread to be further continued for asking the secondary signal lamp of the second shared buffer, wait second to believe if failure
Signal lamp；Write in home server memory space from read block in the second shared buffer if success, release second
Signal lighties；
Step 10：Write thread repeat step 8- step 9 and operate unloading until completing all data.
By the algorithm and flow process of the load shedding instrument of the present invention, data conversion storage fetch-store execution ordering figure as shown in Figure 5, can
To find out, reading and writing parallelization can make to write thread and operate under full load condition that (the reason for producing such is because typically coming
Say, the speed that the speed ratio of reading is write is fast, if reading to write slowly soon reading thread can be made to meet load condition), it is written locally speed
Reach optimal, load shedding speed has and increases substantially.
The above is only the preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, on the premise of without departing from the technology of the present invention principle, some improvement and deformation can also be made, these improve and deform
Also should be regarded as protection scope of the present invention.
Claims (1)
1. a kind of method that data logger mass data high speed unloading is realized based on multithreading, it is characterised in that the method bag
Include following steps：
Step 1：Load shedding instrument hews out first two shared buffers in server memory, and each relief area is full enough
Foot preserves a unloading data block；Two shared buffers divide into the first shared buffer and the second shared buffer；
Step 2：Two shared buffers are set to be accessed by multiple threads, be ensure relief area in data synchronization and
Concordance, is that each relief area is equipped with a signal lighties, respectively the first signal lighties and secondary signal lamp, for controlling to right
Answer the read-write of relief area；Thread only could be written and read operation when obtaining to signal lighties to specified relief area；
Step 3：Two threads of initialization, respectively read thread and write thread, and by the calling-on signal lamp of two shared buffers
It is initialized to read thread control；
Step 4：Load shedding instrument is while two threads of startup optimization；The follow procedures 5- steps 7 in thread running is read
Flow process；The flow process of follow procedures 8- steps 10 in thread running is write；
Step 5：Read the first signal lighties of the shared buffer of thread request first, if failure the first signal lighties are waited, if
It is successful then from data logger read block in the first shared buffer, discharge the first signal lighties；
Step 6：Read thread to be further continued for asking the secondary signal lamp of the second shared buffer, if failure secondary signal is waited
Lamp；Read block in the second shared buffer, discharges secondary signal lamp from data logger if success；
Step 7：Read thread repeat step 5- step 6 to operate until the internal all data read-outs of monitor storage；
Step 8：The first signal lighties of the shared buffer of thread request first are write, if failure the first signal lighties are waited, if
It is successful then write in home server memory space from read block in the first shared buffer, discharge the first signal lighties；
Step 9：Write thread to be further continued for asking the secondary signal lamp of the second shared buffer, if failure secondary signal is waited
Lamp；Write in home server memory space from read block in the second shared buffer if success, release second is believed
Signal lamp；
Step 10：Write thread repeat step 8- step 9 and operate unloading until completing all data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611049332.4A CN106681658A (en) | 2016-11-25 | 2016-11-25 | Method for achieving high-speed transfer of mass data of data recorder on basis of multithreading |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611049332.4A CN106681658A (en) | 2016-11-25 | 2016-11-25 | Method for achieving high-speed transfer of mass data of data recorder on basis of multithreading |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106681658A true CN106681658A (en) | 2017-05-17 |
Family
ID=58866699
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611049332.4A Pending CN106681658A (en) | 2016-11-25 | 2016-11-25 | Method for achieving high-speed transfer of mass data of data recorder on basis of multithreading |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106681658A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109101192A (en) * | 2018-07-02 | 2018-12-28 | 北京东软医疗设备有限公司 | Date storage method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102368210A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Pipeline based metadata journaling method for distributed system |
| CN102999378A (en) * | 2012-12-03 | 2013-03-27 | 中国科学院软件研究所 | Read-write lock implement method |
| CN103885726A (en) * | 2014-03-20 | 2014-06-25 | 东蓝数码股份有限公司 | Efficient multi-thread log writing-in method |
| CN104184817A (en) * | 2014-08-28 | 2014-12-03 | 广州金山网络科技有限公司 | Download optimizing method and device and terminal |
| CN104252405A (en) * | 2013-06-26 | 2014-12-31 | 腾讯科技（深圳）有限公司 | Log information output method and device |
| CN104881600A (en) * | 2015-06-23 | 2015-09-02 | 中国矿业大学(北京) | Technical method for solving data overflowing in ground penetrating radar data collection system |
-
2016
- 2016-11-25 CN CN201611049332.4A patent/CN106681658A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102368210A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Pipeline based metadata journaling method for distributed system |
| CN102999378A (en) * | 2012-12-03 | 2013-03-27 | 中国科学院软件研究所 | Read-write lock implement method |
| CN104252405A (en) * | 2013-06-26 | 2014-12-31 | 腾讯科技（深圳）有限公司 | Log information output method and device |
| CN103885726A (en) * | 2014-03-20 | 2014-06-25 | 东蓝数码股份有限公司 | Efficient multi-thread log writing-in method |
| CN104184817A (en) * | 2014-08-28 | 2014-12-03 | 广州金山网络科技有限公司 | Download optimizing method and device and terminal |
| CN104881600A (en) * | 2015-06-23 | 2015-09-02 | 中国矿业大学(北京) | Technical method for solving data overflowing in ground penetrating radar data collection system |
Non-Patent Citations (1)
| Title |
|---|
| 黄明: "《实时多线程系统的日志实现》", 《软件世界》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109101192A (en) * | 2018-07-02 | 2018-12-28 | 北京东软医疗设备有限公司 | Date storage method and device |
| CN109101192B (en) * | 2018-07-02 | 2021-04-13 | 北京东软医疗设备有限公司 | Data storage method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10754550B2 (en) | Optimized data placement for individual file accesses on deduplication-enabled sequential storage systems | |
| Byna et al. | ExaHDF5: Delivering efficient parallel I/O on exascale computing systems | |
| CN106066890B (en) | Distributed high-performance database all-in-one machine system | |
| US9135262B2 (en) | Systems and methods for parallel batch processing of write transactions | |
| WO2017173919A1 (en) | Concurrent data caching method and structure | |
| CN102637147A (en) | Storage system using solid state disk as computer write cache and corresponding management scheduling method | |
| US10515078B2 (en) | Database management apparatus, database management method, and storage medium | |
| CN104572106A (en) | Concurrent program developing method for processing of large-scale data based on small memory | |
| CN107247624B (en) | Key-Value system oriented collaborative optimization method and system | |
| US20140215127A1 (en) | Apparatus, system, and method for adaptive intent logging | |
| CN106445405A (en) | Flash storage-oriented data access method and apparatus | |
| CN105681402A (en) | Distributed high speed database integration system based on PCIe flash memory card | |
| Ouyang et al. | CRFS: A lightweight user-level filesystem for generic checkpoint/restart | |
| WO2023185111A1 (en) | Quick access method and device for data file | |
| US9223690B2 (en) | Freeing memory safely with low performance overhead in a concurrent environment | |
| CN104239231B (en) | A kind of method and device for accelerating L2 cache preheating | |
| CN102929935A (en) | Transaction-based large-volume data read and write methods | |
| CN103645873B (en) | A kind of method realizing efficient data caching in trend curve system | |
| CN106681658A (en) | Method for achieving high-speed transfer of mass data of data recorder on basis of multithreading | |
| CN107220069A (en) | A kind of Shuffle methods for Nonvolatile memory | |
| US11275689B2 (en) | Data loading method, data loading apparatus, and recording medium | |
| CN104461941A (en) | Memory system structure and management method | |
| Breß et al. | Exploring the design space of a GPU-aware database architecture | |
| WO2023159976A1 (en) | Data segmented writing method, data reading method and apparatus | |
| CN115080206B (en) | High-speed echo data real-time recording system and method based on multithreading mechanism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20170517 |
|
| WD01 | Invention patent application deemed withdrawn after publication |