CN108241617A - A kind of production of hydrocarbons real-time data memory system and method for reading data - Google Patents
A kind of production of hydrocarbons real-time data memory system and method for reading data Download PDFInfo
- Publication number
- CN108241617A CN108241617A CN201611203327.4A CN201611203327A CN108241617A CN 108241617 A CN108241617 A CN 108241617A CN 201611203327 A CN201611203327 A CN 201611203327A CN 108241617 A CN108241617 A CN 108241617A
- Authority
- CN
- China
- Prior art keywords
- write
- unit
- data
- read
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of production of hydrocarbons real-time data memory system and method for reading data, including a kind of production of hydrocarbons real-time data memory system, the system includes real-time data memory unit, the real-time data memory unit includes writing interface and reads interface, the interface of writing connects the achievement data buffer unit being arranged in real-time data memory unit respectively, file system read-write unit, multiple threads that the connection of this document system read-write unit is arranged in real-time data memory unit read buffering, and the reading interface is connect with end achievement data buffer unit.And the method for reading data using above system, it is read the method includes the state of data in monitoring file system read-write unit and by reading interface；Based on behaviour's monitoring, the file read-write condition in file system read-write unit is detected, which is to be read out the demand of data directory in file system read-write unit.
Description
Technical field
The present invention relates to production of hydrocarbons real time datas to read and write technology, especially a kind of production of hydrocarbons real-time data memory system
And method for reading data.
Background technology
The industrialization of the country and information-based depth integration strategic plan are implemented, Petroleum Production enterprise, which gradually increases, in recent years adopts
Oily production scene information promotes building-up work, using technologies such as sensor, Internet of Things, traditional oil recovery production and information is automatic
Change technology depth integration realizes the real-time of the data such as grease well pressure, temperature, flow, electrical parameter in oil recovery production process and adopts
Collection makes it possible that remote monitoring oil-water well production status, realization production scene are unattended, improves the labour of traditional industries
Productivity.Further to develop and use the magnanimity creation data of minute grade, the effect of data resource is played, recovers the oil and produces for optimization
Management measure scheme provides support, needs to extract real-time production data, and with reference to petroleum production engineering theory, establishment computer is soft
Part calculates the variation tendency and delta data that carry out data according to certain model, is analyzed, the abnormal feelings of prediction production
Condition, management measure to be taken to provide guidance in advance.Widespread practice is at present：
（1）Real time data is dumped to from the real-time data base of profession in the relational databases such as Oracle, it is enterprising in Oracle
Row inquiry extraction data are analyzed to application program, and the relational databases such as Oracle as the storage of universal data are limited to
Its versatility, index time unable to do what one wishes to the query analysis of the real time data of magnanimity using Index Algorithms such as B-tree/B+ trees/B- trees
It goes through and rapid downward trend is presented with the rapid growth of data volume.
（2）Real time data is dumped into NoSQL libraries from the real-time data base of profession, inquiry point is carried out on NoSQL libraries
Analysis, the way performance is slightly good and Oracle, but similary because the universal design of NoSQL is to the access performance of magnanimity real time data
It is promoted limited.
（3）Query analysis directly is carried out on the real-time data base of profession, professional real-time database is generally deployed in industry computer
Environment is generally forbidden developing immediately above, and the pressure analyzed and can increased to the database is directly done in the real-time database of profession
Power will influence the stability of industry computer.
The database index algorithm of the prior art needs to carry out traversal index, could the data of needs be loaded into application
In system, for the real time data of magnanimity, extraction efficiency is low, and it is long to expend the time, it is difficult to meet mass data extraction, analysis application
Demand.
Invention content
In view of this, the main object of the present invention is to provide a kind of production of hydrocarbons real-time data memory system and digital independent
Method.
Itself the technical solution adopted is that：
A kind of production of hydrocarbons real-time data memory system, which is characterized in that including
One real-time data memory unit, the real-time data memory unit include writing interface and read interface, and the interface of writing connects respectively
It connects and is arranged on the first achievement data buffer unit being connected with each other in real-time data memory unit, N achievement data buffer units
And end achievement data buffer unit and a thread Write post unit, wherein N are positive number and >=2；
One file system read-write unit, the connection of this document system read-write unit are arranged on multiple lines in real-time data memory unit
Journey reads buffering, and wherein at least one thread reads buffering and is connected to the first achievement data buffer unit, N achievement data buffer units
And any one in the achievement data buffer unit of end, the thread Write post unit are respectively connected to file system read-write
Unit and at least one achievement data buffer unit,
The reading interface is connect with end achievement data buffer unit.
A kind of method for reading data using production of hydrocarbons real-time data memory system, the method includes
It monitors the state of data in file system read-write unit and is read by reading interface；
Based on behaviour's monitoring, the file read-write condition in file system read-write unit is detected, the condition is is read out file
In system read-write unit the demand of data directory and
If such condition is detected, start reproduction process, wherein, the reproduction process includes sending by reading interface
Message is read to the end achievement data buffer unit, the end achievement data buffer unit inquires the write-in of these data
Condition is directly indexed according to the Writing condition to file system read-write unit.
Preferably, the index is included by reading directly to calculate index simultaneously according to the time after interface position to data file
Extract the continuous data of specified quantity.
Preferably, time+time interval * N1, wherein N1 is positive number and >=1 on the basis of the time.
Preferably, the fiducial time for monitoring the Writing condition for including data and matching Writing condition and file rise
Beginning position corresponds to obtain starting index.
Preferably, the fiducial time has system to automatically configure.
Preferably, the reproduction process, which further includes, is read buffering from thread while is referred to the first achievement data buffer unit, N
Scalar data cache unit and end achievement data buffer unit send out response message, if detecting the response message, text
The state of data in it is pushed to thread and reads buffering by part system read-write unit, and the random push of buffering is read by thread.
Preferably, the thread read buffering can to the first achievement data buffer unit, N achievement datas buffer unit with
And any one in the achievement data buffer unit of end individually sends out response message.
Preferably, which is characterized in that said write condition includes
It monitors from the state for writing interface execution write-in data volume, system automatically configures fiducial time and data volume number with 9
Position storage write-in, the interface of writing perform writing commands and send out and write to the first achievement data buffer unit and thread Write post
Enter order, the first achievement data buffer unit receives that data volume is sent to thread Write post, thread Write post after writing commands
Matching write-in indexes, and position+span data * N2, wherein N2 is positive number and >=1 on the basis of the index.
Beneficial effects of the present invention are：For prior art defect, it is contemplated that real time data real-time, continuity, on time
Between equal difference be distributed characteristic, in a manner of fiducial time and reference position, the computation index of data is calculated by offset.
It does not need to store, traverse index, Data Position is directly obtained by calculation, and be aided with caching, memory access speed is fast, more traditional rope
Draw algorithm data read or write speed and promote 10 times or more.
Description of the drawings
Fig. 1 is the frame principle of production of hydrocarbons real-time data memory system in the present invention；
Fig. 2 is the flow chart of method for reading data in the present invention.
Specific embodiment
Below in conjunction with attached drawing and specific embodiment, the present invention will be described in detail, herein illustrative examples of the invention
And explanation is used for explaining the present invention, but not as a limitation of the invention.
Referring to Figures 1 and 2, the invention discloses a kind of production of hydrocarbons real-time data memory system and method for reading data.
Itself the technical solution adopted is that：
A kind of production of hydrocarbons real-time data memory system, which is characterized in that including
One real-time data memory unit 9, the real-time data memory unit 9 include writing interface 1 and read interface 7, and the interface 1 of writing divides
The first achievement data buffer unit 2 being connected with each other in real-time data memory unit 9 Lian Jie be arranged on, N achievement datas delay
Memory cell 3 and end achievement data buffer unit 4 and a thread Write post unit 5, wherein N are positive number and >=2；
One file system read-write unit 6, the connection of this document system read-write unit 6 are arranged on more in real-time data memory unit 9
A thread reads buffering 8, and wherein at least one thread reads buffering 8 and is connected to the first achievement data buffer unit 2, N achievement datas
Any one in buffer unit 3 and end achievement data buffer unit 4, thread Write post Unit 5 is respectively connected to
File system read-write unit 6 and at least one achievement data buffer unit 8,
The reading interface 7 caches Unit 4 with end achievement data and connect.
The production of hydrocarbons real-time data memory system of the present invention will utilize the technologies such as sensor, Internet of Things, and traditional is adopted
Oil production and information automation technology depth integration realize grease well pressure in oil recovery production process, temperature, flow, electricity ginseng
The real-time acquisition of the data such as number, and the real time data of the acquisition is written, when being written, utilize the write-in item of system formation
Part is monitored from the state for writing interface execution write-in data volume, and system automatically configures fiducial time and data volume number with 9
Storage write-in, the interface 1 of writing perform writing commands and are sent out to the first achievement data buffer unit 2 and thread Write post 5
Writing commands, the first achievement data buffer unit 2 receive that data volume is sent to thread Write post 5 after writing commands, and thread is write
5 matching write-in of buffering indexes, and position+span data * N2, wherein N2 is positive number and >=1 on the basis of the index.The index includes
Index is directly calculated according to the time and extract the continuous data of specified quantity after data file is navigated to by reading interface 7.Institute
Time+time interval * N1, wherein N1 is positive number and >=1 on the basis of stating the time.
It states monitoring and includes the Writing condition of data and the fiducial time for matching Writing condition and file initial position correspondence
Obtain starting index.The fiducial time has system to automatically configure.
A kind of method for reading data using production of hydrocarbons real-time data memory system, the method includes
It monitors the state of data in file system read-write unit 6 and is read by reading interface 7；
Based on behaviour's monitoring, the file read-write condition in file system read-write unit 6 is detected, the condition is is read out file
In system read-write unit 6 demand of data directory and
If such condition is detected, start reproduction process, wherein, the reproduction process includes sending out by reading interface 7
It send and reads message to the end achievement data buffer unit 4, the end achievement data buffer unit 4 inquires these data
Writing condition is directly indexed according to the Writing condition to file system read-write unit 6.
The reproduction process further includes from thread and reads buffering 8 simultaneously to the first achievement data buffer unit 2, N index numbers
Response message is sent out according to buffer unit 3 and end achievement data buffer unit 4, if detecting the response message, file
The state of data in it is pushed to thread and reads buffering 8 by system read-write unit, and the push at random of buffering 8 is read by thread.The line
Journey, which reads buffering 8, to be cached to the first achievement data buffer unit 2, N achievement datas buffer unit 3 and end achievement data
Any one in unit 4 individually sends out response message.
The present invention considers real time data real-time, continuity, the characteristic of temporally equal difference distribution, with fiducial time and base
The computation index of data is calculated by offset for the mode that level is put.Technological invention point is when real time data is utilized
Between equal difference be distributed real-time characteristic, data stores with distinctive document form, innovation data access engine, do not need to store,
Traversal index, is directly obtained by calculation Data Position, and be aided with caching, and memory access speed is fast, it realizes rapid extraction, reduces magnetic
Disk is sought, and increases I/O throughput, and more traditional Index Algorithm reading and writing data speed promotes 10 times or more.
The technical solution disclosed in the embodiment of the present invention is described in detail above, specific implementation used herein
Example is expounded the principle and embodiment of the embodiment of the present invention, and the explanation of above example is only applicable to help to understand
The principle of the embodiment of the present invention；Meanwhile for those of ordinary skill in the art, embodiment, is being embodied according to the present invention
There will be changes in mode and application range, in conclusion the content of the present specification should not be construed as the limit to the present invention
System.
Claims (9)
1. a kind of production of hydrocarbons real-time data memory system, which is characterized in that including
One real-time data memory unit, the real-time data memory unit include writing interface and read interface, and the interface of writing connects respectively
It connects and is arranged on the first achievement data buffer unit being connected with each other in real-time data memory unit, N achievement data buffer units
And end achievement data buffer unit and a thread Write post unit, wherein N are positive number and >=2；
One file system read-write unit, the connection of this document system read-write unit are arranged on multiple lines in real-time data memory unit
Journey reads buffering, and wherein at least one thread reads buffering and is connected to the first achievement data buffer unit, N achievement data buffer units
And any one in the achievement data buffer unit of end, the thread Write post unit are respectively connected to file system read-write
Unit and at least one achievement data buffer unit,
The reading interface is connect with end achievement data buffer unit.
2. a kind of method for reading data using production of hydrocarbons real-time data memory system, the method includes
It monitors the state of data in file system read-write unit and is read by reading interface；
Based on behaviour's monitoring, the file read-write condition in file system read-write unit is detected, the condition is is read out file
In system read-write unit the demand of data directory and
If such condition is detected, start reproduction process, wherein, the reproduction process includes sending by reading interface
Message is read to the end achievement data buffer unit, the end achievement data buffer unit inquires the write-in of these data
Condition is directly indexed according to the Writing condition to file system read-write unit.
3. according to the method described in claim 2, it is characterized in that, the index is included by reading interface position to data file
Index is directly calculated according to the time and extract the continuous data of specified quantity afterwards.
4. according to the method described in claim 3, it is characterized in that, time+time interval * N1 on the basis of the time, wherein
N1 is positive number and >=1.
5. according to the method described in claim 2, it is characterized in that, described monitor the Writing condition for including data and matching
The fiducial time of Writing condition and file initial position correspond to obtain starting index.
6. method according to claim 4 or 5, which is characterized in that the fiducial time has system to automatically configure.
7. according to the method described in claim 2, it is characterized in that, the reproduction process further include from thread read buffering simultaneously to
First achievement data buffer unit, N achievement datas buffer unit and end achievement data buffer unit send out response message,
If detecting the response message, the state of the data in it is pushed to thread and reads buffering by file system read-write unit, by
Thread reads the random push of buffering.
8. according to the method described in claim 2, it is characterized in that, thread reading buffering can be cached to the first achievement data
Any one in unit, N achievement datas buffer unit and end achievement data buffer unit individually sends out response message.
9. according to the method described in claim 2~8 wherein any one claim, which is characterized in that said write condition
Including
It monitors from the state for writing interface execution write-in data volume, system automatically configures fiducial time and data volume number with 9
Position storage write-in, the interface of writing perform writing commands and send out and write to the first achievement data buffer unit and thread Write post
Enter order, the first achievement data buffer unit receives that data volume is sent to thread Write post, thread Write post after writing commands
Matching write-in indexes, and position+span data * N2, wherein N2 is positive number and >=1 on the basis of the index.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611203327.4A CN108241617B (en) | 2016-12-23 | 2016-12-23 | Oil and gas production real-time data storage system and data reading method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611203327.4A CN108241617B (en) | 2016-12-23 | 2016-12-23 | Oil and gas production real-time data storage system and data reading method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108241617A true CN108241617A (en) | 2018-07-03 |
| CN108241617B CN108241617B (en) | 2022-05-06 |
Family
ID=62703341
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611203327.4A Active CN108241617B (en) | 2016-12-23 | 2016-12-23 | Oil and gas production real-time data storage system and data reading method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108241617B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1612252A (en) * | 2003-10-31 | 2005-05-04 | 浙江中控技术股份有限公司 | Real-time data on-line compression and decompression method |
| CN102567227A (en) * | 2012-01-13 | 2012-07-11 | 北京邦诺存储科技有限公司 | Double-controller memory system and method for sharing cache equipment |
| CN102902709A (en) * | 2012-08-02 | 2013-01-30 | 何建亿 | Space allocation fixing file memory system and implementation method |
| CN103281362A (en) * | 2013-05-14 | 2013-09-04 | 郭斌 | File transfer protocol |
| US8954663B1 (en) * | 2012-06-25 | 2015-02-10 | Kip Cr P1 Lp | System, method and computer program product for synchronizing data written to tape including writing an index into a data partition so that data can be recovered in case of failure |
| CN104424158A (en) * | 2013-08-19 | 2015-03-18 | 上海芯豪微电子有限公司 | General unit-based high-performance processor system and method |
| CN105912639A (en) * | 2016-04-08 | 2016-08-31 | 浪潮(北京)电子信息产业有限公司 | Automatic test method and apparatus for data write-in file system |
-
2016
- 2016-12-23 CN CN201611203327.4A patent/CN108241617B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1612252A (en) * | 2003-10-31 | 2005-05-04 | 浙江中控技术股份有限公司 | Real-time data on-line compression and decompression method |
| CN102567227A (en) * | 2012-01-13 | 2012-07-11 | 北京邦诺存储科技有限公司 | Double-controller memory system and method for sharing cache equipment |
| US8954663B1 (en) * | 2012-06-25 | 2015-02-10 | Kip Cr P1 Lp | System, method and computer program product for synchronizing data written to tape including writing an index into a data partition so that data can be recovered in case of failure |
| CN102902709A (en) * | 2012-08-02 | 2013-01-30 | 何建亿 | Space allocation fixing file memory system and implementation method |
| CN103281362A (en) * | 2013-05-14 | 2013-09-04 | 郭斌 | File transfer protocol |
| CN104424158A (en) * | 2013-08-19 | 2015-03-18 | 上海芯豪微电子有限公司 | General unit-based high-performance processor system and method |
| CN105912639A (en) * | 2016-04-08 | 2016-08-31 | 浪潮(北京)电子信息产业有限公司 | Automatic test method and apparatus for data write-in file system |
Non-Patent Citations (2)
| Title |
|---|
| YOUMIN CHEN ET AL: "Sprout: A Functional Caching Approach to Minimize Service Latency in Erasure-Coded Storage", 《IEEE》 * |
| 何佳委 等: "多线程技术在数据实时采集分析中的应用", 《世界电子元器件》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108241617B (en) | 2022-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103020204B (en) | A kind of method and its system carrying out multi-dimensional interval query to distributed sequence list | |
| US10496523B2 (en) | Sequential access storage and data de-duplication | |
| CN107016400B (en) | Evaluation of training set | |
| CN105183839A (en) | Hadoop-based storage optimizing method for small file hierachical indexing | |
| CN103793493B (en) | A kind of method and system for handling car-mounted terminal mass data | |
| CN102693164B (en) | Equipment and method for preventing buffer overflow | |
| CN105630810B (en) | A method of mass small documents are uploaded in distributed memory system | |
| CN109299113A (en) | The range query method of hybrid index with storage perception | |
| CN104270605A (en) | Method and device for processing video monitoring data | |
| CN105302668A (en) | Method for optimizing block device data access performance in object-based storage system | |
| CN106528649A (en) | Massive data storage and retrieval system and massive data storage and retrieval methods for new energy vehicles | |
| CN102073727A (en) | Method for describing seismic data | |
| CN103548004B (en) | The method and apparatus of dynamic data attemper is realized in file system | |
| CN104834648A (en) | Log query method and device | |
| CN103593460A (en) | Data hierarchical storage system and data hierarchical storage method | |
| CN103605617A (en) | Method and device for recording changes in stored data | |
| CN106780157B (en) | Ceph-based power grid multi-temporal model storage and management system and method | |
| CN110377598B (en) | Multi-source heterogeneous data storage method based on intelligent manufacturing process | |
| WO2015108922A1 (en) | Loose coupling of metadata and actual data | |
| CN108241617A (en) | A kind of production of hydrocarbons real-time data memory system and method for reading data | |
| CN109189343A (en) | A kind of metadata rule method, apparatus, equipment and computer readable storage medium | |
| CN112650739A (en) | Data storage processing method and device for coal mine data middling station | |
| CN101840436A (en) | Logging data file storage method based on table structure | |
| CN104166524A (en) | Processing method of metadata and data | |
| CN107357915A (en) | A kind of date storage method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |