CN108415854A - Data collecting system based on shared buffer memory and method - Google Patents
Data collecting system based on shared buffer memory and method Download PDFInfo
- Publication number
- CN108415854A CN108415854A CN201810141939.8A CN201810141939A CN108415854A CN 108415854 A CN108415854 A CN 108415854A CN 201810141939 A CN201810141939 A CN 201810141939A CN 108415854 A CN108415854 A CN 108415854A
- Authority
- CN
- China
- Prior art keywords
- data
- shared
- cache area
- attributes
- buffer memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
- G06F12/0813—Multiuser, multiprocessor or multiprocessing cache systems with a network or matrix configuration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
- G06F12/084—Multiuser, multiprocessor or multiprocessing cache systems with a shared cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Abstract
The present invention relates to big data technical fields, disclose a kind of collecting method based on shared buffer memory, which includes：By multi-course concurrency form gathered data, the data are stored in the first shared cache area and package upload；And the data that package uploads are received, the data are stored in the second shared cache area, and database is written in the data in second shared cache area.The present invention can not only carry out various types of data the real-time acquisition and transmission of second grade frequency, it is also ensured that safety and integrality of the data in transmission process mitigate the pressure of acquisition server and network transmission as far as possible.
Description
Technical field
The present invention relates to big data technical fields, more particularly to a kind of data collecting system based on shared buffer memory and side
Method.
Background technology
With the emergence of consumption internet, big data analysis and cloud computing technology are developed, big data industry by
Step moves towards landing from concept, has obtained extensive concern and attention, big data has already started new round business tide.
2017, big data formally entered the planning of national strategy level：Ministry of Industry and Information prints and distributes《Big data industrial development planning
(2016-2020)》, quickening especially set out promotes big data industry application power, to the year two thousand twenty, big data Related product kimonos
Business health service revenue breaks through 1 trillion yuan, and annual average compound growth rate keeps 30% or so.
Country lands around the policy of big data, and by further enterprise to the consciousness of data value, excitation data is handed over
Easy developing market scale；In terms of market, all trades and professions get over the dependence of big data and the property of value of big data industry
Come higher.
Big data is compared with traditional data, and feature is that data volume is big, variation is fast, broad categories, the true and false leave a question open.Big data
It is the analyzing processing to all target datas, is not traditional sampling analysis processing, thus traditional data acquisition treatment method
It has been not suitable for the acquisition demand of big data.For big data, need there is stronger decision edge, to see clearly discovery power using new
Processing magnanimity, high growth rate and diversified information assets are gone with the tupe of process optimization ability.
The acquisition of big data mainly carries out remote data acquisition with Internet technology, by acquisition software in internet
Middle progress wide area, large-scale, acquisition in real time and transmission are ultimately sent to big data and put down by repeatedly converging and forwarding
Data source of the platform as analysis and application.
Target data is mainly collected the process of big data platform by the gatherer process of big data from Various types of data source,
Big data includes mainly two classes：One kind is from the databases such as Oracle, PostgreSQL, management software interface, file, daily record etc.
The word type of structured data of acquisition；Another kind of is from the non-of the acquisitions such as Email, audio and video, picture, webpage, social media
Structural data.
The main problem that the acquisition of big data faces at present is that the data source of big data is abundant and varied.For example,
The acquisition method of structural data and unstructured data is entirely different, and requirement of the different types of data to real-time is also complete
It is different.Specifically, the productivity data demand real time data acquisition of industrial control system, second grade transmission, to acquisition server
Lasting immense pressure is generated with the network link of transmission；And financial category data demand by week, the moon or season taken at regular intervals and passes
It is defeated, the pressure of periodic shock is generated to acquisition server and transmission network, it is necessary to consider at the data in data acquisition
Reason problem.Especially the range of field data is put into after the system scope of wide area network distribution, big data summation is multiple existing
The summation of field data, when collection point, total amount reaches TB (megabit tuple) or even PB (kilomegabit tuple), EB (tril bit group)
When this problem it is more obvious and intractable.
Invention content
The object of the present invention is to provide a kind of data collecting system and method based on shared buffer memory, the data collecting system
Not only can various types of data be carried out with the real-time acquisition and transmission of second grade frequency, it is also ensured that data are in transmission process
Safety and integrality mitigate the pressure of acquisition server and network transmission as far as possible.
To achieve the goals above, the present invention provides a kind of data collecting system and method based on shared buffer memory, the number
Include according to acquisition system：Data collection terminal is stored in for by multi-course concurrency form gathered data, and by the data to the
One shared cache area and package upload；And data receiver, the data for receiving package upload deposit the data
Enter to the second shared cache area, and database is written in the data in second shared cache area.
Optionally, the data collection terminal is additionally operable to：Data filtering carried out to the data of acquisition, only timestamp and
The data that data value changes simultaneously could be stored in first shared cache area.
Optionally, first shared cache area and second shared cache area are only right within the storage timeliness phase of setting
The data are stored.
Optionally, the data structure in first shared cache area and second shared cache area uses Hash table
Key-value modes store the data.
Optionally, the data structure includes Name attributes, Flages attributes, State attributes, Value attributes, Type
Attribute, Message attributes and/or Time attributes.
Optionally, the unstructured data in the data is stored in Message attributes, and structural data, which is stored in, to be removed
In each attribute outside Message attributes.
Correspondingly, the present invention also provides a kind of collecting method based on shared buffer memory, which includes：
By multi-course concurrency form gathered data, the data are stored in the first shared cache area and package upload；And it receives
The data that package uploads, the data are stored in the second shared cache area, and in second shared cache area
Database is written in the data.
Optionally, the described data are stored in the first shared cache area includes：To the data of acquisition into line number
According to filtering, the data that only timestamp and data value change simultaneously could be stored in first shared cache area.
Optionally, first shared cache area and second shared cache area are only right within the storage timeliness phase of setting
The data are stored.
Optionally, the data structure in first shared cache area and second shared cache area uses Hash table
Key-value modes store the data.
Optionally, the data structure includes Name attributes, Flages attributes, State attributes, Value attributes, Type
Attribute, Message attributes and/or Time attributes.
Optionally, the unstructured data in the data structure is stored in Message attributes, and structural data is deposited
It is put into each attribute in addition to Message attributes.
Through the above technical solutions, the present invention is taught skill by using multi-course concurrency form and multi-stage data shared buffer memory
Art, carries out various types of data unified acquisition and transmission, can not only realize the real-time data acquisition of second grade frequency, may be used also
Ensure safety and integrality of the data in transmission process, mitigates the pressure of acquisition server and network transmission as far as possible.
Description of the drawings
Fig. 1 is the flow chart for the data collecting system based on shared buffer memory that one embodiment of the present invention provides；
Fig. 2 is the structure chart of the first shared cache area of the data collection terminal that one embodiment of the present invention provides；
Fig. 3 is the storage organization figure for the first shared cache area that one embodiment of the present invention provides；
Fig. 4 is the structure chart of the second shared cache area of the data receiver that one embodiment of the present invention provides；And
Fig. 5 is the flow chart for the collecting method based on shared buffer memory that one embodiment of the present invention provides.
Reference sign
1 data collection terminal, 2 data receiver
3 database, 4 data receiver program
10 data source, 11 data source
12 data source, 30 Redis data
100 first shared cache area, 200 second shared cache area
1000 capture program, 1001 capture program
1002 capture programs
Specific implementation mode
The specific implementation mode of the present invention is described in detail below in conjunction with attached drawing.It should be understood that this place is retouched
The specific implementation mode stated is merely to illustrate and explain the present invention, and is not intended to restrict the invention.
Fig. 1 is the flow chart for the data collecting system based on shared buffer memory that one embodiment of the invention provides.Such as Fig. 1 institutes
Show, data collecting system provided by the invention may include：Data collection terminal 1, for passing through multi-course concurrency form gathered data,
And the data are stored in the first shared cache area 100 and package upload；And data receiver 2, for receiving package
The data passed, the data are stored in the second shared cache area 200, and described in second shared cache area
Database 3 is written in data.The data collecting system teaches skill art by using multi-course concurrency form and multi-stage data shared buffer memory,
Various types of data are carried out with unified acquisition and transmission, the real-time data acquisition of second grade frequency can be not only realized, can also protect
Safety and integrality of the data in transmission process are demonstrate,proved, mitigates the pressure of acquisition server and network transmission as far as possible.
In addition, the data collection terminal 1 is additionally operable to：Data filtering, only timestamp (generation are carried out to the data of acquisition
The time that the table data are generated) and data value at the same variation data could be stored in first shared cache area 100, it is other
Data can give up.The data acquired are restored again into first shared cache area 100 by data filtering, it is ensured that
Each data are different, make in first shared cache area 100 data close-packed arrays, will not cause space waste.
First shared cache area 100 and second shared cache area 200 are only right within the storage timeliness phase of setting
The data are stored, once more than the storage timeliness phase, the data will be replaced by new data, to realize data more
Newly, the memory space of shared cache area is rationally and effectively utilized.
As shown in Figure 2 and Figure 4, the data knot in first shared cache area 100 and second shared cache area 200
Structure body may include Name attributes, Flages attributes, State attributes, Value attributes, Type attributes, Message attributes and/or
Time attributes.The letters such as the unstructured data in the data, such as Email, audio and video, picture, webpage, social media
Breath, can be stored in Message attributes；And the letters such as structural data, such as database, management software interface, file, daily record
Breath, can be stored in each attribute in addition to Message attributes.The present invention can be according to described in real data type and size setting
The number of attribute and the corresponding byte number of each attribute in structure, such as structural data, it can in data structure
To be not provided with Message attributes, and be selectively arranged Name attributes, Flages attributes, State attributes, Value attributes,
Type attributes and/or Time attributes etc..
As shown in figure 3, the data structure S in first shared cache area 100 and second shared cache area 200
The key-value modes that Hash table can be used store the data.1~N of number represents measuring point the roll-call Name, Name 1 of storage
~N distinguishes corresponding data source 1-N, and arrow is directed to the pointer of next S values.Name in S is as key, and S is inherently
Value.The data acquired pass through data filtering, that is to say, that only when in S Value (or Message) and Time same
In the case of Shi Bianhua, which could enter shared cache area.For example, behind the region that S11 is stored in that Name is 1, storing
Before S12, to judge Value (or Message) in S12 and Time whether in S11 Value (or Message) and
Time is different, if the Value (or Message) and Time of the two are different, stores S12 according to the newer sequences of Time
At next place of S11；If any one of the two variation or both does not change, S12 is given up, it is not stored.
It can ensure that each S is different in this way, make the data close-packed arrays in shared cache area, space waste will not be caused.In addition, being
The pressure for alleviating the second grade variation of industrial big data, discharges memory space in time.S depositing in setting in shared cache area
It is stored in the storage timeliness phase, the data more than the storage timeliness phase can be updated.For example, the storage timeliness phase set as
In the case of one week, when monitoring the Time in S23 more than one week, S23 will be substituted by new data to be updated,
Then the Time in the S24 at next place is monitored, once it is more than one week, replaces update by new data immediately.
The database 3 may include Redis databases 30, which is that one kind also can be lasting based on memory
Log type, the Key-Value databases of change, support includes a variety of value such as character type, chain phenotype, assembly type and ordered set
Therefore the storage of type is suitable for that storage data quantity is big, requirement of real-time is high and the frequent data of update.It is empty to reduce storage
Between, the Value in the Redis databases 30 stores binary number using the Protocol Buffer formats of Google companies
According to.Protocol Buffer are a kind of light weight, efficient structural data storage format, compared with more traditional character format,
Its is small, occupied space is few and data exchange is efficient.The binary data stored in the Redis databases 30 adds
Close property is good so that it is more suitable for the data for being stored with higher level security demand.
The data collection terminal 1 also uses breakpoint transmission technical transmission data with the data receiver 2, wherein described
Breakpoint transmission technology includes：When the communication failure of data collection terminal 1 and data receiver 2, the data that do not transmit are marked；And
When the communication restores normal, institute is uploaded in the case where not influencing normal data upload, then by the data that do not transmit
Data receiver 2 is stated, and/or when the communication failure of the data receiver 2 and the database 3, marks the number not transmitted
According to；And when the communication restores normal, write in the case where not influencing normal data upload, then by the data that do not transmit
Enter the database 3.Specifically, during the data collection terminal 1 and the data receiver 2 communicate, if described
Data receiver 2 has successfully received a message, then needs the data that a byte is fed back to the data collection terminal 1；If not receiving
Message (such as communication failure), then will not feed back the data of a byte.When the data collection terminal 1 and the data receiver
When 2 communication failure, the data receiver 2 does not receive a certain message, then it will not feed back the data of a byte；The number
According to collection terminal 1 in the case where not receiving the data that the data receiver 2 is fed back, the message (number not transmitted is marked
According to), at this point, the data that do not transmit are stored in second shared cache area 200；When communication restores normal, not
It influences in the case that normal data uploads, the data collection terminal 1 will not pass described in second shared cache area 200 again
Defeated data upload to the data receiver 2.The number is written in the data by 2 breakpoint transmission technology of the data receiver
Process according to library 3 is also in this way, being repeated no more in this.The breakpoint transmission technology solves the problems, such as loss of data when network generation
The problem of, it ensure that safety and integrality of the data in transmission process (especially in the transmission of network link length).
Specifically, now explaining this by taking first shared cache area 100 and second shared cache area 200 as an example
The data acquisition based on shared buffer memory provided is provided.
For the data collection terminal 1, as shown in Fig. 2, the data collection terminal 1 can start multiple capture programs (including
Capture program 1000, capture program 1001, capture program 1002 ... etc.) acquisition is from multiple data sources (including data source
10, data source 11, data source 12 ...) data, the data source may include industrial control system and intelligence sensor.It will
The data are grouped according to data measuring point or particular packet rule is drawn, and it (is usually uniform point to give different capture program processing
With).Each capture program is responsible for independent parsing interface protocol, then gathered data carries out data filtering to the data acquired,
Only have the data that timestamp and data value change simultaneously that could uniformly be converted to proprietary data format through protocol conversion, then private
There are the data of data format to be stored in first shared cache area 100.Wherein unstructured data, such as Email, sound regard
Frequently, the information such as picture, webpage, social media can be stored in Message attributes；And structural data, such as database, pipe
The information such as software interface, file, daily record are managed, can be stored in each attribute in addition to Message attributes.After data filtering,
It can ensure that each data are different, make the data close-packed arrays of first shared cache area 100, space will not be caused unrestrained
Take.It is acquired if it is double nets, two-way data comparison need to be increased, it is shared slow that a deposit described first is chosen from identical data
Deposit area 100.First shared cache area 100 is only stored within the storage timeliness phase of setting, is more than the storage timeliness
The data of phase can be updated.Finally, the data in first shared cache area 100 are carried out that package is concentrated to upload to the number
According to receiving terminal 2.The present embodiment uses multi-process form gathered data, to alleviate the pressure of data acquisition interface, realizes second grade
The real-time data acquisition of frequency；And data acquisition and data upload process are isolated by first shared cache area 100, it can
Easily and flexibly to carry out the physics deployment of data collecting system, also certifiable data can be real-time, safe and be completely transmitted.
When the communication failure of the data collection terminal 1 and the data receiver 2, the data collection terminal 1 is not being received
In the case of the feedback data for being successfully connected to a certain message (data that do not transmit) to the data receiver 2, to described
The message in one shared cache area 100 is marked；When communication restores normal, the case where normal data uploads is not being influenced
Under, the message is uploaded to the data receiver 2 by the data collection terminal 1 again.The present invention is at the data collection terminal 1
The breakpoint transmission technology of use solves data loss problem when network failure, from the data collection terminal 1 to the data
Receiving terminal 2 carries out the one-way transmission of data, does not influence the normal operation of original industrial control system or intelligence sensor, ensure that
Safety and integrality of the data in transmission process (especially in the transmission of network link length).
For the data receiver 2, as shown in figure 4, the data receiver program 4 in the data receiver 2 passes through net
Network link receives the data packet that multiple data collection terminals 1 upload, and data packet is write direct second shared cache area
200.The storage mode and related advantages of second shared cache area 200 and phase the case where the first shared cache area 100
Seemingly, it is repeated no more in this.Finally, the Redis databases 30 are written in the data in second shared cache area.Institute
The data stored in Redis databases 30 are stated with small, occupied space is few, data exchange is efficient and level security is good
The advantages of.
When the communication failure of the data receiver 2 and the Redis databases 30, the data receiver 2 is not
In the case of receiving the feedback data that the Redis databases 30 are successfully connected to a certain message (data that do not transmit), to institute
The message stated in the second shared cache area 200 is marked；When communication restores normal, normal data upload is not being influenced
In the case of, the message is uploaded to the Redis databases 30 by the data receiver 2 again.The present invention connects in the data
The breakpoint transmission technology used at receiving end 2 solves data loss problem when network failure, from the data receiver 2 to institute
The one-way transmission that Redis databases 30 carry out data is stated, does not influence the normal fortune of original industrial control system or intelligence sensor
Row ensure that safety and integrality of the data in transmission process (especially in the transmission of network link length).
First shared cache area 100 and second shared cache area 200 provided by the invention acquired data
Data acquisition in journey is uploaded with package, data receiver and write-in database detach, and can easily and flexibly carry out data acquisition
The physics of system is disposed.The data acquisition of the data collection terminal 1 and package upload independently of each other, are acquired when data volume is huge
Process, the first shared cache area, package upload can be deployed to three independent physical machines；The data of the data receiver 2
It receives and write-in database is mutual indepedent, program is received when data volume is huge can be deployed as multiple independent physical machines or void
Quasi- machine, the second shared cache area and write-in database can also individually dispose physical machine.
The data collection terminal 1 in the present invention and the data receiver 2, can be according to real network situation and safety
Situation carries out multi-stage cascade, and Wide Area Network especially very long in transmission link can increase the data in the data receiver 2
The level of receiver, to mitigate the influence to the data source.
In conclusion the present invention teaches skill art by using multi-course concurrency form and multi-stage data shared buffer memory, to various
The data of type carry out unified acquisition and transmission, can not only realize the real-time data acquisition of second grade frequency, it is also ensured that data
Safety in transmission process and integrality mitigate the pressure of acquisition server and network transmission as far as possible.
Correspondingly, the present invention also provides a kind of collecting method based on shared buffer memory, which includes：
By multi-course concurrency form gathered data, the data are stored in the first shared cache area and package upload；And it receives
The data that package uploads, the data are stored in the second shared cache area, and in second shared cache area
Database is written in the data.
Optionally, the described data are stored in the first shared cache area includes：To the data of acquisition into line number
According to filtering, the data that only timestamp and data value change simultaneously could be stored in first shared cache area.
Optionally, first shared cache area and second shared cache area are only right within the storage timeliness phase of setting
The data are stored.
Optionally, the data structure in first shared cache area and second shared cache area uses Hash table
Key-value modes store the data.
Optionally, the data structure includes Name attributes, Flages attributes, State attributes, Value attributes, Type
Attribute, Message attributes and/or Time attributes.
Optionally, the unstructured data in the data structure is stored in Message attributes, and structural data is deposited
It is put into each attribute in addition to Message attributes.
Detail and benefit in relation to the collecting method provided by the invention based on shared buffer memory see above-mentioned
For the description of the data collecting system based on shared buffer memory, repeated no more in this.
The preferred embodiment of the present invention is described in detail above in association with attached drawing, still, the present invention is not limited to above-mentioned realities
The detail in mode is applied, within the scope of the technical concept of the present invention, a variety of letters can be carried out to technical scheme of the present invention
Monotropic type, these simple variants all belong to the scope of protection of the present invention.
It is further to note that specific technical features described in the above specific embodiments, in not lance
In the case of shield, it can be combined by any suitable means.In order to avoid unnecessary repetition, the present invention to it is various can
The combination of energy no longer separately illustrates.
In addition, various embodiments of the present invention can be combined randomly, as long as it is without prejudice to originally
The thought of invention, it should also be regarded as the disclosure of the present invention.
Claims (12)
1. a kind of data collecting system based on shared buffer memory, which is characterized in that the collecting method includes：
Data collection terminal for passing through multi-course concurrency form gathered data, and the data is stored in the first shared buffer memory
Area and package upload；And
Data receiver, the data for receiving package upload, the data are stored in the second shared cache area, and handle
Database is written in the data in second shared cache area.
2. the data collecting system according to claim 1 based on shared buffer memory, which is characterized in that the data collection terminal
It is additionally operable to：Data filtering is carried out to the data of acquisition, the data that only timestamp and data value change simultaneously could be stored in
First shared cache area.
3. the data collecting system according to claim 1 based on shared buffer memory, which is characterized in that described first is shared slow
It deposits area and second shared cache area only stores the data within the storage timeliness phase of setting.
4. the data collecting system according to claim 1 based on shared buffer memory, which is characterized in that described first is shared slow
The data structure deposited in area and second shared cache area stores the data using the key-value modes of Hash table.
5. the data collecting system according to claim 4 based on shared buffer memory, which is characterized in that the data structure
Belong to including Name attributes, Flages attributes, State attributes, Value attributes, Type attributes, Message attributes and/or Time
Property.
6. the data collecting system according to claim 5 based on shared buffer memory, which is characterized in that non-in the data
Structural data is stored in Message attributes, and structural data is stored in each attribute in addition to Message attributes.
7. a kind of collecting method based on shared buffer memory, which is characterized in that the collecting method includes：
It is stored in the first shared cache area and package upload by multi-course concurrency form gathered data, and by the data；With
And
The data that package uploads are received, the data are stored in the second shared cache area, and is shared slow described second
Deposit the data write-in database in area.
8. the collecting method according to claim 7 based on shared buffer memory, which is characterized in that described by the data
Deposit to the first shared cache area includes：
Data filtering is carried out to the data of acquisition, the data that only timestamp and data value change simultaneously could be stored in described
First shared cache area.
9. the collecting method according to claim 7 based on shared buffer memory, which is characterized in that described first is shared slow
It deposits area and second shared cache area only stores the data within the storage timeliness phase of setting.
10. the collecting method according to claim 7 based on shared buffer memory, which is characterized in that described first is shared
Data structure in buffer area and second shared cache area stores the number using the key-value modes of Hash table
According to.
11. the collecting method according to claim 10 based on shared buffer memory, which is characterized in that the data structure
Body includes that Name attributes, Flages attributes, State attributes, Value attributes, Type attributes, Message attributes and/or Time belong to
Property.
12. the collecting method according to claim 11 based on shared buffer memory, which is characterized in that in the data
Unstructured data is stored in Message attributes, and structural data is stored in each attribute in addition to Message attributes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810141939.8A CN108415854A (en) | 2018-02-11 | 2018-02-11 | Data collecting system based on shared buffer memory and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810141939.8A CN108415854A (en) | 2018-02-11 | 2018-02-11 | Data collecting system based on shared buffer memory and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108415854A true CN108415854A (en) | 2018-08-17 |
Family
ID=63128527
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810141939.8A Pending CN108415854A (en) | 2018-02-11 | 2018-02-11 | Data collecting system based on shared buffer memory and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108415854A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113268043A (en) * | 2021-06-28 | 2021-08-17 | 华能鹤岗发电有限公司 | Method and device for realizing information exchange between PLC (programmable logic controller) and DCS (distributed control system) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103440290A (en) * | 2013-08-16 | 2013-12-11 | 曙光信息产业股份有限公司 | Big data loading system and method |
| CN103559078A (en) * | 2013-11-08 | 2014-02-05 | 华为技术有限公司 | GPU (Graphics Processing Unit) virtualization realization method as well as vertex data caching method and related device |
| CN104462225A (en) * | 2014-11-12 | 2015-03-25 | 华为技术有限公司 | Data reading method, device and system |
| CN104461932A (en) * | 2014-11-04 | 2015-03-25 | 浪潮电子信息产业股份有限公司 | Directory cache management method for big data application |
| CN104750720A (en) * | 2013-12-30 | 2015-07-01 | 中国银联股份有限公司 | Method for achieving high-performance data processing under multithread concurrent access environment |
| CN106293490A (en) * | 2015-05-12 | 2017-01-04 | 中兴通讯股份有限公司 | Data storage, the method read, Apparatus and system |
| CN106777110A (en) * | 2016-12-15 | 2017-05-31 | 武汉邮电科学研究院 | A kind of smart city big data integration system and method |
| CN107301048A (en) * | 2017-06-23 | 2017-10-27 | 北京中泰合信管理顾问有限公司 | Using the internal control and management system of response type sharing application framework |
| US20170351620A1 (en) * | 2016-06-07 | 2017-12-07 | Qubole Inc | Caching Framework for Big-Data Engines in the Cloud |
| CN107479514A (en) * | 2017-07-10 | 2017-12-15 | 昆明理工大学 | A kind of industrial big data process control data collection and Transmission system and implementation method |
| CN107483358A (en) * | 2017-08-24 | 2017-12-15 | 深圳市盛路物联通讯技术有限公司 | A kind of data transmission method and device based on Internet of Things |
| CN107528852A (en) * | 2017-09-05 | 2017-12-29 | 郑州升达经贸管理学院 | A kind of big data based on network security implements system and method |
-
2018
- 2018-02-11 CN CN201810141939.8A patent/CN108415854A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103440290A (en) * | 2013-08-16 | 2013-12-11 | 曙光信息产业股份有限公司 | Big data loading system and method |
| CN103559078A (en) * | 2013-11-08 | 2014-02-05 | 华为技术有限公司 | GPU (Graphics Processing Unit) virtualization realization method as well as vertex data caching method and related device |
| CN104750720A (en) * | 2013-12-30 | 2015-07-01 | 中国银联股份有限公司 | Method for achieving high-performance data processing under multithread concurrent access environment |
| CN104461932A (en) * | 2014-11-04 | 2015-03-25 | 浪潮电子信息产业股份有限公司 | Directory cache management method for big data application |
| CN104462225A (en) * | 2014-11-12 | 2015-03-25 | 华为技术有限公司 | Data reading method, device and system |
| CN106293490A (en) * | 2015-05-12 | 2017-01-04 | 中兴通讯股份有限公司 | Data storage, the method read, Apparatus and system |
| US20170351620A1 (en) * | 2016-06-07 | 2017-12-07 | Qubole Inc | Caching Framework for Big-Data Engines in the Cloud |
| CN106777110A (en) * | 2016-12-15 | 2017-05-31 | 武汉邮电科学研究院 | A kind of smart city big data integration system and method |
| CN107301048A (en) * | 2017-06-23 | 2017-10-27 | 北京中泰合信管理顾问有限公司 | Using the internal control and management system of response type sharing application framework |
| CN107479514A (en) * | 2017-07-10 | 2017-12-15 | 昆明理工大学 | A kind of industrial big data process control data collection and Transmission system and implementation method |
| CN107483358A (en) * | 2017-08-24 | 2017-12-15 | 深圳市盛路物联通讯技术有限公司 | A kind of data transmission method and device based on Internet of Things |
| CN107528852A (en) * | 2017-09-05 | 2017-12-29 | 郑州升达经贸管理学院 | A kind of big data based on network security implements system and method |
Non-Patent Citations (2)
| Title |
|---|
| 咕噜大大: "Flume+Kafka+Storm+Redis实时分析系统基本架构", 《HTTPS://BLOG.CSDN.NET/YMH198816/ARTICLE/DETAILS/51998085》 * |
| 郭佳: "面向多类型终端的视频服务平台的设计与实现", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113268043A (en) * | 2021-06-28 | 2021-08-17 | 华能鹤岗发电有限公司 | Method and device for realizing information exchange between PLC (programmable logic controller) and DCS (distributed control system) |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11474673B1 (en) | Handling modifications in programming of an iterative message processing system | |
| US11113353B1 (en) | Visual programming for iterative message processing system | |
| US11003714B1 (en) | Search node and bucket identification using a search node catalog and a data store catalog | |
| US11663176B2 (en) | Data field extraction model training for a data intake and query system | |
| US10776355B1 (en) | Managing, storing, and caching query results and partial query results for combination with additional query results | |
| US11250056B1 (en) | Updating a location marker of an ingestion buffer based on storing buckets in a shared storage system | |
| US11704490B2 (en) | Log sourcetype inference model training for a data intake and query system | |
| US20220036177A1 (en) | Data field extraction by a data intake and query system | |
| CN103930887B (en) | The inquiry stored using raw column data collects generation | |
| CN106487596A (en) | Distributed Services follow the tracks of implementation method | |
| US11567993B1 (en) | Copying buckets from a remote shared storage system to memory associated with a search node for query execution | |
| CN106878092A (en) | A kind of network O＆M monitor in real time of multi-source heterogeneous data fusion is presented platform with analysis | |
| CN105608188A (en) | Data processing method and data processing device | |
| CN109918349A (en) | Log processing method, device, storage medium and electronic device | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| US11620336B1 (en) | Managing and storing buckets to a remote shared storage system based on a collective bucket size | |
| CN107241305A (en) | A kind of network protocol analysis system and its analysis method based on polycaryon processor | |
| US11714823B1 (en) | Generating metric data from log data using metricization rules | |
| CN107092627A (en) | The column-shaped storage of record is represented | |
| CN106817253A (en) | The monitor in real time of journal file and the method and system of alarm | |
| CN106228263A (en) | Materials stream informationization methods based on big data | |
| CN107302569A (en) | A kind of security monitoring Data acquisition and storage method of facing cloud platform | |
| CN109344137A (en) | A kind of log storing method and system | |
| CN108415854A (en) | Data collecting system based on shared buffer memory and method | |
| CN112306407B (en) | Monitoring data storage method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |