CN110334062A - A kind of big data transmission method across public security boundary based on database physical log - Google Patents
A kind of big data transmission method across public security boundary based on database physical log Download PDFInfo
- Publication number
- CN110334062A CN110334062A CN201910623323.9A CN201910623323A CN110334062A CN 110334062 A CN110334062 A CN 110334062A CN 201910623323 A CN201910623323 A CN 201910623323A CN 110334062 A CN110334062 A CN 110334062A
- Authority
- CN
- China
- Prior art keywords
- oplog
- time
- transmission method
- data transmission
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 230000005540 biological transmission Effects 0.000 title claims abstract description 21
- 238000012790 confirmation Methods 0.000 claims abstract description 4
- 230000006837 decompression Effects 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 3
- 238000004321 preservation Methods 0.000 abstract description 2
- 230000008676 import Effects 0.000 abstract 1
- 238000004883 computer application Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of big data transmission method across public security boundary based on database physical log includes the following steps: that oplog imports time confirmation, oplog compressing file, oplog file scan and upload and oplog file download and preservation；The beneficial effects of the present invention are, firstly, big data transmission method of the invention, by parsing MongoDB physics scene by hand, according to time offset, the newly-increased physics scene of every several minutes of readings MongoDB cluster；And the case where may being lost for data, realize the multiple delivery of same part data；Secondly, big data transmission method of the invention when so that physics scene reaching amount, can be derived automatically from specified buffer strip, all data all efficiently imported into public security Intranet from buffer strip by one buffer strip of design.
Description
Technical field
The present invention relates to computer application technology more particularly to a kind of across public security sides based on database physical log
The big data transmission method on boundary.
Background technique
There are two types of current main-stream schemes, including full dose data output method and incremental data back up output method one by one.Entirely
Data output method is measured, refers to and is transferred to public security Intranet after exporting all files of database again, but this method
Real-time is poor, and needs to expend longer time in face of the biggish data of space hold and transmitted.Incremental data is standby
Part output method one by one refers to according to the logic log of variation, being then transmitted to one by one public security Intranet, this method timeliness
It is relatively good, but control logic is more complicated, needs to send increment logic data to public affairs in the position of each data movement
Pacify Intranet, and under cluster environment, the continuity of logical data may be affected, to influence the preservation of global data
And transmission.
Summary of the invention
1. technical problems to be solved
The technical problem to be solved in the invention is, provide it is a kind of based on database physical log across public security boundary
Big data transmission method, by the data lossless of public network, efficiently, be transmitted to order-preserving public security Intranet.
2. technical solution
To achieve the goals above, the technical scheme is that a kind of across public security side based on database physical log
The big data transmission method on boundary, includes the following steps:
Step 1, public network timer timing acquisition MongoDB last time exports the time of oplog, and determines the export time
The time interval between timer current time；
Step 2, oplog file is exported according to the time interval that step 1 determines, oplog file, pressure is named with current time
It shortens tgz format into and saves to public network local disk；
Step 3, the public network local disk of the public network timer timing scan step 2 of step 1, there are tgz format texts for confirmation
After part, it is uploaded to FTP；
Step 4, the tgz formatted file in the FTP of Intranet timer timing scan step 3, then by several tgz formatted files
It sorts and is downloaded to Intranet local disk；
Step 5, the tgz formatted file of depressurization steps 4 is oplog file and saves to Intranet local disk；
Step 6, MongoDB is written into the filename of the oplog file after step 5 decompression, completes data and transmits work.
Preferably, when the public network timer interval time of the step 1 is less than the public network timer interval of the step 3
Between.
Preferably, the every four minutes of the public network timer of the step 1 obtain last time oplog and export the time.
Preferably, public network local disk described in the every four points of half run-downs of the public network timer of the step 3.
Preferably, whether there is file in the every three minutes run-down FTP of the Intranet timer of the step 4.
Preferably, in the step 2, time interval is exported between time and timer current time less than five minutes,
Stop export oplog formatted file.
3. beneficial effect
In conclusion the beneficial effects of the present invention are:
(1) big data transmission method of the invention, by parsing MongoDB physics scene by hand, according to time offset,
The newly-increased physics scene of every several minutes of readings MongoDB cluster；And the case where may being lost for data, realize same number
According to multiple delivery；
(2) big data transmission method of the invention passes through one buffer strip of design, when so that physics scene reaching amount, meeting
It is derived automatically from specified buffer strip, all data all efficiently imported into public security Intranet from buffer strip.
Detailed description of the invention
Fig. 1 is a kind of process of the big data transmission method across public security boundary based on database physical log of the present invention
Figure；
Fig. 2 is that a kind of big data transmission method public network across public security boundary based on database physical log of the present invention uploads
Data to FTP flow chart；
Fig. 3 is a kind of big data transmission method Intranet downloading across public security boundary based on database physical log of the present invention
Data to FTP flow chart.
Specific embodiment
Technical solution of the present invention is further illustrated in conjunction with specific embodiments.
It refers to shown in attached drawing 1 to attached drawing 3, a kind of big data transmission across public security boundary based on database physical log
Method includes the following steps:
Step 1, public network timer timing acquisition MongoDB last time exports the time of oplog, and determines the export time
The time interval between timer current time；
Step 2, oplog file is exported according to the time interval that step 1 determines, oplog file, pressure is named with current time
It shortens tgz format into and saves to public network local disk；
Step 3, the public network local disk of the public network timer timing scan step 2 of step 1, there are tgz format texts for confirmation
After part, it is uploaded to FTP；
Step 4, the tgz formatted file in the FTP of Intranet timer timing scan step 3, then by several tgz formatted files
It sorts and is downloaded to Intranet local disk；
Step 5, the tgz formatted file of depressurization steps 4 is oplog file and saves to Intranet local disk；
Step 6, MongoDB is written into the filename of the oplog file after step 5 decompression, completes data and transmits work.
Preferably, when the public network timer interval time of the step 1 is less than the public network timer interval of the step 3
Between.
Preferably, the every four minutes of the public network timer of the step 1 obtain last time oplog and export the time.
Preferably, public network local disk described in the every four points of half run-downs of the public network timer of the step 3.
Preferably, whether there is file in the every three minutes run-down FTP of the Intranet timer of the step 4.
Preferably, in the step 2, time interval is exported between time and timer current time less than five minutes,
Stop export oplog formatted file.
The above description is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all utilizations
Equivalent structure transformation made by present specification, directly or indirectly with the technology neck for being attached to other Related products
Domain is included within the scope of the present invention.
Claims (6)
1. a kind of big data transmission method across public security boundary based on database physical log, which is characterized in that including as follows
Step:
Step 1, public network timer timing acquisition MongoDB exports the time of oplog for the last time, and determines the export time and determine
When device current time between time interval；
Step 2, oplog file is exported according to the time interval that step 1 determines, oplog file is named with current time, is compressed into
Tgz format is simultaneously saved to public network local disk；
Step 3, the public network local disk of the public network timer timing scan step 2 of step 1, confirmation there are after tgz formatted file,
It is uploaded to FTP；
Step 4, the tgz formatted file in the FTP of Intranet timer timing scan step 3, then several tgz formatted files are sorted
And it is downloaded to Intranet local disk；
Step 5, the tgz formatted file of depressurization steps 4 is oplog file and saves to Intranet local disk；
Step 6, MongoDB is written into the filename of the oplog file after step 5 decompression, completes data and transmits work.
2. a kind of big data transmission method across public security boundary based on database physical log according to claim 1,
It is characterized by: the public network timer interval time of the step 1 is less than the public network timer interval time of the step 3.
3. a kind of big data transmission method across public security boundary based on database physical log according to claim 2,
It is characterized by: the every four minutes of the public network timer of the step 1, which obtain last time oplog, exports the time.
4. a kind of big data transmission method across public security boundary based on database physical log according to claim 3,
It is characterized by: public network local disk described in the every four points of half run-downs of the public network timer of the step 3.
5. a kind of big data transmission method across public security boundary based on database physical log according to claim 1,
It is characterized by: whether having file in the every three minutes run-down FTP of the Intranet timer of the step 4.
6. a kind of big data transmission method across public security boundary based on database physical log according to claim 4,
It is characterized by: exporting time interval between time and timer current time in the step 2 and stopping leading less than five minutes
Oplog formatted file out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910623323.9A CN110334062A (en) | 2019-07-11 | 2019-07-11 | A kind of big data transmission method across public security boundary based on database physical log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910623323.9A CN110334062A (en) | 2019-07-11 | 2019-07-11 | A kind of big data transmission method across public security boundary based on database physical log |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110334062A true CN110334062A (en) | 2019-10-15 |
Family
ID=68146070
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910623323.9A Pending CN110334062A (en) | 2019-07-11 | 2019-07-11 | A kind of big data transmission method across public security boundary based on database physical log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110334062A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110995723A (en) * | 2019-12-11 | 2020-04-10 | 深圳物讯科技有限公司 | Method for preventing loss of big data transmission data across public security boundary |
| CN112256797A (en) * | 2020-10-30 | 2021-01-22 | 深圳物讯科技有限公司 | MongoDB-based data synchronization system and method for public security network and government affair network |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107590054A (en) * | 2017-09-21 | 2018-01-16 | 大连君方科技有限公司 | Ship server log monitoring system |
| CN109286615A (en) * | 2018-09-03 | 2019-01-29 | 中新网络信息安全股份有限公司 | A kind of efficiently across LAN data synchronous method |
-
2019
- 2019-07-11 CN CN201910623323.9A patent/CN110334062A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107590054A (en) * | 2017-09-21 | 2018-01-16 | 大连君方科技有限公司 | Ship server log monitoring system |
| CN109286615A (en) * | 2018-09-03 | 2019-01-29 | 中新网络信息安全股份有限公司 | A kind of efficiently across LAN data synchronous method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110995723A (en) * | 2019-12-11 | 2020-04-10 | 深圳物讯科技有限公司 | Method for preventing loss of big data transmission data across public security boundary |
| CN112256797A (en) * | 2020-10-30 | 2021-01-22 | 深圳物讯科技有限公司 | MongoDB-based data synchronization system and method for public security network and government affair network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110334062A (en) | A kind of big data transmission method across public security boundary based on database physical log | |
| WO2015106605A1 (en) | Method for testing station level of intelligent substation | |
| WO2017067225A1 (en) | System, method and storage device for cim/e model standard compliance test | |
| CA2315278A1 (en) | Method, apparatus and program for the central storage of standardized image data | |
| CN111880430A (en) | Control method and device for intelligent household equipment | |
| US20230244203A1 (en) | Summarization retrieval in a process control environment | |
| CN103457609A (en) | Lossless compressing method and lossless uncompressing method of fault wave record data | |
| CN107819611B (en) | Client test method based on IEC61850 multi-server simulation | |
| CN104682962A (en) | Compression method for massive fuel gas data | |
| CN101902368A (en) | WEB performance test method based on simulation of bulk thin client operation and test system thereof | |
| CN105117654A (en) | Data synchronization method between same systems under the circumstance of network physical isolation | |
| CN106534272A (en) | System and method for processing parameters of coal-fired unit | |
| Krzewicki et al. | ALICE HLT Run 2 performance overview. | |
| CN103457996B (en) | A kind of Protocol Conformance Test System internal data store method and system | |
| CN113448590A (en) | Method and device for quickly generating remote control injection data, computer equipment and medium | |
| CN111770145A (en) | One-way network bilateral data synchronization system and method based on log analysis | |
| CN109377412A (en) | A kind of intensive artwork maintenance system of auxiliary tone and its maintaining method | |
| CN104915362A (en) | Fast intelligent ERP system data migration scheme based on time slices, increment analysis and random disorganizing technology | |
| CN110474895B (en) | Full type self-adaptation debugging support pipe system of distribution automation | |
| CN102842065A (en) | Index modeling-based index data filling method | |
| Graf et al. | CAD2VR or how to efficiently integrate VR into the product development process | |
| CN109039606A (en) | Same frequency sampling method and circuit in continuous variable quantum key distribution system | |
| CN204362201U (en) | Decoding storehouse self adaptation dynamic update system | |
| CN108802657A (en) | A kind of digital oscilloscope automated calibration system and its application method | |
| CN211087029U (en) | Device for realizing remote multi-greenhouse environment control based on Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20191015 |
|
| WD01 | Invention patent application deemed withdrawn after publication |