CN111177083A - Authorization information analysis and file data analysis system - Google Patents
Authorization information analysis and file data analysis system Download PDFInfo
- Publication number
- CN111177083A CN111177083A CN201911315749.4A CN201911315749A CN111177083A CN 111177083 A CN111177083 A CN 111177083A CN 201911315749 A CN201911315749 A CN 201911315749A CN 111177083 A CN111177083 A CN 111177083A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- analysis
- authorization information
- parsing system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 24
- 238000004458 analytical method Methods 0.000 title claims abstract description 23
- 238000013475 authorization Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000012545 processing Methods 0.000 claims abstract description 19
- 230000003993 interaction Effects 0.000 claims abstract description 4
- 238000004422 calculation algorithm Methods 0.000 claims description 23
- 239000000344 soap Substances 0.000 claims description 5
- 238000007620 mathematical function Methods 0.000 claims description 3
- 230000002159 abnormal effect Effects 0.000 abstract description 8
- 238000013481 data capture Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000006854 communication Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1435—Saving, restoring, recovering or retrying at system level using file system or storage system metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
Abstract
The invention relates to the technical field of data analysis, in particular to an authorization information analysis and file data analysis system. The method comprises a man-machine conversation unit, a processing unit and a data unit, and comprises the following steps: the human-computer conversation unit is used as a user interaction platform, receives remote sensing data information and sends a control command to the processing unit; the processing unit carries out data analysis on the remote sensing data information and analyzes the remote sensing data information; the analyzed remote sensing data information is stored in the data unit. In the authorization information analysis and file data analysis system, the processing unit is used for carrying out data analysis on remote sensing data information, abnormal data and repeated data are removed, the effectiveness of the data is guaranteed, the remote sensing data information is analyzed, the integrity of the analyzed data is improved, the data synchronization module is arranged, the analyzed data can be synchronously stored, and meanwhile, the data backup module is arranged, so that data backup is realized, and data loss is prevented.
Description
Technical Field
The invention relates to the technical field of data analysis, in particular to an authorization information analysis and file data analysis system.
Background
The meteorological data are telemetered by a satellite, and are analyzed by parallel telemetered data, data need to be transmitted in the network communication process, at present, the integrity of the analyzed data is low, error codes are easy to occur, meanwhile, the analyzed data cannot be backed up and stored, and when a fault occurs, the data are completely lost.
Disclosure of Invention
The present invention is directed to provide an authorization information analysis and file data analysis system, so as to solve the problems set forth in the background art.
In order to achieve the above object, the present invention provides an authorization information analysis and file data analysis system, which comprises a human-machine conversation unit, a processing unit and a data unit, wherein the method comprises the following steps:
s1, the human-computer conversation unit is used as a user interaction platform, receives remote sensing data information and sends a control command to the processing unit;
s2, the processing unit analyzes the remote sensing data information and analyzes the remote sensing data information;
and S3, storing the analyzed remote sensing data information in a data unit.
Preferably, the processing unit comprises a data access module, an analysis algorithm module and a resolution algorithm module.
Preferably, the access method of the data access module comprises the following steps:
s1.1, compiling a single DA (data acquisition) class for each PD class; such as: compiling a DA (data acquisition) class named CustomerDA (Customer data acquisition) for the Customer class, wherein the DA class can provide data storage and retrieval service for customers;
s1.2, only the PD class calls the method in the DA class.
Preferably, the analysis algorithm module comprises abnormal data analysis and repeated data analysis, the abnormal data is used for analyzing node data such as RTU serial numbers, data IDs and the like in the data, if the abnormal data occurs, the data is invalid, and the node data can be labeled and prompted to ensure the accuracy of the telecontrol data; the repeated data analysis is used for scanning node data such as data ID in the data, and when the repeated data is found, the data is labeled and prompted, so that the reliability of the telecontrol data is ensured.
Preferably, the analytic algorithm module uses a mathematical function F (v, X, T), where v represents the current telemetry data, X is a related variable, and T is time, and the formula is as follows:
where v is the physical quantity of the telemetry data.
f (v) is the value of the telemetering data in a certain relation, the characteristic of time obtained by calculating the source code of the telemetering data is the variation trend of f (v), f (v)1(v) For the values of the previous frame of the telemetry data, the linear relationship of f (v) is as follows:
(iv) av + M … … (formula 1.1);
where a is a coefficient of a physical quantity v of the telemetry data, then:
Δf(v)＝a(v-v1) … … (formula 1.2);
wherein v is1Is a physical quantity of one frame above the physical quantity of the telemetry data.
Preferably, the data unit comprises a data synchronization module, a data backup module and a storage management module.
Preferably, the data synchronization module is implemented based on a Web Service database, and the data synchronization method comprises the following steps:
s2.1, designing a SOAP data synchronization flow; the monitoring of the database is mainly completed by a monitoring program, if the monitoring program changes, the SOAP protocol is generally used for correspondingly updating and packaging the data information and the control information, and the target database is responsible for receiving the data and then synchronizing the data to the Web Service;
s2.2, capturing data; the data capture is internally provided with a control change table and a data capture module, a trigger is used for capturing data, and the data capture is mainly carried out on a part where the trigger is found to be modified;
s2.3, caching the table Temp; the method is mainly used for recording the table number, the table name and the entry time of the data, and the data monitored by the trigger can be modified, deleted and added by people so as to update IP, numbers and the like.
Preferably, the data backup module adopts a multi-server backup mode or a single-server backup mode.
A multi-server backup mode: one or more tape drives are connected to a server to back up data in the server and all other servers and clients in the network.
Single server backup mode: each server is connected with a tape unit and only backs up data in the server; generally, the backup method can only backup data in the local hard disk of the server, but cannot backup data of other servers in the network.
Preferably, the memory management module adopts a page replacement algorithm, and the algorithm comprises the following steps:
s3.1, generating an instruction sequence through a random number; the number of the instructions is 320, and the address of the instruction is generated according to the following principle:
firstly, generating an address of an instruction according to the following principle;
②, 25% of the instructions are uniformly distributed in the front address part;
③, 25% of the instructions are uniformly distributed in the rear address part;
the specific implementation method comprises the following steps:
the method comprises the following steps: randomly selecting a starting point m between the instruction addresses of [0, 319 ];
step two: sequentially executing an instruction, namely executing the instruction with the address of m + l;
step three: randomly selecting an instruction from the previous address [0, m +1] and executing the instruction, wherein the address of the instruction is m';
step four: sequentially executing an instruction with an address of m' + 1;
step five: randomly selecting an instruction from the rear address [ m' +2, 319] and executing;
step six: repeating the above steps one to five until 320 instructions are executed.
S3.2, converting the instruction sequence into a page address stream, ② the page size of ① to be 1K, ② the user memory capacity of ① to be 4 pages of ① to ③ pages, and ② the user virtual memory capacity of ① to be ③ K.
And S3.3, calculating and outputting the hit rate of the following algorithms under different memory capacities.
Compared with the prior art, the invention has the beneficial effects that:
1. in the authorization information analysis and file data analysis system, the processing unit is used for carrying out data analysis on the remote sensing data information, eliminating abnormal data and repeated data, ensuring the validity of the data, analyzing the remote sensing data information and improving the integrity of the analyzed data.
2. In the authorization information analysis and file data analysis system, a data synchronization module is arranged, analyzed data can be synchronously stored, and a data backup module is arranged to realize data backup and prevent data loss.
Drawings
FIG. 1 is a block diagram of the overall structure of the present invention;
FIG. 2 is an overall process flow diagram of the present invention;
FIG. 3 is a flow chart of an accessing method of the data accessing module according to the present invention;
FIG. 4 is a flow chart of a data synchronization method of the present invention;
FIG. 5 is a flowchart of a page replacement algorithm method of the present invention.
The various reference numbers in the figures mean:
1. a human-machine conversation unit;
2. a processing unit; 21. a data access module; 22. an analysis algorithm module; 23. an analysis algorithm module;
3. a data unit; 31. a data synchronization module; 32. a data backup module; 33. and a storage management module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-5, the present invention provides a technical solution:
the invention provides an authorization information analysis and file data analysis system, which comprises a man-machine conversation unit 1, a processing unit 2 and a data unit 3, wherein the method comprises the following steps:
s1, the human-computer conversation unit 1 is used as a user interaction platform, receives remote sensing data information and sends a control command to the processing unit 2;
s2, the processing unit 2 carries out data analysis on the remote sensing data information and analyzes the remote sensing data information;
s3, the remote sensing data information after analysis is stored in the data unit 3.
The processing unit 2 comprises a data access module 21, an analysis algorithm module 22 and an analysis algorithm module 23.
In this embodiment, the access method of the data access module 21 includes the following steps:
s1.1, compiling a single DA (data acquisition) class for each PD class; such as: compiling a DA (data acquisition) class named CustomerDA (Customer data acquisition) for the Customer class, wherein the DA class can provide data storage and retrieval service for customers;
s1.2, only the PD class calls the method in the DA class. Classes that require data storage and retrieval services must call the DA method in the Customer, which in turn calls the CustomerDA method.
Further, the data access module 21 includes four modules of finding a client, adding a client, changing a client, and deleting a client, wherein:
the method for searching the client comprises the following steps: using the telephone number as the unique identification of the client, and returning the reference to the client after finding the client. If the customer cannot be found, the method creates and throws an instance of the custom exception class;
the method for adding the client comprises the following steps: adding a client into the system, firstly determining whether the telephone number of the existing client is the same as the telephone number of the client being added, if so, creating and throwing an example duplificationException of a custom exception class by the method;
wherein, the method for changing the client comprises the following steps: the update method receives client reference in a parameter mode, searches the existing client and changes the address and telephone number value, if the client cannot be found, the method throws out NotFoundException;
the method for deleting the client comprises the following steps: a client is deleted from the system by first looking for a client that receives its reference in the form of a parameter and then deleting it from the data store, if the client cannot be found, the method will throw notfoundation exception.
Specifically, the analysis algorithm module 22 includes abnormal data analysis and repeated data analysis, the abnormal data is used for analyzing node data such as an RTU serial number, a data ID and the like in the data, and if the abnormal data occurs, it indicates that the data is invalid, the node data can be labeled and prompted to ensure the accuracy of the telecontrol data; the repeated data analysis is used for scanning node data such as data ID in the data, and when the repeated data is found, the data is labeled and prompted, so that the reliability of the telecontrol data is ensured.
Still further, the analysis algorithm module 23 employs a mathematical function F (v, X, T), where v represents the current telemetry data, X is a related variable, and T is time, and the formula is as follows:
where v is the physical quantity of the telemetry data.
It is worth to be noted that f (v) is the value of the telemetry data in a certain relation, the characteristic of time obtained by calculation of the source code of the telemetry data is the variation trend of f (v), f (v)1(v) For the values of the previous frame of the telemetry data, the linear relationship of f (v) is as follows:
(iv) av + M … … (formula 1.1);
where a is a coefficient of a physical quantity v of the telemetry data, then:
Δf(v)＝a(v-v1) … … (formula 1.2);
wherein v is1Is a physical quantity of one frame above the physical quantity of the telemetry data.
It should be noted that by calculating the difference, the time characteristics of the telemetry data, including special patterns such as sine and square waves, can be represented and analyzed.
Specifically, the data unit 3 includes a data synchronization module 31, a data backup module 32, and a storage management module 33.
Further, the data synchronization module 31 is implemented based on a Web Service database, and the data synchronization method includes the following steps:
s2.1, designing a SOAP data synchronization flow; the monitoring of the database is mainly completed by a monitoring program, if the monitoring program changes, the SOAP protocol is generally used for correspondingly updating and packaging the data information and the control information, and the target database is responsible for receiving the data and then synchronizing the data to the Web Service;
s2.2, capturing data; the data capture is internally provided with a control change table and a data capture module, a trigger is used for capturing data, and the data capture is mainly carried out on a part where the trigger is found to be modified;
s2.3, caching the table Temp; the method is mainly used for recording the table number, the table name and the entry time of the data, and the data monitored by the trigger can be modified, deleted and added by people so as to update IP, numbers and the like.
Furthermore, in the database synchronization system of the Web Service, the implementation of each software is independent, the data information in different data systems can be repeatedly input, the data of the heterogeneous database can be synchronized by using a trigger technology, the data change condition in the data system can be better and more accurately obtained, several types of trigger mechanisms of delete \ insert \ update are constructed, when the data of the heterogeneous database is updated, the data synchronization change condition can appear in a temporary table, and the processing response to the data is mainly in a client program. The synchronization of the information data in the heterogeneous database is realized in a client program, the client program is regularly scanned for trigger events in the database system, and the synchronous data is analyzed and processed and a service function is applied.
In addition, the data backup module 32 adopts a multi-server backup mode or a single-server backup mode.
Wherein, the multi-server backup mode: one or more tape drives are connected to a server to back up data in the server and all other servers and clients in the network.
Wherein, the single server backup mode: each server is connected with a tape unit and only backs up data in the server; generally, the backup method can only backup data in the local hard disk of the server, and certainly includes other clients in the network, but cannot backup data of other servers in the network.
It should be noted that the storage management module 33 adopts a page replacement algorithm, and the algorithm includes the following steps:
s3.1, generating an instruction sequence through a random number; the number of the instructions is 320, and the address of the instruction is generated according to the following principle:
firstly, generating an address of an instruction according to the following principle;
②, 25% of the instructions are uniformly distributed in the front address part;
③, 25% of the instructions are uniformly distributed in the rear address part;
the specific implementation method comprises the following steps:
the method comprises the following steps: randomly selecting a starting point m between the instruction addresses of [0, 319 ];
step two: sequentially executing an instruction, namely executing the instruction with the address of m + l;
step three: randomly selecting an instruction from the previous address [0, m +1] and executing the instruction, wherein the address of the instruction is m';
step four: sequentially executing an instruction with an address of m' + 1;
step five: randomly selecting an instruction from the rear address [ m' +2, 319] and executing;
step six: repeating the above steps one to five until 320 instructions are executed.
S3.2, converting the instruction sequence into a page address stream, ② the page size of ① to be 1K, ② the user memory capacity of ① to be 4 pages of ① to ③ pages, and ② the user virtual memory capacity of ① to be ③ K.
And S3.3, calculating and outputting the hit rate of the following algorithms under different memory capacities.
The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and the preferred embodiments of the present invention are described in the above embodiments and the description, and are not intended to limit the present invention. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (9)
1. The system for analyzing the authorization information and analyzing the file data comprises a man-machine conversation unit (1), a processing unit (2) and a data unit (3), and the method comprises the following steps:
s1, the human-computer conversation unit (1) is used as a user interaction platform, receives remote sensing data information and sends a control command to the processing unit (2);
s2, the processing unit (2) carries out data analysis on the remote sensing data information and analyzes the remote sensing data information;
and S3, storing the analyzed remote sensing data information in the data unit (3).
2. The authorization information parsing system and file data parsing system according to claim 1, wherein: the processing unit (2) comprises a data access module (21), an analysis algorithm module (22) and an analysis algorithm module (23).
3. The authorization information parsing system and file data parsing system of claim 2, wherein: the access method of the data access module (21) comprises the following steps:
s1.1, compiling a single DA (data acquisition) class for each PD class;
s1.2, only the PD class calls the method in the DA class.
4. The authorization information parsing system and file data parsing system of claim 2, wherein: the analysis algorithm module (22) includes anomaly data analysis and data duplication analysis.
5. The authorization information parsing system and file data parsing system of claim 2, wherein: the analytical algorithm module (23) employs a mathematical function F (v, X, T), where v represents the current telemetry data, X is a related variable, and T is time, and the formula is as follows:
where v is the physical quantity of the telemetry data.
6. The authorization information parsing system and file data parsing system according to claim 1, wherein: the data unit (3) comprises a data synchronization module (31), a data backup module (32) and a storage management module (33).
7. The authorization information parsing system according to claim 6, wherein: the data synchronization module (31) is realized based on a Web Service database, and the data synchronization method comprises the following steps:
s2.1, designing a SOAP data synchronization flow;
s2.2, capturing data;
s2.3, caching the table Temp.
8. The authorization information parsing system according to claim 6, wherein: the data backup module (32) adopts a multi-server backup mode or a single-server backup mode.
9. The authorization information parsing system according to claim 6, wherein: the memory management module (33) adopts a page replacement algorithm, which comprises the following steps:
s3.1, generating an instruction sequence through a random number;
s3.2, converting the instruction sequence into a page address stream;
and S3.3, calculating and outputting the hit rate of the following algorithms under different memory capacities.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911315749.4A CN111177083A (en) | 2019-12-19 | 2019-12-19 | Authorization information analysis and file data analysis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911315749.4A CN111177083A (en) | 2019-12-19 | 2019-12-19 | Authorization information analysis and file data analysis system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111177083A true CN111177083A (en) | 2020-05-19 |
Family
ID=70651988
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911315749.4A Pending CN111177083A (en) | 2019-12-19 | 2019-12-19 | Authorization information analysis and file data analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111177083A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521687A (en) * | 2011-12-01 | 2012-06-27 | 中国资源卫星应用中心 | Miniaturized universal platform for preprocessing remote-sensing satellite data |
| CN104252526A (en) * | 2014-08-29 | 2014-12-31 | 航天东方红卫星有限公司 | Telemetry storage method based on redundancy elimination |
| CN109063258A (en) * | 2018-06-29 | 2018-12-21 | 北京微纳星空科技有限公司 | Satellitosis telemetry parameter storage method and system |
| CN110188093A (en) * | 2019-05-21 | 2019-08-30 | 江苏锐天信息科技有限公司 | A kind of data digging system being directed to AIS information source based on big data platform |
-
2019
- 2019-12-19 CN CN201911315749.4A patent/CN111177083A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521687A (en) * | 2011-12-01 | 2012-06-27 | 中国资源卫星应用中心 | Miniaturized universal platform for preprocessing remote-sensing satellite data |
| CN104252526A (en) * | 2014-08-29 | 2014-12-31 | 航天东方红卫星有限公司 | Telemetry storage method based on redundancy elimination |
| CN109063258A (en) * | 2018-06-29 | 2018-12-21 | 北京微纳星空科技有限公司 | Satellitosis telemetry parameter storage method and system |
| CN110188093A (en) * | 2019-05-21 | 2019-08-30 | 江苏锐天信息科技有限公司 | A kind of data digging system being directed to AIS information source based on big data platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6330600B1 (en) | System for synchronizing configuration information of a network element if received trap sequence number is out-of-sequence | |
| JP6716727B2 (en) | Streaming data distributed processing method and apparatus | |
| EP0952521B1 (en) | Method for tracking configuration changes in networks of computer systems through historical monitoring of configuration status of devices on the network | |
| CN105447046A (en) | Distributed system data consistency processing method, device and system | |
| EP2486698B1 (en) | Method and system for reconstructing transactions in a communication network | |
| CN112040429B (en) | Short message management system and method based on distributed storage | |
| CN110933115B (en) | Analysis object behavior abnormity detection method and device based on dynamic session | |
| CN100514962C (en) | Host performance collection proxy in large-scale network | |
| CN107103014A (en) | The replay method of history pushed information, device and system | |
| CN111061498A (en) | Configuration information management system | |
| CN111740868A (en) | Alarm data processing method and device and storage medium | |
| CN104636395A (en) | Count processing method and device | |
| CN111382334B (en) | Data processing method and device, computer and readable storage medium | |
| CN112328702A (en) | Data synchronization method and system | |
| CN104123102B (en) | A kind of IP hard disks and its data processing method | |
| CN105069029A (en) | Real-time ETL (extraction-transformation-loading) system and method | |
| CN116827965B (en) | Coal mine underground offline scene data storage and synchronization method based on cloud platform | |
| CN108809678A (en) | A kind of method and server of information push | |
| CN111177083A (en) | Authorization information analysis and file data analysis system | |
| CN111221815B (en) | Script-based web service paging data acquisition system | |
| CN114465741A (en) | Anomaly detection method and device, computer equipment and storage medium | |
| CN111966650B (en) | Operation and maintenance big data sharing data table processing method and device and storage medium | |
| CN105791350B (en) | Project data processing method and system | |
| CN109525649B (en) | Data processing method and device for zookeeper client | |
| CN111291127B (en) | Data synchronization method, device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240108 Address after: 410073 Hunan province Changsha Kaifu District, Deya Road No. 109 Applicant after: National University of Defense Technology Address before: No.60, jiagangmen, Qinhuai District, Nanjing, Jiangsu Province, 210000 Applicant before: Hu Youbin |
|
| TA01 | Transfer of patent application right |