CN201252571Y - Automatic data exchanging device with high security - Google Patents
Automatic data exchanging device with high security Download PDFInfo
- Publication number
- CN201252571Y CN201252571Y CNU2008201527591U CN200820152759U CN201252571Y CN 201252571 Y CN201252571 Y CN 201252571Y CN U2008201527591 U CNU2008201527591 U CN U2008201527591U CN 200820152759 U CN200820152759 U CN 200820152759U CN 201252571 Y CN201252571 Y CN 201252571Y
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- processing unit
- workflow
- pick
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Abstract
The utility model discloses an automatic data exchanging device with high security comprising a data extracting unit which extracts data from a data source and transmits the data to a data processing unit, the data processing unit which carries out various processing to the data which is extracted by the data extracting unit, and a data loading unit which loads the data which is processed by the data processing unit to a data-base, wherein the data extracting unit, the data processing unit and the data loading unit are connected through data pipelines, the device exchanges data through the data extracting unit, the data processing unit and the data loading unit, workflow is used for describing the integral data exchanging process, a workflow triggering mechanism with priority level is adopted to simultaneously realize multiple data-exchanging processes, and safety measures, such as data comprehensive detection and the like are inlaid in the workflow to realize the high-safety data exchanging, thereby realizing the aims of the utility model.
Description
Technical field
The utility model relates to a kind of automaticdata switch, particularly a kind of automaticdata switch that is applicable to the high security of field of computer information security.
Background technology
Along with professional development such as E-Government, ecommerce, information realizes automatic, quick, safe exchange between different networks, system and data source, promptly " exchanges data " becomes the subject matter that E-Government, electronic commerce information are shared and information interconnects, and data are most important elements in the data exchange process, guarantee quality, fail safe, consistency and the confidentiality etc. of data in the data exchange process, data are carried out security inspection, are the key problems that exchanges data must solve.
But in existing data exchange process, between trustable network and the untrusted network, the exchanges data between heterogeneous system, the heterogeneous data source all compares difficulty, realizes more complicated; Simultaneously, can't guarantee quality, fail safe, consistency and the confidentiality etc. of data in the data exchange process, data are carried out security inspection.
Therefore, the automaticdata switch that needs a kind of high security especially, exchange profound demand at E-Government, electronic commerce data to the data fail safe, the complexity of data exchange service and the outstanding feature that demand constantly changes have been taken into full account, data to exchange are checked, reach the purpose of commencing business rapidly and alleviating amount of user effort to greatest extent.
The utility model content
Technical problem to be solved in the utility model is to provide a kind of automaticdata switch of high security, in data exchange process, adopt working flow mode, automatically trigger data exchanging workflows journey, and data integrated use wooden horse anti-virus filtered, inspection methods such as file format guarantee the fail safe of data exchange process, guarantee safety of data, realization resource consolidation, information security are shared, and guarantee carrying out smoothly of disparate networks business.
Technical problem to be solved in the utility model can be achieved through the following technical solutions:
A kind of automaticdata switch of high security, it is characterized in that, it comprises: the data that data are extracted and are transferred to the data pick-up unit of described data processing unit from data source, described data pick-up unit extracted are carried out the data processing unit of various processing and data load that described data processing unit the is handled data load unit in the database, connect by the data pipeline between described data pick-up unit, data processing unit and the data load unit.
Described data processing unit comprises functional modules such as data connection, data filter, the processing of viral wooden horse and file format inspection.
Mode by workflow between described data pick-up unit, data processing unit and the data load unit is carried out exchanges data.
Mode by workflow between described data pick-up unit, data processing unit and the data load unit is described and the exchange current process control each data exchange process.
Carry out the triggering of exchanges data by the mode of time and message between described data pick-up unit, data processing unit and the data load unit.
The automaticdata switch of high security of the present utility model, carry out exchanges data by data pick-up unit, data processing unit and data load unit, and a partial data exchange process is described with workflow, the workflow trigger mechanism that employing possesses priority is realized that a plurality of data exchange processes are concurrent and is carried out, and in workflow, embed the exchanges data that safety measure such as aggregation of data inspection realizes high securities, realize the purpose of this utility model.
Description of drawings
Fig. 1 is the system block diagram of the automaticdata switch of high security of the present utility model.
Specific implementation method
For technological means, creation characteristic that the utility model is realized, reach purpose and effect is easy to understand, below in conjunction with concrete diagram, further set forth the utility model.
As shown in Figure 1, a kind of automaticdata switch of high security, it comprises: data pick-up unit 1, data processing unit 2 and data load unit 3 constitute a data exchanging workflows by data pick-up unit 1, data processing unit 2 and data load unit 3; So-called workflow by one group of functional unit according to the operation flow order by data pipe (DataPipe) realization that is serially connected.
A basic exchanges data workflow is by industry unit, a source, a target service unit and a cover exchange regulation are formed, in fact the exchanges data workflow of a complexity can be split as some groups of basic exchanges data workflows, the formation element of basic exchanges data workflow is most basic exchanges data elementary cell, comprise and extract object type and configurable extracting unit, loaded targets type and configurable loading unit and configurable converting unit, requirement according to concrete business object, by selecting, dispose these elementary cells, form a complete exchanges data workflow.
In the present embodiment, data pick-up unit 1 is as extracting unit, and extracted data is come out and is transferred to data processing unit 2 from data source; Data processing unit 2 carries out various processing as converting unit to the data that data extracting unit 1 extracts; Data load unit 3 is as loading unit, the data load that to handle through data processing unit 2 is in database, connect by the data pipeline between data pick-up unit 1, data processing unit 2 and the data load unit 3, constitute a complete exchanges data workflow.
In exchanges data workflow of the present utility model, data pick-up (Extract) unit 1 generally has two, extracts data respectively from database and text; Pass to next connected node (Join) by two data pipelines (Datapipe) by data processing unit 2, connected node is according to the condition of contact of prior design, the join that two ducted data are carried out in the similar Database Systems operates, and data are laterally merged; In data processing unit 2, pass to the next node (Clean) that cleans, the data of importing into are cleaned by a data pipeline (Datapipe); Data after cleaning are sent to switching node (Transformation), and the data format and content is changed and hinted obliquely at; Data after the conversion are distributed to the loading node (Load) and the aggregation (Aggregation) of data load unit 3, and the combined data after the polymerization also is sent to the loading node (Load) of data load unit 3; Loading the responsible detailed data and the combined data that will finally produce of node (Load) is saved in target database, data warehouse, file, the target application system.
The customization of the exchanges data rule of the automaticdata switch of high security of the present utility model and writing comprises data conversion rule and exchanges data operation rule, and data conversion rule is responsible for the transfer process definition of data; The exchanges data operation rule is responsible for connecting source data and target data, realizes the scheduling of data conversion process.
Data pick-up unit 1 has the functional unit of following type:
DBReader: reading of data from the DBMS data source, support incremental data to extract;
FileReader: reading of data from external file;
FTP: reading of data from ftp server;
HTTP/WEB SERVICES: send WEB and ask server, then reading of data from the data flow that the WEB server returns;
SMTP (simple mail transfer protocol): visit e-mail server, reading of data from the E-mail address of appointment;
JMS: reading of data from the JMS messaging bus;
The rank JAVABEANS of enterprise (EJB): call the EJB interface, reading of data from the data flow of returning.
Data processing unit 2 has the functional unit of following type:
Joiner: realize the Join operation of two data sets, comprise connection type common in the database, and the part connection type that does not provide in the database.Utilize the Joiner assembly, can also realize the Join operation in different pieces of information source;
Filter: according to the condition filter data of appointment;
Switcher: according to some characteristic value of data, toward different data pipe dateouts;
Duplicator: duplicate the data of input channel, toward a plurality of data pipe outputs;
Sorter: data are sorted;
Aggregator: data are added up computing, for example polymerization computings such as Sum, Count, Max, Min;
Converter: data are carried out various conversions, comprise that type conversion, data computation, code disappearance detect replacement etc.DataConflux provides a large amount of conversion functions, and the developer can design, add required function as required voluntarily.
Antivirus: data are carried out viral wooden horse killing.
Data load unit 3 has the functional unit of following type:
DBWriter: upgrade data in the target database in the mode of SQL statement;
DBLoader: the mode loading data that loads in batches with external file in target database, the Bulk Copy among the Sybase ASE for example, the Load of Sybase IQ;
FileWriter: data are exported to the external text file, support local text, FTP telefile.
DataConflux: comprised functional unit and transfer function commonly used in the ETL product, and function and major key transfer function that some qualities of data detect, clean have been increased, for example, whether the Check function can detect certain is code value in the standard code table for value, is certain particular code value otherwise it is cleaned.
In the automaticdata switch of high security of the present utility model, carry out the triggering of exchanges data by the mode of time and message between data pick-up unit 1, data processing unit 2 and the data load unit 3.
The triggering of time-based exchanges data, the priority of workflow are based on system priority, and the task scheduling of each time is carried out according to priority.
Based on the triggering of the exchanges data of timer message, to dispatch based on message mechanism at every turn, message can adopt two kinds of timer message and event messages.
The automaticdata switch support of high security of the present utility model triggers flow process by the set time point, by the configuration mode definition dispatching cycle of similar Crontab; Support press fixed time period triggering flow process, as per 3 minutes, per 1 second etc.Minimum interval: 0.1 second; Perhaps, trigger in real time by various interface.
Aggregation of data inspection in the data processing unit 2 comprises that data filter, viral wooden horse are handled, file format is checked contour data of safety inspection method, and based on the above-mentioned inspection method of mode integrated use of workflow.Use anti-virus software that viral wooden horse is handled, have document format data inspection, data traffic dynamic chek duplication check function, all processing are finished with the processing speed near linear speed in internal memory.File format is checked, as required to document classification, as text, picture, sound, video etc., check the form whether file format suits the requirements and transmit, if with exchange files, if text carries out preliminary filtration to the content of text.Aggregation of data inspection method based on workflow.In order to realize data high security inspection in the data exchange process, above-mentioned several data checkings are interconnected to constitute operation flow according to business activity according to sequential or logical relation.Carry out in the process in business, data are according to inspection method and organize standard transmission, handle or carry out, and have realized automation, comprehensive data checks.
Referring to Fig. 1,2 pairs of data of data processing unit are carried out viral wooden horse when checking, adopt the antivirus functional module of the data processing unit 2 of workflow to finish, promptly each trojan horse filtration inspection task (Job) is formed by connecting by data pipe according to certain logical order by some independent functional units, and this is a typical data checks flow process.Trojan horse filters checks it at first is data pick-up unit 1, extracts data from database and text respectively; Pass to by two data pipelines and to read the filtering rule node, be delivered to according to rule by data pipe again and check node, the anti-virus trojan horse program starts, it is investigated virus killing and look into wooden horse according to regular logarithm, then passing to processing node by a data pipeline handles, if find to have virus or wooden horse, data are abandoned, otherwise, the detailed data that has checked and the combined data that finally produce are saved in target database, data warehouse, file, in the target application system, can add fire compartment wall data filter and other inspection method nodes before the trojan horse inspection method as required.
Data format and Content inspection mainly are the data at file type, adopt the tupe of workflow, and file format and data content inspection at first are data pick-up unit 1, from extracting data from database and text; Pass to by data pipe and to read regular node, be delivered to by data pipe according to file format rule inspection node, this moment, program was at first checked first file type of rule, by configuration, file type can be divided into two kinds of blacklist and white lists, the file that meets white list then is delivered to next functional unit, the data that meet blacklist are then abandoned, and then other of traversal rule, so that check the content of file, passing to processing node by a data pipeline at last handles, the data that form and content are correct exchange, and detailed data that has checked and the combined data that finally produces is saved in target database, data warehouse, file, in the target application system.Can add fire compartment wall data filter and other inspection method nodes before the file checking method as required.
More than show and described basic principle of the present utility model and principal character and advantage thereof.The technical staff of the industry should understand; the utility model is not restricted to the described embodiments; that describes in the foregoing description and the specification just illustrates principle of the present utility model; under the prerequisite that does not break away from the utility model spirit and scope; the utility model also has various changes and modifications, and these changes and improvements all fall in claimed the utility model scope.The claimed scope of the utility model is defined by appending claims and equivalent thereof.
Claims (5)
1, a kind of automaticdata switch of high security, it is characterized in that, it comprises: the data that data are extracted and are transferred to the data pick-up unit of described data processing unit from data source, described data pick-up unit extracted are carried out the data processing unit of various processing and data load that described data processing unit the is handled data load unit in the database, connect by the data pipeline between described data pick-up unit, data processing unit and the data load unit.
2, automaticdata switch as claimed in claim 1 is characterized in that, described data processing unit comprises functional modules such as data connection, data filter, the processing of viral wooden horse and file format inspection.
3, automaticdata switch as claimed in claim 1 is characterized in that, the mode by workflow between described data pick-up unit, data processing unit and the data load unit is carried out exchanges data.
4, automaticdata switch as claimed in claim 1 is characterized in that, the mode by workflow between described data pick-up unit, data processing unit and the data load unit is described and the exchange current process control each data exchange process.
5, automaticdata switch as claimed in claim 1 is characterized in that, carries out the triggering of exchanges data by the mode of time and message between described data pick-up unit, data processing unit and the data load unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNU2008201527591U CN201252571Y (en) | 2008-09-05 | 2008-09-05 | Automatic data exchanging device with high security |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNU2008201527591U CN201252571Y (en) | 2008-09-05 | 2008-09-05 | Automatic data exchanging device with high security |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN201252571Y true CN201252571Y (en) | 2009-06-03 |
Family
ID=40748234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNU2008201527591U Expired - Lifetime CN201252571Y (en) | 2008-09-05 | 2008-09-05 | Automatic data exchanging device with high security |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN201252571Y (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010145109A1 (en) * | 2009-06-18 | 2010-12-23 | 中兴通讯股份有限公司 | Sp access control method and system based on wap |
| CN101945116A (en) * | 2010-09-25 | 2011-01-12 | 公安部第三研究所 | Method for exchanging cross-domain video data safely |
| CN104104738A (en) * | 2014-08-06 | 2014-10-15 | 江苏瑞中数据股份有限公司 | FTP-based (file transfer protocol-based) data exchange system |
-
2008
- 2008-09-05 CN CNU2008201527591U patent/CN201252571Y/en not_active Expired - Lifetime
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010145109A1 (en) * | 2009-06-18 | 2010-12-23 | 中兴通讯股份有限公司 | Sp access control method and system based on wap |
| CN101945116A (en) * | 2010-09-25 | 2011-01-12 | 公安部第三研究所 | Method for exchanging cross-domain video data safely |
| CN104104738A (en) * | 2014-08-06 | 2014-10-15 | 江苏瑞中数据股份有限公司 | FTP-based (file transfer protocol-based) data exchange system |
| CN104104738B (en) * | 2014-08-06 | 2018-02-02 | 江苏瑞中数据股份有限公司 | A kind of data exchange system based on FTP |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107995169B (en) | Mobile game server framework system based on micro-service architecture | |
| CN101257671B (en) | Method for real time filtering large scale rubbish SMS based on content | |
| US7908160B2 (en) | System and method for producing audit trails | |
| CN104104738A (en) | FTP-based (file transfer protocol-based) data exchange system | |
| CN103246735A (en) | Abnormal data processing method and abnormal data processing system | |
| CN110069495A (en) | Date storage method, device and terminal device | |
| CN110865997A (en) | Online identification method for hidden danger of power system equipment and application platform thereof | |
| WO2017008598A1 (en) | Big data exchange method and device | |
| CN112737926B (en) | Mail sending method and device, electronic equipment and storage medium | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN201252571Y (en) | Automatic data exchanging device with high security | |
| CN105681426B (en) | Heterogeneous system | |
| CN114528458A (en) | Logistics track pushing method, device, equipment and storage medium | |
| CN102055620B (en) | Method and system for monitoring user experience | |
| CN112990746A (en) | Cloud platform intelligent management system of inspection and detection mechanism | |
| CN101729259A (en) | Highly-safe data checking method for data switching system | |
| CN105468502A (en) | Log collection method, device and system | |
| CN103530297A (en) | Method and device capable of automatically carrying out website analysis | |
| CN111401819B (en) | Intersystem data pushing method and system | |
| CN114297841A (en) | Simulation model resource system construction system and construction method thereof | |
| CN107451910A (en) | A kind of city one-card data collecting system | |
| CN114741404A (en) | Early warning information aggregation system and method and electronic equipment | |
| Racka | Apache Nifi As A Tool For Stream Processing Of Measurement Data | |
| CN112965948A (en) | Management service center system based on data | |
| CN110909086A (en) | Mail archiving method, system, computer device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CX01 | Expiry of patent term | ||
| CX01 | Expiry of patent term |
Granted publication date: 20090603 |