CN104142942A - Mass heterogeneous data accessing and standardization method - Google Patents
Mass heterogeneous data accessing and standardization method Download PDFInfo
- Publication number
- CN104142942A CN104142942A CN201310165749.7A CN201310165749A CN104142942A CN 104142942 A CN104142942 A CN 104142942A CN 201310165749 A CN201310165749 A CN 201310165749A CN 104142942 A CN104142942 A CN 104142942A
- Authority
- CN
- China
- Prior art keywords
- thread
- data
- client
- access
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
- Communication Control (AREA)
Abstract
The invention provides a mass heterogeneous data accessing and standardization method. The method is characterized by comprising the following step of arranging two sub-modules, namely, a protocol standardization sub-module and a communication adapting sub-module, wherein the protocol standardization sub-module is mainly used for providing an user-friendly interface to a client to fill personalized data protocol conversion rules, and compiling the data protocol conversion rules filled by the client into a serialized file for the communication adaption sub-module; the communication sub-module is mainly used for performing multi-threading scheduling when mass data access and are used for communicating, so as to link, receive and send data information of the client.
Description
Technical field
The present invention relates to a kind of access and standardized method of isomeric data, be specifically related to access and the standardisation process of field magnanimity isomeric data.
Background technology
In security field, the access that need to face magnanimity heterogeneous data information is processed, these data are data type isomery not only, as different data messages such as warning, location, portraits, and data type of the same race is because the field being employed or the classification difference that adheres to separately also have different data layouts, as the data type of UNICOM's well lid in well lid data and traffic control well lid difference to some extent just.These isomeric datas have caused system information interaction and interoperability very difficult, and Data Update can not be synchronous, and on stream, the problem such as program multiplicity is higher, and personnel's input amount is large, difficult in maintenance.
In mass data communication, conventionally adopt at present multithreading, and have many packaged multithreading storehouses, but in use also there are some problems in these multithreading storehouses.For example the DarwinInclude of Apple company, is mainly the library file for streaming media server, and its encapsulation to Socket is lower on the one hand, and developer is simplified not; On the other hand, this storehouse is not the access for mass data in security field, therefore in business demand, has a large amount of secondary overlapping development, has increased manpower and time cost.The shortcomings such as current ubiquitous use xml technology solves the otherness of isomeric data, but xml technology exists data volume large aspect the isomeric data access that solves this real-time magnanimity, and the time is slow and comparatively complicated.
Summary of the invention
The object of the present invention is to provide a kind of solution of access and standardized method of magnanimity isomeric data, this scheme, mainly towards developer, solves the problems such as the current magnanimity isomeric data access construction cycle is long, and performance history is complicated, and repeatability exploitation is too much.
The access of a kind of magnanimity isomeric data provided by the invention and standardized method, it is characterized in that: two submodules are set, be respectively consensus standard beggar module and the aptamer module of communicating by letter, wherein consensus standard beggar module is mainly responsible for offering user-friendly interface of client, make client can fill in personalized data protocol transformation rule, and the data protocol transformation rule that client is filled in is compiled into the file of a serializing, for communication aptamer module; Multithread scheduling when communication aptamer module is mainly responsible for mass data access and communication, completes the link of accepting client, receives and send data message.
Described multithreading has four kinds of thread types, is respectively to accept thread, management thread, worker thread, protocol conversion thread; Wherein, describedly accept the link that thread is responsible for accepting client; Described management thread is responsible for linked server, and has Auto-reconnect function; Described worker thread is responsible for receiving and sending data; Described protocol conversion thread is responsible for the data that receive, according to the file of serializing, converting the data of respective protocol form to.
The present invention is directed to the developer in security field, owing to encapsulating socket and buffer with ID, for upper strata provides interface, therefore the differentiation that has shielded Socket between communication, simplifies communication process, therefore makes performance history simpler, construction cycle is shorter, safeguards simpler.The link that improper communication disconnects, has Auto-reconnect function.And this programme is used serializing mechanism to solve data heterogeneous question, make program have better extensibility and operational effect faster.
Accompanying drawing explanation
Fig. 1 is modular structure block diagram of the present invention
Fig. 2 is data access of the present invention and standardization flow process figure
Embodiment
As shown in Figure 1, solution of the present invention is mainly provided with two submodules: consensus standard beggar module and the aptamer module of communicating by letter.
Consensus standard beggar module is mainly responsible for offering user-friendly interface of client, make client can fill in personalized data protocol transformation rule, and the data protocol transformation rule that client is filled in is compiled into the file of a serializing, for communication aptamer module.
Multithread scheduling when communication aptamer module is mainly responsible for mass data access and communication, completes the link of accepting client, receives and send data message.
The present invention is packaged into ID by socket and buffer, with ID, shows a session.ID allocation scheme is as follows:
| ID | The machine |
| 1——100 | Service end (TCP) |
| 101——1000 | Client (TCP) |
| 1001——3000 | ？ |
| 3001——3200 | Service end (UDP) |
| 3200——3500 | Client (UDP) |
As shown in Figure 2, solution of the present invention has four kinds of thread types, is respectively to accept thread, management thread, worker thread, protocol conversion thread.
Accept thread: be responsible for accepting the link of client.
Management thread: be responsible for linked server, and there is Auto-reconnect function.
Worker thread: be responsible for receiving and sending data.
Protocol conversion thread: be responsible for the data that receive, according to the file of serializing, converting the data of respective protocol form to.
Program operation, four sub-threads of main thread initializes: accept thread, management thread, worker thread, protocol conversion thread.
In worker thread, there is a thread pool, exist many can invoked thread, thread during initialization in thread pool is in blocked state, waiting event triggers.If there is Event triggered, the thread that worker thread calls in idle condition from thread pool completes event response.
Accept thread and always keep a plurality of sockets in listening state, therefore can accept the link of a plurality of clients simultaneously.Accept thread and when initialization, set up some sockets, for monitoring the link of external client, the event of accepting in response main thread, accepts event if exist, and accepts thread and will set up new socket, for establishing the link and keep communicating by letter with external client.
Management thread can communication non-artificial off-state under Auto-reconnect server.
Protocol conversion thread needs two inputs, the file of a serializing, the data of a paragraph format, the data layout that output protocol was changed.
Claims (6)
1. the access of a magnanimity isomeric data and standardized method, it is characterized in that: two submodules are set, be respectively consensus standard beggar module and the aptamer module of communicating by letter, wherein consensus standard beggar module is mainly responsible for offering user-friendly interface of client, make client can fill in personalized data protocol transformation rule, and the data protocol transformation rule that client is filled in is compiled into the file of a serializing, for communication aptamer module; Multithread scheduling when communication aptamer module is mainly responsible for mass data access and communication, completes the link of accepting client, receives and send data message.
2. the access of magnanimity isomeric data according to claim 1 and standardized method, is characterized in that: described multithreading has four kinds of thread types, is respectively to accept thread, management thread, worker thread, protocol conversion thread; Wherein, describedly accept the link that thread is responsible for accepting client; Described management thread is responsible for linked server, and has Auto-reconnect function; Described worker thread is responsible for receiving and sending data; Described protocol conversion thread is responsible for the data that receive, according to the file of serializing, converting the data of respective protocol form to.
3. the access of magnanimity isomeric data according to claim 2 and standardized method, it is characterized in that: in described worker thread, have a thread pool, exist many can invoked thread, thread during initialization in thread pool is in blocked state, waiting event triggers; If there is Event triggered, the thread that worker thread calls in idle condition from thread pool completes event response.
4. the access of magnanimity isomeric data according to claim 2 and standardized method, is characterized in that: the described thread of accepting always keeps a plurality of sockets in listening state, can accept the link of a plurality of clients simultaneously; Accept thread and when initialization, set up some sockets, for monitoring the link of external client, the event of accepting in response main thread, accepts event if exist, and accepts thread and will set up new socket, for establishing the link and keep communicating by letter with external client.
5. the access of magnanimity isomeric data according to claim 5 and standardized method, is characterized in that: described management thread is Auto-reconnect server under the non-artificial off-state of communication.
6. the access of magnanimity isomeric data according to claim 2 and standardized method, is characterized in that: described protocol conversion thread has two inputs, the file of a serializing, the data of a paragraph format, the data layout that output protocol was changed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310165749.7A CN104142942A (en) | 2013-05-08 | 2013-05-08 | Mass heterogeneous data accessing and standardization method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310165749.7A CN104142942A (en) | 2013-05-08 | 2013-05-08 | Mass heterogeneous data accessing and standardization method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104142942A true CN104142942A (en) | 2014-11-12 |
Family
ID=51852116
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310165749.7A Pending CN104142942A (en) | 2013-05-08 | 2013-05-08 | Mass heterogeneous data accessing and standardization method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104142942A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105897797A (en) * | 2014-12-03 | 2016-08-24 | 北京航天长峰科技工业集团有限公司 | Massive heterogeneous data access and standardization system |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
| CN110113374A (en) * | 2019-03-15 | 2019-08-09 | 平安科技（深圳）有限公司 | Streaming media server executes multitask method, device and storage medium, terminal device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090063504A1 (en) * | 2007-08-29 | 2009-03-05 | Richard Banister | Bi-directional replication between web services and relational databases |
| CN102566984A (en) * | 2010-12-07 | 2012-07-11 | 北大方正集团有限公司 | Method and device for configuring parameters |
-
2013
- 2013-05-08 CN CN201310165749.7A patent/CN104142942A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090063504A1 (en) * | 2007-08-29 | 2009-03-05 | Richard Banister | Bi-directional replication between web services and relational databases |
| CN102566984A (en) * | 2010-12-07 | 2012-07-11 | 北大方正集团有限公司 | Method and device for configuring parameters |
Non-Patent Citations (3)
| Title |
|---|
| 孙安健: "通用ETL工具的研究与设计", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 师硕等: "《Visual C#.NET实用教程》", 31 July 2012 * |
| 杨起帆: "《.NET架构与应用开发实践》", 28 February 2013 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105897797A (en) * | 2014-12-03 | 2016-08-24 | 北京航天长峰科技工业集团有限公司 | Massive heterogeneous data access and standardization system |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
| CN110113374A (en) * | 2019-03-15 | 2019-08-09 | 平安科技（深圳）有限公司 | Streaming media server executes multitask method, device and storage medium, terminal device |
| CN110113374B (en) * | 2019-03-15 | 2022-03-25 | 平安科技（深圳）有限公司 | Method and device for executing tasks by streaming media server, storage medium and terminal equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102790776B (en) | Heartbeat connection normalizing processing method, terminal, server and communication system | |
| US20150382132A1 (en) | Method and Device for Monitoring | |
| CN102202093B (en) | A kind of method and system realizing transducer adaptation | |
| CN105721543A (en) | Cross-protocol distribution-type cloud storage system based on operation control unit and data management method | |
| CN109980776A (en) | A kind of intelligence distribution transforming system and the systematic difference method | |
| CN103987071B (en) | Electric power TD LTE Radio Service systems | |
| CN102855143B (en) | All purpose communication framework in a kind of SCADA system | |
| CN110650203B (en) | Data transmission method, device and system, computer storage medium and electronic equipment | |
| CN104601723B (en) | Power Marketing Management System SOA framework based on internal services bus | |
| CN102810069A (en) | JAVA object requesting and responding methods, devices and systems and terminal | |
| WO2022252717A1 (en) | Homogeneous-heterogeneous hybrid multi-core chip architecture for implementing electric power data processing | |
| CN109660537A (en) | A method of real time monitoring and maintenance cloud platform physical resource service operation state | |
| CN104092581A (en) | Quick message processing method in message recording and analyzing system of intelligent substation | |
| CN103199968A (en) | Sending method of messages and system | |
| CN103647706A (en) | Self-adaptation protocol communication gateway and communication platform | |
| CN102711139A (en) | Data flow counting method and terminal | |
| CN111294235A (en) | Data processing method, device, gateway and readable storage medium | |
| CN103067388A (en) | Cloud computing foundation framework resource automatization method and system | |
| CN104142942A (en) | Mass heterogeneous data accessing and standardization method | |
| Terrazas et al. | A cloud-based framework for shop floor big data management and elastic computing analytics | |
| CN105022627A (en) | Communication method among Views in Android system | |
| CN107248981A (en) | Internet of Things intelligent equipment protocol storehouse | |
| CN110851252A (en) | Protocol conversion equipment and protocol conversion method based on TSN (traffic service network) architecture | |
| Simoncelli et al. | Stream-monitoring with blockmon: convergence of network measurements and data analytics platforms | |
| CN107070862B (en) | Data distribution method, data distribution device and the gateway system of gateway |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20141112 |
|
| WD01 | Invention patent application deemed withdrawn after publication |