CN113542249A - Multi-protocol data issuing system - Google Patents

Multi-protocol data issuing system Download PDF

Info

Publication number
CN113542249A
CN113542249A CN202110771035.5A CN202110771035A CN113542249A CN 113542249 A CN113542249 A CN 113542249A CN 202110771035 A CN202110771035 A CN 202110771035A CN 113542249 A CN113542249 A CN 113542249A
Authority
CN
China
Prior art keywords
module
file
data
management module
protocol
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110771035.5A
Other languages
Chinese (zh)
Inventor
李冬萍
杨迎春
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kunming University
Original Assignee
Kunming University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kunming University filed Critical Kunming University
Priority to CN202110771035.5A priority Critical patent/CN113542249A/en
Publication of CN113542249A publication Critical patent/CN113542249A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/18Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/48Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806Task transfer initiation or dispatching
    • G06F9/4843Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08Configuration management of networks or network elements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/12Applying verification of the received information
    • H04L63/123Applying verification of the received information received data contents, e.g. message integrity
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/161Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
    • H04L69/162Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields involving adaptations of sockets based mechanisms

Abstract

The invention discloses a multi-protocol data release system which comprises an operation control module, a thread management module, a configuration file management module, a file release module rule file identification module, a file name generation module, a log management module and a database operation module. Through scientific and reasonable system design, the invention not only realizes the correct and complete release of the data file as required, but also can dynamically manage a plurality of different projects at the same time, realizes an important capability in data life cycle management, establishes a unified and standard data release management mode, improves the data quality and normalization, has high intelligent degree and is worthy of popularization and use.

Description

Multi-protocol data issuing system
Technical Field
The invention relates to the technical field of computer communication, in particular to a multi-protocol data issuing system.
Background
In the world, digital transformation of various industries is a great trend, a new foundation of the digital transformation is core infrastructure construction which is developed around data acquisition, transmission, storage and calculation, data information acquired from a data source enters a system and is stored in a file, the data information needs to be sent to hosts of different processing links for processing, target hosts have different transmission protocols, the data volume is very large, the operation steps are different, and a multi-protocol data distribution system is needed for management.
Disclosure of Invention
The invention aims to solve the defects in the prior art and provide a multi-protocol data distribution system which can support different protocols to be connected with a target server and well realize the transmission capability of a data file.
A multi-protocol data distribution system comprising: the system comprises an operation control module, a thread management module, a configuration file management module, a file release module, a rule file identification module, a file name generation module, a log management module and a database operation module;
the operation control module is used for managing the starting, stopping and inquiring of the system;
the thread management module is used for starting and exiting the project thread by receiving an instruction sent by the operation control module, the started thread runs in a system memory, and communication management is performed through a socket;
a configuration sub item of the configuration file management module is called an item, the running content of the system dynamically increases and quits a corresponding data publishing thread according to the acquired configuration information and each item, and each file publishing module in the system is called to realize the publishing of the data file;
the file publishing module is used for establishing remote network connection with a target server through a corresponding protocol;
the rule file identification module is used for detecting whether a file needing to be issued exists in a directory specified by the current project or not and feeding back a detection result to the file issuing module;
the file name generating module is used for generating a file name according to the configuration information when the data file is issued to the remote server;
the log management module is used for recording the detailed condition of the system operation so as to facilitate maintenance personnel to check the operation condition;
the database operation module is used for writing a part of required information data into the database, and is convenient for query, operation and maintenance, monitoring and use.
Further, in the multi-protocol data distribution system as described above, the file distribution module includes: an FTP protocol file issuing unit and an SMB protocol file issuing unit;
further, in the multi-protocol data distribution system, the thread management module can match files required by the project under a required directory, so that the data files are distributed to a remote server.
Further, in the multi-protocol data distribution system, the file name generation module specifies the file name to be matched with the current item through a regular expression.
Further, in the multiprotocol data distribution system as described above, the information recorded by the log management module includes processing information and error information.
Further, in the multi-protocol data distribution system, the database operation module supports both ORACLE and MYSQL databases.
Has the advantages that:
through scientific and reasonable system design, the system and the remote host can interact according to a protocol supported by configuration, and the transmission capability of the data file in data operation life cycle management is realized. The project is dynamically managed in a multithreading parallel mode, one of FTP and SMB protocols can be selected according to the actual situation of a remote server to realize the function of sending data files to a specified directory of the remote server as required, the functions of intelligent file identification, data safety guarantee and audit verification are provided, the data source and transmission mode of each project host are managed in a configurable mode, the personalized process control is realized, and the operation and maintenance are very convenient and intelligent. According to the development idea of software modularization, the design of each functional module is completed, and the reusability and maintainability of the program are improved.
Drawings
FIG. 1 is a block diagram of the system of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention are described clearly and completely below, and it is obvious that the described embodiments are some, not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, the multi-protocol data distribution system provided by the present invention includes an operation control module, a thread management module, a configuration file management module, an FTP protocol file distribution module, an SMB protocol file distribution module, a rule file identification module, a file name generation module, a log management module, and a database operation module;
the operation control module is a script set for starting and quitting the system and is used for starting and quitting a main thread or starting and quitting a thread of a specified item; after the main thread is started, the main thread runs in a system background, socket monitoring is started, a client sends out operations such as quitting and inquiring, the communication with a server is carried out through the socket, and instructions and result information are transmitted;
the thread management module is a core for system operation, and the system generates a sub-thread for each configured project to complete a corresponding data distribution task by adopting a multi-thread concurrent mode. The distribution thread is responsible for the work related to specific transmission, and the main thread is responsible for tasks such as system management and information reporting. The main thread interacts with the sub-thread through a global data structure (configuration information, state information, control information);
the configuration file management module provides project configuration information and an access method required by program operation, and audits and verifies the integrity of the configuration information, wherein the configuration file is transfer _ conf.ini and is placed in a conf directory of a directory where the program is located. The first section [ COMMON ] of the configuration file can not be defaulted, the section stores information and program information which are COMMON to all transmissions, the profile section is configured below [ COMMON ], each section corresponds to the name of a batch of items and is separated by commas, then each transmission information is configured below the section corresponding to [ ], and the case and case are configured and distinguished;
configuration content example:
Figure BDA0003153407050000041
Figure BDA0003153407050000051
the FTP protocol file publishing module is characterized in that the remote end is an FTP server, the protocol type is configured in the configuration file as FTP, and the FTP protocol file publishing module is configured according to the user name, password and remotedir set by the configuration FTP server.
And calling the bottom layer method and the interface provided by the FTPClient by the program to realize the data file publishing function in the FTP mode.
The method mainly comprises the following steps:
(1) and (4) connect: connecting to the FTP server through a specified IP address and a specified port;
(2) logic: setting a user name and a password for logging in the FTP server;
(3) ispotitivecompletion: detecting whether a connection is successful
(4) disconnect remote server link
(5) Switching between changeWorkingDirectory and appointed directory on remote server
(6) A storeFile: retrieving the named file from the server and writing it to the given OutputStream;
(7) rnfr, rnto: renaming a file on a remote FTP server, the two commands together completing the renaming of the file
The SMB protocol file publishing module can support the remote end as an SMB protocol server. And configuring protocol type as SMB in the configuration file, and configuring user name, password and remotedir set by the SMB server.
And the program calls a JCIFS packet to realize SMB protocol file release, and the client is widely used for production on a large intranet.
The method mainly comprises the following steps:
(1) SmbFile (RemoteUrl): initializing classes by a construction method, wherein RemoteUrl parameters comprise a remote host address, a user name, a password and a shared directory;
description of the drawings: remoteUrl ═ smb:// "+ userName +": "+ passswerd +" @ "+ host + remotedDir;
SmbFile (RemoteUrl) initializes the class through a construction method, the RemoteUrl parameter adds a file name to generate an SmbFile object containing a remote file name for the SmbFileOutputStream output stream object to use;
(2) SmbFileOutputStream (remoteFile): uploading files to a shared directory;
(3) listFiles: the SmbFile object lists the files under the remote directory.
The rule file identification module intelligently identifies files needing to be issued in terms by configuring regular expressions in the configuration files, and can match the files needing to be issued according to the regular expressions by configuring localfilterule items in the configuration files.
When the file name generation module issues the data file to the remote server, the stored file needs to have a name, the file name is not necessarily consistent with the local source file name, and the file name can be referred to as required. The module generates a file name according to configuration information, wherein the configuration information comprises two fields, remotefilerly and localfierlle, and the file name which needs to be matched by the system is determined through a regular expression. localfilterule indicates which files need to be uploaded to a remote server under a locally specified directory, and is described by a regular expression, for example: ^ GG1 \ \ d {10}. BIL $.
remotefileresult represents what the filename of a file fetched by the native machine to the remote server directory is described by the algorithm rules defined by the present system plus a set of regular expressions, for example: LINUX _ @ \ GG1 \ \ d {10}. BIL > _ yyyyMMddHHmmss >
Any one character representation matches any corresponding character, e.g., a match a, 7 match 7, _ match _.
{ n } denotes n occurrences; \ \ d represents a number, \ \ turn a meaning, represents \; ^ represents the beginning; and $ represents the end.
< yyyyMMddHHmms > indicates that a string of current time is generated, a timestamp can be added to the file name, and the system also defines timestamp formats of other formats.
The other < > containing part in remotefileresult is a regular formula string description taken from localfilterresult, which represents the character string in the source file name
The database operation module supports ORACLE and MYSQL databases, and writes the file information which is successfully transmitted and the information which is reported by mistake in the processing process into the database, thereby facilitating query and operation and maintenance monitoring. The database is configured through keys such as driverUrl, url, userName, and passWord under section [ database ] in the configuration file transfer _ conf. And operating the database in a JDBC mode.
For the processing log of the system in the running process, the log module is written into the log file, so that the program can be conveniently checked and used during debugging, operation and maintenance.
Through scientific and reasonable system design, the invention not only realizes the correct and complete release of the data file as required, but also can dynamically manage a plurality of different projects at the same time, realizes an important capability in data life cycle management, establishes a unified and standard data release management mode, improves the data quality and normalization, has high intelligent degree and is worthy of popularization and use.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (6)

1. A multi-protocol data distribution system, comprising: the system comprises an operation control module, a thread management module, a configuration file management module, a file release module, a rule file identification module, a file name generation module, a log management module and a database operation module;
the operation control module is used for managing the starting, stopping and inquiring of the system;
the thread management module is used for starting and exiting the project thread by receiving an instruction sent by the operation control module, the started thread runs in a system memory, and communication management is performed through a socket;
a configuration sub item of the configuration file management module is called an item, the running content of the system dynamically increases and quits a corresponding data publishing thread according to the acquired configuration information and each item, and each file publishing module in the system is called to realize the publishing of the data file;
the file publishing module is used for establishing remote network connection with a target server through a corresponding protocol;
the rule file identification module is used for detecting whether a file needing to be issued exists in a directory specified by the current project or not and feeding back a detection result to the file issuing module;
the file name generating module is used for generating a file name according to the configuration information when the data file is issued to the remote server;
the log management module is used for recording the detailed condition of the system operation so as to facilitate maintenance personnel to check the operation condition;
the database operation module is used for writing a part of required information data into the database, and is convenient for query, operation and maintenance, monitoring and use.
2. The multi-protocol data distribution system of claim 1, wherein the file distribution module comprises: an FTP protocol file issuing unit and an SMB protocol file issuing unit.
3. The multi-protocol data distribution system of claim 1, wherein the thread management module is capable of matching files required by the project under the required directory to enable distribution of the data files to the remote server.
4. The multi-protocol data distribution system according to claim 1, wherein in the file name generation module, the configuration information includes two fields, remotefilerly and localfillerule, and the file name of the current item to be matched is specified by a regular expression.
5. The multi-protocol data distribution system of claim 1, wherein the information recorded by the log management module includes process information and error information.
6. The multi-protocol data distribution system of claim 1, wherein the database operation module supports both ORACLE and MYSQL databases.
CN202110771035.5A 2021-07-08 2021-07-08 Multi-protocol data issuing system Pending CN113542249A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110771035.5A CN113542249A (en) 2021-07-08 2021-07-08 Multi-protocol data issuing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110771035.5A CN113542249A (en) 2021-07-08 2021-07-08 Multi-protocol data issuing system

Publications (1)

Publication Number Publication Date
CN113542249A true CN113542249A (en) 2021-10-22

Family

ID=78127082

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110771035.5A Pending CN113542249A (en) 2021-07-08 2021-07-08 Multi-protocol data issuing system

Country Status (1)

Country Link
CN (1) CN113542249A (en)

Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101635640A (en) * 2009-09-04 2010-01-27 江苏天智互联科技有限公司 Method for automatically releasing terminal program version of WEB network station system server
CN103941675A (en) * 2014-03-27 2014-07-23 北京卓越经纬测控技术有限公司 Safety monitoring management system based on wireless network
CN104125235A (en) * 2014-08-11 2014-10-29 浪潮电子信息产业股份有限公司 Integrated multi-protocol storage method
CN104426880A (en) * 2013-09-03 2015-03-18 中国银联股份有限公司 Network-based centralized automatic file collection and distribution device
CN105262820A (en) * 2015-10-29 2016-01-20 浪潮电子信息产业股份有限公司 Cluster multi-computer mutual backup method based on Linux operating system
CN105607904A (en) * 2015-12-18 2016-05-25 南京熊猫电子股份有限公司 Linux platform-based rapid development framework of communication software and using method thereof
CN106060066A (en) * 2016-06-28 2016-10-26 积成电子股份有限公司 Embedded linux-based method for realizing terminal maintenance and information release
CN107688881A (en) * 2016-08-03 2018-02-13 中国铁道科学研究院铁道建筑研究所 A kind of span bridge construction monitoring management system based on BIM technology
CN108269071A (en) * 2018-03-15 2018-07-10 叶枫 The method that buildings model is shared cloud platform and project management is carried out using platform
CN109391644A (en) * 2017-08-03 2019-02-26 蓝盾信息安全技术有限公司 It is a kind of based on singly lead equipment intelligent file upload, downloading and administrative skill
CN111277425A (en) * 2018-12-05 2020-06-12 中国移动通信集团湖南有限公司 Centralized data transmission management device
CN111581175A (en) * 2020-04-29 2020-08-25 上海爱数信息技术股份有限公司 File storage gateway system and data migration method thereof
CN112039904A (en) * 2020-09-03 2020-12-04 福州林科斯拉信息技术有限公司 Network traffic analysis and file extraction system and method
CN112070388A (en) * 2020-09-04 2020-12-11 南京三方化工设备监理有限公司 Petrochemical engineering supervision, inspection, detection and consultation management system
CN112835568A (en) * 2019-11-25 2021-05-25 北京京东尚科信息技术有限公司 Project construction method and device
CN112835747A (en) * 2021-02-24 2021-05-25 上海泰宇信息技术股份有限公司 Remote backup method without client

Patent Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101635640A (en) * 2009-09-04 2010-01-27 江苏天智互联科技有限公司 Method for automatically releasing terminal program version of WEB network station system server
CN104426880A (en) * 2013-09-03 2015-03-18 中国银联股份有限公司 Network-based centralized automatic file collection and distribution device
CN103941675A (en) * 2014-03-27 2014-07-23 北京卓越经纬测控技术有限公司 Safety monitoring management system based on wireless network
CN104125235A (en) * 2014-08-11 2014-10-29 浪潮电子信息产业股份有限公司 Integrated multi-protocol storage method
CN105262820A (en) * 2015-10-29 2016-01-20 浪潮电子信息产业股份有限公司 Cluster multi-computer mutual backup method based on Linux operating system
CN105607904A (en) * 2015-12-18 2016-05-25 南京熊猫电子股份有限公司 Linux platform-based rapid development framework of communication software and using method thereof
CN106060066A (en) * 2016-06-28 2016-10-26 积成电子股份有限公司 Embedded linux-based method for realizing terminal maintenance and information release
CN107688881A (en) * 2016-08-03 2018-02-13 中国铁道科学研究院铁道建筑研究所 A kind of span bridge construction monitoring management system based on BIM technology
CN109391644A (en) * 2017-08-03 2019-02-26 蓝盾信息安全技术有限公司 It is a kind of based on singly lead equipment intelligent file upload, downloading and administrative skill
CN108269071A (en) * 2018-03-15 2018-07-10 叶枫 The method that buildings model is shared cloud platform and project management is carried out using platform
CN111277425A (en) * 2018-12-05 2020-06-12 中国移动通信集团湖南有限公司 Centralized data transmission management device
CN112835568A (en) * 2019-11-25 2021-05-25 北京京东尚科信息技术有限公司 Project construction method and device
CN111581175A (en) * 2020-04-29 2020-08-25 上海爱数信息技术股份有限公司 File storage gateway system and data migration method thereof
CN112039904A (en) * 2020-09-03 2020-12-04 福州林科斯拉信息技术有限公司 Network traffic analysis and file extraction system and method
CN112070388A (en) * 2020-09-04 2020-12-11 南京三方化工设备监理有限公司 Petrochemical engineering supervision, inspection, detection and consultation management system
CN112835747A (en) * 2021-02-24 2021-05-25 上海泰宇信息技术股份有限公司 Remote backup method without client

Similar Documents

Publication Publication Date Title
US11055302B2 (en) Method and system for implementing target model configuration metadata for a log analytics system
US20210405629A1 (en) System and method for interoperable communication of an automation system component with multiple information sources
US9286368B2 (en) Linking framework for information technology management
CN112600891B (en) Information physical fusion-based edge cloud cooperative system and working method
CN106685691A (en) Efficient configuration update method and system of terminal device in Internet of things system
CN105074698A (en) Executing continuous event processing (CEP) queries in parallel
WO2016161381A1 (en) Method and system for implementing a log parser in a log analytics system
WO2014127628A1 (en) Firmware upgrade method and system
CN106164897A (en) For depositing the system and method for memory database
CN112988485A (en) Simulation test method and device for power Internet of things equipment
CN114564852B (en) Operation method of FMEA (failure mode and effects analysis) data node and electronic equipment
US20240045751A1 (en) Lighting Method and Apparatus Based on AMD Platform, Device and Readable Medium
CN113542249A (en) Multi-protocol data issuing system
CN108228417B (en) Internet of vehicles log processing method and device
CN113138794A (en) Management method and management system for code version
US20040068575A1 (en) Method and apparatus for achieving a high transfer rate with TCP protocols by using parallel transfers
CN112905676A (en) Data file importing method and device
CN116150273A (en) Data processing method, device, computer equipment and storage medium
CN104965917A (en) Industrial real-time database interface method
CN114385458A (en) Sensor monitoring method and device based on event bus model
de Martel et al. Machine Learning Platform: Deploying and Managing Models in the CERN Control System
CN115827009B (en) Method and system for deploying Ambari based on automation script
WO2023236806A1 (en) Data burning method and apparatus, and computer device and computer-readable storage medium
CN116886691B (en) File management control method, device, system and storage medium
US11953995B1 (en) Centralized data backup platform supporting multiple data environments

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20211022