CN115361455A - Data transmission and storage method and device and computer equipment - Google Patents
Data transmission and storage method and device and computer equipment Download PDFInfo
- Publication number
- CN115361455A CN115361455A CN202211005704.9A CN202211005704A CN115361455A CN 115361455 A CN115361455 A CN 115361455A CN 202211005704 A CN202211005704 A CN 202211005704A CN 115361455 A CN115361455 A CN 115361455A
- Authority
- CN
- China
- Prior art keywords
- data
- target data
- station
- packet
- transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 77
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000008521 reorganization Effects 0.000 claims abstract description 16
- 230000004044 response Effects 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims description 17
- 238000004458 analytical method Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000012795 verification Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 2
- 238000004891 communication Methods 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000013480 data collection Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 239000000523 sample Substances 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 101001094649 Homo sapiens Popeye domain-containing protein 3 Proteins 0.000 description 1
- 101000608234 Homo sapiens Pyrin domain-containing protein 5 Proteins 0.000 description 1
- 101000578693 Homo sapiens Target of rapamycin complex subunit LST8 Proteins 0.000 description 1
- 102100027802 Target of rapamycin complex subunit LST8 Human genes 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005684 electric field Effects 0.000 description 1
- 238000000855 fermentation Methods 0.000 description 1
- 230000004151 fermentation Effects 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 231100000279 safety data Toxicity 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/163—In-band adaptation of TCP data exchange; In-band control procedures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9057—Arrangements for supporting packet reassembly or resequencing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/083—Network architectures or network communication protocols for network security for authentication of entities using passwords
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/143—Termination or inactivation of sessions, e.g. event-controlled end of session
- H04L67/145—Termination or inactivation of sessions, e.g. event-controlled end of session avoiding end of session, e.g. keep-alive, heartbeats, resumption message or wake-up for inactive or interrupted session
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
- H04L69/162—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields involving adaptations of sockets based mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application relates to a data transmission and storage method, a data transmission and storage device and computer equipment, wherein the data transmission and storage method comprises the steps of obtaining a socket request sent by a station, establishing TCP connection with the station, and forming a transmission channel in long connection with the station; acquiring target data sent by the station on the transmission channel, and correspondingly receiving the acquired target data based on a load balancing server; determining whether the obtained target data is reorganized packet data; if so, splicing the reassembled packet data to generate the target data; storing the acquired target data based on a preset rule; through TCP transmission mode, can reduce extra data packet transmission, and use load balancing server to distribute, the central side can faster response, receives the reorganization package data of a plurality of split simultaneously to can realize that thousands of stations transmit data safety, fast to the central side, the central side is saved it.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a data transmission and storage method and apparatus, and a computer device.
Background
At present, with continuous innovation of new technologies, the application of the industrial internet is greatly improved, and the safety problem of the industrial internet is gradually highlighted. Industrial internet is widely used in industries such as industry, energy, traffic, water conservancy and the like, which means that once a security incident occurs in such industries, the fermentation is very easy to continue, and even has a great influence on personal safety, social development and national stability. Therefore, the method strengthens the monitoring of the national industrial internet log data, prevents the lawless persons attacking the industrial internet in time, and finds out clues and evidences of crime. And the method is particularly important in the initial step of establishing the security situation awareness, namely collecting reliable and secure situation awareness data.
However, in the current industrial internet, in the scene facing data collection, the scene requires a large amount of data concurrency, two parties need to interact in real time, a C/S (client/server) architecture is mostly adopted at present to upload collected data, and an http protocol is used, so that the method has the advantages of high development speed and convenience in use of the existing application level interface, but the method has the advantages of low transmission speed, less effective data transmission, poor transmission safety performance, real-time interaction implementation and high server pressure; the method is characterized in that a TCP mode is partially adopted, data are uploaded and collected through a user-defined TCP data protocol packaging and unpacking method, and although the method is safe, data transmission, accurate, reliable and early-sending, has no size limitation, the method has the problems of low efficiency, time consumption in connection, poor readability and the like; therefore, the prior art can not meet the requirements of collecting, processing and storing high-concurrency data to be processed by the industrial internet, and simultaneously the problem of ensuring the safety of data collection is solved.
Disclosure of Invention
Based on the above, the application provides a data transmission and storage method, a data transmission and storage device and computer equipment, so as to achieve the effects of collecting, processing and storing high-concurrency data in the scene of industrial internet safety data collection.
In a first aspect, a data transmission and storage method is provided, where the method includes: acquiring a socket request sent by a station, establishing TCP connection with the station, and forming a transmission channel in long connection with the station; acquiring target data sent by the station on the transmission channel, and correspondingly receiving the acquired target data based on a load balancing server; determining whether the obtained target data is reorganized packet data; if so, splicing the recombined packet data to generate the target data; and storing the acquired target data based on a preset rule.
Optionally, the acquiring a socket request sent by a station, establishing a TCP connection with the station, and before forming a transmission channel of a long connection with the station, further includes: sending unique identification account number and password information to each station; when a socket request sent by a station is received, the account and the password are identified and verified; and if the verification is passed, establishing a TCP connection with the station.
Optionally, before the acquiring the target data sent by the station on the transmission channel, the method further includes: establishing a heartbeat mechanism between the station and a central side; the station sends a detection packet to the central side at regular time based on a preset period, and starts an overtime timer at the same time; if the central side returns a response packet within a preset time, the connection is normal, and the overtime timer is deleted; if the central side does not return a response packet within the preset time, the connection is disconnected, and reconnection is needed.
Optionally, the obtaining target data sent by the station on the transmission channel, and correspondingly receiving the obtained target data based on a load balancing server includes: the station sends the encrypted target data to the load balancing server; the load balancing server receives the target data and then carries out transcoding analysis processing; and the load balancing server determines a corresponding server node according to a balancing strategy and sends the decrypted target data to the corresponding server node.
Optionally, the obtaining target data sent by the station on the transmission channel, and correspondingly receiving the obtained target data based on the load balancing server further includes: the station judges the size of the target data; if the size of the target data is larger than a preset value, splitting the target data into a plurality of small packet data with preset sizes, and sending all the small packet data and the MD5 values of the target data to the center side; and if the size of the target data is not larger than a preset value, performing data transmission on the target data according to the size of the original data.
Optionally, the determining whether the acquired target data is reassembly packet data; if so, splicing the reassembled packet data to generate the target data comprises: if the target data is determined to be the reorganization packet data, further determining whether a reorganization packet folder already exists, and if so, reading all small packet data under the reorganization packet folder; and if the data does not exist, creating a reorganization package folder, and storing the reorganization package data in the reorganization package folder.
Optionally, the reading all the small package data in the reassembled package folder specifically includes: adding the byte lengths of the data packets of all the small packet data in the recombined packet folder and the byte length of the data packet of the newly received small packet data, splicing all the small packet data to generate large packet data if the added byte length of the data packet is equal to the byte length of the data packet of the target data, calculating the MD5 value of the large packet data, and indicating that the transmission is successful if the MD5 value of the large packet data is consistent with the MD5 value of the target data sent by the field station.
Optionally, the storing the obtained target data based on a preset rule includes: classifying the acquired target data according to the marks carried by the target data; storing the classified target data in the log files of the corresponding center sides respectively; storing the target data stored in the log file in a storage medium through flash.
In a second aspect, there is provided a data transmission storage device, the device comprising: a first obtaining module: the system comprises a receiving station, a receiving station and a transmitting station, wherein the receiving station is used for acquiring a socket request sent by the station, establishing a TCP connection with the station and forming a transmission channel in long connection with the station; a second obtaining module: the system comprises a transmission channel, a load balancing server and a field station, wherein the transmission channel is used for transmitting target data to the field station; the determining module: the data processing device is used for determining whether the acquired target data is reorganized packet data; if so, splicing the reassembled packet data to generate the target data; a storage module: and storing the acquired target data based on a preset rule.
In a third aspect, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method when executing the computer program.
According to the technical content provided by the embodiment of the application, firstly, the station can transmit data only after being encrypted when sending the data, so that the safety of the data is ensured, meanwhile, a heartbeat mechanism is introduced between the station and the center side, the center side and the station monitor the safety state of the other side at any time, and if abnormity is found, the station actively disconnects with the other side; secondly, the TCP transmission mode is used, extra data packet transmission is reduced, the Nginx server is used for load balancing, the central side can respond faster, a large data packet is split into a plurality of small data packets when data are transmitted, the small data packets are transmitted respectively, and data channels cannot be blocked, so that the method is beneficial to thousands of stations to safely and quickly transmit the data to the central side and store the data.
Drawings
FIG. 1 is a diagram illustrating an exemplary embodiment of a data transfer storage method;
FIG. 2 is a flow chart illustrating a method for data transfer and storage according to an embodiment;
FIG. 3 is a block flow diagram illustrating a method for data transfer storage according to one embodiment;
fig. 4 is a schematic flow chart showing a new access station at the center side in one embodiment;
FIG. 5 is a schematic flow chart illustrating a station verification connection in one embodiment;
FIG. 6 is a flowchart illustrating the determination of reassembled packet data according to one embodiment;
FIG. 7 is a schematic block diagram illustrating a computer device in one embodiment.
Detailed Description
The present application will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
For ease of understanding, the system to which the present application is applied will first be described. The data transmission and storage method provided by the application can be applied to the system architecture shown in fig. 1. The system comprises: a user space file server 103 and a terminal device 101, wherein the terminal device 101 communicates with the user space file server 103 through a network. The user space file server 103 may be a file server based on NFSv3\ v4 protocol, operating in Linux environment, and NFS (network file system) is a network abstraction over a file system, which may allow a remote client operating on the terminal device 101 to access through a network in a similar manner to a local file system. The terminal device 101 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, etc., and the user space file server 103 may be implemented by an independent server or a server cluster composed of a plurality of servers.
Fig. 2 is a flowchart illustrating a data transmission and storage method according to an embodiment of the present application, where the method may be executed by a user space file server in the system shown in fig. 1. As shown in fig. 2 and 3, the method may include the following steps:
step 201: acquiring a socket request sent by a station, establishing a TCP connection with the station, and forming a transmission channel in long connection with the station;
in this embodiment, it should be noted that a station refers to a general term for defining a management range according to region division, for example: dunhuang photovoltaic power station, chongli wind farm, etc., certainly not limited to electric field, power station can be any new type of industrial economy infrastructure with new type of integration of industrial internet. The central side is a central server or a central server cluster with functions of storing, ETL data processing, chart displaying, task issuing, software updating, strategy updating and the like, and is mainly used for receiving and processing requests sent by a plurality of stations. Broadly speaking, the station described in this embodiment may be a client, and the central side may be a server.
When data needs to be transmitted, the field station firstly sends a socket request, and after receiving the request, the central side establishes a TCP (Transmission Control Protocol) connection with the field station, so that a long-connection Transmission channel is formed between the central side and the field station.
Step 203: and acquiring target data sent by the station on the transmission channel, and correspondingly receiving the acquired target data based on a load balancing server.
In this embodiment, it should be noted that the load balancing server is configured to deploy a Nginx server at the center side, where the Nginx is a high-performance HTTP and reverse proxy server, and is also an IMAP/POP3/SMTP server. The Nginx serves as a load balancing server, and can directly support the Rails and the PHP program to serve the outside inside and also can support the HTTP proxy server to serve the outside. And the plurality of stations transmit the target data to the load balancing server, and the load balancing server distributes the target data to the server nodes in the center side for receiving according to the polling rules.
Referring to fig. 4, when the load balancing server receives a new station link from the listening port, it immediately executes the routing scheduling algorithm to obtain the service ip specified to be connected, and then creates a new upstream connection to connect to the server node specified by the center side, thereby completing the TCP connection between the station and the center side.
In this embodiment, a Nginx server is used as a load balancing server, the center side can respond faster, and send connection requests at thousands of stations at the same time, and the center side can dynamically realize flexible connection with station configuration according to the load of each server node of the current center side through effective connection configuration of the Nginx server, thereby achieving the purpose of load sharing. Preferably, in this embodiment, the load balancing server using the nginnx server allows a group of services monitoring the TCP connection to be configured, and also allows a plurality of TCP connections to be configured, so as to relieve the pressure on the central side from the site.
Referring to fig. 2, step 205: determining whether the obtained target data is reorganized packet data; if so, splicing the reassembled packet data to generate the target data;
in this embodiment, it should be noted that when the central side receives the target data sent by the station, it is further required to determine whether the received target data is reassembly packet data; if the packet data is recombined, all the received data can be spliced to generate target data, so that the data can be quickly transmitted without causing the blockage of a transmission channel.
Referring to fig. 2 and 3, step 207: storing the acquired target data based on a preset rule;
in this embodiment, it should be noted that in the embodiment of the present application, a TCP connection with a station is established by obtaining a socket request sent by the station, so as to form a transmission channel connected with the station, after the station establishes the transmission channel with a central side, the station first sends target data to a load balancing server, and the load balancing server distributes the target data to a server node in the central side according to a polling rule for receiving. The target data transmitted by thousands of stations are distributed and received in a balanced manner through the load balancing server based on the bearing state of the server node in the center side, so that the high-efficiency transmission of the data between the stations and the center side is realized, meanwhile, due to the arrangement of the load balancing server, the stations and the center side are not in direct communication, namely, the information of the center side is hidden, and the safe communication is further realized.
In this embodiment, it should be further noted that, after receiving the target data, the center side backs up the local files according to the category of the target data and performs classified storage.
Referring to fig. 2 and 5, in some embodiments, in step 201, before receiving a socket request sent by a station, establishing a TCP connection with the station, and forming a transmission channel of a long connection with the station, the method further includes:
step 301: sending unique identification account number and password information to each station;
step 303: identifying and verifying the account number and the password when receiving a socket request sent by a station;
step 305: and if the verification is passed, establishing a TCP connection with the station.
In this embodiment, it should be noted that before establishing a TCP connection, the central side configures a uniform ip and port for each station in advance, and each station can establish a connection with the central side through a socket. Meanwhile, in order to facilitate the later-stage monitoring of the transmission channel, the center side allocates unique account numbers and passwords to each station in advance, the stations can transmit the corresponding account numbers and the corresponding passwords to the center side after the stations are preliminarily connected with the center side through unified ip and ports which are configured on the center side in advance, and the TCP connection with the center side is finally realized after the stations are verified by the center side. In this embodiment, the account and the password are in one-to-one correspondence with the site, and cannot be changed without authorization, and if the account and the password information are changed, the authentication will fail at the center side, resulting in a failure in connection establishment.
In some embodiments, to improve processing efficiency, a message queue is introduced, wherein the message queue is a linked list of messages that can be treated as a record, having a specific format and a specific priority. A process having write permission on a message queue can add a new message to the message queue according to a certain rule; a process that has read permission for a message queue may then read the message from the message queue. Therefore, the station can firstly push target data to be sent to the message queue, and then the timing task system acquires the target data from the message queue within a specific time period. Message queues are important components in distributed systems, and are mainly used for improving system performance and peak clipping and reducing system coupling through asynchronous processing.
Referring to fig. 2, in some embodiments, before acquiring the target data sent by the station on the transmission channel in step 203, a heartbeat mechanism is further established between the station and the central side, the station sends a probe packet to the central side every other preset period, and starts a timeout timer at the same time as the probe packet is sent. If the central side returns a response packet within the preset time, the connection is normal, and the overtime timer is deleted; if the central side does not return the response packet within the preset time, the connection is disconnected, and reconnection is needed. By establishing a heartbeat mechanism between the central side and the station, whether the station survives or not can be judged so as to ensure the safe transmission of data.
For convenience of understanding, the following description illustrates that the site sends a probe packet to the central side every 10 minutes while starting a timeout timer, the preset time is set to 60 seconds, and if the central side returns a response packet within 60 seconds, the site and the central side are kept connected; if the central side does not return the response packet within 60 seconds, the station is disconnected from the central side. Here, if the station is disconnected from the center side, a re-authentication connection may be attempted.
Referring to fig. 2 and 3, in some embodiments, in step 203, acquiring target data sent by the station on the transmission channel, and correspondingly receiving the acquired target data based on the load balancing server further includes:
the station encrypts the target data before sending the target data, wherein the target data is encrypted by using a national secret SM2 algorithm, and meanwhile, an encryption machine and a firewall are arranged at the center side, and the target data to be transmitted is further safely protected by using IPSec VPN (Internet Protocol Security Virtual Private Network) safety technology and the like. And the load balancing server receives the encrypted target data and then carries out transcoding analysis processing on the encrypted target data, determines a corresponding server node according to a balancing strategy and sends the decrypted target data to the corresponding server node.
In this embodiment, it should be noted that a network protocol connection is established between the site and the load balancing server deployed on the center side, that is, a TCP connection is established between the site and the load balancing server, after the connection is established, the load balancing server converts the virtual IP address and the destination IP address when distributing the service request, and then forwards the packet to the server node having the corresponding destination address through the route; then the station sends the request service which needs to be sent to the virtual IP address to a load balancing server, and the load balancing server receives the request of the station; the virtual IP address here refers to a public network IP address used by the server cluster on the center side when the station requests a service. After receiving a request of a station, a load balancing server selects a real server node responsible for responding and processing the request through the steps of matching a strategy, scheduling a load balancing algorithm and the like; after the real server node is selected, the data transmission can be carried out without directly establishing connection between the site and the server node in the real center side. And then the load balancing node device carries out transcoding analysis on the acquired target data and transmits the target data after analysis and decoding to a server node at the center side correspondingly connected with the site. The safety and the high efficiency of data transmission of the station and the center side are realized through the load balancing server.
In some embodiments, further comprising: the station judges the size of the target data, if the size of the target data is larger than a preset value, the target data is split into a plurality of small packet data with preset sizes, and all the small packet data and the MD5 value of the target data are sent to a center side; and if the file size of the target data is not larger than the preset value, the target data is subjected to data transmission according to the original file size. Here MD5 is equivalent to the ID of the file, its value being unique.
For example, if the preset value is 1024 bytes, the target data is 2048KB, and the target data is larger than 1024, the target data can be split into 2 1024 bytes, that is, the target data is split into 2 data fragments. After the station sequentially sends the two data segments, the center side calculates the MD5 value of the target data according to all the received data segments.
Referring to fig. 2 and 6, step 205: determining whether the obtained target data is reorganized packet data; if so, splicing the reassembled packet data to generate the target data specifically includes: when the central side receives the target data sent by the station and determines that the target data is the reorganized packet data; further determining whether a regrouping packet folder already exists, and if so, directly reading all packet data under the regrouping packet folder; if the small package data do not exist in the reorganization package folder, creating a reorganization package folder, and storing the small package data in the reorganization package folder; the data transmitted by the station are judged and identified repeatedly.
In this embodiment, it should be noted that, first, whether the packet data is reassembled is determined by obtaining a reassembly identifier in a message header of the target data, if so, the identifier of the reassembled packet data is equal to 1, if not, the reassembly identifier of the reassembled packet data is equal to 0, if it is determined that the packet data is reassembled, it is further determined whether a reassembled packet folder already exists, if so, it is determined whether a folder named as "field station identity identifier + reassembled packet identifier + byte length of a combined data packet" exists under the reassembled packet folder, if not, a new folder is created, named as "field station identity identifier + reassembled packet identifier + byte length of a combined data packet", and the packet data exists under the folder; if the data packet exists, reading all small packet data with the format of [ data packet identification + data packet byte length + data packet offset ] under the folder, taking out the data packet byte length inside, adding the data packet byte length to the data packet byte length of the newly received small packet data, comparing the data packet byte length obtained after the addition with the data packet byte length sum of the target data sent by the field station, if the data packet byte length is equal to the data packet byte length sum, taking out all small packet data in the folder, sequentially combining the small packet data according to the offset of the data packet to generate large packet data, indicating that the data recombination packet is uploaded, and deleting the folder; if the data is less than the preset threshold value, storing and receiving the packet data, naming the data packet identifier, the data packet byte length and the data packet offset, storing the data packet identifier, the data packet byte length and the data packet offset under the folder, and waiting for re-receiving the packet data for the next time for combination.
When all the small packet data are combined, the byte length of the data packet is verified to be equal, the combined large packet data needs to be verified, after all the small packet data are received by the isocenter side, the large packet data are combined and generated, the MD5 value of the large packet data is calculated, and if the calculated MD5 value is consistent with the MD5 value of the target data sent by the station, successful transmission is indicated, and storage is performed. If not, the transmission is not successful.
By dividing the target data into a plurality of packet data for transmission, the problem that the data transmission is not efficient and safe due to the fact that a transmission channel is occupied by one packet for a long time due to network fluctuation is further effectively avoided.
Referring to fig. 2, in some embodiments, in step 207, storing the obtained target data based on a preset rule further includes: classifying the received target data according to the marks of the target data, respectively storing the classified target data in the log files of the corresponding servers, and storing the target data stored in the log files in a storage medium through flash.
The received target data are classified according to the marks of the target data, and different data are stored in respective log files based on different data attributes.
In one example, the log file also sets the caching time, the file backup size and the function of automatic deletion when due so as not to occupy too large server resources. For example, after the acquired target data is stored in the corresponding log file, the time for storing the target data in the date file backup is seven days, and the cached target data is deleted after seven days, so that the server space is effectively released, and the service efficiency of the center side is improved.
In this embodiment, it should be noted that, when receiving target data, the center side checks the target data, mainly checks whether the target data conforms to data in a predetermined json (JavaScript Object notification) format, whether a garbled code exists, cannot be correctly analyzed, and the like, and if the check finds a problem, returns an error prompt message to the station, discards the target data, and prompts the station to resend the target data.
In this embodiment, it should be further noted that storing the target data stored in the log file in the centralized storage by using flash means storing the target data stored in the fixed directory in the centralized storage by using flash. The Flume refers to a distributed framework for collecting and aggregating event stream data, and can connect a plurality of nodes, collect initial data sources and store the initial data sources in a final storage system. In this case, the target data stored in the log folder from the center side is stored in an external medium such as HDFS (Hadoop Distributed File System), HBase, ES, kafka, or the like in the storage medium.
The present application also provides a data transmission and storage method and apparatus, where the apparatus may include: the device comprises a first acquisition module, a second acquisition module, a determination module and a storage module. The main functions of each component module are as follows:
the first acquisition module is used for acquiring a socket request sent by a station, establishing TCP connection with the station and forming a transmission channel in long connection with the station.
A second obtaining module, configured to obtain target data sent by the station on the transmission channel, and correspondingly receive the obtained target data based on a load balancing server;
a determination module: the target data is used for determining whether the acquired target data is reorganized packet data or not; if so, splicing the reassembled packet data to generate the target data;
and the storage module is used for storing the acquired target data based on a preset rule.
According to an embodiment of the present application, a computer device and a computer-readable storage medium are also provided.
Fig. 7 is a block diagram of a computer device according to an embodiment of the present application. Computer apparatus is intended to represent various forms of digital computers or mobile devices. Which may include desktop computers, laptop computers, workstations, personal digital assistants, servers, mainframe computers, and other suitable computers. The mobile device may include a tablet, smartphone, wearable device, and the like.
As shown in fig. 7, the apparatus 600 includes a calculation unit 601, a ROM 602, a RAM 603, a bus 604, and an input/output (I/O) interface 605, the calculation unit 601, the ROM 602, and the RAM 603 being connected to each other via the bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The calculation unit 601 may perform various processes in the method embodiments of the present application according to computer instructions stored in a Read Only Memory (ROM) 602 or computer instructions loaded from a storage unit 608 into a Random Access Memory (RAM) 603. The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. The computing unit 601 may include, but is not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. In some embodiments, the methods provided by embodiments of the present application may be implemented as a computer software program tangibly embodied in a computer-readable storage medium, such as storage unit 608.
The RAM 603 can also store various programs and data required for the operation of the device 600. Part or all of the computer program may be loaded and/or installed on the device 600 via the ROM 602 and/or the communication unit 609.
An input unit 606, an output unit 607, a storage unit 608 and a communication unit 609 in the device 600 may be connected to the I/O interface 605. The input unit 606 may be, for example, a keyboard, a mouse, a touch screen, a microphone, or the like; the output unit 607 may be, for example, a display, a speaker, an indicator lamp, or the like. The device 600 is capable of exchanging information, data, etc. with other devices via the communication unit 609.
It should be noted that the device may also include other components necessary to achieve proper operation. It may also contain only the components necessary to implement the solution of the present application and not necessarily all of the components shown in the figures.
Various implementations of the systems and techniques described here can be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof.
Computer instructions for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer instructions may be provided to the computing unit 601 such that the computer instructions, when executed by the computing unit 601, such as a processor, cause the steps involved in the method embodiments of the present application to be performed.
The computer-readable storage media provided herein may be tangible media that may contain, or store, computer instructions for performing various steps involved in method embodiments of the present application. The computer readable storage medium may include, but is not limited to, storage media in the form of electronic, magnetic, optical, electromagnetic, and so forth.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method for data transmission and storage, the method comprising:
acquiring a socket request sent by a station, establishing a TCP connection with the station, and forming a transmission channel in long connection with the station;
acquiring target data sent by the station on the transmission channel, and correspondingly receiving the acquired target data based on a load balancing server;
determining whether the obtained target data is reorganized packet data; if so, splicing the reassembled packet data to generate the target data;
and storing the acquired target data based on a preset rule.
2. The data transmission and storage method according to claim 1, wherein before acquiring a socket request sent by a station, establishing a TCP connection with the station, and forming a transmission channel of a long connection with the station, the method further comprises:
sending unique identification account number and password information to each station;
when a socket request sent by a station is received, the account and the password are identified and verified;
and if the verification is passed, establishing a TCP connection with the station.
3. The data transmission storage method according to claim 1,
before the acquiring the target data sent by the station on the transmission channel, the method further includes: establishing a heartbeat mechanism between the station and a central side;
the station sends a detection packet to the central side at regular time based on a preset period, and starts an overtime timer at the same time;
if the central side returns a response packet within a preset time, the connection is normal, and the overtime timer is deleted;
and if the central side does not return a response packet within the preset time, the connection is disconnected and needs to be reconnected.
4. The data transmission and storage method according to claim 3, wherein the obtaining target data sent by the station on the transmission channel and correspondingly receiving the obtained target data based on a load balancing server comprises:
the station sends the encrypted target data to the load balancing server;
the load balancing server receives the target data and then carries out transcoding analysis processing;
and the load balancing server determines a corresponding server node according to a balancing strategy and sends the decrypted target data to the corresponding server node.
5. The data transmission and storage method according to claim 1, wherein the acquiring target data sent by the station on the transmission channel, and correspondingly receiving the acquired target data based on a load balancing server comprises:
the station judges the size of the target data;
if the size of the target data is larger than a preset value, splitting the target data into a plurality of small packet data with preset sizes, and sending all the small packet data and the MD5 values of the target data to the center side;
and if the size of the target data is not larger than a preset value, the target data is subjected to data transmission according to the size of the original data.
6. The data transmission and storage method according to claim 5, wherein the determination is made whether the acquired target data is reassembly packet data; if so, splicing the reassembled packet data to generate the target data comprises:
if the target data is determined to be the reorganization packet data, further determining whether a reorganization packet folder already exists, and if so, reading all small packet data under the reorganization packet folder;
and if the data does not exist, creating a reorganization package folder, and storing the reorganization package data in the reorganization package folder.
7. The data transmission and storage method according to claim 6, wherein the reading all the small package data in the regrouping package folder specifically includes:
adding the byte lengths of the data packets of all the small packet data in the recombined packet folder and the byte length of the data packet of the newly received small packet data, splicing all the small packet data to generate large packet data if the added byte length of the data packet is equal to the byte length of the data packet of the target data, calculating the MD5 value of the large packet data, and indicating that the transmission is successful if the MD5 value of the large packet data is consistent with the MD5 value of the target data sent by the field station.
8. The data transmission and storage method according to claim 1, wherein the storing the acquired target data based on a preset rule includes:
classifying the acquired target data according to the label of the target data;
storing the classified target data in the log files of the corresponding center sides respectively; storing the target data stored in the log file in a storage medium through flash.
9. A data transmission storage device, the device comprising:
a first acquisition module: the system comprises a station, a data processing unit and a control unit, wherein the data processing unit is used for acquiring a socket request sent by the station, establishing TCP connection with the station and forming a transmission channel in long connection with the station;
a second obtaining module: the system comprises a transmission channel, a load balancing server and a field station, wherein the transmission channel is used for transmitting target data to the field station;
a determination module: the target data is used for determining whether the acquired target data is reorganized packet data or not; if so, splicing the recombined packet data to generate the target data;
a storage module: and the target data is stored based on a preset rule, wherein the preset rule is a rule for classifying the target data according to categories.
10. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 8 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211005704.9A CN115361455B (en) | 2022-08-22 | 2022-08-22 | Data transmission storage method and device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211005704.9A CN115361455B (en) | 2022-08-22 | 2022-08-22 | Data transmission storage method and device and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115361455A true CN115361455A (en) | 2022-11-18 |
| CN115361455B CN115361455B (en) | 2024-01-23 |
Family
ID=84001785
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211005704.9A Active CN115361455B (en) | 2022-08-22 | 2022-08-22 | Data transmission storage method and device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115361455B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116522472A (en) * | 2023-03-15 | 2023-08-01 | 中国船舶集团有限公司第七一九研究所 | Ship multi-specialty general data packet construction and transmission method |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015096025A1 (en) * | 2013-12-24 | 2015-07-02 | Telefonaktiebolaget L M Ericsson (Publ) | Methods and apparatus for load balancing in a network |
| CN106936796A (en) * | 2015-12-30 | 2017-07-07 | 华耀(中国)科技有限公司 | A kind of server load balancing method and apparatus based on SNI |
| CN107819828A (en) * | 2017-10-16 | 2018-03-20 | 平安科技（深圳）有限公司 | Data transmission method, device, computer equipment and storage medium |
| US20190053319A1 (en) * | 2017-08-10 | 2019-02-14 | Comcast Cable Communications, Llc | Activation of Grant-Free Transmission |
| CN109660639A (en) * | 2018-12-11 | 2019-04-19 | 深圳市网心科技有限公司 | A kind of data uploading method, equipment, system and medium |
| CN111083425A (en) * | 2019-11-25 | 2020-04-28 | 视联动力信息技术股份有限公司 | Video stream processing method and device, server, electronic equipment and storage medium |
| CN111163130A (en) * | 2019-12-06 | 2020-05-15 | 深圳智链物联科技有限公司 | Network service system and data transmission method thereof |
| CN112787404A (en) * | 2021-01-29 | 2021-05-11 | 国电电力内蒙古新能源开发有限公司 | Regional monitoring system based on intelligent operation and network security deep fusion |
| CN112866361A (en) * | 2021-01-06 | 2021-05-28 | 戴振卿 | Safe transmission method of industrial data |
| CN112947333A (en) * | 2021-02-05 | 2021-06-11 | 天津市普迅电力信息技术有限公司 | Socket long connection-based balanced load fragmentation method |
| US20210243676A1 (en) * | 2020-01-31 | 2021-08-05 | Dell Products, Lp | System and method of managing data connections to a communication network using tiered devices |
| CN113438499A (en) * | 2021-06-29 | 2021-09-24 | 北京奇艺世纪科技有限公司 | Data uploading and receiving method, device, equipment and storage medium |
| CN114489995A (en) * | 2022-02-15 | 2022-05-13 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
-
2022
- 2022-08-22 CN CN202211005704.9A patent/CN115361455B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015096025A1 (en) * | 2013-12-24 | 2015-07-02 | Telefonaktiebolaget L M Ericsson (Publ) | Methods and apparatus for load balancing in a network |
| CN106936796A (en) * | 2015-12-30 | 2017-07-07 | 华耀(中国)科技有限公司 | A kind of server load balancing method and apparatus based on SNI |
| US20190053319A1 (en) * | 2017-08-10 | 2019-02-14 | Comcast Cable Communications, Llc | Activation of Grant-Free Transmission |
| CN107819828A (en) * | 2017-10-16 | 2018-03-20 | 平安科技（深圳）有限公司 | Data transmission method, device, computer equipment and storage medium |
| CN109660639A (en) * | 2018-12-11 | 2019-04-19 | 深圳市网心科技有限公司 | A kind of data uploading method, equipment, system and medium |
| CN111083425A (en) * | 2019-11-25 | 2020-04-28 | 视联动力信息技术股份有限公司 | Video stream processing method and device, server, electronic equipment and storage medium |
| CN111163130A (en) * | 2019-12-06 | 2020-05-15 | 深圳智链物联科技有限公司 | Network service system and data transmission method thereof |
| US20210243676A1 (en) * | 2020-01-31 | 2021-08-05 | Dell Products, Lp | System and method of managing data connections to a communication network using tiered devices |
| CN112866361A (en) * | 2021-01-06 | 2021-05-28 | 戴振卿 | Safe transmission method of industrial data |
| CN112787404A (en) * | 2021-01-29 | 2021-05-11 | 国电电力内蒙古新能源开发有限公司 | Regional monitoring system based on intelligent operation and network security deep fusion |
| CN112947333A (en) * | 2021-02-05 | 2021-06-11 | 天津市普迅电力信息技术有限公司 | Socket long connection-based balanced load fragmentation method |
| CN113438499A (en) * | 2021-06-29 | 2021-09-24 | 北京奇艺世纪科技有限公司 | Data uploading and receiving method, device, equipment and storage medium |
| CN114489995A (en) * | 2022-02-15 | 2022-05-13 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
Non-Patent Citations (1)
| Title |
|---|
| 汪姗姗: "面向工业互联网的安全加密管控系统分析", 《科学与信息化》, pages 41 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116522472A (en) * | 2023-03-15 | 2023-08-01 | 中国船舶集团有限公司第七一九研究所 | Ship multi-specialty general data packet construction and transmission method |
| CN116522472B (en) * | 2023-03-15 | 2023-11-24 | 中国船舶集团有限公司第七一九研究所 | Ship multi-specialty general data packet construction and transmission method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115361455B (en) | 2024-01-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9246819B1 (en) | System and method for performing message-based load balancing | |
| JP6858749B2 (en) | Devices and methods for establishing connections in load balancing systems | |
| Armenia et al. | A flexible phasor data concentrator design leveraging existing software technologies | |
| US20140330982A1 (en) | Facilitating secure network traffic by an application delivery controller | |
| CN105991412B (en) | Information push method and device | |
| CN112468518B (en) | Access data processing method and device, storage medium and computer equipment | |
| CN112150141A (en) | Block chain consensus method, device and system | |
| CN106464596A (en) | Openflow communication method, system, controller, and service gateway | |
| KR20220074971A (en) | Blockchain-based data processing method, apparatus and device, and readable storage medium | |
| CN114679407A (en) | Link aggregation method, device, equipment and storage medium | |
| CN115361455B (en) | Data transmission storage method and device and computer equipment | |
| CN110545230B (en) | Method and device for forwarding VXLAN message | |
| CN108494848B (en) | Enterprise message pushing method based on MQTT | |
| CN110650097A (en) | Data broadcasting method and device and computer readable storage medium | |
| Vithanage et al. | A Secure corroboration protocol for internet of things (IoT) devices using MQTT version 5 and LDAP | |
| US11444882B2 (en) | Methods for dynamically controlling transmission control protocol push functionality and devices thereof | |
| CN110912904B (en) | Malicious device identification method and device, storage medium and computer device | |
| CN105100101A (en) | Method, apparatus and system based on SSL session | |
| US11044350B1 (en) | Methods for dynamically managing utilization of Nagle's algorithm in transmission control protocol (TCP) connections and devices thereof | |
| US20140108523A1 (en) | Connection sharing across entities in a distributed messaging system | |
| CN107104964B (en) | Network security terminal and use method | |
| Lin et al. | An efficient source authentication for multicast based on Merkle hash tree | |
| CN116781764A (en) | Long-connection task execution method and device and related equipment | |
| CN111935090B (en) | Big data transmission and persistence method and system for industrial intelligent Internet of things | |
| CN117041272B (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |