CN113507530A - Data forwarding method, related system and device and storage medium - Google Patents
Data forwarding method, related system and device and storage medium Download PDFInfo
- Publication number
- CN113507530A CN113507530A CN202111057281.0A CN202111057281A CN113507530A CN 113507530 A CN113507530 A CN 113507530A CN 202111057281 A CN202111057281 A CN 202111057281A CN 113507530 A CN113507530 A CN 113507530A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- picture
- message
- picture storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000004806 packaging method and process Methods 0.000 claims abstract description 68
- 238000012545 processing Methods 0.000 claims abstract description 31
- 238000004590 computer program Methods 0.000 claims description 7
- 238000004891 communication Methods 0.000 claims description 5
- 238000012544 monitoring process Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 abstract description 26
- 238000006243 chemical reaction Methods 0.000 abstract description 8
- 238000012856 packing Methods 0.000 description 13
- 238000005192 partition Methods 0.000 description 7
- 230000002093 peripheral effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/06—Message adaptation to terminal or network requirements
- H04L51/066—Format adaptation, e.g. format conversion or compression
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/07—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail characterised by the inclusion of specific contents
- H04L51/10—Multimedia information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data forwarding method, which comprises the steps of converting acquired original picture data into picture data in a Protobuf protocol format through a data front end and storing the picture data in a database server, wherein a first message server sends the picture data and a front-end picture storage message to a data packaging server; the data packaging server sends the picture storage file generated by packaging to a shared directory folder in the FTP server; the data back end sends the acquired picture storage file to an unpacking server; the unpacking server unpacks the image data; the second message server stores the picture data and generates a back-end picture storage message; and the application server acquires the back-end picture storage message and implements preset processing measures. The invention also provides a data forwarding system, a data forwarding method device and a computer readable storage medium. The technical scheme of the invention realizes the rapid cross-platform transmission of data and has high conversion efficiency of data transmission.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to a data forwarding method, a data forwarding system, a data forwarding device, and a computer-readable storage medium.
Background
With the wider application of camera monitoring, the image data acquired by the front-end intelligent camera is not only used in a peripheral network for acquiring data, but also provided for different professional video private network applications. The image data acquired by the front-end intelligent camera needs a set of complete forwarding platform, and the image data can be forwarded to the video private network from a peripheral network of acquired data. Only one piece of picture data is acquired by the front-end camera, and in order to receive the picture data simultaneously in a video private network and a peripheral network, a forwarding platform is required to realize cross-boundary data sharing, so that the high efficiency and the rapidness of data transmission of the forwarding platform are important indexes.
At present, a data forwarding platform of the related art adopts a JSON format to transmit picture data.
However, in the conventional data forwarding process, when there is a certain loss rate for receiving the picture data and the data size of the picture data is large and large, the performance is severely reduced by using the JSON format to transmit the picture data. Specifically, on the one hand: data are transmitted by using a JSON format, and the time efficiency and the space efficiency of data analysis are low; on the other hand: the security of data transmission by using the JSON format is low, the data transmission is easy to be attacked, and the reliability of data transmission is low.
Therefore, there is a need to provide a new method and related system and device to solve the above technical problems.
Disclosure of Invention
The present invention is to overcome the above technical problems, and provide a data forwarding method, a data forwarding system, a data forwarding device, and a computer-readable storage medium, which achieve fast cross-platform data transmission and high conversion efficiency of data transmission.
In order to achieve the above object, the present invention provides a data forwarding method, where the data forwarding method is applied to a data front end, and the data front end includes a database server, a first message server, a data packaging server, and an FTP server, which are sequentially connected by data, and the method includes the following steps:
s11, obtaining original picture data of an external image acquisition device, converting the original picture data into picture data in a Protobuf protocol format, and storing the converted picture data in the database server;
s12, calling the picture data currently stored in the database server out and sending the picture data to the first message server, wherein the first message server is a kafka message server;
s13, the first message server stores the received picture data, generates a front-end picture storage message corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage message to the data packaging server;
s14, the data packaging server packages the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server;
the picture storage file is used for acquiring the picture data at the data back end: the data back end comprises an unpacking server, a second message server and an application server which are sequentially connected in a data mode, the data back end acquires the picture storage file from a shared directory folder in the FTP server and sends the picture storage file to the unpacking server, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain picture data and sends the unpacked picture data to the second message server, the second message server is a kafka message server, the second message server stores the picture data, a back end picture storage message corresponding to the picture data is generated according to a preset second message rule, the application server acquires the back end picture storage message and implements a preset processing measure corresponding to the back end picture storage message according to the acquired back end picture storage message, the preset processing measures comprise the step of obtaining the picture data corresponding to the back-end picture storage message.
Preferably, in the step S11, the data front end further includes an interceptor, and the interceptor is configured to be in data connection with the external image capturing device, and monitor and acquire the original picture data generated by the external image capturing device.
Preferably, after acquiring the original picture data in step S11, the method further includes:
judging whether the original picture data is data generated by external image acquisition equipment in a preset list: if so, processing the original picture data into the picture data; if not, the interceptor continues to monitor and acquire the original picture data.
Preferably, the database server is a rocksDB database; and the data packaging server performs data packaging on a plurality of corresponding picture data and a plurality of corresponding picture storage messages and generates one picture storage file.
The invention also provides a data forwarding method, which is applied to a data back end, wherein the data back end comprises an unpacking server, a second message server and an application server which are sequentially connected in a data manner, and the method comprises the following steps:
s21, acquiring a picture storage file in a shared directory folder in the FTP server, and sending the acquired picture storage file to the unpacking server;
wherein the picture storage file is obtained from the data front end: the data front end comprises a database server, a first message server, a data packaging server and an FTP server which are sequentially connected in a data mode, the data front end acquires original picture data of external image acquisition equipment, converts the original picture data into picture data in a Protobuf protocol format, stores the converted picture data in the database server, extracts the picture data currently stored in the database server and sends the picture data to the first message server, the first message server is a kafka message server, stores the picture data, generates front-end picture storage information corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage information to the data packaging server, the data packaging server performs data packaging on the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server;
s22, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain the picture data, and sends the unpacked picture data to the second message server, wherein the second message server is a kafka message server;
s23, the second message server stores the picture data, and generates a back-end picture storage message corresponding to the picture data according to a preset second message rule;
and S24, the application server acquires the back-end picture storage message of the second message server, and implements preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise acquiring the picture data corresponding to the back-end picture storage message.
Preferably, in the step S21, the data backend acquires the picture storage file in the shared directory folder in the FTP server through an FTP protocol.
Preferably, the database server is a rocksDB database; and the picture storage file is generated by data packaging of a plurality of picture data and a plurality of picture storage messages which correspond to each other.
The invention also provides a data forwarding system, which comprises a data front end and a data back end in communication connection with the data front end,
the data front end comprises:
the interceptor is used for being in data connection with the external image acquisition equipment and monitoring and acquiring original picture data generated by the external image acquisition equipment;
the front-end processor is used for converting the original picture data into the picture data in a Protobuf protocol format;
the database server is used for storing the picture data sent by the front-end processor;
the first message server is used for storing the picture data which is called out from the database server and sent to the front-end processor, and generating a front-end picture storage message corresponding to the picture data according to a preset first message rule, wherein the first message server is a kafka message server;
the data packaging server is used for carrying out data packaging on the picture data and the front-end picture storage message sent by the first message server according to a preset packaging rule and generating a picture storage file;
the FTP server is used for storing the picture storage file sent by the data packaging server into a shared directory folder;
the interceptor is in data connection with the front-end processor, the front-end processor is in data connection with the database server and the first message server respectively, the first message server is in data connection with the data packaging server, and the data packaging server is in data connection with the FTP server;
the data backend comprises:
the back-end processor is used for acquiring the picture storage files in the shared directory folder in the FTP server;
the unpacking server is used for unpacking the picture storage file sent by the back-end processor according to a preset unpacking rule to obtain the picture data;
the second message server is used for storing the picture data sent by the unpacking server and generating a back-end picture storage message corresponding to the picture data according to a preset second message rule, and the second message server is a kafka message server;
the application server is used for acquiring the back-end picture storage message of the second message server and implementing preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise the acquisition of the picture data corresponding to the back-end picture storage message;
the back-end processor is in communication connection with the FTP server, the back-end processor is in data connection with the unpacking server, the unpacking server is in data connection with the second message server, and the second message server is in data connection with the application server.
The present invention also provides a data forwarding apparatus comprising a processor and a memory, wherein the processor is configured to read a program in the memory and execute the steps in the data forwarding method according to any one of the above.
The present invention also provides a computer readable storage medium storing a computer program comprising program instructions which, when executed by a processor, implement the steps in the data forwarding method as described in any one of the above.
Compared with the prior art, the data forwarding method comprises the following steps: s11, obtaining original picture data of an external image acquisition device, converting the original picture data into picture data in a Protobuf protocol format, and storing the converted picture data in the database server; s12, calling the picture data currently stored in the database server out and sending the picture data to the first message server, wherein the first message server is a kafka message server; s13, the first message server stores the received picture data, generates a front-end picture storage message corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage message to the data packaging server; s14, the data packaging server packages the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server; s21, acquiring a picture storage file in a shared directory folder in the FTP server, and sending the acquired picture storage file to the unpacking server; s22, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain the picture data, and sends the unpacked picture data to the second message server, wherein the second message server is a kafka message server; s23, the second message server stores the picture data, and generates a back-end picture storage message corresponding to the picture data according to a preset second message rule; and S24, the application server acquires the back-end picture storage message of the second message server, and implements preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise acquiring the picture data corresponding to the back-end picture storage message. The database server is a rocksDB database, and the rocksDB database is used for storing the picture data to realize efficient storage of the data, so that the data can be rapidly off the disk, and zero loss of the data is realized. The original picture data are converted into the picture data in the Protobuf protocol format, so that the proto protocol is used for transmitting data, the safety and the conversion efficiency of the data are greatly improved, and the time efficiency and the space efficiency are higher than those of the data transmitted in the JSON format. The kafka message server is used as a middleware, the picture storage file is sent to a shared directory folder in the FTP server, data transmission is achieved between the data back end and the data front end through the FTP, cross-boundary safe sharing of data is achieved, the throughput rate of the whole system is improved, and the data can be used in a cross-platform mode. Therefore, the data forwarding method, the data forwarding system, the data forwarding device and the computer readable storage medium of the invention realize the rapid cross-platform transmission of data and have high conversion efficiency of data transmission.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without inventive efforts, wherein,
FIG. 1 is a flow chart of a data forwarding method according to the present invention;
fig. 2 is a flow chart of step S11 of the data forwarding method of the present invention;
FIG. 3 is a flow chart of another data forwarding method of the present invention;
FIG. 4 is a block diagram of a modular architecture of the data forwarding system of the present invention;
fig. 5 is a schematic structural diagram of the data forwarding device of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "comprising" and "having," and any variations thereof, in the description and claims of this application and the description of the figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or the accompanying drawings are used for distinguishing between different objects and not for describing a particular order. Reference herein to "an embodiment or this implementation" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
The invention provides a data forwarding method. The data forwarding method is applied to a data front end.
The data front end is a peripheral network for collecting picture data. The data front end comprises a database server, a first message server, a data packaging server and an FTP server which are sequentially connected in a data mode.
Referring to fig. 1, fig. 1 is a flow chart of a data forwarding method according to the present invention.
The data forwarding method comprises the following steps:
s11, obtaining original picture data of an external image acquisition device, converting the original picture data into picture data in a Protobuf protocol format, and storing the converted picture data in the database server.
In the step S11, the data front end further includes an interceptor, and the interceptor is configured to be in data connection with the external image capturing device, and monitor and acquire the original picture data generated by the external image capturing device.
After acquiring the original picture data in the step S11, the method further includes:
judging whether the original picture data is data generated by external image acquisition equipment in a preset list: if so, processing the original picture data into the picture data; if not, the interceptor continues to monitor and acquire the original picture data.
Referring to fig. 2, fig. 2 is a flowchart illustrating step S11 of the data forwarding method according to the present invention. Specifically, the step S11 includes the following steps:
and step S111, acquiring original picture data of the external image acquisition equipment.
Step S112, determining whether the original picture data is data generated by an external image capturing device in a preset list:
if yes, go to step S113;
if not, returning to the step S111, and the interceptor continuously monitors and acquires the original picture data. When the original picture data is acquired at the next time, the data is ignored by the front end of the data and is not adopted.
And step S113, converting the original picture data into picture data in a Protobuf protocol format. The original picture data are converted into the picture data in the Protobuf protocol format, so that proto protocol transmission data are achieved, the safety and the conversion efficiency of the data are greatly improved, the time efficiency and the space efficiency are higher than those of JSON format transmission data, and specifically, the time efficiency and the space efficiency of the proto protocol transmission data are 3-5 times those of the JSON format transmission data.
And step S114, storing the converted picture data in the database server.
In this embodiment, the database server is a RocksDB database. And storing the picture data by adopting a RocksDB database to realize efficient storage of the data, so that the data can be quickly dropped and zero loss of the data can be realized.
And S12, calling the picture data currently stored in the database server out and sending the picture data to the first message server. The first message server is a kafka message server.
The Kafka message server is a high-throughput, distributed, publish-subscribe messaging system-based server that can build large-scale messaging systems on inexpensive PC servers using Kafka technology. Kafka in the Kafka message server classifies messages by topics (topic), the topics are independent of each other, each topic comprises one or more partitions (partitions), the number of the partitions can be dynamically modified, the Kafka ensures that the messages are ordered in one partition, and each message in the partition has a unique offset (offset). A partition may contain multiple copies of the partition at the same time: one leader (leader) copy and one or more follower (follower) copies, only the leader copy being responsible for the reception and transmission of messages, the remaining copies being responsible for maintaining synchronization with the leader (leader) copy, thereby achieving high availability.
And S13, the first message server stores the received picture data, generates a front-end picture storage message corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage message to the data packaging server.
Wherein the first message rule is a generation message rule commonly used by the kafka message server in the art.
And S14, the data packaging server performs data packaging on the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server.
Wherein, the packing rule is a packing rule commonly used by the data packing server in the field.
In this embodiment, the data packaging server performs data packaging on a plurality of corresponding picture data and a plurality of corresponding picture storage messages to generate one picture storage file.
The picture storage file is used for the data back end to acquire the picture data: the back end of the data is a video private network. The data back end comprises an unpacking server, a second message server and an application server which are connected in sequence. In this embodiment, the data backend includes a plurality of data backend. And obtaining the picture storage file from a shared directory folder in the FTP server and sending the picture storage file to the unpacking server. The kafka message server is used as a middleware, the picture storage file is sent to a shared directory folder in the FTP server, data transmission is achieved between the data back end and the data front end through the FTP, cross-boundary safe sharing of data is achieved, the throughput rate of the whole system is improved, and the data can be used in a cross-platform mode.
And the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain the picture data and sends the unpacked picture data to the second message server.
Wherein, the unpacking rule is an unpacking rule commonly used by the data packing server in the field.
The second message server is a kafka message server. And the second message server stores the picture data and generates a rear-end picture storage message corresponding to the picture data according to a preset second message rule.
Wherein the second message rule is a generation message rule commonly used by the kafka message server in the art.
And the application server acquires the back-end picture storage message and implements a preset processing measure corresponding to the back-end picture storage message according to the acquired back-end picture storage message. Wherein the preset processing measure includes acquiring the picture data corresponding to the back-end picture storage message.
The invention also provides a data forwarding method. The data forwarding method is applied to a data back end.
The back end of the data is a video private network. The data back end comprises an unpacking server, a second message server and an application server which are connected in sequence. In this embodiment, the data backend includes a plurality of data backend.
Referring to fig. 3, fig. 3 is a flow chart of another data forwarding method according to the present invention.
The data forwarding method comprises the following steps:
s21, obtaining the picture storage file in the shared directory folder in the FTP server, and sending the obtained picture storage file to the unpacking server.
In this embodiment, in the step S21, the data backend acquires the picture storage file in the shared directory folder in the FTP server through an FTP protocol.
Wherein the picture storage file is obtained from the data front end: the data front end is a peripheral network for collecting picture data. The data front end comprises a database server, a first message server, a data packaging server and an FTP server which are sequentially connected in a data mode. The method comprises the steps that original picture data of external image acquisition equipment at the front end of the data are converted into picture data in a Protobuf protocol format, the converted picture data are stored in a database server, and the picture data stored in the database server at present are called out and sent to a first message server.
In this embodiment, the database server is a RocksDB database. The database server is a rocksDB database, and the rocksDB database is used for storing the picture data to realize efficient storage of the data, so that the data can be rapidly off the disk, and zero loss of the data is realized.
In this embodiment, the original picture data is processed into the picture data in the Protobuf protocol format, so that proto protocol transmission data is realized, and the security and the conversion efficiency of the data are greatly improved, so that the time efficiency and the space efficiency are both higher than those of JSON format transmission data, and specifically, the time efficiency and the space efficiency of proto protocol transmission data are both 3-5 times that of JSON format transmission data.
The first message server is a kafka message server. The first message server stores the picture data, generates a front-end picture storage message corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage message to the data packaging server.
Wherein the first message rule is a generation message rule commonly used by the kafka message server in the art.
And the data packaging server performs data packaging on the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server.
Wherein, the packing rule is a packing rule commonly used by the data packing server in the field.
The kafka message server is used as a middleware, the picture storage file is sent to a shared directory folder in the FTP server, data transmission is achieved between the data back end and the data front end through the FTP, cross-boundary safe sharing of data is achieved, the throughput rate of the whole system is improved, and the data can be used in a cross-platform mode.
In this embodiment, the picture storage file is generated by data packaging of a plurality of picture data and a plurality of picture storage messages that correspond to each other. The picture storage file can realize the sorting and induction of data, and is favorable for displaying the data in a shared directory folder in the FTP server so as to be convenient for a user to find and select.
And S22, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain the picture data, and sends the unpacked picture data to the second message server. The second message server is a kafka message server.
Wherein, the unpacking rule is an unpacking rule commonly used by the data packing server in the field.
And S23, the second message server stores the picture data and generates a back-end picture storage message corresponding to the picture data according to a preset second message rule.
Wherein the second message rule is a generation message rule commonly used by the kafka message server in the art.
And S24, the application server acquires the back-end picture storage message of the second message server, and implements preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise acquiring the picture data corresponding to the back-end picture storage message.
The present invention also provides a data forwarding system 100.
Referring to fig. 4, fig. 4 is a block diagram of a module structure of the data forwarding system 100 according to the present invention.
The data forwarding system 100 includes a data front end 1 and a data back end 2 communicatively connected to the data front end 1.
The data front end 1 comprises:
the interceptor 11 is used for being in data connection with the external image acquisition equipment and monitoring and acquiring original picture data generated by the external image acquisition equipment;
the front-end processor 12 is configured to process the raw picture data into the picture data in the protocol format.
The database server 13 is configured to store the picture data sent by the front-end processor 12.
The first message server 14 is configured to store the picture data that the front-end processor 12 retrieves from the database server 13 and sends the picture data to, and generate a front-end picture storage message corresponding to the picture data according to a preset first message rule, where the first message server 14 is a kafka message server.
The data packing server 15 is configured to perform data packing on the picture data and the front-end picture storage message sent by the first message server 14 according to a preset packing rule, and generate a picture storage file.
The FTP server 16 is configured to store the picture storage files sent by the data packaging server 15 in a shared directory folder.
Wherein the interceptor 11 is in data connection with the front-end processor 12. The front-end processor 12 is in data connection with the database server 13 and the first message server 14, respectively. The first message server 14 is in data connection with the data packaging server 15. The data packaging server 15 is in data connection with the FTP server 16.
The data backend 2 comprises:
the back-end processor 21 is configured to obtain a picture storage file in a shared directory folder in the FTP server 16.
The unpacking server 22 is configured to unpack the picture storage file sent by the back-end processor 21 according to a preset unpacking rule to obtain the picture data.
The second message server 23 is configured to store the picture data sent by the unpacking server 22, and generate a back-end picture storage message corresponding to the picture data according to a preset second message rule, where the second message server 23 is a kafka message server.
The application server 24 is configured to acquire the back-end image storage message of the second message server 23, and implement a preset processing measure corresponding to the back-end image storage message according to the acquired back-end image storage message. The preset processing measures comprise the step of obtaining the picture data corresponding to the back-end picture storage message.
Wherein, the back-end processor 21 is connected with the FTP server 16 in communication. The back-end processor 21 is in data connection with the unpacking server 22. The unpacking server 22 is in data connection with the second message server 23. The second message server 23 is in data connection with the application server 24.
It should be noted that the first message rule, the second message rule, the packing rule, and the unpacking rule are all common rules in the art. The interceptor 11, the front-end processor 12, the database server 13, the first message server 14, the data packing server 15, the FTP server 16, the back-end processor 21, the unpacking server 22, the second message server 23, and the application server 24 are all modules and devices commonly used in the art, and specific models and indexes are selected according to specific use conditions, which are not described in detail herein.
The present invention also provides a data forwarding device 1000. Referring to fig. 5, fig. 5 is a schematic structural diagram of a data forwarding apparatus 1000 according to the present invention.
The data forwarding apparatus 1000 includes a processor 1001, a memory 1002, a network interface 1003, and a computer program stored on the memory 1002 and operable on the processor 1001, where the processor 1001 is configured to read the program in the memory 1002, and the processor 1001, when executing the computer program, implements the steps in the data forwarding method provided by the embodiment of the present invention. I.e. the processor 1001 performs the steps in the data forwarding method.
Specifically, the processor 1001 is configured to perform the following steps:
s11, obtaining original picture data of an external image acquisition device, converting the original picture data into picture data in a Protobuf protocol format, and storing the converted picture data in the database server.
The data front end also comprises an interceptor, and the interceptor is used for being in data connection with the external image acquisition equipment and monitoring and acquiring the original picture data generated by the external image acquisition equipment.
After the original picture data is obtained, the method further comprises the following steps:
judging whether the original picture data is data generated by external image acquisition equipment in a preset list: if so, processing the original picture data into the picture data; if not, the interceptor continues to monitor and acquire the original picture data.
The database server is a rocksDB database.
S12, calling the picture data currently stored in the database server and sending the picture data to the first message server, wherein the first message server is a kafka message server.
And S13, the first message server stores the received picture data, generates a front-end picture storage message corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage message to the data packaging server.
And S14, the data packaging server performs data packaging on the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server.
And the data packaging server performs data packaging on a plurality of corresponding picture data and a plurality of corresponding picture storage messages and generates one picture storage file.
S21, obtaining the picture storage file in the shared directory folder in the FTP server, and sending the obtained picture storage file to the unpacking server.
And the data back end acquires the picture storage file in the shared directory folder in the FTP server through an FTP protocol. And the picture storage file is generated by data packaging of a plurality of picture data and a plurality of picture storage messages which correspond to each other.
And S22, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain the picture data, and sends the unpacked picture data to the second message server. The second message server is a kafka message server.
And S23, the second message server stores the picture data and generates a back-end picture storage message corresponding to the picture data according to a preset second message rule.
And S24, the application server acquires the back-end picture storage message of the second message server, and implements preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise acquiring the picture data corresponding to the back-end picture storage message.
The data forwarding device 1000 provided in the embodiment of the present invention can implement each implementation manner in the data forwarding method embodiment, and has corresponding beneficial effects, and for avoiding repetition, details are not described here again.
It is noted that only 1001-1003 with components are shown in FIG. 5, but it is understood that not all of the illustrated components are required and that more or fewer components may alternatively be implemented. As will be understood by those skilled in the art, the data forwarding apparatus 1000 is an apparatus capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded apparatus, and the like.
The memory 1002 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 1002 may be an internal storage unit of the data forwarding device 1000, such as a hard disk or a memory of the data forwarding device 1000. In other embodiments, the memory 1002 may also be an external storage device of the data forwarding device 1000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the data forwarding device 1000. Of course, the memory 1002 may also include both internal and external memory units of the data forwarding device 1000. In this embodiment, the memory 1002 is generally used for storing an operating system installed in the data forwarding apparatus 1000 and various types of application software, such as program codes of a data forwarding method of the data forwarding apparatus 1000. The memory 1002 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 1001 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 1001 is generally configured to control the overall operation of the data forwarding device 1000. In this embodiment, the processor 1001 is configured to run a program code stored in the memory 1002 or process data, for example, a program code of a data forwarding method of the data forwarding apparatus 1000.
The network interface 1003 may include a wireless network interface or a wired network interface, and the network interface 1003 is generally used to establish a communication connection between the data forwarding device 1000 and other electronic devices.
The present invention further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, the computer program includes program instructions, and the program instructions, when executed by the processor 1001, implement the steps in the data forwarding method, and can achieve the same technical effects, and are not described herein again to avoid repetition.
It will be understood by those skilled in the art that all or part of the processes in the data forwarding method of the data forwarding apparatus 100 according to the embodiments may be implemented by a computer program to instruct related hardware, and the program may be stored in a computer-readable storage medium, and when executed, may include processes according to the embodiments of the methods. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The present embodiment mentioned in the examples of the present invention is for convenience of description. The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.
Compared with the prior art, the data forwarding method comprises the following steps: s11, obtaining original picture data of an external image acquisition device, converting the original picture data into picture data in a Protobuf protocol format, and storing the converted picture data in the database server; s12, calling the picture data currently stored in the database server out and sending the picture data to the first message server, wherein the first message server is a kafka message server; s13, the first message server stores the received picture data, generates a front-end picture storage message corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage message to the data packaging server; s14, the data packaging server packages the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server; s21, acquiring a picture storage file in a shared directory folder in the FTP server, and sending the acquired picture storage file to the unpacking server; s22, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain the picture data, and sends the unpacked picture data to the second message server, wherein the second message server is a kafka message server; s23, the second message server stores the picture data, and generates a back-end picture storage message corresponding to the picture data according to a preset second message rule; and S24, the application server acquires the back-end picture storage message of the second message server, and implements preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise acquiring the picture data corresponding to the back-end picture storage message. The database server is a rocksDB database, and the rocksDB database is used for storing the picture data to realize efficient storage of the data, so that the data can be rapidly off the disk, and zero loss of the data is realized. The original picture data are converted into the picture data in the Protobuf protocol format, so that the proto protocol is used for transmitting data, the safety and the conversion efficiency of the data are greatly improved, and the time efficiency and the space efficiency are higher than those of the data transmitted in the JSON format. The kafka message server is used as a middleware, the picture storage file is sent to a shared directory folder in the FTP server, data transmission is achieved between the data back end and the data front end through the FTP, cross-boundary safe sharing of data is achieved, the throughput rate of the whole system is improved, and the data can be used in a cross-platform mode. Therefore, the data forwarding method, the data forwarding system, the data forwarding device and the computer readable storage medium of the invention realize the rapid cross-platform transmission of data and have high conversion efficiency of data transmission.
While the foregoing is directed to embodiments of the present invention, it will be understood by those skilled in the art that various changes may be made without departing from the spirit and scope of the invention.
Claims (10)
1. A data forwarding method is applied to a data front end, wherein the data front end comprises a database server, a first message server, a data packaging server and an FTP server which are sequentially connected in a data mode, and the method comprises the following steps:
s11, obtaining original picture data of an external image acquisition device, converting the original picture data into picture data in a Protobuf protocol format, and storing the converted picture data in the database server;
s12, calling the picture data currently stored in the database server out and sending the picture data to the first message server, wherein the first message server is a kafka message server;
s13, the first message server stores the received picture data, generates a front-end picture storage message corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage message to the data packaging server;
s14, the data packaging server packages the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server;
the picture storage file is used for acquiring the picture data at the data back end: the data back end comprises an unpacking server, a second message server and an application server which are sequentially connected in a data mode, the data back end acquires the picture storage file from a shared directory folder in the FTP server and sends the picture storage file to the unpacking server, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain picture data and sends the unpacked picture data to the second message server, the second message server is a kafka message server, the second message server stores the picture data, a back end picture storage message corresponding to the picture data is generated according to a preset second message rule, the application server acquires the back end picture storage message and implements a preset processing measure corresponding to the back end picture storage message according to the acquired back end picture storage message, the preset processing measures comprise the step of obtaining the picture data corresponding to the back-end picture storage message.
2. The data forwarding method according to claim 1, wherein in the step S11, the data front end further includes an interceptor, and the interceptor is configured to be in data connection with the external image capturing device and monitor and acquire the original picture data generated by the external image capturing device.
3. The data forwarding method according to claim 2, wherein the step S11, after acquiring the original picture data, further comprises:
judging whether the original picture data is data generated by external image acquisition equipment in a preset list: if so, processing the original picture data into the picture data; if not, the interceptor continues to monitor and acquire the original picture data.
4. The data forwarding method of claim 1, wherein the database server is a rocksDB database; and the data packaging server performs data packaging on a plurality of corresponding picture data and a plurality of corresponding picture storage messages and generates one picture storage file.
5. A data forwarding method is characterized in that the data forwarding method is applied to a data back end, the data back end comprises an unpacking server, a second message server and an application server which are sequentially connected in a data mode, and the method comprises the following steps:
s21, acquiring a picture storage file in a shared directory folder in the FTP server, and sending the acquired picture storage file to the unpacking server;
wherein the picture storage file is obtained from the data front end: the data front end comprises a database server, a first message server, a data packaging server and an FTP server which are sequentially connected in a data mode, the data front end acquires original picture data of external image acquisition equipment, converts the original picture data into picture data in a Protobuf protocol format, stores the converted picture data in the database server, extracts the picture data currently stored in the database server and sends the picture data to the first message server, the first message server is a kafka message server, stores the picture data, generates front-end picture storage information corresponding to the picture data according to a preset first message rule, and sends the picture data and the front-end picture storage information to the data packaging server, the data packaging server performs data packaging on the received picture data and the front-end picture storage message according to a preset packaging rule to generate a picture storage file, and sends the picture storage file to a shared directory folder in the FTP server;
s22, the unpacking server unpacks the received picture storage file according to a preset unpacking rule to obtain the picture data, and sends the unpacked picture data to the second message server, wherein the second message server is a kafka message server;
s23, the second message server stores the picture data, and generates a back-end picture storage message corresponding to the picture data according to a preset second message rule;
and S24, the application server acquires the back-end picture storage message of the second message server, and implements preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise acquiring the picture data corresponding to the back-end picture storage message.
6. The data forwarding method according to claim 5, wherein in the step S21, the data backend obtains the picture storage file in the shared directory folder in the FTP server through an FTP protocol.
7. The data forwarding method of claim 5, wherein the database server is a rocksDB database; and the picture storage file is generated by data packaging of a plurality of picture data and a plurality of picture storage messages which correspond to each other.
8. A data forwarding system, the data forwarding system comprising a data front end and a data back end communicatively coupled to the data front end:
the data front end comprises:
the interceptor is used for being in data connection with the external image acquisition equipment and monitoring and acquiring original picture data generated by the external image acquisition equipment;
the front-end processor is used for converting the original picture data into the picture data in a Protobuf protocol format;
the database server is used for storing the picture data sent by the front-end processor;
the first message server is used for storing the picture data which is called out from the database server and sent to the front-end processor, and generating a front-end picture storage message corresponding to the picture data according to a preset first message rule, wherein the first message server is a kafka message server;
the data packaging server is used for carrying out data packaging on the picture data and the front-end picture storage message sent by the first message server according to a preset packaging rule and generating a picture storage file;
the FTP server is used for storing the picture storage file sent by the data packaging server into a shared directory folder;
the interceptor is in data connection with the front-end processor, the front-end processor is in data connection with the database server and the first message server respectively, the first message server is in data connection with the data packaging server, and the data packaging server is in data connection with the FTP server;
the data backend comprises:
the back-end processor is used for acquiring the picture storage files in the shared directory folder in the FTP server;
the unpacking server is used for unpacking the picture storage file sent by the back-end processor according to a preset unpacking rule to obtain the picture data;
the second message server is used for storing the picture data sent by the unpacking server and generating a back-end picture storage message corresponding to the picture data according to a preset second message rule, and the second message server is a kafka message server;
the application server is used for acquiring the back-end picture storage message of the second message server and implementing preset processing measures corresponding to the back-end picture storage message according to the acquired back-end picture storage message, wherein the preset processing measures comprise the acquisition of the picture data corresponding to the back-end picture storage message;
the back-end processor is in communication connection with the FTP server, the back-end processor is in data connection with the unpacking server, the unpacking server is in data connection with the second message server, and the second message server is in data connection with the application server.
9. A data forwarding device comprising a processor and a memory, the processor being configured to read a program in the memory, to perform a step in a data forwarding method according to any one of claims 1 to 4, or to perform a step in a data forwarding method according to any one of claims 5 to 7.
10. A computer-readable storage medium, characterized in that it stores a computer program comprising program instructions which, when executed by a processor, implement the steps in the data forwarding method according to any one of claims 1 to 4, or implement the steps in the data forwarding method according to any one of claims 5 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111057281.0A CN113507530B (en) | 2021-09-09 | 2021-09-09 | Data forwarding method, related system and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111057281.0A CN113507530B (en) | 2021-09-09 | 2021-09-09 | Data forwarding method, related system and device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113507530A true CN113507530A (en) | 2021-10-15 |
| CN113507530B CN113507530B (en) | 2022-01-04 |
Family
ID=78016942
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111057281.0A Active CN113507530B (en) | 2021-09-09 | 2021-09-09 | Data forwarding method, related system and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113507530B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115883658A (en) * | 2022-12-07 | 2023-03-31 | 长安大学 | Cross-platform image information transmission method, system, equipment and readable storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106973073A (en) * | 2016-01-13 | 2017-07-21 | 杭州海康威视系统技术有限公司 | The transmission method and equipment of multi-medium data |
| CN109308224A (en) * | 2017-07-27 | 2019-02-05 | 阿里巴巴集团控股有限公司 | The method, apparatus and system of cross-platform data communication, cross-platform data processing |
| CN110401724A (en) * | 2019-08-22 | 2019-11-01 | 北京旷视科技有限公司 | File management method, ftp server and storage medium |
| CN110505307A (en) * | 2019-08-30 | 2019-11-26 | 公安部交通管理科学研究所 | The exchange method and system of traffic flow data between a kind of net |
| CN110719332A (en) * | 2019-10-17 | 2020-01-21 | 北京旷视科技有限公司 | Data transmission method, device, system, computer equipment and storage medium |
| CN111290696A (en) * | 2020-02-26 | 2020-06-16 | 平安银行股份有限公司 | Flow control method and device for application program assembly |
| CN111694828A (en) * | 2020-06-08 | 2020-09-22 | 山东伏羲智库互联网研究院 | Data processing method, device, system and storage medium |
| CN112187898A (en) * | 2020-09-18 | 2021-01-05 | 佳都新太科技股份有限公司 | Data access system, method and device based on public security network |
| CN112187911A (en) * | 2020-09-23 | 2021-01-05 | 苏州科达科技股份有限公司 | Message transmission method, device, storage medium and system based on network isolation |
-
2021
- 2021-09-09 CN CN202111057281.0A patent/CN113507530B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106973073A (en) * | 2016-01-13 | 2017-07-21 | 杭州海康威视系统技术有限公司 | The transmission method and equipment of multi-medium data |
| CN109308224A (en) * | 2017-07-27 | 2019-02-05 | 阿里巴巴集团控股有限公司 | The method, apparatus and system of cross-platform data communication, cross-platform data processing |
| CN110401724A (en) * | 2019-08-22 | 2019-11-01 | 北京旷视科技有限公司 | File management method, ftp server and storage medium |
| CN110505307A (en) * | 2019-08-30 | 2019-11-26 | 公安部交通管理科学研究所 | The exchange method and system of traffic flow data between a kind of net |
| CN110719332A (en) * | 2019-10-17 | 2020-01-21 | 北京旷视科技有限公司 | Data transmission method, device, system, computer equipment and storage medium |
| CN111290696A (en) * | 2020-02-26 | 2020-06-16 | 平安银行股份有限公司 | Flow control method and device for application program assembly |
| CN111694828A (en) * | 2020-06-08 | 2020-09-22 | 山东伏羲智库互联网研究院 | Data processing method, device, system and storage medium |
| CN112187898A (en) * | 2020-09-18 | 2021-01-05 | 佳都新太科技股份有限公司 | Data access system, method and device based on public security network |
| CN112187911A (en) * | 2020-09-23 | 2021-01-05 | 苏州科达科技股份有限公司 | Message transmission method, device, storage medium and system based on network isolation |
Non-Patent Citations (2)
| Title |
|---|
| 丁雷: "无锡市智慧城市公共安全视频监控网络设计与建设", 《电视技术》 * |
| 甘冉等: "FTP协议剥落与重组技术在网闸系统中的应用", 《计算机安全》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115883658A (en) * | 2022-12-07 | 2023-03-31 | 长安大学 | Cross-platform image information transmission method, system, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113507530B (en) | 2022-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109547474A (en) | A kind of data transmission method and device | |
| CN112685248A (en) | Intelligent network card monitoring log obtaining method and device, electronic equipment and storage medium | |
| CN113872967B (en) | Vehicle information transmission method and device, electronic equipment and storage medium | |
| CN113835902A (en) | Data processing method and device, computer equipment and storage medium | |
| CN113507530B (en) | Data forwarding method, related system and device and storage medium | |
| CN108289165B (en) | Method and device for realizing camera control based on mobile phone and terminal equipment | |
| CN115344522B (en) | Message conversion channel, message conversion device, electronic equipment and exchange equipment | |
| CN108924773B (en) | Message processing method and device | |
| CN113608889B (en) | Message data processing method, device, equipment and storage medium | |
| CN111104119A (en) | MPI program conversion method and device for memory center type computer | |
| CN112783724B (en) | Log grabbing and debugging method, system, intelligent device and storage medium | |
| CN112511636B (en) | Data transmission system, method, device, computer equipment and storage medium | |
| CN113055350B (en) | Data transmission method, device, equipment and readable storage medium | |
| CN109995589B (en) | Log collection method and system | |
| CN111526075A (en) | Intelligent device control method, storage medium and electronic device | |
| CN113327302A (en) | Picture processing method and device, storage medium and electronic device | |
| CN112416843A (en) | Backboard communication equipment, control method thereof and storage medium | |
| CN112651212A (en) | Method and terminal for remotely acquiring word stock | |
| JP2739830B2 (en) | Data communication device for multiprocessor system | |
| CN111078477A (en) | Interface test method and device, storage medium and electronic equipment | |
| CN110266814A (en) | Transmission method and transmitting device | |
| CN116132214B (en) | Event transmission method, device, equipment and medium based on event bus model | |
| CN114745367B (en) | Data transmission method, device, electronic equipment and storage medium | |
| CN114513561B (en) | Information processing method, information processing device, electronic equipment and computer readable storage medium | |
| CN114362999B (en) | Data transmission method, system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PP01 | Preservation of patent right | ||
| PP01 | Preservation of patent right |
Effective date of registration: 20240109 Granted publication date: 20220104 |