CN113572854B - Data transmission method and system based on Kafka component - Google Patents
Data transmission method and system based on Kafka component Download PDFInfo
- Publication number
- CN113572854B CN113572854B CN202110914309.1A CN202110914309A CN113572854B CN 113572854 B CN113572854 B CN 113572854B CN 202110914309 A CN202110914309 A CN 202110914309A CN 113572854 B CN113572854 B CN 113572854B
- Authority
- CN
- China
- Prior art keywords
- key information
- data
- component
- original data
- theme
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 57
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 14
- 239000000284 extract Substances 0.000 claims abstract description 4
- 238000004458 analytical method Methods 0.000 claims description 9
- 238000004891 communication Methods 0.000 abstract description 9
- 238000000605 extraction Methods 0.000 abstract description 5
- 238000003860 storage Methods 0.000 description 8
- 238000012806 monitoring device Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000004904 shortening Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data transmission method and system based on a Kafka component, and relates to the technical field of communication. The method comprises the following steps: the data acquisition device acquires original data to be transmitted; the data processing device extracts key information of the original data; the Producer component sends the original data and the key information to different topics of a corresponding type of Broker component for caching according to the type of the original data; and the user terminal acquires key information from the corresponding theme of the Broker component by using the Consumer component, and judges whether to acquire the original data according to the key information. The application is suitable for high-efficiency high-quality transmission of a large amount of data, can meet the real-time requirement in large-scale data transmission, and the extraction of key information is equivalent to the high-efficiency simplification of the original data, meets the data quality requirement, and shortens the data transmission link to increase the stability of the system.
Description
Technical Field
The application relates to the technical field of communication, in particular to a data transmission method and system based on a Kafka component.
Background
With the development of experimental environments and higher performance sensors for large data volumes, large amounts of data need to be processed and transmitted.
Conventional data transmission systems are classified into two types, real-time data transmission and non-real-time data transmission. When the received data quantity exceeds the processing capacity, the traditional real-time data transmission system accumulates the buffer memory to temporarily store the data, so that packet loss or memory overflow is easy to cause, and the normal function of the system is affected; non-real-time transmission systems cannot fully utilize data because of data delays, which are not supported by many data applications.
Therefore, the existing data transmission method cannot meet the stable transmission of large-batch data.
Disclosure of Invention
The application aims to solve the technical problem of providing a data transmission method and system based on a Kafka component aiming at the defects of the prior art.
The technical scheme for solving the technical problems is as follows:
a data transmission method based on a Kafka assembly, the Kafka assembly comprising: a Producer component, a Broker component, and a Consumer component, the Broker component comprising at least one topic for each type of data, the data transmission method comprising:
the data acquisition device acquires original data to be transmitted;
the data processing device extracts key information of the original data;
the Producer component sends the original data and the key information to different topics of a corresponding type of Broker component for caching according to the type of the original data;
and the user terminal uses the Consumer component to acquire the key information from the corresponding theme of the Broker component, and judges whether to acquire the original data according to the key information.
The other technical scheme for solving the technical problems is as follows:
a data transmission system based on a Kafka assembly, the Kafka assembly comprising: a Producer component, a Broker component, and a Consumer component, the Broker component comprising at least one topic for each type of data, the data transmission system comprising:
the data acquisition device is used for acquiring the original data to be transmitted;
the data processing device is used for extracting key information of the original data;
the Producer component is used for sending the original data and the key information to different topics of a corresponding type of Broker component for caching according to the type of the original data;
and the user terminal is used for acquiring the key information from the corresponding theme of the Broker component by using the Consumer component, and judging whether to acquire the original data according to the key information.
The beneficial effects of the application are as follows: the data transmission method and the system provided by the application are suitable for high-efficiency and high-quality transmission of a large amount of data, the data quality requirement can be met by extracting key information of the data, which is equivalent to high-efficiency simplification of the original data, and then the data is classified and stored into different topics of Kafka Broker according to the data type.
Additional aspects of the application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the application.
Drawings
Fig. 1 is a schematic flow chart of a data transmission method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a data transmission system according to an embodiment of the present application.
Detailed Description
The principles and features of the present application are described below with reference to the drawings, the illustrated embodiments are provided for illustration only and are not intended to limit the scope of the present application.
Kafka is a distributed publish-subscribe messaging system written in the Scala language and later becomes part of the Apache project. The Kafka messaging system consists essentially of Producer, broker and Consumer. The Producer is responsible for generating messages to send to the Broker, which stores the messages temporarily and then forwards them to the Consumer Consumer, who receives and processes the messages.
Based on this, the application proposes a data transmission method based on Kafka components, which is further described below.
As shown in fig. 1, a flow is provided for an embodiment of a data transmission method according to the present application, which is implemented based on a Kafka module, the Kafka module including: the data transmission method comprises the following steps of a Producer component, a Broker component and a Consumer component, wherein the Broker component comprises at least one theme of each type of data, and the data transmission method comprises the following steps:
s1, acquiring original data to be transmitted by a data acquisition device;
it should be noted that, the raw data may be data from a sensor, a test device, a monitoring device, or other devices or instruments that need to be stored, and the data acquisition device needs to determine the IP and the port of the transmitting and receiving end, and collect specific information through a network.
For example, taking the vehicle monitoring information collected by the image monitoring device as an example, the raw data may include: the image monitoring device itself includes state information such as temperature and usage rate, type of the vehicle such as maneuvering or non-maneuvering, basic information of the vehicle such as license plate number and vehicle height, movement information of the vehicle such as speed and direction, appearance information of the vehicle such as color histogram, deep learning extraction feature and optical flow information, and position information of the vehicle such as coordinate position in the area.
Optionally, the data acquisition device may be implemented in a hardware or software manner, if the data acquisition device is implemented in a hardware manner, the processor chip, the wireless communication unit and the communication interface may be packaged into an independent acquisition device, where the processor chip performs the functions of receiving and processing data, and is connected with a communication interface of a device or an instrument such as a sensor, a test device, a monitoring device, etc. through the communication interface and a data line, so as to acquire data of the device or the instrument such as the sensor, the test device, the monitoring device, etc. or may also establish a wireless communication link with the device or the instrument such as the sensor, the test device, the monitoring device, etc. through the wireless communication unit by a wireless communication method such as WIFI, 4G/5G, NFC or Zigbee, etc. so as to acquire the data. If the method is implemented by software, corresponding programs can be written in the processors of devices or instruments such as sensors, test equipment, monitoring equipment and the like, and acquired data can be directly sent to the data processing device.
For example, the data acquisition device listens to a specific port (8888), and the sender sends data to the port under the condition of having authority, and the data received by the software is the original data.
It should be appreciated that the original data format must be known to the server side and be clearly distinguishable from the type for subsequent processing.
S2, extracting key information of original data by the data processing device;
note that the key information refers to attribute information of the data set in advance, and preferably may include a unique identifier and some necessary attribute information, for example, the key information may include an ID of the data, a sender, a transmission time, a reception time, a content size, and the like. And meanwhile, the specified type is allocated to the data according to the key information so as to be pushed to the Kafka component in the subsequent step.
Optionally, after the data processing device receives the original data, the following information may be extracted after parsing: the type of original data (information 1/information 2/.) and the identification (ID information in the original data is used or ID is formulated according to time + type), the transmission time and the reception time (time/, time minutes second) are 256 bytes in length, wherein ID should be consistent with the original data ID or have unique correlation according to time + type.
For example, the key information extracted therefrom includes the type of the data (pedestrian/motor vehicle/non-motor vehicle/.), identification (license plate number, pedestrian ID), transmission time and reception time (20×0×0×time-division second), length is in bytes.
Optionally, the data processing device may be adapted to UDP/TCP message receiving protocol, and may be accessed and successfully parsed to be valid original data.
By extracting key information of the data, the received data can be accurately and efficiently utilized, and the requirements of speed and accuracy are met.
S3, the Producer component sends the original data and the key information to different topics of the corresponding type of Broker component for caching according to the type of the original data;
it should be understood that the Kafka component needs to establish at least one Broker with different topics in advance, taking 4 topics as an example for illustration, and for each type of data, establish 4 topics, and after the data processing device extracts the key information, send the data and the key information to the Broker with the corresponding topic of the corresponding type of data respectively for storage.
It should be understood that the theme of the Broker component may be set according to actual requirements, for example, assuming that the user terminal may be a local terminal and a remote terminal, and both have requirements for retrieving data, the theme may be set to 4 themes, which are a local key information theme, a local original data theme, a remote key information theme, and a remote original data theme, respectively, and store the local key information for being retrieved by the local terminal, the local original data for being retrieved by the local terminal, the key information for being retrieved by the remote terminal, and the original data for being retrieved by the remote terminal, respectively.
For another example, if the data is only available for local retrieval, then the topics may also be set to 2, i.e., key information topics and raw data.
And S4, the user terminal uses the Consumer component to acquire key information from the corresponding theme of the Broker component, and judges whether to acquire the original data according to the key information.
Specifically, the user terminal uses the Consumer component to acquire key information from the corresponding theme of the Broker component, and if the content in the key information no longer meets the use requirement, the user can pull data from the original data cached in the Broker component for analysis.
According to the data types, the data can be classified and stored in different topics of the Broker component, and the Kafka has the characteristic of high throughput, can meet the production and consumption of millions of level information per second, and the Broker can persist the data without memory pressure, so that the Broker can be used as a data storage center with extremely strong real-time performance for caching the data. For data in different topics, key information data can be consumed in close to real time, original data content can be cached, and a user can configure the caching time of the original data in the Broker according to actual use requirements.
By using the Consumer component to pull key information from the corresponding theme of the Broker component, whether to pull the original data to the local is judged according to the actual demand and the key information, so that the storage pressure of the database is reduced, the data use efficiency is improved, meanwhile, the user can set the automatic data deletion rule in the Broker component, set the time for data deletion according to the demand, and for example, can set to automatically delete data exceeding 10 days.
The data transmission method provided by the embodiment is suitable for high-efficiency and high-quality transmission of a large amount of data, the key information of the data is extracted, the high-efficiency and simplified of the original data are equivalent, the data quality requirement can be met, the data is classified and stored into different topics of Kafka Broker according to the data type, the production and consumption of millions of messages per second can be met due to the fact that Kafka has the characteristic of high throughput, the data are superior to a traditional database, the Broker can last data without memory pressure, the data can be used as a data storage center with extremely high real-time performance, the data are cached, each application department can take according to the service requirement, the high throughput performance of Kafka can meet the real-time requirement of large-scale data transmission, the key information extraction is equivalent to the high-efficiency and simplified of the original data, the data quality requirement is met, and the shortening of a data transmission link also increases the stability of the system.
Optionally, in some possible embodiments, the Producer component sends the original data and the key information to different topics of the corresponding type of Broker component for caching according to the type of the original data, which specifically includes:
the Producer component sends the original data and the key information to 4 topics of the corresponding type of the Broker component for caching according to the type of the original data, wherein the 4 topics are respectively: the remote key information system comprises a local key information theme, a local original data theme, a remote key information theme and a remote original data theme, wherein the local key information theme and the remote key information theme are the same in storage content, the local key information theme is used for storing local-oriented key information, and the remote key information theme is used for storing remote-oriented key information; the local original data theme is used for storing the original data facing the local, and the remote original data theme is used for storing the original data facing the remote.
Optionally, in some possible embodiments, the user terminal uses the Consumer component to obtain key information from the corresponding theme of the Broker component, and determines whether to obtain the original data according to the key information, which specifically includes:
when the user terminal is a local terminal, the local program uses a Consumer component to extract key information from a local key information theme of the Broker component and store the key information into a local database;
when the user terminal is far-end, the far-end program uses the Consumer component to fetch key information from the far-end key information theme of the Broker component and store the key information in the far-end data center.
Optionally, in some possible embodiments, when the user terminal is a local terminal, the method further includes:
when the business logic changes, the local program uses the Consumer component to take out the original data from the local original data theme of the Broker component for analysis.
For example, the service logic change may be: for example, if the actual business needs to add a new key information field, namely whether the motor vehicle is driving at overspeed in daytime, a Consumer component can be used for extracting data from a local daytime motor vehicle original data Broker component for analysis and filling.
It should be noted that, the local end stores the key information of the data into the local database through the Consumer, uses the key information to perform data analysis and application, and if the content in the key information no longer meets the use requirement, the local end can pull the data from the original data cached in the Kafka Broker to perform analysis. Since the definition of key information of data tends to be standardized after the business of each data application department is mature, the probability of using original data is smaller and smaller, and the data stored in a database in large-scale message transmission can cause performance loss and resource waste, so temporary storage in a Broker component according to configuration duration can become a better choice.
Optionally, in some possible embodiments, when the user terminal is a remote end, the method further includes:
when a new data field needs to be queried and seen, the remote program uses the Consumer component to take out the original data from the remote original data subject of the Broker component for analysis.
For example, the service logic change may be: when a new data field needs to be checked, namely whether the motor vehicle is driving at overspeed in daytime, a Consumer can be used for taking out data from the original data Broker of the motor vehicle in the remote daytime for analysis and filling.
It should be noted that, the remote end stores the key information of the data into the remote data center through the Consumer, uses the key information to perform data analysis and application, and if the content in the key information no longer meets the use requirement, the remote end can pull the data from the original data cached in the Broker component to perform analysis. Usually, the remote data center also has real-time requirement on data, so the data transmission delay is increased by writing into a read database, and the problem is more obvious in large-scale message transmission. While the high throughput performance of kafka can meet the real-time requirement of large-scale message transmission, the extraction of key information is equivalent to the efficient simplification of the original data, and the data quality requirement is met. Shortening the data transmission link also increases the stability of the system.
It will be appreciated that in some embodiments, some or all of the above embodiments may be included.
As shown in fig. 2, a schematic structural framework is provided for an embodiment of the data transmission system of the present application, which is implemented based on a Kafka assembly 30, which includes: the Producer component 31, the Broker component 32, and the Consumer component 33, the Broker component 32 comprising at least one topic for each type of data, the data transmission system comprising:
the data acquisition device 10 is used for acquiring original data to be transmitted;
a data processing device 20 for extracting key information of the original data;
the Producer component 31 is configured to send the original data and the key information to different topics of the corresponding type of Broker component 32 for caching according to the type of the original data;
the user terminal 40 is configured to use the Consumer component 33 to obtain key information from the corresponding theme of the Broker component 32, and determine whether to obtain the original data according to the key information.
The data transmission system provided by the embodiment is suitable for high-efficiency and high-quality transmission of a large amount of data, the key information of the data is extracted, the high-efficiency and simplified of the original data are equivalent, the data quality requirement can be met, the data is classified and stored into different topics of Kafka Broker according to the data type, the production and consumption of millions of messages per second can be met due to the fact that Kafka has the characteristic of high throughput, the data are superior to a traditional database, the Broker can last data without memory pressure, the data can be used as a data storage center with extremely high real-time performance, data are cached, each application department can take according to the service requirement, the high throughput performance of Kafka can meet the real-time requirement of large-scale data transmission, the key information extraction is equivalent to the high-efficiency and simplified of the original data, the data quality requirement is met, and the shortening of a data transmission link also increases the stability of the system.
Optionally, in some possible embodiments, the Producer component 31 is specifically configured to send, according to the type of the original data, the original data and the key information to the cache in 4 topics of the corresponding type of Broker component, where the 4 topics are respectively: the remote key information system comprises a local key information theme, a local original data theme, a remote key information theme and a remote original data theme, wherein the local key information theme and the remote key information theme are the same in storage content, the local key information theme is used for storing local-oriented key information, and the remote key information theme is used for storing remote-oriented key information; the local original data theme is used for storing the original data facing the local, and the remote original data theme is used for storing the original data facing the remote.
Optionally, in some possible embodiments, when the user terminal 40 is a local terminal, the user terminal 40 is specifically configured to invoke a local program to use the Consumer component 33 to fetch key information from the local key information topic of the Broker component 32 and store the key information in the local database;
when the user terminal 40 is remote, the user terminal 40 is specifically configured to invoke a remote program to use the Consumer component 33 to retrieve key information from the remote key information theme of the Broker component 32 and store the key information in the remote data center.
Optionally, in some possible embodiments, when the user terminal 40 is a local terminal, the user terminal 40 is further configured to use the Consumer component 33 to extract the original data from the local original data topic of the Broker component 32 for parsing when the service logic changes.
Optionally, in some possible embodiments, when the user terminal 40 is remote, the user terminal 40 is further configured to use the Consumer component 33 to fetch the original data from the remote original data theme of the Broker component 32 for parsing when the new data field needs to be queried.
It will be appreciated that in some embodiments, some or all of the above embodiments may be included.
It should be noted that, each of the foregoing embodiments is a product example corresponding to the previous method example, and for the description of the product embodiment, reference may be made to the corresponding description in each of the foregoing method embodiments, which is not repeated herein.
The reader will appreciate that in the description of this specification, a description of terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the method embodiments described above are merely illustrative, e.g., the division of steps is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple steps may be combined or integrated into another step, or some features may be omitted or not performed.
The above-described method, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RAM, randomAccessMemory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The present application is not limited to the above embodiments, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the present application, and these modifications and substitutions are intended to be included in the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (8)
1. A data transmission method based on a Kafka assembly, the Kafka assembly comprising: the data transmission method comprises the following steps of a Producer component, a Broker component and a Consumer component, and is characterized in that the Broker component comprises at least one theme of each type of data, and the data transmission method comprises the following steps:
the data acquisition device acquires original data to be transmitted;
the data processing device extracts key information of the original data;
the Producer component sends the original data and the key information to different topics of a corresponding type of Broker component for caching according to the type of the original data;
the user terminal uses the Consumer component to acquire the key information from the corresponding theme of the Broker component, and judges whether to acquire the original data according to the key information;
the Producer component sends the original data and the key information to different topics of a corresponding type of Broker component for caching according to the type of the original data, and specifically comprises the following steps:
the Producer component sends the original data and the key information to 4 topics of a corresponding type of Broker component for caching according to the type of the original data, wherein the 4 topics are respectively: the remote key information management system comprises a local key information theme, a local original data theme, a remote key information theme and a remote original data theme, wherein the local key information theme and the remote key information theme are the same in stored content, the local key information theme is used for storing the local-oriented key information, and the remote key information theme is used for storing the remote-oriented key information; the local original data theme is the same as the content stored by the remote original data theme, the local original data theme is used for storing the original data facing to the local, and the remote original data theme is used for storing the original data facing to the remote.
2. The data transmission method based on the Kafka component according to claim 1, wherein the user terminal uses the Consumer component to obtain the key information from the corresponding theme of the Broker component, and determines whether to obtain the original data according to the key information, which specifically includes:
when the user terminal is a local terminal, a local program uses the Consumer component to extract key information from the local key information subject of the Broker component and store the key information into a local database;
and when the user terminal is far-end, the far-end program uses the Consumer component to extract key information from the far-end key information subject of the Broker component and store the key information in a far-end data center.
3. The data transmission method based on Kafka components according to claim 2, wherein when the user terminal is a local terminal, further comprising:
when business logic changes, the local program uses the Consumer component to take out original data from the local original data theme of the Broker component for analysis.
4. The Kafka component-based data transmission method according to claim 2, further comprising, when the user terminal is a remote end:
when a new data field needs to be queried and seen, the remote program uses the Consumer component to take out the original data from the remote original data subject of the Broker component for analysis.
5. A data transmission system based on a Kafka assembly, the Kafka assembly comprising: the data transmission system comprises a Producer component, a Broker component and a Consumer component, and is characterized in that the Broker component comprises at least one theme of each type of data, and the data transmission system comprises:
the data acquisition device is used for acquiring the original data to be transmitted;
the data processing device is used for extracting key information of the original data;
the Producer component is used for sending the original data and the key information to different topics of a corresponding type of Broker component for caching according to the type of the original data;
the user terminal is used for acquiring the key information from the corresponding theme of the Broker component by using the Consumer component, and judging whether to acquire the original data according to the key information;
the Producer component is specifically configured to send the original data and the key information to 4 topics of a Broker component of a corresponding type for caching according to the type of the original data, where the 4 topics are respectively: the remote key information management system comprises a local key information theme, a local original data theme, a remote key information theme and a remote original data theme, wherein the local key information theme and the remote key information theme are the same in stored content, the local key information theme is used for storing the local-oriented key information, and the remote key information theme is used for storing the remote-oriented key information; the local original data theme is the same as the content stored by the remote original data theme, the local original data theme is used for storing the original data facing to the local, and the remote original data theme is used for storing the original data facing to the remote.
6. The data transmission system based on a Kafka component according to claim 5, wherein when the user terminal is a local terminal, the user terminal is specifically configured to invoke a local program to use the Consumer component to extract key information from the local key information topic of the Broker component and store the key information in a local database;
when the user terminal is a far-end, the user terminal is specifically used for calling a far-end program to use the Consumer component to take out key information from the far-end key information subject of the Broker component and store the key information in a far-end data center.
7. The data transmission system based on a Kafka component according to claim 6, wherein when the user terminal is a local terminal, the user terminal is further configured to use the Consumer component to fetch raw data from the local raw data topic of the Broker component for parsing when a service logic change occurs.
8. The data transmission system based on a Kafka component according to claim 6, wherein when the user terminal is a far-end, the user terminal is further configured to use the Consumer component to fetch raw data from the far-end raw data topic of the Broker component for parsing when a new data field needs to be queried.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110914309.1A CN113572854B (en) | 2021-08-10 | 2021-08-10 | Data transmission method and system based on Kafka component |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110914309.1A CN113572854B (en) | 2021-08-10 | 2021-08-10 | Data transmission method and system based on Kafka component |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113572854A CN113572854A (en) | 2021-10-29 |
| CN113572854B true CN113572854B (en) | 2023-11-14 |
Family
ID=78171133
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110914309.1A Active CN113572854B (en) | 2021-08-10 | 2021-08-10 | Data transmission method and system based on Kafka component |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113572854B (en) |
Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105530327A (en) * | 2014-10-27 | 2016-04-27 | 任子行网络技术股份有限公司 | DNS (domain name system) key information processing method and system |
| DE202015000534U1 (en) * | 2015-01-26 | 2016-04-28 | a.r.t associated researchers + trendsetters gmbh | Computer-implemented information and knowledge delivery system |
| CN106326452A (en) * | 2016-08-26 | 2017-01-11 | 宁波薄言信息技术有限公司 | Method for human-machine dialogue based on contexts |
| CN107562810A (en) * | 2017-08-10 | 2018-01-09 | 北京时代云英科技有限公司 | Video big data is classified storage method |
| CN107844325A (en) * | 2017-10-27 | 2018-03-27 | 上海斐讯数据通信技术有限公司 | The acquisition methods and system of a kind of distributed data |
| CN108737177A (en) * | 2018-05-21 | 2018-11-02 | 中国联合网络通信有限公司重庆市分公司 | A kind of implementation method mobile Internet real-time streaming data acquisition and analyzed |
| CN109218379A (en) * | 2017-07-05 | 2019-01-15 | 维布络有限公司 | Data processing method and system in environment of internet of things |
| CN109614401A (en) * | 2018-12-06 | 2019-04-12 | 航天恒星科技有限公司 | Transmitted data on network storage system based on ElasticSearch and Hbase technology |
| CN109729130A (en) * | 2018-04-04 | 2019-05-07 | 中国平安人寿保险股份有限公司 | Information analysis method, service server, storage medium and device |
| CN109783182A (en) * | 2019-02-15 | 2019-05-21 | 百度在线网络技术（北京）有限公司 | A kind of method of adjustment, device, equipment and the medium of page subject matter tone |
| CN110225347A (en) * | 2019-06-24 | 2019-09-10 | 北京大米科技有限公司 | Method of transmitting video data, device, electronic equipment and storage medium |
| CN110727722A (en) * | 2019-08-30 | 2020-01-24 | 安徽四创电子股份有限公司 | Massive concurrent radar data storage method |
| CN110851473A (en) * | 2018-07-25 | 2020-02-28 | 中兴通讯股份有限公司 | Data processing method, device and system |
| US10735530B1 (en) * | 2020-02-18 | 2020-08-04 | Capital One Services, Llc | Aggregated service status reporter |
| CN111797150A (en) * | 2020-05-25 | 2020-10-20 | 武汉烽火众智数字技术有限责任公司 | Method and system for high concurrent data docking and forwarding |
| CN111818112A (en) * | 2019-04-11 | 2020-10-23 | 中国移动通信集团四川有限公司 | Kafka system-based message sending method and device |
| CN111897792A (en) * | 2020-08-11 | 2020-11-06 | 北京无线电测量研究所 | Distributed file access method, system, medium and device |
| CN112463772A (en) * | 2021-02-02 | 2021-03-09 | 北京信安世纪科技股份有限公司 | Log processing method and device, log server and storage medium |
| CN112685385A (en) * | 2020-12-31 | 2021-04-20 | 广西中科曙光云计算有限公司 | Big data platform for smart city construction |
| CN112929430A (en) * | 2021-01-29 | 2021-06-08 | 光控特斯联(上海)信息科技有限公司 | Data transmission method and system based on communication of Internet of things |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10540119B2 (en) * | 2016-08-26 | 2020-01-21 | Vmware, Inc. | Distributed shared log storage system having an adapter for heterogenous big data workloads |
-
2021
- 2021-08-10 CN CN202110914309.1A patent/CN113572854B/en active Active
Patent Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105530327A (en) * | 2014-10-27 | 2016-04-27 | 任子行网络技术股份有限公司 | DNS (domain name system) key information processing method and system |
| DE202015000534U1 (en) * | 2015-01-26 | 2016-04-28 | a.r.t associated researchers + trendsetters gmbh | Computer-implemented information and knowledge delivery system |
| CN106326452A (en) * | 2016-08-26 | 2017-01-11 | 宁波薄言信息技术有限公司 | Method for human-machine dialogue based on contexts |
| CN109218379A (en) * | 2017-07-05 | 2019-01-15 | 维布络有限公司 | Data processing method and system in environment of internet of things |
| CN107562810A (en) * | 2017-08-10 | 2018-01-09 | 北京时代云英科技有限公司 | Video big data is classified storage method |
| CN107844325A (en) * | 2017-10-27 | 2018-03-27 | 上海斐讯数据通信技术有限公司 | The acquisition methods and system of a kind of distributed data |
| CN109729130A (en) * | 2018-04-04 | 2019-05-07 | 中国平安人寿保险股份有限公司 | Information analysis method, service server, storage medium and device |
| CN108737177A (en) * | 2018-05-21 | 2018-11-02 | 中国联合网络通信有限公司重庆市分公司 | A kind of implementation method mobile Internet real-time streaming data acquisition and analyzed |
| CN110851473A (en) * | 2018-07-25 | 2020-02-28 | 中兴通讯股份有限公司 | Data processing method, device and system |
| CN109614401A (en) * | 2018-12-06 | 2019-04-12 | 航天恒星科技有限公司 | Transmitted data on network storage system based on ElasticSearch and Hbase technology |
| CN109783182A (en) * | 2019-02-15 | 2019-05-21 | 百度在线网络技术（北京）有限公司 | A kind of method of adjustment, device, equipment and the medium of page subject matter tone |
| CN111818112A (en) * | 2019-04-11 | 2020-10-23 | 中国移动通信集团四川有限公司 | Kafka system-based message sending method and device |
| CN110225347A (en) * | 2019-06-24 | 2019-09-10 | 北京大米科技有限公司 | Method of transmitting video data, device, electronic equipment and storage medium |
| CN110727722A (en) * | 2019-08-30 | 2020-01-24 | 安徽四创电子股份有限公司 | Massive concurrent radar data storage method |
| US10735530B1 (en) * | 2020-02-18 | 2020-08-04 | Capital One Services, Llc | Aggregated service status reporter |
| CN111797150A (en) * | 2020-05-25 | 2020-10-20 | 武汉烽火众智数字技术有限责任公司 | Method and system for high concurrent data docking and forwarding |
| CN111897792A (en) * | 2020-08-11 | 2020-11-06 | 北京无线电测量研究所 | Distributed file access method, system, medium and device |
| CN112685385A (en) * | 2020-12-31 | 2021-04-20 | 广西中科曙光云计算有限公司 | Big data platform for smart city construction |
| CN112929430A (en) * | 2021-01-29 | 2021-06-08 | 光控特斯联(上海)信息科技有限公司 | Data transmission method and system based on communication of Internet of things |
| CN112463772A (en) * | 2021-02-02 | 2021-03-09 | 北京信安世纪科技股份有限公司 | Log processing method and device, log server and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| "KOHA: Building a Kafka-Based Distributed Queue System on the Fly in a Hadoop Cluster";Cao Ngoc Nguyen et al;《2016 IEEE 1st International Workshops on Foundations and Applications of Self* Systems (FAS*W)》;全文 * |
| "基于海量数据的消息队列的性能对比与优化方案";刘峰等;《软件》;第1节，第4节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113572854A (en) | 2021-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106452819B (en) | Data acquisition system and data acquisition method | |
| KR101745456B1 (en) | Ecu for transmitting large data in hil test environment, system including the same and method thereof | |
| US8001150B2 (en) | Device management method using nodes having additional attribute and device management client thereof | |
| CN110752943B (en) | Distributed fault diagnosis system and method for power transmission line | |
| CN107786623B (en) | Message asynchronous processing method and device | |
| CN113242165B (en) | National standard data processing and uploading method for new energy vehicles | |
| WO2014173151A1 (en) | Method, device and terminal for data processing | |
| CN111726414A (en) | Vehicle reported data processing method and vehicle data reporting system | |
| CN109861998B (en) | Plug-in type dynamic analysis system and method based on Beidou short message protocol | |
| CN112702259B (en) | Delay message pushing method and device and message middleware | |
| CN111723115A (en) | Data statistics method, device, terminal and storage medium based on multi-level cache | |
| CN113572854B (en) | Data transmission method and system based on Kafka component | |
| CN104104568A (en) | Method, device and system for acquiring data from communication bus | |
| CN103581119B (en) | System and method for displaying production process data at high speed | |
| CN110888790B (en) | Log management method and device, electronic equipment and storage medium | |
| CN114363304B (en) | RTP video stream storage and playing method and device | |
| CN113992609B (en) | Method and system for processing multilink service data disorder | |
| CN115473858A (en) | Data transmission method and streaming data transmission system | |
| CN114885277A (en) | Emergency data transmission method based on Beidou third short message | |
| CN108989467B (en) | Centralized addressing method | |
| CN109471898B (en) | Method and system for sharing and distributing data | |
| CN114691455A (en) | Method and device for monitoring alarm data and dynamically pushing multiple platforms | |
| CN108075932B (en) | Data monitoring method and device | |
| CN116112482A (en) | Buried point data processing method and device, electronic equipment and storage medium | |
| KR100653038B1 (en) | Apparatus And Method For Transmission/Receiving Large Volumes Of Message In Satellite Mobile Data ServiceSMDS System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |