CN113420048A - Data aggregation method and device, electronic equipment and storage medium - Google Patents
Data aggregation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113420048A CN113420048A CN202110548797.9A CN202110548797A CN113420048A CN 113420048 A CN113420048 A CN 113420048A CN 202110548797 A CN202110548797 A CN 202110548797A CN 113420048 A CN113420048 A CN 113420048A
- Authority
- CN
- China
- Prior art keywords
- data
- kafka
- mqtt
- converting
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000002776 aggregation Effects 0.000 title claims abstract description 40
- 238000004220 aggregation Methods 0.000 title claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 37
- 230000008447 perception Effects 0.000 claims abstract description 29
- 230000002085 persistent effect Effects 0.000 claims abstract description 9
- 238000004806 packaging method and process Methods 0.000 claims abstract description 8
- 230000007246 mechanism Effects 0.000 claims description 14
- 230000015654 memory Effects 0.000 claims description 12
- 238000005111 flow chemistry technique Methods 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims 1
- 238000004590 computer program Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 10
- 238000012544 monitoring process Methods 0.000 description 10
- 238000013500 data storage Methods 0.000 description 9
- 238000010276 construction Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 108010001267 Protein Subunits Proteins 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 239000000758 substrate Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 229910002056 binary alloy Inorganic materials 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24561—Intermediate data storage techniques for performance improvement
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data aggregation method, including: collecting road perception data, and caching the road perception data in a persistent message queue storage mode; reading the road perception data, and packaging the road perception data into an MQTT data packet; converting the MQTT data packet into a Kafka message queue based on a Kafka stream processing platform; and converting the Kafka messages in the Kafka message queue into entity objects, then performing data processing, and storing the processed data in a time sequence database. The disclosure also provides a data aggregation device, an electronic device and a readable storage medium.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a data aggregation method and apparatus, an electronic device, and a storage medium.
Background
In recent years, with rapid development of information technologies such as sensors, internet of things, cloud computing, big data, artificial intelligence and the like, the demonstration application and test of an automatic driving technology are promoted, and the automatic driving technology is called as a smart vehicle by the traffic industry. For better service and support of automatic driving development, the field which is also concerned with smart vehicles is the construction of intelligent roads, and the construction of intelligent roads becomes a new hot field. At present, the definition of the intelligent road is not known in the industry, and the connotation and the scope of the intelligent road are rapidly developed. Meanwhile, the road facility and environment data are collected in real time, a road perception system is built, and the realization of all-round road perception is a necessary link for the currently accepted intelligent road construction.
Currently, road infrastructure awareness includes: a health monitoring system for special or particularly important structures such as bridges, tunnels, slopes and the like collects various response states of the structures by mounting various sensors such as temperature, humidity, displacement, strain, vibration, pressure and the like on the structures so as to master the safety states of the structures. The road meteorological environment perception comprises the following steps: the monitoring of environmental information such as wind speed and direction, visibility, environment humiture, rainfall, atmospheric pressure mainly through building automatic meteorological station, gathers multiple meteorological environment parameter set to master road along-line meteorological environment state, especially in the area that mountain area weather is changeable and geological disasters are frequent. In urban road, also road ponding monitoring, road surface wet and slippery situation monitoring, noise monitoring, air quality monitoring etc. and to road subsidiary facilities, if: the core thinking of the monitoring system such as well lid monitoring, street lamp monitoring and the like is that the monitoring system collects the states of facilities and equipment by arranging various sensors and transmits the states to a central platform through a wired or wireless network.
The road perception system is often independently and dispersedly built and self-organized due to differences of acquisition modes, communication modes, data formats and the like, and the systems are difficult to interconnect and intercommunicate and data are difficult to share. On the other hand, the systems all have highly similar technical ideas, and have the problems of repeated construction and resource waste. The road sensing system mainly transmits sensing data based on TCP/IP and UDP protocols through an optical fiber network and a 4G, GPRS network, when the network fluctuation is abnormal, the data reliability is not high, the data is difficult to continuously transmit, the data loss problem is caused, the transmission process lacks safety verification, a private network is often required to be built, and the transmission cost is high; data storage usually adopts files or relational databases for storage, and for scenes with high sampling frequency and continuous and uninterrupted acquisition, a large-scale business relational database with high price, such as Oracle, needs to be selected, but for data query with the data scale of 10 hundred million or more, the query time is long, and the performance is poor.
BRIEF SUMMARY OF THE PRESENT DISCLOSURE
In view of the foregoing technical problems, the present disclosure provides a data aggregation method, an apparatus, an electronic device, and a storage medium, which are used to at least partially solve the above technical problems.
One aspect of the present disclosure provides a data aggregation method, including: collecting road perception data, and caching the road perception data in a persistent message queue storage mode; reading road perception data, and packaging the road perception data into an MQTT data packet; converting the MQTT data packet into a Kafka message queue based on a Kafka stream processing platform; and converting the Kafka messages in the Kafka message queue into entity objects, then performing data processing, and storing the processed data.
According to an embodiment of the disclosure, converting the MQTT data packet into the Kafka message based on the Kafka stream processing platform includes: adopting an open-source or commercial MQTT publishing and subscribing platform to subscribe the MQTT data packet; based on a Kafka message engine, the MQTT data packet is converted into a Kafka message queue through a serialization mechanism.
According to an embodiment of the present disclosure, converting the Kafka message queue into an entity object includes: subscribing to a Kafka message; the Kafka message is converted into an entity object through an deserialization mechanism.
According to an embodiment of the present disclosure, storing the processed data includes: and writing the processed data into a time sequence database for storage.
Another aspect of the present disclosure provides a data aggregation apparatus, including: the acquisition module is used for acquiring the road perception data and caching the road perception data in a persistent message queue storage mode; the convergence module is used for reading the road perception data, packaging the road perception data into an MQTT data packet, converting the MQTT data packet into a Kafka message queue based on a Kafka flow processing platform, and converting Kafka messages in the Kafka message queue into entity objects for data processing; and the storage module is used for storing the processed data.
According to the embodiment of the disclosure, the converting, by the convergence module, the MQTT data packet into the Kafka message based on the Kafka stream processing platform includes: adopting an open-source or commercial MQTT publishing and subscribing platform to subscribe the MQTT data packet; based on a Kafka message engine, the MQTT data packet is converted into a Kafka message queue through a serialization mechanism.
According to an embodiment of the present disclosure, the converting, by the aggregation module, the Kafka message queue into the entity object includes: subscribing to a Kafka message; the Kafka message is converted into an entity object through an deserialization mechanism.
According to an embodiment of the present disclosure, the storing the processed data by the storage module includes: and writing the processed data into a time sequence database for storage.
Another aspect of the present disclosure also provides an electronic device, including: one or more processors; memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1 to 4.
Another aspect of the disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 4.
Drawings
FIG. 1 schematically illustrates a flow chart of a method of data aggregation in accordance with an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow chart of a data acquisition method according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a data acquisition method according to an embodiment of the present disclosure;
FIG. 4 schematically illustrates a flow chart of a data storage method according to an embodiment of the present disclosure;
FIG. 5 schematically illustrates a block diagram of a data aggregation device, according to an embodiment of the present disclosure;
fig. 6 schematically shows a block diagram of an electronic device adapted to implement the above described method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
Some block diagrams and/or flow diagrams are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations thereof, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, which execute via the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks. The techniques of this disclosure may be implemented in hardware and/or software (including firmware, microcode, etc.). In addition, the techniques of this disclosure may take the form of a computer program product on a computer-readable storage medium having instructions stored thereon for use by or in connection with an instruction execution system.
The embodiment of the disclosure provides a data aggregation method and device, which are used for a road intelligent perception system of intelligent traffic, and belong to the technical field of digital traffic infrastructure and intelligent traffic. The data aggregation method is realized from data acquisition, data aggregation and data storage.
Fig. 1 schematically shows a flow chart of a data aggregation method according to an embodiment of the present disclosure.
As shown in fig. 1, the data aggregation method may include operations S101 to S103, for example.
In operation S101, collecting road sensing data, and caching the road sensing data in a persistent message queue storage manner; reading the road perception data, and packaging the road perception data into an MQTT data packet.
Fig. 2 schematically shows a flow chart of a data acquisition method according to an embodiment of the present disclosure.
As shown in FIG. 2, the collection of sensory data may include collection and caching of data, data transformation, and publication.
In the embodiment of the present disclosure, the data collection and cache can collect or receive data packets sent by other collection programs in real time, such as: UDP packets, TCP protocol packets, and the like. The binary system form can be adopted, the binary data packet is cached to the message queue, and the message queue can be stored based on the memory and the hard disk in a persistent mode, so that the data are not lost when the equipment is powered off. The acquisition device may include, for example, sensors, collectors, acquisition software, and the like.
In the embodiment of the disclosure, the data conversion and publishing may be to take out the message load from the message queue, encapsulate the message into MQTT protocol data packet payload, and publish the MQTT protocol data packet payload to the MQTT Broker server. The MQTT protocol supports three quality guarantees with a Qos of 0/1/2, in order to ensure that data is not lost, Qos of 1 or 2 can be selected, for a scenario with Qos of 1, data may have a situation of repeated transmission, and data deduplication needs to be performed at an application end, and for a scenario with Qos of 2, data can be accurately transmitted only 1 time without subsequent deduplication.
In operation S102, based on the Kafka stream processing platform, the MQTT data packet is converted into a Kafka message queue, and the Kafka messages in the Kafka message queue are converted into entity objects for data processing.
Aggregation of data may be achieved based on operation S102. The data aggregation can be realized based on an MQTT Broker server, a data parsing and publishing module and a Kafka message engine of a Kafka stream processing platform.
Fig. 3 schematically illustrates a flow chart of a data acquisition method according to an embodiment of the present disclosure.
As shown in fig. 3, in the embodiment of the present disclosure, the MQTT Broker server may employ an open-source or commercial MQTT publish-subscribe platform to implement publish and subscribe for high-performance MQTT data packets. The adopted MQTT Broker platform can support distributed transverse expansion and can be matched with load balancing equipment or middleware to realize load balancing under the condition that massive terminals are accessed into a high-concurrency scene.
And the data analysis and release module can subscribe the MQTT data packet and convert the MQTT data packet into a Kafka Message queue (Kafka Message) through a serialization mechanism. The process calls a protocol analysis SDK or a protocol analysis class library of the data message, analyzes the data into a general data model, and provides a basis for the subsequent data standardization processing.
The Kafka message engine is an open-source, distributed, multi-partition, multi-copy and multi-subscriber distributed high-performance message engine based on zookeeper coordination, and is widely applied to scenes such as large data stream processing, message queues and log systems. The Kafka message engine is selected as a core message queue due to ultra-high throughput, data persistence, a data partitioning mechanism and distributed support, and on one hand, the Kafka message engine can be conveniently and seamlessly butted with a large data stream processing platform, such as Apache flash/Spark; on the other hand, the system has higher stability and persistent storage capacity based on the hard disk, can ensure reliable data caching, balance message processing pressure and decouple production and consumption capacity, and has higher flexibility. Therefore, the disclosed embodiment converts the MQTT data packet into the Kafka message queue through a serialization mechanism based on the Kafka message engine.
In operation S103, the processed data is stored.
FIG. 4 schematically shows a flow chart of a data storage method according to an embodiment of the present disclosure.
As shown in fig. 4, in the embodiment of the present disclosure, the storage of data may be accomplished based on data subscription and processing and a time-series database.
The data subscription and processing can subscribe Kafka messages in real time, deserialize the Kafka messages, convert the Kafka messages into entity objects and process the data according to the service requirements, the Kafka has good adaptability with a large data stream processing platform comprising Flink and Spark, the stream processing application can be conveniently and quickly developed and deployed, and the data adaptation difficulty is greatly reduced. After the data processing is finished, mass time sequence data storage is realized by writing the data into the time sequence database in batches.
The time series database is also referred to as a time series database. The time series database is mainly used for processing data with time tags (which are changed in time sequence, i.e., time-sequenced), and the data with time tags is also called time series data. At present, a relational database mode is often adopted for storage and processing of time series big data, but the relational database cannot be efficiently stored and queried due to inherent disadvantages of the relational database. The time sequence big data solution enables the time sequence big data to be efficiently stored and quickly process the mass time sequence big data by using a special storage mode, and is an important technology for solving the mass data processing. The technology adopts a special data storage mode, greatly improves the processing capacity of time-related data, and has less storage space and greatly improved query speed compared with a relational database. The superior query performance of the time sequence function is far superior to that of a relational database, such as an InfluxDB database which is well known in the industry, a domestic TDengine database, and a TSDB database provided by Aliskiu and the like. Therefore, in the embodiment of the present disclosure, the processed data is written into the time-series database for storage.
According to the data aggregation method provided by the embodiment of the disclosure, in the data acquisition stage, the message queue based on hard disk persistence is adopted to cache data to be transmitted, so that the problems of data temporary storage and data loss caused by unexpected power failure of a road perception acquisition end when the network bandwidth is insufficient can be solved, and the data conversion and release module can support the function of continuous transmission of a disconnected network. And the data of the outfield sensor is collected to the central server platform, the MQTT protocol is adopted for transmission, and the problem of data loss when the network quality is poor can be solved through a bidirectional data confirmation mechanism. In the data aggregation stage, distributed expansion and ultrahigh data throughput are supported based on the Kafka flow processing platform, a data caching effect is achieved, the problem of rear-end data processing downtime caused by front-end burst flow is avoided, and the reliability of the system is improved. In the data storage stage, the time sequence database is adopted to store the road perception data, the problem of mass original data storage can be solved, the query and storage performance is high, and the data storage space is saved under the same condition.
Based on the same inventive concept, the embodiment of the disclosure also provides a data aggregation device.
Fig. 5 schematically shows a block diagram of a data aggregation apparatus according to an embodiment of the present disclosure.
As shown in fig. 5, the data aggregation apparatus 500 may include, for example, an acquisition module 510, an aggregation module 520, and a storage module 530.
The collecting module 510 is configured to collect the road sensing data, and cache the road sensing data in a persistent message queue storage manner.
The convergence module 520 is configured to read the road sensing data, encapsulate the road sensing data into MQTT data packets, convert the MQTT data packets into Kafka message queues based on the Kafka stream processing platform, and convert the Kafka messages in the Kafka message queues into entity objects for data processing.
And a storage module 530 for storing the processed data.
In the embodiment of the present disclosure, the converging module converts the MQTT data packet into the Kafka message based on the Kafka stream processing platform, which may include: adopting an open-source or commercial MQTT publishing and subscribing platform to subscribe the MQTT data packet; and converting the MQTT data packet into a Kafka message queue through a serialization mechanism based on a Kafka message engine.
In the embodiment of the present disclosure, the converting, by the aggregation module, the Kafka message queue into an entity object includes: subscribing to a Kafka message; and converting the Kafka message into an entity object through an deserialization mechanism.
In an embodiment of the present disclosure, the storing the processed data by the storage module includes: and writing the processed data into a time sequence database for storage.
Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
For example, any plurality of the acquisition module 510, the aggregation module 520, and the storage module 530 may be combined and implemented in one module/unit/sub-unit, or any one of the modules/units/sub-units may be split into a plurality of modules/units/sub-units. Alternatively, at least part of the functionality of one or more of these modules/units/sub-units may be combined with at least part of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. According to an embodiment of the present disclosure, at least one of the acquisition module 510, the aggregation module 520, and the storage module 530 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable manner of integrating or packaging a circuit, or may be implemented in any one of three implementations of software, hardware, and firmware, or in a suitable combination of any of them. Alternatively, at least one of the acquisition module 510, the aggregation module 520 and the storage module 530 may be at least partially implemented as a computer program module, which when executed, may perform a corresponding function.
It should be noted that, the data aggregation device portion in the embodiment of the present disclosure corresponds to the data aggregation method portion in the embodiment of the present disclosure, and the specific implementation details and the technical effects thereof are also the same, and are not described herein again.
Fig. 6 schematically shows a block diagram of an electronic device adapted to implement the above described method according to an embodiment of the present disclosure. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, an electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. Processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 601 may also include onboard memory for caching purposes. Processor 601 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the disclosure.
In the RAM603, various programs and data necessary for the operation of the electronic apparatus 600 are stored. The processor 601, the ROM602, and the RAM603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM602 and/or RAM 603. It is to be noted that the programs may also be stored in one or more memories other than the ROM602 and RAM 603. The processor 601 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program, when executed by the processor 601, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM602 and/or RAM603 described above and/or one or more memories other than the ROM602 and RAM 603.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The above-mentioned embodiments are intended to illustrate the objects, aspects and advantages of the present disclosure in further detail, and it should be understood that the above-mentioned embodiments are only illustrative of the present disclosure and are not intended to limit the present disclosure, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (10)
1. A method of data aggregation, comprising:
collecting road perception data, and caching the road perception data in a persistent message queue storage mode;
reading the road perception data, and packaging the road perception data into an MQTT data packet;
converting the MQTT data packet into a Kafka message queue based on a Kafka stream processing platform;
and converting the Kafka messages in the Kafka message queue into entity objects, then performing data processing, and storing the processed data.
2. The data aggregation method according to claim 1, wherein the Kafka-based stream processing platform converting the MQTT data packet into the Kafka message comprises:
adopting an open-source or commercial MQTT publishing and subscribing platform to subscribe the MQTT data packet;
and converting the MQTT data packet into a Kafka message queue through a serialization mechanism based on a Kafka message engine.
3. The data aggregation method according to claim 1, wherein the converting the Kafka message queue into an entity object comprises:
subscribing to the Kafka message;
and converting the Kafka message into an entity object through an deserialization mechanism.
4. The data aggregation method according to claim 1, wherein the storing the processed data comprises:
and writing the processed data into a time sequence database for storage.
5. A data aggregation device, comprising:
the acquisition module is used for acquiring road perception data and caching the road perception data in a persistent message queue storage mode;
the convergence module is used for reading the road perception data, packaging the road perception data into an MQTT data packet, converting the MQTT data packet into a Kafka message queue based on a Kafka flow processing platform, and converting Kafka messages in the Kafka message queue into entity objects for data processing;
and the storage module is used for storing the processed data.
6. The data aggregation device of claim 5, wherein the aggregation module is based on a Kafka stream processing platform, and the transforming the MQTT data packet into the Kafka message comprises:
adopting an open-source or commercial MQTT publishing and subscribing platform to subscribe the MQTT data packet;
and converting the MQTT data packet into a Kafka message queue through a serialization mechanism based on a Kafka message engine.
7. The data aggregation device of claim 5, wherein the aggregation module to convert the Kafka message queue into an entity object comprises:
subscribing to the Kafka message;
and converting the Kafka message into an entity object through an deserialization mechanism.
8. The data aggregation apparatus according to claim 5, wherein the storage module stores the processed data comprises:
and writing the processed data into a time sequence database for storage.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-4.
10. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110548797.9A CN113420048A (en) | 2021-05-19 | 2021-05-19 | Data aggregation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110548797.9A CN113420048A (en) | 2021-05-19 | 2021-05-19 | Data aggregation method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113420048A true CN113420048A (en) | 2021-09-21 |
Family
ID=77712612
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110548797.9A Pending CN113420048A (en) | 2021-05-19 | 2021-05-19 | Data aggregation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113420048A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114461150A (en) * | 2022-02-09 | 2022-05-10 | 马来西亚明试国际有限公司 | Method, system and storage medium for automatic test equipment data aggregation |
| CN115037729A (en) * | 2022-04-21 | 2022-09-09 | 中国建设银行股份有限公司 | Data aggregation method and device, electronic equipment and computer readable medium |
| CN115914360A (en) * | 2022-09-15 | 2023-04-04 | 成都飞机工业（集团）有限责任公司 | Time sequence data storage method, device, equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108038201A (en) * | 2017-12-12 | 2018-05-15 | 无锡华云数据技术服务有限公司 | A kind of data integrated system and its distributed data integration system |
| CN108519987A (en) * | 2018-02-24 | 2018-09-11 | 国家计算机网络与信息安全管理中心 | A kind of data persistence method and apparatus |
| CN108768826A (en) * | 2018-05-16 | 2018-11-06 | 华南理工大学 | Based on the message route method under MQTT and Kafka high concurrent scenes |
| CN108985461A (en) * | 2018-06-29 | 2018-12-11 | 深圳昂云鼎科技有限公司 | A kind of method, apparatus and terminal device of autonomous machine study |
| CN109871502A (en) * | 2019-01-18 | 2019-06-11 | 北京赛思信安技术股份有限公司 | A kind of flow data canonical matching process based on Storm |
| CN111131348A (en) * | 2018-10-31 | 2020-05-08 | 顺丰科技有限公司 | Internet of vehicles system and operation method thereof |
| CN112099948A (en) * | 2020-09-10 | 2020-12-18 | 西安交通大学 | Method for standardizing digital twin manufacturing unit protocol and integrating industrial big data in real time |
| CN112769897A (en) * | 2020-12-21 | 2021-05-07 | 北京百度网讯科技有限公司 | Synchronization method and device for edge calculation message, electronic equipment and storage medium |
-
2021
- 2021-05-19 CN CN202110548797.9A patent/CN113420048A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108038201A (en) * | 2017-12-12 | 2018-05-15 | 无锡华云数据技术服务有限公司 | A kind of data integrated system and its distributed data integration system |
| CN108519987A (en) * | 2018-02-24 | 2018-09-11 | 国家计算机网络与信息安全管理中心 | A kind of data persistence method and apparatus |
| CN108768826A (en) * | 2018-05-16 | 2018-11-06 | 华南理工大学 | Based on the message route method under MQTT and Kafka high concurrent scenes |
| CN108985461A (en) * | 2018-06-29 | 2018-12-11 | 深圳昂云鼎科技有限公司 | A kind of method, apparatus and terminal device of autonomous machine study |
| CN111131348A (en) * | 2018-10-31 | 2020-05-08 | 顺丰科技有限公司 | Internet of vehicles system and operation method thereof |
| CN109871502A (en) * | 2019-01-18 | 2019-06-11 | 北京赛思信安技术股份有限公司 | A kind of flow data canonical matching process based on Storm |
| CN112099948A (en) * | 2020-09-10 | 2020-12-18 | 西安交通大学 | Method for standardizing digital twin manufacturing unit protocol and integrating industrial big data in real time |
| CN112769897A (en) * | 2020-12-21 | 2021-05-07 | 北京百度网讯科技有限公司 | Synchronization method and device for edge calculation message, electronic equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114461150A (en) * | 2022-02-09 | 2022-05-10 | 马来西亚明试国际有限公司 | Method, system and storage medium for automatic test equipment data aggregation |
| CN115037729A (en) * | 2022-04-21 | 2022-09-09 | 中国建设银行股份有限公司 | Data aggregation method and device, electronic equipment and computer readable medium |
| CN115037729B (en) * | 2022-04-21 | 2024-05-28 | 中国建设银行股份有限公司 | Data aggregation method, device, electronic equipment and computer readable medium |
| CN115914360A (en) * | 2022-09-15 | 2023-04-04 | 成都飞机工业（集团）有限责任公司 | Time sequence data storage method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113420048A (en) | Data aggregation method and device, electronic equipment and storage medium | |
| US20220021917A1 (en) | Dynamic load balancing for video analytics pipelines | |
| Lim et al. | State of data platforms for connected vehicles and infrastructures | |
| JP2024041823A (en) | SYSTEM AND METHOD FOR MANAGING VEHICLE DATA - Patent application | |
| US11879749B2 (en) | Enhanced high definition maps for a vehicle | |
| CN109377778B (en) | Collaborative automatic driving system and method based on multipath RDMA and V2X | |
| Yu et al. | RTIC-C: A big data system for massive traffic information mining | |
| US11689623B2 (en) | Adaptive real-time streaming for autonomous vehicles | |
| Rito et al. | Aveiro tech city living lab: A communication, sensing and computing platform for city environments | |
| CN111131348A (en) | Internet of vehicles system and operation method thereof | |
| CN113808389A (en) | Vehicle-road cooperation system, edge computing unit, central cloud platform and information processing method | |
| CN110704491B (en) | Data query method and device | |
| Liu | Big data analytics architecture for internet-of-vehicles based on the spark | |
| CN112671839A (en) | Data processing method and device, cloud system and storage medium | |
| CN112907942A (en) | Vehicle scheduling method, device, equipment and medium based on edge calculation | |
| Niaz et al. | Smart city IoT application for road infrastructure safety and monitoring by using digital twin | |
| CN115909716A (en) | Traffic intersection scheduling system, method and equipment based on internet cloud control platform | |
| CN113051323B (en) | Water environment big data exchange method | |
| US11538293B2 (en) | Preparation of vehicle bulk data for high speed terahertz offloading | |
| Badidi | A fog node architecture for real-time processing of urban IoT data streams | |
| CN205541467U (en) | Public vehicle's operation condition monitoring system | |
| CN207938180U (en) | A kind of road traffic signal lamp situation intelligent monitor system | |
| CN102446156B (en) | Communication behavior obtaining device for parallel multiprocessor system | |
| Alamgir Hossain et al. | Detecting situations from heterogeneous internet of things data in smart city context | |
| WO2024108608A1 (en) | Communication method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |