CN113568587A - Smart city real-time data processing method, system and storage medium - Google Patents
Smart city real-time data processing method, system and storage medium Download PDFInfo
- Publication number
- CN113568587A CN113568587A CN202111104385.2A CN202111104385A CN113568587A CN 113568587 A CN113568587 A CN 113568587A CN 202111104385 A CN202111104385 A CN 202111104385A CN 113568587 A CN113568587 A CN 113568587A
- Authority
- CN
- China
- Prior art keywords
- data
- smart city
- time data
- real
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
- G06F11/1012—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices using codes or arrangements adapted for a specific type of error
- G06F11/102—Error in check bits
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F5/00—Methods or arrangements for data conversion without changing the order or content of the data handled
- G06F5/06—Methods or arrangements for data conversion without changing the order or content of the data handled for changing the speed of data flow, i.e. speed regularising or timing, e.g. delay lines, FIFO buffers; over- or underrun control therefor
- G06F5/065—Partitioned buffers, e.g. allowing multiple independent queues, bidirectional FIFO's
Abstract
The invention discloses a smart city real-time data processing method, a system and a storage medium, wherein the method comprises the following steps: after acquiring smart city real-time data uploaded by a client from a network port, returning a message to the client through an asynchronous non-blocking network IO model; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
Description
Technical Field
The invention relates to the technical field of smart city monitoring, in particular to a smart city real-time data processing method, a smart city real-time data processing system and a smart city real-time data processing storage medium.
Background
In the current smart city strong real-time data transmission mode, most systems are based on spring MVC as a receiving program on a server, and a common synchronous blocking scheme is adopted, because the concurrency capability of a tomcat container is limited, sensor data which can be simultaneously received in unit time is limited, so that delay is increased, the requirement of second-level real-time transmission cannot be met, the problem of strong real-time data is usually solved by adding hardware, and the problems of huge hardware cost and high initial investment threshold are brought. In order to control the cost, the processing capability is reduced, the delay is increased, the stability is reduced, and the reliability is insufficient.
Disclosure of Invention
The invention mainly aims to provide a smart city real-time data processing method, a system and a storage medium, aiming at reducing smart city real-time data delay and improving data processing capacity, stability and reliability.
In order to achieve the purpose, the invention provides a smart city real-time data processing method, which comprises the following steps:
after acquiring smart city real-time data uploaded by a client from a network port, returning a message to the client through an asynchronous non-blocking network IO model;
decoding the smart city real-time data;
storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue;
and coding the data in the high-speed data queue according to a preset mode, writing the data into a time sequence database, and performing CSV persistence processing.
Optionally, the step of decoding the smart city real-time data further comprises:
and checking the real-time data of the smart city, and if the real-time data passes the checking, executing the following steps: and decoding the smart city real-time data.
Optionally, the step of verifying the smart city real-time data includes:
and verifying the real-time data of the smart city by adopting a length verification mode and an effectiveness verification mode.
Optionally, the step of verifying the smart city real-time data in a length verification manner includes:
adopt length check-up formula, right smart city real-time data checks up, and whether the check smart city real-time data package is complete, the length check-up formula is: the packet byte number = the packet header byte number + the data byte number + the check byte number.
Optionally, the step of verifying the smart city real-time data in a validity verification manner includes:
checking whether the smart city real-time data are valid or not by adopting a validity check formula, wherein the validity check formula is as follows: the check bit = performing bit and operation on all bytes from the packet header to the front of the check bit.
Optionally, the step of decoding the smart city real-time data includes:
extracting a data packet from the smart city real-time data;
splitting the extracted data packet into single sensor data items to be placed in the high-speed data queue;
the step of storing the decoded data in a high-speed data queue, in which the data is managed based on the sensor data items, includes:
storing each decoded data packet into the high-speed data queue in a classified manner in a sensor data item manner;
in the high-speed data queue, data is managed in a manner that a sensor organizes and manages a sub-queue based on sensor data items.
Optionally, the step of encoding the data in the high-speed data queue according to a preset manner includes:
and coding the data in the high-speed data queue according to a data block reorganizing mode of the sensor and a data block reorganizing mode required by a time sequence database and a CSV file.
The embodiment of the invention also provides a smart city real-time data processing system, which comprises: a client and a web server, wherein:
the client is used for acquiring real-time data of the smart city and uploading the real-time data to the network server;
the network server is used for returning a message to the client through an asynchronous non-blocking network IO model after acquiring the smart city real-time data uploaded by the client from a network port; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; and coding the data in the high-speed data queue according to a preset mode, writing the data into a time sequence database, and performing CSV persistence processing.
The embodiment of the invention also provides a smart city real-time data processing system, which comprises a memory, a processor and a smart city real-time data processing program stored on the memory, wherein the smart city real-time data processing program is executed by the smart city real-time data processing method when being operated by the processor.
The embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a smart city real-time data processing program, and when the smart city real-time data processing program is executed by a processor, the smart city real-time data processing method is executed.
According to the smart city real-time data processing method, the system and the storage medium, after smart city real-time data uploaded by a client are obtained from a network port, a message is returned to the client through an asynchronous non-blocking network IO model; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
Drawings
Fig. 1 is a schematic functional block diagram of a terminal device to which a smart city monitoring real-time data processing apparatus according to an embodiment of the present invention belongs;
FIG. 2 is a schematic flow chart illustrating a smart city real-time data processing method according to a first embodiment of the present invention;
FIG. 3 is a diagram illustrating the structure of data in a high speed data queue according to an embodiment of the present invention;
FIG. 4 is a schematic flowchart illustrating a smart city real-time data processing method according to a second embodiment of the present invention;
FIG. 5 is a data lane diagram of an embodiment of the smart city real-time data processing method according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The main solution of the embodiment of the invention is as follows: after acquiring smart city real-time data uploaded by a client from a network port, returning a message to the client through an asynchronous non-blocking network IO model; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
In the embodiment of the invention, a common synchronous blocking scheme in a smart city strong real-time data transmission mode in the existing related scheme is limited in concurrency capability, and sensor data which can be simultaneously received in unit time is limited, so that delay is increased, and the requirement of second-level real-time transmission cannot be met.
Specifically, referring to fig. 1, fig. 1 is a schematic diagram of functional modules of a terminal device to which the smart city real-time data processing apparatus belongs. The smart city real-time data processing device can be a device which is independent of the terminal equipment and can process data, and the device can be borne on the terminal equipment in a hardware or software mode. The terminal device may be a network device such as a server.
In this embodiment, the terminal device to which the smart city real-time data processing apparatus belongs at least includes an output module 110, a processor 120, a memory 130 and a communication module 140.
The memory 130 stores an operating system and a real-time data processing program of the smart city, and the output module 110 may be a display screen, a speaker, etc. The communication module 140 may include a WIFI module, a mobile communication module, a bluetooth module, and the like, and communicates with an external device or a server through the communication module 140.
As an embodiment, the smart city real-time data processing program in the memory 130 implements the following steps when executed by the processor:
after acquiring smart city real-time data uploaded by a client from a network port, returning a message to the client through an asynchronous non-blocking network IO model;
decoding the smart city real-time data;
storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue;
and coding the data in the high-speed data queue according to a preset mode, writing the data into a time sequence database, and performing CSV persistence processing.
Further, the smart city real-time data processing program in the memory 130, when executed by the processor, further implements the following steps:
and checking the real-time data of the smart city, and if the real-time data passes the checking, executing the following steps: and decoding the smart city real-time data.
Further, the smart city real-time data processing program in the memory 130, when executed by the processor, further implements the following steps:
and verifying the real-time data of the smart city by adopting a length verification mode and an effectiveness verification mode.
Further, the smart city real-time data processing program in the memory 130, when executed by the processor, further implements the following steps:
adopt length check-up formula, right smart city real-time data checks up, and whether the check smart city real-time data package is complete, the length check-up formula is: the packet byte number = the packet header byte number + the data byte number + the check byte number.
Further, the smart city real-time data processing program in the memory 130, when executed by the processor, further implements the following steps:
checking whether the smart city real-time data are valid or not by adopting a validity check formula, wherein the validity check formula is as follows: the check bit = performing bit and operation on all bytes from the packet header to the front of the check bit.
Further, the smart city real-time data processing program in the memory 130, when executed by the processor, further implements the following steps:
extracting a data packet from the smart city real-time data;
splitting the extracted data packet into single sensor data items to be placed in the high-speed data queue;
the step of storing the decoded data in a high-speed data queue, in which the data is managed based on the sensor data items, includes:
storing each decoded data packet into the high-speed data queue in a classified manner in a sensor data item manner;
in the high-speed data queue, data is managed in a manner that a sensor organizes and manages a sub-queue based on sensor data items.
Further, the smart city real-time data processing program in the memory 130, when executed by the processor, further implements the following steps:
and coding the data in the high-speed data queue according to a data block reorganizing mode of the sensor and a data block reorganizing mode required by a time sequence database and a CSV file.
According to the scheme, after the smart city real-time data uploaded by the client is obtained from the network port, the message is returned to the client through the asynchronous non-blocking network IO model; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
Specifically, as shown in fig. 2, the first embodiment of the smart city real-time data processing method of the present invention includes the following steps:
and step S10, after the smart city real-time data uploaded by the client is obtained from the network port, returning a message to the client through the asynchronous non-blocking network IO model.
The execution main body of the method can be a network server interacting with the client, a network port is arranged on the network server, and the network server receives data uploaded by the client through the network port.
In this embodiment, after obtaining the data uploaded by the client from the network port, an asynchronous non-blocking IO mode is adopted to send a message to the client immediately to indicate that the data has been received, so that resources can be issued for the next packet.
Compared with a common synchronous blocking mode, the data is returned to the client after being processed, so that the client needs to wait for a long time, when subsequent data arrives, the time delay is too long due to the fact that the data is waited for the previous data processing, the data of the corresponding segment needs to be obtained according to the sensor after the subsequent data processing, and therefore the structure is different from that when the data comes in, the algorithm of a program is very complex, the system stability is caused, the reliability is very poor, the system is easy to crash due to the fact that resources are excessively occupied, and faults such as memory overflow and the like occur.
In the embodiment, after the asynchronous non-blocking IO mode is adopted, the client does not need to wait for a long time, the complexity in data processing is reduced, the number of concurrent links of the client is greatly increased, and the number of concurrent links can reach 1 ten thousand times of that of a synchronous blocking IO mode.
Step S20, decoding the smart city real-time data;
as an embodiment, before the step of decoding the smart city real-time data, the method may further include:
and checking the real-time data of the smart city, and if the real-time data passes the checking, executing the following steps: and decoding the smart city real-time data.
The data check is divided into two modes, namely length check and validity check. The formula adopted for length check is as follows: and (4) the number of bytes of the packet is = the number of bytes of the packet header + the number of bytes of data + the number of check bytes, and whether the packet is complete is checked. The validity check adopts the formula as follows: the check bit = performing bit and operation on all bytes from the packet header to the front of the check bit, and checking whether the bytes are valid.
And decoding the data passing the verification, namely extracting the data in the packet, splitting the data into single sensor data items and preparing the data items for being placed in a high-speed data queue.
Specifically, the step of decoding the smart city real-time data includes:
extracting a data packet from the smart city real-time data;
and splitting the extracted data packet into single sensor data items to be placed into the high-speed data queue.
Step S30, storing the decoded data into a high-speed data queue, and managing the data in the high-speed data queue based on the sensor data item;
storing each decoded data packet into the high-speed data queue in a classified manner in a sensor data item manner;
in the high-speed data queue, data is managed in a manner that a sensor organizes and manages a sub-queue based on sensor data items.
As shown in fig. 3, in this embodiment, the high-speed data queue manages the sub-queues respectively according to the sensor organization classification, thereby completely avoiding confusion caused by data intersection for high-frequency data, reducing complexity of subsequent operations, and increasing processing speed.
And step S40, encoding the data in the high-speed data queue according to a preset mode, writing the data into a time sequence database, and performing CSV persistence processing.
And after the decoded data are stored in a high-speed data queue, encoding the data in the high-speed data queue according to a data block reorganizing mode of a sensor and a data block reorganizing mode required by a time sequence database and a CSV file.
The data coding is to reorganize the data blocks according to the requirements of the database and the CSV file, so that batch writing is realized, the writing times are reduced, the whole block writing is realized, and the processing speed is improved.
Therefore, starting with the IO mode of the software network, after the asynchronous non-blocking IO mode is adopted, the client does not need to wait for a long time, meanwhile, data processing and data receiving are decoupled, complexity in data processing is reduced, the number of concurrent links of the client is greatly increased, 1 ten thousand times of a synchronous blocking type can be achieved, delay can be controlled within 100 milliseconds, thousands of sensors send data stably at 100HZ, received data packets pass through quick verification, and the probability of occurrence of problems such as packet sticking, packet error, packet time sequence disorder and the like is less than 1%. Meanwhile, the received data passes through the links of decoding, queuing and encoding, so that the complexity of data processing is reduced, the stability and the reliability are improved, and the problem of memory occupation is solved.
According to the technical scheme, after smart city real-time data uploaded by a client is obtained from a network port, a message is returned to the client through an asynchronous non-blocking network IO model; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
Further, referring to fig. 4, fig. 4 is a schematic flow chart of a smart city real-time data processing method according to a second embodiment of the present invention, which is different from the first embodiment shown in fig. 2 in that in this embodiment, the step S10 is followed by further comprising:
s101, checking the smart city real-time data, and specifically comprising the following steps:
and verifying the real-time data of the smart city by adopting a length verification mode and an effectiveness verification mode.
Data verification is a verification operation performed to ensure the integrity of data, a designated algorithm is usually used to calculate a verification value for original data, a receiver calculates a verification value for the original data by using the same algorithm, if the verification values obtained by the two calculations are the same, the data is complete, otherwise, the data is incomplete.
Specifically, in this embodiment, the step of verifying the smart city real-time data by using length verification includes:
the formula is adopted: and (3) the number of bytes of the packet is = the number of bytes of the packet header + the number of bytes of data + the number of check bytes, the real-time data of the smart city is checked, and whether the real-time data packet of the smart city is complete or not is checked.
The step of verifying the smart city real-time data by adopting validity verification comprises the following steps:
the formula is adopted: and (4) checking a bit and an operation result of all bytes from the packet header to the front of the check bit, and checking whether the real-time data of the smart city is valid.
According to the technical scheme, after smart city real-time data uploaded by a client is obtained from a network port, a message is returned to the client through an asynchronous non-blocking network IO model; checking the real-time data of the smart city by adopting length checking and validity checking; if the verification is passed, decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
The smart city real-time data processing method of the present invention will be further described in detail below.
In a common synchronous blocking mode, data is returned to a client after being processed, so that the client needs to wait for a long time, when subsequent data arrives, too much time delay is caused by waiting for the previous data processing, the subsequent data processing needs to obtain data of corresponding segments according to a sensor, and thus, different from the structure of the incoming data, the algorithm of a program is very complex, the system stability is caused, the reliability is very poor, and the faults such as collapse, memory overflow and the like are easily caused due to the fact that resources are excessively occupied.
As shown in fig. 5, after the asynchronous non-blocking IO mode is adopted, the client does not need to wait for a long time, and meanwhile, the data processing and data receiving are decoupled, so that the complexity in the data processing is reduced, the number of concurrent links of the client is greatly increased, which can reach 1 ten thousand times of the synchronous blocking type, and meanwhile, the received data passes through the decoding, queuing and encoding links, so that the complexity of the data processing is reduced, the stability and reliability are improved, and the problem of memory occupation is solved.
According to the smart city real-time data processing method provided by the embodiment of the invention, after smart city real-time data uploaded by a client is obtained from a network port, a message is returned to the client through an asynchronous non-blocking network IO model; checking the real-time data of the smart city by adopting length checking and validity checking; if the verification is passed, decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
In order to achieve the above object, the present invention further provides a smart city real-time data processing system, which includes: a client and a web server, wherein:
the client is used for acquiring real-time data of the smart city and uploading the real-time data to the network server;
the network server is used for returning a message to the client through an asynchronous non-blocking network IO model after acquiring the smart city real-time data uploaded by the client from a network port; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; and coding the data in the high-speed data queue according to a preset mode, writing the data into a time sequence database, and performing CSV persistence processing.
For the principle of implementing real-time data processing of smart city, please refer to the above embodiments, which are not described herein again.
In order to achieve the above object, the present invention further provides a smart city real-time data processing system, which includes a memory, a processor, and a smart city real-time data processing program stored on the memory, wherein the smart city real-time data processing program is executed by the processor to perform the method according to the above embodiment.
For the principle of implementing real-time data processing of smart city, please refer to the above embodiments, which are not described herein again.
In order to achieve the above object, the present invention further provides a computer-readable storage medium, where a smart city real-time data processing program is stored, and when the smart city real-time data processing program is executed by a processor, the steps of the method according to the above embodiment are performed, which is not described herein again.
For the principle of implementing real-time data processing of smart city, please refer to the above embodiments, which are not described herein again.
According to the smart city real-time data processing method, the system and the storage medium, after smart city real-time data uploaded by a client are obtained from a network port, a message is returned to the client through an asynchronous non-blocking network IO model; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; the data in the high-speed data queue is encoded according to a preset mode, written into a time sequence database and subjected to CSV persistence processing, so that concurrency capacity is increased through an asynchronous non-blocking network IO mode, data throughput is improved, strong real-time data transmission delay is controlled to be in a millisecond level, requirements of low delay, stability and reliability of strong real-time data can be met, and delay problems, stability and reliability problems in the prior art are solved.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a recommendation effect evaluation system (e.g., a mobile phone, a computer, a server, a controlled terminal, or a network device) to execute the method of each embodiment of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A smart city real-time data processing method is characterized by comprising the following steps:
after acquiring smart city real-time data uploaded by a client from a network port, returning a message to the client through an asynchronous non-blocking network IO model;
decoding the smart city real-time data;
storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue;
and coding the data in the high-speed data queue according to a preset mode, writing the data into a time sequence database, and performing CSV persistence processing.
2. The smart city real-time data processing method as claimed in claim 1, wherein the step of decoding the smart city real-time data further comprises:
and checking the real-time data of the smart city, and if the real-time data passes the checking, executing the following steps: and decoding the smart city real-time data.
3. The smart city real-time data processing method as claimed in claim 2, wherein the step of verifying the smart city real-time data comprises:
and verifying the real-time data of the smart city by adopting a length verification mode and an effectiveness verification mode.
4. The smart city real-time data processing method as claimed in claim 3, wherein the step of checking the smart city real-time data by using a length check method comprises:
adopt length check-up formula, right smart city real-time data checks up, and whether the check smart city real-time data package is complete, the length check-up formula is: the packet byte number = the packet header byte number + the data byte number + the check byte number.
5. The smart city real-time data processing method as claimed in claim 3, wherein the step of verifying the smart city real-time data by validity verification comprises:
checking whether the smart city real-time data are valid or not by adopting a validity check formula, wherein the validity check formula is as follows: the check bit = performing bit and operation on all bytes from the packet header to the front of the check bit.
6. The smart city real-time data processing method as claimed in claim 1, wherein the step of decoding the smart city real-time data comprises:
extracting a data packet from the smart city real-time data;
splitting the extracted data packet into single sensor data items to be placed in the high-speed data queue;
the step of storing the decoded data in a high-speed data queue, in which the data is managed based on the sensor data items, includes:
storing each decoded data packet into the high-speed data queue in a classified manner in a sensor data item manner;
in the high-speed data queue, data is managed in a manner that a sensor organizes and manages a sub-queue based on sensor data items.
7. The smart city real-time data processing method as claimed in any one of claims 1 to 6, wherein the step of encoding the data in the high speed data queue in a predetermined manner includes:
and coding the data in the high-speed data queue according to a data block reorganizing mode of the sensor and a data block reorganizing mode required by a time sequence database and a CSV file.
8. A smart city real-time data processing system, the system comprising: a client and a web server, wherein:
the client is used for acquiring real-time data of the smart city and uploading the real-time data to the network server;
the network server is used for returning a message to the client through an asynchronous non-blocking network IO model after acquiring the smart city real-time data uploaded by the client from a network port; decoding the smart city real-time data; storing the decoded data into a high-speed data queue, and managing the data based on the sensor data items in the high-speed data queue; and coding the data in the high-speed data queue according to a preset mode, writing the data into a time sequence database, and performing CSV persistence processing.
9. A smart city real-time data processing system, comprising a memory, a processor, and a smart city real-time data processing program stored on the memory, wherein the smart city real-time data processing program is executed by the processor to perform the smart city real-time data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a smart city real-time data processing program, which when executed by a processor performs the smart city real-time data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111104385.2A CN113568587B (en) | 2021-09-22 | 2021-09-22 | Smart city real-time data processing method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111104385.2A CN113568587B (en) | 2021-09-22 | 2021-09-22 | Smart city real-time data processing method, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113568587A true CN113568587A (en) | 2021-10-29 |
| CN113568587B CN113568587B (en) | 2021-12-24 |
Family
ID=78173882
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111104385.2A Active CN113568587B (en) | 2021-09-22 | 2021-09-22 | Smart city real-time data processing method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113568587B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117076467A (en) * | 2023-10-18 | 2023-11-17 | 公诚管理咨询有限公司 | Smart city identification method, system, equipment and medium based on MA system |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203014864U (en) * | 2012-12-11 | 2013-06-19 | 黎智添 | Monitoring system based on wireless sensor network |
| WO2014202848A1 (en) * | 2013-06-20 | 2014-12-24 | Highlands Technologies Solutions | Communication method for an interactive application between terminals in a screen projection room |
| CN104317800A (en) * | 2014-09-19 | 2015-01-28 | 山东大学 | Hybrid storage system and method for mass intelligent power utilization data |
| CN104700221A (en) * | 2015-03-18 | 2015-06-10 | 浪潮集团有限公司 | Method for acquiring mass data change logs in system in SAAS (software as a service) mode |
| CN104717283A (en) * | 2015-02-13 | 2015-06-17 | 腾讯科技（深圳）有限公司 | File downloading control method, terminal and logic processing server |
| CN105426326A (en) * | 2015-11-05 | 2016-03-23 | 上海斐讯数据通信技术有限公司 | High-concurrency queue storage method and system |
| CN105760452A (en) * | 2016-02-04 | 2016-07-13 | 深圳市嘉力达实业有限公司 | Method and system for collection, processing and storage of high-concurrency mass data |
| CN106530438A (en) * | 2016-09-26 | 2017-03-22 | 浙江大学宁波理工学院 | Massive and real-time tachograph data collection method and system |
| CN111277672A (en) * | 2020-03-31 | 2020-06-12 | 上海积成能源科技有限公司 | Non-blocking input and output model-based energy Internet of things data acquisition method and software gateway |
-
2021
- 2021-09-22 CN CN202111104385.2A patent/CN113568587B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203014864U (en) * | 2012-12-11 | 2013-06-19 | 黎智添 | Monitoring system based on wireless sensor network |
| WO2014202848A1 (en) * | 2013-06-20 | 2014-12-24 | Highlands Technologies Solutions | Communication method for an interactive application between terminals in a screen projection room |
| CN104317800A (en) * | 2014-09-19 | 2015-01-28 | 山东大学 | Hybrid storage system and method for mass intelligent power utilization data |
| CN104717283A (en) * | 2015-02-13 | 2015-06-17 | 腾讯科技（深圳）有限公司 | File downloading control method, terminal and logic processing server |
| CN104700221A (en) * | 2015-03-18 | 2015-06-10 | 浪潮集团有限公司 | Method for acquiring mass data change logs in system in SAAS (software as a service) mode |
| CN105426326A (en) * | 2015-11-05 | 2016-03-23 | 上海斐讯数据通信技术有限公司 | High-concurrency queue storage method and system |
| CN105760452A (en) * | 2016-02-04 | 2016-07-13 | 深圳市嘉力达实业有限公司 | Method and system for collection, processing and storage of high-concurrency mass data |
| CN106530438A (en) * | 2016-09-26 | 2017-03-22 | 浙江大学宁波理工学院 | Massive and real-time tachograph data collection method and system |
| CN111277672A (en) * | 2020-03-31 | 2020-06-12 | 上海积成能源科技有限公司 | Non-blocking input and output model-based energy Internet of things data acquisition method and software gateway |
Non-Patent Citations (3)
| Title |
|---|
| TILKOV S: "Node.js:Using JavaScript to Build High-Performance Network Programs", 《IEEE INTERNET COMPUTING》 * |
| 戴大蒙: "基于非阻塞式 Winsock的多线程网络通信机制", 《计算机工程》 * |
| 许会元: "NodeJS的异步非阻塞I/O研究", 《工业控制计算机》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117076467A (en) * | 2023-10-18 | 2023-11-17 | 公诚管理咨询有限公司 | Smart city identification method, system, equipment and medium based on MA system |
| CN117076467B (en) * | 2023-10-18 | 2024-02-27 | 公诚管理咨询有限公司 | Smart city identification method, system, equipment and medium based on MA system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113568587B (en) | 2021-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110445653B (en) | Network state prediction method, device, equipment and medium | |
| CN102510323B (en) | Frame identifying method for serial data | |
| CN113568587B (en) | Smart city real-time data processing method, system and storage medium | |
| CN111813573B (en) | Communication method of management platform and robot software and related equipment thereof | |
| CN112822260B (en) | File transmission method and device, electronic equipment and storage medium | |
| CN111262868A (en) | Message sending method and receiving method based on intelligent network card | |
| US11671678B2 (en) | Method and device, equipment, and storage medium for data processing | |
| CN114710224A (en) | Frame synchronization method and device, computer readable medium and electronic device | |
| CN110955390B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN110795166B (en) | Data processing method and device | |
| CN110503180B (en) | Model processing method and device and electronic equipment | |
| CN114584560A (en) | Fragmented frame recombination method and device | |
| CN111553652A (en) | Service processing method and device | |
| CN113645316B (en) | Upgrade data processing method, upper computer, lower computer, device and storage medium | |
| CN114615325A (en) | Message pushing method and device, computer equipment and storage medium | |
| CN113760799B (en) | Scalable communication method, device, computer equipment and storage medium of UPP interface | |
| CN114661772B (en) | Data processing method and related device | |
| CN113691532B (en) | Parallel analysis method and device for tera-megaphone communication data based on FAST protocol | |
| CN114531269B (en) | Tbox judging method and device | |
| CN109597576B (en) | Method, device, readable storage medium and system for improving response speed of NCQ command | |
| CN110061902A (en) | A kind of method, apparatus regularly sending message and storage medium | |
| CN114827312B (en) | Method and device for self-adapting delay and throughput rate requirement in intelligent network card/DPU | |
| CN117278439B (en) | Communication quick verification method based on compression algorithm | |
| CN113905336B (en) | Text recognition-based short message sending method, device, equipment and storage medium | |
| CN115118795B (en) | Data processing method, expansion device, control system, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |