WO2020259017A1 - 时序数据处理方法、装置、系统、服务器及可读存储介质 - Google Patents

时序数据处理方法、装置、系统、服务器及可读存储介质 Download PDF

Info

Publication number
WO2020259017A1
WO2020259017A1 PCT/CN2020/084962 CN2020084962W WO2020259017A1 WO 2020259017 A1 WO2020259017 A1 WO 2020259017A1 CN 2020084962 W CN2020084962 W CN 2020084962W WO 2020259017 A1 WO2020259017 A1 WO 2020259017A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
result
queue
processing
cache queue
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/084962
Other languages
English (en)
French (fr)
Inventor
李元景
李荐民
戴俊娣
朱文涛
牛雄飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nuctech Technology (jiangsu) Co Ltd
Nuctech Co Ltd
Original Assignee
Nuctech Technology (jiangsu) Co Ltd
Nuctech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nuctech Technology (jiangsu) Co Ltd, Nuctech Co Ltd filed Critical Nuctech Technology (jiangsu) Co Ltd
Publication of WO2020259017A1 publication Critical patent/WO2020259017A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00—Packet switching elements
    • H04L49/90—Buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2474—Sequence data queries, e.g. querying versioned data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/62—Establishing a time schedule for servicing the requests

Definitions

  • the present invention relates to the field of data processing, and in particular, to a time series data processing method, device, system, server and readable storage medium.
  • the first category Big data cluster computing methods based on typical stream/batch processing frameworks such as Hadoop, Spark, Storm, etc., which are mainly suitable for high-speed processing of massive data. It has the advantages of high parallelism and high throughput, and can support thousands of units. Data processing of IoT devices. But at the same time, the big data cluster processing framework requires high resource requirements for its own operation. According to the scale effect, only when the cluster reaches a certain scale can its performance advantages be exerted. For example, a cluster framework generally requires at least three servers to build. Such an architecture is difficult to deploy, complex to implement, and costly;
  • the second category simple data processing methods implemented by programming, mainly suitable for home use or small-scale equipment, and the scale of supported equipment is less than hundreds. Due to the simple design basis, it is difficult for general program design to achieve both timing and high-performance data processing purposes, so it is usually only suitable for scenarios that process non-timing sensitive data.
  • Timing sensitivity means that in the data processing project, the processing of the data monitored at the current time must depend on the data processing results at the previous time, that is, the data monitored at the previous time must be processed before the data monitored at the current time.
  • two pieces of data may be allocated to different processing threads. Therefore, the data monitored at the current moment will often be processed before the data monitored at the last moment, resulting in complete Wrong processing result.
  • a temperature sensor is used to monitor the temperature in the truck compartment in real time.
  • Serialization can ensure the orderly processing of data, but it will inevitably fail to take advantage of multi-core computing, resulting in a greater waste of resources.
  • the existing dedicated time series database generally only focuses on storing time series data and providing data query services, and cannot directly complete the aforementioned logic processing.
  • the present invention provides a time series data processing method, device, system, server and readable storage medium.
  • a time-series data processing method including: receiving multiple pieces of to-be-processed data sent by multiple devices, the multiple pieces of to-be-processed data respectively including the time-series data collected by the multiple devices and the The identification codes of multiple devices; according to the grouping results of the multiple devices, the multiple pieces of to-be-processed data are respectively imported into the corresponding task queue for sequential processing, and multiple pieces of result data are correspondingly obtained; and the result The data is sequentially stored in the first buffer queue.
  • the grouping result is obtained according to the following steps: inputting the identification code into a pre-constructed hash function, and using the output result of the hash function as the identification code corresponding device Group the results.
  • the multiple pieces of to-be-processed data are respectively imported into the corresponding task queue for sequential processing, and correspondingly obtaining multiple pieces of result data includes: Grouping results, importing the time series data in the multiple pieces of to-be-processed data into the task queues of the corresponding thread pools, each of the thread pools includes a processing thread; and in each of the thread pools , By sequentially processing the time series data imported into the task queue of the thread pool by the processing thread, and correspondingly obtaining the multiple pieces of result data.
  • the method further includes: generating a second cache queue according to the first cache queue, the result data stored in the second cache queue and the current storage in the first cache queue The result data of are completely the same; the result data in the second cache queue is sent to the database; and when the second cache queue is empty, the second cache queue is deleted.
  • the new result data obtained through the sequential processing is received and stored in the first buffer queue.
  • a time-series data processing device including: a data receiving module, configured to receive multiple pieces of to-be-processed data sent by multiple devices, and the multiple pieces of to-be-processed data respectively include the multiple devices The collected time series data and the identification codes of the multiple devices; the data processing module is used to import the multiple pieces of data to be processed into the corresponding task queues for sequential processing according to the grouping results of the multiple devices , Correspondingly obtain multiple pieces of result data; and a data buffer module, configured to sequentially store the result data in the first buffer queue.
  • the device further includes: a queue generation module, configured to generate a second cache queue according to the first cache queue, the result data stored in the second cache queue and the The result data currently stored in the first buffer queue is completely the same; a data sending module is used to send the result data in the second buffer queue to the database; and a queue deletion module is used to serve as the second buffer When the queue is empty, delete the second cache queue.
  • a queue generation module configured to generate a second cache queue according to the first cache queue, the result data stored in the second cache queue and the The result data currently stored in the first buffer queue is completely the same
  • a data sending module is used to send the result data in the second buffer queue to the database
  • a queue deletion module is used to serve as the second buffer When the queue is empty, delete the second cache queue.
  • a server including: a memory, a processor, and executable instructions stored in the memory and capable of being run in the processor, and the processor executes the executable instructions Realize any of the above-mentioned sequential data processing methods.
  • a time-series data processing system including: a plurality of devices, a server according to the above, and a database; wherein the server receives a plurality of pieces of to-be-processed data from the The pieces of result data are sent to the database.
  • a computer-readable storage medium having computer-executable instructions stored thereon, and when the executable instructions are executed by a processor, any one of the above-mentioned sequential data processing methods is implemented.
  • time series data processing method based on the grouping result of the device, high-performance time series processing of large batches of data can be realized in parallel, and the advantages of multi-core computing can be exerted.
  • a single machine can process the data processing of tens of thousands of devices, avoiding the establishment of complex computer groups, and significantly reducing deployment costs.
  • the time series data processing method provided by the present invention adopts mutually independent data caching technology and data transmission technology, which can continuously and stably realize the batch transmission and accurate storage of massive time series result data.
  • Fig. 1 is a flow chart showing a method for processing time series data according to an exemplary embodiment.
  • Fig. 2 is a flowchart showing another method for processing time series data according to an exemplary embodiment.
  • Fig. 3 is a flowchart showing yet another method for processing time series data according to an exemplary embodiment.
  • Fig. 4 is a block diagram showing a sequential data processing device according to an exemplary embodiment.
  • Fig. 5 is a schematic diagram showing the structure of a server according to an exemplary embodiment.
  • Fig. 6 is a schematic structural diagram showing a time series data processing system according to an exemplary embodiment.
  • Fig. 7 is a schematic diagram showing a computer-readable storage medium according to an exemplary embodiment.
  • Fig. 8 is a schematic diagram showing the flow of time series data according to an example.
  • Fig. 9 is a schematic diagram of scheduling and processing time series data according to an example.
  • Fig. 10 is a schematic diagram of storing result data according to an example.
  • first and second are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined with “first” and “second” may explicitly or implicitly include one or more of these features. In the description of the present invention, "a plurality of” means at least two, such as two, three, etc., unless otherwise specifically defined.
  • the second type of data processing method can easily realize the serial processing of time series data, but it cannot take advantage of the multi-core operation and seriously waste computer resources;
  • the first type of data processing method can process time series data in parallel, but it is difficult to deploy the architecture Large, complicated to implement and high cost.
  • the present invention proposes a time series data processing method that combines the advantages of the above two types of methods.
  • the time series data processing method provided by the embodiments of the present invention based on the grouping result of the device, high-performance time series processing of large batches of data can be realized in parallel, and the advantages of multi-core computing can be exerted.
  • a single machine can process the data processing of tens of thousands of devices, avoiding the establishment of complex computer groups, and significantly reducing deployment costs.
  • Fig. 8 is a schematic diagram showing the flow of time series data according to an example.
  • the time series data collected by a terminal device in real time can be first received by the data receiver/dispatcher module in the server, and transferred to the data processor module in the server for corresponding processing according to the configuration strategy. unit.
  • the data processor module logically processes the sequence data, it can temporarily store the obtained result data in the data buffer module in the server, and the data submitter module in the server obtains the result data in batches and submits it to the database system for storage . Later, through the data interface, operations such as calling, exporting, querying, testing, and statistics can be performed on the result data stored in the database system.
  • Fig. 9 is a schematic diagram of scheduling and processing time series data according to an example.
  • the data receiver/scheduler module provides configuration strategies for terminal devices, which can group n devices, and continuously and stably import time series data collected from the same device at different times into the same thread pool in the data processor module;
  • Fig. 10 is a schematic diagram of storing result data according to an example.
  • the data processor module processes time series data
  • the result data obtained will finally be submitted to the database system through the data submitter module.
  • As a "transit" data buffer module it can provide two data queues, a buffer queue and a generation based on the buffer queue
  • the temporary queues can respectively continuously and independently perform dynamic caching and dynamic submission of result data.
  • Fig. 1 is a flow chart showing a method for processing time series data according to an exemplary embodiment.
  • the time series data processing method shown in FIG. 1 can be applied to the server side in the data processing scenario of the Internet of Things, for example, the server composed of a data receiver/scheduler, a data processor, a data buffer, and a data submitter in FIG. 8.
  • the time series data processing method 10 includes:
  • step S102 multiple pieces of to-be-processed data sent by multiple devices are received.
  • the multiple pieces of to-be-processed data respectively include time series data collected by multiple devices and identification codes of multiple devices.
  • IoT devices such as physical sensors can collect time series data with corresponding physical quantity indicators that change over time and wait for processing.
  • the server receives the to-be-processed data sent by each IoT device.
  • the structure of the to-be-processed data also includes a fixed unique identification code corresponding to the IoT device.
  • the information interaction between the IoT device and the server can be achieved through protocols such as MQTT (Message Queuing Telemetry Transport), HTTP (HyperText Transfer Protocol), TCP (Transmission Control Protocol, Transmission Control Protocol), etc. Transmission to achieve.
  • MQTT Message Queuing Telemetry Transport
  • HTTP HyperText Transfer Protocol
  • TCP Transmission Control Protocol, Transmission Control Protocol
  • step S104 according to the grouping results of multiple devices, multiple pieces of to-be-processed data are respectively imported into corresponding task queues for sequential processing, and multiple pieces of result data are correspondingly obtained.
  • the grouping result is obtained according to the following steps: inputting the identification code into a pre-built hash function, and taking the output result of the hash function as the grouping result of the device corresponding to the identification code.
  • the balanced hash function can satisfy the random sampling of the keyword key or the keyword key randomly generated according to the distribution characteristics of the KEY space.
  • the probability of entering any one of the n hash addresses is 1/n; the idempotent hash function It can be satisfied that the same result can be obtained when the function is repeatedly executed with the same parameter key, and the repeated execution of the function will not change the system state.
  • Identity function and constant function are the two simplest idempotent functions.
  • the grouping results of multiple devices can be determined based on the hash function being allocated in advance once, and it is not necessary to repeatedly execute the hash function every time sequence data is received and processed; It is determined by repeatedly executing the hash function when receiving and processing time series data: Since the pre-constructed hash function has the nature of an idempotent function, repeated execution can ensure that the grouping result of each time remains unchanged; it can also be based on actual needs To adjust the structure of the hash function at any time to change any one or more device grouping results, the present invention is not limited to this.
  • the above grouping is only illustrative, and the present invention is not
  • step S106 the result data are sequentially stored in the first buffer queue.
  • time series data processing method based on the grouping result of the device, high-performance time series processing of large batches of data can be realized in parallel, and the advantages of multi-core computing can be exerted.
  • a single machine can process the data processing of tens of thousands of devices, avoiding the establishment of complex computer groups, and significantly reducing deployment costs.
  • Fig. 2 is a flowchart showing another method for processing time series data according to an exemplary embodiment.
  • the difference from the method 10 shown in FIG. 1 is that the method 20 shown in FIG. 2 further provides a method for sequentially processing the data to be processed to obtain the result data according to the device grouping result, that is, the step S104 in the above method 10 One embodiment.
  • the time series data processing method shown in FIG. 2 can also be applied to the server side in the data processing scenario of the Internet of Things, for example.
  • step S104 in method 10 further includes:
  • step S202 according to the grouping result, the time series data in the multiple pieces of to-be-processed data are respectively imported into the task queue of the corresponding thread pool.
  • each thread pool includes a processing thread.
  • step S204 in each thread pool, the time series data imported into the task queue of the thread pool are sequentially processed by processing threads, and multiple pieces of result data are correspondingly obtained.
  • each thread pool on the server side reserves only one unique processing thread.
  • the unique processing thread can be based on different things. The order in which the networked devices collect/send time series data is queued to ensure the orderliness of the thread pool when processing the time series data from multiple IoT devices.
  • Fig. 3 is a flowchart showing yet another method for processing time series data according to an exemplary embodiment.
  • the difference from the method 10 shown in FIG. 1 is that the method 30 shown in FIG. 3 further provides a method for storing the result data in a database.
  • the time series data processing method shown in FIG. 3 can also be applied to the server side in the data processing scenario of the Internet of Things, for example.
  • the time series data processing method 30 includes:
  • step S302 a second cache queue is generated according to the first cache queue.
  • the result data stored in the second cache queue is completely the same as the result data currently stored in the first cache queue.
  • the server side sequentially stores the result data in the first cache queue, for example, in a cache queue named QUEUE-RESULT . While storing the result data in QUEUE-RESULT sequentially, the sending, receiving and processing of time series data can be carried out dynamically and continuously.
  • the server In the process of storing the result data in the database, the server continuously processes the time series data collected/sent by multiple IoT devices at different times, and then stores the obtained result data in QUEUE-RESULT; at the same time, it also stores The result data in QUEUE-RESULT is sent in batches and stored in the database, so it is easy to cause data disorder or even loss.
  • the server can generate a second cache queue according to the first cache queue, so that two different cache queues are processed at the same time on the server side, and the process of caching the result data and transmitting the result data is independent of each other. Do not interfere with each other, avoiding the disorder and loss of data.
  • generating the second cache queue according to the first cache queue may be implemented by duplicating the first cache queue in real time, or by renaming the first cache queue.
  • renaming the first cache queue for example, QUEUE-RESULT is renamed QUEUE-TEMP, that is, only the pointer to the first address of the array formed by the result data in real time is modified to ensure that the second cache queue QUEUE-TEMP is stored
  • the result data is exactly the same as the structure and content of the result data currently stored in the first buffer queue QUEUE-RESULT.
  • step S304 the result data in the second buffer queue is sent to the database.
  • a database storing massive results data can provide services such as Internet of Things data query, statistics, and report generation.
  • step S306 when the second cache queue is empty, the second cache queue is deleted.
  • the method 30 further includes: receiving new result data obtained through sequential processing and storing it in the first buffer queue.
  • the server side transmits and stores all the result data in the second cache queue QUEUE-TEMP to the database, that is, when the QUEUE-TEMP becomes empty, the server side deletes the QUEUE-TEMP immediately.
  • the current first buffer queue QUEUE-RESULT has stored the new result data obtained through sequential processing. Repeat the above steps S302-S306, and generate a second buffer queue every cycle, which can continuously and stably realize the batch transmission and accurate storage of massive timing result data.
  • a computer program executed by a CPU.
  • the computer program executes the aforementioned functions defined by the aforementioned method provided by the present invention.
  • the program can be stored in a computer-readable storage medium, which can be a read-only memory, a magnetic disk, or an optical disk.
  • Fig. 4 is a block diagram showing a sequential data processing device according to an exemplary embodiment.
  • the time series data processing device 40 includes: a data receiving module 402, a data processing module 404 and a data buffering module 406.
  • the data receiving module 402 is used to receive multiple pieces of to-be-processed data sent by multiple devices.
  • the multiple pieces of to-be-processed data respectively include time series data collected by multiple devices and identification codes of multiple devices.
  • the data processing module 404 is configured to import multiple pieces of to-be-processed data into corresponding task queues for sequential processing according to the grouping results of multiple devices, and correspondingly obtain multiple pieces of result data.
  • the data buffer module 406 is used for storing the result data in the first buffer queue in sequence.
  • the data processing module 404 may further include a data import unit and a data processing unit.
  • the data import unit is used for importing multiple pieces of time series data in the to-be-processed data into the task queue of the corresponding thread pool according to the grouping result.
  • Each thread pool includes a processing thread; the data processing unit is used in each thread pool to sequentially process the time series data imported into the task queue of the thread pool through the processing threads, and correspondingly obtain multiple pieces of result data.
  • a virtual machine with a 4-core CPU, 16GB of memory, and the above-mentioned data receiving module 402, data processing module 404, and data caching module 406 can process up to 12,000 IoT devices at a frequency of 30 seconds/time
  • the timing data sent, and the processing logic includes complex logic such as correlation timing judgment.
  • the time series data processing device 40 may further include: a queue generating module 408, a data sending module 410, and a queue deleting module 412.
  • the queue generation module 408 is configured to generate a second cache queue according to the first cache queue.
  • the result data stored in the second cache queue is exactly the same as the result data currently stored in the first cache queue.
  • the data sending module 410 is configured to send the result data in the second buffer queue to the database.
  • the queue deletion module 412 is configured to delete the second cache queue when the second cache queue is empty.
  • the data caching module 406 may also be used to receive the new result data obtained through sequential processing by the data processing module 404 and store it in the first cache queue.
  • the sequential data processing device provided by the embodiment of the present invention, based on the grouping result of the device, high-performance sequential processing of large batches of data can be realized in parallel, and the advantages of multi-core computing can be exerted.
  • a single machine can process the data processing of tens of thousands of devices, avoiding the construction of complex computer groups, and significantly reducing deployment costs.
  • the time series data processing device provided by the present invention adopts mutually independent data caching technology and data transmission technology, which can continuously and stably realize batch transmission and precise storage of massive time series result data.
  • Fig. 5 is a schematic diagram showing the structure of a server according to an exemplary embodiment. It should be noted that the server shown in FIG. 5 is only an example, and should not bring any limitation to the function and scope of use of the embodiment of the present invention. The server shown in FIG. 5 can be applied to, for example, an IoT data processing scenario.
  • the server 100 is represented in the form of a general-purpose computer device.
  • the components of the server 100 include: at least one central processing unit (CPU) 1001, which can be loaded into a random access memory (RAM) 1003 according to program codes stored in a read only memory (ROM) 1002 or from at least one storage unit 1008
  • the program code executes various appropriate actions and processing.
  • the program code may be executed by the central processing unit 1001, so that the central processing unit 1001 executes the steps according to various exemplary embodiments of the present invention described in the above method embodiment section of this specification.
  • the central processing unit 1001 may execute the steps shown in FIGS. 1, 2, and 3.
  • the RAM 1003 also stores various programs and data required for the operation of the server 100.
  • the CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004.
  • An input/output (I/O) interface 1005 is also connected to the bus 1004.
  • the following components are connected to the I/O interface 1005: an input unit 1006 including a keyboard, a mouse, etc.; an output unit 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and speakers, etc.; a storage unit 1008 including a hard disk, etc. ; And a communication unit 1009 including a network interface card such as a LAN card, a modem, and the like. The communication unit 1009 performs communication processing via a network such as the Internet.
  • the driver 1010 is also connected to the I/O interface 1005 as needed.
  • a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that the computer program read from it can be installed in the storage unit 1008 as needed.
  • Fig. 6 is a schematic structural diagram showing a time series data processing system according to an exemplary embodiment.
  • the time series data processing system shown in FIG. 6 can be applied to, for example, an IoT data processing scenario.
  • the time series data processing system 60 includes a plurality of devices 602, the aforementioned server 100, and a database 604.
  • the server 100 is configured to receive multiple pieces of data to be processed from multiple devices 602 and send multiple pieces of result data to the database 604.
  • Fig. 7 is a schematic diagram showing a computer-readable storage medium according to an exemplary embodiment.
  • a program product 700 configured to implement the above method according to an embodiment of the present invention is described. It can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be installed in a terminal device, For example, running on a personal computer.
  • the program product of the present invention is not limited to this.
  • the readable storage medium can be any tangible medium that contains or stores a program.
  • the program can be used by or in combination with an instruction execution system, device, or device.
  • the above-mentioned computer-readable medium carries one or more programs.
  • the computer-readable medium realizes the functions shown in FIGS. 1, 2, and 3.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Mathematical Physics (AREA)
  • Fuzzy Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本发明公开一种时序数据处理方法、装置、系统、服务器及可读存储介质。所述方法包括：接收多个设备发送的多条待处理数据，所述多条待处理数据分别包括所述多个设备采集的时序数据以及所述多个设备的标识码；根据对所述多个设备的分组结果，将所述多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据；以及将所述结果数据依序存储至第一缓存队列中。根据本发明提供的时序数据处理方法，基于设备的分组结果，能够并行地实现大批量数据的高性能时序处理，发挥多核运算的优势。同时，单机即可处理上万个设备的数据处理，避免搭建复杂的计算机群，显著降低了部署成本。

Description

时序数据处理方法、装置、系统、服务器及可读存储介质 技术领域
本发明涉及数据处理领域，具体而言，涉及一种时序数据处理方法、装置、系统、服务器及可读存储介质。
背景技术
随着物联网技术的发展，物联网数据的处理量和处理难度越来越大。当前，物联网领域常用的数据处理方法主要分为以下两类：
第一类：基于Hadoop、Spark、Storm等典型流/批处理框架的大数据集群计算方法，主要适用于海量数据的高速处理，具有高并行、高吞吐量等优点，可支持成千上万台物联网设备的数据处理。但同时，大数据集群处理框架对自身运行所需的资源要求较高。根据规模效应，只有当集群达到一定的规模才能发挥出其性能优势，比如集群框架一般需要至少三台服务器才能搭建。这样的架构部署难度大，实现复杂且成本较高；
第二类：通过编程实现的简单数据处理方法，主要适用于家用或设备规模较小等场景，可支持的设备规模在数百台以内。由于设计基础简单，一般的程序设计很难达到既保证时序性，又保证高性能的数据处理目的，因此通常仅适用于处理非时序敏感数据的场景。
时序敏感是指在数据处理工程中，处理当前时刻监测到的数据必须依赖于上一时刻的数据处理结果，也即上一时刻监测到的数据必须在当前时刻监测到的数据之前处理完成。在现有的多线程模型中，两条数据有可能会被分配给不同的处理线程，因此往往会出现当前时刻监测到的数据会比上一时刻监测到的数据先处理的情况，进而导致完全错误的处理结果。例如，在冷链物流业务中使用温度传感器实时监测货车车厢内的温度，当温度不低于20℃时需要发送预警提示，但当温度连续不低于20℃时只需发送一次预警提示：假设连续5个时刻监测到的温度为17℃、19℃、20℃、19℃、21℃，按照前述业务规则，正常情况下将在第3、第5时刻各触发一次预警提示。然而，当不能确保数据按时间顺序有序处理时，例如第4时刻的数据比第3时刻的数据先处理，则仅在处理第3时刻数据(20℃)时触发一次预警提示，从而导致运输人员延迟发觉车厢温控系统的失灵。
串行化可以确保实现数据的有序处理，但势必无法发挥多核运算的优势，造成较大的资源浪费。此外，现有的专用时序数据库一般也只是侧重于存储时序数据并提供数据查询服务，并不能直接完成前述的逻辑处理。
在所述背景技术部分公开的上述信息仅用于加强对本发明的背景的理解，因此它可以包括不构成对本领域普通技术人员已知的现有技术的信息。
发明内容
有鉴于此，本发明提供一种时序数据处理方法、装置、系统、服务器及可读存储介质。
本发明的其他特性和优点将通过下面的详细描述变得显然，或部分地通过本发明的实践而习得。
根据本发明的一方面，提供一种时序数据处理方法，包括：接收多个设备发送的多条待处理数据，所述多条待处理数据分别包括所述多个设备采集的时序数据以及所述多个设备的标识码；根据对所述多个设备的分组结果，将所述多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据；以及将所述结果数据依序存储至第一缓存队列中。
根据本发明的一实施方式，所述分组结果根据下述步骤获得：将所述标识码输入预构造的哈希函数，以所述哈希函数的输出结果作为所述标识码对应设备的所述分组结果。
根据本发明的一实施方式，根据对所述多个设备的分组结果，将所述多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据包括：根据所述分组结果，将所述多条待处理数据中的所述时序数据分别导入相应的线程池的所述任务队列中，每个所述线程池包括一个处理线程；以及在每个所述线程池中，通过所述处理线程依次处理导入所述线程池的所述任务队列中的所述时序数据，对应获得所述多条结果数据。
根据本发明的一实施方式，所述方法还包括：根据所述第一缓存队列，生成第二缓存队列，所述第二缓存队列中存储的所述结果数据与所述第一缓存队列当前存储的所述结果数据完全相同；将所述第二缓存队列中的所述结果数据发送至数据库；以及当所述第二缓存队列为空时，删除所述第二缓存队列。
根据本发明的一实施方式，接收经所述依序处理获得的新的所述结果数据，并存 储至所述第一缓存队列中。
根据本发明的另一方面，提供一种时序数据处理装置，包括：数据接收模块，用于接收多个设备发送的多条待处理数据，所述多条待处理数据分别包括所述多个设备采集的时序数据以及所述多个设备的标识码；数据处理模块，用于根据对所述多个设备的分组结果，将所述多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据；以及数据缓存模块，用于将所述结果数据依序存储至第一缓存队列中。
根据本发明的一实施方式，所述装置还包括：队列生成模块，用于根据所述第一缓存队列，生成第二缓存队列，所述第二缓存队列中存储的所述结果数据与所述第一缓存队列当前存储的所述结果数据完全相同；数据发送模块，用于将所述第二缓存队列中的所述结果数据发送至数据库；以及队列删除模块，用于当所述第二缓存队列为空时，删除所述第二缓存队列。
根据本发明的再一方面，提供一种服务器，包括：存储器、处理器及存储在所述存储器中并可在所述处理器中运行的可执行指令，所述处理器执行所述可执行指令时实现上述任一种时序数据处理方法。
根据本发明的再一方面，提供一种时序数据处理系统，包括：多个设备、根据上述的服务器以及数据库；其中，所述服务器从所述多个设备接收多条待处理数据，并将多条结果数据发送至所述数据库。
根据本发明的再一方面，提供一种计算机可读存储介质，其上存储有计算机可执行指令，所述可执行指令被处理器执行时实现上述任一种时序数据处理方法。
根据本发明实施方式提供的时序数据处理方法，基于设备的分组结果，能够并行地实现大批量数据的高性能时序处理，发挥多核运算的优势。同时，根据本发明实施方式提供的时序数据处理方法，单机即可处理上万个设备的数据处理，避免搭建复杂的计算机群，显著降低了部署成本。
另外，根据一些实施例，本发明提供的时序数据处理方法采用相互独立的数据缓存技术和数据传送技术，能够连续、稳定地实现海量时序结果数据的批量传送和精准存储。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性的，并不能限制本发明。
附图说明
通过参照附图详细描述其示例实施例，本发明的上述和其它目标、特征及优点将变得更加显而易见。
图1是根据一示例性实施方式示出的一种时序数据处理方法的流程图。
图2是根据一示例性实施方式示出的另一种时序数据处理方法的流程图。
图3是根据一示例性实施方式示出的再一种时序数据处理方法的流程图。
图4是根据一示例性实施方式示出的一种时序数据处理装置的框图。
图5是根据一示例性实施方式示出的一种服务器的结构示意图。
图6是根据一示例性实施方式示出的一种时序数据处理系统的结构示意图。
图7是根据一示例性实施方式示出的一种计算机可读存储介质的示意图。
图8是根据一示例示出的一种时序数据的流向示意图。
图9是根据一示例示出的一种调度并处理时序数据的示意图。
图10是根据一示例示出的一种存储结果数据的示意图。
具体实施方式
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本发明将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。附图仅为本发明的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。
此外，所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。在下面的描述中，提供许多具体细节从而给出对本发明的实施方式的充分理解。然而，本领域技术人员将意识到，可以实践本发明的技术方案而省略所述特定细节中的一个或更多，或者可以采用其它的方法、装置、步骤等。在其它情况下，不详细示出或描述公知结构、方法、装置、实现或者操作以避免喧宾夺主而使得本发明的各方面变得模糊。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本发明的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
如上所述，第二类数据处理方法可容易地实现时序数据的串行处理，但无法发挥多核运算的优势，严重浪费计算机资源；第一类数据处理方法可以并行处理时序数据，但架构部署难度大，实现复杂且成本高。
因此，本发明提出一种综合了上述两类方法优点的时序数据处理方法。根据本发明实施方式提供的时序数据处理方法，基于设备的分组结果，能够并行地实现大批量数据的高性能时序处理，发挥多核运算的优势。同时，根据本发明实施方式提供的时序数据处理方法，单机即可处理上万个设备的数据处理，避免搭建复杂的计算机群，显著降低了部署成本。
图8是根据一示例示出的一种时序数据的流向示意图。如图8中箭头指向所示，一个终端设备实时采集到的时序数据可以首先被服务器中的数据接收/调度器模块接收，并依配置策略被调拨至服务器中的数据处理器模块中相应的处理单元。数据处理器模块对时序数据进行逻辑处理后，可以将获得的结果数据暂存于服务器中的数据缓存器模块，由服务器中的数据提交器模块从中批量获取结果数据并提交至数据库系统中进行存储。后续通过数据接口，可以对数据库系统中存储的结果数据执行调用、导出、查询、测试、统计等操作。
本发明整体的方案即是根据如图8所示的时序数据流向所提出的。参考图9及图10，可以将时序数据的完整流向划分为前后两部分，以体现本发明整体方案所作出的具体创新：
图9是根据一示例示出的一种调度并处理时序数据的示意图。数据接收/调度器模块提供针对终端设备的配置策略，可对n个设备进行分组，并将来自同一设备在不同时刻采集的时序数据持续、稳定地导入数据处理器模块中的同一线程池；
图10是根据一示例示出的一种存储结果数据的示意图。考虑到数据处理器模块处理时序数据，所获得的结果数据最终要经数据提交器模块提交至数据库系统，作为“中转”的数据缓存器模块可以提供两个数据队列，缓存队列及根据缓存队列生成的临时队列可分别连续、独立地执行结果数据的动态缓存和动态提交。
图1是根据一示例性实施方式示出的一种时序数据处理方法的流程图。如图1所示的时序数据处理方法例如可以应用于物联网数据处理场景中的服务器端，如图8中由数据接收/调度器、数据处理器、数据缓存器及数据提交器组成的服务器。
参考图1，时序数据处理方法10包括：
在步骤S102中，接收多个设备发送的多条待处理数据。
其中，多条待处理数据分别包括多个设备采集的时序数据以及多个设备的标识码。
在物联网数据处理场景中，例如物理传感器等物联网设备可采集对应物理量指标随着时间不断变化的时序数据等待处理。服务器端接收到各物联网设备发送的待处理数据，待处理数据的结构体中除了物联网设备采集到的时序数据外，还包括对应物联网设备固定不变的唯一标识码。物联网设备与服务器端间的信息交互可以通过例如MQTT(Message Queuing Telemetry Transport，消息队列遥测传输)、HTTP(HyperText Transfer Protocol，超文本传输协议)、TCP(Transmission Control Protocol，传输控制协议)等协议传输来实现。
在步骤S104中，根据对多个设备的分组结果，将多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据。
在一些实施例中，分组结果根据下述步骤获得：将标识码输入预构造的哈希函数，以哈希函数的输出结果作为标识码对应设备的分组结果。
承上述，假设服务器端可处理m个物联网设备采集的时序数据，各设备的唯一标识码分别为S1、S2、S3、……、Sm。预先构造一个均衡且幂等的哈希函数addr＝h(key)：
均衡的哈希函数可满足随机采样得到的关键字key或者按照KEY空间的分布特性随机产生的关键字key进入n个哈希地址中任意一个的概率均为1/n；幂等的哈希函数可满足使用相同参数key重复执行函数时获得相同的结果，函数的重复执行不会使系统状态产生变化。恒等函数和常数函数即为两种最简单的幂等函数。
预构造哈希函数addr＝h(key)可将关键字key和它的存储位置addr之间建立一个确定的对应关系，使每个关键字和一个唯一的存储位置相对应。因此，均衡且幂等的哈希函数addr＝h(key)可实现将一定分布的流标识映射到不同的哈希地址中，例如将上述m个物联网设备的标识码分为n组映射到n个哈希地址中，进一步保证了在后续过程中来自同一个物联网设备的时序数据每次都能进入到服务器端的同一个线程池中进行处理，每个线程池对应匹配的物联网设备数均约等于m/n。
在本发明的实施方式中，多个设备的分组结果可以是基于哈希函数经一次预先分配而确定的，不必在每一次接收和处理时序数据时重复执行哈希函数；也可以是在每一次接收和处理时序数据时重复执行哈希函数而确定的：由于预构造的哈希函数具备幂等函数的性质，因此将其重复执行可以确保每次的分组结果保持不变；还可以根据 实际需求，随时调整哈希函数的构造以改变任意一次或多次的设备分组结果，本发明并不以此为限。
分组数n的取值例如可以参考服务器端的CPU核心数量取经验值：假设m＝6，n＝3，一种最简单的分组情况为：h(S1)＝0、h(S2)＝1、h(S3)＝2、h(S4)＝0、h(S5)＝1、h(S6)＝2，即将6个物联网设备均匀地分为3组，其中物联网设备S1和S4对应于第一个哈希地址，来自S1和S4的时序数据每次都进入到服务器端的第一个线程池中进行处理；物联网设备S2和S5对应于第二个哈希地址，来自S2和S5的时序数据每次都进入到服务器端的第二个线程池中进行处理；物联网设备S3和S6对应于第三个哈希地址，来自S3和S6的时序数据每次都进入到服务器端的第三个线程池中进行处理。以上分组仅作示例性说明，本发明不以其中的设备数、分组数及分组结果为限。
在步骤S106中，将结果数据依序存储至第一缓存队列中。
根据本发明实施方式提供的时序数据处理方法，基于设备的分组结果，能够并行地实现大批量数据的高性能时序处理，发挥多核运算的优势。同时，根据本发明实施方式提供的时序数据处理方法，单机即可处理上万个设备的数据处理，避免搭建复杂的计算机群，显著降低了部署成本。
应清楚地理解，本发明描述了如何形成和使用特定示例，但本发明的原理不限于这些示例的任何细节。相反，基于本发明公开的内容的教导，这些原理能够应用于许多其它实施方式。
图2是根据一示例性实施方式示出的另一种时序数据处理方法的流程图。与图1所示方法10的不同之处在于，图2所示的方法20进一步提供了根据设备分组结果，依序处理待处理数据以对应获得结果数据的方法，即上述方法10中步骤S104的一实施例。同样地，如图2所示的时序数据处理方法例如也可以应用于物联网数据处理场景中的服务器端。
参考图2，方法10中的步骤S104还包括：
在步骤S202中，根据分组结果，将多条待处理数据中的时序数据分别导入相应的线程池的任务队列中。
其中，每个线程池包括一个处理线程。
在步骤S204中，在每个线程池中，通过处理线程依次处理导入线程池的任务队列中的时序数据，对应获得多条结果数据。
在一些实施例中，服务器端中的每个线程池仅保留一个唯一的处理线程，当一个线程池需接收多个物联网设备发送的多条待处理数据时，唯一的处理线程可依不同物联网设备采集/发送时序数据的先后顺序进行排队处理，保证了线程池处理来自多个物联网设备的时序数据时的有序性。
图3是根据一示例性实施方式示出的再一种时序数据处理方法的流程图。与图1所示方法10的不同之处在于，图3所示的方法30进一步提供了将结果数据存储至数据库的方法。同样地，如图3所示的时序数据处理方法例如也可以应用于物联网数据处理场景中的服务器端。
参考图3，时序数据处理方法30包括：
在步骤S302中，根据第一缓存队列，生成第二缓存队列。
其中，第二缓存队列中存储的结果数据与第一缓存队列当前存储的结果数据完全相同。
仍以上述物联网数据处理场景为例进行说明：在对应获得多条结果数据后，服务器端将结果数据依序存储至第一缓存队列中，例如存储至一个名为QUEUE-RESULT的缓存队列中。在将结果数据依序存入QUEUE-RESULT的同时，时序数据的发送、接收和处理可动态、持续地进行。
在将结果数据存储至数据库的过程中，服务器端源源不断地对多个物联网设备于不同时刻采集/发送的时序数据进行处理后，将获得的结果数据存储至QUEUE-RESULT；同时，又将QUEUE-RESULT中的结果数据批量传送并存储至数据库，因此极易导致数据紊乱甚至丢失。在本发明的实施方式中，服务器端可根据第一缓存队列生成第二缓存队列，使服务器端中同时进行处理的是两个不同的缓存队列，缓存结果数据与传送结果数据的过程相互独立、互不干扰，避免了数据的紊乱和丢失。
在一些实施例中，根据第一缓存队列生成第二缓存队列可以通过实时复制第一缓存队列，或者通过对第一缓存队列进行改名实现。通过对第一缓存队列进行改名的方式，例如将QUEUE-RESULT改名为QUEUE-TEMP，即仅修改指向结果数据实时形成的数组的首地址的指针，保证了第二缓存队列QUEUE-TEMP中存储的结果数据与第一缓存队列QUEUE-RESULT当前存储的结果数据的结构及内容均完全相同。
在步骤S304中，将第二缓存队列中的结果数据发送至数据库。
存储有海量结果数据的数据库例如可以提供物联网数据查询、统计及报表生成等 服务。
在步骤S306中，当第二缓存队列为空时，删除第二缓存队列。
在一些实施例中，方法30还包括：接收经依序处理获得的新的结果数据，并存储至第一缓存队列中。
承上述，服务器端将第二缓存队列QUEUE-TEMP中的结果数据全部传送并存储至数据库后，也即QUEUE-TEMP变为空时，服务器端随即将QUEUE-TEMP删除。与此同时，当前第一缓存队列QUEUE-RESULT又已存入了经依序处理获得的新的结果数据。重复上述步骤S302-S306，每一个周期生成一个第二缓存队列，可连续、稳定地实现海量时序结果数据的批量传送和精准存储。
本领域技术人员可以理解实现上述实施方式的全部或部分步骤被实现为由CPU执行的计算机程序。在该计算机程序被CPU执行时，执行本发明提供的上述方法所限定的上述功能。所述的程序可以存储于一种计算机可读存储介质中，该存储介质可以是只读存储器，磁盘或光盘等。
此外，需要注意的是，上述附图仅是根据本发明示例性实施方式的方法所包括的处理的示意性说明，而不是限制目的。易于理解，上述附图所示的处理并不表明或限制这些处理的时间顺序。另外，也易于理解，这些处理可以是例如在多个模块中同步或异步执行的。
下述为本发明装置实施例，可以用于执行本发明方法实施例。对于本发明装置实施例中未披露的细节，请参照本发明方法实施例。
图4是根据一示例性实施方式示出的一种时序数据处理装置的框图。
参考图4，时序数据处理装置40包括：数据接收模块402、数据处理模块404以及数据缓存模块406。
其中，数据接收模块402用于接收多个设备发送的多条待处理数据。
其中多条待处理数据分别包括多个设备采集的时序数据以及多个设备的标识码。
数据处理模块404用于根据对多个设备的分组结果，将多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据。
数据缓存模块406用于将结果数据依序存储至第一缓存队列中。
在一些实施例中，数据处理模块404可进一步包括数据导入单元以及数据处理单元。其中，数据导入单元用于根据分组结果，将多条待处理数据中的时序数据分别导 入相应的线程池的任务队列中。其中每个线程池包括一个处理线程；数据处理单元用于在每个线程池中，通过处理线程依次处理导入线程池的任务队列中的时序数据，对应获得多条结果数据。
在一实施例中，一个具有4核CPU、16GB内存以及上述数据接收模块402、数据处理模块404和数据缓存模块406的虚拟机，可处理多达12000个物联网设备以30秒/次的频率发送的时序数据，且处理逻辑包括关联时序判断等复杂逻辑。
在一些实施例中，时序数据处理装置40还可包括：队列生成模块408、数据发送模块410以及队列删除模块412。
其中，队列生成模块408用于根据第一缓存队列，生成第二缓存队列。其中第二缓存队列中存储的结果数据与第一缓存队列当前存储的结果数据完全相同。
数据发送模块410用于将第二缓存队列中的结果数据发送至数据库。
队列删除模块412用于当第二缓存队列为空时，删除第二缓存队列。
在一些实施例中，数据缓存模块406还可用于接收经数据处理模块404依序处理获得的新的结果数据，并存储至第一缓存队列中。
根据本发明实施方式提供的时序数据处理装置，基于设备的分组结果，能够并行地实现大批量数据的高性能时序处理，发挥多核运算的优势。同时，根据本发明实施方式提供的时序数据处理装置，单机即可处理上万个设备的数据处理，避免搭建复杂的计算机群，显著降低了部署成本。
另外，根据一些实施例，本发明提供的时序数据处理装置采用相互独立的数据缓存技术和数据传送技术，能够连续、稳定地实现海量时序结果数据的批量传送和精准存储。
需要注意的是，上述附图中所示的框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
图5是根据一示例性实施方式示出的一种服务器的结构示意图。需要说明的是，图5示出的服务器仅仅是一个示例，不应对本发明实施例的功能和使用范围带来任何限制。如图5所示的服务器例如可以应用于物联网数据处理场景。
如图5所示，服务器100以通用计算机设备的形式表现。服务器100的组件包括： 至少一个中央处理单元(CPU)1001，其可以根据存储在只读存储器(ROM)1002中的程序代码或者从至少一个存储单元1008加载到随机访问存储器(RAM)1003中的程序代码而执行各种适当的动作和处理。
特别地，根据本发明的实施例，所述程序代码可以被中央处理单元1001执行，使得中央处理单元1001执行本说明书上述方法实施例部分中描述的根据本发明各种示例性实施方式的步骤。例如，中央处理单元1001可以执行如图1、2、3中所示的步骤。
在RAM 1003中，还存储有服务器100操作所需的各种程序和数据。CPU 1001、ROM 1002以及RAM 1003通过总线1004彼此相连。输入/输出(I/O)接口1005也连接至总线1004。
以下部件连接至I/O接口1005：包括键盘、鼠标等的输入单元1006；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出单元1007；包括硬盘等的存储单元1008；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信单元1009。通信单元1009经由诸如因特网的网络执行通信处理。驱动器1010也根据需要连接至I/O接口1005。可拆卸介质1011，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器1010上，以便于从其上读出的计算机程序根据需要被安装入存储单元1008。
图6是根据一示例性实施方式示出的一种时序数据处理系统的结构示意图。如图6所示的时序数据处理系统例如可以应用于物联网数据处理场景。
参考图6，时序数据处理系统60包括：多个设备602、上述服务器100以及数据库604。
其中，服务器100用于从多个设备602接收多条待处理数据，并将多条结果数据发送至数据库604。
图7是根据一示例性实施方式示出的一种计算机可读存储介质的示意图。
参考图7所示，描述了根据本发明的实施方式的设置为实现上述方法的程序产品700，其可以采用便携式紧凑盘只读存储器(CD-ROM)并包括程序代码，并可以在终端设备，例如个人电脑上运行。然而，本发明的程序产品不限于此，在本文件中，可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置 或者器件使用或者与其结合使用。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被一个该设备执行时，使得该计算机可读介质实现如图1、2、3中所示的功能。
以上具体地示出和描述了本发明的示例性实施方式。应可理解的是，本发明不限于这里描述的详细结构、设置方式或实现方法；相反，本发明意图涵盖包含在所附权利要求的精神和范围内的各种修改和等效设置。

Claims (10)

  1. 一种时序数据处理方法，其特征在于，包括：
    接收多个设备发送的多条待处理数据，所述多条待处理数据分别包括所述多个设备采集的时序数据以及所述多个设备的标识码；
    根据对所述多个设备的分组结果，将所述多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据；以及
    将所述结果数据依序存储至第一缓存队列中。
  2. 根据权利要求1所述的方法，其特征在于，所述分组结果根据下述步骤获得：将所述标识码输入预构造的哈希函数，以所述哈希函数的输出结果作为所述标识码对应设备的所述分组结果。
  3. 根据权利要求1所述的方法，其特征在于，根据对所述多个设备的分组结果，将所述多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据包括：
    根据所述分组结果，将所述多条待处理数据中的所述时序数据分别导入相应的线程池的所述任务队列中，每个所述线程池包括一个处理线程；以及
    在每个所述线程池中，通过所述处理线程依次处理导入所述线程池的所述任务队列中的所述时序数据，对应获得所述多条结果数据。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，所述方法还包括：
    根据所述第一缓存队列，生成第二缓存队列，所述第二缓存队列中存储的所述结果数据与所述第一缓存队列当前存储的所述结果数据完全相同；
    将所述第二缓存队列中的所述结果数据发送至数据库；以及
    当所述第二缓存队列为空时，删除所述第二缓存队列。
  5. 根据权利要求4所述的方法，其特征在于，还包括：接收经所述依序处理获得的新的所述结果数据，并存储至所述第一缓存队列中。
  6. 一种时序数据处理装置，其特征在于，包括：
    数据接收模块，用于接收多个设备发送的多条待处理数据，所述多条待处理数据分别包括所述多个设备采集的时序数据以及所述多个设备的标识码；
    数据处理模块，用于根据对所述多个设备的分组结果，将所述多条待处理数据分别导入相应的任务队列中进行依序处理，对应获得多条结果数据；以及
    数据缓存模块，用于将所述结果数据依序存储至第一缓存队列中。
  7. 根据权利要求6所述的方法，其特征在于，所述装置还包括：
    队列生成模块，用于根据所述第一缓存队列，生成第二缓存队列，所述第二缓存队列中存储的所述结果数据与所述第一缓存队列当前存储的所述结果数据完全相同；
    数据发送模块，用于将所述第二缓存队列中的所述结果数据发送至数据库；以及
    队列删除模块，用于当所述第二缓存队列为空时，删除所述第二缓存队列。
  8. 一种服务器，包括：存储器、处理器及存储在所述存储器中并可在所述处理器中运行的可执行指令，其特征在于，所述处理器执行所述可执行指令时实现如权利要求1-5任一项所述的方法。
  9. 一种时序数据处理系统，其特征在于，包括：多个设备、根据权利要求8所述的服务器以及数据库；其中，所述服务器从所述多个设备接收多条待处理数据，并将多条结果数据发送至所述数据库。
  10. 一种计算机可读存储介质，其上存储有计算机可执行指令，其特征在于，所述可执行指令被处理器执行时实现如权利要求1-5任一项所述的方法。
PCT/CN2020/084962 2019-06-24 2020-04-15 时序数据处理方法、装置、系统、服务器及可读存储介质 Ceased WO2020259017A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910549734.8 2019-06-24
CN201910549734.8A CN112134909B (zh) 2019-06-24 2019-06-24 时序数据处理方法、装置、系统、服务器及可读存储介质

Publications (1)

Publication Number Publication Date
WO2020259017A1 true WO2020259017A1 (zh) 2020-12-30

Family

ID=73849297

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/084962 Ceased WO2020259017A1 (zh) 2019-06-24 2020-04-15 时序数据处理方法、装置、系统、服务器及可读存储介质

Country Status (2)

Country Link
CN (1) CN112134909B (zh)
WO (1) WO2020259017A1 (zh)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113220780A (zh) * 2021-04-29 2021-08-06 北京字跳网络技术有限公司 一种数据处理方法、装置、设备及介质
CN113377501A (zh) * 2021-06-08 2021-09-10 中国农业银行股份有限公司 数据处理方法、装置、设备、介质及程序产品
CN113961641A (zh) * 2021-09-27 2022-01-21 北京百度网讯科技有限公司 数据库同步方法、装置、设备和存储介质
CN114265841A (zh) * 2021-11-23 2022-04-01 中国联合网络通信集团有限公司 多模型串联的数据处理方法、设备及存储介质
CN114282125A (zh) * 2021-12-28 2022-04-05 湖北三江航天红峰控制有限公司 一种等效器多路点火时序数据处理方法
CN114500673A (zh) * 2022-02-15 2022-05-13 江苏提米智能科技有限公司 一种基于时序数据传输的编码方法和解码方法、装置、电子设备及存储介质
CN115982160A (zh) * 2022-12-26 2023-04-18 北京明朝万达科技股份有限公司 数据处理方法、服务器、电子设备及计算机存储介质
CN116192985A (zh) * 2023-02-08 2023-05-30 广州市保伦电子有限公司 一种数据传输方法、装置及系统
CN116719870A (zh) * 2023-06-16 2023-09-08 平安科技（深圳）有限公司 一种时序数据库集群的数据管理方法、装置、设备及介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113282604B (zh) * 2021-07-14 2021-10-22 北京远舢智能科技有限公司 基于消息队列实现的高可用时序数据库集群系统
CN114416787B (zh) * 2021-12-13 2025-11-21 国汽智控(北京)科技有限公司 多计算节点的数据实时同步方法及装置
CN114553700B (zh) * 2022-02-24 2024-06-28 树根互联股份有限公司 设备分组方法、装置、计算机设备及存储介质
CN114900507B (zh) * 2022-04-29 2024-10-08 阿里巴巴（中国）有限公司 Rtc音频数据的处理方法、装置、设备以及存储介质
CN115543660A (zh) * 2022-10-08 2022-12-30 哈尔滨海邻科信息技术有限公司 Gps数据处理方法、装置、计算机设备和存储介质
CN116089414B (zh) * 2023-04-10 2023-09-08 之江实验室 基于海量数据场景的时序数据库写入性能优化方法及装置

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150234882A1 (en) * 2013-04-01 2015-08-20 Hitachi, Ltd. Computer system, data management method, and computer
CN107193539A (zh) * 2016-03-14 2017-09-22 北京京东尚科信息技术有限公司 多线程并发处理方法和多线程并发处理系统
CN108459917A (zh) * 2018-03-15 2018-08-28 欧普照明股份有限公司 一种消息分配部件、消息处理系统以及消息分配方法

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7891004B1 (en) * 1999-10-06 2011-02-15 Gelvin David C Method for vehicle internetworks
AU2008225072B2 (en) * 2007-03-12 2013-01-17 Citrix Systems, Inc. Systems and methods for dynamic bandwidth control by proxy
US20120054362A1 (en) * 2010-08-31 2012-03-01 Canon Kabushiki Kaisha Mechanism for autotuning mass data transfer from a sender to a receiver over parallel connections
CN107391719A (zh) * 2017-07-31 2017-11-24 南京邮电大学 一种云环境中分布式流数据处理方法及系统
CN108228730A (zh) * 2017-12-11 2018-06-29 深圳市买买提信息科技有限公司 数据导入方法、装置、计算机设备及可读存储介质
CN109145051A (zh) * 2018-07-03 2019-01-04 阿里巴巴集团控股有限公司 分布式数据库的数据汇总方法及装置和电子设备

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150234882A1 (en) * 2013-04-01 2015-08-20 Hitachi, Ltd. Computer system, data management method, and computer
CN107193539A (zh) * 2016-03-14 2017-09-22 北京京东尚科信息技术有限公司 多线程并发处理方法和多线程并发处理系统
CN108459917A (zh) * 2018-03-15 2018-08-28 欧普照明股份有限公司 一种消息分配部件、消息处理系统以及消息分配方法

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113220780A (zh) * 2021-04-29 2021-08-06 北京字跳网络技术有限公司 一种数据处理方法、装置、设备及介质
CN113220780B (zh) * 2021-04-29 2023-12-05 北京字跳网络技术有限公司 一种数据处理方法、装置、设备及介质
CN113377501A (zh) * 2021-06-08 2021-09-10 中国农业银行股份有限公司 数据处理方法、装置、设备、介质及程序产品
CN113961641A (zh) * 2021-09-27 2022-01-21 北京百度网讯科技有限公司 数据库同步方法、装置、设备和存储介质
CN114265841A (zh) * 2021-11-23 2022-04-01 中国联合网络通信集团有限公司 多模型串联的数据处理方法、设备及存储介质
CN114282125A (zh) * 2021-12-28 2022-04-05 湖北三江航天红峰控制有限公司 一种等效器多路点火时序数据处理方法
CN114500673A (zh) * 2022-02-15 2022-05-13 江苏提米智能科技有限公司 一种基于时序数据传输的编码方法和解码方法、装置、电子设备及存储介质
CN115982160A (zh) * 2022-12-26 2023-04-18 北京明朝万达科技股份有限公司 数据处理方法、服务器、电子设备及计算机存储介质
CN116192985A (zh) * 2023-02-08 2023-05-30 广州市保伦电子有限公司 一种数据传输方法、装置及系统
CN116192985B (zh) * 2023-02-08 2023-11-21 广东保伦电子股份有限公司 一种数据传输方法、装置及系统
CN116719870A (zh) * 2023-06-16 2023-09-08 平安科技（深圳）有限公司 一种时序数据库集群的数据管理方法、装置、设备及介质

Also Published As

Publication number Publication date
CN112134909A (zh) 2020-12-25
CN112134909B (zh) 2022-04-19

Similar Documents

Publication Publication Date Title
CN112134909B (zh) 时序数据处理方法、装置、系统、服务器及可读存储介质
CN110019350B (zh) 基于配置信息的数据查询方法和装置
CN109656963B (zh) 元数据获取方法、装置、设备及计算机可读存储介质
WO2019114766A1 (zh) 一种数据脱敏方法、服务器、终端及计算机可读存储介质
EP4207688A1 (en) Asynchronous bookkeeping method and apparatus for blockchain, medium, and electronic device
CN116719622A (zh) 业务流程的编排方法及业务流程编排系统
CN113778700A (zh) 消息处理方法、系统、介质和计算机系统
CN113421109A (zh) 一种业务核对方法、装置、电子设备和存储介质
WO2022218248A1 (zh) 用户画像数据处理方法和装置、电子设备和可读存储介质
CN114201508A (zh) 数据处理方法、数据处理装置、电子设备和存储介质
CN112417016A (zh) 一种数据交换方法、系统、设备及存储介质
US20250131005A1 (en) Data processing method and apparatus based on data lake and data warehouse integration, and electronic device
CN114201297A (zh) 数据处理方法、装置、电子设备及存储介质
CN115220131A (zh) 气象数据质检方法及系统
CN115080606A (zh) 业务数据查询方法、装置、设备及介质
CN111858576A (zh) 实时消息处理方法、系统、装置、存储介质及电子设备
CN111651509B (zh) 基于Hbase数据库的数据导入方法、装置、电子设备及介质
US12566790B2 (en) Lakehouse metadata change determination method, device, and medium
CN110705935B (zh) 一种物流单据的处理方法和装置
CN110120959B (zh) 大数据推送方法、装置、系统、设备及可读存储介质
CN112100414A (zh) 数据处理方法、装置、系统与计算机可读存储介质
CN112069179A (zh) 一种基于Hbase查询性能优化的装置及其方法
CN115757120B (zh) 压力测试方法、装置、电子设备及可读存储介质
CN117632683A (zh) 一种分布式拨测监控方法、系统、电子设备及存储介质
US12032630B2 (en) Entity resolution data structure system and method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20830889

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20830889

Country of ref document: EP

Kind code of ref document: A1