WO2021115368A1 - 物联网设备性能采集方法、装置、设备及可读存储介质 - Google Patents

物联网设备性能采集方法、装置、设备及可读存储介质 Download PDF

Info

Publication number
WO2021115368A1
WO2021115368A1 PCT/CN2020/135175 CN2020135175W WO2021115368A1 WO 2021115368 A1 WO2021115368 A1 WO 2021115368A1 CN 2020135175 W CN2020135175 W CN 2020135175W WO 2021115368 A1 WO2021115368 A1 WO 2021115368A1
Authority
WO
WIPO (PCT)
Prior art keywords
performance data
collection
performance
service
service agent
Prior art date
Application number
PCT/CN2020/135175
Other languages
English (en)
French (fr)
Inventor
符凯
Original Assignee
中盈优创资讯科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中盈优创资讯科技有限公司 filed Critical 中盈优创资讯科技有限公司
Publication of WO2021115368A1 publication Critical patent/WO2021115368A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/50—Testing arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching

Definitions

  • the present invention relates to the technical field of the Internet of Things, in particular to a method, device, equipment and readable storage medium for collecting the performance of Internet of Things equipment.
  • performance collection is usually to pull data from devices through active polling.
  • this collection method is a one-to-one question-and-answer interaction between the collection service agent and the device, which virtually increases the network delay and the processing overhead of the device.
  • the embodiment of the present invention provides a method for collecting the performance of an Internet of Things device to solve the technical problems of increasing network delay and increasing equipment processing overhead in the collection method in the prior art.
  • the method includes:
  • the virtual server in the service cluster receives the request for reporting performance data sent by the device to be collected based on the telemetry protocol, and distributes the request for reporting performance data to the collection service agent in the service cluster, where the request for reporting performance data Including performance data;
  • the collection service agent in the service cluster establishes a connection with the device to be collected and receives performance data.
  • the embodiment of the present invention also provides a device for collecting performance of Internet of Things equipment to solve the technical problems of increasing network delay and increasing equipment processing overhead in the collection method in the prior art.
  • the device includes:
  • the access receiving module is configured to receive, through the virtual server, a request for reporting performance data sent by the device to be collected based on the telemetry protocol, and distribute the request for reporting performance data to the collection service agent in the service cluster;
  • a plurality of data receiving modules are configured to establish a connection with the device to be collected through the collection service agent and receive performance data, wherein a virtual server and a plurality of collection service agents form a service cluster.
  • An embodiment of the present invention also provides a computer device including a memory, a processor, and a computer program stored in the memory and capable of running on the processor.
  • the processor implements the above-mentioned Internet of Things device performance when the computer program is executed.
  • the collection method solves the technical problems of increasing network delay and increasing equipment processing overhead in the collection method in the prior art.
  • the embodiment of the present invention also provides a computer-readable storage medium that stores a computer program that executes any of the above-mentioned methods for collecting performance of Internet of Things equipment, so as to solve the problem of increasing the network in the collection method in the prior art.
  • the virtual server in the service cluster receives the request for reporting performance data sent by the device to be collected based on the telemetry protocol, and distributes the request for reporting performance data to the collection service agent in the service cluster, Furthermore, the performance data is received through the collection service agent in the service cluster, which realizes that the device to be collected actively reports the performance parameters to the collection service agent.
  • the performance collection of this application is The mode is changed from pull to push, which avoids the question-and-answer interaction between the device and the collection service agent, which is beneficial to reduce network delays, reduce equipment processing overhead, and achieve more real-time and high-speed performance data Collection;
  • the request for reporting performance data is received through the virtual server in the service cluster and distributed to the collection service agent in the service cluster.
  • the use of the service cluster makes it possible to support the high-concurrency reporting situation of the device, and it is also beneficial to increase the collection service agent
  • the number of collection services improves the horizontal scalability of the collection service agent, which in turn helps to support the collection requirements of more collection equipment.
  • Fig. 1 is a schematic diagram of a framework of performance data collection in the prior art
  • FIG. 2 is a flowchart of a method for collecting performance of an Internet of Things device according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of a framework for realizing performance data collection according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a framework of performance data receiving and data processing provided by an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of a framework for implementing task issuance according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a framework for implementing the foregoing method for collecting performance of Internet of Things equipment according to an embodiment of the present invention
  • Figure 7 is a structural block diagram of a computer device provided by an embodiment of the present invention.
  • Fig. 8 is a structural block diagram of an apparatus for collecting performance of Internet of Things equipment provided by an embodiment of the present invention.
  • a method for collecting the performance of an Internet of Things device includes:
  • Step 202 Receive, through the virtual server in the service cluster, a request for reporting performance data sent by the device to be collected based on the telemetry protocol, and distribute the request for reporting performance data to the collection service agent in the service cluster;
  • Step 204 Establish a connection with the device to be collected through the collection service agent in the service cluster and receive performance data.
  • the virtual server in the service cluster receives the request for reporting performance data sent by the device to be collected based on the telemetry protocol, and distributes the request for reporting performance data.
  • the device to be collected actively reports performance parameters to the collection service agent, and the performance is pulled by polling in the prior art.
  • the performance collection mode of this application is changed from pull to push, avoiding the question-and-answer interaction between the device and the collection service agent, which is beneficial to reduce network delay, and is beneficial to reducing the processing overhead of the device.
  • the request for reporting performance data is received through the virtual server in the service cluster and distributed to the collection service agent in the service cluster.
  • the use of the service cluster makes it possible to support high-concurrency reporting of equipment
  • the situation is also conducive to increasing the number of collection service agents, improving the horizontal scalability of collection service agents, and thus helping to support the collection requirements of more collection equipment.
  • Telemetry is a remote technology that collects data from physical devices or virtual devices at high speed.
  • the device to be collected can actively send a request for reporting performance data in Push Mode based on the telemetry protocol, and it can also actively send a request for reporting performance data periodically in Push Mode, the request for reporting performance data It can include performance information such as device interface traffic statistics, CPU or memory data.
  • this embodiment adopts the push mode (Push Mode).
  • the collection machine acts as a client to initiate collection requests. As long as the number of collection machines is increased, more collection objects can be supported. After the conversion to the telmetry passive collection mode, the collection The machine is used as a server to receive the data pushed by the collection object. Telemetry reports two ways: UDP (User Datagram Protocol) and TCP (Transmission Control Protocol) (GRPC). It is oriented to millions of devices in the Internet of Things. A single server is It is impossible to withstand the impact of such connections, so it is necessary to solve the concurrency problem caused by the telemetry report of a large number of devices and improve the high availability and scalability of the service.
  • UDP User Datagram Protocol
  • TCP Transmission Control Protocol
  • the above-mentioned service cluster is composed of a virtual server and multiple collection service agents.
  • the virtual server can be an LVS (Linux Virtual Server), which provides a unified floating access address to the outside world.
  • LVS Local Virtual Server
  • the request for reporting performance data is sent through the access address, and the virtual server distributes the access request to multiple collection service agents, and realizes horizontal expansion through multi-machine clusters. In this way, by increasing the number of collection service agents, more collection objects can be supported. High availability and scalability.
  • two virtual servers can be set up in the service cluster, a primary virtual server and a backup virtual server.
  • an event-driven network programming model can be used, which can greatly improve the parallel processing capability of a single collection service agent and support TCP/UDP connections for massive devices.
  • the timing report of performance data is initiated by the device. It is very likely that a large number of devices simultaneously report at a unified time, forming a data tide. It is obviously not cost-effective to configure the number of collection service agents according to the peak data volume. This requires the service cluster to have a certain buffer capacity, so that when the data peaks, the received data is saved first, and the data processing is completed when the data is low.
  • the above also includes:
  • a message queue is set up on each collection service agent in the service cluster, and the buffering of performance data is implemented by using a real-time distributed message queue NSQ.
  • NSQ is an open source message queuing software, which is characterized by high throughput and is very suitable for scenarios where large amounts of data are peak-cut and valley-filled.
  • NSQ mainly includes two core components, nsqd (NSQ message service) and nsqlookupd (NSQ message queue discovery service).
  • nsqd is responsible for queue read and write operations, and nsqlookupd is responsible for routing of nsqd nodes.
  • NSQ's nsqd can be directly deployed on the collection service agent, so that the performance data received by the collection service agent can be enqueued nearby and reduce the impact of network delay.
  • nsqlookupd will collect the service agent.
  • the nsqd node is routed to the data processing service, so that the data processing service reads performance data from the message queue of the collection service agent and performs data processing. It can be seen that the reception of collected data and the processing of collected data are deployed separately.
  • the collection service agent is responsible for enqueuing the received data into the queue nearby, while the data processing service uses nsqlookupd to pull the collected data from nsqd for processing to achieve data reception. Decoupling from data processing helps avoid peak overloads.
  • the network manager may have different collection cycle requirements for different indicators, and sometimes temporarily issue some collection tasks.
  • the acquisition frequency can be controlled by the acquisition machine; while in the passive acquisition mode, the data push cycle is determined by the device side. If the whole network static configuration push cycle scheme is adopted, it will undoubtedly be greatly improved. The flexibility of the collection task is limited. Therefore, in this embodiment, the above method further includes:
  • the collection task management module sends the collection tasks originally sent to the collection machine to the device through the controller. For each performance data collection task, only the collection interval needs to be adjusted. At the same time, it is necessary to re-issue the performance data collection task to the device, so that the device can actively report performance parameters while flexibly setting the reporting period of the device. Therefore, the semi-active collection mode is realized, which takes into account the passive collection. Efficiency and flexibility of active acquisition.
  • the device can support the issuance of telemetry tasks with different collection intervals, if there is no restriction, it will cause greater collection pressure on the device. Therefore, the sampling interval corresponding to each performance data collection task issued by the device is the smallest sampling interval among all required sampling intervals. For example, if the required sampling interval has indicators such as 1 minute, 5 minutes, 10 minutes, 15 minutes, 30 minutes, etc., the sampling interval for the performance data collection task issued by the device is the minimum 1 minute, and the other 5 minutes, 10 minutes, etc.
  • the data of indicators such as 15 minutes and 30 minutes can be aggregated through preprocessing based on the original collected results.
  • the data processing service can be responsible for data aggregation processing of different time granularities. At the same time, in the preprocessing link, a field of data can also be screened and cleaned. Different granularities of aggregated data can adopt different retention periods.
  • a computer device including a memory 702, a processor 704, and a computer program stored in the memory and running on the processor, and the processor executes the computer
  • the program implements any of the above-mentioned methods for collecting the performance of the Internet of Things equipment.
  • the computer equipment may be a computer terminal, a server, or a similar computing device.
  • a computer-readable storage medium stores a computer program that executes any of the foregoing methods for collecting the performance of an Internet of Things device.
  • computer-readable storage media includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology.
  • the information can be computer-readable instructions, data structures, program modules, or other data.
  • Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only Memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage , Magnetic cassette tape, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable storage media does not include transitory media, such as modulated data signals and carrier waves.
  • the embodiments of the present invention also provide a device for collecting the performance of the Internet of Things equipment, as described in the following embodiments. Since the problem-solving principle of the IoT device performance collection device is similar to the IoT device performance collection method, the implementation of the IoT device performance collection device can refer to the implementation of the IoT device performance collection method, and the repetition will not be repeated.
  • the term "unit” or "module” can be a combination of software and/or hardware that implements a predetermined function.
  • the devices described in the following embodiments are preferably implemented by software, implementation by hardware or a combination of software and hardware is also possible and conceived.
  • Fig. 8 is a structural block diagram of an apparatus for collecting performance of an Internet of Things equipment according to an embodiment of the present invention. As shown in Fig. 8, the apparatus includes:
  • the access receiving module 802 is configured to receive, through the virtual server, a request for reporting performance data sent by the device to be collected based on the telemetry protocol, and distribute the request for reporting performance data to the collection service agent in the service cluster;
  • Multiple data receiving modules 804 are configured to establish a connection with the device to be collected through the collection service agent and receive performance data, where a virtual server and multiple collection service agents form a service cluster.
  • it further includes:
  • the cache module is set on each collection service agent, and is used to cache the performance data received by each collection service agent through the message queue;
  • Multiple data processing modules are used to read performance data from the message queue of the collection service agent through the data processing service and perform data processing according to the message queue of the collection service agent routed by nsqlookupd.
  • the caching module is also used for sharing performance data by collecting the message queue of the service agent.
  • it further includes:
  • the task issuing module is used to issue performance data collection tasks with different sampling intervals to the device to be collected, and trigger the device to be collected to periodically send the request for reporting performance data according to the sampling interval, where different sampling intervals correspond to different Performance data collection task.
  • the sampling interval corresponding to each performance data collection task is the smallest sampling interval among all required sampling intervals.
  • the virtual server in the service cluster receives a request for reporting performance data sent by the device to be collected based on the telemetry protocol, and distributes the request for reporting performance data to the collection in the service cluster
  • the service agent receives performance data through the collection service agent in the service cluster, that is, the device to be collected actively reports performance parameters to the collection service agent.
  • this application The performance collection mode is changed from pull to push, avoiding the question-and-answer interaction between the device and the collection service agent, which is beneficial to reduce network delay, reduce the processing overhead of the device, and achieve more real-time and high-speed Performance data collection;
  • the request for reporting performance data is received through the virtual server in the service cluster and distributed to the collection service agent in the service cluster.
  • the use of the service cluster makes it possible to support the high-concurrency reporting of equipment, which is also conducive to increase
  • the number of collection service agents improves the horizontal scalability of collection service agents, thereby helping to support the collection requirements of more collection equipment.
  • modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, and they can be concentrated on a single computing device or distributed among multiple computing devices.
  • they can be implemented by the program code executable by the computing device, so that they can be stored in the storage device for execution by the computing device, and in some cases, they can be different from here
  • the steps shown or described are executed in the order of, or they are respectively fabricated into individual integrated circuit modules, or multiple modules or steps of them are fabricated into a single integrated circuit module to achieve. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Abstract

本发明实施例提供了一种物联网设备性能采集方法、装置、设备及可读存储介质，其中，该方法包括：通过服务集群中的虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将上报性能数据的请求分发给服务集群中的采集服务代理，上报性能数据的请求包括性能数据；通过服务集群中的采集服务代理与待采集设备建立连接并接收性能数据。该方案性能采集模式由拉转变为推，避免了设备与采集服务代理之间一问一答的交互方式，有利于降低网络延时、有利于降低设备的处理开销，有利于实现更实时更高速的性能数据的采集；服务集群的使用可以支持设备高并发的上报情况，提高了采集服务代理的横向可扩展性。

Description

物联网设备性能采集方法、装置、设备及可读存储介质 技术领域
本发明涉及物联网技术领域，特别涉及一种物联网设备性能采集方法、装置、设备及可读存储介质。
背景技术
在传统的网管系统中，性能采集通常是通过主动轮询的方式从设备拉取数据。例如，如图1所示，这种采集方式是采集服务代理和设备之间一对一地进行一问一答的交互，使得无形中增大了网络时延和设备的处理开销。
在新的SDN(软件定义网络)中，例如5G物联网网络中，由于虚拟设备数量巨大，网络情况不稳定，如果使用传统的采集方式，将会使增大网络时延和增大设备的处理开销这个问题变得更加突出。
发明内容
本发明实施例提供了一种物联网设备性能采集方法，以解决现有技术中采集方式存在增大网络时延、增大设备处理开销的技术问题。该方法包括：
通过服务集群中的虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理，其中，所述上报性能数据的请求包括性能数据；
通过服务集群中的采集服务代理与所述待采集设备建立连接并接收性能数据。
本发明实施例还提供了一种物联网设备性能采集装置，以解决现有技术中采集方式存在增大网络时延、增大设备处理开销的技术问题。该装置包括：
访问接收模块，用于通过虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理；
多个数据接收模块，用于通过所述采集服务代理与所述待采集设备建立连接并接收性能数据，其中，虚拟服务器和多个采集服务代理组成服务集群。
本发明实施例还提供了一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述的物联网 设备性能采集方法，以解决现有技术中采集方式存在增大网络时延、增大设备处理开销的技术问题。
本发明实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有执行上述任意的物联网设备性能采集方法的计算机程序，以解决现有技术中采集方式存在增大网络时延、增大设备处理开销的技术问题。
在本发明实施例中，提出了通过服务集群中的虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理，进而通过服务集群中的采集服务代理接收性能数据，即实现了待采集设备主动向采集服务代理上报性能参数，与现有技术中通过轮询方式来拉去性能数据相比，本申请的性能采集模式由拉转变为推，避免了设备与采集服务代理之间一问一答的交互方式，有利于降低网络延时、有利于降低设备的处理开销，有利于实现更实时更高速的性能数据的采集；同时，上报性能数据的请求是通过服务集群中的虚拟服务器接收并分发给服务集群中的采集服务代理，服务集群的使用使得可以支持设备高并发的上报情况，也有利于增加采集服务代理的数量，提高了采集服务代理的横向可扩展性，进而有利于支持更多的采集设备的采集需求。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。在附图中：
图1是现有技术中的一种性能数据采集的框架示意图；
图2是本发明实施例提供的一种物联网设备性能采集方法的流程图；
图3是本发明实施例提供的一种实现性能数据采集的框架示意图；
图4是本发明实施例提供的一种性能数据接收和数据处理的框架示意图；
图5是本发明实施例提供的一种实现任务下发的框架示意图；
图6是本发明实施例提供的一种实施上述物联网设备性能采集方法的框架示意图；
图7是本发明实施例提供的一种计算机设备的结构框图；
图8是本发明实施例提供的一种物联网设备性能采集装置的结构框图。
具体实施方式
为使本发明实施例的目的、技术方案和优点更加清楚明白，下面结合附图对本发明实施例做进一步详细说明。在此，本发明的示意性实施例及其说明用于解释本发明，但并不作为对本发明的限定。
在本发明实施例中，提供了一种物联网设备性能采集方法，如图2所示，该方法包括：
步骤202：通过服务集群中的虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理；
步骤204：通过服务集群中的采集服务代理与所述待采集设备建立连接并接收性能数据。
由图2所示的流程可知，在本发明实施例中，提出了通过服务集群中的虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理，进而通过服务集群中的采集服务代理接收性能数据，即实现了待采集设备主动向采集服务代理上报性能参数，与现有技术中通过轮询方式来拉去性能数据相比，本申请的性能采集模式由拉转变为推，避免了设备与采集服务代理之间一问一答的交互方式，有利于降低网络延时、有利于降低设备的处理开销，有利于实现更实时更高速的性能数据的采集；同时，上报性能数据的请求是通过服务集群中的虚拟服务器接收并分发给服务集群中的采集服务代理，服务集群的使用使得可以支持设备高并发的上报情况，也有利于增加采集服务代理的数量，提高了采集服务代理的横向可扩展性，进而有利于支持更多的采集设备的采集需求。
具体实施时，Telemetry(遥测)是一项远程的从物理设备或虚拟设备上高速采集数据的技术。具体的，待采集设备可以基于telemetry协议以推模式(Push Mode)主动发送上报性能数据的请求，还可以推模式(Push Mode)周期性的主动发送上报性能数据的请求，该上报性能数据的请求可以包括设备的接口流量统计、CPU或内存数据等性能信息。相对传统的拉模式(Pull Mode)的一问一答式交互，本实施例以推模式(Push
Mode)周期性的主动发送上报性能数据的请求，提供了更实时更高速的数据采集功能，在5G网络中，面对海量的物联网设备，通过引入telemetry技术，可以实时准确地获取设备的性能信息，提升采集效率。
具体实施时，传统的性能数据主动采集方式中，采集机是作为客户端发起采集请求，只要增加采集机的数量，就可以支持更多的采集对象，而在转变为telmetry被动采集方式后，采集机是作为服务端接收采集对象推送的数据，Telemetry上报的方式有UDP(用户数据报协议)和TCP(传输控制协议)(GRPC)两种，面向物联网上百万的设备，单台服务器是无法承受这样的连接冲击的，因此需要解决海量设备telemetry上报带来的并发问题，提高服务的高可用性和可扩展性。在本实施例中，如图3所示，上述服务集群由虚拟服务器和多个采集服务代理组成，虚拟服务器可以为LVS(Linux Virtual Server，linux虚拟服务器)，对外提供统一的浮动访问地址，设备通过访问地址发送上报性能数据的请求，虚拟服务器将访问请求分发给多个采集服务代理，通过多机集群实现横向扩展，这样通过增加采集服务代理的数量，就可以支持更多的采集对象，实现高可用性和可扩展性。
具体实施时，服务集群中可以设置两个虚拟服务器，一个主虚拟服务器和一个备用虚拟服务器。针对服务集群中的每个采集服务代理，可以采用基于事件驱动的网络编程模型，可以大大提升单个采集服务代理的并行处理能力，支持海量设备的TCP/UDP连接。
具体实施时，上述物联网设备性能采集方法转换为被动采集模式后，就是性能数据的定时上报是由设备主动发起的，很有可能出现海量设备在统一时刻同时上报的情况，形成数据潮汐。按照峰值数据量配置采集服务代理数量显然是不合算的。这就要求服务集群具备一定的缓冲能力，使得在数据峰值的时候，将接收到的数据先保存下来，在数据谷值的时候完成数据处理。例如，上述还包括：
在服务集群中的每个采集服务代理上设置消息队列，每个采集服务代理将接收的性能数据缓存在自身的消息队列中；
设置多个数据处理服务，通过消息队列发现服务(nsqlookupd)将采集服务代理的消息队列路由给数据处理服务，通过数据处理服务从采集服务代理的消息队列中读取性能数据并进行数据处理。
具体实施时，在服务集群中的每个采集服务代理上设置消息队列，性能数据的缓冲采用实时分布式消息队列NSQ实现。NSQ是一个开源的消息队列软件，特点是吞吐量高，非常适合用于大数据量削峰填谷的场景。NSQ主要包含nsqd(NSQ消息服务)和nsqlookupd(NSQ消息队列发现服务)两个核心组件，其中nsqd负责队列的读写操作，nsqlookupd则负责nsqd节点的路由。如图4、图6所示，可以把NSQ的nsqd直接部署 在采集服务代理上，这样采集服务代理接收的性能数据就可以实现就近入队，减少网络时延的影响，nsqlookupd将采集服务代理上的nsqd节点路由给数据处理服务，进而实现数据处理服务从采集服务代理的消息队列中读取性能数据并进行数据处理。可见，采集数据的接收和采集数据的处理是分离部署的，采集服务代理负责将接收到的数据就近入队，而数据处理服务则通过nsqlookupd从nsqd拉取采集到的数据进行处理，实现数据接收和数据处理的解耦，有利于避免出现峰值过载。
具体实施时，在存在多个网管系统的情况下，很有可能出现多个网管系统对设备的重复采集，造成不必要的设备压力。由于设备通过telemetry上报的信息是一致的，完全可以通过采集服务代理将性能数据共享给多个网管系统，避免多个网管系统对设备的重复采集，减轻设备的采集压力。例如，可以通过利用nsq本身的订阅的功能，实现原始采集信息的直接共享；也可以采集数据处理服务进行处理和汇聚之后的数据，通过NSQ进行分享。
具体实施时，上述物联网设备性能采集方法转换为被动采集模式后，就是网管对不同的指标可能有不同的采集周期需求，有时还会临时下发一些采集任务。在传统的主动采集模式下，可以由采集机控制采集的频度；而在被动采集方式下，数据推送的周期是由设备侧决定的，如果采用全网静态配置推送周期的方案，无疑会大大限制采集任务的灵活性，因此，在本实施例中，上述方法还包括：
将不同采样间隔的性能数据采集任务下发给待采集设备，触发待采集设备根据采样间隔周期性地发送所述上报性能数据的请求，其中，不同的采样间隔对应不同的性能数据采集任务，即实现了不同的性能数据采集任务可以配置不同的采样频率，提高了性能数据采集任务的配置灵活性。
具体的，如图5、图6所示，采集任务管理模块将原来下发到采集机的采集任务，通过控制器下发到设备上，针对每个性能数据采集任务，只有在采集间隔需要调整时，才需要重新下发该性能数据采集任务给设备，使得实现设备主动上报性能参数的同时，还可以灵活设置设备上报的周期，因此，实现了半主动采集模式，这样就兼顾了被动式采集的效率和主动式采集的灵活性。
具体实施时，虽然设备可以支持下发不同的采集间隔的telemetry任务，但是如果不加限制，会对设备造成较大的采集压力。所以，对设备下发的每个性能数据采集任务对应的采样间隔为所有所需采样间隔中最小的采样间隔。例如，所需采样间隔有1分钟、5分钟、10分钟、15分钟、30分钟等指标，则对设备下发的性能数据采集任务的采样间 隔为最小的1分钟，其他5分钟、10分钟、15分钟、30分钟等指标的数据可以基于采集的原始结果通过预处理汇聚得到，例如，可以通过数据处理服务负责不同时间粒度的数据汇聚处理。同时在预处理环节，还可以对一场数据进行筛选和清洗。不同粒度的汇聚数据可以采用不同的保留周期。
在本实施例中，提供了一种计算机设备，如图7所示，包括存储器702、处理器704及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述任意的物联网设备性能采集方法。
具体的，该计算机设备可以是计算机终端、服务器或者类似的运算装置。
在本实施例中，提供了一种计算机可读存储介质，所述计算机可读存储介质存储有执行上述任意的物联网设备性能采集方法的计算机程序。
具体的，计算机可读存储介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机可读存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读存储介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
基于同一发明构思，本发明实施例中还提供了一种物联网设备性能采集装置，如下面的实施例所述。由于物联网设备性能采集装置解决问题的原理与物联网设备性能采集方法相似，因此物联网设备性能采集装置的实施可以参见物联网设备性能采集方法的实施，重复之处不再赘述。以下所使用的，术语“单元”或者“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
图8是本发明实施例的物联网设备性能采集装置的一种结构框图，如图8所示，该装置包括：
访问接收模块802，用于通过虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理；
多个数据接收模块804，用于通过所述采集服务代理与所述待采集设备建立连接并接收性能数据，其中，虚拟服务器和多个采集服务代理组成服务集群。
在一个实施例中，还包括：
缓存模块，设置在每个采集服务代理上，用于通过消息队列缓存每个采集服务代理接收的性能数据；
多个数据处理模块，用于根据nsqlookupd路由的采集服务代理的消息队列，通过数据处理服务从采集服务代理的消息队列中读取性能数据并进行数据处理。
在一个实施例中，所述缓存模块，还用于通过采集服务代理的消息队列进行性能数据的共享。
在一个实施例中，还包括：
任务下发模块，用于将不同采样间隔的性能数据采集任务下发给待采集设备，触发待采集设备根据采样间隔周期性地发送所述上报性能数据的请求，其中，不同的采样间隔对应不同的性能数据采集任务。
在一个实施例中，每个性能数据采集任务对应的采样间隔为所有所需采样间隔中最小的采样间隔。
本发明实施例实现了如下技术效果：提出了通过服务集群中的虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理，进而通过服务集群中的采集服务代理接收性能数据，即实现了待采集设备主动向采集服务代理上报性能参数，与现有技术中通过轮询方式来拉去性能数据相比，本申请的性能采集模式由拉转变为推，避免了设备与采集服务代理之间一问一答的交互方式，有利于降低网络延时、有利于降低设备的处理开销，有利于实现更实时更高速的性能数据的采集；同时，上报性能数据的请求是通过服务集群中的虚拟服务器接收并分发给服务集群中的采集服务代理，服务集群的使用使得可以支持设备高并发的上报情况，也有利于增加采集服务代理的数量，提高了采集服务代理的横向可扩展性，进而有利于支持更多的采集设备的采集需求。
显然，本领域的技术人员应该明白，上述的本发明实施例的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们 中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明实施例不限制于任何特定的硬件和软件结合。
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明实施例可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种物联网设备性能采集方法，其特征在于，包括：
    通过服务集群中的虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理，其中，所述上报性能数据的请求包括性能数据；
    通过服务集群中的采集服务代理与所述待采集设备建立连接并接收性能数据。
  2. 如权利要求1所述的物联网设备性能采集方法，其特征在于，还包括：
    在服务集群中的每个采集服务代理上设置消息队列，每个采集服务代理将接收的性能数据缓存在自身的消息队列中；
    设置多个数据处理服务，通过消息队列发现服务将采集服务代理的消息队列路由给数据处理服务，通过数据处理服务从采集服务代理的消息队列中读取性能数据并进行数据处理。
  3. 如权利要求2所述的物联网设备性能采集方法，其特征在于，还包括：
    通过采集服务代理的消息队列进行性能数据的共享。
  4. 如权利要求1至3中任一项所述的物联网设备性能采集方法，其特征在于，还包括：
    将不同采样间隔的性能数据采集任务下发给待采集设备，触发待采集设备根据采样间隔周期性地发送所述上报性能数据的请求，其中，不同的采样间隔对应不同的性能数据采集任务。
  5. 如权利要求4所述的物联网设备性能采集方法，其特征在于，每个性能数据采集任务对应的采样间隔为所有所需采样间隔中最小的采样间隔。
  6. 一种物联网设备性能采集装置，其特征在于，包括：
    访问接收模块，用于通过虚拟服务器接收待采集设备基于telemetry协议发送的上报性能数据的请求，并将所述上报性能数据的请求分发给服务集群中的采集服务代理；
    多个数据接收模块，用于通过所述采集服务代理与所述待采集设备建立连接并接收性能数据，其中，虚拟服务器和多个采集服务代理组成服务集群。
  7. 如权利要求6所述的物联网设备性能采集装置，其特征在于，还包括：
    缓存模块，设置在每个采集服务代理上，用于通过消息队列缓存每个采集服务代理接收的性能数据；
    多个数据处理模块，用于根据消息队列发现服务路由的采集服务代理的消息队列，通过数据处理服务从采集服务代理的消息队列中读取性能数据并进行数据处理。
  8. 如权利要求7所述的物联网设备性能采集装置，其特征在于，所述缓存模块，还用于通过采集服务代理的消息队列进行性能数据的共享。
  9. 一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至5中任一项所述的物联网设备性能采集方法。
  10. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有执行权利要求1至5中任一项所述的物联网设备性能采集方法的计算机程序。
PCT/CN2020/135175 2019-12-10 2020-12-10 物联网设备性能采集方法、装置、设备及可读存储介质 WO2021115368A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911257737.0 2019-12-10
CN201911257737.0A CN112953784B (zh) 2019-12-10 2019-12-10 物联网设备性能采集方法、装置、设备及可读存储介质

Publications (1)

Publication Number Publication Date
WO2021115368A1 true WO2021115368A1 (zh) 2021-06-17

Family

ID=76225522

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/135175 WO2021115368A1 (zh) 2019-12-10 2020-12-10 物联网设备性能采集方法、装置、设备及可读存储介质

Country Status (2)

Country Link
CN (1) CN112953784B (zh)
WO (1) WO2021115368A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114338484A (zh) * 2021-12-29 2022-04-12 中国电信股份有限公司 光网络性能数据融合采集方法、装置、设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160234087A1 (en) * 2015-02-06 2016-08-11 Ustream, Inc. Techniques for managing telemetry data for content delivery and/or data transfer networks
US20180091475A1 (en) * 2016-09-27 2018-03-29 International Business Machines Corporation Reducing data connections for transmitting secured data
CN109474487A (zh) * 2018-10-17 2019-03-15 Ut斯达康通讯有限公司 网络性能监测方法、网络设备及网络性能监测系统
CN110546606A (zh) * 2017-04-14 2019-12-06 微软技术许可有限责任公司 租户升级分析

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8316110B1 (en) * 2003-12-18 2012-11-20 Symantec Operating Corporation System and method for clustering standalone server applications and extending cluster functionality
CN108769121B (zh) * 2018-04-24 2021-11-30 浙江正泰电器股份有限公司 智能工业装备物联网数据采集系统及采集数据的上传方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160234087A1 (en) * 2015-02-06 2016-08-11 Ustream, Inc. Techniques for managing telemetry data for content delivery and/or data transfer networks
US20180091475A1 (en) * 2016-09-27 2018-03-29 International Business Machines Corporation Reducing data connections for transmitting secured data
CN110546606A (zh) * 2017-04-14 2019-12-06 微软技术许可有限责任公司 租户升级分析
CN109474487A (zh) * 2018-10-17 2019-03-15 Ut斯达康通讯有限公司 网络性能监测方法、网络设备及网络性能监测系统

Also Published As

Publication number Publication date
CN112953784B (zh) 2023-10-31
CN112953784A (zh) 2021-06-11

Similar Documents

Publication Publication Date Title
CN109271233B (zh) 基于Kubernetes组建Hadoop集群的实现方法
CN108268208B (zh) 一种基于rdma的分布式内存文件系统
US9942339B1 (en) Systems and methods for providing messages to multiple subscribers
CN109308221B (zh) 一种基于WebSocket长连接的Nginx动态负载均衡方法
US8572241B2 (en) Integrating external and cluster heat map data
US8108623B2 (en) Poll based cache event notifications in a distributed cache
CN112887433B (zh) 基于quic协议的云端访问边缘服务的方法和系统
Wang et al. The design and implementation of the NCTUns network simulation engine
Pipatsakulroj et al. mumq: A lightweight and scalable mqtt broker
CN105607954A (zh) 一种有状态容器在线迁移的方法和装置
WO2020019743A1 (zh) 流量控制方法及装置
CN112631788B (zh) 数据传输方法及数据传输服务器
CN105592163A (zh) 一种通信方法及系统
CN112583895A (zh) Tcp通信方法、系统及装置
WO2021115368A1 (zh) 物联网设备性能采集方法、装置、设备及可读存储介质
CN112751847A (zh) 接口调用请求的处理方法、装置、电子设备及存储介质
Javed et al. Cutting the tail: Designing high performance message brokers to reduce tail latencies in stream processing
WO2020206783A1 (zh) 一种数据传输调度方法和系统
Xu et al. Building a high-performance key–value cache as an energy-efficient appliance
CN109428745B (zh) Jms消息转发方法、装置、电子设备和存储介质
CN110289990B (zh) 基于gpu的网络功能虚拟化系统、方法及存储介质
Liu et al. A high performance, scalable dns service for very large scale container cloud platforms
CN114884830A (zh) 基于广域网的分布式并行仿真推演系统
Solanti et al. POCL-R: distributed OpenCL runtime for low latency remote offloading
CN113630366A (zh) 一种物联网设备接入方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20900082

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20900082

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20900082

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 23/01/2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20900082

Country of ref document: EP

Kind code of ref document: A1