WO2021012331A1 - 一种边缘计算系统及数据存储方法 - Google Patents
一种边缘计算系统及数据存储方法 Download PDFInfo
- Publication number
- WO2021012331A1 WO2021012331A1 PCT/CN2019/100922 CN2019100922W WO2021012331A1 WO 2021012331 A1 WO2021012331 A1 WO 2021012331A1 CN 2019100922 W CN2019100922 W CN 2019100922W WO 2021012331 A1 WO2021012331 A1 WO 2021012331A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- storage unit
- module
- filtering
- snapshot
- Prior art date
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000001914 filtration Methods 0.000 claims abstract description 79
- 238000001514 detection method Methods 0.000 claims description 30
- 238000004140 cleaning Methods 0.000 claims description 25
- 238000004891 communication Methods 0.000 claims description 17
- 238000000605 extraction Methods 0.000 claims description 15
- 230000003993 interaction Effects 0.000 claims description 15
- 238000011084 recovery Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 11
- 239000000284 extract Substances 0.000 claims description 11
- 238000012545 processing Methods 0.000 abstract description 8
- 230000007812 deficiency Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 3
- 238000013480 data collection Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013144 data compression Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
Definitions
- the invention relates to the technical field of edge device data storage, and more specifically, to an edge computing system and a data storage method.
- Data is an important resource in the cloud platform.
- the acquisition of high-resolution data will result in a lot of costs for data collection, storage, and transmission, such as communication bandwidth, data gateway and platform data caching and processing capabilities; on the other hand, massive, high-resolution data Data analysis also requires the consumption of platform processing power and resources.
- the purpose of the present invention is to overcome the deficiencies in the prior art that the collection, storage and processing of high-resolution data require huge computing resources, and to provide an edge computing system and data storage method that can efficiently process high-resolution data , And can optimize resource utilization.
- An edge computing system of the present invention includes a cloud platform, a data filtering storage unit, and a snapshot data storage unit.
- the cloud platform is connected to the snapshot data storage unit through the data filtering storage unit; wherein the cloud platform sends configuration instructions to the data filtering storage unit.
- the data filtering storage unit filters and stores data according to the configured rules, so that the computing center of the cloud platform can more centralize data processing, and the overall computing pressure will be reduced and balanced, which can further enhance edge computing The cache capability of the device.
- the snapshot data storage unit stores the data according to the configured rules, so that the snapshot data can be efficiently processed.
- the data filtering storage unit includes an offline detection module, a recovery detection module and a business data buffer area.
- the offline detection module is connected to the business data buffer area through the data cleaning filtering module, and the business data buffer area and the recovery detection module are respectively connected to the data interaction module ; Among them, the data interaction module is connected to the cloud platform.
- the snapshot data storage unit includes a judgment module, an extraction module, and a storage module.
- the judgment module is connected to the storage module through the extraction module; and the judgment module is connected to the extraction module through a timer.
- the time for the extraction module to extract data can be specified by setting the timer. .
- the snapshot data storage unit also includes a snapshot data buffer area, which is connected to the storage module; and the snapshot data buffer area is connected to the cloud platform through the data interaction module, so that the snapshot data buffer area can transmit the stored data to the cloud platform.
- the edge computing system of the present invention also includes a business data calculation module, which is connected with the judgment module, the offline detection module and the recovery detection module, and the business data module can perform periodic calculations on the business data.
- a data storage method of the present invention adopts the aforementioned edge computing system.
- the cloud platform sends configuration instructions to the data filtering storage unit and the snapshot data storage unit, and the data filtering storage unit and the snapshot data storage unit are configured according to the configuration instructions. ;
- the data filtering storage unit filters and stores the data according to the configured rules;
- the snapshot data storage unit records and stores the data according to the configured rules.
- the offline detection module of the data filtering storage unit detects a communication interruption
- the data cleaning filtering module of the data filtering storage unit records the communication interruption time point, and the data cleaning filtering module starts the data cleaning filtering service; then the data cleaning filtering module will satisfy The data points of the filtering rules are filtered, and the data points that do not meet the filtering rules are stored in the business data buffer area; when the recovery detection module of the data filtering storage unit detects communication recovery, the business data buffer area will store the data points through the data
- the interaction module is transmitted to the cloud platform; when the business data buffer area transmits all the stored data points to the cloud platform, the data cleaning and filtering module stops the data cleaning and filtering service.
- the snapshot data storage unit configures the data channel, trigger duration, and trigger condition according to the configuration instructions.
- the trigger duration includes the pre-trigger duration T1 and the post-trigger duration T2; at the same time, the timer's timing duration is set to T2 and the timer is initialized.
- the judgment module of the snapshot data storage unit judges that the reference object meets the trigger rule, the judgment module starts the timer to start timing, and at the same time the extraction module of the snapshot data storage unit extracts the data and stores it in the storage module; when the timer expires, the storage module will The stored data is transmitted to the snapshot data cache area, and then the snapshot data cache area transmits the data to the cloud platform through the data interaction module, which can reduce the pressure of snapshot data storage and calculation, and optimize the utilization of resources.
- Figure 1 is a schematic structural diagram of an edge computing system of the present invention
- FIG. 2 is a schematic diagram of the storage process flow diagram of the data filtering storage unit of the present invention.
- FIG. 3 is a schematic diagram of the storage process flow of the snapshot data storage unit of the present invention.
- Fig. 4 is a schematic diagram 1 of trigger-based snapshot data storage of the present invention.
- Figure 5 is the second schematic diagram of trigger-based snapshot data storage of the present invention.
- an edge computing system of the present invention includes a cloud platform and an edge computing device, and the cloud platform is connected to the edge device.
- the cloud platform is used to send configuration instructions to edge devices and receive data transmitted by edge devices.
- the edge computing device includes a data filtering storage unit and a snapshot data storage unit.
- the cloud platform is connected to the snapshot data storage unit through the data filtering storage unit; wherein the cloud platform sends configuration instructions to the data filtering storage unit and the snapshot data storage unit respectively,
- the data filtering storage unit and the snapshot data storage unit are respectively configured according to the configuration instructions; it is worth noting that the data filtering storage unit filters and stores data according to the configured rules, so that the computing center of the cloud platform can more centralize data processing.
- the snapshot data storage unit of the present invention extracts and stores data according to the configured rules, so that the snapshot data can be efficiently processed. It is worth noting that the snapshot data in the present invention refers to high-resolution data.
- the data filtering storage unit of the present invention includes an offline detection module, a restoration detection module, and a business data buffer area.
- the offline detection module is connected to the business data buffer area through the data cleaning and filtering module.
- the business data buffer area and the restoration detection module are respectively connected to the data
- the interaction module is connected; the data interaction module is connected to the cloud platform.
- the offline detection module and the data cleaning and filtering module enable the collection and filtering of business data after the edge computing device communication is interrupted, and after the edge computing device communication is restored, the data interaction module can transmit the stored data to the cloud platform.
- the snapshot data storage unit of the present invention includes a judgment module, an extraction module, a storage module, and a snapshot data buffer area.
- the judgment module is connected to the storage module through the extraction module; and the judgment module is connected to the extraction module through a timer, and the timing is set.
- the device can specify the time for the extraction module to extract data.
- the snapshot data buffer area is connected with the storage module; and the snapshot data buffer area is connected with the cloud platform through the data interaction module, so that the snapshot data buffer area can transmit the stored data to the cloud platform.
- an edge computing system of the present invention further includes a business data calculation module, the business data calculation module is arranged in the edge computing device, the business data calculation module is connected to the data filtering storage unit and the snapshot data storage unit, specifically, the business The data calculation module is connected with the judgment module, the offline detection module and the recovery detection module. It is worth noting that the business data calculation module periodically calculates the data and transmits the data to the data filtering storage unit and the snapshot data storage unit.
- a data storage method of the present invention uses the above-mentioned edge computing system.
- the cloud platform sends configuration instructions to the data filtering storage unit and the snapshot data storage unit, and the data filtering storage unit and the snapshot data storage unit respectively complete the configuration according to the configuration instructions.
- the data filtering storage unit detects a communication interruption
- the data filtering storage unit filters and stores the data according to the configured rules
- the snapshot data storage unit performs data processing according to the configured rules Record storage.
- the storage process of the data filtering storage unit is as follows:
- the cloud platform sends configuration instructions to the data filtering storage unit.
- the configured instructions include communication interruption and connection recovery detection configuration, data cleaning filtering rule configuration, communication interruption triggering data offline cleaning filtering configuration and registering offline cleaning filtering service ; Afterwards, the data filtering storage unit completes the configuration according to the configuration instructions. Further, the data filtering storage unit exchanges heartbeat messages with the cloud platform.
- the offline detection module of the data filtering storage unit receives the synchronous heartbeat message of the cloud platform; when the offline detection module detects a communication interruption, the offline detection module continues The cloud platform synchronization heartbeat message is not received within 3 seconds, the offline detection module records the communication interruption time and status, and the data cleaning filter module starts the data cleaning filter service, and the data cleaning filter module collects the data of the access device and interrupts the data set of the data. Point to filter, mark the data points that meet the filter conditions, and cache the filtered interrupted data set in the business data buffer area. It should be noted that the data point filtering process is as follows:
- the filtering algorithm adopted by the data cleaning filtering module of the present invention is a dynamic threshold cleaning filtering algorithm. Specifically, the algorithm parameters are first obtained, then the algorithm engine is initialized, and the engine state is set to the normal state. It is worth noting that the state of the algorithm engine includes normal State, upper out-of-bounds state and lower out-of-bounds state. Algorithm parameters include reference value y, upper boundary threshold value y1, lower boundary threshold value y2, and hysteresis value z. The values of algorithm parameters are preset according to relevant factors of the actual data collection process.
- the upper boundary threshold y1 is set to 1.1 times the reference value
- the lower boundary threshold y2 is set to 0.9 times the reference value correspondingly.
- the current state is the upper boundary state
- the data point is greater than the upper boundary threshold y1 minus the hysteresis value z
- the data point is a point that satisfies the filtering rule
- the current upper boundary state is maintained and the data point is filtered.
- the data point is the lower boundary state
- the data point is less than the lower boundary threshold y2 plus the hysteresis value z
- the data point is the point that satisfies the filtering rules
- the current lower boundary state is maintained and the data point is filtered; in other cases, only the state is updated It is normal.
- the recovery detection module detects that the communication connection has been restored, the recovery detection module records the communication interruption recovery time point, and the service data buffer area uploads the filtered interrupted data set to the cloud platform through the data interaction module, and the interrupted data set to be filtered After all the data points are uploaded, the data cleaning and filtering module stops the data filtering service.
- the cloud platform first sends configuration instructions to the snapshot data storage unit.
- the configuration instructions include data channel, trigger time configuration and trigger rule configuration.
- the snapshot data storage unit configures the data channel, trigger time and trigger according to the configuration instructions Condition: It is worth noting that the trigger duration includes the pre-trigger duration T1 and the post-trigger duration T2 (as shown in Figure 4). Further, the snapshot data storage unit first initializes the snapshot data cache according to the characteristics of the data, then sets the timing duration of the timer to T2, and initializes the timer.
- the judgment module of the snapshot data storage unit judges whether the reference object triggers the threshold, that is, whether the snapshot trigger condition is satisfied; it is worth noting that the reference object refers to the object used for trigger condition judgment, the reference object is configured by the judgment module, and the reference object Corresponding to a reference value. If the trigger rule is met, that is, when the reference value triggers the above trigger condition at time t, the extraction module extracts the data D1 within the duration of [t-T1,t] and starts the timer to start timing. During the timer timing process, extract The module then extracts data D2 until the timer returns to zero. Finally, both D1 and D2 storage modules are stored in the snapshot data buffer area.
- the snapshot data buffer area uploads the data to the cloud data center through the data interaction module, and the cloud data center analyzes and calculates the data D1 and D2.
- time t is the trigger time
- the trigger time is the time when the reference value rises from less than the threshold to greater than the threshold.
- the reference object triggers the threshold only at time t; the other type is within [t,t+T2], the reference object triggers the threshold at least twice; the specific description is as follows:
- the thresholds and trigger rules are preset, and there are different reference values for different reference objects, so the trigger conditions for each reference object are not the same.
- the specific settings depend on the actual situation. Depending on the situation, generally speaking, such as data delay or interruption time, data resolution can be set as trigger conditions.
- the threshold is regarded as a trigger condition.
- Figure 5(a) shows that within the duration of [t,t+T2], the reference object only triggers the threshold at time t, and the extraction module extracts data D2 within the duration of [t,t+T2] Until the timer returns to zero.
- Figures 5(b) and 5(c) show that the reference object triggers the threshold more than once within the duration of [t,t+T2].
- the timing will be at ti
- the device must be initialized to restart the timing, respectively extract the snapshot data D21, D22,..., D2 within the duration of [t, t0), [t0, t1),..., [t(i-1),ti) (i+1), and extract the snapshot data D2(i+2) within the duration of [ti,ti+T2], until the timer returns to zero, we get
- Figure 5(c) shows that after the reference object triggers the threshold at time t, until the time tN drops below the threshold, the snapshot data D21, D22,..., D2(N+1) within the duration of [t, tN) are respectively extracted, And the snapshot data D2(N+2) within the duration of [tN,tN+T2], then we get Among them, N is a natural number.
- the triggering process described in Figure 5(c) may only occur part of the time period [t,t+T2], and the process may occur cyclically.
- T1 in FIG. 5 represents the duration of the snapshot data before the trigger condition is triggered
- T2 is the duration of the snapshot data after the trigger
- the length settings of T1 and T2 are also determined according to the actual situation of the reference object.
- the edge computing device of the present invention will trigger the trigger condition set by the snapshot data storage unit when the data transmission process is delayed or interrupted, start data storage, and record all the corresponding high-resolution data in the snapshot data buffer area. Upload the data storage to the cloud data center, so that the cloud data center can analyze the data, and then analyze the cause of the data transmission failure.
Abstract
一种边缘计算系统及数据存储方法，属于边缘设备数据存储领域。包括云平台、数据过滤存储单元和快照数据存储单元，云平台通过数据过滤存储单元与快照数据存储单元连接。方法为：首先云平台发送配置指令至数据过滤存储单元和快照数据存储单元，数据过滤存储单元和快照数据存储单元分别根据配置指令进行配置；而后数据过滤存储单元根据配置的规则对数据进行过滤并存储；快照数据存储单元根据配置的规则对数据进行记录存储。目的在于克服现有技术中，高分辨数据的采集、存储及处理需要消耗巨大的计算资源的不足，可以对高分辨数据进行高效的处理，并且可以优化资源的利用率。
Description
本发明涉及边缘设备数据存储技术领域，更具体地说，涉及一种边缘计算系统及数据存储方法。
数据是云平台中的重要资源。数据分辨率越高，数据量越大，从数据中挖掘出的价值越大。可是，一方面高分辨率数据的获取会导致数据采集、存储以及传输需要付出很多的代价，例如通讯带宽、数据网关和平台的数据缓存及处理能力等；另一方面海量的、高分辨率数据的数据分析也需要消耗的平台处理能力和资源。
目前主要有两种通用解决方法，一种是扩展硬件资源，增加数据网关的存储和计算能力，并把数据分析的能力下放到数据采集网关，但是该方法会因为数据量的增加而影响数据的检索速度，并且增加了运行成本。另一种是优化数据存储及分析算法，例如数据压缩算法，提高传输和计算效率，但是该方法会消耗计算能力，从而会降低数据网关的在线数据检索效率。
发明内容
1.要解决的问题
本发明的目的在于克服现有技术中，高分辨数据的采集、存储及处理需要消耗巨大的计算资源的不足，提供了一种边缘计算系统及数据存储方法，可以对高分辨数据进行高效的处理，并且可以优化资源的利用率。
2.技术方案
为了解决上述问题，本发明所采用的技术方案如下：
本发明的一种边缘计算系统，包括云平台、数据过滤存储单元和快照数据存储单元，所述云平台通过数据过滤存储单元与快照数据存储单元连接；其中，云平台发送配置指令至数据过滤存储单元和快照数据存储单元；数据过滤存储单元根据配置的规则对数据进行过滤并存储，从而使得云平台的计算中心对数据处理更加集中，整体计算压力会下降且平衡化，进一步地可以增强边缘计算设备的缓存能力。快照数据存储单元根据配置的规则对数据进行存储，从而可以对快照数据进行高效的处理。
进一步地，数据过滤存储单元包括离线检测模块、恢复检测模块和业务数据缓存区，离线检测模块通过数据清洗过滤模块与业务数据缓存区连接，业务数据缓存区和恢复检测模块分别与数据交互模块连接；其中，数据交互模块与云平台互相连接。
进一步地，快照数据存储单元包括判断模块、抽取模块和存储模块，判断模块通过抽取 模块与存储模块连接；且判断模块通过定时器与抽取模块连接，通过设置定时器可以规定抽取模块抽取数据的时间。
进一步地，快照数据存储单元还包括快照数据缓存区，快照数据缓存区与存储模块连接；且快照数据缓存区通过数据交互模块与云平台连接，从而快照数据缓存区可以将存储的数据传输至云平台。本发明的边缘计算系统还包括业务数据计算模块，该业务数据计算模块与判断模块、离线检测模块和恢复检测模块连接，业务数据模块可以对业务数据进行周期性计算。
本发明的一种数据存储方法，采用上述的一种边缘计算系统，首先云平台发送配置指令至数据过滤存储单元和快照数据存储单元，数据过滤存储单元和快照数据存储单元分别根据配置指令进行配置；而后数据过滤存储单元根据配置的规则对数据进行过滤并存储；快照数据存储单元根据配置的规则对数据进行记录存储。
其中，当数据过滤存储单元的离线检测模块检测到通讯中断时，数据过滤存储单元的数据清洗过滤模块记录通讯中断时间点，并且数据清洗过滤模块启动数据清洗过滤服务；而后数据清洗过滤模块将满足过滤规则的数据点进行过滤，并将不满足过滤规则的数据点存储至业务数据缓存区；当数据过滤存储单元的恢复检测模块检测到通讯恢复时，业务数据缓存区将存储的数据点经数据交互模块传输至云平台；当业务数据缓存区将存储的数据点全部传输至云平台后，数据清洗过滤模块停止数据清洗过滤服务。通过上述过滤存储方法，从而使得云平台的计算中心对数据处理更加集中，整体计算压力会下降且平衡化，进一步地可以增强边缘计算设备的缓存能力。
此外，快照数据存储单元根据配置指令配置数据通道、触发时长和触发条件，触发时长包括触发前时长T1和触发后时长T2；同时将定时器的定时时长设为T2，并对定时器进行初始化。当快照数据存储单元的判断模块判断参考对象满足触发规则时，判断模块启动定时器开始计时，同时快照数据存储单元的抽取模块抽取数据并存储至存储模块；当定时器计时完毕时，存储模块将存储的数据传输至快照数据缓存区，而后快照数据缓存区将数据经数据交互模块传输至云平台，从而可以减小快照数据存储和计算的压力，也优化了资源的利用率。
图1为本发明的一种边缘计算系统结构示意图；
图2为本发明的数据过滤存储单元的存储过程流程示意图；
图3为本发明的快照数据存储单元的存储过程流程示意图；
图4为本发明的基于触发的快照数据存储示意图一；
图5为本发明的基于触发的快照数据存储示意图二。
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例；而且，各个实施例之间不是相对独立的，根据需要可以相互组合，从而达到更优的效果。因此，以下对在附图中提供的本发明的实施例的详细描述并非旨在限制要求保护的本发明的范围，而是仅仅表示本发明的选定实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
为进一步了解本发明的内容，结合附图和实施例对本发明作详细描述。
实施例1
结合图1所示，本发明的一种边缘计算系统，包括云平台和边缘计算设备，云平台与边缘设备相连接。需要说明的是，云平台用于对边缘设备发送配置指令以及接收边缘设备传输的数据。具体地，边缘计算设备包括数据过滤存储单元和快照数据存储单元，云平台通过数据过滤存储单元与快照数据存储单元连接；其中，云平台分别发送配置指令至数据过滤存储单元和快照数据存储单元，数据过滤存储单元和快照数据存储单元分别根据配置指令完成配置；值得说明的是，数据过滤存储单元根据配置的规则对数据进行过滤并存储，从而使得云平台的计算中心对数据处理更加集中，整体计算压力会下降且平衡化，进一步地可以增强边缘计算设备的缓存能力。此外，本发明的快照数据存储单元根据配置的规则对数据进行抽取存储，从而可以对快照数据进行高效的处理。值得说明的是，本发明的快照数据指的是高分辨率数据。
进一步地，本发明的数据过滤存储单元包括离线检测模块、恢复检测模块和业务数据缓存区，离线检测模块通过数据清洗过滤模块与业务数据缓存区连接，业务数据缓存区和恢复检测模块分别与数据交互模块连接；数据交互模块与云平台互相连接。通过离线检测模块和数据清洗过滤模块使得在边缘计算设备通讯中断后，能够采集并过滤业务数据，且在边缘计算设备通讯恢复后，数据交互模块能够将存储的数据传输至云平台。
更进一步地，本发明的快照数据存储单元包括判断模块、抽取模块、存储模块和快照数据缓存区，判断模块通过抽取模块与存储模块连接；且判断模块通过定时器与抽取模块连接，通过设置定时器可以规定抽取模块抽取数据的时间。快照数据缓存区与存储模块连接；且快照数据缓存区通过数据交互模块与云平台连接，从而快照数据缓存区可以将存储的数据传输至云平台。此外，本发明的一种边缘计算系统还包括业务数据计算模块，该业务数据计算模块设置于边缘计算设备内，业务数据计算模块与数据过滤存储单元和快照数据存储单元连接，具体地，该业务数据计算模块与判断模块、离线检测模块和恢复检测模块连接。值得说明的 是，业务数据计算模块对数据进行周期性计算并将数据传输至数据过滤存储单元和快照数据存储单元。
本发明的一种数据存储方法，采用上述的一种边缘计算系统，首先云平台发送配置指令至数据过滤存储单元和快照数据存储单元，数据过滤存储单元和快照数据存储单元分别根据配置指令完成配置；而后当数据过滤存储单元检测到通讯中断时，数据过滤存储单元根据配置的规则对数据进行过滤并存储；当快照数据存储单元的触发规则满足时，快照数据存储单元根据配置的规则对数据进行记录存储。具体过程如下：
A、数据过滤存储单元的存储过程如下：
结合图2所示，首先云平台发送配置指令至数据过滤存储单元，配置的指令包括通讯中断和连接恢复检测配置、数据清洗过滤规则配置、通讯中断触发数据离线清洗过滤配置和注册离线清洗过滤服务；之后数据过滤存储单元根据配置指令完成配置。进一步地，数据过滤存储单元与云平台交互心跳报文，具体地，数据过滤存储单元的离线检测模块接收云平台的同步心跳报文；当离线检测模块检测到通讯中断时，即离线检测模块连续3秒内没收到云平台同步心跳报文，离线检测模块记录通讯中断时间点和状态，并且数据清洗过滤模块启动数据清洗过滤服务，数据清洗过滤模块采集接入设备数据，对中断数据集的数据点进行过滤，并标记满足过滤条件的数据点，并将过滤后的中断数据集缓存至业务数据缓存区。需要说明的是，数据点过滤过程具体如下：
本发明数据清洗过滤模块采取的过滤算法为动态阈值清洗过滤算法，具体地，首先获取算法参数，再初始化算法引擎，并将引擎状态设置为正常状态，值得说明的是，算法引擎的状态包括正常状态、上越界状态和下越界状态。算法参数包括参考值y、上边界阈值y1、下边界阈值y2和迟滞值z，算法参数的值都是根据实际数据采集过程的相关因素预设的，比如若参考值y为100，可将上边界阈值y1设为参考值的1.1倍，对应地将下边界阈值y2设为参考值的0.9倍。当通讯中断时，获取中断数据集的数据点。当前状态为正常状态时，若数据点大于上边界阈值y1，那么该数据点是满足过滤规则的点，则将改数据点过滤并将引擎状态更新为上越界状态；若数据点小于下边界阈值y2，那么数据点是满足过滤规则的点，则将该数据点过滤且将引擎状态更新为下越界状态。
当前状态为上越界状态时，若数据点大于上边界阈值y1减去迟滞值z，那么数据点是满足过滤规则的点，保持当前的上越界状态并过滤该数据点。当前状态为下越界状态时，若数据点小于下边界阈值y2加上迟滞值z，那么数据点是满足过滤规则的点，保持当前的下越界状态并过滤该数据点；其余的情况仅更新状态为正常状态。
而后恢复检测模块检测到通讯连接已恢复时，恢复检测模块记录通讯中断恢复时间点， 并且业务数据缓存区将过滤后的中断数据集经数据交互模块上传至云平台，待过滤后的中断数据集的数据点全部上传完毕后，数据清洗过滤模块停止数据过滤服务。
B、快照数据存储单元的存储过程如下：
结合图3所示，首先云平台发送配置指令至快照数据存储单元，配置指令包括数据的数据通道、触发时长配置和触发规则配置，而后快照数据存储单元根据配置指令配置数据通道、触发时长和触发条件；值得说明的是，触发时长包括触发前时长T1和触发后时长T2(如图4所示)。进一步地，快照数据存储单元根据数据的特性，先对快照数据缓存去进行初始化，再将定时器的定时时长设为T2，并且对定时器进行初始化。之后快照数据存储单元的判断模块判断参考对象是否触发阈值，即是否满足快照触发条件；值得说明的是，参考对象是指用于触发条件判断的对象，该参考对象由判断模块配置，并且参考对象对应有参考值。如果满足触发规则，即当参考值在t时刻触发上述触发条件时，抽取模块抽取[t-T1,t]时长内的数据D1，同时启动定时器开始定时，在定时器的定时过程中，抽取模块再抽取数据D2，直至定时器归零。最后将D1和D2存储都存储模块并存储至快照数据缓存区，快照数据缓存区将数据经数据交互模块上传至云数据中心，云数据中心对数据D1和D2进行分析计算。其中，t时刻为触发时刻，该触发时刻为参考值从小于阈值上升至大于阈值的时刻。
值得说明的是，在定时器开始定时且抽取模块再抽取数据D2的过程中，包括以下两类情况：
一类是在[t,t+T2]时长内，参考对象仅在t时刻触发阈值；另一类是在[t,t+T2]时长内，参考对象至少两次触发阈值；具体描述如下：
由于触发条件一般包括阈值和触发规则，阈值和触发规则都是预先设定的，针对不同的参考对象有不同的参考值，因而每个参考对象的触发条件也不尽相同，具体设定视实际情况而定，一般来讲，诸如数据延迟或中断时间、数据的分辨率都可以设为触发条件。以下，为了方便描述，将阈值即视为触发条件。
结合图5所示，图5(a)表示的是，在[t,t+T2]时长内，参考对象仅在t时刻触发阈值，抽取模块抽取[t,t+T2]时长内的数据D2，直至定时器归零。图5(b)和5(c)表示的是在[t,t+T2]时长内，参考对象不止一次触发阈值，若t时刻和ti时刻为触发时刻，i为自然数，则在ti时刻定时器都要初始化重新开始定时，分别抽取[t,t0),[t0,t1),...,[t(i-1),ti)时长内的快照数据D21,D22,...,D2(i+1)，并抽取[ti,ti+T2]时长内的快照数据D2(i+2)，直至定时器归零，得到
具体地，图5(b)表示参考对象在t时刻触发阈值后，直至t0时刻下降至小于阈值，则抽 取[t,t0)时长内的快照数据D21，以及[t0,t0+T2]时长内的快照数据D22，得到D2＝D21+D22。
图5(c)表示参考对象在t时刻触发阈值后，直至tN时刻下降至小于阈值，分别抽取[t,tN)时长内的快照数据D21,D22,...,D2(N+1)，以及[tN,tN+T2]时长内的快照数据D2(N+2)，则得到
其中，N为自然数。图5(c)描述的触发过程在[t,t+T2]时长内可能只出现其中一部分，该过程可能循环发生。
值得说明的是，图5中T1表示的是触发触发条件前的快照数据时长，T2则是触发后的快照数据时长，T1和T2的长度设置也是根据参考对象的实际情况而定。
本发明的边缘计算设备在数据传输过程发生故障延迟或中断时，就会触发快照数据存储单元设置的触发条件，开启数据的存储，将相应的高分辨率数据全部记录到快照数据缓存区，最终将数据存储上传至云数据中心，从而云数据中心可以对数据进行分析，进而可以分析数据传输发生故障的原因。在上文中结合具体的示例性实施例详细描述了本发明。但是，应当理解，可在不脱离由所附权利要求限定的本发明的范围的情况下进行各种修改和变型。详细的描述和附图应仅被认为是说明性的，而不是限制性的，如果存在任何这样的修改和变型，那么它们都将落入在此描述的本发明的范围内。此外，背景技术旨在为了说明本技术的研发现状和意义，并不旨在限制本发明或本申请和本发明的应用领域。
Claims (10)
- 一种边缘计算系统，其特征在于：包括云平台、数据过滤存储单元和快照数据存储单元，所述云平台通过数据过滤存储单元与快照数据存储单元连接；其中，云平台发送配置指令至数据过滤存储单元和快照数据存储单元；数据过滤存储单元根据配置的规则对数据进行过滤并存储；快照数据存储单元根据配置的规则对数据进行存储。
- 根据权利要求1所述的一种边缘计算系统，其特征在于：所述数据过滤存储单元包括离线检测模块、恢复检测模块和业务数据缓存区，离线检测模块通过数据清洗过滤模块与业务数据缓存区连接，业务数据缓存区和恢复检测模块分别与数据交互模块连接；其中，数据交互模块与云平台互相连接。
- 根据权利要求2所述的一种边缘计算系统，其特征在于：所述快照数据存储单元包括判断模块、抽取模块和存储模块，判断模块通过抽取模块与存储模块连接；且判断模块通过定时器与抽取模块连接。
- 根据权利要求3所述的一种边缘计算系统，其特征在于：所述快照数据存储单元还包括快照数据缓存区，所述快照数据缓存区与存储模块连接；且快照数据缓存区通过数据交互模块与云平台连接。
- 根据权利要求3所述的一种边缘计算系统，其特征在于：还包括业务数据计算模块，所述业务数据计算模块与判断模块、离线检测模块和恢复检测模块连接。
- 一种数据存储方法，其特征在于：采用权利要求1～5任一项所述的一种边缘计算系统，首先云平台发送配置指令至数据过滤存储单元和快照数据存储单元，数据过滤存储单元和快照数据存储单元分别根据配置指令进行配置；而后数据过滤存储单元根据配置的规则对数据进行过滤并存储；快照数据存储单元根据配置的规则对数据进行记录存储。
- 根据权利要求6所述的一种数据存储方法，其特征在于：当数据过滤存储单元的离线检测模块检测到通讯中断时，数据过滤存储单元的数据清洗过滤模块记录通讯中断时间点，并且数据清洗过滤模块启动数据清洗过滤服务；而后数据清洗过滤模块将满足过滤规则的数据点进行过滤，并将不满足过滤规则的数据点存储至业务数据缓存区；当数据过滤存储单元的恢复检测模块检测到通讯恢复时，业务数据缓存区将存储的数据点经数据交互模块传输至云平台。
- 根据权利要求6所述的一种数据存储方法，其特征在于：当快照数据存储单元的判断模块判断参考对象满足触发规则时，判断模块启动定时器开始计时，同时快照数据存储单元的抽取模块抽取数据并存储至存储模块；当定时器计时完毕时，存储模块将存储的数据传输至快照数据缓存区，而后快照数据缓存区将数据经数据交互模块传输至云平台。
- 根据权利要求7所述的一种数据存储方法，其特征在于：当业务数据缓存区将存储的 数据点全部传输至云平台后，数据清洗过滤模块停止数据清洗过滤服务。
- 根据权利要求8所述的一种数据存储方法，其特征在于：快照数据存储单元根据配置指令配置数据通道、触发时长和触发条件，所述触发时长包括触发前时长T1和触发后时长T2；同时将定时器的定时时长设为T2，并对定时器进行初始化。
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910669324.7A CN110572427A (zh) | 2019-07-24 | 2019-07-24 | 应用于数据网关的离线数据过滤存储的方法及系统 |
| CN201910669262.XA CN110543282A (zh) | 2019-07-24 | 2019-07-24 | 一种应用于边缘计算网关的触发快照存储方法及系统 |
| CN201910669324.7 | 2019-07-24 | ||
| CN201910669262.X | 2019-07-24 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2021012331A1 true WO2021012331A1 (zh) | 2021-01-28 |
Family
ID=74192474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2019/100922 WO2021012331A1 (zh) | 2019-07-24 | 2019-08-16 | 一种边缘计算系统及数据存储方法 |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2021012331A1 (zh) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103259845A (zh) * | 2013-04-12 | 2013-08-21 | 赵利林 | 基于网络中断的数据备份任务的改进方法 |
| CN103902422A (zh) * | 2012-12-25 | 2014-07-02 | 中国电信股份有限公司 | 对用户操作行为进行监控的方法、装置与云电脑系统 |
| US20160321455A1 (en) * | 2015-04-29 | 2016-11-03 | International Business Machines Corporation | Security policy generation based on snapshots of similar virtual machines |
| CN109034731A (zh) * | 2018-07-02 | 2018-12-18 | 泛亚汽车技术中心有限公司 | 用于园区员工管理的动态数据处理系统及方法 |
| CN110045696A (zh) * | 2019-03-29 | 2019-07-23 | 山东钢铁集团日照有限公司 | 一种基于移动互联技术的智能环保与能源监控预警平台 |
-
2019
- 2019-08-16 WO PCT/CN2019/100922 patent/WO2021012331A1/zh active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902422A (zh) * | 2012-12-25 | 2014-07-02 | 中国电信股份有限公司 | 对用户操作行为进行监控的方法、装置与云电脑系统 |
| CN103259845A (zh) * | 2013-04-12 | 2013-08-21 | 赵利林 | 基于网络中断的数据备份任务的改进方法 |
| US20160321455A1 (en) * | 2015-04-29 | 2016-11-03 | International Business Machines Corporation | Security policy generation based on snapshots of similar virtual machines |
| CN109034731A (zh) * | 2018-07-02 | 2018-12-18 | 泛亚汽车技术中心有限公司 | 用于园区员工管理的动态数据处理系统及方法 |
| CN110045696A (zh) * | 2019-03-29 | 2019-07-23 | 山东钢铁集团日照有限公司 | 一种基于移动互联技术的智能环保与能源监控预警平台 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103200046B (zh) | 监控网元设备性能的方法及系统 | |
| CN1795446B (zh) | 计算机集群及其中的计算机节点和传送状态信息的方法 | |
| CN103763396A (zh) | 基于多协议并行采集技术的能耗数据采集器及其采集方法 | |
| CN105610913A (zh) | 通信网络中长连接的心跳保活方法及装置 | |
| CN105357061A (zh) | 一种基于大数据流处理技术的运维监控分析系统 | |
| CN103297543A (zh) | 一种基于计算机集群作业调度的方法 | |
| EP3611879A1 (en) | Data traffic statistics method and device based on block chain | |
| CN100571170C (zh) | 一种通讯链路故障的检测及恢复方法 | |
| CN106331047A (zh) | 一种集群设备性能同步统计方法及系统 | |
| CN108566363A (zh) | 基于流式计算的暴力破解确定方法和系统 | |
| TW202127834A (zh) | 用於行動通訊系統之威脅偵測系統及其中心裝置與本地裝置 | |
| CN101141292B (zh) | 一种实现网络管理系统中告警自动刷新的方法 | |
| CN108833366B (zh) | 基于as6802协议的控制帧压缩方法 | |
| WO2021012331A1 (zh) | 一种边缘计算系统及数据存储方法 | |
| CN109831316A (zh) | 海量日志实时分析系统、实时分析方法及可读存储介质 | |
| CN107800711A (zh) | 一种OpenFlow控制器抵御DDoS攻击的方法 | |
| CN102413191A (zh) | 一种基于浏览器的web即时通讯方法 | |
| CN102546659B (zh) | 面向远过程调用的持久tcp连接方法 | |
| CN104917768A (zh) | 基于多协议并行采集技术的能耗数据采集器及其采集方法 | |
| CN106959885A (zh) | 一种虚拟机高可用实现系统及其实现方法 | |
| CN204631259U (zh) | 一种基于精密时钟同步协议的矿震监测装置 | |
| CN105357060A (zh) | 一种获取网络设备性能数据的方法、装置和系统 | |
| CN104333469B (zh) | 一种存储系统光纤通道fc会话管理方法和系统 | |
| CN116010169A (zh) | 一种基于云原生技术的云平台rds数据库迁移容灾方法 | |
| CN205693803U (zh) | 基于无线网络的通信和图像采集的监控装置 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19938381 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 19938381 Country of ref document: EP Kind code of ref document: A1 |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 19938381 Country of ref document: EP Kind code of ref document: A1 |
|
| 32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 09.11.2022) |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 19938381 Country of ref document: EP Kind code of ref document: A1 |