CN114979122B - Big data storage rate optimization method based on industrial Internet - Google Patents
Big data storage rate optimization method based on industrial Internet Download PDFInfo
- Publication number
- CN114979122B CN114979122B CN202210606901.XA CN202210606901A CN114979122B CN 114979122 B CN114979122 B CN 114979122B CN 202210606901 A CN202210606901 A CN 202210606901A CN 114979122 B CN114979122 B CN 114979122B
- Authority
- CN
- China
- Prior art keywords
- data
- control unit
- central control
- rate
- transfer server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000013500 data storage Methods 0.000 title claims abstract description 37
- 238000005457 optimization Methods 0.000 title claims abstract description 8
- 238000012546 transfer Methods 0.000 claims abstract description 156
- 230000001934 delay Effects 0.000 claims abstract 3
- 230000005540 biological transmission Effects 0.000 claims description 58
- 230000003111 delayed effect Effects 0.000 claims 4
- 230000001105 regulatory effect Effects 0.000 claims 4
- 238000003672 processing method Methods 0.000 description 9
- 238000011161 development Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 101001121408 Homo sapiens L-amino-acid oxidase Proteins 0.000 description 1
- 102100026388 L-amino-acid oxidase Human genes 0.000 description 1
- 101100012902 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) FIG2 gene Proteins 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0894—Packet rate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
Description
技术领域Technical Field
本发明涉及大数据存储领域,尤其涉及一种基于工业互联网的大数据存储速率优化方法。The present invention relates to the field of big data storage, and in particular to a big data storage rate optimization method based on industrial Internet.
背景技术Background Art
“大数据”通常指的是那些数量巨大、难于收集、处理、分析的数据集,亦指那些在传统基础设施中长期保存的数据。随着大数据应用的爆发性增长,它已经衍生出了自己独特的架构,而且也直接推动了存储、网络以及计算技术的发展。毕竟处理大数据这种特殊的需求是一个新的挑战。大数据存储和传统的数据存储的不同。数据通常以每年增长50%的速度快速激增,尤其是非结构化数据。随着科技的进步,有越来越多的传感器采集数据、移动设备、社交多媒体等等,所以数据只可能继续增长,"Big data" usually refers to data sets that are huge in number and difficult to collect, process, and analyze, and also refers to data that is stored for a long time in traditional infrastructure. With the explosive growth of big data applications, it has derived its own unique architecture and has directly promoted the development of storage, network, and computing technologies. After all, dealing with the special needs of big data is a new challenge. Big data storage is different from traditional data storage. Data usually surges rapidly at a rate of 50% per year, especially unstructured data. With the advancement of technology, there are more and more sensors collecting data, mobile devices, social multimedia, etc., so data can only continue to grow.
随着信息社会的发展,越来越多的信息被数字化,特别是随着互联网的发展,数据呈爆炸式增长。从存储服务的发展趋势来看,一方面,对数据存储的需求越来越大;另一方面,它对有效的数据管理提出了更高的要求,同时存储容量的快速扩张,对数据的传输速率提出了更大的需求,在数据传输过程中,数据中转服务器中的数据接收速率与数据上传速率之间的不匹配导致数据的大面积丢失。With the development of the information society, more and more information is digitized, especially with the development of the Internet, data is growing explosively. From the development trend of storage services, on the one hand, the demand for data storage is increasing; on the other hand, it puts forward higher requirements for effective data management. At the same time, the rapid expansion of storage capacity has put forward greater requirements for data transmission rate. During the data transmission process, the mismatch between the data receiving rate and the data upload rate in the data transfer server leads to large-scale data loss.
发明内容Summary of the invention
为此,本发明提供一种基于工业互联网的大数据存储速率优化方法,用以克服现有技术中数据接收速率与数据上传速率之间的不匹配问题。To this end, the present invention provides a big data storage rate optimization method based on industrial Internet, so as to overcome the mismatch problem between data receiving rate and data uploading rate in the prior art.
为实现上述目的,本发明一种基于工业互联网的大数据存储速率优化方法,包括:To achieve the above object, the present invention provides a method for optimizing the storage rate of big data based on the industrial Internet, comprising:
步骤s1,数据终端将其内部的数据上传至数据中转服务器;Step s1, the data terminal uploads its internal data to the data transfer server;
步骤s2,所述数据中转服务器接收所述数据终端上传的数据,接收过程中,中控单元根据所述数据的重要程度依次对各数据建立对应的优先级标签,中控单元检测数据中转服务器在接收所述终端数据过程中的实际接收速率、判定该接收速率是否合格并在判定接收速率过快或过慢时根据结果判定数据中转服务器接收速率过快或过慢的实际原因;Step s2, the data transfer server receives the data uploaded by the data terminal. During the receiving process, the central control unit establishes corresponding priority tags for each data in turn according to the importance of the data. The central control unit detects the actual receiving rate of the data transfer server in the process of receiving the terminal data, determines whether the receiving rate is qualified, and determines the actual reason why the receiving rate of the data transfer server is too fast or too slow according to the result when the receiving rate is determined to be too fast or too slow;
步骤s3,所述中控单元在判定所述数据中转服务器接收速率过过快或过慢时调节所述数据终端的上传速率或调节所述数据中转服务器的上传速率或根据数据的优先级标签对对应的数据进行推迟上传以使数据中转服务器的接收速率符合标准,中控单元计算单个周期内的综合传输速率并检测该速率是否符合标准;Step s3, when the central control unit determines that the receiving rate of the data transfer server is too fast or too slow, it adjusts the upload rate of the data terminal or the upload rate of the data transfer server or postpones the upload of the corresponding data according to the priority tag of the data so that the receiving rate of the data transfer server meets the standard. The central control unit calculates the comprehensive transmission rate in a single cycle and detects whether the rate meets the standard;
步骤s4,所述中控单元在判定所述综合传输速率你符合标准时调节数据中转服务器的接收速率和上传速率以使综合传输速率符合预设值,中控单元计算数据的完整度并检测数据的完整度是否符合标准;Step s4, when the central control unit determines that the comprehensive transmission rate meets the standard, the central control unit adjusts the receiving rate and the uploading rate of the data transfer server so that the comprehensive transmission rate meets the preset value, and the central control unit calculates the integrity of the data and detects whether the integrity of the data meets the standard;
步骤s5,所述中控单元在判定所述数据的完整度不符合标准时,所述中控单元调节综合传输速率以使所述数据的完整度符合预设标准;Step s5, when the central control unit determines that the integrity of the data does not meet the standard, the central control unit adjusts the comprehensive transmission rate so that the integrity of the data meets the preset standard;
步骤s6,当所述中控单元完成对对应的参数的调节时,中控单元重新检测数据中转服务器在接收所述终端数据过程中的实际接收速率,若该接收速率合格,中控单元判定针对该数据的存储速率的优化完成。Step s6, when the central control unit completes the adjustment of the corresponding parameters, the central control unit re-detects the actual receiving rate of the data transfer server in the process of receiving the terminal data. If the receiving rate is qualified, the central control unit determines that the optimization of the storage rate of the data is completed.
进一步地,所述中控单元在所述数据中转服务器接收单个所述数据终端上传的数据时将数据中转服务器针对该数据终端的数据接收速率记为Qa并根据Qa判定数据中转服务器针对该数据终端的数据接收速率是否合格,所述中控单元中设有预设接收速率Qa0,Furthermore, when the data transfer server receives data uploaded by a single data terminal, the central control unit records the data receiving rate of the data transfer server for the data terminal as Qa and determines whether the data receiving rate of the data transfer server for the data terminal is qualified according to Qa. The central control unit is provided with a preset receiving rate Qa0.
若0.9×Qa0≤Qa≤1.1×Qa0,所述中控单元初步判定所述数据中转服务器针对所述数据终端的数据接收速率合格,中控单元检测数据中转服务器的上传速率并计算综合数据传输速率记为Q并检测数据的完整度记为S;If 0.9×Qa0≤Qa≤1.1×Qa0, the central control unit preliminarily determines that the data receiving rate of the data transfer server for the data terminal is qualified, and the central control unit detects the upload rate of the data transfer server and calculates the comprehensive data transmission rate recorded as Q and detects the integrity of the data recorded as S;
若Qa<0.9×Qa0,所述中控单元初步判定所述数据中转服务器针对所述数据终端的数据接收速率不符合标准,中控单元计算Qa与Qa0的差值△Qax并根据△Qax确定该接收速率低于预设标准的实际原因,设定△Qax=Qa0-Qa;If Qa<0.9×Qa0, the central control unit preliminarily determines that the data receiving rate of the data transfer server for the data terminal does not meet the standard, and the central control unit calculates the difference △Qax between Qa and Qa0 and determines the actual reason why the receiving rate is lower than the preset standard according to △Qax, and sets △Qax=Qa0-Qa;
若Qa>1.1×Qa0,所述中控单元判定所述数据中转服务器针对所述数据终端的数据接收速率过快,中控单元计算数据接收速率与预设数据接收速率的差值△Qay并根据△Qay判定是否对所述数据中转服务器的实际接收速率Qa进行限制,设定△Qay=Qa-Qa0。If Qa>1.1×Qa0, the central control unit determines that the data receiving rate of the data relay server for the data terminal is too fast. The central control unit calculates the difference △Qay between the data receiving rate and the preset data receiving rate and determines whether to limit the actual receiving rate Qa of the data relay server based on △Qay, and sets △Qay=Qa-Qa0.
进一步地,所述中控单元初步判定所述数据中转服务器针对所述数据终端的数据接收速率不符合标准时,中控单元根据△Qax对该接收速率低于预设标准的实际原因进行判定,中控单元内设有第一预设数据接收速率差值△Qax1、第二预设数据接收速率差值△Qax2、第二预设数据接收速率差值△Qax3、第一预设推迟任务比例b1和第二预设推迟任务比例b2,其中△Qax1<△Qax2<△Qax3,b1<b2<1,Further, when the central control unit preliminarily determines that the data receiving rate of the data transfer server for the data terminal does not meet the standard, the central control unit determines the actual reason why the receiving rate is lower than the preset standard according to △Qax, and the central control unit is provided with a first preset data receiving rate difference △Qax1, a second preset data receiving rate difference △Qax2, a second preset data receiving rate difference △Qax3, a first preset postponed task ratio b1 and a second preset postponed task ratio b2, wherein △Qax1<△Qax2<△Qax3, b1<b2<1,
若△Qax≤△Qax1,所述中控单元二次判定数据接收速率在可允许范围内,中控单元检测所述数据终端的上传速率qb并根据qb判定所述数据中转服务器接收数据的接收速率与该终端上传数据的上传速率是否平衡;If △Qax≤△Qax1, the central control unit determines that the data receiving rate is within the allowable range, and the central control unit detects the upload rate qb of the data terminal and determines whether the receiving rate of the data transfer server receiving data is balanced with the upload rate of the terminal uploading data according to qb;
若△Qax1<△Qax≤△Qax2,中控单元判定所述数据中转服务器实际负载高于预设标准、根据数据优先级控制数据中转服务器推迟接收对应所述终端输出的数据以降低数据中转服务器的运行负载并将推迟数据比例设置为b1,数据接收总数量记为M,设定推迟数据接收数量为M’,设定M’=M×b1;If △Qax1<△Qax≤△Qax2, the central control unit determines that the actual load of the data transfer server is higher than the preset standard, controls the data transfer server to postpone receiving the data outputted by the corresponding terminal according to the data priority to reduce the operation load of the data transfer server, and sets the postponed data ratio to b1, the total number of data received is recorded as M, the number of postponed data received is set to M', and M'=M×b1;
若△Qax2<△Qax≤△Qax3,中控单元判定所述数据中转服务器实际负载高于预设标准,根据数据优先级推迟数据中转服务器的对应数据以降低数据中转服务器的运行负载并将推迟数据比例设置为b2,设定M’=M×b1;If △Qax2<△Qax≤△Qax3, the central control unit determines that the actual load of the data transfer server is higher than the preset standard, postpones the corresponding data of the data transfer server according to the data priority to reduce the operation load of the data transfer server and sets the postponed data ratio to b2, and sets M'=M×b1;
若△Qax>△Qax3,所述中控单元判定网路环境出现问题、检测所述数据中转服务器上传速率Qb并根据Qa与Qb的差值判定所述数据中转服务器的接收速率与数据中转服务器的上传速率是否平衡。If ΔQax>ΔQax3, the central control unit determines that there is a problem with the network environment, detects the upload rate Qb of the data transfer server, and determines whether the receiving rate of the data transfer server and the upload rate of the data transfer server are balanced according to the difference between Qa and Qb.
进一步地,所述数据中转服务器接收各所述终端上传的数据时,中控单元根据数据中转服务器中上传的数据的重要程度依次对各数据建立对应的优先级标签;所述数据的重要程度由高到低的顺序包括一级、二级、三级和四级;当所述中控单元判定需根据数据优先级推迟所述数据中转服务器推迟接收对应所述终端输出的数据以降低数据中转服务器的运行负载,中控单元按照四级至一级的数据标签的顺序推迟接收对应级别的数据。Furthermore, when the data transfer server receives the data uploaded by each of the terminals, the central control unit establishes corresponding priority tags for each data in turn according to the importance of the data uploaded in the data transfer server; the importance of the data includes level one, level two, level three and level four in order from high to low; when the central control unit determines that it is necessary to postpone the data transfer server to postpone receiving the data output by the corresponding terminal according to the data priority to reduce the operating load of the data transfer server, the central control unit postpones receiving the data of the corresponding level in the order of data tags from level four to level one.
进一步地,所述中控单元二次判定所述数据接收速率在可允许范围内时,中控单元检测所述数据终端上传速率qb、计算qa和qb的差值△q并根据△q判定是否需将qb调节至对应值,设定△q=Qa-qb,中控单元内设有所述数据中转服务器的第一上传数据速率调节系数α1和第二上传数据速率调节系数α2,其中,0<α2<1<α1,Further, when the central control unit determines for the second time that the data receiving rate is within the allowable range, the central control unit detects the data terminal upload rate qb, calculates the difference △q between qa and qb, and determines whether qb needs to be adjusted to the corresponding value according to △q, and sets △q=Qa-qb. The central control unit is provided with a first upload data rate adjustment coefficient α1 and a second upload data rate adjustment coefficient α2 of the data transfer server, wherein 0<α2<1<α1,
若△q>0,所述中控单元判定使用α1调节qb以将qb增加至对应值;If Δq>0, the central control unit determines to use α1 to adjust qb to increase qb to the corresponding value;
若△q<0,所述中控单元判定使用α2调节Qb以将qb减小至对应值;If Δq<0, the central control unit determines to use α2 to adjust Qb to reduce qb to a corresponding value;
当所述中控单元使用αi对qb进行调节时,设定i=1,2,调节后的数据上传速率记为qb’,设定qb’=qb×αi;所述中控单元控制所述数据中转服务器使用qb’的速率上传数据并计算对应综合传输速率Q。When the central control unit uses αi to adjust qb, i is set to 1, 2, the adjusted data upload rate is recorded as qb’, and qb’=qb×αi is set; the central control unit controls the data transfer server to upload data at the rate of qb’ and calculates the corresponding comprehensive transmission rate Q.
进一步地,所述中控单元判定网路环境出现问题时,中控单元计算计算Qa和Qb的差值△Q’并根据△Q’判定是否需将Qb调节至对应值,设定△Q’=Qa-Qb,中控单元内设有所述数据中转服务器第一上传数据速率预设差值△Q1、第二上传数据速率预设差值△Q2、第一上传数据速率调节系数β1和第二上传数据速率调节系数β2,其中△Q1<△Q2,β1<β2<1;Further, when the central control unit determines that there is a problem with the network environment, the central control unit calculates the difference △Q' between Qa and Qb and determines whether Qb needs to be adjusted to the corresponding value based on △Q', and sets △Q'=Qa-Qb. The central control unit is provided with a first upload data rate preset difference △Q1, a second upload data rate preset difference △Q2, a first upload data rate adjustment coefficient β1, and a second upload data rate adjustment coefficient β2 of the data transfer server, wherein △Q1<△Q2, β1<β2<1;
若△Q’<△Q1,所述中控单元判定使用β1调节Qb;If ΔQ’<ΔQ1, the central control unit determines to use β1 to adjust Qb;
若△Q1≤△Q’<△Q2,所述中控单元判定使用β2调节Qb;If △Q1≤△Q’<△Q2, the central control unit determines to use β2 to adjust Qb;
若△Q’≥△Q2,所述中控单元判定网络故障并发出故障警报;If △Q’≥△Q2, the central control unit determines that the network is faulty and issues a fault alarm;
当所述中控单元使用βi对Qb进行调节时,设定i=1,2,调节后的数据上传速率记为Qb”,设定Qb”=Qb×βi;所述中控单元控制所述数据中转服务器使用Qb”的速率上传数据并计算对应综合传输速率Q。When the central control unit uses βi to adjust Qb, i is set to 1, 2, and the adjusted data upload rate is recorded as Qb", and Qb"=Qb×βi is set; the central control unit controls the data transfer server to upload data at the rate of Qb" and calculates the corresponding comprehensive transmission rate Q.
进一步地,所述中控单元判定数据接收速率过快时,中控单元根据△Qay判定是否对中转点的实际接收速率Qa进行限制,中控单元内设有第一预设数据接收速率差值△Qay1、第二预设数据接收速率差值△Qay2、第一数据接收速率调节系数γ1和第一数据接收速率调节系数γ2,其中,△Qay1<△Qay2,0<γ1<γ2<1,Further, when the central control unit determines that the data receiving rate is too fast, the central control unit determines whether to limit the actual receiving rate Qa of the transfer point according to ΔQay, and the central control unit is provided with a first preset data receiving rate difference ΔQay1, a second preset data receiving rate difference ΔQay2, a first data receiving rate adjustment coefficient γ1 and a first data receiving rate adjustment coefficient γ2, wherein ΔQay1<ΔQay2, 0<γ1<γ2<1,
若△Qay<△Qay1,所述中控单元判定使用γ1调节Qa;If △Qay<△Qay1, the central control unit determines to use γ1 to adjust Qa;
若△Qay1≤△Qay<△Qay2,所述中控单元判定使用γ2调节Qa;If △Qay1≤△Qay<△Qay2, the central control unit determines to use γ2 to adjust Qa;
若△Qay≥△Qay2,所述中控单元判定数据中转服务器发生故障并发出警报;If △Qay≥△Qay2, the central control unit determines that the data transfer server fails and issues an alarm;
当所述中控单元使用γi对Qa进行调节时,设定i=1,2,调节后的数据上传速率记为Qa’,设定Qa’=Qa×βi;所述中控单元控制所述数据中转服务器使用Qa’的速率接并与所述预设数据上传速率重新比对,若Qa’>1.1×Qa0,所述中控单元判定数据中转服务器发生故障并发出警报。When the central control unit uses γi to adjust Qa, i is set to 1, 2, the adjusted data upload rate is recorded as Qa’, and Qa’=Qa×βi is set; the central control unit controls the data transfer server to use the rate of Qa’ and re-compares it with the preset data upload rate. If Qa’>1.1×Qa0, the central control unit determines that the data transfer server has a fault and issues an alarm.
进一步地,所述中控单元判定数据接收速率合格时,中控单元针对单个周期内的数据综合传输速率Q进行计算并根据Q判定所述数据中转服务器针对单个周期内的综合传输速率是否合格,设定Q=ra×Qa+rb×Qb,中控单元内设有预设综合传输速率标准Q0,其中ra为接收速率权重系数,rb为上传速率权重系数,ra<1,rb<1且ra+rb=1,Furthermore, when the central control unit determines that the data receiving rate is qualified, the central control unit calculates the comprehensive data transmission rate Q within a single cycle and determines whether the comprehensive transmission rate of the data transfer server within a single cycle is qualified according to Q, and sets Q=ra×Qa+rb×Qb. The central control unit is provided with a preset comprehensive transmission rate standard Q0, where ra is the receiving rate weight coefficient, rb is the upload rate weight coefficient, ra<1, rb<1 and ra+rb=1,
若Q≥Q0,所述中控单元判定数据综合传输速率符合预设值,中控单元计算数据的完整度S;If Q≥Q0, the central control unit determines that the comprehensive data transmission rate meets the preset value, and the central control unit calculates the integrity S of the data;
若Q<Q0,所述中控单元判定根据Qa和Qb之间的传输速率比对将Qa或Qb增加至对应值;在增加过程中所述中控单元实时检测Qa与Qb之间的关系,若在增加Qa时导致Qb<Qa,中控单元判定增加Qb以保证Qa与Qb保持平衡;若在增加Qb时导致Qa<Qb,中控单元判定增加Qa以保证Qa与Qb保持平衡。If Q<Q0, the central control unit determines to increase Qa or Qb to the corresponding value based on the transmission rate comparison between Qa and Qb; during the increase process, the central control unit detects the relationship between Qa and Qb in real time. If increasing Qa causes Qb<Qa, the central control unit determines to increase Qb to ensure that Qa and Qb are balanced; if increasing Qb causes Qa<Qb, the central control unit determines to increase Qa to ensure that Qa and Qb are balanced.
进一步地,所述中控单元判定数据综合传输速率速率符合预设值并检测数据的完整度S时并根据S判定所述数据中转服务器接收所述数据终端上传的数据完整度是否合格,中控单元内设有预设数据完整度S0,Further, when the central control unit determines that the data comprehensive transmission rate meets the preset value and detects the integrity S of the data, it determines whether the integrity of the data uploaded by the data terminal received by the data transfer server is qualified according to S. The central control unit is provided with a preset data integrity S0.
若S≥S0,所述中控单元判定数据完整度符合预设值;If S≥S0, the central control unit determines that the data integrity meets the preset value;
若S<S0,所述中控单元计算S与S0的差值△S并根据△S判定是否调节Q,设定△S=S0-S。If S<S0, the central control unit calculates the difference △S between S and S0 and determines whether to adjust Q according to △S, setting △S=S0-S.
进一步地,所述中控单元判定S<S0时,中控单元计算计算S0和S的差值并根据△S判定是否需将Q调节至对应值,设定△S=S0-S,中控单元内设有第一预设数据完整度差值△Sa、第二预设数据完整度差值△Sb、第一综合传输速率调节系数e1和第二综合传输速率调节系数e2,其中△Sa<△Sb,1<e1<e2,Further, when the central control unit determines that S<S0, the central control unit calculates the difference between S0 and S and determines whether Q needs to be adjusted to the corresponding value according to △S, and sets △S=S0-S. The central control unit is provided with a first preset data integrity difference △Sa, a second preset data integrity difference △Sb, a first comprehensive transmission rate adjustment coefficient e1 and a second comprehensive transmission rate adjustment coefficient e2, wherein △Sa<△Sb, 1<e1<e2,
若△S≤△Sa,所述中控单元二次判定数据完整度差值在可允许范围内;If △S≤△Sa, the central control unit determines that the data integrity difference is within the allowable range;
若△Sa<△S≤△Sb,所述中控单元判定使用e1调节Q;If △Sa<△S≤△Sb, the central control unit determines to use e1 to adjust Q;
若△S>△Sb,所述中控单元判定使用e2调节Q;If △S>△Sb, the central control unit determines to use e2 to adjust Q;
当所述中控单元使用ei对Q行调节时,设定i=1,2,调节后的数据综合传输速率记为Q’,设定Q’=Q×ei;调节完成后,所述中控单元再次检测数据完整度记为S’,若S’<S0,中控单元判定所述数据终端采集数据不稳定并发出警报。When the central control unit uses ei to adjust Q, i=1, 2 is set, and the adjusted comprehensive data transmission rate is recorded as Q', and Q'=Q×ei is set; after the adjustment is completed, the central control unit detects the data integrity again and records it as S'. If S'<S0, the central control unit determines that the data collected by the data terminal is unstable and issues an alarm.
与现有技术相比,本发明的有益效果在于,所述基于工业互联网的大数据存储速率进行优化时,其中包括与中控单元相连的云平台和数据中转服务器以及和数据中转服务器相连的数据终端;通过中控单元对数据终端采集到的数据通过数据中转服务器的接收速率和上传速率进行检测,针对检测结果与中控单元内预设的对应数据值进行比对,根据比对结果选择对应的处理方式,能够有效的避免数据中转服务器中的数据接收速率与数据上传速率之间的不匹配导致数据的大面积丢失的情况发生。本发明通过中控单元针对单个周期内的数据中转服务器实际负载高于预设标准时,通过对数据的重要程度对数据建立优先级标签,中控单元根据数据的优先级由低到高的顺序对数据进行推迟,能够有效的解决数据中转服务器负载过高导致数据丢失的问题,同时,中控单元检测上传数据的完整度与预设完成度进行比对并对不符合预设数据完整度的数据有针对性的调节,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Compared with the prior art, the beneficial effect of the present invention is that when the storage rate of big data based on the industrial Internet is optimized, it includes a cloud platform and a data transfer server connected to the central control unit and a data terminal connected to the data transfer server; the central control unit detects the receiving rate and upload rate of the data transferred by the data terminal through the data transfer server, compares the detection result with the corresponding data value preset in the central control unit, and selects the corresponding processing method according to the comparison result, which can effectively avoid the mismatch between the data receiving rate and the data upload rate in the data transfer server, resulting in a large-scale data loss. When the actual load of the data transfer server in a single cycle is higher than the preset standard, the central control unit establishes a priority label for the data according to the importance of the data, and the central control unit postpones the data according to the order of the priority of the data from low to high, which can effectively solve the problem of data loss caused by excessive load on the data transfer server. At the same time, the central control unit detects the completeness of the uploaded data and compares it with the preset completion degree and adjusts the data that does not meet the preset data completeness in a targeted manner, effectively optimizing the data storage rate and further improving the integrity of the data during the storage process.
进一步地,所述中控单元检测所述数据中转服务器实际接收速率与预设接收速率进行比对,根据比对结果选择对应的处理方式,能够对实际接收速率所在范围有针对性的进行处理,能够有效的避免数据中转服务器中的数据接收速率与数据上传速率之间的不匹配导致数据的大面积丢失的情况发生,进一步的提高了数据在存储过程中的完整性。Furthermore, the central control unit detects the actual receiving rate of the data transfer server and compares it with the preset receiving rate, and selects the corresponding processing method according to the comparison result, so as to be able to carry out targeted processing within the range of the actual receiving rate, and can effectively avoid the mismatch between the data receiving rate and the data upload rate in the data transfer server, thereby causing large-scale data loss, and further improves the integrity of the data during the storage process.
进一步地,所述中控单元判定所述数据接收速率不符合标准时,计算数据接收速率和预设数据接收速率的差值根据差值所在预设差值范围选择对应的处理方式,能够准确的判断出数据中转服务器实际负责高于预设标准并有针对性的进行处理,能够有效的避免数据中转服务器中的数据接收速率与数据上传速率之间的不匹配导致数据的大面积丢失的情况发生,进一步的提高了数据在存储过程中的完整性。Furthermore, when the central control unit determines that the data receiving rate does not meet the standard, the difference between the calculated data receiving rate and the preset data receiving rate is selected according to the preset difference range in which the difference lies. It can accurately determine that the data transfer server is actually responsible for exceeding the preset standard and perform targeted processing. It can effectively avoid the mismatch between the data receiving rate and the data upload rate in the data transfer server, which may lead to large-scale data loss, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元控制所述数据中转服务器中上传的数据根据数据的重要程度建立对应的优先级标签,中控单元根据数据的由低到高的顺序对数据进行推迟,能够有效的解决数据中转服务器负载过高导致数据丢失的问题,进一步的提高了数据在存储过程中的完整性。Furthermore, the central control unit controls the data uploaded in the data transfer server to establish corresponding priority tags according to the importance of the data. The central control unit postpones the data in order from low to high, which can effectively solve the problem of data loss caused by excessive load on the data transfer server and further improve the integrity of the data during the storage process.
进一步地,所述中控单元二次判定所述数据接收速率在可允许范围内时,中控单元检测所述数据中转服务器的数据上传速率并与数据接收速率进行比对,根据比对结果选择对应的调节系数对终端上传速率进行调节,能够有效的保证了数据中转服务器中的数据接收速率与终端上传速率之间的平衡关系,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Furthermore, when the central control unit determines for the second time that the data receiving rate is within the allowable range, the central control unit detects the data upload rate of the data transfer server and compares it with the data receiving rate, and selects a corresponding adjustment coefficient according to the comparison result to adjust the terminal upload rate, which can effectively ensure the balance between the data receiving rate in the data transfer server and the terminal upload rate, effectively optimize the data storage rate, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元对所述数据上传速率进行检测时初步判定网络环境出现问题时,中控单元检测所述数据中转服务器的上传速率并根据数据接收速率与数据上传速率的差值所在范围选择对应的调节系数对数据上传速率进行调节,能够有效的保证了数据中转服务器中的数据接收速率与数据上传速率之间的平衡关系,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Furthermore, when the central control unit detects the data upload rate and preliminarily determines that there is a problem with the network environment, the central control unit detects the upload rate of the data transfer server and selects a corresponding adjustment coefficient to adjust the data upload rate according to the range of the difference between the data receiving rate and the data upload rate. This can effectively ensure the balance between the data receiving rate and the data upload rate in the data transfer server, effectively optimize the data storage rate, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元判定数据接收速率过快时,中控单元计算数据接收速率与预设接收速率的差值并根据差值所在范围选择对应的调节系数对数据接收速率进行调节,能够有针对性的数据接收速率进行控制的同时提高了数据存储过程中的完整性。Furthermore, when the central control unit determines that the data receiving rate is too fast, the central control unit calculates the difference between the data receiving rate and the preset receiving rate and selects a corresponding adjustment coefficient to adjust the data receiving rate according to the range of the difference. This can control the data receiving rate in a targeted manner while improving the integrity of the data storage process.
进一步地,所述中控单元所述中控单元判定数据接收速率合格时,中控单元针对单个周期内的数据综合传输速率进行计算并与预设综合传输速率进行比对,根据比对结果选择对应的处理方式,能够检测数据在接收与上传过程中的综合速率值,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Furthermore, when the central control unit determines that the data receiving rate is qualified, the central control unit calculates the comprehensive data transmission rate within a single cycle and compares it with the preset comprehensive transmission rate, and selects the corresponding processing method according to the comparison result. It can detect the comprehensive rate value of the data during the reception and upload process, effectively optimize the data storage rate, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元判定数据综合传输速率速率符合预设值并检测数据的完整度并与预设完整度进行比对,根据比对结果选择对应的处理方式,能够有针对性的检测单个周期内数据存储过程中是否有丢失,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Furthermore, the central control unit determines whether the comprehensive data transmission rate meets the preset value, detects the integrity of the data and compares it with the preset integrity, selects the corresponding processing method according to the comparison result, and can specifically detect whether there is any loss during the data storage process within a single cycle, effectively optimizes the data storage rate, and further improves the integrity of the data during the storage process.
进一步地,所述中控单元判定实际数据完整度小于预设数据完整度时,计算预设数据完整度与实际数据完整度的差值,根据差值所在范围选择对应的调节系数对数据综合传输速率进行调节,能够有效的解决数据在存储过程中完整度不符合标准时进行调节,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Furthermore, when the central control unit determines that the actual data integrity is less than the preset data integrity, it calculates the difference between the preset data integrity and the actual data integrity, and selects the corresponding adjustment coefficient according to the range of the difference to adjust the comprehensive data transmission rate. This can effectively solve the problem of data integrity not meeting the standards during storage, effectively optimize the data storage rate, and further improve the integrity of the data during storage.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1为本发明所述的一种基于工业互联网的大数据存储速率优化方法的结构示意图;FIG1 is a schematic diagram of the structure of a method for optimizing the data storage rate of a large data storage based on the industrial Internet according to the present invention;
图2为本发明所述的一种基于工业互联网的大数据存储速率优化方法的流程图。FIG2 is a flow chart of a method for optimizing the storage rate of big data based on the industrial Internet described in the present invention.
具体实施方式DETAILED DESCRIPTION
为了使本发明的目的和优点更加清楚明白,下面结合实施例对本发明作进一步描述;应当理解,此处所描述的具体实施例仅仅用于解释本发明,并不用于限定本发明。In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
下面参照附图来描述本发明的优选实施方式。本领域技术人员应当理解的是,这些实施方式仅仅用于解释本发明的技术原理,并非在限制本发明的保护范围。The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.
需要说明的是,在本发明的描述中,术语“上”、“下”、“左”、“右”、“内”、“外”等指示的方向或位置关系的术语是基于附图所示的方向或位置关系,这仅仅是为了便于描述,而不是指示或暗示所述装置或元件必须具有特定的方位、以特定的方位构造和操作,因此不能理解为对本发明的限制。It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.
此外,还需要说明的是,在本发明的描述中,除非另有明确的规定和限定,术语“安装”、“相连”、“连接”应做广义理解,例如,可以是固定连接,也可以是可拆卸连接,或一体地连接;可以是机械连接,也可以是电连接;可以是直接相连,也可以通过中间媒介间接相连,可以是两个元件内部的连通。对于本领域技术人员而言,可根据具体情况理解上述术语在本发明中的具体含义。In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
请参阅图2所示,其为本发明所述一种基于工业互联网的大数据存储速率优化方法的流程图。Please refer to Figure 2, which is a flow chart of a method for optimizing the storage rate of big data based on the industrial Internet described in the present invention.
本发明所述基于工业互联网的大数据存储速率优化方法,包括:The method for optimizing the storage rate of big data based on the industrial Internet of the present invention comprises:
步骤s1,数据终端将其内部的数据上传至数据中转服务器;Step s1, the data terminal uploads its internal data to the data transfer server;
步骤s2,所述数据中转服务器接收所述数据终端上传的数据,接收过程中,中控单元根据所述数据的重要程度依次对各数据建立对应的优先级标签,中控单元检测数据中转服务器在接收所述终端数据过程中的实际接收速率、判定该接收速率是否合格并在判定接收速率过快或过慢时根据结果判定数据中转服务器接收速率过快或过慢的实际原因;Step s2, the data transfer server receives the data uploaded by the data terminal. During the receiving process, the central control unit establishes corresponding priority tags for each data in turn according to the importance of the data. The central control unit detects the actual receiving rate of the data transfer server in the process of receiving the terminal data, determines whether the receiving rate is qualified, and determines the actual reason why the receiving rate of the data transfer server is too fast or too slow according to the result when the receiving rate is determined to be too fast or too slow;
步骤s3,所述中控单元在判定所述数据中转服务器接收速率过过快或过慢时调节所述数据终端的上传速率或调节所述数据中转服务器的上传速率或根据数据的优先级标签对对应的数据进行推迟上传以使数据中转服务器的接收速率符合标准,中控单元计算单个周期内的综合传输速率并检测该速率是否符合标准;Step s3, when the central control unit determines that the receiving rate of the data transfer server is too fast or too slow, it adjusts the upload rate of the data terminal or the upload rate of the data transfer server or postpones the upload of the corresponding data according to the priority tag of the data so that the receiving rate of the data transfer server meets the standard. The central control unit calculates the comprehensive transmission rate in a single cycle and detects whether the rate meets the standard;
步骤s4,所述中控单元在判定所述综合传输速率你符合标准时调节数据中转服务器的接收速率和上传速率以使综合传输速率符合预设值,中控单元计算数据的完整度并检测数据的完整度是否符合标准;Step s4, when the central control unit determines that the comprehensive transmission rate meets the standard, the central control unit adjusts the receiving rate and the uploading rate of the data transfer server so that the comprehensive transmission rate meets the preset value, and the central control unit calculates the integrity of the data and detects whether the integrity of the data meets the standard;
步骤s5,所述中控单元在判定所述数据的完整度不符合标准时,所述中控单元调节综合传输速率以使所述数据的完整度符合预设标准;Step s5, when the central control unit determines that the integrity of the data does not meet the standard, the central control unit adjusts the comprehensive transmission rate so that the integrity of the data meets the preset standard;
步骤s6,当所述中控单元完成对对应的参数的调节时,中控单元重新检测数据中转服务器在接收所述终端数据过程中的实际接收速率,若该接收速率合格,中控单元判定针对该数据的存储速率的优化完成。Step s6, when the central control unit completes the adjustment of the corresponding parameters, the central control unit re-detects the actual receiving rate of the data transfer server in the process of receiving the terminal data. If the receiving rate is qualified, the central control unit determines that the optimization of the storage rate of the data is completed.
具体而言,本发明的有益效果在于,所述基于工业互联网的大数据存储速率进行优化时,其中包括与中控单元相连的云平台和数据中转服务器以及和数据中转服务器相连的数据终端;通过中控单元对数据终端采集到的数据通过数据中转服务器的接收速率和上传速率进行检测,针对检测结果与中控单元内预设的对应数据值进行比对,根据比对结果选择对应的处理方式,能够有效的避免数据中转服务器中的数据接收速率与数据上传速率之间的不匹配导致数据的大面积丢失的情况发生。本发明通过中控单元针对单个周期内的数据中转服务器实际负载高于预设标准时,通过对数据的重要程度对数据建立优先级标签,中控单元根据数据的优先级由低到高的顺序对数据进行推迟,能够有效的解决数据中转服务器负载过高导致数据丢失的问题,同时,中控单元检测上传数据的完整度与预设完成度进行比对并对不符合预设数据完整度的数据有针对性的调节,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Specifically, the beneficial effect of the present invention is that when the storage rate of big data based on the industrial Internet is optimized, it includes a cloud platform and a data transfer server connected to the central control unit and a data terminal connected to the data transfer server; the central control unit detects the receiving rate and upload rate of the data collected by the data terminal through the data transfer server, compares the detection result with the corresponding data value preset in the central control unit, and selects the corresponding processing method according to the comparison result, which can effectively avoid the mismatch between the data receiving rate and the data upload rate in the data transfer server, resulting in a large-scale data loss. The present invention establishes a priority label for the data according to the importance of the data when the actual load of the data transfer server in a single cycle is higher than the preset standard, and the central control unit postpones the data according to the order of the data priority from low to high, which can effectively solve the problem of data loss caused by excessive load on the data transfer server. At the same time, the central control unit detects the completeness of the uploaded data and compares it with the preset completion degree and makes targeted adjustments to the data that does not meet the preset data completeness, effectively optimizing the data storage rate and further improving the integrity of the data during the storage process.
进一步地,所述中控单元在所述数据中转服务器接收单个所述数据终端上传的数据时将数据中转服务器针对该数据终端的数据接收速率记为Qa并根据Qa判定数据中转服务器针对该数据终端的数据接收速率是否合格,所述中控单元中设有预设接收速率Qa0,Furthermore, when the data transfer server receives data uploaded by a single data terminal, the central control unit records the data receiving rate of the data transfer server for the data terminal as Qa and determines whether the data receiving rate of the data transfer server for the data terminal is qualified according to Qa. The central control unit is provided with a preset receiving rate Qa0.
若0.9×Qa0≤Qa≤1.1×Qa0,所述中控单元初步判定所述数据中转服务器针对所述数据终端的数据接收速率合格,中控单元检测数据中转服务器的上传速率并计算综合数据传输速率记为Q并检测数据的完整度记为S;If 0.9×Qa0≤Qa≤1.1×Qa0, the central control unit preliminarily determines that the data receiving rate of the data transfer server for the data terminal is qualified, and the central control unit detects the upload rate of the data transfer server and calculates the comprehensive data transmission rate recorded as Q and detects the integrity of the data recorded as S;
若Qa<0.9×Qa0,所述中控单元初步判定所述数据中转服务器针对所述数据终端的数据接收速率不符合标准,中控单元计算Qa与Qa0的差值△Qax并根据△Qax确定该接收速率低于预设标准的实际原因,设定△Qax=Qa0-Qa;If Qa<0.9×Qa0, the central control unit preliminarily determines that the data receiving rate of the data transfer server for the data terminal does not meet the standard, and the central control unit calculates the difference △Qax between Qa and Qa0 and determines the actual reason why the receiving rate is lower than the preset standard according to △Qax, and sets △Qax=Qa0-Qa;
若Qa>1.1×Qa0,所述中控单元判定所述数据中转服务器针对所述数据终端的数据接收速率过快,中控单元计算数据接收速率与预设数据接收速率的差值△Qay并根据△Qay判定是否对所述数据中转服务器的实际接收速率Qa进行限制,设定△Qay=Qa-Qa0。If Qa>1.1×Qa0, the central control unit determines that the data receiving rate of the data relay server for the data terminal is too fast. The central control unit calculates the difference △Qay between the data receiving rate and the preset data receiving rate and determines whether to limit the actual receiving rate Qa of the data relay server based on △Qay, and sets △Qay=Qa-Qa0.
具体而言,所述中控单元检测所述数据中转服务器实际接收速率与预设接收速率进行比对,根据比对结果选择对应的处理方式,能够对实际接收速率所在范围有针对性的进行处理,能够有效的避免数据中转服务器中的数据接收速率与数据上传速率之间的不匹配导致数据的大面积丢失的情况发生,进一步的提高了数据在存储过程中的完整性。Specifically, the central control unit detects the actual receiving rate of the data transfer server and compares it with the preset receiving rate, and selects the corresponding processing method according to the comparison result. It can perform targeted processing on the range of the actual receiving rate, and can effectively avoid the mismatch between the data receiving rate and the data upload rate in the data transfer server, which may lead to large-scale data loss, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元初步判定所述数据中转服务器针对所述数据终端的数据接收速率不符合标准时,中控单元根据△Qax对该接收速率低于预设标准的实际原因进行判定,中控单元内设有第一预设数据接收速率差值△Qax1、第二预设数据接收速率差值△Qax2、第二预设数据接收速率差值△Qax3、第一预设推迟任务比例b1和第二预设推迟任务比例b2,其中△Qax1<△Qax2<△Qax3,b1<b2<1,Further, when the central control unit preliminarily determines that the data receiving rate of the data transfer server for the data terminal does not meet the standard, the central control unit determines the actual reason why the receiving rate is lower than the preset standard according to △Qax, and the central control unit is provided with a first preset data receiving rate difference △Qax1, a second preset data receiving rate difference △Qax2, a second preset data receiving rate difference △Qax3, a first preset postponed task ratio b1 and a second preset postponed task ratio b2, wherein △Qax1<△Qax2<△Qax3, b1<b2<1,
若△Qax≤△Qax1,所述中控单元二次判定数据接收速率在可允许范围内,中控单元检测所述数据终端的上传速率qb并根据qb判定所述数据中转服务器接收数据的接收速率与该终端上传数据的上传速率是否平衡;If △Qax≤△Qax1, the central control unit determines that the data receiving rate is within the allowable range, and the central control unit detects the upload rate qb of the data terminal and determines whether the receiving rate of the data transfer server receiving data is balanced with the upload rate of the terminal uploading data according to qb;
若△Qax1<△Qax≤△Qax2,中控单元判定所述数据中转服务器实际负载高于预设标准、根据数据优先级控制数据中转服务器推迟接收对应所述终端输出的数据以降低数据中转服务器的运行负载并将推迟数据比例设置为b1,数据接收总数量记为M,设定推迟数据接收数量为M’,设定M’=M×b1;If △Qax1<△Qax≤△Qax2, the central control unit determines that the actual load of the data transfer server is higher than the preset standard, controls the data transfer server to postpone receiving the data outputted by the corresponding terminal according to the data priority to reduce the operation load of the data transfer server, and sets the postponed data ratio to b1, the total number of data received is recorded as M, the number of postponed data received is set to M', and M'=M×b1;
若△Qax2<△Qax≤△Qax3,中控单元判定所述数据中转服务器实际负载高于预设标准,根据数据优先级推迟数据中转服务器的对应数据以降低数据中转服务器的运行负载并将推迟数据比例设置为b2,设定M’=M×b1;If △Qax2<△Qax≤△Qax3, the central control unit determines that the actual load of the data transfer server is higher than the preset standard, postpones the corresponding data of the data transfer server according to the data priority to reduce the operation load of the data transfer server and sets the postponed data ratio to b2, and sets M'=M×b1;
若△Qax>△Qax3,所述中控单元判定网路环境出现问题、检测所述数据中转服务器上传速率Qb并根据Qa与Qb的差值判定所述数据中转服务器的接收速率与数据中转服务器的上传速率是否平衡。If ΔQax>ΔQax3, the central control unit determines that there is a problem with the network environment, detects the upload rate Qb of the data transfer server, and determines whether the receiving rate of the data transfer server and the upload rate of the data transfer server are balanced according to the difference between Qa and Qb.
具体而言,所述中控单元判定所述数据接收速率不符合标准时,计算数据接收速率和预设数据接收速率的差值根据差值所在预设差值范围选择对应的处理方式,能够准确的判断出数据中转服务器实际负责高于预设标准并有针对性的进行处理,能够有效的避免数据中转服务器中的数据接收速率与数据上传速率之间的不匹配导致数据的大面积丢失的情况发生,进一步的提高了数据在存储过程中的完整性。Specifically, when the central control unit determines that the data receiving rate does not meet the standard, it calculates the difference between the data receiving rate and the preset data receiving rate and selects the corresponding processing method according to the preset difference range in which the difference lies. It can accurately determine that the data transfer server is actually responsible for exceeding the preset standard and perform targeted processing. It can effectively avoid the mismatch between the data receiving rate and the data upload rate in the data transfer server, which may lead to large-scale data loss, and further improve the integrity of the data during the storage process.
进一步地,所述数据中转服务器接收各所述终端上传的数据时,中控单元根据数据中转服务器中上传的数据的重要程度依次对各数据建立对应的优先级标签;所述数据的重要程度由高到低的顺序包括一级、二级、三级和四级;当所述中控单元判定需根据数据优先级推迟所述数据中转服务器推迟接收对应所述终端输出的数据以降低数据中转服务器的运行负载,中控单元按照四级至一级的数据标签的顺序推迟接收对应级别的数据。Furthermore, when the data transfer server receives the data uploaded by each of the terminals, the central control unit establishes corresponding priority tags for each data in turn according to the importance of the data uploaded in the data transfer server; the importance of the data includes level one, level two, level three and level four in order from high to low; when the central control unit determines that it is necessary to postpone the data transfer server to postpone receiving the data output by the corresponding terminal according to the data priority to reduce the operating load of the data transfer server, the central control unit postpones receiving the data of the corresponding level in the order of data tags from level four to level one.
具体而言,所述中控单元控制所述数据中转服务器中上传的数据根据数据的重要程度建立对应的优先级标签,中控单元根据数据的由低到高的顺序对数据进行推迟,能够有效的解决数据中转服务器负载过高导致数据丢失的问题,进一步的提高了数、据在存储过程中的完整性。Specifically, the central control unit controls the data uploaded in the data transfer server to establish corresponding priority tags according to the importance of the data. The central control unit postpones the data in order from low to high, which can effectively solve the problem of data loss caused by excessive load on the data transfer server, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元二次判定所述数据接收速率在可允许范围内时,中控单元检测所述数据终端上传速率qb、计算qa和qb的差值△q并根据△q判定是否需将qb调节至对应值,设定△q=Qa-qb,中控单元内设有所述数据中转服务器的第一上传数据速率调节系数α1和第二上传数据速率调节系数α2,其中,0<α2<1<α1,Further, when the central control unit determines for the second time that the data receiving rate is within the allowable range, the central control unit detects the data terminal upload rate qb, calculates the difference △q between qa and qb, and determines whether qb needs to be adjusted to the corresponding value according to △q, and sets △q=Qa-qb. The central control unit is provided with a first upload data rate adjustment coefficient α1 and a second upload data rate adjustment coefficient α2 of the data transfer server, wherein 0<α2<1<α1,
若△q>0,所述中控单元判定使用α1调节qb以将qb增加至对应值;If Δq>0, the central control unit determines to use α1 to adjust qb to increase qb to the corresponding value;
若△q<0,所述中控单元判定使用α2调节Qb以将qb减小至对应值;If Δq<0, the central control unit determines to use α2 to adjust Qb to reduce qb to a corresponding value;
当所述中控单元使用αi对qb进行调节时,设定i=1,2,调节后的数据上传速率记为qb’,设定qb’=qb×αi;所述中控单元控制所述数据中转服务器使用qb’的速率上传数据并计算对应综合传输速率Q。When the central control unit uses αi to adjust qb, i is set to 1, 2, the adjusted data upload rate is recorded as qb’, and qb’=qb×αi is set; the central control unit controls the data transfer server to upload data at the rate of qb’ and calculates the corresponding comprehensive transmission rate Q.
具体而言,所述中控单元二次判定所述数据接收速率在可允许范围内时,中控单元检测所述数据中转服务器的数据上传速率并与数据接收速率进行比对,根据比对结果选择对应的调节系数对终端上传速率进行调节,能够有效的保证了数据中转服务器中的数据接收速率与终端上传速率之间的平衡关系,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Specifically, when the central control unit determines for the second time that the data receiving rate is within the allowable range, the central control unit detects the data upload rate of the data transfer server and compares it with the data receiving rate, and selects the corresponding adjustment coefficient according to the comparison result to adjust the terminal upload rate. This can effectively ensure the balance between the data receiving rate in the data transfer server and the terminal upload rate, effectively optimize the data storage rate, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元判定网路环境出现问题时,中控单元计算计算Qa和Qb的差值△Q’并根据△Q’判定是否需将Qb调节至对应值,设定△Q’=Qa-Qb,中控单元内设有所述数据中转服务器第一上传数据速率预设差值△Q1、第二上传数据速率预设差值△Q2、第一上传数据速率调节系数β1和第二上传数据速率调节系数β2,其中△Q1<△Q2,β1<β2<1;Further, when the central control unit determines that there is a problem with the network environment, the central control unit calculates the difference △Q' between Qa and Qb and determines whether Qb needs to be adjusted to the corresponding value based on △Q', and sets △Q'=Qa-Qb. The central control unit is provided with a first upload data rate preset difference △Q1, a second upload data rate preset difference △Q2, a first upload data rate adjustment coefficient β1, and a second upload data rate adjustment coefficient β2 of the data transfer server, wherein △Q1<△Q2, β1<β2<1;
若△Q’<△Q1,所述中控单元判定使用β1调节Qb;If ΔQ’<ΔQ1, the central control unit determines to use β1 to adjust Qb;
若△Q1≤△Q’<△Q2,所述中控单元判定使用β2调节Qb;If △Q1≤△Q’<△Q2, the central control unit determines to use β2 to adjust Qb;
若△Q’≥△Q2,所述中控单元判定网络故障并发出故障警报;If △Q’≥△Q2, the central control unit determines that the network is faulty and issues a fault alarm;
当所述中控单元使用βi对Qb进行调节时,设定i=1,2,调节后的数据上传速率记为Qb”,设定Qb”=Qb×βi;所述中控单元控制所述数据中转服务器使用Qb”的速率上传数据并计算对应综合传输速率Q。When the central control unit uses βi to adjust Qb, i is set to 1, 2, and the adjusted data upload rate is recorded as Qb", and Qb"=Qb×βi is set; the central control unit controls the data transfer server to upload data at the rate of Qb" and calculates the corresponding comprehensive transmission rate Q.
具体而言,所述中控单元对所述数据上传速率进行检测时初步判定网络环境出现问题时,中控单元检测所述数据中转服务器的上传速率并根据数据接收速率与数据上传速率的差值所在范围选择对应的调节系数对数据上传速率进行调节,能够有效的保证了数据中转服务器中的数据接收速率与数据上传速率之间的平衡关系,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Specifically, when the central control unit detects the data upload rate and preliminarily determines that there is a problem with the network environment, the central control unit detects the upload rate of the data transfer server and selects a corresponding adjustment coefficient to adjust the data upload rate according to the range of the difference between the data receiving rate and the data upload rate. This can effectively ensure the balance between the data receiving rate and the data upload rate in the data transfer server, effectively optimize the data storage rate, and further improve the integrity of the data during the storage process.
进一步地,所述中控单元判定数据接收速率过快时,中控单元根据△Qay判定是否对中转点的实际接收速率Qa进行限制,中控单元内设有第一预设数据接收速率差值△Qay1、第二预设数据接收速率差值△Qay2、第一数据接收速率调节系数γ1和第一数据接收速率调节系数γ2,其中,△Qay1<△Qay2,0<γ1<γ2<1,Further, when the central control unit determines that the data receiving rate is too fast, the central control unit determines whether to limit the actual receiving rate Qa of the transfer point according to ΔQay, and the central control unit is provided with a first preset data receiving rate difference ΔQay1, a second preset data receiving rate difference ΔQay2, a first data receiving rate adjustment coefficient γ1 and a first data receiving rate adjustment coefficient γ2, wherein ΔQay1<ΔQay2, 0<γ1<γ2<1,
若△Qay<△Qay1,所述中控单元判定使用γ1调节Qa;If △Qay<△Qay1, the central control unit determines to use γ1 to adjust Qa;
若△Qay1≤△Qay<△Qay2,所述中控单元判定使用γ2调节Qa;If △Qay1≤△Qay<△Qay2, the central control unit determines to use γ2 to adjust Qa;
若△Qay≥△Qay2,所述中控单元判定数据中转服务器发生故障并发出警报;If △Qay≥△Qay2, the central control unit determines that the data transfer server fails and issues an alarm;
当所述中控单元使用γi对Qa进行调节时,设定i=1,2,调节后的数据上传速率记为Qa’,设定Qa’=Qa×βi;所述中控单元控制所述数据中转服务器使用Qa’的速率接并与所述预设数据上传速率重新比对,若Qa’>1.1×Qa0,所述中控单元判定数据中转服务器发生故障并发出警报。When the central control unit uses γi to adjust Qa, i is set to 1, 2, the adjusted data upload rate is recorded as Qa’, and Qa’=Qa×βi is set; the central control unit controls the data transfer server to use the rate of Qa’ and re-compares it with the preset data upload rate. If Qa’>1.1×Qa0, the central control unit determines that the data transfer server has a fault and issues an alarm.
具体而言,所述中控单元判定数据接收速率过快时,中控单元计算数据接收速率与预设接收速率的差值并根据差值所在范围选择对应的调节系数对数据接收速率进行调节,能够有针对性的数据接收速率进行控制的同时提高了数据存储过程中的完整性。Specifically, when the central control unit determines that the data receiving rate is too fast, the central control unit calculates the difference between the data receiving rate and the preset receiving rate and selects a corresponding adjustment coefficient to adjust the data receiving rate according to the range of the difference. This can control the data receiving rate in a targeted manner while improving the integrity of the data storage process.
进一步地,所述中控单元判定数据接收速率合格时,中控单元针对单个周期内的数据综合传输速率Q进行计算并根据Q判定所述数据中转服务器针对单个周期内的综合传输速率是否合格,设定Q=ra×Qa+rb×Qb,中控单元内设有预设综合传输速率标准Q0,其中ra为接收速率权重系数,rb为上传速率权重系数,ra<1,rb<1且ra+rb=1,Furthermore, when the central control unit determines that the data receiving rate is qualified, the central control unit calculates the comprehensive data transmission rate Q within a single cycle and determines whether the comprehensive transmission rate of the data transfer server within a single cycle is qualified according to Q, and sets Q=ra×Qa+rb×Qb. The central control unit is provided with a preset comprehensive transmission rate standard Q0, where ra is the receiving rate weight coefficient, rb is the upload rate weight coefficient, ra<1, rb<1 and ra+rb=1,
若Q≥Q0,所述中控单元判定数据综合传输速率符合预设值,中控单元计算数据的完整度S;If Q≥Q0, the central control unit determines that the comprehensive data transmission rate meets the preset value, and the central control unit calculates the integrity S of the data;
若Q<Q0,所述中控单元判定根据Qa和Qb之间的传输速率比对将Qa或Qb增加至对应值;在增加过程中所述中控单元实时检测Qa与Qb之间的关系,若在增加Qa时导致Qb<Qa,中控单元判定增加Qb以保证Qa与Qb保持平衡;若在增加Qb时导致Qa<Qb,中控单元判定增加Qa以保证Qa与Qb保持平衡。If Q<Q0, the central control unit determines to increase Qa or Qb to the corresponding value based on the transmission rate comparison between Qa and Qb; during the increase process, the central control unit detects the relationship between Qa and Qb in real time. If increasing Qa causes Qb<Qa, the central control unit determines to increase Qb to ensure that Qa and Qb are balanced; if increasing Qb causes Qa<Qb, the central control unit determines to increase Qa to ensure that Qa and Qb are balanced.
具体而言所述中控单元所述中控单元判定数据接收速率合格时,中控单元针对单个周期内的数据综合传输速率进行计算并与预设综合传输速率进行比对,根据比对结果选择对应的处理方式,能够检测数据在接收与上传过程中的综合速率值,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Specifically, when the central control unit determines that the data receiving rate is qualified, the central control unit calculates the comprehensive data transmission rate within a single cycle and compares it with the preset comprehensive transmission rate, and selects the corresponding processing method according to the comparison result. It can detect the comprehensive rate value of data during the reception and upload process, effectively optimize the data storage rate, and further improve the integrity of data during the storage process.
进一步地,所述中控单元判定数据综合传输速率速率符合预设值并检测数据的完整度S时并根据S判定所述数据中转服务器接收所述数据终端上传的数据完整度是否合格,中控单元内设有预设数据完整度S0,Further, when the central control unit determines that the data comprehensive transmission rate meets the preset value and detects the integrity S of the data, it determines whether the integrity of the data uploaded by the data terminal received by the data transfer server is qualified according to S. The central control unit is provided with a preset data integrity S0.
若S≥S0,所述中控单元判定数据完整度符合预设值;If S≥S0, the central control unit determines that the data integrity meets the preset value;
若S<S0,所述中控单元计算S与S0的差值△S并根据△S判定是否调节Q,设定△S=S0-S。If S<S0, the central control unit calculates the difference △S between S and S0 and determines whether to adjust Q according to △S, setting △S=S0-S.
具体而言,所述中控单元判定数据综合传输速率速率符合预设值并检测数据的完整度并与预设完整度进行比对,根据比对结果选择对应的处理方式,能够有针对性的检测单个周期内数据存储过程中是否有丢失,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Specifically, the central control unit determines whether the comprehensive data transmission rate meets the preset value, detects the integrity of the data and compares it with the preset integrity, selects the corresponding processing method according to the comparison result, and can specifically detect whether there is any loss during the data storage process within a single cycle, effectively optimizes the data storage rate, and further improves the integrity of the data during the storage process.
进一步地,所述中控单元判定S<S0时,中控单元计算计算S0和S的差值并根据△S判定是否需将Q调节至对应值,设定△S=S0-S,中控单元内设有第一预设数据完整度差值△Sa、第二预设数据完整度差值△Sb、第一综合传输速率调节系数e1和第二综合传输速率调节系数e2,其中△Sa<△Sb,1<e1<e2,Further, when the central control unit determines that S<S0, the central control unit calculates the difference between S0 and S and determines whether Q needs to be adjusted to the corresponding value according to △S, and sets △S=S0-S. The central control unit is provided with a first preset data integrity difference △Sa, a second preset data integrity difference △Sb, a first comprehensive transmission rate adjustment coefficient e1 and a second comprehensive transmission rate adjustment coefficient e2, wherein △Sa<△Sb, 1<e1<e2,
若△S≤△Sa,所述中控单元二次判定数据完整度差值在可允许范围内;If △S≤△Sa, the central control unit determines that the data integrity difference is within the allowable range;
若△Sa<△S≤△Sb,所述中控单元判定使用e1调节Q;If △Sa<△S≤△Sb, the central control unit determines to use e1 to adjust Q;
若△S>△Sb,所述中控单元判定使用e2调节Q;If △S>△Sb, the central control unit determines to use e2 to adjust Q;
当所述中控单元使用ei对Q行调节时,设定i=1,2,调节后的数据综合传输速率记为Q’,设定Q’=Q×ei;调节完成后,所述中控单元再次检测数据完整度记为S’,若S’<S0,中控单元判定所述数据终端采集数据不稳定并发出警报。When the central control unit uses ei to adjust Q, i=1, 2 is set, and the adjusted comprehensive data transmission rate is recorded as Q', and Q'=Q×ei is set; after the adjustment is completed, the central control unit detects the data integrity again and records it as S'. If S'<S0, the central control unit determines that the data collected by the data terminal is unstable and issues an alarm.
具体而言,所述中控单元判定实际数据完整度小于预设数据完整度时,计算预设数据完整度与实际数据完整度的差值,根据差值所在范围选择对应的调节系数对数据综合传输速率进行调节,能够有效的解决数据在存储过程中完整度不符合标准时进行调节,有效的优化了数据存储的速率,进一步的提高了数据在存储过程中的完整性。Specifically, when the central control unit determines that the actual data integrity is less than the preset data integrity, it calculates the difference between the preset data integrity and the actual data integrity, and selects the corresponding adjustment coefficient according to the range of the difference to adjust the comprehensive data transmission rate. This can effectively solve the problem of data integrity not meeting the standards during storage, effectively optimize the data storage rate, and further improve the integrity of the data during storage.
至此,已经结合附图所示的优选实施方式描述了本发明的技术方案,但是,本领域技术人员容易理解的是,本发明的保护范围显然不局限于这些具体实施方式。在不偏离本发明的原理的前提下,本领域技术人员可以对相关技术特征做出等同的更改或替换,这些更改或替换之后的技术方案都将落入本发明的保护范围之内。So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.
以上所述仅为本发明的优选实施例,并不用于限制本发明;对于本领域的技术人员来说,本发明可以有各种更改和变化。凡在本发明的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本发明的保护范围之内。The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims (9)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210606901.XA CN114979122B (en) | 2022-05-31 | 2022-05-31 | Big data storage rate optimization method based on industrial Internet |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210606901.XA CN114979122B (en) | 2022-05-31 | 2022-05-31 | Big data storage rate optimization method based on industrial Internet |
Publications (2)
Publication Number | Publication Date |
---|---|
CN114979122A CN114979122A (en) | 2022-08-30 |
CN114979122B true CN114979122B (en) | 2024-09-20 |
Family
ID=82956937
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210606901.XA Active CN114979122B (en) | 2022-05-31 | 2022-05-31 | Big data storage rate optimization method based on industrial Internet |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114979122B (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116186142B (en) * | 2023-03-08 | 2023-12-22 | 深圳欧税通技术有限公司 | Cross-border tax data service management system based on Internet of things |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105187490A (en) * | 2015-08-05 | 2015-12-23 | 广东长天思源环保科技股份有限公司 | Method for transferring data of IOT (Internet of Things) |
CN111163471A (en) * | 2019-12-26 | 2020-05-15 | 北京微智信业科技有限公司 | Method, device, equipment and storage medium for protecting integrity of service data |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
RU2013122421A (en) * | 2013-05-16 | 2014-11-27 | Открытое Акционерное Общество Междугородной И Международной Электрической Связи "Ростелеком" (Оао "Ростелеком") | VIDEO DATA TRANSFER SYSTEM FROM MANY SIMULTANEOUS OCCURRING TERRITORALLY DISTRIBUTED EVENTS |
CN105933326A (en) * | 2016-06-08 | 2016-09-07 | 乐视控股(北京)有限公司 | Remote terminal data reporting method and device |
CN109272752B (en) * | 2018-10-11 | 2021-03-02 | 南威软件股份有限公司 | Transmission method and transmission system of intersection vehicle picture acquisition system |
CN111505997A (en) * | 2020-05-14 | 2020-08-07 | 周丽君 | Intelligent Internet of things data acquisition terminal equipment |
-
2022
- 2022-05-31 CN CN202210606901.XA patent/CN114979122B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105187490A (en) * | 2015-08-05 | 2015-12-23 | 广东长天思源环保科技股份有限公司 | Method for transferring data of IOT (Internet of Things) |
CN111163471A (en) * | 2019-12-26 | 2020-05-15 | 北京微智信业科技有限公司 | Method, device, equipment and storage medium for protecting integrity of service data |
Also Published As
Publication number | Publication date |
---|---|
CN114979122A (en) | 2022-08-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110365529B (en) | A kind of edge computing intelligent gateway business processing method and edge computing intelligent gateway | |
CN116365536B (en) | A low-voltage reactive power precision compensation method based on power distribution | |
CN114979122B (en) | Big data storage rate optimization method based on industrial Internet | |
CN117914786A (en) | Smart IoT data processing method and management platform for cloud-edge collaboration | |
CN115688014A (en) | Method, device and equipment for determining running state of relay protection equipment | |
CN111986046A (en) | Energy internet collaborative optimization processing system and processing method based on power network | |
CN118399586A (en) | Intelligent detection control system and method for power distribution network ring main unit | |
CN112712606A (en) | Automatic inspection management method and system based on operation and maintenance service | |
CN115378841B (en) | Method and device for detecting state of equipment accessing cloud platform, storage medium and terminal | |
WO2024021559A1 (en) | Power data synchronous uploading method and system, storage and computing device | |
CN111674266A (en) | A state monitoring method, system and related device of a suspension control system | |
CN116450349A (en) | A method, device, system, equipment and medium for improving system availability | |
CN108039717B (en) | Capacity configuration method and device for converter station in multi-terminal flexible direct-current power grid | |
CN213041272U (en) | Intelligent control box display management system based on Beidou | |
CN111429070A (en) | Warehouse management system easy for classification recording | |
CN113221333A (en) | Single-device relay protection reliability assessment method suitable for localization of core components | |
CN113433897B (en) | Product debugging method, service industrial personal computer and collaboration server | |
CN118337815A (en) | Edge fusion low-redundancy data acquisition and processing method | |
CN117170878B (en) | A method to dynamically adjust CPU and GPU cache | |
CN119202091B (en) | Method and device for adjusting polling strategy through Markov chain | |
CN221200305U (en) | Intelligent temperature and liquid level control system | |
CN118199260A (en) | Power grid broadband oscillation monitoring method, system, device and computer equipment | |
CN109738846B (en) | Meter calibration method based on meter active self-calibration | |
CN114968709A (en) | Server performance real-time analysis method based on load data processing | |
CN119359208A (en) | A material-based intelligent inventory adaptive adjustment system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CB03 | Change of inventor or designer information |
Inventor after: Huang Lin Inventor after: Shan Shitai Inventor after: Yang Shaokun Inventor after: Ji Feng Inventor before: Ji Feng |