CN116186004A

CN116186004A - Data cleaning method, device, computer equipment and storage medium

Info

Publication number: CN116186004A
Application number: CN202211645984.XA
Authority: CN
Inventors: 胡冉; 厉冰; 马楠; 许志锋; 刘国伟; 黄湛华
Original assignee: Shenzhen Power Supply Co ltd
Current assignee: Shenzhen Power Supply Co ltd
Priority date: 2022-12-20
Filing date: 2022-12-20
Publication date: 2023-05-30

Abstract

The present disclosure relates to the field of power grid monitoring technologies, and in particular, to a data cleaning method, a data cleaning device, a computer device, and a storage medium. The method comprises the following steps: acquiring initial temperature data and initial current data of a target cable line at each detection moment; determining abnormal temperature data in the initial temperature data; determining a temperature data observation group comprising abnormal temperature data from the initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data; determining a target current data observation group associated with the temperature data observation group from each initial current data; determining a second change trend corresponding to the temperature data observation group according to the first change trend corresponding to the target current data observation group; and compensating abnormal temperature data in the temperature data observation group according to the second change trend. The method and the device can improve the accuracy of data.

Description

Data cleaning method, device, computer equipment and storage medium

Technical Field

The present disclosure relates to the field of power grid monitoring technologies, and in particular, to a data cleaning method, a data cleaning device, a computer device, and a storage medium.

Background

Temperature data of a power transmission cable line can cause damage to a power system or even serious accidents to occur beyond a limit value, so that a cable line temperature monitoring technology is an important point of the long-term study of the power system.

At present, the line temperature monitoring technology relies on a steady-state heat balance equation of a specific meteorological environment wire, a temperature sensing optical fiber and the like are buried in a cable to directly measure the temperature of the conductor, or a cable temperature field is calculated to obtain the corresponding relation between the surface temperature of the cable and the temperature of the conductor, and then the measured value of the surface temperature is combined to obtain the temperature of a cable core of the cable.

However, the existing line temperature monitoring technology is greatly affected by factors such as meteorological environment, electrical performance, mechanical performance and thermal phenomenon of a wire, and the like, and abnormal data can occur, so that the accuracy of the obtained temperature data is reduced, and therefore, improvement is needed.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data cleansing method, apparatus, computer device, and storage medium capable of improving data accuracy.

In a first aspect, the present application provides a data cleansing method, the method comprising:

acquiring initial temperature data and initial current data of a target cable line at each detection moment;

Determining abnormal temperature data in the initial temperature data;

determining a temperature data observation group comprising abnormal temperature data from the initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data;

determining a target current data observation group associated with the temperature data observation group from each initial current data;

determining a second change trend corresponding to the temperature data observation group according to the first change trend corresponding to the target current data observation group;

and compensating abnormal temperature data in the temperature data observation group according to the second change trend.

In one embodiment, determining a set of target current data observations associated with a set of temperature data observations from each initial current data, comprises:

determining at least one candidate current data observation group from each initial current data according to the abnormality detection time;

and determining a target current data observation group associated with the temperature data observation group from the candidate current data observation groups according to the similarity between the candidate current data observation groups and the temperature data observation groups.

In one embodiment, the data cleansing method further comprises:

determining effective detection time corresponding to initial temperature data except abnormal temperature data in the temperature data observation group aiming at each candidate current data observation group;

And determining initial current data corresponding to the effective detection time in the candidate current data observation group, and similarity between the initial current data corresponding to the effective detection time in the temperature data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, wherein the similarity is used as the similarity between the candidate current data observation group and the temperature data observation group.

In one embodiment, determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group includes:

determining a comprehensive observation matrix according to initial current data corresponding to effective detection time in the candidate current data observation group and initial temperature data corresponding to effective detection time in the temperature data observation group;

determining a covariance matrix according to the comprehensive observation matrix;

determining the maximum value data from the comprehensive observation matrix, wherein the maximum value data comprises a temperature maximum value, a temperature minimum value, a current maximum value and a current minimum value;

and determining initial current data corresponding to the effective detection time in the candidate current data observation group according to the covariance matrix and the maximum value data, and similarity between the initial current data and initial temperature data corresponding to the effective detection time in the temperature data observation group.

In one embodiment, determining abnormal temperature data in each initial temperature data includes:

for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, the initial temperature data corresponding to the detection time is taken as abnormal temperature data.

In one embodiment, compensating for abnormal temperature data in the temperature data observation set according to the second variation trend includes:

according to the second change trend, determining a compensation value corresponding to abnormal temperature data in the temperature data observation group;

if the error between the compensation value and the abnormal temperature data meets the preset condition, compensating the abnormal temperature data in the temperature data observation group according to the compensation value.

In a second aspect, the present application also provides a data cleansing apparatus, the apparatus comprising:

the acquisition module is used for acquiring initial temperature data and initial current data of the target cable line at each detection moment;

the screening module is used for determining abnormal temperature data in the initial temperature data;

the temperature combination module is used for determining a temperature data observation group comprising abnormal temperature data from the initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data;

The current combination module is used for determining a target current data observation group associated with the temperature data observation group from all initial current data;

the calculation module is used for determining a second change trend corresponding to the temperature data observation group according to the first change trend corresponding to the target current data observation group;

and the compensation module is used for compensating abnormal temperature data in the temperature data observation group according to the second change trend.

In a third aspect, the present application also provides a computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

determining abnormal temperature data in the initial temperature data;

And compensating abnormal temperature data in the temperature data observation group according to the second change trend. In a fourth aspect, the present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

determining abnormal temperature data in the initial temperature data;

and compensating abnormal temperature data in the temperature data observation group according to the second change trend. In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of:

determining abnormal temperature data in the initial temperature data;

and compensating abnormal temperature data in the temperature data observation group according to the second change trend. According to the data cleaning method, the device, the computer equipment and the storage medium, when the abnormal temperature data exists in the initial temperature number, the temperature data observation group comprising the abnormal temperature data is formed based on the abnormal detection time corresponding to the abnormal temperature data, the target current data observation group related to the existence of the temperature data observation group is determined, the second variation trend of the temperature data observation group can be determined based on the first variation trend of the target current data observation group based on the principle that the variation of the temperature data lags behind the variation of the current data, the compensation value corresponding to the abnormal detection time can be calculated based on the second variation trend, the abnormal temperature data is compensated, and the cleaning of the abnormal temperature data is realized.

Drawings

FIG. 1 is a diagram of an application environment for a data cleansing method in one embodiment;

FIG. 2 is a flow chart of a method of cleaning data in one embodiment;

FIG. 3 is a schematic diagram showing a hysteresis relationship between a current variation trend and a temperature variation trend in one embodiment;

FIG. 4 is a flow chart of determining a set of target current data observations in one embodiment;

FIG. 5 is a flow chart of determining similarity between a candidate current data observation set and a temperature data observation set in one embodiment;

FIG. 6 is a flow chart of another embodiment for determining similarity between a candidate set of current data observations and a temperature data observation set;

FIG. 7 is a flow chart of compensating for abnormal temperature data in one embodiment;

FIG. 8 is a flow chart of a method of cleaning data in another embodiment;

FIG. 9 is a block diagram showing the structure of a data washer in one embodiment;

fig. 10 is an internal structural view of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

The data cleaning method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the data acquisition device 102 (e.g., temperature sensor and current sensor) communicates with the server 104 over a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. For example, the server 104 acquires initial temperature data and initial current data of the target cabling at each detection time; determining abnormal temperature data in the initial temperature data; determining a temperature data observation group comprising abnormal temperature data from the initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data; determining a target current data observation group associated with the temperature data observation group from each initial current data; determining a second change trend corresponding to the temperature data observation group according to the first change trend corresponding to the target current data observation group; and compensating abnormal temperature data in the temperature data observation group according to the second change trend. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.

In one embodiment, as shown in fig. 2, a data cleansing method is provided, and the method is applied to the server 104 in fig. 1 for illustration, and includes the following steps:

s201, acquiring initial temperature data and initial current data of a target cable line at each detection moment.

The temperature sensor acquires initial temperature data of each detection moment at preset sampling intervals, and the current sensor acquires initial current data of each detection moment at the same preset sampling intervals. Specifically, the temperature sensor may have a coding error during the data transmission process, and a part of the initial temperature data is coded into error data (for example, the data is not aligned), and in addition, the coding error may also cause an abnormality in the timestamp of the initial temperature data.

In this embodiment, in the case where there is an abnormality in the time stamp of a part of the initial temperature data, each initial temperature data is first preprocessed, that is, the arrangement order of each initial temperature data is determined according to the reading timing (transmission timing or reception timing) of each initial temperature data, in which case the detection timing corresponding to the ordered initial temperature data is calibrated, but there may be erroneous data in a part of the initial temperature data.

It can be understood that, since the change of the temperature data is caused by the change of the current data, the embodiment acquires the initial current data of each detection time at the same preset sampling interval, so as to compensate the abnormal temperature data by using the complete initial current data.

S202, determining abnormal temperature data in the initial temperature data.

It can be appreciated that if the target cabling is affected by extreme weather conditions or severe operating conditions occur, or if both the temperature sensor and the current sensor fail, both the initial temperature data and the initial current data at the same detection time are abnormal. In this case, it may be determined that the target cable line is faulty, and the server 104 should send an alarm signal to the remote decision platform to obtain relevant control policies to regulate and solve the fault, without performing cleaning of erroneous data (abnormal data).

In contrast, when there is an abnormality in the temperature sensor (or the initial temperature data) and there is no abnormality in the current sensor (initial current data), the initial temperature data can be predictively compensated according to the complete (correct) initial current data.

Therefore, in this embodiment, when determining whether any initial temperature data is abnormal temperature data, it needs to determine that the initial temperature data is abnormal, and meanwhile, the initial current data corresponding to the initial temperature data at the same time is not abnormal, and in this case, the abnormal temperature data can be compensated by using the complete initial current data.

S203, determining a temperature data observation group comprising abnormal temperature data from the initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data.

The verification module in the server 104 detects the abnormal condition of the data in the initial temperature data at a preset frequency, and the verification module in the server 104 determines the abnormal temperature data corresponding to the detection time as the abnormal detection time.

It will be appreciated that as shown in fig. 3, the change in temperature data is due to the change in current data, and the change in temperature data lags the change in current data; therefore, the temperature data forms a corresponding temperature variation trend within a certain period of time, and the current variation trend in the previous period of time should be similar to that in the previous period of time. Therefore, in the present embodiment, initial temperature data in a certain period of time is extracted from each initial temperature data, and a temperature data observation group including abnormal temperature data is formed; further, the temperature data is used to observe the temperature change trend of the group, and the current data group corresponding to the similar current change trend is calculated.

S204, determining a target current data observation group associated with the temperature data observation group from the initial current data.

Wherein there is a correlation between the target current data observation group and the temperature data observation group, the correlation may satisfy a preset correlation condition for similarity between data changes (i.e., a temperature change trend and a current change trend).

Specifically, when the target current data observation group is selected, according to the principle that the change of the temperature data lags behind the current change data, the time period corresponding to the current data observation group should be equal to or lead the time period corresponding to the temperature data observation group, and in order to ensure the correlation accuracy, the number of initial current data in the current data observation group should be the same as the number of initial temperature data in the temperature data observation group.

For example, the number is represented by the sequence of the detection moments (the larger the number is, the later the time is), if the detection time period corresponding to the temperature data observation group includes 100 detection moments, the numbers of the detection moments are {201,202,203, 204..300 }; correspondingly, the detection time period corresponding to the target current data observation group also includes 100 detection times, and the number of each detection time may be {201,202,203, 204..once again..300 } (indicating a time period corresponding to the temperature data observation group) or {101,102,103, 104..once again..200 } (indicating a time period corresponding to the preceding temperature data observation group).

S205, determining a second change trend corresponding to the temperature data observation group according to the first change trend corresponding to the target current data observation group.

Specifically, when determining the first change trend corresponding to the target current data observation group, connecting the current data at each detection time in the target current data observation group, determining the slope corresponding to each initial current data (detection time) in the target current data observation group based on each connecting line, and determining the first change trend corresponding to the target current data observation group based on the slope of each initial current data (detection time).

Further, the first change trend, that is, the slope corresponding to each initial current data (detection time) is respectively assigned to the initial temperature data corresponding to the corresponding detection time in the temperature data observation group; in this case, the abnormality detection timing is also given a corresponding slope; at this time, the slope of each initial temperature data (detection timing) in the temperature data observation group is determined as the second trend corresponding to the temperature data observation group.

S206, compensating abnormal temperature data in the temperature data observation group according to the second change trend.

Specifically, based on the second variation trend, for the abnormal temperature data in the temperature data observation group, knowing the normal temperature data before the abnormal temperature data, the normal temperature data after the abnormal temperature data, and the slope of the abnormal temperature data, the compensation value corresponding to the abnormal temperature data can be calculated. Further, based on the compensation value, the abnormal temperature data is compensated, and cleaning of the abnormal temperature data is achieved.

In the above data cleaning method, when it is determined that abnormal temperature data exists in the initial temperature number, a temperature data observation group including abnormal temperature data is formed based on an abnormal detection time corresponding to the abnormal temperature data, a target current data observation group associated with the existence of the temperature data observation group is determined, based on the principle that the change of the temperature data lags behind the change of the current data, a second change trend of the temperature data observation group can be determined based on a first change trend of the target current data observation group, and a compensation value corresponding to the abnormal detection time can be calculated based on the second change trend, so that the abnormal temperature data is compensated, and cleaning of the abnormal temperature data is realized.

In one embodiment, this embodiment provides an alternative way of determining the abnormal temperature data in the initial temperature data, i.e. a way of refining S201. The specific implementation process can comprise the following steps: for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, the initial temperature data corresponding to the detection time is taken as abnormal temperature data.

The verification module in the server 104 detects initial temperature data at each detection time, if it recognizes that the timestamp of the initial temperature data is abnormal, it determines that the initial temperature data corresponding to the detection time is abnormal, and further, the server 104 determines whether the initial current data corresponding to the detection time is abnormal, so as to verify whether the reason for the abnormal temperature data is a fault of the whole target cable line (including a temperature sensor and a current sensor), or is only a coding error of the temperature sensor.

In this embodiment, when there is no abnormality in the initial current data corresponding to the abnormality detection time, it is determined that the initial temperature data corresponding to the detection time is abnormal temperature data, so as to ensure the effectiveness of the subsequent data cleaning process.

Further, the temperature data observation set may include at least one abnormal temperature data when the temperature data observation set is compared with the target current data observation set. That is, after determining the abnormality detection time, the verification module in the server 104 may record the number of abnormality detection times, and execute the subsequent compensation process when the number of abnormality detection times reaches the compensation condition, where in this case, the plurality of abnormality temperature data observation groups may be incorporated into the same temperature data observation group, so as to realize synchronous cleaning.

In order to determine an accurate target current data observation set, as shown in fig. 4, the present embodiment provides an alternative way to determine a target current data observation set associated with a temperature data observation set from the initial current data, that is, a way to refine S204. The specific implementation process can comprise the following steps:

s401, determining at least one candidate current data observation group from the initial current data according to the abnormality detection time.

Under the condition that the lag time between the current change trend and the temperature change trend is unknown, a plurality of candidate current data observation groups are required to be selected from initial current data, and the candidate current data observation groups and the temperature data observation groups are compared one by one; in addition, there may or may not be a coincidence between the respective candidate current data observation sets, such as the two candidate current data observation sets illustrated in S204 described above.

S402, determining a target current data observation group associated with the temperature data observation group from the candidate current data observation groups according to the similarity between the candidate current data observation groups and the temperature data observation groups.

In calculating the similarity between the respective candidate current data observation sets and the temperature data observation set, as shown in fig. 5, in one embodiment, the data cleaning method further includes:

s501, for each candidate current data observation group, a valid detection time corresponding to initial temperature data except for abnormal temperature data in the temperature data observation group is determined.

When determining the similarity between the candidate current data observation group and the temperature data observation group, the same number of data is required to be ensured. In addition, since the abnormal temperature data in the temperature data observation group is unavailable data, the number of the abnormal temperature data is 5 by way of example, and the rest 95 are available data; the available data in the corresponding candidate current data observation set should also be 95.

In the present embodiment, the valid detection times corresponding to the initial temperature data other than the abnormal temperature data in the temperature data observation group are determined, and as exemplified above, the temperature data observation group corresponds to 95 valid detection times.

S502, determining initial current data corresponding to effective detection time in the candidate current data observation group, similarity between the initial current data corresponding to effective detection time in the temperature data observation group and similarity between the initial temperature data corresponding to effective detection time in the temperature data observation group, and taking the similarity as similarity between the candidate current data observation group and the temperature data observation group.

When determining initial current data corresponding to effective detection time in the candidate current data observation group, the initial current data can be determined according to a time difference deltat between the candidate current data observation group and the temperature data observation group, wherein the time difference deltat can be represented by a number difference value corresponding to the detection time.

As exemplified in S204, Δt=0, 100; then, correspondingly, assuming that in the temperature data observation group, the time corresponding to one effective detection time is t1, the effective detection time is as follows at the detection time t1' corresponding to the candidate current data observation group: t1' =t1- Δt.

Further, in one embodiment, when determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation set and the initial temperature data corresponding to the effective detection time in the temperature data observation set, as shown in fig. 6, the method specifically may include the following steps:

S601, determining a comprehensive observation matrix according to initial current data corresponding to effective detection time in the candidate current data observation group and initial temperature data corresponding to effective detection time in the temperature data observation group.

For convenience of description, determining an effective detection time in the temperature data observation group as a first effective detection time, and determining a detection time corresponding to the first effective detection time in the candidate current data observation group as a second effective detection time; each first detection instant corresponds to each second detection instant, i.e. the first detection instant is delayed by a time Δt compared to the second detection instant.

Specifically, for any group of first effective detection time and second effective detection time with corresponding relation, the initial current data corresponding to the first effective detection time and the initial temperature data corresponding to the second effective detection time form an observation vector, and the initial current data corresponding to the first effective detection time and the initial current data corresponding to the second effective detection time of each group form a plurality of observation vectors X ₁ 、X ₂ 、X ₃ ......X _95. Forming a comprehensive observation matrix [ X ] based on each observation vector]。

The number of total initial temperature data in the temperature data observation group is N, the number of abnormal temperature data is z, and the number of initial temperature data corresponding to the first effective detection time is N-z, wherein the observation vector X ₁ 、X ₂ The following is shown:

the comprehensive observation matrix X is shown below,

taking any observation vector X1 as an example, where X1 is initial temperature data corresponding to a first effective detection time, and y1 is initial current data corresponding to a second effective detection time.

S602, determining a covariance matrix according to the comprehensive observation matrix.

Specifically, the process of calculating the covariance matrix is as follows:

first, based on the mean vector M of the respective observation vectors, illustratively, n=100:

secondly, let the

k=1, 2,3. (100-z), then there is a new matrix B:

then, the covariance matrix S can be obtained from the matrix B as:

in this embodiment, the covariance matrix S is a 2-row and 2-column matrix; the purpose of forming the comprehensive observation matrix is to calculate the corresponding Covariance (Covariance) of the observation matrix, the Covariance is used for measuring the total error of two variables in probability theory and statistics, and the Covariance matrix only represents the Covariance relation of all variables in the form of a matrix.

S603, determining the most value data from the comprehensive observation matrix.

Wherein the maximum data includes a temperature maximum value, a temperature minimum value, a current maximum value, and a current minimum value. Specifically, the maximum value of the temperature x _max To integrate the maximum value, the minimum value x of the temperature in all initial temperature data in the observation matrix _min The minimum value in all initial temperature data in the comprehensive observation matrix is set; maximum value of current y _max To integrate the maximum value, the current minimum value y, in all initial current data in the observation matrix _min Is the minimum value of all initial current data in the comprehensive observation matrix.

S604, determining initial current data corresponding to effective detection time in the candidate current data observation group and similarity between initial temperature data corresponding to effective detection time in the temperature data observation group according to the covariance matrix and the target data.

Specifically, based on the maximum value x of the temperature _max And a current maximum value y _max Form a maximum value vector X _max I.e.

Based on the maximum value x of the temperature _min And a current maximum value y _min Form a maximum value vector X _min I.e.

Wherein, the process of calculating the similarity delta d between the candidate current data observation group and the temperature data observation group is as follows:

wherein S is the covariance matrix in S602, and M is the mean vector M.

Further, the similarity Δd, Δd=d is calculated _max (T，I)-d _min (T，I)。

Specifically, the smaller Δd, the higher the similarity between the candidate current data observation set and the temperature data observation set. In determining the similarity between the candidate current data observation set and the temperature data observation set, only the similarity between the data is considered, and the influence of the time difference is not considered.

As shown in fig. 7, the present embodiment provides an alternative way to compensate for abnormal temperature data in the temperature data observation group according to the second trend of variation, that is, a way to refine S206. The specific implementation process can comprise the following steps:

s701, according to the second change trend, determining a compensation value corresponding to abnormal temperature data in the temperature data observation group.

When determining the corresponding compensation value of the abnormal temperature data corresponding to the temperature data observation group according to the second variation trend, the method specifically may be: for any abnormal temperature data, according to the initial temperature data corresponding to one effective detection time before the abnormal temperature data, the initial temperature data corresponding to one effective detection time after the abnormal temperature data and the slope corresponding to the abnormal temperature data, the compensation value corresponding to the abnormal temperature data can be calculated.

S702, if the error between the compensation value and the abnormal temperature data meets the preset condition, compensating the abnormal temperature data in the temperature data observation group according to the compensation value.

If the difference between the compensation value and the abnormal temperature data (coding error data) is smaller than 10%, compensating the abnormal temperature data in the temperature data observation group according to the compensation value; otherwise, the abnormal temperature data is reserved.

Illustratively, on the basis of the above-described embodiments, this embodiment provides an alternative example of a data cleaning method. As shown in fig. 8, the specific implementation process includes:

s801, initial temperature data and initial current data of a target cable line at each detection moment are acquired.

S802, regarding any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, the initial temperature data corresponding to the detection time is taken as abnormal temperature data.

S803, according to the abnormality detection time corresponding to the abnormal temperature data, a temperature data observation group comprising the abnormal temperature data is determined from the initial temperature data.

S804, determining at least one candidate current data observation group from the initial current data according to the abnormality detection time.

S805, determining a target current data observation group associated with the temperature data observation group from among the candidate current data observation groups according to the similarity between the candidate current data observation groups and the temperature data observation groups.

Specifically, for each candidate current data observation group, determining effective detection time corresponding to initial temperature data except abnormal temperature data in the temperature data observation group; and determining initial current data corresponding to the effective detection time in the candidate current data observation group, and similarity between the initial current data corresponding to the effective detection time in the temperature data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, wherein the similarity is used as the similarity between the candidate current data observation group and the temperature data observation group.

Further, determining the similarity between the initial current data corresponding to the effective detection time in the candidate current data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group includes: determining a comprehensive observation matrix according to initial current data corresponding to effective detection time in the candidate current data observation group and initial temperature data corresponding to effective detection time in the temperature data observation group; determining a covariance matrix according to the comprehensive observation matrix; determining the maximum value data from the comprehensive observation matrix, wherein the maximum value data comprises a temperature maximum value, a temperature minimum value, a current maximum value and a current minimum value; and determining initial current data corresponding to the effective detection time in the candidate current data observation group according to the covariance matrix and the maximum value data, and similarity between the initial current data and initial temperature data corresponding to the effective detection time in the temperature data observation group.

S806, determining a second change trend corresponding to the temperature data observation group according to the first change trend corresponding to the target current data observation group.

S807, according to the second variation trend, determining a compensation value corresponding to the abnormal temperature data in the temperature data observation group.

S808, if the error between the compensation value and the abnormal temperature data meets the preset condition, compensating the abnormal temperature data in the temperature data observation group according to the compensation value.

The specific process of S801 to S808 may refer to the description of the foregoing method embodiment, and its implementation principle and technical effect are similar, and are not repeated herein.

It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data cleaning device for realizing the above related data cleaning method. The implementation of the solution provided by the device is similar to that described in the above method, so the specific limitation of one or more embodiments of the data cleaning device provided below may be referred to the limitation of the data cleaning method hereinabove, and will not be repeated here.

In one embodiment, as shown in fig. 9, there is provided a data cleaning apparatus comprising: the system comprises an acquisition module 11, a screening module 12, a temperature combination module 13, a current combination module 14, a calculation module 15 and a compensation module 16, wherein:

an obtaining module 11, configured to obtain initial temperature data and initial current data of the target cable line at each detection time;

a screening module 12, configured to determine abnormal temperature data in each initial temperature data;

a temperature combination module 13, configured to determine a temperature data observation group including abnormal temperature data from the initial temperature data according to an abnormal detection time corresponding to the abnormal temperature data;

a current combining module 14 for determining a target current data observation group associated with the temperature data observation group from each initial current data;

The calculating module 15 is configured to determine a second variation trend corresponding to the temperature data observation set according to the first variation trend corresponding to the target current data observation set;

and the compensation module 16 is used for compensating abnormal temperature data in the temperature data observation group according to the second variation trend.

In one embodiment, the screening module 12 is further configured to: determining at least one candidate current data observation group from each initial current data according to the abnormality detection time;

In one embodiment, the data cleansing apparatus further includes a similarity calculation module 15, and the similarity calculation module 15 includes:

the effective screening submodule is used for determining effective detection time corresponding to initial temperature data except abnormal temperature data in the temperature data observation group for each candidate current data observation group;

and the similarity calculation submodule is used for determining initial current data corresponding to the effective detection time in the candidate current data observation group, similarity between the initial current data corresponding to the effective detection time in the temperature data observation group and similarity between the initial current data corresponding to the effective detection time in the temperature data observation group, and the similarity is used as the similarity between the candidate current data observation group and the temperature data observation group.

In one embodiment, the similarity calculation submodule is further configured to: determining a comprehensive observation matrix according to initial current data corresponding to effective detection time in the candidate current data observation group and initial temperature data corresponding to effective detection time in the temperature data observation group;

In one embodiment, the screening module 12 is further configured to: for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, the initial temperature data corresponding to the detection time is taken as abnormal temperature data.

In one embodiment, the compensation module 16 is further configured to: according to the second change trend, determining a compensation value corresponding to abnormal temperature data in the temperature data observation group;

The respective modules in the above-described data cleaning apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 10. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store XX data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data cleansing method.

It will be appreciated by those skilled in the art that the structure shown in fig. 10 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:

determining abnormal temperature data in the initial temperature data;

In one embodiment, when the processor executes the logic of the computer program to determine the target current data observation group associated with the temperature data observation group from each initial current data, the following steps are specifically implemented: determining at least one candidate current data observation group from each initial current data according to the abnormality detection time; and determining a target current data observation group associated with the temperature data observation group from the candidate current data observation groups according to the similarity between the candidate current data observation groups and the temperature data observation groups.

In one embodiment, the processor when executing the computer program further performs the steps of: determining effective detection time corresponding to initial temperature data except abnormal temperature data in the temperature data observation group aiming at each candidate current data observation group; and determining initial current data corresponding to the effective detection time in the candidate current data observation group, and similarity between the initial current data corresponding to the effective detection time in the temperature data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, wherein the similarity is used as the similarity between the candidate current data observation group and the temperature data observation group.

In one embodiment, when the processor executes logic for determining the similarity between the initial current data corresponding to the valid detection time in the candidate current data observation set and the initial temperature data corresponding to the valid detection time in the temperature data observation set, the following steps are specifically implemented: determining a comprehensive observation matrix according to initial current data corresponding to effective detection time in the candidate current data observation group and initial temperature data corresponding to effective detection time in the temperature data observation group; determining a covariance matrix according to the comprehensive observation matrix; determining the maximum value data from the comprehensive observation matrix, wherein the maximum value data comprises a temperature maximum value, a temperature minimum value, a current maximum value and a current minimum value; and determining initial current data corresponding to the effective detection time in the candidate current data observation group according to the covariance matrix and the maximum value data, and similarity between the initial current data and initial temperature data corresponding to the effective detection time in the temperature data observation group.

In one embodiment, the processor, when executing logic for determining abnormal temperature data in each initial temperature data by a computer program, performs the steps of: for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, the initial temperature data corresponding to the detection time is taken as abnormal temperature data.

In one embodiment, when the processor executes the logic for compensating for the abnormal temperature data in the temperature data observation group according to the second variation trend, the following steps are specifically implemented: according to the second change trend, determining a compensation value corresponding to abnormal temperature data in the temperature data observation group; if the error between the compensation value and the abnormal temperature data meets the preset condition, compensating the abnormal temperature data in the temperature data observation group according to the compensation value.

In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:

determining abnormal temperature data in the initial temperature data;

In one embodiment, the computer program determines, from each initial current data, a target current data observation set associated with a temperature data observation set, logic that when executed by a processor, specifically performs the steps of: determining at least one candidate current data observation group from each initial current data according to the abnormality detection time; and determining a target current data observation group associated with the temperature data observation group from the candidate current data observation groups according to the similarity between the candidate current data observation groups and the temperature data observation groups.

In one embodiment, the computer program when executed by the processor further performs the steps of: determining effective detection time corresponding to initial temperature data except abnormal temperature data in the temperature data observation group aiming at each candidate current data observation group; and determining initial current data corresponding to the effective detection time in the candidate current data observation group, and similarity between the initial current data corresponding to the effective detection time in the temperature data observation group and the initial temperature data corresponding to the effective detection time in the temperature data observation group, wherein the similarity is used as the similarity between the candidate current data observation group and the temperature data observation group.

In one embodiment, the computer program determines initial current data corresponding to the effective detection time in the candidate current data observation set, and logic for similarity between initial temperature data corresponding to the effective detection time in the temperature data observation set is executed by the processor, and specifically implements the following steps: determining a comprehensive observation matrix according to initial current data corresponding to effective detection time in the candidate current data observation group and initial temperature data corresponding to effective detection time in the temperature data observation group; determining a covariance matrix according to the comprehensive observation matrix; determining the maximum value data from the comprehensive observation matrix, wherein the maximum value data comprises a temperature maximum value, a temperature minimum value, a current maximum value and a current minimum value; and determining initial current data corresponding to the effective detection time in the candidate current data observation group according to the covariance matrix and the maximum value data, and similarity between the initial current data and initial temperature data corresponding to the effective detection time in the temperature data observation group.

In one embodiment, the logic of the computer program determining the abnormal temperature data in each initial temperature data, when executed by the processor, specifically implements the steps of: for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, the initial temperature data corresponding to the detection time is taken as abnormal temperature data.

In one embodiment, the computer program specifically implements the following steps when the logic for compensating for abnormal temperature data in the temperature data observation group according to the second variation trend is executed by the processor: according to the second change trend, determining a compensation value corresponding to abnormal temperature data in the temperature data observation group; if the error between the compensation value and the abnormal temperature data meets the preset condition, compensating the abnormal temperature data in the temperature data observation group according to the compensation value.

In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:

determining abnormal temperature data in the initial temperature data;

It should be noted that, user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A method of data cleansing, the method comprising:

determining abnormal temperature data in the initial temperature data;

determining a temperature data observation group comprising the abnormal temperature data from the initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data;

determining a second variation trend corresponding to the temperature data observation group according to the first variation trend corresponding to the target current data observation group;

2. The method of claim 1, wherein said determining a set of target current data observations associated with said set of temperature data observations from each initial current data comprises:

3. The method according to claim 2, wherein the method further comprises:

determining effective detection time corresponding to initial temperature data except for the abnormal temperature data in the temperature data observation group aiming at each candidate current data observation group;

And determining initial current data corresponding to the effective detection time in the candidate current data observation group, and using the similarity between the initial current data corresponding to the effective detection time in the temperature data observation group as the similarity between the candidate current data observation group and the temperature data observation group.

4. A method according to claim 3, wherein said determining initial current data corresponding to valid detection times in the candidate current data observation set, and the similarity between initial temperature data corresponding to valid detection times in the temperature data observation set, comprises:

determining the maximum data from the comprehensive observation matrix, wherein the maximum data comprises a temperature maximum value, a temperature minimum value, a current maximum value and a current minimum value;

and determining initial current data corresponding to the effective detection time in the candidate current data observation group according to the covariance matrix and the maximum value data, and determining the similarity between the initial current data corresponding to the effective detection time in the temperature data observation group and the initial temperature data corresponding to the effective detection time.

5. The method of claim 1, wherein determining abnormal temperature data in each initial temperature data comprises:

and for any detection time, if the initial temperature data corresponding to the detection time is abnormal and the initial current data corresponding to the detection time is normal, taking the initial temperature data corresponding to the detection time as the abnormal temperature data.

6. The method of claim 1, wherein compensating for abnormal temperature data in the temperature data observation set according to the second trend of variation comprises:

determining a compensation value corresponding to abnormal temperature data in the temperature data observation group according to the second change trend;

and if the error between the compensation value and the abnormal temperature data meets a preset condition, compensating the abnormal temperature data in the temperature data observation group according to the compensation value.

7. A data cleansing apparatus, the apparatus comprising:

The temperature combination module is used for determining a temperature data observation group comprising the abnormal temperature data from the initial temperature data according to the abnormal detection time corresponding to the abnormal temperature data;

a current combination module for determining a target current data observation group associated with the temperature data observation group from each initial current data;

and the compensation module is used for compensating the abnormal temperature data in the temperature data observation group according to the second change trend.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.