CN111324600A - Data cleaning method and device - Google Patents

Abstract

The invention provides a data cleaning method and a device, wherein the method comprises the following steps: the method comprises the steps that terminal equipment obtains detection data of the detection equipment in a first collection period, data cleaning is conducted on the detection data according to a threshold range of the detection data, the data cleaning is used for discarding invalid values in the detection data, the threshold range of the detection data is indicated by a server, and if available data exist after the data cleaning, the terminal equipment reports the available data to the server in a first reporting period. In the first updating period, the server generates a first range of the detection data according to the available data reported by the terminal equipment in the first reporting period, and sends the first range of the detection data to the terminal equipment. By the mode, the terminal equipment can update the range of the detection data in real time according to the indication of the server, and the data cleaning effect is improved.

Description

Data cleaning method and device

technical field

The present invention relates to the field of information technology, and in particular, to a data cleaning method and device.

Background technique

During data collection, sensors are usually used to collect data and send it to edge terminals such as edge gateways and edge controllers, and then the edge terminals send the collected data to the server for users to download and use. Due to the influence of the surrounding environment of the sensor, the collected data usually contains a lot of invalid data. Therefore, data cleaning needs to be performed before using the data collected by the sensor.

In the prior art, edge processing is usually used for data cleaning. Specifically, the data cleaning algorithm can be pre-programmed to the edge end, and the data collected by the sensor can be cleaned by the edge end through the cleaning algorithm.

However, due to limited resources at the edge, it is often unable to provide the computing power required to update the cleaning algorithm alone. Therefore, in the prior art, the threshold range of the data corresponding to the data cleaning algorithm is often fixed, and cannot be dynamically adjusted according to the actual situation, resulting in a poor cleaning effect of the existing data cleaning method by the edge.

SUMMARY OF THE INVENTION

The present invention provides a data cleaning method and device to solve the technical problem of poor data cleaning effect in the prior art.

A first aspect of the present invention provides a data cleaning method, applied to a terminal device, the method includes:

Acquire the detection data of the detection equipment in the first collection period;

According to the threshold range of the detection data, data cleaning is performed on the detection data, and the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server;

If there is available data after data cleaning, the available data is reported to the server in the first reporting cycle.

In an optional implementation manner, after performing data cleaning on the detection data, the method further includes:

If there is no available data after data cleaning, linear interpolation is performed on the historically available data to predict the available data in the first collection period;

The predicted available data is reported to the server in the first reporting period.

In an optional implementation manner, after the available data is reported to the server in the first reporting period, the method further includes:

receiving a first range of the detection data sent by the server, where the first range is determined according to available data in the first collection period;

According to the first range, the threshold range of the detection data is updated.

A second aspect of the present invention provides a data cleaning method, which is applied to a server, and the method includes:

Receive the available data reported by the terminal device in the first reporting cycle, where the available data is obtained after the terminal device performs data cleaning on the detection data of the detection device in the first collection cycle according to the threshold range of the detection data, and the Data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server;

In the first update period, the first range of the detection data is generated according to the available data reported by the terminal device in the first reporting period;

Sending the first range of detection data to the terminal device.

In an optional implementation manner, before the sending the first update value of the threshold range of the detection data to the terminal device, the method further includes:

receiving the available data reported by the terminal device in a second reporting period, the second reporting period being later than the first reporting period;

The sending the first range of the detection data to the terminal device includes:

If the available data reported in the second reporting period is not within the first range, the first range is sent to the terminal device.

In an optional implementation manner, before the sending the first range of the detection data to the terminal device, the method further includes:

Obtain the unupdated duration of the threshold range of the detection data;

The sending the first range of the detection data to the terminal device includes:

If the unupdated duration exceeds the duration threshold, the first range of the detection data is sent to the terminal device.

A third aspect of the present invention provides a data cleaning device, the device includes: a receiving module, a processing module and a sending module;

The receiving module is used to obtain the detection data of the detection device in the first collection period;

The processing module is configured to perform data cleaning on the detection data according to the threshold range of the detection data, the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is determined by the server instruct;

The sending module is configured to report the available data to the server in the first reporting period if there is available data after data cleaning.

In an optional implementation manner, the processing module is further configured to perform linear interpolation on the historically available data if there is no available data after data cleaning, and predict the available data in the first collection period;

The sending module is further configured to report the predicted available data to the server in the first reporting period.

In an optional implementation manner, the receiving module is further configured to receive a first range of the detection data sent by the server, where the first range is determined according to available data in the first collection period of;

The processing module is further configured to update the threshold range of the detection data according to the first range.

A fourth aspect of the present invention provides a data cleaning device, the device includes: a receiving module, a processing module and a sending module;

The receiving module is configured to receive the available data reported by the terminal device in the first reporting period, and the available data is that the terminal device performs data analysis on the detection data of the detection device in the first collection period according to the threshold range of the detection data. data obtained after cleaning, the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server;

The processing module is configured to, in the first update period, generate the first range of the detection data according to the available data reported by the terminal device in the first reporting period;

The sending module is configured to send the first range of the detection data to the terminal device.

In an optional implementation manner, the receiving module is further configured to receive the available data reported by the terminal device in a second reporting period, and the second reporting period is later than the first reporting period;

The sending module is further configured to send the first range to the terminal device if the available data reported by the second reporting period is not within the first range.

In an optional implementation manner, the receiving module is further configured to acquire the unupdated duration of the threshold range of the detection data;

The sending module is further configured to send the first range of the detection data to the terminal device if the unupdated duration exceeds a duration threshold.

In a fifth aspect of the embodiments of the present invention, a terminal device is provided, including: a memory, a processor, and a computer program, where the computer program is stored in the memory, and the processor executes the computer program to execute the first embodiment of the present invention Aspects and various optional data cleaning methods of the first aspect.

A sixth aspect of the embodiments of the present invention provides a server, including: a memory, a processor, and a computer program, where the computer program is stored in the memory, and the processor executes the computer program to execute the second aspect of the present invention and various optional data cleaning methods in the second aspect.

A seventh aspect of the present invention provides a storage medium, where a computer program is stored in the readable storage medium, and the computer program is used to execute the first aspect and various optional data cleaning methods of the first aspect.

An eighth aspect of the present invention provides a storage medium, where a computer program is stored in the readable storage medium, and the computer program is used to execute the second aspect and various optional data cleaning methods of the second aspect.

In the data cleaning method and device provided by the present invention, the terminal device obtains the detection data of the detection device in the first collection period, and performs data cleaning on the detection data according to the threshold range of the detection data, and the data cleaning is used to discard invalid detection data. The threshold range of the detected data is indicated by the server. If there is available data after data cleaning, the terminal device reports the available data to the server in the first reporting cycle. In the first update cycle, the server generates a first range of detection data according to the available data reported by the terminal device in the first reporting cycle, and sends the first range of detection data to the terminal device. In the above manner, the terminal device can update the range of detection data in real time according to the instruction of the server, which improves the data cleaning effect.

Description of drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are the For some embodiments of the invention, for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic diagram of a scenario of a data cleaning method provided by an embodiment of the present application;

FIG. 2 is a signaling interaction diagram of a data cleaning method provided by an embodiment of the present application;

3 is a schematic structural diagram of a DAE-LSTM model provided by an embodiment of the present application;

FIG. 4 is a signaling interaction diagram of another data cleaning method provided by an embodiment of the present application;

FIG. 5 is a signaling interaction diagram of still another data cleaning method provided by an embodiment of the present application;

6 is a schematic diagram of a network architecture of a data cleaning method provided by an embodiment of the present application;

7 is a schematic diagram of a four-level data cleaning mechanism provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a data cleaning apparatus provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of another data cleaning apparatus provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of still another data cleaning apparatus provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

During data collection, sensors are usually used to collect data and send it to edge terminals such as edge gateways and edge controllers, and then the edge terminals send the collected data to the server for users to download and use. Due to the influence of the surrounding environment of the sensor, the collected data usually contains a lot of invalid data. Therefore, data cleaning needs to be performed before using the data collected by the sensor. In the prior art, edge processing is usually used for data cleaning. Specifically, the data cleaning algorithm can be pre-programmed to the edge end, and the data collected by the sensor can be cleaned by the edge end through the cleaning algorithm. However, due to limited resources at the edge, it is often unable to provide the computing power required to update the cleaning algorithm alone. Therefore, in the prior art, the threshold range of the data corresponding to the data cleaning algorithm is often fixed, and cannot be dynamically adjusted according to the actual situation, resulting in a poor cleaning effect of the existing data cleaning method by the edge.

In view of the above problems, the present invention provides a data cleaning method and device, through which the cleaning process of the detection data is completed by the terminal device, and the determination of the threshold range of the detection data is completed by the server, and the terminal device uploads the detection data to the server. , the server sends the threshold range to the terminal device in real time, so that the terminal device updates the range of the detection data in real time according to the instruction of the server, thereby improving the data cleaning effect.

FIG. 1 is a schematic diagram of a scenario of a data cleaning method provided by an embodiment of the present application. As shown in FIG. 1 , a detection device 101 , a terminal device 102 and a server 103 are included, and the terminal device 102 is connected to the detection device 101 and the server 103 respectively. The detection device 101 is used to collect detection data and send the detection data to the terminal device 102 . The terminal device 102 performs data cleaning on the detection data, and sends the cleaned data to the server 103 . The server 103 determines the threshold range of the new detection data according to the received cleaned detection data.

The detection device 101 may be a sensor, or a controller or a driving device. The detection device 101 is used to collect sensor data such as temperature and position, and may also be used to collect control signals or drive signals.

The terminal device 102 may also be referred to as an edge terminal, and the above-mentioned terminal device 102 may be an edge gateway, an edge controller, or the like. Exemplarily, since an embedded programmable logic controller (ePLC) has easy programming and high reliability, and is widely used in various industrial fields, the terminal device 102 may use an ePLC.

The server 103 may be a server, or a server in a cloud service platform. The server 103 may receive the detection data sent by the terminal device 102, and then determine the threshold range of the detection data according to the received detection data.

It should be noted that the technical solutions of the present application can be applied to the above application scenarios, but are not limited thereto, and can also be applied to other scenarios that require data cleaning.

The technical solutions of the embodiments of the present application will be described in detail below with specific embodiments by taking terminal devices and servers integrated or installed with relevant execution codes as examples. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

FIG. 2 is a signaling interaction diagram of a data cleaning method provided by an embodiment of the present application. This embodiment relates to the specific process of how the terminal device and the server clean the detection data. As shown in Figure 2, the method includes:

S201. The terminal device acquires the detection data of the detection device in the first collection period.

Wherein, the embodiment of the present application does not limit the first collection period, which may be specifically set according to the detection frequency of the detection device. Exemplarily, the detection device may take ten seconds as a collection period, and the terminal device acquires the detection data of the detection device every ten seconds.

In some embodiments, the terminal device may also configure the corresponding acquisition protocol and data dictionary table according to the application environment of the detection device. Among them, the application environment can be exemplified by the environment such as temperature, humidity and the voltage of the monitoring equipment, and the acquisition protocol can be exemplified by serial communication protocol (Modbus), Remote Terminal Unit (Remote Terminal Unit, RTU) protocol, etc., data A dictionary is a collection of information describing data, a collection of definitions for all data elements used in the system. The terminal device collects the detection data of the detection device through the acquisition protocol, and determines the type of the collected detection data through the data dictionary table.

S202. According to the threshold range of the detection data, the terminal device performs data cleaning on the detection data, the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server.

In this step, after the terminal device acquires the detection data of the detection device in the first collection period, before uploading the detection data to the server, the terminal device may also perform data cleaning on the detection data according to the threshold range of the detection data, and discard the detection data. Invalid value in data.

Since the normal detection data usually increases or decreases within a certain range, the detection data whose increase or decrease is too large can be regarded as invalid values. Therefore, data cleaning can be performed through the threshold range of the detection data, and by judging whether the detection data is within the threshold range, it is determined whether the detection data needs to be cleaned. Exemplarily, if the detection data is not within the threshold range, it is determined that the detection data is an invalid value, and the detection data is discarded; if the detection data is within the threshold range, it is determined that the detection data is valid, and then saved in the terminal device. the detection data.

In some embodiments, the terminal device may also detect the consistency of the available data, or check whether the available data has missing values.

S203: If there is available data after data cleaning, the terminal device reports the available data to the server in the first reporting period.

In this step, after the terminal device cleans the detection data, if there is available data after the data cleaning, the terminal device reports the available data to the server in the first reporting period.

Exemplarily, every time t is used as an upload cycle, and the data cleaning program in the terminal device will check whether the latest detection data is discarded. If it is detected that the discard flag is False, the available data that exists after cleaning is uploaded to the server. The embodiments of the present application do not limit how to upload the available data. Exemplarily, the data can be uploaded through a 3G network or a 4G network.

In the embodiment of the present application, since the uploaded available data is the data after preliminary cleaning by the terminal device, it is possible to avoid the detection data of massive heterogeneous sensors from being directly uploaded to the server for processing, which reduces the load of the network and also reduces the cost. .

S204. In the first update period, the server generates a first range of detection data according to the available data reported by the terminal device in the first reporting period.

In some embodiments, the server generates a new threshold range of detection data every update period. The embodiments of the present application do not limit how to generate the threshold range of the detection data. An optional implementation may generate the detection data through a denoising autoencoder long short-term memory (DAE-LSTM) model. The threshold range of the DAE-LSTM model is obtained by inputting the available data to the DAE-LSTM model to obtain the threshold range of the detection data output by the DAE-LSTM model. The DAE-LSTM model is established by training from the historical available data and the historical threshold range.

FIG. 3 is a schematic structural diagram of a DAE-LSTM model provided by an embodiment of the present application. As shown in FIG. 3 , the input of the DAE-LSTM model is currently processed available data, and the output is the predicted threshold range at the next moment, And through the settings can have a 5% fluctuation.

The model consists of a fully connected network (factorization machine supported neural network, FNN) and a long short-term memory (LSTM) network, and denoising autoencoder (DAE) is used to initialize the parameters of the fully connected layer. . Input the available data X _t into the FNN, and get the output data _rt of the FNN, then input _rt into the LSTM, and finally the LSTM outputs h _t . The DAE-LSTM model is a time series based network, so a time series should be fed into the network.

The input format formula (1) of the model is given below, where N _x is the number of input sequences:

Regarding the training process, the loss function of the LSTM part is as formula (2):

Among them, η ₂ is the adjustment coefficient.

In the DAE-LSTM model, Dense(x, f) represents a fully connected layer with x neurons and using f as the activation function. The expression formula (3) of the linear rectification function (Relu) is as follows:

f(x)=max(x, 0) (3)

The formula (4) of the hyperbolic tangent function (Tanh) is as follows:

S205, the server sends the first range of the detection data to the terminal device.

In this step, if the first range of detection data is generated, the server may directly send the first range of detection data to the terminal device, or may send the first range of detection data to the terminal device when certain conditions are met.

In some optional implementations, the server may receive the available data reported by the terminal device in the second reporting period before sending the first update value of the threshold range of the detection data to the terminal device, and the second reporting period is later than the first reporting cycle. If the available data reported in the second reporting period is not within the first range, the server sends the first range to the terminal device.

In some embodiments, as a fourth-level cleaning mechanism, if the threshold range of possible improper detection data is not updated for a long time, the threshold value of detection data may be obtained before sending the first range of detection data to the terminal device The unupdated duration of the range. If the unupdated duration exceeds the duration threshold, the server sends the first range of detection data to the terminal device.

In some embodiments, before sending the first range of detection data to the terminal device, the user may access the server and intervene whether to send the first range of detection data to the terminal device by setting parameters.

In the data cleaning method provided by the embodiment of the present application, the terminal device acquires the detection data of the detection device in the first collection period, and performs data cleaning on the detection data according to the threshold range of the detection data, and the data cleaning is used to discard invalid detection data. The threshold range of the detected data is indicated by the server. If there is available data after data cleaning, the terminal device reports the available data to the server in the first reporting cycle. In the first update cycle, the server generates a first range of detection data according to the available data reported by the terminal device in the first reporting cycle, and sends the first range of detection data to the terminal device. In the above manner, the terminal device can update the range of detection data in real time according to the instruction of the server, which improves the data cleaning effect.

On the basis of the above embodiment, when the terminal device cleans the detection data and discards invalid values, it may discard all the detection data in the first collection cycle. At this time, the terminal device needs to predict the detection data in the first collection cycle Upload, the following describes how to predict the detection data in the first collection period. FIG. 4 is a signaling interaction diagram of another data cleaning method provided by an embodiment of the present application. As shown in FIG. 4 , the method further includes:

S301. The terminal device acquires the detection data of the detection device in the first collection period.

S302. According to the threshold range of the detection data, the terminal device performs data cleaning on the detection data, and the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server.

The technical terms, technical effects, technical features, and optional implementations of steps S301-S302 can be understood with reference to steps S201-S202 shown in FIG. 2 , and repeated content will not be repeated here.

S303. If there is no available data after data cleaning, the terminal device performs linear interpolation on the historically available data to predict the available data in the first collection period.

In some embodiments, a possible straight-line approach is to compute a linear interpolation of the first two available data in the local database as the available data for the first acquisition cycle. Among them, the formula (5) of linear interpolation is as follows:

Among them, t _n is the n-th collection period, and x _n is the detection data collected in the n-th collection period.

S304. The terminal device reports the predicted available data to the server in the first reporting period.

According to the data cleaning method provided by the embodiment of the present application, if there is no available data after data cleaning, the terminal device performs linear interpolation on the historically available data, predicts the available data in the first collection period, and estimates the available data in the first reporting period. report to the server. In this way, the integrity of the detection data can be improved.

The following describes how the terminal device updates the threshold range of the detection data. FIG. 5 is a signaling interaction diagram of still another data cleaning method provided by an embodiment of the present application. As shown in FIG. 5 , the method further includes:

S401. The terminal device acquires the detection data of the detection device in the first collection period.

S402. According to the threshold range of the detection data, the terminal device performs data cleaning on the detection data, and the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server.

S403. If there is available data after data cleaning, the terminal device reports the available data to the server in the first reporting period.

S404. In the first update period, the server generates a first range of detection data according to the available data reported by the terminal device in the first reporting period.

S405. The server sends the first range of detection data to the terminal device.

S406. According to the first range, the terminal device updates the threshold range of the detection data.

In the data cleaning method provided by the embodiment of the present application, the terminal device acquires the detection data of the detection device in the first collection period, and performs data cleaning on the detection data according to the threshold range of the detection data, and the data cleaning is used to discard invalid detection data. The threshold range of the detected data is indicated by the server. If there is available data after data cleaning, the terminal device reports the available data to the server in the first reporting cycle. In the first update cycle, the server generates a first range of detection data according to the available data reported by the terminal device in the first reporting cycle, and sends the first range of detection data to the terminal device. In the above manner, the terminal device can update the range of detection data in real time according to the instruction of the server, which improves the data cleaning effect.

FIG. 6 is a schematic diagram of a network architecture of a data cleaning method provided by an embodiment of the present application. As shown in FIG. 6 , the server in the above embodiment may be an intelligent decision server, which is used to generate a threshold range of detection data according to available data, and The available data is stored in the database. When uploading the available data, the terminal device will not directly send the information to the intelligent decision server, but intermediately forward it through the message queuing telemetry transport (MQTT) server. In addition, when the user needs to interact with the intelligent decision server, it also needs to be forwarded through a website (web) server.

FIG. 7 is a schematic diagram of a four-level data cleaning mechanism provided by an embodiment of the present application. As shown in FIG. 7 , in the first-level cleaning mechanism, the terminal device collects detection data in the collection period _of the time interval T1, and performs The detection data is cleaned, the cleaned data is stored in the local storage, and the data that has not been cleaned is discarded.

In the _second -level cleaning mechanism, the terminal device uploads the available data retained after cleaning in the reporting period of the time interval T2.

In the third-level cleaning mechanism, the terminal device generates a model with the latest available data threshold and obtains the first range of the current detection data output by the threshold generation model during the update period _of the time interval T3. Then obtain the latest available data, if the available data is within the current first range, the first range will not be sent to the terminal device to update the threshold range, and if the available data is not within the current first range, the first range will be sent Update threshold ranges for end devices.

In the _fourth -level cleaning mechanism, the terminal device is forced to update the threshold range when the unupdated time exceeds the time threshold in the mandatory update period of the time interval T4.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, execute It includes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

FIG. 8 is a schematic structural diagram of a data cleaning apparatus according to an embodiment of the present application. The data transmission device may be implemented by software, hardware or a combination of the two, and may be the aforementioned terminal equipment.

As shown in FIG. 8 , the data cleaning device 500 includes: a receiving module 501, a processing module 502 and a sending module 503;

The receiving module 501 is configured to acquire detection data of the detection device in the first collection period.

The processing module 502 is configured to perform data cleaning on the detection data according to the threshold range of the detection data, and the data cleaning is used for discarding invalid values in the detection data, and the threshold range of the detection data is indicated by the server.

The sending module 503 is configured to report the available data to the server in the first reporting cycle if there is available data after data cleaning.

In an optional implementation manner, the processing module 502 is further configured to perform linear interpolation on the historically available data if there is no available data after data cleaning, and predict the available data in the first collection period; the sending module 503 is further configured to use for reporting the predicted available data to the server in the first reporting cycle.

In an optional implementation manner, the receiving module 501 is further configured to receive a first range of detection data sent by the server, where the first range is determined according to the available data in the first collection period; the processing module 502 is further configured to According to the first range, the threshold range of the detection data is updated.

The data cleaning apparatus provided in the embodiment of the present application can execute the data cleaning method on the terminal device side in the above method embodiment, and the implementation principle and technical effect thereof are similar, and details are not described herein again.

FIG. 9 is a schematic structural diagram of another data cleaning apparatus according to an embodiment of the present application. The data transmission device may be implemented by software, hardware or a combination of the two, and may be the aforementioned server.

As shown in FIG. 9 , the data cleaning device 600 includes: a receiving module 601, a processing module 602 and a sending module 603;

The receiving module 601 is used to receive the available data reported by the terminal device in the first reporting period. The available data is the data obtained after the terminal device performs data cleaning on the detection data of the detection device in the first collection period according to the threshold range of the detection data. , data cleaning is used to discard invalid values in detection data, and the threshold range of detection data is indicated by the server.

The processing module 602 is configured to, in the first update period, generate a first range of detection data according to the available data reported by the terminal device in the first reporting period.

The sending module 603 is configured to send the first range of detection data to the terminal device.

In an optional implementation manner, the receiving module 601 is further configured to receive the available data reported by the terminal device in the second reporting period, and the second reporting period is later than the first reporting period; the sending module 603 is further configured to receive the available data if the first reporting period is The available data reported in the second reporting period is not within the first range, and the first range is sent to the terminal device.

In an optional implementation manner, the receiving module 601 is further configured to obtain the unupdated duration of the threshold range of the detection data; the sending module 603 is further configured to obtain the first unupdated duration of the detection data if the unupdated duration exceeds the duration threshold The range is sent to the end device.

The data cleaning apparatus provided in the embodiment of the present application can execute the data cleaning method on the server side in the above method embodiment, and the implementation principle and technical effect thereof are similar, and are not repeated here.

FIG. 10 is a schematic structural diagram of still another data cleaning apparatus provided by an embodiment of the present application. As shown in FIG. 10 , the data transmission apparatus may include: at least one processor 701 and a memory 702 . FIG. 10 shows an electronic device with a processor as an example.

The memory 702 is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions.

Memory 702 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

The processor 701 is configured to execute the computer-executed instructions stored in the memory 702, so as to realize the above-mentioned data cleaning method on the terminal device side;

Alternatively, the processor 701 is configured to execute computer-executed instructions stored in the memory 702 to implement the above-mentioned server-side data cleaning method;

The processor 701 may be a central processing unit (Central Processing Unit, referred to as CPU), or a specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC), or is configured to implement one or more of the embodiments of the present application. multiple integrated circuits.

Optionally, in terms of specific implementation, if the communication interface, the memory 702 and the processor 701 are implemented independently, the communication interface, the memory 702 and the processor 701 can be connected to each other through a bus and complete mutual communication. The bus may be an Industry Standard Architecture (ISA for short) bus, a Peripheral Component (PCI for short) bus or an Extended Industry Standard Architecture (EISA for short) bus or the like. Buses can be divided into address bus, data bus, control bus, etc., but it does not mean that there is only one bus or one type of bus.

Optionally, in terms of specific implementation, if the communication interface, the memory 702 and the processor 701 are integrated on one chip, the communication interface, the memory 702 and the processor 701 can communicate through an internal interface.

The present application also provides a program product including execution instructions stored in a readable storage medium. At least one processor of the data cleaning apparatus can read the execution instruction from the readable storage medium, and the at least one processor executes the execution instruction to cause the data cleaning apparatus to implement the data cleaning methods provided in the above-mentioned various embodiments.

The present invention also provides a computer-readable storage medium, the computer-readable storage medium may include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) Various media that can store program codes, such as a magnetic disk, a magnetic disk, or an optical disk, specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the foregoing embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (10)

1. a data cleaning method, applied to terminal equipment, is characterized in that, described method comprises: Acquire the detection data of the detection equipment in the first collection period; According to the threshold range of the detection data, data cleaning is performed on the detection data, and the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server; If there is available data after data cleaning, the available data is reported to the server in the first reporting cycle, so that the server generates a first range of the detection data according to the available data, and the first range is used for The threshold value of the threshold value range of the detection data is updated. 2. The method according to claim 1, wherein after the data cleaning is performed on the detection data, the method further comprises: If there is no available data after data cleaning, linear interpolation is performed on the historically available data to predict the available data in the first collection period; The predicted available data is reported to the server in the first reporting period. 3. The method according to claim 2, characterized in that, after reporting the available data to the server in the first reporting period, the method further comprises: receiving a first range of the detection data sent by the server, where the first range is determined according to available data in the first collection period; According to the first range, the threshold range of the detection data is updated. 4. A data cleaning method, applied to a server, wherein the method comprises: Receive the available data reported by the terminal device in the first reporting cycle, where the available data is obtained after the terminal device performs data cleaning on the detection data of the detection device in the first collection cycle according to the threshold range of the detection data, and the Data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server; In the first update period, the first range of the detection data is generated according to the available data reported by the terminal device in the first reporting period; Sending the first range of detection data to the terminal device. 5. The method according to claim 4, wherein before the sending the first update value of the threshold range of the detection data to the terminal device, the method further comprises: receiving the available data reported by the terminal device in a second reporting period, the second reporting period being later than the first reporting period; The sending the first range of the detection data to the terminal device includes: If the available data reported in the second reporting period is not within the first range, the first range is sent to the terminal device. 6. The method according to claim 4, wherein before the sending the first range of the detection data to the terminal device, the method further comprises: Obtain the unupdated duration of the threshold range of the detection data; The sending the first range of the detection data to the terminal device includes: If the unupdated duration exceeds the duration threshold, the first range of the detection data is sent to the terminal device. 7. A data cleaning device, characterized in that the device comprises: a receiving module, a processing module and a sending module; The receiving module is used to obtain the detection data of the detection device in the first collection period; The processing module is configured to perform data cleaning on the detection data according to the threshold range of the detection data, the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is determined by the server instruct; The sending module is configured to report the available data to the server in the first reporting period if there is available data after data cleaning. 8. A data cleaning device, characterized in that the device comprises: a receiving module, a processing module and a sending module; The receiving module is configured to receive the available data reported by the terminal device in the first reporting period, and the available data is that the terminal device performs data analysis on the detection data of the detection device in the first collection period according to the threshold range of the detection data. data obtained after cleaning, the data cleaning is used to discard invalid values in the detection data, and the threshold range of the detection data is indicated by the server; The processing module is configured to, in the first update period, generate the first range of the detection data according to the available data reported by the terminal device in the first reporting period; The sending module is configured to send the first range of the detection data to the terminal device. 9. A storage medium on which a computer program is stored, characterized in that, comprising: when the program is executed by a processor, the method according to any one of claims 1-3 is implemented. 10. A storage medium on which a computer program is stored, characterized in that, comprising: when the program is executed by a processor, the method according to any one of claims 4-6 is implemented.

Images