CN115145902A - Data processing method, data processing apparatus, storage medium, and electronic device - Google Patents
Data processing method, data processing apparatus, storage medium, and electronic device Download PDFInfo
- Publication number
- CN115145902A CN115145902A CN202210780052.XA CN202210780052A CN115145902A CN 115145902 A CN115145902 A CN 115145902A CN 202210780052 A CN202210780052 A CN 202210780052A CN 115145902 A CN115145902 A CN 115145902A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- cleaning rule
- value
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/08—Construction
Abstract
The application discloses a data processing method, a data processing device, a storage medium and electronic equipment. Wherein, the method comprises the following steps: collecting target data reported by target equipment; calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule; determining the attribute type of the target data according to the target cleaning rule; different data processing strategies are executed according to the attribute types. The method and the device solve the technical problems that in the related technology, only data can be stored based on a third-party IOT monitoring tool, accurate data cannot be acquired due to the fact that wrong data cannot be rejected, follow-up reasonable and timely early warning cannot be conducted, and prediction analysis cannot be conducted accurately.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data processing method, apparatus, storage medium, and electronic device.
Background
The building intellectualization is a building which takes a building as a platform, integrates architecture, system, application, management and optimization combination into a whole based on the comprehensive application of various intelligent information, has comprehensive intelligent capabilities of perception, transmission, memory, reasoning, judgment and decision, forms an integration body in which people, the building and the environment are mutually coordinated, and provides a safe, efficient, convenient and sustainable development functional environment for people.
However, in the building intellectualization construction, a large number of IoT devices and gateways are required to be used for data collection depending on data support, and once a gateway or an IoT device fails, even a network fails, data loss or wrong data is generated, reliability of the data is lost, and an erroneous decision is easily made. The existing data processing technology is mainly a third party IoT monitoring tool and platform and is stored correspondingly. The method can only realize simple data storage, and can not remove wrong data or supplement the data, which is unreasonable. Therefore, accurate data cannot be acquired in the intelligent process, the fault early warning guidance function cannot be achieved, reasonable and timely early warning cannot be performed, whether a fault occurs can be found only when the fault occurs, prediction analysis is difficult to perform, and intelligent building decision making is not facilitated.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing device, a storage medium and electronic equipment, and aims to at least solve the technical problems that in the related technology, data can only be stored based on a third-party IOT monitoring tool, accurate data cannot be acquired due to the fact that wrong data cannot be rejected, follow-up reasonable and timely early warning cannot be performed, and prediction analysis is difficult to accurately perform.
According to an aspect of an embodiment of the present application, there is provided a data processing method, including: collecting target data reported by target equipment; calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule; determining the attribute type of the target data according to the target cleaning rule; different data processing strategies are executed according to the attribute types.
Optionally, the acquiring target data reported by the target device includes: detecting target acquisition time for acquiring target data reported by target equipment; acquiring a time length difference value between target acquisition time and last acquisition time; and executing different storage operations on the target data according to the time length difference.
Optionally, different storage operations are performed on the target data according to the time length difference, including: determining that the target data is expired and rejecting the target data under the condition that the time length difference is smaller than a first preset threshold; under the condition that the time length difference value is greater than a first preset threshold value and equal to a second preset threshold value, storing historical data corresponding to the last acquisition time; and under the condition that the time length difference value is larger than a second preset threshold value, storing historical data corresponding to the last acquisition time and target data, and triggering a completion data notification.
Optionally, the completion data is implemented as follows: respectively determining a first target value corresponding to target acquisition time, a second target value corresponding to last acquisition time and a data period to be supplemented; determining the number of points to be supplemented according to the time length difference and the data period to be supplemented; and obtaining a difference value to be compensated according to the first target value and the second target value, and determining the increment of the points to be compensated according to the difference value to be compensated and the number of the points to be compensated.
Optionally, determining the number of points to be supplemented according to the time length difference and the data cycle to be supplemented includes: and determining a first ratio of the time length difference value to the data period to be supplemented, and determining a difference value of the first ratio and a first preset value to obtain the number of points to be supplemented.
Optionally, determining an increment of the point to be compensated according to the difference to be compensated and the number of the point to be compensated includes: adding the number of the points to be compensated and a second preset value to obtain a sum of the number of the points to be compensated and the second preset value; and determining the ratio of the difference value to be compensated to the sum value to obtain the increment of the point to be compensated.
Optionally, different data processing policies are executed according to the attribute type, including: and under the condition that the attribute type is increment or fluctuation, checking whether the data value range in the target data is within a preset range to obtain a first checking result, and determining that the data cleaning is not passed and rejecting the target data if the first checking result indicates that the data value range in the target data is not within the preset range.
Optionally, different data processing policies are executed according to the attribute type, including: and under the condition that the attribute type is enumeration, checking whether the data in the target data is in an enumeration value list or not to obtain a second checking result, and determining that the data cannot be cleaned and rejecting the target data if the second checking result indicates that the data is not in the enumeration value list.
According to another aspect of the embodiments of the present application, there is also provided a data processing apparatus, including: the acquisition module is used for acquiring target data reported by the target equipment; the calling module is used for calling data cleaning rule mapping and acquiring a target cleaning rule corresponding to target data according to the identification of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identification and the cleaning rule; the determining module is used for determining the attribute type of the target data according to the target cleaning rule; and the execution module is used for executing different data processing strategies according to the attribute types.
According to another aspect of the embodiments of the present application, there is also provided a non-volatile storage medium, where the storage medium includes a stored program, and where the program is executed to control a device in which the storage medium is located to execute any one of the data processing methods.
According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the data processing methods.
In the embodiment of the application, a cleaning mode based on a data cleaning rule is adopted, and target data reported by target equipment are collected; calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule; determining the attribute type of the target data according to the target cleaning rule; different data processing strategies are executed according to the attribute types, the purpose of filtering and cleaning data is achieved, and therefore the technical effects of eliminating error data, improving the accuracy of data prediction and analysis and accurately and timely alarming are achieved, and the technical problems that in the related technology, only data can be stored based on a third-party IOT monitoring tool, accurate data cannot be obtained due to the fact that the error data cannot be eliminated, follow-up reasonable and timely early warning cannot be conducted, and prediction and analysis are difficult to accurately conduct are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a schematic flow chart diagram of an alternative data processing method according to an embodiment of the present application;
FIG. 2 is a block diagram of an alternative data processing apparatus according to an embodiment of the present application;
FIG. 3 shows a schematic block diagram of an example electronic device 300 that may be used to implement embodiments of the present application.
Detailed Description
In order to make the technical solutions of the present application better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the accompanying drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
To facilitate understanding of embodiments related to the present application by those skilled in the art, technical terms or partial terms that may be referred to in the present application are explained as follows:
the Internet of things (IoT) refers to the billions of physical devices that are not normally expected to have an Internet connection, are now connected to the Internet (and/or each other), collect and share data. Almost all physical objects you can think can be converted into IoT devices, such as coffee machines, washing machines, headsets, lights, wearable devices and machine components. The internet of things describes a world that can be connected and communicate in an almost intelligent manner. With the internet of things, the physical world is becoming a large information system.
In accordance with an embodiment of the present application, there is provided an embodiment of a data processing method, it should be noted that the steps shown in the flowchart of the figure may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in an order different than here.
Fig. 1 is a data processing method according to an embodiment of the present application, as shown in fig. 1, the method including the steps of:
step S102, collecting target data reported by target equipment;
step S104, calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule, and the identifier can be the code of the equipment;
step S106, determining the attribute type of the target data according to the target cleaning rule;
and step S108, executing different data processing strategies according to the attribute types.
In the data processing method, target data reported by target equipment are collected; calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule; determining the attribute type of the target data according to the target cleaning rule; different data processing strategies are executed according to attribute types, the purpose of filtering and cleaning data is achieved, and therefore the technical effects of eliminating error data, improving the accuracy of data prediction and analysis and accurately and timely alarming are achieved, and the technical problems that in the related technology, only data can be stored based on a third-party IOT monitoring tool, accurate data cannot be obtained due to the fact that the error data cannot be eliminated, follow-up reasonable and timely early warning cannot be carried out, and prediction and analysis are difficult to accurately carry out are solved.
In some embodiments of the present application, acquiring target data reported by a target device includes: detecting target acquisition time for acquiring target data reported by target equipment; acquiring a time length difference value between target acquisition time and last acquisition time; different storage operations are executed on the target data according to the time length difference, and it is easy to notice that the data normalization processing can be completed through the technical scheme.
Specifically, different storage operations are performed on the target data according to the time difference, and the method can be implemented by the following steps: determining that the target data is overdue and rejecting the target data under the condition that the time length difference value is smaller than (including equal to) a first preset threshold value; under the condition that the time length difference value is larger than a first preset threshold value and equal to a second preset threshold value, storing historical data corresponding to the last acquisition time; and under the condition that the time length difference value is greater than a second preset threshold value, storing historical data corresponding to the last acquisition time and target data, and triggering a completion data notification, wherein the first preset threshold value is smaller than the second preset threshold value, for example, the first preset threshold value may be 0 minute, and the second preset threshold value may be 5 minutes.
The data normalization process will be described with reference to a specific embodiment.
First, data initialization may be performed, calculating a corresponding 5 MINUTE time period Map per MINUTE, and constructing a time formatting Map FORMAT _ field _ MINUTE _ Map with a data structure Map < current MINUTEs, 5 MINUTE period MINUTEs >. Secondly, the current time may be formatted according to FORMAT _ live _ MINUTE _ MAP, and the difference between the number of MINUTEs of the current acquisition point time and the last acquisition point time, diffminimum, may be calculated.
If the last acquisition point time is empty, the current data point is stored and an IoT completion data notification is sent. If diffMinute is less than 0, the current data is invalid and does not participate in the subsequent processing. If diffMinute is equal to 0, it indicates that the current data point and the previous data point are in the same 5-minute period and do not participate in the subsequent processing. If diffMinute is greater than 0 and equal to 5, indicating that the current data time point has elapsed a 5 minute period, the last data point is stored. It can be understood that if diffminite is greater than 0 and greater than 5, indicating that the current data time point has passed the 10 minute period, the last data point and the current data point are stored, and an IoT complementary data notification, an IoT data breakpoint alert, is sent. In addition, storage strategies of 5 minutes, 15 minutes, 1 hour, 1 day and the like can be calculated according to the current data time point, and the data are stored in the database. Finally, the data point can be updated, and the previous data point is updated to be the current data point.
It should be noted that, in the embodiment related to the present application, the completion data is implemented as follows: respectively determining a first target value corresponding to target acquisition time, a second target value corresponding to last acquisition time and a data period to be supplemented; determining the number of points to be supplemented according to the time length difference and the data period to be supplemented; and obtaining a difference value to be compensated according to the first target value and the second target value, and determining the increment of the points to be compensated according to the difference value to be compensated and the number of the points to be compensated.
Optionally, determining the number of points to be complemented according to the time length difference and the data cycle to be complemented includes: and determining a first ratio of the time length difference value to the data period to be supplemented, and determining a difference value of the first ratio and a first preset value to obtain the number of points to be supplemented.
In some optional embodiments of the present application, determining an increment of a point to be compensated according to the difference to be compensated and the number of the point to be compensated includes: adding the number of the points to be compensated and a second preset value to obtain a sum of the number of the points to be compensated and the second preset value; and determining the ratio of the difference value to be compensated to the sum value to obtain the increment of the point to be compensated.
The process of completing data processing is described with reference to an embodiment, and specifically, it may be defined that the value of the current collection point is currentValue, the time of the current collection point is currentTime, the value of the previous collection point is latestValue, the time of the previous collection point is latestTime, and the period of completing data is fillPeriod.
Calculate the value of the current acquisition point minus the value diffValue of the last acquisition point (i.e., the completion difference) = currentValue-latestValue).
Calculating the time of the current acquisition point minus the time diffTime (instant length difference) = currentTime-latestTime of the last acquisition point.
And calculating the number of completion points diffSize = diffTime/fillPeriod-1.
The completion point increment fillValue = diffValue/(diffSize + 1) is calculated.
Result data:
1 st completion point: completion point time fillTime = latestTime +1 fillperiod;
the completion point data fillData = latestData +1 fillvue.
Second completion point: completion point time fillTime = latestTime +2 fillperiod;
the completion point data fillData = latestData +2 fillvue.
Nth completion point: completion point time fillTime = latestTime + n × fillPeriod;
the completion point data fillData = latestData + n × fillValue.
In some embodiments of the present application, the performing different data processing policies according to the attribute type includes: and under the condition that the attribute type is increment or fluctuation, checking whether the data value range in the target data is within a preset range to obtain a first checking result, and determining that the data cleaning is not passed and rejecting the target data if the first checking result indicates that the data value range in the target data is not within the preset range.
In other alternative embodiments of the present application, the performing different data processing policies according to attribute types includes: and under the condition that the attribute type is enumeration, checking whether the data in the target data is in an enumeration value list or not to obtain a second checking result, and determining that the data cannot be cleaned and rejecting the target data if the second checking result indicates that the data is not in the enumeration value list.
With reference to a specific embodiment, to describe the performing of different data processing policies according to attribute types, the target device may be an IoT device, which is specifically: the method can be initialized firstly, acquires all IoT equipment and point location attributes configured by the system, and constructs a data structure as Map < IoT equipment code, map < IoT attribute code and CLEANING rule > > CLEANING rule mapping CLEANING _ INFO _ MAP, wherein the CLEANING rule comprises information such as IoT attribute data type, maximum allowable value, minimum allowable value and enumeration value list.
In this embodiment, the processing procedure may be to receive IoT reported data, obtain an IoT cleansing rule according to the clearingjnfo _ MAP, and obtain an IoT attribute data type according to the IoT cleansing rule.
It will be appreciated that if the data type is cumulative (incremental) or fluctuating, it is checked whether the current value is within the minimum and maximum allowable values. If the data type is enumeration, then it is checked whether the current value is in the list of enumeration values. Finally, if the cleansing process does not pass, an IoT data exception alert may be sent and the data does not participate in subsequent processing.
It can be understood that three data exception processing modes of increment, fluctuation and enumeration can be used in the method, error data can be removed, accurate data can be obtained, and compared with an original simple data storage mode, the method is beneficial to guiding the fault early warning and is more beneficial to making intelligent building decisions. And various data normalization processing modes such as 5 minutes, 15 minutes, 1 hour, 1 day and the like can be set, and the data can be complemented by combining a related algorithm, so that continuous and normalized data can be obtained.
Fig. 2 is a data processing apparatus according to an embodiment of the present application, and as shown in fig. 2, the apparatus includes:
the acquisition module 40 is used for acquiring target data reported by the target equipment;
a calling module 42, configured to call a data cleansing rule mapping, and obtain a target cleansing rule corresponding to target data according to an identifier of a target device and the data cleansing rule mapping, where the data cleansing rule mapping is at least used to indicate a mapping relationship between a device identifier and a cleansing rule;
a determining module 44, configured to determine an attribute type of the target data according to the target cleaning rule;
and the execution module 46 is used for executing different data processing strategies according to the attribute types.
In the data processing device, an acquisition module 40 is used for acquiring target data reported by target equipment; a calling module 42, configured to call a data cleansing rule mapping, and obtain a target cleansing rule corresponding to target data according to an identifier of a target device and the data cleansing rule mapping, where the data cleansing rule mapping is at least used to indicate a mapping relationship between a device identifier and a cleansing rule; a determining module 44, configured to determine an attribute type of the target data according to the target cleaning rule; the execution module 46 is used for executing different data processing strategies according to the attribute types, and achieving the purpose of filtering and cleaning data, so that the error data can be rejected, the accuracy of data prediction analysis is improved, the technical effects of accurate and timely alarming are achieved, and the technical problems that in the related technology, only data can be stored based on a third-party IOT monitoring tool, accurate data cannot be acquired due to the fact that the error data cannot be rejected, follow-up reasonable and timely early warning cannot be performed, and accurate prediction analysis is difficult to perform are solved.
According to another aspect of the embodiments of the present application, there is also provided a non-volatile storage medium, where the storage medium includes a stored program, and when the program runs, a device in which the storage medium is located is controlled to execute any one of the data processing methods.
Specifically, the storage medium is used for storing program instructions of the following functions, and the following functions are realized:
collecting target data reported by target equipment; calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule; determining the attribute type of the target data according to the target cleaning rule; different data processing strategies are executed according to the attribute types.
Alternatively, in the present embodiment, the storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the aforementioned storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the aforementioned.
In an exemplary embodiment of the application, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the data processing method of any of the above.
Optionally, the computer program may, when executed by a processor, implement the steps of:
collecting target data reported by target equipment; calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule; determining the attribute type of the target data according to the target cleaning rule; different data processing strategies are executed according to the attribute types.
An embodiment according to the present application provides an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the data processing methods described above.
Optionally, the electronic device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
FIG. 3 illustrates a schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 3, the apparatus 300 comprises a computing unit 301 which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 302 or a computer program loaded from a storage unit 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the device 300 can also be stored. The calculation unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
Various components in device 300 are connected to I/O interface 305, including: an input unit 306 such as a keyboard, a mouse, or the like; an output unit 307 such as various types of displays, speakers, and the like; a storage unit 308 such as a magnetic disk, optical disk, or the like; and a communication unit 309 such as a network card, modem, wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 301 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 301 executes the respective methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded onto and/or installed onto device 300 via ROM 302 and/or communications unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions/acts specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.
In the related embodiment of the application, a cleaning mode based on a data cleaning rule is adopted, and target data reported by target equipment are collected; calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule; determining the attribute type of the target data according to the target cleaning rule; different data processing strategies are executed according to the attribute types, the purpose of filtering and cleaning data is achieved, and therefore the technical effects of eliminating error data, improving the accuracy of data prediction and analysis and accurately and timely alarming are achieved, and the technical problems that in the related technology, only data can be stored based on a third-party IOT monitoring tool, accurate data cannot be obtained due to the fact that the error data cannot be eliminated, follow-up reasonable and timely early warning cannot be conducted, and prediction and analysis are difficult to accurately conduct are solved.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be an indirect coupling or communication connection through some interfaces, units or modules, and may be electrical or in other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (11)
1. A method of data processing, comprising:
collecting target data reported by target equipment;
calling data cleaning rule mapping, and acquiring a target cleaning rule corresponding to the target data according to the identifier of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identifier and the cleaning rule;
determining the attribute type of the target data according to a target cleaning rule;
and executing different data processing strategies according to the attribute types.
2. The method of claim 1, wherein collecting target data reported by a target device comprises:
detecting target acquisition time for acquiring the target data reported by the target equipment;
acquiring a time length difference value between the target acquisition time and the last acquisition time;
and executing different storage operations on the target data according to the duration difference.
3. The method of claim 2, wherein performing different storage operations on the target data according to the duration difference comprises:
determining that the target data is expired and rejecting the target data under the condition that the time length difference is smaller than a first preset threshold;
under the condition that the time length difference value is larger than a first preset threshold and equal to a second preset threshold, storing historical data corresponding to the last acquisition time;
and under the condition that the time length difference value is larger than the second preset threshold value, storing historical data corresponding to the last acquisition time and the target data, and triggering a completion data notification.
4. The method of claim 2, wherein the completion data is implemented by:
respectively determining a first target value corresponding to the target acquisition time, a second target value corresponding to the last acquisition time and a data period to be supplemented;
determining the number of points to be supplemented according to the time length difference and the data period to be supplemented;
and obtaining a difference value to be compensated according to the first target value and the second target value, and determining the increment of the points to be compensated according to the difference value to be compensated and the number of the points to be compensated.
5. The method of claim 4, wherein determining the number of points to be complemented according to the duration difference and the data period to be complemented comprises:
and determining a first ratio of the time length difference value to the data period to be supplemented, and determining a difference value of the first ratio and a first preset value to obtain the number of the points to be supplemented.
6. The method of claim 4, wherein determining the number of points to be compensated based on the difference to be compensated and the number of points to be compensated comprises:
adding the number of the points to be compensated and a second preset value to obtain a sum of the number of the points to be compensated and the second preset value;
and determining the ratio of the difference value to be compensated to the sum value to obtain the point increment to be compensated.
7. The method of claim 1, wherein implementing different data processing policies based on the attribute type comprises:
and if the attribute type is increment or fluctuation, checking whether the data value range in the target data is within a preset range to obtain a first checking result, and if the first checking result indicates that the data value range in the target data is not within the preset range, determining that the data is not cleaned and rejecting the target data.
8. The method of claim 1, wherein implementing different data processing policies based on the attribute type comprises:
and if the attribute type is enumeration, checking whether the data in the target data is in an enumeration value list to obtain a second checking result, and if the second checking result indicates that the data is not in the enumeration value list, determining that the data is not cleaned and rejecting the target data.
9. A data processing apparatus, comprising:
the acquisition module is used for acquiring target data reported by the target equipment;
the calling module is used for calling data cleaning rule mapping and acquiring a target cleaning rule corresponding to the target data according to the identification of the target equipment and the data cleaning rule mapping, wherein the data cleaning rule mapping is at least used for indicating the mapping relation between the equipment identification and the cleaning rule;
the determining module is used for determining the attribute type of the target data according to a target cleaning rule;
and the execution module is used for executing different data processing strategies according to the attribute types.
10. A non-volatile storage medium, characterized in that the storage medium includes a stored program, wherein when the program runs, the device where the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 8.
11. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the data processing method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210780052.XA CN115145902A (en) | 2022-07-04 | 2022-07-04 | Data processing method, data processing apparatus, storage medium, and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210780052.XA CN115145902A (en) | 2022-07-04 | 2022-07-04 | Data processing method, data processing apparatus, storage medium, and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115145902A true CN115145902A (en) | 2022-10-04 |
Family
ID=83410404
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210780052.XA Pending CN115145902A (en) | 2022-07-04 | 2022-07-04 | Data processing method, data processing apparatus, storage medium, and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115145902A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116069774A (en) * | 2023-04-03 | 2023-05-05 | 北京全路通信信号研究设计院集团有限公司 | Data cleaning method, device and medium based on wireless timeout intelligent analysis |
-
2022
- 2022-07-04 CN CN202210780052.XA patent/CN115145902A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116069774A (en) * | 2023-04-03 | 2023-05-05 | 北京全路通信信号研究设计院集团有限公司 | Data cleaning method, device and medium based on wireless timeout intelligent analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116049146B (en) | Database fault processing method, device, equipment and storage medium | |
| CN112615742A (en) | Method, device, equipment and storage medium for early warning | |
| CN112532435B (en) | Operation and maintenance method, operation and maintenance management platform, equipment and medium | |
| CN115145902A (en) | Data processing method, data processing apparatus, storage medium, and electronic device | |
| CN115904656A (en) | State control method, device and equipment applied to chip and storage medium | |
| CN115425760A (en) | Digital power grid system based on digital twinning | |
| CN115951123B (en) | Electric energy metering method and system based on wireless communication | |
| CN107525955A (en) | A kind of ammeter data processing method and processing device | |
| CN110750425A (en) | Database monitoring method, device and system and storage medium | |
| CN115687406A (en) | Sampling method, device and equipment of call chain data and storage medium | |
| CN115459434A (en) | Intelligent electricity consumption monitoring method and system for industrial enterprise | |
| CN114327392A (en) | Data processing method and device and electronic equipment | |
| CN114885014A (en) | Method, device, equipment and medium for monitoring external field equipment state | |
| CN114329238A (en) | Data processing method, device, equipment and storage medium | |
| CN110211678A (en) | A kind of data monitoring method, device, equipment and storage medium | |
| CN115190045B (en) | Monitoring method and device for express cabinet system service, electronic equipment and storage medium | |
| CN116302370A (en) | Method, device, equipment and medium for generating return code abnormality alarm | |
| CN114844920A (en) | Internet of things equipment checking method and device, electronic equipment and storage medium | |
| CN116743508B (en) | Method, device, equipment and medium for detecting network attack chain of power system | |
| CN117687816A (en) | Service reliability evaluation method, device, equipment and storage medium | |
| CN115102850B (en) | Configuration comparison method, device, electronic equipment and storage medium | |
| CN117608904A (en) | Fault positioning method and device, electronic equipment and storage medium | |
| CN117749614A (en) | Protocol rule determining method and device, electronic equipment and storage medium | |
| CN115204424A (en) | Intelligent operation and maintenance method, device, equipment and storage medium of cloud native system | |
| CN117608905A (en) | Alarm root cause positioning method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |