WO2022127259A1

WO2022127259A1 - Data cleaning method, apparatus and device, and storage medium

Info

Publication number: WO2022127259A1
Application number: PCT/CN2021/120043
Authority: WO
Inventors: 孟宪奎; 程强; 万月亮
Original assignee: 北京锐安科技有限公司
Priority date: 2020-12-16
Filing date: 2021-09-24
Publication date: 2022-06-23
Also published as: CN112612761A; CN112612761B

Abstract

A data cleaning method, apparatus and device, and a storage medium. The data cleaning method comprises: obtaining data to be cleaned, and determining a target data extractor corresponding to the data (110); parsing the data, and extracting, by means of the target data extractor, target data comprised in the data (120), the target data comprising at least one of an attribute name, attribute data, or label text data; and decoding the target data, and screening the decoded target data according to multiple pieces of reference data uploaded by a client, so as to clean the data (130).

Description

Data cleaning method, device, equipment and storage medium

This application claims the priority of the Chinese Patent Application No. 202011490975.9 filed with the China Patent Office on December 16, 2020, the entire contents of which are incorporated herein by reference.

technical field

The embodiments of the present application relate to the technical field of data cleaning, for example, to a data cleaning method, apparatus, device, and storage medium.

Background technique

With the continuous development of the Internet, a variety of data have shown explosive growth; at the same time, redundant data that do not meet the specifications are also increasing, especially the post data in the Internet, which contains a lot of redundant data. The overall value of data cannot be effectively reflected without structured information.

In the related art, for post data, the post data is usually forwarded directly, or the post data is word-segmented and stored, which not only consumes a lot of storage space, but also requires redundant data in the post data during the data transmission process. Forward.

Therefore, it is necessary to study a solution to clean redundant data in post data to save storage space and improve data transmission efficiency.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a data cleaning method, apparatus, device, and storage medium, so as to clean redundant data in data, save storage space, and improve data transmission efficiency.

In a first aspect, an embodiment of the present application provides a data cleaning method, including:

acquiring data to be cleaned, and determining a target data extractor corresponding to the data to be cleaned;

Parsing the data to be cleaned, and extracting target data contained in the data to be cleaned by the target data extractor, where the target data includes at least one of attribute name, attribute data or label text data;

The target data is decoded, and the decoded target data is screened according to multiple reference data uploaded by the client, so as to clean the data to be cleaned.

In a second aspect, an embodiment of the present application further provides a data cleaning device, including:

a data extractor determination module, configured to acquire the data to be cleaned, and determine a target data extractor corresponding to the data to be cleaned;

A target data extraction module, configured to parse the data to be cleaned, and extract target data contained in the data to be cleaned through the target data extractor, where the target data includes attribute names, attribute data or label text data at least one;

The target data screening module is configured to decode the target data, and screen the decoded target data according to a plurality of reference data uploaded by the client, so as to clean the data to be cleaned.

In a third aspect, an embodiment of the present application further provides a data cleaning device, where the data cleaning device includes:

one or more processors;

storage means arranged to store one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the data cleaning method according to any one of the embodiments of this application.

In a fourth aspect, the embodiments of the present application further provide a storage medium containing computer-executable instructions, where the computer-executable instructions, when executed by a computer processor, are used to execute any one of the embodiments of the present application. data cleaning method.

Description of drawings

1 is a flowchart of a data cleaning method in an embodiment of the present application;

2 is a flowchart of a data cleaning method in another embodiment of the present application;

3 is a flowchart of a data cleaning method in another embodiment of the present application;

4 is a flowchart of a data cleaning method in another embodiment of the present application;

5 is a flowchart of a data cleaning method in another embodiment of the present application;

Fig. 6 is the composition diagram of a kind of post data in the embodiment of the present application;

7 is a schematic structural diagram of a data cleaning apparatus in an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a data cleaning device in an embodiment of the present application.

Detailed ways

The embodiments of the present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that, the specific embodiments described herein are only used to explain the embodiments of the present application, but are not intended to limit the embodiments of the present application. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all of the structures related to the embodiments of the present application.

FIG. 1 is a flowchart of a data cleaning method in an embodiment of the present application. This embodiment is applicable to the case of filtering out redundant data in data. The method can be executed by a data cleaning device, and the device can It is implemented by means of software and/or hardware, and is integrated in the data cleaning device that executes the method. In this embodiment, the data cleaning device may be a server, a computer, or a tablet computer, etc., which is not added in this embodiment. limited. Referring to Figure 1, the method includes the following steps:

Step 110: Acquire the data to be cleaned, and determine a target data extractor corresponding to the data to be cleaned.

In this embodiment, the data to be cleaned may be post data or get data, which is not limited in this embodiment.

In an example implementation of this embodiment, after the data to be cleaned is acquired, a target data extractor corresponding to the data to be cleaned may be determined. It should be noted that the post data can be composed of three parts: request url (uniform resource locator, uniform resource locator), request cookie and request body. In the embodiment, these three parts of post data all need to be processed by target data. extract. Among them, url and cookie are generally key-value pair data, and the body can contain xml (extensible markup language, extensible markup language), JSON (JavaScript Object Notation, JS object notation) and key-value pair data in three data formats. .

In an example implementation of this embodiment, after the data to be cleaned is obtained, that is, after the post data to be cleaned is obtained, the data format type contained in the post data to be cleaned can be identified, and the data format type contained in the post data can be identified according to the data format of the post data. The type selects the target data extractor to extract the target data contained in the post data.

Exemplarily, if it is identified that the post data to be cleaned contains only key-value pair data, a target data extractor that matches the key-value pair data can be selected to extract the key-value pair data contained in the post data; If the post data to be cleaned only contains xml data, the target data extractor that matches the xml data can be selected to extract the xml data contained in the post data; if it is identified that the post data to be cleaned contains key-value pair data and xml data , the target data extractor that matches the key-value pair data and the target data extractor that matches the xml data can be selected to extract the key-value pair data and the xml data contained in the post data.

Step 120: Analyze the data to be cleaned, and extract the target data contained in the data to be cleaned through a target data extractor.

The target data includes at least one item of attribute name, attribute data or label text data.

In an example implementation of this embodiment, after the target data extractor corresponding to the data to be cleaned is determined, the data to be cleaned may be parsed to determine the encoding mode of the data to be cleaned. In this embodiment, the data to be cleaned The encoding mode may be a base64 encoding mode, a decoder encoding mode, or an encryption encoding mode, which is not limited in this embodiment. For example, the target data contained in the data to be cleaned can be extracted by the selected target data extractor.

Step 130: Decode the target data, and filter the decoded target data according to multiple reference data uploaded by the client to clean the data to be cleaned.

In an example implementation of this embodiment, after the target data included in the data to be cleaned is extracted by the target data extractor, the extracted target data can be decoded according to the parsed encoding mode of the data to be cleaned, The decoded target data is screened according to multiple reference data uploaded by the client, so as to clean the data to be cleaned and filter out redundant data contained in the data to be cleaned.

Wherein, the plurality of reference data uploaded by the client may be data related to user requirements, for example, keywords related to user requirements, etc., which are not limited in this embodiment.

Exemplarily, after the target data is decoded to obtain target data that is easy to identify, the target data can be compared with the reference data uploaded by the client, the target data corresponding to the reference data can be retained, and the target data that is different from the reference data can be filtered out. Corresponding target data, so as to filter the target data to filter out redundant data that is not related to user needs.

In the solution of this embodiment, the data to be cleaned is obtained and the target data extractor corresponding to the data to be cleaned is determined; the data to be cleaned is parsed, and the target data contained in the data to be cleaned is extracted through the target data extractor, and the target data includes attributes At least one of name, attribute data, or label text data; decode the target data, and filter the decoded target data according to multiple reference data uploaded by the client to clean the data to be cleaned, realizing the The redundant data is cleaned, which can save storage space and improve data transmission efficiency.

FIG. 2 is a flowchart of a data cleaning method in another embodiment of the present application. This embodiment is a refinement of the above technical solution. The technical solution in this embodiment may be the same as that in one or more of the above embodiments. Example scenarios combined. As shown in Figure 2, the data cleaning method may include the following steps:

Step 210: Acquire data to be cleaned.

Step 220: Identify the data format type contained in the data to be cleaned, and determine the target data extractor according to the data format type.

The data format types include: key-value pair, xml, JSON, etc., which are not limited in this embodiment.

In an example implementation of this embodiment, after the data to be cleaned is obtained, that is, after the post data to be cleaned is obtained, the data format type contained in the post data to be cleaned can be identified, and the data contained in the post data can be identified. The format type may be one or more of key-value pair, xml, and JSON data format types, which are not limited in this embodiment.

The target data extractor may be determined according to the data format type contained in the identified data to be cleaned, where the target data extractor may include a key-value pair extractor, an xml extractor, or a JSON extractor.

Exemplarily, if it is identified that the data format types included in the data to be cleaned are key-value pair and xml data format types, the key-value pair extractor and the xml extractor can be determined as the target data extractor.

Step 230: Analyze the data to be cleaned, and extract the target data contained in the data to be cleaned by using a target data extractor.

In an example implementation of this embodiment, extracting the target data included in the data to be cleaned by the target data extractor may include: extracting key-value pairs, xml data or JSON information contained in the data to be cleaned by the target data extractor.

Extracting the key-value pairs, xml data or JSON information contained in the data to be cleaned by the target data extractor may include: extracting the key-value pairs contained in the data to be cleaned by the key-value pair extractor; or, extracting the data to be cleaned by using the xml extractor The xml data contained in the cleaning data; or, the JSON information contained in the data to be cleaned is extracted through a JSON extractor.

In an example implementation of this embodiment, the key-value pairs contained in the data to be cleaned can also be extracted by the key-value pair extractor, the xml data contained in the data to be cleaned can be extracted by the xml extractor, and the JSON extractor can be used to extract the xml data contained in the data to be cleaned at the same time. Extract the JSON information contained in the data to be cleaned.

Step 240: Decode the target data, and filter the decoded target data according to multiple reference data uploaded by the client to clean the data to be cleaned.

In the solution of this embodiment, by identifying the data format type contained in the data to be cleaned, and determining a target data extractor according to the data format type; extracting key-value pairs, xml data or JSON information contained in the data to be cleaned by the target data extractor, Provide a basis for subsequent cleaning of redundant data in post data.

FIG. 3 is a flowchart of a data cleaning method in another embodiment of the present application. This embodiment is a refinement of the above technical solution. The technical solution in this embodiment may be the same as that in one or more of the above embodiments. Example scenarios combined. As shown in Figure 3, the data cleaning method may include the following steps:

Step 310: Acquire data to be cleaned.

Step 320: Identify the data format type contained in the data to be cleaned, and determine the target data extractor according to the data format type.

Step 330: Analyze the data to be cleaned.

In an example implementation of this embodiment, parsing the data to be cleaned may include identifying an encoding mode of the data to be cleaned, where the encoding mode of the data to be cleaned may be a base64 encoding mode, a decoder encoding mode, or an encrypted encoding mode, etc., It is not limited in this embodiment.

Step 340: Extract the target data contained in the data to be cleaned by using the target data extractor.

Step 350: Select a target decoder according to the parsed coding mode corresponding to the target data, and decode the target data.

In an example implementation of this embodiment, after the target data included in the data to be cleaned is extracted by the target data extractor, the target can be selected according to the encoding mode of the target data obtained by analysis, that is, the encoding mode of the data to be cleaned obtained by analysis Decoder, and decode the extracted target data according to the target decoder to convert the target data from unrecognizable characters to characters that are easy to understand, for example, convert the target data from "%%" to "abc" and other characters .

Exemplarily, if it is identified that the encoding mode of the data to be cleaned is the base64 encoding mode, that is, the encoding mode of the target data is the base64 encoding mode, then the target decoder corresponding to the base64 encoding mode can be selected to decode the target data; If the encoding mode of the data is the decoder encoding mode, that is, the encoding mode of the target data is the decoder encoding mode, the target decoder corresponding to the decoder encoding mode can be selected to decode the target data.

In an example implementation of this embodiment, if the encoding mode is the encryption encoding mode, an encryption key corresponding to the encryption encoding mode is obtained, and the target data is decoded according to the encryption key.

If it is identified that the encoding mode of the data to be cleaned is the encryption encoding mode, that is, the encoding mode of the target data is the encryption encoding mode, the encryption key corresponding to the encryption encoding mode can be obtained, and the target data can be decoded according to the encryption key.

Step 360: Screen the decoded target data according to the multiple reference data uploaded by the client to clean the data to be cleaned.

In the solution of this embodiment, the data to be cleaned is parsed to determine the encoding mode of the data to be cleaned; a target decoder is selected according to the parsed encoding mode corresponding to the target data, and the target data is decoded, thereby converting the target data into A form of expression that is easy to identify, and provides a basis for subsequent cleaning of redundant data in post data.

FIG. 4 is a flowchart of a data cleaning method in another embodiment of the present application. This embodiment is a refinement of the above technical solution. The technical solution in this embodiment may be the same as that in one or more of the above embodiments. Example scenarios combined. As shown in Figure 4, the data cleaning method may include the following steps:

Step 410: Acquire data to be cleaned.

Step 420: Identify the data format type contained in the data to be cleaned, and determine the target data extractor according to the data format type.

Step 430: Analyze the data to be cleaned.

Step 440: Extract the target data contained in the data to be cleaned by using the target data extractor.

Step 450: Select a target decoder according to the parsed coding mode corresponding to the target data, and decode the target data.

Step 460: Compare the decoded target data with multiple reference data output by the data model uploaded by the client; if the first data in the target data matches the multiple reference data, keep the first data; if If the second data in the target data does not match the multiple reference data, the second data is filtered out.

Wherein, the first data in the target data and the second data in the target data are any data in the target data, which are only for the convenience of describing this embodiment, rather than limiting the embodiment of this application.

In an example implementation of this embodiment, after the target data is decoded, the decoded target data may be compared with multiple reference data output from the data model uploaded by the client. Any reference data in the reference data matches, for example, the similarity is greater than a set threshold (for example, 0.9, 0.85 or 0.99, etc., which are not limited in this embodiment), the first data can be retained; if the second The data does not match any reference data among the multiple reference data. For example, if the similarity with any reference data is less than the set threshold, it can be determined that the second data is redundant data, and can be filtered out. .

It should be noted that the data model involved in this embodiment may be a data model obtained by training on the client or computer in advance; in this embodiment, the process of training the data model may include: the sample data may be Labeling, wherein the sample data can be a large amount of post data, which is not limited in this embodiment; a data model can be constructed through naive Bayesian training, and all user-required data can be output, and the output data can be normalized on this basis. Normalization (data standardization, case removal, etc.), and then highly aggregate the normalized data, refine the data content, and reduce the number of data pieces. reference data.

In the solution of this embodiment, after decoding the target data, the decoded target data can be compared with multiple reference data output by the data model uploaded by the client; If the reference data matches, the first data is retained; if the second data in the target data does not match multiple reference data, the second data is filtered out, which realizes the filtering of redundant data contained in the post data. , which can save the storage space of post data and improve the transmission efficiency of post data.

In order to make those skilled in the art better understand the data cleaning method involved in this embodiment, an example is used for description below. Referring to FIG. 5 , the process of the data cleaning method includes:

Step 510, post data identification.

Among them, the post data identification mainly identifies the data structure of the post, including identifying the data encoding mode, identifying the cookie part data, identifying the header data of the request header, and identifying the body data of the request body. On this basis, identify the data type (mainly including json, xml, key-value pair type data). Through the identification of the data pattern, the subsequent processing logic control is simplified, and it is convenient to apply the corresponding extractor for each data format to perform data extraction.

Step 520, post data extraction.

The post data consists of three parts: request url, cookie, and body. Each part needs to be extracted. At the same time, the request url data and cookie data are generally key-value pair data, and the body contains xml, JSON, and key-value data. For the three modes, as shown in Figure 6.

In the design of post data extraction, combined with the results of data identification technology, the responsibilities are assigned to specific extractors, and each extractor is responsible for the extraction of corresponding types of data.

Xml extractor

Perform data analysis for xml format data, extract attributes and attribute data, and extract label text data according to the characteristics of xml data structure.

cookie extractor

Perform data analysis on cookie data, and extract attribute names and attribute data based on the cookie data structure.

JSON extractor

Data analysis is carried out for JSON format data, and attributes and attribute data are extracted according to the characteristics of JSON data structure, and label text data is extracted.

Step 530: Decode the data in reverse.

The data reverse decoding technology mainly identifies the data pattern from the extracted data. The current stage mainly supports base64 encoding and ordinary decoder encoding mode data. Under the condition of identifying the corresponding data encoding mode, the standard decoder is called for decoding, so as to realize the restoration of the encoded data, thereby realizing the data restoration capability as much as possible and improving the data quality.

In actual combat data, there are multiple encoding modes, including JSON nested in xml or xml nested in JSON. At the same time, the node text value of xml is encoded JSON data, and the JSON data field value adopts encoding technology. Through the design of data identification, data extraction, and data reverse coding components, the above components are combined and nested. The implementation support of multiple complex scenes is well completed.

Step 540, data cleaning.

Among them, the data cleaning technology mainly includes two parts:

1. Sample data annotation and keyword extraction:

In different regions, the value direction of the data is different. The sample data is labeled according to the user's requirements, the data model is constructed through Naive Bayesian training, and all the user-required data is output. On this basis, the output data is normalized (data standardization, Remove case, etc.), and then highly aggregate the normalized data, refine the data content, and reduce the number of data items.

2. Data matching

Find all matching patterns in a string, such as finding which phrases in a dictionary match a piece of text. Data matching adopts Aho-Corasick automata algorithm. Its core idea is to skillfully transform character comparisons into state transitions through finite automata. The AC automaton does not need backtracking when matching, and the time complexity is O(n), that is, the time complexity is independent of the size of the dictionary. The overall guarantee of efficient processing of data.

In the Aho-Corasick automata algorithm, the number of keywords has a great influence on efficient processing. Therefore, under the premise of meeting business needs, the number of keywords should be reduced as much as possible. In this study, the keywords are highly aggregated and refined.

The embodiments of the present application can analyze post data in the Internet in real time and extract valuable data, reduce distributed file (hadoop) and retrieval engine data storage, improve the retrieval engine's ability to read and write data, and extract valuable data in post data.

FIG. 7 is a schematic structural diagram of a data cleaning apparatus in an embodiment of the present application, and the apparatus can execute the data cleaning method involved in the above-mentioned embodiments. Referring to FIG. 7 , a data extractor determination module 710 , a target data extraction module 720 , and a target data screening module 730 .

Wherein, the data extractor determination module 710 is configured to obtain the data to be cleaned, and to determine the target data extractor corresponding to the data to be cleaned;

The target data extraction module 720 is configured to parse the data to be cleaned, and extract target data contained in the data to be cleaned through the target data extractor, where the target data includes attribute names, attribute data or label text data at least one of the

The target data screening module 730 is configured to decode the target data, and screen the decoded target data according to multiple reference data uploaded by the client, so as to clean the data to be cleaned.

In the solution of this embodiment, the data to be cleaned is obtained through the data extractor determination module, and the target data extractor corresponding to the to-be-cleaned data is determined; the target data extraction module is used to parse the to-be-cleaned data, and the The target data extractor extracts the target data contained in the data to be cleaned; the target data is decoded by the target data screening module, and the decoded target data is screened according to a plurality of reference data uploaded by the client, so as to Cleaning the data to be cleaned realizes cleaning of redundant data in the data, which can save storage space and improve data transmission efficiency.

A data extractor determination module 710, configured to identify the data format type contained in the data to be cleaned, and determine a target data extractor according to the data format type;

Wherein, the data format type includes key-value pair, extensible markup language xml, or JS object notation JSON;

The target data extractor includes: a key-value pair extractor, an xml extractor or a JSON extractor.

The target data extraction module 720 is configured to extract key-value pairs, xml data or JSON information contained in the data to be cleaned through the target data extractor;

The target data extraction module 720 is further configured to extract the key-value pairs contained in the data to be cleaned through the key-value pair extractor;

Or, extract the xml data contained in the data to be cleaned by using the xml extractor;

Alternatively, the JSON information contained in the data to be cleaned is extracted by the JSON extractor.

The target data screening module 730, including a decoding module, is configured to select a target decoder according to the parsed encoding mode corresponding to the target data, and decode the target data;

The encoding mode includes: base64 encoding mode, decoder encoding mode or encryption encoding mode.

The decoding module is further configured to obtain an encryption key corresponding to the encryption encoding mode if the encoding mode is an encryption encoding mode, and decode the target data according to the encryption key.

The target data screening module 730 is configured to compare the decoded target data with a plurality of reference data output by the data model uploaded by the client;

If the first data in the target data matches a plurality of the reference data, retaining the first data;

If the second data in the target data does not match a plurality of the reference data, the second data is filtered out.

The data to be cleaned involved in this embodiment is post data.

The data cleaning apparatus provided in the embodiment of the present application can execute the data cleaning method provided by any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

FIG. 8 is a schematic structural diagram of a data cleaning device provided by an embodiment of the application. As shown in FIG. 8 , the data cleaning device includes a processor 80, a memory 81, an input device 82, and an output device 83; the processor in the data cleaning device The number of 80 can be one or more, and one processor 80 is taken as an example in FIG. 8; the processor 80, the memory 81, the input device 82 and the output device 83 in the data cleaning device can be connected by a bus or other means, as shown in FIG. 8 Take the connection through the bus as an example.

As a computer-readable storage medium, the memory 81 can be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data cleaning method in the embodiments of the present application (for example, data extraction in the data cleaning device). Detector determination module 710, target data extraction module 720, and target data screening module 730). The processor 80 executes various functional applications and data processing of the data cleaning device by running the software programs, instructions and modules stored in the memory 81 , that is, the above-mentioned data cleaning method is implemented.

The memory 81 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Additionally, memory 81 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory 81 may include memory located remotely from processor 80, which may be connected to the data cleaning device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 82 may be configured to receive input numerical or character information and to generate key signal input related to user settings and function control of the data cleaning apparatus. The output device 83 may include a display device such as a display screen.

Embodiments of the present application further provide a storage medium containing computer-executable instructions, where the computer-executable instructions are configured to execute a data cleaning method when executed by a computer processor, and the method includes:

Of course, a storage medium containing computer-executable instructions provided by the embodiments of the present application, the computer-executable instructions of which are not limited to the above-mentioned method operations, and can also execute the data cleaning methods provided in any embodiment of the present application. related operations. The storage medium may be a non-transitory computer-readable storage medium.

From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general-purpose hardware, and certainly can also be implemented by hardware. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence or the parts that make contributions to related technologies, and the computer software products can be stored in a computer-readable storage medium, such as a computer floppy disk, Read-Only Memory (ROM), Random Access Memory (RAM), Flash Memory (FLASH), hard disk or CD, etc., including several instructions to make a computer device (which can be a personal computer, A server, or a network device, etc.) executes the methods described in the various embodiments of the present application.

It is worth noting that, in the above-mentioned embodiment of the data cleaning device, the multiple units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, The specific names of the multiple functional units are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application.

Claims

A data cleaning method, comprising:

acquiring data to be cleaned, and determining a target data extractor corresponding to the data to be cleaned;

Parsing the data to be cleaned, and extracting target data contained in the data to be cleaned by the target data extractor, where the target data includes at least one of attribute name, attribute data or label text data;

The target data is decoded, and the decoded target data is screened according to multiple reference data uploaded by the client, so as to clean the data to be cleaned.
The method according to claim 1, wherein the determining a target data extractor corresponding to the data to be cleaned comprises:

Identifying the data format type contained in the data to be cleaned, and determining a target data extractor according to the data format type;

Wherein, the data format type includes key-value pair, extensible markup language xml, or JS object notation JSON;

The target data extractor includes: a key-value pair extractor, an xml extractor or a JSON extractor.
The method according to claim 2, wherein the extracting the target data contained in the data to be cleaned by the target data extractor comprises:

Extract key-value pairs, xml data or JSON information contained in the data to be cleaned by using the target data extractor;

The extraction of key-value pairs, xml data or JSON information contained in the data to be cleaned by the target data extractor includes one of the following:

Extract the key-value pairs contained in the data to be cleaned by using the key-value pair extractor;

Extract the xml data contained in the data to be cleaned by the xml extractor; and

Extract the JSON information contained in the data to be cleaned by using the JSON extractor.
The method of claim 1, wherein the decoding the target data comprises:

Select a target decoder according to the parsed encoding mode corresponding to the target data, and decode the target data;

The encoding mode includes: base64 encoding mode, decoder encoding mode or encryption encoding mode.
The method of claim 4, wherein the decoding the target data further comprises:

In response to determining that the encoding mode is an encryption encoding mode, an encryption key corresponding to the encryption encoding mode is obtained, and the target data is decoded according to the encryption key.
The method according to claim 1, wherein the filtering of the decoded target data according to a plurality of reference data uploaded by the client to clean the data to be cleaned comprises:

Compare the decoded target data with a plurality of reference data output by the data model uploaded by the client;

In response to determining that first data in the target data matches the plurality of reference data, retaining the first data;

In response to determining that second data in the target data does not match the plurality of reference data, filtering out the second data.
The method according to any one of claims 1-6, wherein the data to be cleaned is post data.
A data cleaning device, comprising:

a data extractor determination module, configured to obtain data to be cleaned, and to determine a target data extractor corresponding to the data to be cleaned;

A target data extraction module, configured to parse the data to be cleaned, and extract target data contained in the data to be cleaned through the target data extractor, where the target data includes attribute names, attribute data or label text data at least one;

The target data screening module is configured to decode the target data, and screen the decoded target data according to a plurality of reference data uploaded by the client, so as to clean the data to be cleaned.
A data cleaning device, comprising:

one or more processors;

storage means arranged to store one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the data cleaning method according to any one of claims 1-7.
A storage medium containing computer-executable instructions, when executed by a computer processor, for performing the data cleaning method according to any one of claims 1-7.