WO2022062834A1

WO2022062834A1 - Data exploration method and apparatus, electronic device and storage medium

Info

Publication number: WO2022062834A1
Application number: PCT/CN2021/114969
Authority: WO
Inventors: 王如意; 谢永恒; 万月亮
Original assignee: 北京锐安科技有限公司
Priority date: 2020-09-27
Filing date: 2021-08-27
Publication date: 2022-03-31
Also published as: CN112131296B; CN112131296A

Abstract

Disclosed are a data exploration method and apparatus, an electronic device and a storage medium. The data exploration method comprises: acquiring a data extraction rule, a segmentation rule and an exploration requirement, which correspond to a data source; on the basis of the data extraction rule and the data source, determining data to be processed; on the basis of the segmentation rule, segmenting the data to be processed to obtain segmented data; and carrying out exploration on the segmented data on the basis of the exploration requirement to obtain an exploration result.

Description

Data exploration method, apparatus, electronic device and storage medium

This application claims the priority of the Chinese patent application with application number 202011036080.8 filed with the China Patent Office on September 27, 2020, the entire contents of which are incorporated herein by reference.

technical field

The embodiments of the present application relate to the field of big data processing, for example, to a data exploration method, apparatus, electronic device, and storage medium.

Background technique

With the development of the Internet of Everything, we have entered the era of big data, and the storage structure of data is also changing with each passing day. The huge amount, rapid growth, and variety of data have become a reality that enterprises have to face in the era of big data. Being able to quickly and accurately grasp the data structure information of data from various sources can greatly reduce the investment in data analysis. cost, and quickly tap the value of data. Data exploration is a major trend in the field of big data processing in the future.

The data exploration method is usually to obtain data to be processed from different data sources and use the fields of the data to be processed to perform data exploration. The related art has at least the following disadvantages: the related art can only probe according to the default fields of the data itself, and for some data sources, the default fields cannot be probed to obtain useful data information, which reduces the efficiency of data utilization.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a data exploration method, apparatus, electronic device, and storage medium, so as to implement data exploration of different data sources and improve data utilization efficiency.

An embodiment of the present application provides a data exploration method, including: acquiring data extraction rules, segmentation rules, and exploration requirements corresponding to a data source; determining data to be processed based on the data extraction rules and the data source; The splitting rule splits the data to be processed to obtain split data; the split data is probed based on the probe requirement to obtain a probe result.

The embodiment of the present application also provides a data exploration device, the data exploration device includes: an acquisition module, configured to acquire data extraction rules, segmentation rules, and detection requirements corresponding to the data source; a data to be processed determination module, configured to be based on the data source. The data extraction rule and the data source are used to determine the data to be processed; the segmentation module is configured to segment the to-be-processed data based on the segmentation rule to obtain segmented data; the detection module is configured to perform segmentation based on the detection requirement. The above segmentation data is probed to obtain the probe results.

An embodiment of the present application further provides an electronic device, the electronic device includes: one or more processors; a storage device configured to store one or more programs, when the one or more programs are stored by the one or more programs A plurality of processors execute such that the one or more processors implement a data exploration method as described in any embodiment of the present application.

Embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, implements the data exploration method provided by any embodiment of the present application.

Description of drawings

1 is a schematic flowchart of a data detection method provided in Embodiment 1 of the present application;

2 is a schematic flowchart of a data detection method provided in Embodiment 2 of the present application;

3 is a schematic diagram of a method for detecting multi-source heterogeneous data provided in Embodiment 3 of the present application;

4 is a structural block diagram of a data detection device provided in Embodiment 4 of the present application;

FIG. 5 is a schematic structural diagram of an electronic device according to Embodiment 5 of the present application.

detailed description

The present application will be described below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all the structures related to the present application.

Example 1

FIG. 1 is a schematic flowchart of a data detection method provided in Embodiment 1 of the present application. This embodiment can be applied to the situation where data of different structures from different data sources is probed. The method may be performed by a data detection apparatus, which may be implemented in software and/or hardware, and integrated into the electronic device provided in the embodiments of the present application, for example, may be configured in a computer, which is not limited herein.

As shown in FIG. 1 , the data detection method provided by this embodiment includes the following steps.

S110. Acquire data extraction rules, segmentation rules, and exploration requirements corresponding to the data source.

The data extraction rules corresponding to the data of different data sources are different, and the data extraction rules may include the data extraction methods of different data sources, the amount of data to be extracted, and so on.

Optionally, the data extraction rule is determined according to the type of the data source.

Different types of data sources have different data structures, and corresponding data extraction rules are determined according to different data structures.

Exemplarily, when the type of the data source is a database table, the data type at this time is structured data, and the extraction rule of the database table may be, using the join method to formulate and extract the data of the database table to be extracted, and/or extract the data in the database table. The data of the field to be extracted, in which the database table to be extracted and the field to be extracted can be selected and set according to the exploration requirements; when the type of the data source is file, the data type at this time is unstructured data, and the extraction rules of the file can be Yes, use the extraction method of machine learning to determine the number of data files to be read according to the data files. When the data in each file is small, read more data files. If the amount of data in the file is large, then To read fewer data files, the embodiment of the present application does not limit the data extraction rules.

In the data extraction rule, the data extraction method corresponding to the data source can be set, so that when extracting the data of the data source, the configured extraction method can be read, and the quantity of read data can be set when configuring the data extraction method.

The data extraction rules are determined according to the type of the data source, which can realize the rapid extraction of data from different data sources.

The segmentation rule refers to the rules for how to perform row segmentation and column segmentation for the extracted data. The extraction data is segmented based on the segmentation rules, which can facilitate subsequent data exploration. For example, the division rule may be to perform row division and column division of the data by setting different delimiters, or may be to set the delimiter according to the requirement attribute contained in the data, and the embodiment of the present application does not limit the division rules.

Probing requirements refer to the data attribute information that needs to be understood for data from different data sources. By obtaining the probing requirements, the data attribute information of the data source corresponding to the probing requirements can be determined. For example, for a database table, the probing requirements can be: Field name, field type and other information; for file data, the detection requirement may be field length, semantics, etc., and the embodiment of this application does not limit the detection requirement.

Data extraction rules, segmentation rules, and detection requirements can be preset according to the type of data source, and data extraction rules, segmentation rules, and detection requirements of different data sources can be different.

Data extraction rules, segmentation rules, and exploration requirements are pre-set and configured in the system file, and can be directly read from the system file during data exploration.

S120. Determine the data to be processed based on the data extraction rule and the data source.

Based on data extraction rules corresponding to different data sources, JAVA multithreading can be used to extract data from different data sources, and the extracted data is determined as data to be processed. Pending data is used for data exploration.

Exemplarily, when using the corresponding data extraction method to extract data from data of different structures in different systems, it may be full extraction, that is, the data in the data source is extracted from the data system intact, or it may be based on incremental data extraction. Volume extraction, that is, extraction of newly added, modified, and deleted data in the data system since the last extraction, and the data extraction method is not limited in this embodiment of the present application.

S130. Segment the data to be processed based on the segmentation rule to obtain segmented data.

The data extracted from different data sources may have different data structures, and it is necessary to normalize the data to be processed with different structures into the same structural form. The segmentation rule refers to a rule used to normalize the data to be processed with different structures. The segmentation rules are used to segment the data to be processed with different structures into the same form to obtain the segmented data corresponding to different data sources.

S140. Investigate the segmented data based on the probing requirement, and obtain a probing result.

The exploration requirements corresponding to different data sources include data attribute information about different data sources. Different data attribute information can be formulated into exploration items, and the segmented data can be probed according to the exploration items. When the segmented data is probed, it can be considered that data exploration is performed on the segmented data according to a probe rule, wherein the probe rule may include the probe item and the segment location information contained in the segment rule. Exemplarily, in the segmentation rule, the data to be processed is segmented according to the delimiter, then the detection rule may include the location information of the delimiter, and the segmented data is examined according to the location information of the delimiter and the detection item, to obtain data detection of different data sources. information.

Data exploration is mainly to realize the function of exploration and analysis of table data and file data, etc., and build the requirements related to data quality as exploration items to detect whether the data meets the data quality requirements of users.

Optionally, the split data is probed based on the probe requirements, and after the probe results are obtained, the method further includes: collecting statistics on the probe results, obtaining the statistical results, and outputting a probe report based on the probe requirements and the statistical results.

After the data to be processed is divided according to the division rules, the division data is probed according to the division position information contained in the probe rules, so the probed information is the field information divided by multiple delimiters, not the information of the entire extracted data, but also the information of the fields. The data information after the probe needs to be counted to obtain the overall probe result of the extracted data, and then a probe report is output based on the overall probe result and the probe items included in the probe requirements. The probe report may include probe information on multiple fields, or may include the statistical results of the entire data source probe, and may also be a distribution map of data probe items. This embodiment of the present application does not limit the content of the probe report.

The data exploration method provided by the embodiment of the present application can extract, explore, analyze, and process data with different data structures such as various databases and data files, and perform statistics and output on the exploration results, so as to quickly grasp the data content and data quality. It provides an important basis for data analysis and data cleaning of big data. Moreover, this method can support functions such as data exploration result preview, detailed viewing of exploration results, and generation of data distribution graph, which can help data analysts to understand the content, structure and characteristics of source data faster and better, and quickly discover anomalies in source data. Data, better data analysis and data cleaning can greatly speed up the extraction of data value.

A method for data exploration provided by an embodiment of the present application, by acquiring data extraction rules, segmentation rules and detection requirements corresponding to data sources, determining data to be processed based on the data extraction rules and data sources, and performing processing on the data to be processed based on the segmentation rules. Segmentation, obtains segmented data, probes the segmented data based on the exploration requirements, and obtains the exploration results, realizes that the data of different data sources is segmented according to the corresponding segmentation rules, and then probes according to the segmentation results. The problem that useful data information cannot be obtained when the default fields are probed, which improves the efficiency of data utilization.

Embodiment 2

FIG. 2 is a schematic flowchart of a data detection method provided in Embodiment 2 of the present application. In this embodiment, on the basis of the above-mentioned multiple embodiments, the segmentation rules are refined into row segmentation rules and column segmentation rules, and row detection is performed according to the row segmentation rules, and column detection is performed according to the column segmentation rules. The data is probed through custom determined fields according to the probe requirements. The explanations of terms that are the same as or corresponding to the above-mentioned multiple embodiments are not repeated here.

Referring to FIG. 2 , the data detection method provided by this embodiment includes the following steps.

S210: Acquire data extraction rules, segmentation rules, and exploration requirements corresponding to the data source.

S220. Determine the data to be processed based on the data extraction rule and the data source.

S230. Based on the row segmentation rule, perform row segmentation on the data to be processed to obtain row segmentation data.

The row division rule can be considered as a rule for row division for the data to be processed which is normalized into a determinant. The row division rule may be, for example, that according to the row separator in the loaded row division rule, the data to be processed is divided into each row of data, and after division, the data is represented as row division data in the form of row by row. Optionally, the line separator can be the Tab key by default. In this case, the Tab key is used as the line dividing rule. Exemplarily, for file data, a period can be used as a delimiter to divide each sentence into a line; for data in a database table, a tab can be used as a delimiter to perform data division to obtain line-segmented data, an embodiment of the present application. There is no restriction on the manner of row division of the data to be processed.

S240. Based on the column division rule, perform column division on the row division data to obtain a row and column division array.

The data to be processed is divided into rows based on the row division rules, and after the row division data is obtained, each row of data can be divided into columns based on the column separators in the column division rules, and finally a row and column division array is obtained. Therefore, a row-column-split array is an array obtained by dividing the data to be processed in the form of a determinant by a row separator and a column separator, wherein each array element can be regarded as a field, and the lengths of different fields can be different.

S250. Perform row exploration on the row-column segmented array based on the exploration requirement, to obtain a row exploration result.

Each row of data in the row-column split array can be probed according to the row probe items contained in the probe requirements, and the probe results of the row probe items of each row of data can be obtained.

Optionally, at least one of the following row detection is performed on the row-column split array: abnormal data filtering, data sorting, data repetition rate, and the number of data records.

Perform data filtering, sorting, and record data repetition rates for each row of data in the row-column-divided array, and count the number of records of data to be processed. Data filtering can be used to screen abnormal data by judging abnormal data. Data sorting can be to re-sort the data of different rows according to the length of each row. The data repetition rate can be to determine the number of rows that duplicate data of different rows. . The data repetition rate can be determined according to the total number of data rows and the number of repeated data rows. Each row of data is a piece of data and also a record. The number of records is the total number of rows of data. The number of records can be used to determine how many pieces of data have been extracted from the corresponding data source. This embodiment of the present application does not make any changes to the content of the row detection item. limit.

The row-column-divided array is row-detected according to the row-detection rules, and the information of each row of data can be determined, and then the quality of each row of data can be analyzed.

S260. Perform column exploration on the row-column divided array based on the exploration requirement, to obtain a column exploration result.

The data of each column of the row-column divided array can be probed separately according to the column probe items included in the probe requirements, and the probe results of the column probe items of each column of data can be obtained. Column profiling is actually to perform profiling items on each column of data based on the column delimiter, wherein the field is determined by dividing each row of data according to the column delimiter in the above steps.

Optionally, perform at least one of the following column exploration on the row-column-divided array: format, type, length, value, null rate, maximum value, minimum value, average value, and range distribution.

After row exploration is performed on the data, the data format, data type, data length, value, null rate, maximum value, minimum value, average value, and value range distribution of the corresponding column are probed for each column of each row of data according to the column separator. and other information.

The null value rate may refer to the ratio of unknown data items in the values of the data corresponding to the column to the total amount of data items, and the value range distribution may refer to the range of value ranges included in the maximum and minimum values of each column of data.

Exemplarily, the data format may be decimal, or binary, etc., and the data type may be numerical data, character data, or code type data. For numerical data, you can explore specific numerical values, null value rates, and value range distributions. This embodiment of the present application does not limit the content of the column probe item.

Column exploration is performed on the row-column division array to obtain the information of each column of data, which is convenient for exploration of each column of each row of data, and then analyzes the quality of each column of data divided according to the column division rules.

S270. Statistical analysis is performed on the exploration results to obtain the statistical results, and based on the inspection requirements and the statistical results, an inspection report is output.

The embodiments provided in this application extract, probe, analyze, and process data with different data structures such as various databases and data files, and perform statistics and output on the probe results, so as to quickly grasp the data content, data quality, and data structure. Data analysis and data cleaning of big data provide an important basis.

In the technical solution of this embodiment, the data to be processed is determined based on the data extraction rules and the data source by acquiring the data extraction rules, segmentation rules and exploration requirements corresponding to the data source, and the data to be processed is divided into rows based on the row segmentation rules, and the result is obtained. Row segmentation data, based on the column segmentation rules, perform column segmentation on the row segmentation data to obtain a row and column segmentation array, perform row detection on the row and column segmentation array based on the detection requirements, and obtain the row detection result. Column exploration results, make statistics on the exploration results, get the statistical results, and output the exploration report based on the exploration requirements and the statistical results. Data usage efficiency.

Embodiment 3

FIG. 3 is a schematic diagram of a method for detecting multi-source heterogeneous data according to Embodiment 3 of the present application. The explanations of terms that are the same as or corresponding to the above-mentioned multiple embodiments are not repeated here.

As shown in FIG. 3 , an exemplary process of detecting multi-source heterogeneous data is as follows.

S310, data extraction. For data of different structures of database, file system, File Transfer Protocol (FTP) system and message middleware system, extract data according to corresponding data extraction rules.

S320. Read the detection rule. Detect the repetition rate of multi-line data, the number of rows of recorded data, etc. according to the row detection rules in the read detection rules, and perform field detection according to the field detection rules in the detection rules, that is, column detection rules. Field detection can include detection fields completeness, uniqueness, correctness, consistency or validity. Optionally, the integrity of the field can be determined by checking whether the field has a null value, whether the value of a field is unique, whether the length and type of the field are correct, and whether the length of the field is the same as that specified by the field rule. Consistent, etc., you can also probe whether the value of the field is valid for the probe requirements.

S330, data operation. When data exploration is performed, data operations such as analysis, sorting, data structure exploration, and filtering are performed on the data based on data exploration rules.

S340, quality statistics. According to the probing results, the quality is evaluated through quality statistics. The quality statistics can be row statistics and field statistics. The row statistics can be statistics on the repetition rate and the number of records. The field statistics can be field type statistics, field value statistics, and field statistics. Length statistics, field null value statistics and proportion, field maximum value statistics, field minimum value statistics, field average statistics, field value repetition rate statistics and field value range analysis, etc.

S350, outputting an investigation report. Finally, output the final data exploration report based on the statistical information of the actual data exploration and the exploration requirements for data analysis and data cleaning.

The data detection method provided by this embodiment can detect data of different data sources according to the row detection rule and the field detection rule, and perform quality statistics according to the detection result, which improves the data utilization efficiency.

The implementation manner of the technical features described in any embodiment of the present disclosure can be applied to this embodiment on the premise that there is no contradiction with this embodiment, and details are not repeated here.

Embodiment 4

FIG. 4 is a structural block diagram of a data detection apparatus provided in Embodiment 4 of the present application, and this embodiment may be applied to the case where data of different structures from different data sources is probed. The apparatus for applying data detection can implement the data detection method provided by any embodiment of the present application. As shown in FIG. 4 , the data detection device includes: an acquisition module 410, configured to acquire data extraction rules, segmentation rules and detection requirements corresponding to the data source; a data to be processed determination module 420, configured to be based on the data extraction rules and all The data source is used to determine the data to be processed; the segmentation module 430 is configured to segment the data to be processed based on the segmentation rules to obtain segmented data; the detection module 440 is configured to perform segmentation on the segmented data based on the detection requirements. Probe and get the probing results.

Optionally, acquiring the data extraction rule corresponding to the data source includes: determining the data extraction rule according to the type of the data source.

Splitting rules include row splitting rules and column splitting rules.

The segmentation module 430 includes: a row segmentation unit, configured to perform row segmentation on the data to be processed based on a row segmentation rule to obtain row segmentation data; a column segmentation unit, configured to perform column segmentation on the row segmentation data based on the column segmentation rule to obtain row-column segmentation array.

The probe module 440 includes: a row probe module configured to perform row probe on the row-column split array based on probe requirements to obtain row probe results; a column probe module configured to perform column probe on the row-column split array based on probe requirements to obtain column probe results.

Perform at least one of the following row profiling on row-column split arrays: abnormal data filtering, data sorting, data repetition rate, and number of data records; perform at least one of the following column profiling on row-column split arrays: format, type, length, value, null Value rate, percentage of nulls, maximum, minimum, mean, and range distribution.

Optionally, the data detection device further includes: a statistics module, configured to perform statistics on the detection results to obtain the statistical results; and an output module, configured to output a detection report based on the detection requirements and the statistical results.

In the data exploration device provided by the embodiment of the present application, the acquisition module is configured to acquire data extraction rules, segmentation rules and detection requirements corresponding to the data source, and the data to be processed determination module is configured to be based on the data extraction rules and the data source, Determine the data to be processed, the segmentation module is set to segment the data to be processed based on the segmentation rules to obtain the segmented data, and the detection module is set to perform detection on the segmented data based on the detection requirements to obtain the detection result.

The data detection apparatus provided by the embodiment of the present application can execute the data detection method of any embodiment of the present application, and has functional modules corresponding to the execution method. For technical details that are not described, reference may be made to the data exploration method provided in any embodiment of the present application.

Embodiment 5

FIG. 5 is a schematic structural diagram of an electronic device provided in Embodiment 5 of the present application. Figure 5 shows a block diagram of an exemplary electronic device 12 suitable for use in implementing any of the embodiments of the present application. The electronic device 12 shown in FIG. 5 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

As shown in FIG. 5, the electronic device 12 takes the form of a general-purpose computing device. Components of the electronic device 12 may include, but are not limited to, one or more processors or processing units 16 , a memory 28 , and a bus 18 connecting the various components including the memory 28 and the processing unit 16 .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, enhanced ISA bus, Video Electronics Standards Association (Video Electronics Standards Association) Association, VESA) local bus and Peripheral Component Interconnect (PCI) bus.

Electronic device 12 includes a variety of computer-readable media. These media can be any available media that can be accessed by electronic device 12, including both volatile and non-volatile media, removable and non-removable media.

Memory 28 may include computer device readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32 . Electronic device 12 may also include other removable/non-removable, volatile/non-volatile computer storage media. For example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 5, commonly referred to as a "hard drive"). Although not shown in FIG. 5, storage system 34 may provide disk drives configured to read and write to removable non-volatile magnetic disks (eg, "floppy disks"), as well as removable non-volatile optical disks (eg, optical disks) (Compact DiscRead-Only Memory, CD-ROM), digital video disc (Digital Video DiscRead-Only Memory, DVD-ROM) or other optical media) optical disc drive for reading and writing. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. The memory 28 may include at least one program product 40 having a set of program modules 42 configured to perform the functions of various embodiments of the present application. Program product 40, which may be stored, for example, in memory 28, such program modules 42 including, but not limited to, one or more application programs, other program modules, and program data, each or some combination of these examples may include a network environment realization. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The electronic device 12 may also communicate with one or more external devices 14 (eg, keyboard, mouse, camera, etc., and display), with one or more devices that enable a user to interact with the electronic device 12, and/or with Any device (eg, network card, modem, etc.) that enables the electronic device 12 to communicate with one or more other computing devices. Such communication may take place through an input/output (I/O) interface 22 . Furthermore, the electronic device 12 can also communicate with one or more networks (eg, Local Area Network (LAN), Wide Area Network, WAN) and/or public networks such as the Internet through the network adapter 20. As shown in FIG. 5 , the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18 . It should be understood that, although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) devices, tape drives, and data backup storage devices, etc.

The processor 16 executes a variety of functional applications and data processing by running the programs stored in the memory 28, for example, to implement the data detection method provided by the above-mentioned embodiments of the present application, the method includes: acquiring data extraction rules corresponding to the data source , segmentation rules and exploration requirements; determine the data to be processed based on the data extraction rules and the data source; segment the to-be-processed data based on the segmentation rules to obtain segmented data; based on the exploration requirements, probe the segmented data to obtain the detection results .

Of course, those skilled in the art can understand that the processor can also implement the data detection method provided by any embodiment of the present application.

Embodiment 6

Embodiment 6 of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, implements the data detection method provided by any embodiment of the present application, the method comprising: acquiring and data Data extraction rules, segmentation rules, and exploration requirements corresponding to the source; determine the data to be processed based on the data extraction rules and the data source; segment the to-be-processed data based on the segmentation rules to obtain segmentation data; Get the probe results.

Certainly, the computer program stored on a computer-readable storage medium provided by the embodiments of the present application is not limited to the above method instructions, and can also execute the data detection method provided by any embodiment of the present application.

The computer storage medium of the embodiments of the present application may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, device, or device, or a combination of any of the above. Examples (non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, RAM, Read-Only Memory (ROM), erasable Programmable read-only memory (Electrically Programmable Read-Only Memory, EPROM) or flash memory, optical fiber, portable CD-ROM, optical storage device, magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution apparatus, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution apparatus, apparatus, or device .

The program code embodied on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the above.

Computer program code for executing the instructions of the present application may be written in one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programs, or a combination thereof A design language such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any kind of network, including a LAN or WAN, or may be connected to an external computer (eg, through the Internet using an Internet service provider).

Claims

A data profiling method comprising:

Obtain data extraction rules, segmentation rules and exploration requirements corresponding to the data source;

Determine the data to be processed based on the data extraction rule and the data source;

Splitting the data to be processed based on the splitting rule to obtain split data;

The segmented data is probed based on the probe requirement to obtain a probe result.
The method of claim 1, wherein the partitioning rules comprise row partitioning rules and column partitioning rules;

The step of dividing the data to be processed based on the dividing rule to obtain the dividing data includes: dividing the data to be processed by row based on the dividing rule to obtain row dividing data; and dividing the data based on the column dividing rule , perform column segmentation on the row segmentation data to obtain a row-column segmentation array;

The performing a search on the segmented data based on the search requirement to obtain a search result includes: performing a row search on the row and column division array based on the search requirement to obtain a row search result; Divide the array to perform column profiling to obtain column profiling results.
The method of claim 2, further comprising:

Perform at least one of the following row explorations on the row-column split array: abnormal data filtering, data sorting, data repetition rate and data record number;

Perform at least one of the following column explorations on the row-column partitioned array: format, type, length, value, null rate, maximum value, minimum value, average value, and range distribution.
The method according to claim 1, wherein the acquiring the data extraction rule corresponding to the data source comprises:

The data extraction rule is determined according to the type of the data source.
According to the method of claim 1, after the detection of the segmented data based on the detection requirement, and after obtaining the detection result, the method further comprises:

Statistics are performed on the probe results to obtain statistical results, and a probe report is output based on the probe requirements and the statistical results.
A data exploration device, comprising:

The acquisition module is set to acquire the data extraction rules, segmentation rules and exploration requirements corresponding to the data source;

a data to be processed determination module, configured to determine the data to be processed based on the data extraction rule and the data source;

A segmentation module, configured to segment the to-be-processed data based on the segmentation rule to obtain segmented data;

The exploration module is configured to probe the segmented data based on the probe requirement, and obtain a probe result.
The apparatus of claim 6, wherein the division rules comprise row division rules and column division rules;

The segmentation module includes:

a row segmentation unit, configured to perform row segmentation on the data to be processed based on the row segmentation rule to obtain row segmentation data;

a column division unit, configured to perform column division on the row division data based on the column division rule to obtain a row and column division array;

The probe module includes:

a row detection unit, configured to perform row detection on the row-column segmented array based on the detection requirement, to obtain a row detection result;

A column detection unit, configured to perform column detection on the row-column divided array based on the detection requirement, to obtain a column detection result.
The apparatus of claim 6, further comprising:

A statistical module, configured to count the probe results, and obtain the statistical results;

The output module is configured to output a probe report based on the probe requirement and the statistical result.
An electronic device comprising:

at least one processor;

a memory, arranged to store at least one program;

When the at least one program is executed by the at least one processor, the at least one processor is caused to implement the data exploration method of any one of claims 1-5.
A computer-readable storage medium on which a computer program is stored, which implements the data exploration method according to any one of claims 1-5 when the program is executed by a processor.