WO2022121337A1

WO2022121337A1 - Data exploration method and apparatus, and electronic device and storage medium

Info

Publication number: WO2022121337A1
Application number: PCT/CN2021/109589
Authority: WO
Inventors: 霍康; 万月亮; 火一莽
Original assignee: 北京锐安科技有限公司
Priority date: 2020-12-11
Filing date: 2021-07-30
Publication date: 2022-06-16
Also published as: CN112559523A

Abstract

Disclosed are a data exploration method and apparatus, and an electronic device and a storage medium. The method comprises: according to a condition for an exploration range, matching at least one target data table from a database in a connected state; acquiring a data structure of the at least one target data table, and determining fields, to be explored, in the at least one target data table; and on the basis of a preset exploration rule, respectively exploring said fields, and determining an exploration result.

Description

Data exploration method, device, electronic device, and storage medium

The present disclosure claims the priority of a Chinese patent application with application number 202011462110.1 filed with the Chinese Patent Office on December 11, 2020, the entire contents of which are incorporated herein by reference.

technical field

The embodiments of the present application relate to the technical field of data analysis, for example, to a data exploration method, apparatus, electronic device, and storage medium.

Background technique

With the advent of the era of big data, the application of data is becoming more and more important, and more and more applications and services are built based on data. Moreover, data quality is the basis for the validity and accuracy of data analysis and data mining conclusions, as well as the premise for data-driven decision-making.

In the related art, a plurality of independent detection scripts are generally written manually to probe the raw data table by table to analyze the quality of the raw data; and a plurality of independent detection scripts are directly used to execute corresponding task-based queries. However, every time data detection is performed, repeated detection scripts need to be manually written, which will consume a lot of manpower and reduce data detection efficiency.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a data exploration method, device, electronic device, and storage medium, so as to automatically complete the exploration and analysis of data quality in data tables of different database types, which is convenient to operate and improves the efficiency of data exploration.

In a first aspect, an embodiment of the present application provides a data detection method, which is applied to a data table detection device, and the method includes:

matching at least one target data table from at least one database in the connected state according to the probe scope condition;

acquiring the data structure of the at least one target data table, and determining the fields to be probed in the at least one target data table according to the data structure of the at least one target data table;

Based on the preset detection rules, the fields to be detected are respectively detected, and the detection results are determined, wherein the detection rules include field filling detection rules, feature value detection rules, field length detection rules, and field dictionary code detection rules. at least one of.

In a second aspect, an embodiment of the present application further provides a data table detection device, including:

a target data table determination module, configured to match at least one target data table from at least one database in a connected state according to the probe scope condition;

A to-be-explored field determination module, configured to acquire a data structure of the at least one target data table, and to determine a to-be-explored field in the at least one target data table according to the data structure of the at least one target data table;

The probe result determination module is configured to probe the fields to be probed respectively based on the preset probe rules, and determine the probe results, wherein the probe rules include field fill probe rules, feature value probe rules, and field length probe rules and at least one of the field dictionary code detection rules.

In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising:

one or more processors;

storage means arranged to store one or more programs,

The one or more processors are configured to execute the one or more programs to implement the data detection method provided by any embodiment of the present application.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data provided by any embodiment of the present application is implemented Probing method.

Description of drawings

In order to illustrate the embodiments of the present application, the following briefly introduces the accompanying drawings used in describing the embodiments.

FIG. 1 is a schematic flowchart of a data exploration method provided in Embodiment 1 of the present application;

2 is a schematic structural diagram of a data table detection device provided in Embodiment 2 of the present application;

FIG. 3 is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present application.

Detailed ways

The present application will be described below with reference to the accompanying drawings and embodiments.

Example 1

FIG. 1 is a flowchart of a data exploration method provided in Embodiment 1 of the present application, and this embodiment may be applied to a situation in which performance testing is performed in software testing. The method may be performed by a data table look-up device, which may be implemented by means of software and/or hardware.

Before introducing the first embodiment of the present application, the application scenario in which the first embodiment is located is introduced.

In the data analysis scenario, before analyzing the data, the validity and accuracy of the data quality can be ensured. In order to ensure data quality and data availability, in the related art, technicians usually write detection scripts, and use the detection scripts to detect the original data quality of the database that needs to be probed, and the detection scripts cannot be used in multiple different types of databases. , which consumes a lot of manpower and material resources, making data detection inefficient.

The data exploration method provided in the first embodiment of the present application can cope with the above-mentioned low data detection efficiency. Integrate the profiling results, and generate a profiling report by combining the profiling results and the data table to be profiled, which is convenient for data analysts to analyze the data quality of the profiling data. In order to ensure the versatility and flexibility of the data table detection device, the configuration file of the data table detection device is pre-configured, so that when the data detection method executes the data table detection device, the configuration parameters in the configuration file can be directly read, and the data can be successfully executed. Probing method. The configuration file includes configuration parameters such as the data type of the source data, the database instance name, the database connection method, the database instance, the database user name, and the password.

In one embodiment, the configuration file is read and executed, and at least one database is connected according to the connection mode of at least one database in the configuration file. Among them, the database includes Oracle, mysql, mpp, hive, txt, excel, csv, word and other databases. For example, ADO (Active Data Objects, Active Data Objects) can be used for database connection, and there are many connection methods.

In Embodiment 1 of the present application, as shown in FIG. 1 , the method includes the following steps.

S110. Match at least one target data table from at least one database in a connected state according to the condition of the probe scope.

In this embodiment of the present application, the probe scope condition may be a probe condition for obtaining at least one target database set according to requirements. For example, when querying the all_tab_comments table to obtain a list of library tables, a table name filter condition of table_name may be added. The target data table may be a database to be probed determined from a plurality of databases in a connected state according to a probe scope condition of the database. The number of at least one target data table may be one or more.

In one embodiment, the probe scope condition and the matching mode may be acquired, a matching instruction may be generated based on the probe scope condition and the matching mode, and the matching instruction may be executed to determine at least one target data table in at least one database, Wherein, the matching mode includes any one of exact matching parameters, fuzzy matching parameters, exact exclusion parameters and fuzzy exclusion parameters. The exact match parameter is extract_match, the fuzzy match parameter is fuzzy_match, the exact exclusion parameter is exact_not_match, and the fuzzy exclusion parameter is fuzzy_not_match. The matching instruction may be a database matching instruction generated based on the matching mode and the conditions of the detection range, and the matching instruction is used to determine at least one target data table to be detected.

For example, when it is not necessary to perform probe analysis on all the data tables in the database, at least one target data table to be probed can be determined based on the matching mode and the database matching instruction generated based on the probe range condition. Exemplarily, when it is desired to probe for tables starting with T_ and G_ or named PERSON_INFO in the Oracle library, but not including tables with _TMP identified in the table name, the following configuration can be used: extract_match=PERSON_INFO; fuzzy_match=T_,G_; exact_not_match=_TMP.

S120. Acquire the data structure of the at least one target data table, and determine the fields to be probed in the at least one target data table according to the data structure of the at least one target data table.

In this embodiment, the data structure may be a way of storing the data to be probed in the data table. The fields to be probed may be fields in the data structure of the target data table, such as fields such as field name, field description, field type, and field length. For example, multiple fields in the data structure can be obtained, preset fields to be explored can be obtained as fields to be probed, all fields can be used as fields to be probed, and fields to be probed can also be set according to actual conditions.

In one embodiment, the number of the fields to be probed is determined, and when the number of the fields to be probed is greater than a preset number, the data of the fields to be probed is sampled, and the sampled data is determined as the to-be-explored field. The probed data corresponding to the probed field. For example, when the number of fields to be probed in the data table exceeds the preset number of data, and there is a large amount of duplicate data in the data of the fields to be probed, random sampling is performed on the data in the fields to be probed, and the sampled fields are Conduct data exploration. The random sampling method can ensure the validity of the exploration results, and reducing the amount of data can also reduce the computational complexity of the data exploration and improve the efficiency of the exploration.

S130. Based on the preset detection rules, respectively perform detection on the to-be-detected fields, and determine the detection results, wherein the detection rules include field filling detection rules, feature value detection rules, field length detection rules, and field dictionary code detection at least one of the rules.

The detection rule is a detection indicator for the detection and analysis of the field to be explored, and the detection rule includes at least one of a field filling detection rule, a feature value detection rule, a field length detection rule, and a field dictionary code detection rule. For example, before performing data profiling, set profiling rules in advance. The field to be probed is probed and analyzed according to the probe rules, and the probe result of the field to be probed is determined. The detection result includes at least one of the detection filling rate, the feature value coincidence rate, the maximum field length, and the dictionary code corresponding to the field to be detected.

In one embodiment, check whether the data corresponding to the fields to be checked is filled, determine the number of filled fields, and determine the percentage of the filled fields in the fields to be checked to the number of fields to be checked, and determine the Probe fill rate.

The number of padding fields may be the number of fields with padding values in the fields to be probed. In one embodiment, the determined probe fill rate is presented in a probe report. For example, the formula to calculate the probe fill rate could be:

R1 represents the probe fill rate, N1 represents the number of padding fields in the fields to be probed, and M1 represents the number of fields to be probed.

In one embodiment, the validity of at least one feature value corresponding to the field to be probed is probed, the percentage of the number of valid feature values in the field to be probed to the number of fields to be probed is determined, and the Eigenvalue coincidence rate.

In the embodiment of the present application, the feature value means content capable of identifying real-world entity information, and each feature value has a unique corresponding feature value type. The eigenvalue conformity rate is to detect the conformity of the normative content of the analysis data, and when the eigenvalue conforms to the norm, it is an effective eigenvalue. For example, before calculating the feature value coincidence rate, the feature type corresponding to the feature value in the current field to be probed may be identified to determine the feature value in the current field to be probed.

Exemplarily, when it is determined that the feature type corresponding to the feature value in the field to be probed is hotel address according to the field description (for example, hotel location, hotel address, hotel details) in the field to be probed, and the current to be probed is determined by the hotel address. The only feature value in the field is Hotel. For another example, when it is determined by the field description (license plate number, license plate number) in the field to be probed that the feature type corresponding to the feature value in the field to be probed is the license plate number, and the unique feature value in the field to be probed is determined by the license plate number. for the license plate.

In one embodiment, in response to determining that the field to be probed belongs to a preset feature type, a feature value of the field to be probed is acquired, and a feature value check is performed on the feature value. For example, the feature type that can perform feature value verification is preset as a preset feature type, and when it is determined that the feature type of the current feature value belongs to the preset feature type, the feature value verification is performed on the feature value belonging to the preset feature type. The preset eigenvalue type is an eigenvalue type conforming to a regular expression. When the eigenvalue verification is performed on the eigenvalue, the verification method corresponding to the current eigenvalue can be defined first, and the verification method can be defined in advance or defined according to the actual situation.

Exemplarily, when it is determined that the license plate number in the above embodiment is a preset feature type, the feature value in the to-be-explored field corresponding to the feature type is determined according to a predefined verification method (for example, the check_carnum verification method) for the feature. value to check.

In one embodiment, the coincidence rate of the feature value of the current field to be probed is calculated according to the valid feature value of the successful verification and all the feature values participating in the verification. In one embodiment, when the obtained eigenvalue coincidence rate does not meet the preset threshold, the eigenvalue coincidence rate is displayed in the generated exploration report. Among them, the calculation formula of the eigenvalue coincidence rate can be:

R2 represents the eigenvalue coincidence rate, N2 represents the field number of valid eigenvalues in the to-be-explored field, and M2 represents the number of all eigenvalues in the to-be-explored field.

In one embodiment, the content length corresponding to the field to be probed is probed, and the maximum value of the field length of the field to be probed is determined according to the content length.

The maximum value of the field length includes the longest value of the field and the shortest value of the field. For example, determine the content length of data corresponding to multiple fields to be probed, and compare the content lengths of multiple fields to determine the longest value or shortest value of multiple fields to be probed, and then generate a probe structure report displayed in.

In one embodiment, the description information of the field to be searched is searched, and the dictionary code corresponding to the field to be searched is determined according to the description information.

The dictionary code may be a gender code, a certificate type code, or the like. For example, a preset identification method is used to identify the dictionary code in the description information in the field to be probed, and the identification result is displayed in the probe report. Wherein, the preset identification code may be a neural network identification model, or the identification result may be determined according to the input identification information. In one embodiment, the dictionary code can be displayed in the probe report in an enumerated manner.

In one embodiment, in order to facilitate the profiling analyst to obtain all profiling data results more conveniently, after determining the profiling results of the field to be queried, the profiling results are integrated, and based on the integrated profiling results and the target Data tables generate profiling reports.

Among them, the exploration report can be displayed in the form of an Excel table. For example, a probe report includes a probe catalog summary and a probe detail. Wherein, the general table of exploration catalogue includes the target data table, the data quantity of each target data table, the number of fields, the quantity of feature types, and the feature type information. The probe directory summary table is used to represent the overall statistical information of the current data results to be probed, which is convenient for data analysts to understand the data table to be probed and the basic information of each field in the data table in the current data probe and analysis process. The probe detailed list includes: probe analysis results of the fields to be probed and sample data of the probe analysis results. The probe list is used to represent the detailed information of multiple probe results in the probe analysis results of the current field to be probed, which is convenient for opening and convenient for data analysts to analyze and optimize multiple fields to be probed through the detailed information of the probe results. data performance.

In this embodiment of the present application, at least one target data table is matched from at least one database in a connected state according to the probe scope condition; the data structure of the at least one target data table is acquired, and the target data table is determined according to the data structure of the at least one target data table Describe the fields to be probed in at least one target data table; based on the preset probe rules, probe the fields to be probed respectively, determine the probe results, and fill in the probe results, eigenvalue probe results, fields according to the fields in the probe results The length detection result and the field dictionary code detection result determine the data quality of the field to be detected. In the embodiment of the present application, by setting up a data table inquiry device, connecting the data table device with at least one database, and performing efficient inquiry on the data structure and data content of the data table in the database, it realizes the automatic completion of the data table in different database types. The exploration and analysis of data quality is easy to operate and improves the efficiency of data exploration.

The following is an example of the data table detection device provided by the embodiment of the present application, which belongs to the same inventive concept as the data detection method of the above-mentioned embodiment. For details in the embodiment of the data table detection device, please refer to the above data detection method. Examples of methods.

Embodiment 2

FIG. 2 is a schematic structural diagram of a data table detection apparatus provided in Embodiment 2 of the present application, and this embodiment can be applied to a situation in which performance testing is performed in software testing. The data table inquiry device includes: a target data table determination module 210 , a field to be inspected field determination module 220 , and an inquiry result determination module 230 . in:

The target data table determination module 210 is configured to match at least one target data table from at least one database in the connected state according to the condition of the probe scope.

The to-be-explored field determination module 220 is configured to acquire the data structure of the at least one target data table, and determine the to-be-explored field in the at least one target data table according to the data structure of the at least one target data table.

The probe result determination module 230 is configured to probe the fields to be probed respectively based on the preset probe rules, and determine the probe results, wherein the probe rules include field fill probe rules, feature value probe rules, and field length probes At least one of a rule and a field dictionary code detection rule.

In this embodiment of the present application, at least one target data table is matched from at least one database in a connected state according to the probe scope condition; the data structure of the at least one target data table is acquired, and the target data table is determined according to the data structure of the at least one target data table Describe the fields to be probed in at least one target data table; based on the preset probe rules, probe the fields to be probed respectively, determine the probe results, and fill in the probe results, eigenvalue probe results, fields according to the fields in the probe results The length detection result and the field dictionary code detection result determine the data quality of the field to be detected. In the embodiment of the present application, by setting up a data table inquiry device, connecting the data table device with at least one database, and efficiently inspecting the structure and data content of the data table in the database, the automatic completion of the data table in different database types is realized. The exploration and analysis of data quality is easy to operate and improves the efficiency of data exploration.

On the basis of the above embodiment, the data table detection device further includes:

The database connection unit is configured to read the configuration file, and connect the at least one database according to the connection mode of the at least one database in the configuration file.

On the basis of the above embodiment, the target data table determination module 210 includes:

A target data table determination unit, configured to acquire the probe range condition and the matching mode, generate a matching instruction based on the probe range condition and the matching mode, and execute the matching instruction to determine at least one target data table in at least one database , wherein the matching mode includes any one of exact matching parameters, fuzzy matching parameters, exact exclusion parameters and fuzzy exclusion parameters.

A probe data determination unit, configured to determine the number of the fields to be probed, in response to the number of the fields to be probed being greater than a preset number, sampling the data of the fields to be probed, and determining the data obtained by sampling as the The data to be probed corresponding to the field to be probed.

On the basis of the above embodiment, wherein, the detection result determination module 230 includes:

A first probing result determining unit, configured to probe whether the probed data corresponding to the fields to be probed is filled, determine the number of filled fields, and determine that the number of filled fields in the fields to be probed accounts for the number of fields to be probed The percentage of determining the probe fill rate;

a second detection result determination unit, configured to detect the validity of at least one feature value corresponding to the field to be detected, and to determine the percentage of the number of valid feature values in the field to be detected accounting for the number of fields to be detected, determining the eigenvalue coincidence rate;

a third detection result determination unit, configured to detect the content length corresponding to the field to be probed, and to determine the maximum value of the field length of the field to be probed according to the content length;

The fourth detection result determination unit is configured to detect the description information of the field to be searched, and determine the dictionary code corresponding to the field to be searched according to the description information.

On the basis of the above embodiment, wherein, the second detection result determination unit includes:

A feature value checking unit, configured to, in response to determining that the field to be checked belongs to a preset feature type, obtain a feature value of the field to be checked, and perform feature value check on the feature value.

On the basis of the above-mentioned embodiment, the data table detection device is further configured to integrate the detection results, and generate a detection report based on the integrated detection results and the target data table.

The data table detection apparatus provided by the embodiment of the present application can execute the data detection method provided by any embodiment of the present application, and has functional modules corresponding to the execution method.

The multiple units and modules included in the above-mentioned embodiments of the data table detection apparatus are only divided according to functional logic, but are not limited to the above-mentioned division manner; in addition, the names of all functional units are only for the convenience of distinguishing from each other.

Embodiment 3

FIG. 3 is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present application. FIG. 3 shows a block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present application. The electronic device 12 shown in FIG. 3 is only one example.

As shown in FIG. 3, the electronic device 12 takes the form of a general-purpose computing electronic device. The components of the electronic device 12 may include: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting the various system components including the system memory 28 and the processing unit 16.

System memory 28 may be memory 28 .

The bus 18 represents at least one type of bus structure, eg, the bus 18 includes a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) ) local bus and peripheral component interconnect (peripheral component interconnect, PCI) bus.

Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including both volatile and non-volatile media, removable and non-removable media.

Memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache 32 . Electronic device 12 may include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media, not shown in FIG. 3, commonly referred to as hard disk drives. Although not shown in FIG. 3, a magnetic disk drive for reading and writing to removable non-volatile magnetic disks, such as floppy disks, and an optical disk drive for reading and writing to removable non-volatile optical disks, such as removable non-volatile optical disks, may be provided For example CD-ROM, DVD-ROM or other optical media. In these cases, each drive may be connected to bus 18 through one or more data media interfaces. The memory 28 may include at least one program product having a set of, eg, at least one program module configured to perform the functions of the embodiments of the present application.

A program/utility 40 having, for example, a set of at least one program module 42, which may be stored, for example, in memory 28, such program module 42 including an operating system, one or more application programs, other program modules, and program data, in these examples Each or some combination of may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The electronic device 12 may also communicate with one or more external devices 14 (eg, a keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with the electronic device 12, and/or with Any device (eg, network card, modem, etc.) that enables the electronic device 12 to communicate with one or more other computing devices. Such communication may take place through input/output (I/O) interface 22 . Also, the electronic device 12 may communicate with one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through a network adapter 20. As shown in FIG. 3 , the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18 . Although not shown in FIG. 3, other hardware and/or software modules may be used in conjunction with electronic device 12, including: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays ( Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems, etc.

The processing unit 16 executes a variety of functional applications and sample data acquisition operations by running the program stored in the memory 28, for example, to implement the steps of a data detection method provided by the embodiment of the present application, and the data detection method includes:

Embodiment 4

The fourth embodiment provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the data detection method provided by the foregoing embodiments of the present application is implemented. For example, data exploration methods include:

The computer storage medium provided by the embodiments of the present application may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. Examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM) ), Erasable Programmable Read-Only Memory (EPROM), memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable of the above The combination. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by, or in combination with, an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied in the computer-readable signal medium. Such propagated data signals may take a variety of forms, including electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

The program code embodied on the computer-readable medium may be transmitted by any suitable medium, including: wireless, wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the above.

Computer program code for carrying out the operations of this application may be written in one or more programming languages, including object-oriented programming languages, such as Java, Smalltalk, C++, or a combination of programming languages. , but also conventional procedural programming languages - such as C or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer, such as through the Internet using an Internet service provider Connect to an external computer.

Those of ordinary skill in the art should understand that the above-mentioned multiple modules or multiple steps of the present application can be implemented by a general-purpose computing device, and they can be centralized on a single computing device or distributed on a network composed of multiple computing devices . In one embodiment, they can be implemented with program codes executable by a computer device, they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into a plurality of integrated circuit modules, or some of them can be combined. Multiple modules or steps are implemented as a single integrated circuit module. In this way, the embodiments of the present application exist in various forms of combinations of hardware and software.

Claims

A data detection method, applied to a data table detection device, includes:

matching at least one target data table from at least one database in the connected state according to the probe scope condition;

acquiring the data structure of the at least one target data table, and determining the fields to be probed in the at least one target data table according to the data structure of the at least one target data table;

Based on the preset detection rules, the fields to be detected are respectively detected, and the detection results are determined, wherein the detection rules include field filling detection rules, feature value detection rules, field length detection rules, and field dictionary code detection rules. at least one of.
The method of claim 1, before matching at least one target data table from at least one database in a connected state according to a probe scope condition, further comprising:

The configuration file is read, and the at least one database is connected according to the connection mode of the at least one database in the configuration file.
The method of claim 1, wherein the matching at least one target data table from at least one database in a connected state according to a probe scope condition comprises:

Acquire the probe scope condition and the matching mode, generate a matching instruction based on the probe scope condition and the matching mode, and execute the matching instruction to determine at least one target data table in at least one database, wherein the matching mode includes Any of exact match parameters, fuzzy match parameters, exact exclude parameters, and fuzzy exclude parameters.
The method according to claim 1, after determining the fields to be probed in the at least one target data table according to the data structure of the at least one target data table, the method further comprises:

Determining the number of fields to be probed, in response to the number of fields to be probed being greater than a preset number, sampling the data of the fields to be probed, and determining the sampled data as the corresponding fields to be probed to be probed The data.
The method according to claim 4, wherein, based on a preset detection rule, the fields to be detected are respectively detected, and the detection result is determined, comprising:

Investigate whether the probed data corresponding to the field to be probed is filled, determine the number of filled fields, and determine the percentage of the number of filled fields in the field to be probed to the number of fields to be probed, and determine the probe fill rate ;and / or,

Investigating the validity of at least one feature value corresponding to the field to be probed, determining the percentage of the number of valid feature values in the field to be probed to the number of fields to be probed, and determining the feature value coincidence rate; and /or,

Investigate the content length corresponding to the field to be probed, and determine the maximum value of the field length of the field to be probed according to the content length; and/or,

The description information of the field to be searched is searched, and the dictionary code corresponding to the field to be searched is determined according to the description information.
The method according to claim 5, wherein the detecting the validity of at least one characteristic value corresponding to the field to be detected comprises:

In response to determining that the field to be probed belongs to a preset feature type, a feature value of the field to be probed is acquired, and feature value verification is performed on the feature value.
The method of claim 1, after determining the probe result, the method further comprising:

The probe results are integrated, and a probe report is generated based on the integrated probe results and the target data table.
A data table probe device comprising:

a target data table determination module, configured to match at least one target data table from at least one database in a connected state according to the probe scope condition;

A to-be-explored field determination module, configured to acquire the data structure of the at least one target data table, and to determine the to-be-explored field in the at least one target data table according to the data structure of the at least one target data table;

The probe result determination module is configured to probe the fields to be probed respectively based on the preset probe rules, and determine the probe results, wherein the probe rules include field fill probe rules, feature value probe rules, and field length probe rules and at least one of the field dictionary code detection rules.
The data table exploration device of claim 8, further comprising:

The database connection unit is configured to read the configuration file, and connect the at least one database according to the connection mode of the at least one database in the configuration file.
The data table detection device according to claim 8, wherein the target data table determination module comprises:

A target data table determination unit, configured to acquire the probe range condition and the matching mode, generate a matching instruction based on the probe range condition and the matching mode, and execute the matching instruction to determine at least one target data table in at least one database , wherein the matching mode includes any one of exact matching parameters, fuzzy matching parameters, exact exclusion parameters and fuzzy exclusion parameters.
The data table exploration device of claim 8, further comprising:

A probe data determination unit, configured to determine the number of the fields to be probed, in response to the number of the fields to be probed being greater than a preset number, sampling the data of the fields to be probed, and determining the data obtained by sampling as the The data to be probed corresponding to the field to be probed.
The data table probe device of claim 11, wherein the probe result determination module comprises:

A first probing result determining unit, configured to probe whether the probed data corresponding to the fields to be probed is filled, determine the number of filled fields, and determine that the number of filled fields in the fields to be probed accounts for the number of fields to be probed The percentage of determining the probe fill rate;

a second probing result determining unit, configured to probe the validity of at least one feature value corresponding to the field to be probed, and determine the percentage of the number of valid feature values in the field to be probed to the number of fields to be probed, determining the eigenvalue coincidence rate;

a third detection result determination unit, configured to detect the content length corresponding to the field to be probed, and to determine the maximum value of the field length of the field to be probed according to the content length;

The fourth detection result determination unit is configured to detect the description information of the field to be searched, and determine the dictionary code corresponding to the field to be searched according to the description information.
The data table inquiry apparatus according to claim 12, wherein the second inquiry result determination unit comprises:

A feature value checking unit, configured to, in response to determining that the field to be checked belongs to a preset feature type, obtain a feature value of the field to be checked, and perform feature value checking on the feature value.
The data table exploration device according to claim 8, further configured to integrate the inspection results and generate an inspection report based on the integrated inspection results and the target data table.
An electronic device comprising:

one or more processors;

storage means arranged to store one or more programs,

The one or more processors configured to execute the one or more programs to implement the data exploration method of any one of claims 1-7.
A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the data exploration method according to any one of claims 1-7.