CN110569234A - Data checking method and device, electronic equipment and computer readable storage medium - Google Patents
Data checking method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN110569234A CN110569234A CN201910692872.1A CN201910692872A CN110569234A CN 110569234 A CN110569234 A CN 110569234A CN 201910692872 A CN201910692872 A CN 201910692872A CN 110569234 A CN110569234 A CN 110569234A
- Authority
- CN
- China
- Prior art keywords
- data
- inspection
- instruction
- verification
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data inspection method and device, electronic equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring a data inspection instruction; responding to the data inspection instruction, and acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction; inputting the multiple items of data to be detected into a hadoop data detection model; and carrying out parallel inspection processing on the multiple items of data to be inspected through a preset rule in the hadoop data inspection model. By the technical scheme, the parallel inspection of the data of different data sources is realized, the compatibility and the working efficiency of the data inspection system are improved, meanwhile, the flexible setting and the change of inspection rules are ensured in the process of the parallel inspection of the data, the flexibility and the practicability of the data inspection are further improved, and the accuracy of the data inspection is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data inspection method and apparatus, an electronic device, and a computer-readable storage medium.
Background
In the data inspection process, a simple inspection rule can be set for a certain amount of data, and whether the data meet the requirements is checked through the inspection rule.
However, as the development of scientific technology increases, the amount of concurrent data in different types of systems increases, and a simple audit rule cannot cope with a large amount of data with high concurrence. Meanwhile, due to the variety of data sources, the existing data inspection mode can only respectively inspect the data aiming at the data of different data sources. The data inspection process is time-consuming and labor-consuming, the consumption cost is extremely high, and the inspection efficiency of harvesting is low.
Therefore, how to improve the practicability of data inspection becomes a technical problem to be solved urgently at present.
Disclosure of Invention
in view of the above problems, the present invention provides a data inspection method and apparatus, an electronic device, and a computer-readable storage medium, which can perform concurrent inspection processing on data of different data sources through a hadoop framework, aiming at the technical problem of low cost efficiency in data inspection and inspection in the related art.
in a first aspect, an embodiment of the present invention provides a data inspection method, including: acquiring a data inspection instruction; responding to the data inspection instruction, and acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction; inputting the multiple items of data to be detected into a hadoop data detection model; and carrying out parallel inspection processing on the multiple items of data to be inspected through a preset rule in the hadoop data inspection model.
In the above embodiment of the present invention, optionally, the step of obtaining, in response to the data verification instruction, a plurality of items of data to be verified in a plurality of databases corresponding to the data verification instruction specifically includes: determining a target database to which the data requested to be checked by the data checking instruction belongs according to the data information in the data checking instruction; judging whether the target database is a connected database of the data inspection system; responding to the target database as the connected database, and directly acquiring corresponding data to be checked from the target database; and responding to the fact that the target database is not the connected database, establishing connection between the target database and the data inspection system through a preset data source expansion interface, and acquiring corresponding data to be inspected from the target database after the connection is established.
In the above embodiment of the present invention, optionally, before the step of obtaining the data verification instruction, the method further includes: and according to a rule parameter adjustment command, performing parameter adjustment on the preset rule in the hadoop data inspection model.
In the above embodiment of the present invention, optionally, before the step of inputting the plurality of pieces of data to be inspected into the hadoop data inspection model, the method further includes: determining a required target verification rule according to the attribute information of the plurality of items of data to be verified; judging whether the preset rules in the hadoop data inspection model contain all the target verification rules or not; wherein, in response to the preset rule containing all the target verification rules, the multiple items of data to be verified are subjected to parallel verification processing through the preset rule; in response to the predetermined rule not including all of the target validation rules, defining a number of items of the target validation rules not included in the predetermined rule through a predetermined standard packet interface.
in a second aspect, an embodiment of the present invention provides a data verification apparatus, including: the instruction acquisition unit is used for acquiring a data inspection instruction; the data acquisition unit is used for responding to the data inspection instruction and acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction; the model input unit is used for inputting the plurality of items of data to be detected into the hadoop data detection model; and the data inspection unit is used for carrying out parallel inspection processing on the multiple items of data to be inspected through a preset rule in the hadoop data inspection model.
in the above embodiment of the present invention, optionally, the data obtaining unit includes: the database determining unit is used for determining a target database to which the data requested to be checked by the data checking instruction belongs according to the data information in the data checking instruction; the first judgment unit is used for judging whether the target database is a connected database of the data inspection system; the first execution unit is used for responding to the situation that the target database is the connected database, and directly acquiring corresponding data to be checked from the target database; and the second execution unit is used for responding to the condition that the target database is not the connected database, establishing the connection between the target database and the data inspection system through a preset data source expansion interface, and acquiring corresponding data to be inspected from the target database after establishing the connection.
in the above embodiment of the present invention, optionally, the method further includes: and the rule updating unit is used for carrying out parameter adjustment on the preset rule in the hadoop data inspection model according to a rule parameter adjustment command before the instruction obtaining unit obtains the data inspection instruction.
In the above embodiment of the present invention, optionally, the method further includes: the rule determining unit is used for determining a required target verification rule according to the attribute information of the data to be detected before the data to be detected are input into the hadoop data detection model by the model input unit; the second judging unit is used for judging whether the preset rules in the hadoop data inspection model contain all the target verification rules; the data checking unit is specifically configured to: and in response to the fact that the preset rules contain all the target verification rules, performing parallel verification processing on the multiple items of data to be verified through the preset rules, and in response to the fact that the preset rules do not contain all the target verification rules, defining a plurality of items of the target verification rules which are not contained in the preset rules through a preset standard packet interface.
In a third aspect, an embodiment of the present invention provides an electronic device, including: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of the first aspects above.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the method flow described in any one of the first aspect.
The technical scheme aims at the technical problem that the data inspection cost in the related technology is high in efficiency, and particularly, when the data inspection instruction is received, the source of the data required to be inspected by the data inspection instruction can be determined, so that a plurality of corresponding data to be inspected are obtained from a plurality of databases of the source, and therefore the data of different data sources can be obtained simultaneously for inspection, the application range of the data inspection is expanded, and the data inspection efficiency is further improved.
Then, the multiple items of data to be checked are input into a hadoop data checking model, the hadoop data checking model is established on the basis of a hadoop system framework, the hadoop system framework is a software framework capable of performing distributed processing on a large amount of data, the hadoop system framework works in a parallel mode, the processing speed is accelerated through parallel processing, and meanwhile, the hadoop system framework assumes that computing elements and storage fail, so that multiple working data copies are maintained, and the redistribution processing for failed nodes is ensured. Therefore, the hadoop system framework has the advantages of high reliability, high expansibility, high efficiency, high fault tolerance and low cost. Certainly, the hadoop data inspection model is provided with preset rules, the preset rules are used for inspecting and processing a plurality of items of data to be inspected, and based on the high expansibility of the hadoop system framework, the preset rules can be added, deleted or parameters can be changed according to actual requirements.
Through the technical scheme, the parallel inspection of the data of different data sources is realized, the compatibility and the working efficiency of a data inspection system are improved, meanwhile, the flexible setting and the change of inspection rules are also ensured in the process of the parallel inspection of the data, the flexibility and the practicability of the data inspection are further improved, and the accuracy of the data inspection is favorably improved.
Drawings
in order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 shows a flow diagram of a data verification method according to one embodiment of the invention.
FIG. 2 shows a flow diagram of a data verification method according to another embodiment of the invention.
FIG. 3 shows a block diagram of a data verification device, according to one embodiment of the present invention.
FIG. 4 shows a block diagram of an electronic device according to one embodiment of the invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.
in some of the flows described in the present specification and claims and in the above figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, with the order of the operations being indicated as 101, 102, etc. merely to distinguish between the various operations, and the order of the operations by themselves does not represent any order of performance. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
FIG. 1 shows a flow diagram of a data verification method according to one embodiment of the invention.
As shown in fig. 1, a flow of a data verification method according to an embodiment of the present invention includes:
Step 102, a data verification instruction is obtained.
and 104, responding to the data inspection instruction, and acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction.
after the data inspection instruction is received, the source of the data required to be inspected by the data inspection instruction can be determined, and a plurality of corresponding data to be inspected are obtained from a plurality of databases of the source, so that the data of different data sources can be obtained simultaneously for inspection, the application range of data inspection is expanded, and the data inspection efficiency is further improved.
And 106, inputting the multiple items of data to be detected into a hadoop data detection model.
Inputting the multiple items of data to be detected into a hadoop data detection model, wherein the hadoop data detection model is established on the basis of a hadoop system frame, the hadoop system frame is a software frame capable of performing distributed processing on a large amount of data, the hadoop system frame works in a parallel mode, the processing speed is accelerated through parallel processing, and meanwhile, the hadoop system frame assumes that computing elements and storage fail, so that the hadoop system frame maintains multiple working data copies and ensures that the working data copies can be redistributed and processed aiming at failed nodes. Therefore, the hadoop system framework has the advantages of high reliability, high expansibility, high efficiency, high fault tolerance and low cost.
and 108, carrying out parallel inspection processing on the multiple items of data to be inspected through a preset rule in the hadoop data inspection model.
The hadoop data inspection model is provided with preset rules, the preset rules are used for inspecting and processing a plurality of items of data to be inspected, and based on the high expansibility of the hadoop system framework, the preset rules can be added, deleted or changed according to actual requirements.
through the technical scheme, the parallel inspection of the data of different data sources is realized, the compatibility and the working efficiency of a data inspection system are improved, meanwhile, the flexible setting and the change of inspection rules are also ensured in the process of the parallel inspection of the data, the flexibility and the practicability of the data inspection are further improved, and the accuracy of the data inspection is favorably improved.
FIG. 2 shows a flow diagram of a data verification method according to another embodiment of the invention.
As shown in fig. 2, a flow of a data verification method according to another embodiment of the present invention includes:
At step 202, a data verification instruction is obtained.
After the data inspection instruction is received, the source of the data required to be inspected by the data inspection instruction can be determined, and a plurality of corresponding data to be inspected are obtained from a plurality of databases of the source, so that the data of different data sources can be obtained simultaneously for inspection, the application range of data inspection is expanded, and the data inspection efficiency is further improved.
And step 204, determining a target database to which the data requested to be checked by the data checking instruction belongs according to the data information in the data checking instruction.
Step 206, judging whether the target database is a connected database of the data inspection system, if so, entering step 208, otherwise, entering step 210.
in this case, the target database to which the data requested to be checked by the data checking instruction belongs may not be a connected database of the data checking system, and in this case, the target database needs to be connected in real time based on the high expansibility of the hadoop system framework, so as to ensure the validity of the data checking.
and step 208, directly acquiring corresponding data to be checked from the target database.
Step 210, establishing connection between the target database and the data inspection system through a predetermined data source expansion interface.
step 212, obtaining the corresponding data to be checked from the target database after establishing the connection.
the preset data source expansion interfaces are reserved in the data inspection system, the number of the preset data source expansion interfaces is one or more, and when a target database which is not connected with the database is to be connected, a new connection can be established between the preset data source expansion interface and the target database, so that the data to be inspected in the target database can be successfully acquired, and the validity of the data inspection result is ensured.
Step 214, determining the required target verification rule according to the attribute information of the plurality of items of data to be verified.
Different sources and different types of data have different rules for verification, so that the target verification rule matched with the attribute information of each item of data to be verified can be determined according to the attribute information of the items of data to be verified.
In an implementation manner of the present invention, optionally, the method further includes: and according to a rule parameter adjustment command, performing parameter adjustment on the preset rule in the hadoop data inspection model. That is, the actual verification parameters of each predetermined rule can be adjusted according to the actual requirements, so as to adapt to the data inspection requirements of the user to the maximum extent.
step 216, determining whether the predetermined rule in the hadoop data inspection model includes all the target verification rules, if so, entering step 218, otherwise, entering step 220.
And step 218, performing parallel inspection processing on the plurality of items of data to be inspected through the preset rule.
If the preset rules in the hadoop data inspection model comprise all the target verification rules, and the rules do not need to be added, the multiple items of data to be inspected can be subjected to parallel inspection processing directly through the preset rules.
Step 220, defining a plurality of target verification rules which are not included in the predetermined rules through a predetermined standard packet interface.
if the predetermined rule in the hadoop data inspection model does not include all the target verification rules, it indicates that at least one target verification rule required by the inspection is not set in the hadoop data inspection model, at this time, the at least one target verification rule needs to be defined in real time through a predetermined standard packet interface, and then the defined target verification rule is added for use.
Through the technical scheme, the flexibility and the practicability of the setting of the data inspection verification rule are improved, the concurrent, flexible and accurate inspection can be performed on the data of different data sources, and the practicability and the efficiency of the data inspection are greatly improved.
FIG. 3 shows a block diagram of a data verification device, according to one embodiment of the present invention.
As shown in fig. 3, a data verifying apparatus 300 according to an embodiment of the present invention includes: an instruction obtaining unit 302, configured to obtain a data verification instruction; a data obtaining unit 304, configured to, in response to the data checking instruction, obtain a plurality of items of data to be checked in a plurality of databases corresponding to the data checking instruction; the model input unit 306 is configured to input the plurality of items of data to be checked into a hadoop data check model; and the data inspection unit 308 is used for performing parallel inspection processing on the plurality of items of data to be inspected through a preset rule in the hadoop data inspection model.
The data checking device 300 uses the scheme described in any one of the embodiments shown in fig. 1 and fig. 2, and therefore, has all the technical effects described above, and will not be described again. The data verification device 300 also has the following technical features:
In the above embodiment of the present invention, optionally, the data obtaining unit 304 includes: the database determining unit is used for determining a target database to which the data requested to be checked by the data checking instruction belongs according to the data information in the data checking instruction; the first judgment unit is used for judging whether the target database is a connected database of the data inspection system; the first execution unit is used for responding to the situation that the target database is the connected database, and directly acquiring corresponding data to be checked from the target database; and the second execution unit is used for responding to the condition that the target database is not the connected database, establishing the connection between the target database and the data inspection system through a preset data source expansion interface, and acquiring corresponding data to be inspected from the target database after establishing the connection.
In the above embodiment of the present invention, optionally, the method further includes: a rule updating unit, configured to perform parameter adjustment on the predetermined rule in the hadoop data inspection model according to a rule parameter adjustment command before the instruction obtaining unit 302 obtains the data inspection instruction.
In the above embodiment of the present invention, optionally, the method further includes: a rule determining unit, configured to determine a required target verification rule according to attribute information of the multiple items of data to be verified before the model input unit 306 inputs the multiple items of data to be verified into the hadoop data verification model; the second judging unit is used for judging whether the preset rules in the hadoop data inspection model contain all the target verification rules; the data checking unit 308 is specifically configured to: and in response to the fact that the preset rules contain all the target verification rules, performing parallel verification processing on the multiple items of data to be verified through the preset rules, and in response to the fact that the preset rules do not contain all the target verification rules, defining a plurality of items of the target verification rules which are not contained in the preset rules through a preset standard packet interface.
FIG. 4 shows a block diagram of an electronic device according to one embodiment of the invention.
An electronic device is further provided in the embodiment of the present invention, as shown in fig. 4, for convenience of description, only a portion related to the embodiment of the present invention is shown, and details of the specific technology are not disclosed, please refer to the method portion in the embodiment of the present invention. The electronic device may be any electronic device including a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (point of Sales), a vehicle-mounted computer, and the like, taking the electronic device as the mobile phone as an example:
fig. 4 is a block diagram illustrating a partial structure of a mobile phone related to an electronic device provided by an embodiment of the present invention. Referring to fig. 4, the handset includes: radio Frequency (RF) circuitry 410, memory 420, input unit 430, display unit 440, sensor 450, audio circuitry 460, wireless-fidelity (Wi-Fi) module 470, processor 480, and power supply 490. Those skilled in the art will appreciate that the handset configuration shown in fig. 4 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
The following describes each component of the mobile phone in detail with reference to fig. 4:
The RF circuit 410 may be used for receiving and transmitting signals during information transmission and reception or during a call, and in particular, receives downlink information of a base station and then processes the received downlink information to the processor 480; in addition, the data for designing uplink is transmitted to the base station. In general, the RF circuit 410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuitry 410 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to global system for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), and the like.
The memory 420 may be used to store software programs and modules, and the processor 480 executes various functional applications and data processing of the mobile phone by operating the software programs and modules stored in the memory 420. The memory 420 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 420 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
the input unit 430 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the cellular phone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also called a touch screen, may collect touch operations of a user on or near the touch panel 431 (e.g., operations of the user on or near the touch panel 431 using any suitable object or accessory such as a finger or a stylus) and drive the corresponding connection device according to a preset program. Alternatively, the touch panel 431 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 480, and receives and executes commands sent from the processor 480. In addition, the touch panel 431 may be implemented in various types, such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit 430 may include other input devices 432 in addition to the touch panel 431. In particular, other input devices 432 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit 440 may be used to display information input by the user or information provided to the user and various menus of the cellular phone. The display unit 440 may include a display panel 441, and optionally, the display panel 441 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel 431 may cover the display panel 441, and when the touch panel 431 detects a touch operation on or near the touch panel 431, the touch panel is transmitted to the processor 480 to determine the type of the touch event, and then the processor 480 provides a corresponding visual output on the display panel 441 according to the type of the touch event. Although the touch panel 431 and the display panel 441 are shown in fig. 4 as two separate components to implement the input and output functions of the mobile phone, in some embodiments, the touch panel 431 and the display panel 441 may be integrated to implement the input and output functions of the mobile phone.
the handset may also include at least one sensor 450, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display panel 441 according to the brightness of ambient light, and a proximity sensor that turns off the display panel 441 and/or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured on the mobile phone, further description is omitted here.
Audio circuit 460, speaker 461, microphone 462 may provide an audio interface between the user and the cell phone. The audio circuit 460 may transmit the electrical signal converted from the received audio data to the speaker 461, and convert the electrical signal into a sound signal for output by the speaker 461; on the other hand, the microphone 462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 460 and converted into audio data, which is then processed by the audio data output processor 480 and then transmitted to, for example, another cellular phone via the RF circuit 410, or output to the memory 420 for further processing.
WiFi belongs to short-distance wireless transmission technology, and the mobile phone can help a user to receive and send e-mails, browse webpages, access streaming media and the like through the WiFi module 470, and provides wireless broadband Internet access for the user. Although fig. 4 shows the WiFi module 470, it is understood that it does not belong to the essential constitution of the handset, and can be omitted entirely as needed within the scope not changing the essence of the invention.
The processor 480 is a control center of the mobile phone, connects various parts of the entire mobile phone by using various interfaces and lines, and performs various functions of the mobile phone and processes data by operating or executing software programs and/or modules stored in the memory 420 and calling data stored in the memory 420, thereby integrally monitoring the mobile phone. Optionally, processor 480 may include one or more processing units; preferably, the processor 480 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 480.
The handset also includes a power supply 490 (e.g., a battery) for powering the various components, which may preferably be logically connected to the processor 480 via a power management system, so that the power management system may perform functions such as managing charging, discharging, and power consumption.
Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which are not described herein.
In this embodiment of the present invention, the processor 480 included in the electronic device further has the following functions: acquiring a data inspection instruction; responding to the data inspection instruction, and acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction; inputting the multiple items of data to be detected into a hadoop data detection model; and carrying out parallel inspection processing on the multiple items of data to be inspected through a preset rule in the hadoop data inspection model.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic or optical disk, or the like.
It will be understood by those skilled in the art that all or part of the steps in the method for implementing the above embodiments may be implemented by hardware that is instructed to implement by a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
While the electronic device provided by the present invention has been described in detail, those skilled in the art will appreciate that the various modifications, additions, substitutions, and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.
Claims (10)
1. A method of data verification, comprising:
acquiring a data inspection instruction;
Responding to the data inspection instruction, and acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction;
Inputting the multiple items of data to be detected into a hadoop data detection model;
And carrying out parallel inspection processing on the multiple items of data to be inspected through a preset rule in the hadoop data inspection model.
2. the data inspection method according to claim 1, wherein the step of acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction in response to the data inspection instruction specifically comprises:
Determining a target database to which the data requested to be checked by the data checking instruction belongs according to the data information in the data checking instruction;
judging whether the target database is a connected database of the data inspection system;
Responding to the target database as the connected database, and directly acquiring corresponding data to be checked from the target database;
And responding to the fact that the target database is not the connected database, establishing connection between the target database and the data inspection system through a preset data source expansion interface, and acquiring corresponding data to be inspected from the target database after the connection is established.
3. The data verification method of claim 1, further comprising, prior to the step of obtaining data verification instructions:
and according to a rule parameter adjustment command, performing parameter adjustment on the preset rule in the hadoop data inspection model.
4. the data inspection method according to any one of claims 1 to 3, further comprising, before the step of inputting the plurality of items of data to be inspected into a hadoop data inspection model:
Determining a required target verification rule according to the attribute information of the plurality of items of data to be verified;
judging whether the preset rules in the hadoop data inspection model contain all the target verification rules or not;
Wherein the content of the first and second substances,
In response to the fact that the preset rules contain all the target verification rules, performing parallel verification processing on the multiple items of data to be verified through the preset rules;
In response to the predetermined rule not including all of the target validation rules, defining a number of items of the target validation rules not included in the predetermined rule through a predetermined standard packet interface.
5. A data verification apparatus, comprising:
the instruction acquisition unit is used for acquiring a data inspection instruction;
The data acquisition unit is used for responding to the data inspection instruction and acquiring a plurality of items of data to be inspected in a plurality of databases corresponding to the data inspection instruction;
The model input unit is used for inputting the plurality of items of data to be detected into the hadoop data detection model;
And the data inspection unit is used for carrying out parallel inspection processing on the multiple items of data to be inspected through a preset rule in the hadoop data inspection model.
6. The data verification apparatus according to claim 5, wherein the data acquisition unit includes:
The database determining unit is used for determining a target database to which the data requested to be checked by the data checking instruction belongs according to the data information in the data checking instruction;
The first judgment unit is used for judging whether the target database is a connected database of the data inspection system;
The first execution unit is used for responding to the situation that the target database is the connected database, and directly acquiring corresponding data to be checked from the target database;
and the second execution unit is used for responding to the condition that the target database is not the connected database, establishing the connection between the target database and the data inspection system through a preset data source expansion interface, and acquiring corresponding data to be inspected from the target database after establishing the connection.
7. The data verification apparatus of claim 5, further comprising:
And the rule updating unit is used for carrying out parameter adjustment on the preset rule in the hadoop data inspection model according to a rule parameter adjustment command before the instruction obtaining unit obtains the data inspection instruction.
8. The data verification device of any one of claims 5 to 7, further comprising:
The rule determining unit is used for determining a required target verification rule according to the attribute information of the data to be detected before the data to be detected are input into the hadoop data detection model by the model input unit;
the second judging unit is used for judging whether the preset rules in the hadoop data inspection model contain all the target verification rules;
The data checking unit is specifically configured to:
and in response to the fact that the preset rules contain all the target verification rules, performing parallel verification processing on the multiple items of data to be verified through the preset rules, and in response to the fact that the preset rules do not contain all the target verification rules, defining a plurality of items of the target verification rules which are not contained in the preset rules through a preset standard packet interface.
9. An electronic device, comprising:
One or more processors;
A memory;
One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of claims 1-4.
10. A computer-readable storage medium having stored thereon computer-executable instructions for performing the method flow of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910692872.1A CN110569234A (en) | 2019-07-30 | 2019-07-30 | Data checking method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910692872.1A CN110569234A (en) | 2019-07-30 | 2019-07-30 | Data checking method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110569234A true CN110569234A (en) | 2019-12-13 |
Family
ID=68773635
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910692872.1A Pending CN110569234A (en) | 2019-07-30 | 2019-07-30 | Data checking method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110569234A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391903A (en) * | 2014-11-14 | 2015-03-04 | 广州科腾信息技术有限公司 | Distributed storage and parallel calculation-based power grid data quality detection method |
| CN107092694A (en) * | 2017-04-25 | 2017-08-25 | 杭州数梦工场科技有限公司 | The inspection task creating method and device of the quality of data |
| CN107818509A (en) * | 2017-11-24 | 2018-03-20 | 泰康保险集团股份有限公司 | Business datum method of calibration, device, storage medium and electronic equipment |
| CN108647358A (en) * | 2018-05-17 | 2018-10-12 | 东软集团股份有限公司 | Quality of data method of calibration, device, storage medium and electronic equipment |
| CN109656917A (en) * | 2018-12-18 | 2019-04-19 | 深圳前海微众银行股份有限公司 | Data detection method, device, equipment and the readable storage medium storing program for executing of multi-data source |
| CN110020381A (en) * | 2018-02-23 | 2019-07-16 | 中国平安财产保险股份有限公司 | Method of calibration, device, equipment and computer storage medium based on configuration file |
-
2019
- 2019-07-30 CN CN201910692872.1A patent/CN110569234A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391903A (en) * | 2014-11-14 | 2015-03-04 | 广州科腾信息技术有限公司 | Distributed storage and parallel calculation-based power grid data quality detection method |
| CN107092694A (en) * | 2017-04-25 | 2017-08-25 | 杭州数梦工场科技有限公司 | The inspection task creating method and device of the quality of data |
| CN107818509A (en) * | 2017-11-24 | 2018-03-20 | 泰康保险集团股份有限公司 | Business datum method of calibration, device, storage medium and electronic equipment |
| CN110020381A (en) * | 2018-02-23 | 2019-07-16 | 中国平安财产保险股份有限公司 | Method of calibration, device, equipment and computer storage medium based on configuration file |
| CN108647358A (en) * | 2018-05-17 | 2018-10-12 | 东软集团股份有限公司 | Quality of data method of calibration, device, storage medium and electronic equipment |
| CN109656917A (en) * | 2018-12-18 | 2019-04-19 | 深圳前海微众银行股份有限公司 | Data detection method, device, equipment and the readable storage medium storing program for executing of multi-data source |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104852885B (en) | Method, device and system for verifying verification code | |
| CN108268366B (en) | Test case execution method and device | |
| CN104516812A (en) | Method and device for testing software | |
| CN106649126B (en) | Method and device for testing application program | |
| CN108073495B (en) | Method and device for positioning crash reason of application program | |
| CN111078556B (en) | Application testing method and device | |
| CN110597793A (en) | Data management method and device, electronic equipment and computer readable storage medium | |
| WO2018161540A1 (en) | Fingerprint registration method and related product | |
| US10324134B2 (en) | Method and device for ascertaining required charging time | |
| CN104809055B (en) | Application program testing method and device based on cloud platform | |
| CN110213313B (en) | File uploading method, related device and system | |
| CN115904950A (en) | Test case generation method, device, equipment and storage medium | |
| CN105550316A (en) | Pushing method and device of audio list | |
| CN116303085A (en) | Test reason analysis method, device, equipment and storage medium | |
| CN106777383B (en) | File sorting method and intelligent terminal | |
| CN115794654A (en) | Test case distribution processing method, system, equipment and storage medium | |
| CN114155977A (en) | Data processing method and device for clinical research project | |
| CN114049195A (en) | Square carton splitting method and device | |
| CN110569234A (en) | Data checking method and device, electronic equipment and computer readable storage medium | |
| CN107302446B (en) | Banknote-based algorithm simulation verification method, client and terminal equipment | |
| CN114490307A (en) | Unit testing method, device and storage medium | |
| CN108073508B (en) | Compatibility detection method and device | |
| CN106610971A (en) | Identifier determination method and apparatus for ZIP files | |
| CN113961380A (en) | Cross-application repair method, device, equipment and storage medium | |
| CN112667868A (en) | Data detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191213 |