CN111143434A - Intelligent data checking method, device, equipment and storage medium - Google Patents
Intelligent data checking method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111143434A CN111143434A CN201911315244.8A CN201911315244A CN111143434A CN 111143434 A CN111143434 A CN 111143434A CN 201911315244 A CN201911315244 A CN 201911315244A CN 111143434 A CN111143434 A CN 111143434A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- data file
- preset
- service field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000013515 script Methods 0.000 claims abstract description 45
- 238000012544 monitoring process Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 abstract description 2
- 230000003252 repetitive effect Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000004801 process automation Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an intelligent data checking method, and belongs to the technical field of data reports. The method comprises the following steps: executing a preset event operation script, and downloading a first data file and a second data file to be checked; reading the first data file and the second data file line by line, and analyzing data in the first data file and the second data file into the same data check file according to a preset file format, wherein the data check file comprises a table, and the table comprises a service field and a corresponding value; and calling a value of a preset service field from the data check file for checking according to the preset service field, and writing the checking result into the data check file. The RPA technology is adopted to replace a repetitive manual processing process, so that the dependence on manual operation is reduced, and the risk possibly brought by the manual operation is effectively reduced.
Description
Technical Field
The invention relates to the technical field of data reports, in particular to a method, a device, equipment and a storage medium for intelligently checking data.
Background
In various service scenarios, data is usually required to be checked, especially, some services involve checking of multiple system data, that is, involve different data sources, and due to factors such as recording mode and recording time, results of different data sources for the same data recording are different, which requires checking of data recorded by different data sources, thereby finding potential risks caused by data recording errors.
In the prior art, data checking is usually performed for T +1 day, and first, each service system needs to be manually logged in, data of each service system for T day is downloaded, and the data is summarized and compared first and then item by item. Because the operation of the existing manual checking process relates to account login of multiple systems, the process is complicated and long, the risk of manual operation is high, the automation efficiency is low, and the stagnation and delay of the associated business process are easily caused under the condition of abnormity or negligence, so the business circulation efficiency is low, and the manpower is consumed.
Disclosure of Invention
The technical problem to be solved by the invention is to overcome the problem of high operational risk of manually checking data in the prior art, and provide an intelligent data checking method based on RPA (robot process automation), an electronic device, a computer device and a storage medium.
The invention solves the technical problems through the following technical scheme:
an intelligent data checking method comprises the following steps:
executing a preset event operation script, and downloading a first data file and a second data file to be checked;
reading the first data file and the second data file line by line, and analyzing data in the first data file and the second data file into the same data check file according to a preset file format, wherein the data check file comprises a table, and the table comprises a service field and a corresponding value;
and calling a value corresponding to the preset service field from the data check file for checking according to the preset service field, and writing the checking result into the data check file.
Preferably, the generation of the event operation script comprises the following steps:
acquiring an operation event and an operation sequence corresponding to the operation event in the process of manually downloading the first data file and the second data file in an event monitoring mode;
and respectively and automatically generating event operation scripts corresponding to the first data file and the second data file according to the acquired operation events and the operation sequence.
Preferably, the reading the first data file and the second data file line by line, and parsing the data in the first data file and the second data file to the same data check file according to a preset file format includes the following steps:
reading each line of the first data file and the second data file line by using a method in a python native package;
respectively identifying the value under each service field in each line of the first data file and the second data file according to a preset service field sequence;
and writing the values under the service fields in the first data file and the second data file into the same data check file.
Preferably, the writing of the values in the service fields of the first data file and the second data file into the same data check file includes the following steps:
judging whether the same identification value is recorded in the first data file and the second data file under an identification service field;
when the same identification value is recorded, the service field writes the value under each service field in the row where the same identification value in the first data file and the second data file is located, into the same row in the same data collation file corresponding to each service field.
Preferably, the values under each service field in the first data file and the second data file are separated by a preset separator.
Preferably, the automatically generating an event operation script according to the acquired operation event and the operation sequence includes the following steps:
calling a preset implementation method code corresponding to the type of the operation event according to the acquired operation event;
writing the called codes into corresponding service fields of a preset script file;
and when the operation event contains input content, writing the input content into a corresponding service field of the code.
The invention also discloses a data intelligent checking device, which comprises:
the file downloading module is used for executing a preset event operation script and downloading a first data file and a second data file to be checked;
the file analysis module is used for reading the first data file and the second data file line by line, and analyzing data in the first data file and the second data file into the same data check file according to a preset file format, wherein the data check file comprises a table, and the table comprises a service field and a corresponding value;
and the data checking module is used for calling a value corresponding to a preset service field from the data checking file for checking according to the preset service field, and writing the checking result into the data checking file.
Preferably, the apparatus further comprises:
the script generation module is used for acquiring operation events and operation sequences corresponding to the operation events in the process of manually downloading the first data file and the second data file in an event monitoring mode; and respectively and automatically generating event operation scripts corresponding to the first data file and the second data file according to the acquired operation events and the operation sequence.
The invention also discloses computer equipment which comprises a memory and a processor, wherein the memory is stored with a computer program, and the computer program realizes the steps of the intelligent data checking method when being executed by the processor.
The invention also discloses a computer readable storage medium, which stores a computer program, wherein the computer program can be executed by at least one processor to realize the steps of the aforementioned data intelligent checking method.
The positive progress effects of the invention are as follows: by adopting the RPA technology to replace a repetitive manual processing process, the dependence on manual operation is reduced, and the risk possibly brought by the manual operation is effectively reduced.
Drawings
FIG. 1 is a flow chart of a first embodiment of the intelligent data checking method of the present invention;
FIG. 2 is a flow chart of a second embodiment of the intelligent data checking method of the present invention;
FIG. 3 is a block diagram showing a first embodiment of the intelligent collating device for data of the present invention;
FIG. 4 is a block diagram showing a second embodiment of the intelligent collating device for data of the present invention;
fig. 5 shows a hardware architecture diagram of an embodiment of the computer apparatus of the present invention.
Detailed Description
The invention is further illustrated by the following examples, which are not intended to limit the scope of the invention.
Firstly, the invention provides an intelligent data checking method.
In one embodiment, as shown in fig. 1, the intelligent data checking method includes the following steps:
step 10: and executing a preset event operation script, and downloading a first data file and a second data file to be checked.
The event operation script records codes simulating the operation process of manually downloading the data file, and the required file can be automatically downloaded by executing the codes. The operation process may specifically include logging in a platform, selecting downloaded data, clicking to download a generated file, and the like, and the whole operation process may involve operations of a mouse and a keyboard, which are all recorded in an event operation script in the form of codes, and the required data file may be automatically downloaded by running the codes.
Each type of data file corresponds to a preset event operation script, namely one set of event operation scripts can only realize the downloading of one data file. Therefore, here, one event operation script needs to be preset for downloading the first data file and the second data file, respectively, that is, two event operation scripts need to be preset in total.
The first data file can be an online banking reconciliation file, the second data file can be a loan reconciliation file, the two data files are usually stored in different platforms, so that operation permissions of the two platforms are required for downloading the two data files, the data leakage risk is easily increased if the same operator has the operation permissions of the two platforms, and the operation flow is complicated and long if the two operators respectively operate the two platforms to download the data files. The data file is automatically downloaded by executing the event operation script, so that the risk of data leakage is greatly avoided, and the operation flow is simplified.
Step 20: reading the first data file and the second data file line by line, and analyzing data in the first data file and the second data file to the same data check file according to a preset file format, wherein the data check file comprises a table, and the table comprises a service field and a corresponding value.
The step 20 specifically comprises the following steps:
step 21: each row of the first data file and the second data file is read line by line using the method in python native packages.
For example, a readline function in the python native package is used to implement a line-by-line reading function, when line break marks 'n', 'r' (enter) are read, line breaks are followed, data of the line is returned in a character string form, and null is returned when all data are read, so that data of each line can be distinguished.
In order to distinguish the values, the values in the service fields in the first data file and the second data file are separated by a preset separator, for example, a separator "|", and the separators are also shown in the character string when reading line by line. Here, the traffic field is a header of each column.
Step 22: and respectively identifying the value under each service field in each line in the first data file and the second data file according to a preset service field sequence aiming at each read line.
The service field sequence refers to the sequence of the header, and by taking the preset service field sequence of 'merchant order number, service serial number, transaction type, transaction time, transaction amount, customer name' as an example, the value before the first separator in each row can be distinguished from the service field 'merchant order number' by identifying the separator, the value between the first separator and the second separator is distinguished from the service field 'service serial number', and so on, the value under each service field in each row in the file is distinguished.
Step 23: and writing the values under the service fields in the first data file and the second data file into the same data check file.
When the same data check file is written, the data from two data files need to be processed in parallel, which specifically comprises the following steps:
step 231: it is determined whether the same identification value is recorded under the identification service field in the first data file and the second data file, and if the same identification value is recorded in the service field, the next step 232 is executed.
The service identification field may be agreed upon according to actual situations, for example, when the first data file and the second data file are an online banking reconciliation file and a loan reconciliation file, respectively, the service field "contract number" may be agreed upon as the service identification field, that is, when the values under the service field "contract number" in the two files are the same, the next step 232 is executed.
Step 232: and writing the value under each service field in the row where the same identification value in the first data file and the second data file is located into the same row in the same data check file corresponding to each service field.
Each service field in the data check file is set corresponding to each service field in the first data file and the second data file, and the same service field is merged and recorded only once, for example, both the first data file and the second data file have a contract number for identifying the service field, and the contract number for identifying the service field in the data check file only appears once, that is, only occupies one service field, and the corresponding identification value is recorded only once.
Step 30: and calling a value corresponding to the preset service field from the data check file for checking according to the preset service field, and writing the checking result into the data check file.
The business fields to be checked are preset according to business requirements, for example, when the first data file and the second data file are an online banking reconciliation file and a loan reconciliation file respectively, the number of records in the two files and the total amount are compared preliminarily. If the preliminary comparison result is inconsistent, the running records can be compared one by one, for example, corresponding records from two files are read in sequence according to the contract number, if the running information exists on the two sides, the values in the money amount service fields are compared to determine whether the values are different, if so, the difference condition is recorded in the difference service fields in sequence, and [ money amount difference ] characters are recorded in the difference reason service fields; if there is running information and there is no running information, then the [ unilateral account ] word is recorded under the field of the business of the difference reason. The difference condition is usually difference statistics, the difference reason is filled in association according to judgment, if yes, whether running water information from two files corresponding to the same contract number can be read or not is further judged, and if yes, the difference condition is further judged, and if yes, the amount difference is filled in; and if only the stream information from each file corresponding to the same contract number can be read, filling the unilateral account.
All the checked results are recorded in corresponding service fields in the data check file, and finally, a data table capable of displaying the data records and the check results item by item is presented.
In the second embodiment, based on the first embodiment, as shown in fig. 2, the intelligent data checking method includes the following steps:
step 01: and respectively acquiring an operation event and an operation sequence corresponding to the operation event in the process of manually downloading the first data file and the second data file in an event monitoring mode.
The monitoring utilizes a Pyhook interface to create event monitoring of a windows operating system, wherein the Pyhook is a Python-based hook library and is mainly used for monitoring the current mouse and keyboard events on a computer. The mouse hooks are used for receiving mouse events, can call back required events, and can selectively receive only left button pressing events, mouse moving events, or all mouse events and the like according to requirements; the Keyboard Hooks are used for receiving Keyboard events, the operation mode is the same as that of Mouse Hooks, and only returned information is different.
And returning information such as mouse coordinates, mouse click buttons, keyboard click buttons, menu handle acquisition, pull-down bar handle acquisition and the like and events such as execution sequence and the like by monitoring operations of executing a login platform, downloading a data file, saving the data file and the like of a user.
Step 02: and respectively and automatically generating event operation scripts corresponding to the first data file and the second data file according to the acquired operation events and the operation sequence.
The step 02 specifically comprises the following steps:
step 021: and calling a preset implementation method code corresponding to the type of the operation event according to the acquired operation event.
The automatic generation of the script requires setting corresponding implementation method codes in advance according to different types of operation events, for example: the mouse right key operation event corresponds to the code MouseRightClick (), and the keyboard input operation event corresponds to KeyboardInput ($ para).
Step 022: and writing the called codes into corresponding service fields of a preset script file.
When a user clicks a mouse or a keyboard, Pyhook captures an operation event, and a method code corresponding to the event is written into a corresponding service field of the script file.
Step 023: and when the operation event contains input content, writing the input content into a corresponding service field of the code.
When the user taps the keyboard to input the content, the content input by the user is transmitted into the corresponding service field of the code corresponding to the keyboard input operation event through the $ para.
The aforementioned steps 021 and 023 are executed circularly, and each time an operation event is executed, a part of codes corresponding to the operation event are automatically generated, until the user completes all operations, an event operation script for automatically downloading a data file is generated.
Steps 10 to 30 are the same as those in the first embodiment, and are not described herein again.
Secondly, the invention provides a data intelligent checking device, and the device 20 can be divided into one or more modules.
For example, fig. 3 shows a block diagram of a first embodiment of the intelligent data collation apparatus 20, and in this embodiment, the apparatus 20 may be divided into a file download module 201, a file parsing module 202 and a data collation module 203. The following description will specifically describe the specific functions of the module 201 and 203.
The file downloading module 201 is configured to execute a preset event operation script, and download a first data file and a second data file to be checked.
The event operation script records codes simulating the operation process of manually downloading the data file, and the required file can be automatically downloaded by executing the codes. The operation process may specifically include logging in a platform, selecting downloaded data, clicking to download a generated file, and the like, and the whole operation process may involve operations of a mouse and a keyboard, which are all recorded in an event operation script in the form of codes, and the required data file may be automatically downloaded by running the codes.
An event operation script needs to be preset corresponding to each data file to be downloaded, namely, one set of event operation script can only realize the downloading of one data file. Therefore, here, one event operation script needs to be preset for downloading the first data file and the second data file, respectively, that is, two event operation scripts need to be preset in total.
The file parsing module 202 is configured to read the first data file and the second data file line by line, and parse data in the first data file and the second data file into a same data check file according to a preset file format, where the data check file includes a table, and the table includes a service field and a corresponding value.
This module 202 reads each row of the first data file and the second data file line by line using the method in python native packages.
For example, a readline function in the python native package is used to implement a line-by-line reading function, when line break marks 'n', 'r' (enter) are read, line breaks are followed, data of the line is returned in a character string form, and null is returned when all data are read, so that data of each line can be distinguished.
In order to distinguish the values, the values in the service fields in the first data file and the second data file are separated by a preset separator, for example, a separator "|", and the separators are also shown in the character string when reading line by line.
And then, for each read line, respectively identifying a value under each service field in each line in the first data file and the second data file according to a preset service field sequence.
By taking the preset sequence of business fields, namely 'merchant order number, business serial number, transaction type, transaction time, transaction amount and customer name', the value before the first separator in each row can be identified to correspond to the business field 'merchant order and number' by identifying the separator, the value between the first separator and the second separator corresponds to the business field 'business serial number', and so on, the value under each business field in each row in the file is identified.
And then writing the values under the service fields in the first data file and the second data file into the same data check file.
When writing in the same data check file, the data from two data files need to be processed in parallel: by judging whether the first data file and the second data file have the same identification value or not, wherein the identification value is recorded under an identification service field, the identification service field can be agreed according to the actual situation, for example, when the first data file and the second data file are an internet bank account checking file and a loan account checking file respectively, a service field 'contract number' can be agreed as the identification service field, that is, when the values under the service fields 'contract numbers' in the two files are the same, the value under each service field in the row where the same identification value is located in the first data file and the second data file is written into the same row in the same data check file corresponding to each service field, that is, each service field in the data check file is set corresponding to each service field in the first data file and the second data file, the same service field is merged and recorded only once, for example, in the data check file, the service field is identified to occupy only one service field, and the identification value is recorded only once.
The data checking module 203 is configured to retrieve a value corresponding to a preset service field from the data checking file for checking according to the preset service field, and write the checking result into the data checking file.
The business fields to be checked are preset according to business requirements, for example, when the first data file and the second data file are an online banking reconciliation file and a loan reconciliation file respectively, the number of records in the two files and the total amount are compared preliminarily. If the initial comparison result is inconsistent, the running records can be compared one by one, for example, corresponding records from two files are read according to the contract number, if the running information exists on both sides, the money amount is compared to determine whether the money amount is different, the difference condition is recorded, and account checking difference reason records (money amount difference) are recorded; if there is one side and there is no one side, record [ one-side account ].
All the checked results are recorded in corresponding service fields in the data check file, and finally, a data table capable of displaying the data records and the check results item by item is presented.
For another example, fig. 4 shows a block diagram of a second embodiment of the intelligent data checking device 20, in this embodiment, the intelligent data checking device 20 may be further divided into a file downloading module 201, a file analyzing module 202, a data checking module 203 and a script generating module 204.
The modules 201 and 203 are the same as those of the first embodiment, and are not described herein again.
The script generating module 204 is configured to obtain an operation event and an operation sequence corresponding to the operation event in the process of manually downloading the first data file and the second data file in an event monitoring manner; and respectively and automatically generating event operation scripts corresponding to the first data file and the second data file according to the acquired operation events and the operation sequence.
The monitoring utilizes a Pyhook interface to create event monitoring of a windows operating system, wherein the Pyhook is a Python-based hook library and is mainly used for monitoring the current mouse and keyboard events on a computer. The mouse hooks are used for receiving mouse events, can call back required events, and can selectively receive only left button pressing events, mouse moving events, or all mouse events and the like according to requirements; the Keyboard Hooks are used for receiving Keyboard events, the operation mode is the same as that of Mouse Hooks, and only returned information is different.
And returning information such as mouse coordinates, mouse click buttons, keyboard click buttons, menu handle acquisition, pull-down bar handle acquisition and the like and events such as execution sequence and the like by monitoring operations of executing a login platform, downloading a data file, saving the data file and the like of a user.
The automatic generation of the script requires setting corresponding implementation method codes in advance according to different types of operation events, for example: the mouse right key operation event corresponds to the code MouseRightClick (), and the keyboard input operation event corresponds to KeyboardInput ($ para). When a user clicks a mouse or a keyboard, Pyhook captures an operation event, and a method code corresponding to the event is written into a corresponding service field of the script file. When the user taps the keyboard to input the content, the content input by the user is transmitted into the corresponding service field of the code corresponding to the keyboard input operation event through the $ para. And automatically generating a part of codes corresponding to the operation event every time an operation event is executed, and generating an event operation script for automatically downloading a data file until the user finishes all operations.
The invention further provides computer equipment.
Fig. 5 is a schematic diagram of a hardware architecture of an embodiment of the computer device according to the present invention. In the present embodiment, the computer device 2 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a preset or stored instruction. For example, the server may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a rack server (including an independent server or a server cluster composed of a plurality of servers). As shown, the computer device 2 includes, but is not limited to, at least a memory 21, a processor 22, and a network interface 23 communicatively coupled to each other via a system bus. Wherein:
the memory 21 includes at least one type of computer-readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the computer device 2, such as a hard disk or a memory of the computer device 2. In other embodiments, the memory 21 may also be an external storage device of the computer device 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device 2. Of course, the memory 21 may also comprise both an internal storage unit of the computer device 2 and an external storage device thereof. In this embodiment, the memory 21 is generally used for storing an operating system installed in the computer device 2 and various types of application software, such as a computer program for implementing the intelligent data checking method. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is generally configured to control the overall operation of the computer device 2, such as performing control and processing related to data interaction or communication with the computer device 2. In this embodiment, the processor 22 is configured to run a program code stored in the memory 21 or process data, for example, run a computer program for implementing the intelligent data checking method.
The network interface 23 may comprise a wireless network interface or a wired network interface, and the network interface 23 is typically used to establish a communication connection between the computer device 2 and other computer devices. For example, the network interface 23 is used to connect the computer device 2 to an external terminal through a network, establish a data transmission channel and a communication connection between the computer device 2 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, and the like.
It is noted that fig. 5 only shows the computer device 2 with components 21-23, but it is to be understood that not all shown components are required to be implemented, and that more or less components may be implemented instead.
In this embodiment, the computer program stored in the memory 21 for implementing the intelligent data checking method may be executed by one or more processors (in this embodiment, the processor 22) to perform the following steps:
step 01: respectively acquiring an operation event and an operation sequence corresponding to the operation event in the process of manually downloading the first data file and the second data file in an event monitoring mode;
step 02: according to the obtained operation events and the operation sequence, automatically generating event operation scripts corresponding to a first data file and a second data file respectively;
step 10: executing a preset event operation script, and downloading a first data file and a second data file to be checked;
step 20: reading the first data file and the second data file line by line, and analyzing data in the first data file and the second data file into the same data check file according to a preset file format, wherein the data check file comprises a table, and the table comprises a service field and a corresponding value;
step 30: and calling a value corresponding to the preset service field from the data check file for checking according to the preset service field, and writing the checking result into the data check file.
Furthermore, the present invention relates to a computer-readable storage medium, which is a non-volatile readable storage medium, and a computer program is stored in the non-volatile readable storage medium, and the computer program can be executed by at least one processor to implement the operations of the above-mentioned data intelligent checking method or apparatus.
The computer-readable storage medium includes, among others, a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the computer device. Of course, the computer-readable storage medium may also include both internal and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is generally used for storing an operating system and various types of application software installed in the computer device, such as the aforementioned computer program for implementing the data intelligent checking method. Further, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.
Claims (10)
1. An intelligent data checking method is characterized by comprising the following steps:
executing a preset event operation script, and downloading a first data file and a second data file to be checked;
reading the first data file and the second data file line by line, and analyzing data in the first data file and the second data file into the same data check file according to a preset file format, wherein the data check file comprises a table, and the table comprises a service field and a corresponding value;
and calling a value corresponding to the preset service field from the data check file for checking according to the preset service field, and writing the checking result into the data check file.
2. The intelligent data checking method according to claim 1, wherein the generation of the event operation script comprises the following steps:
acquiring an operation event and an operation sequence corresponding to the operation event in the process of manually downloading the first data file and the second data file in an event monitoring mode;
and respectively and automatically generating event operation scripts corresponding to the first data file and the second data file according to the acquired operation events and the operation sequence.
3. The intelligent data checking method according to claim 1, wherein the reading the first data file and the second data file line by line, and parsing the data in the first data file and the second data file into the same data checking file according to a preset file format comprises the following steps:
reading each line of the first data file and the second data file line by using a method in a python native package;
respectively identifying the value under each service field in each line of the first data file and the second data file according to a preset service field sequence;
and writing the values under the service fields in the first data file and the second data file into the same data check file.
4. The intelligent data reconciliation method of claim 3 wherein said writing the values under each service field in said first data file and said second data file into said same data reconciliation file comprises the steps of:
judging whether the same identification value service field is recorded under an identification service field in the first data file and the second data file;
when the same identification value is recorded, writing the value under each service field in the row where the same identification value in the first data file and the second data file is located into the same row in the same data check file corresponding to each service field.
5. The intelligent data checking method according to claim 3, wherein the values under each service field in the first data file and the second data file are separated by a preset separator.
6. The intelligent data checking method according to claim 2, wherein the automatically generating an event operation script according to the acquired operation events and the operation sequence comprises the following steps:
calling a preset implementation method code corresponding to the type of the operation event according to the acquired operation event;
writing the called codes into corresponding service fields of a preset script file;
and when the operation event contains input content, writing the input content into a corresponding service field of the code.
7. An intelligent data checking device, comprising:
the file downloading module is used for executing a preset event operation script and downloading a first data file and a second data file to be checked;
the file analysis module is used for reading the first data file and the second data file line by line, and analyzing data in the first data file and the second data file into the same data check file according to a preset file format, wherein the data check file comprises a table, and the table comprises a service field and a corresponding value;
and the data checking module is used for calling a value corresponding to a preset field from the data checking file for checking according to the preset service field, and writing the checking result into the data checking file.
8. The intelligent data collating device according to claim 7, further comprising:
the script generation module is used for acquiring operation events and operation sequences corresponding to the operation events in the process of manually downloading the first data file and the second data file in an event monitoring mode; and respectively and automatically generating event operation scripts corresponding to the first data file and the second data file according to the acquired operation events and the operation sequence.
9. A computer device comprising a memory and a processor, characterized in that the memory has stored thereon a computer program which, when executed by the processor, carries out the steps of the intelligent data collation method according to any one of claims 1 to 6.
10. A computer-readable storage medium, having stored therein a computer program executable by at least one processor to perform the steps of the intelligent data reconciliation method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911315244.8A CN111143434A (en) | 2019-12-19 | 2019-12-19 | Intelligent data checking method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911315244.8A CN111143434A (en) | 2019-12-19 | 2019-12-19 | Intelligent data checking method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111143434A true CN111143434A (en) | 2020-05-12 |
Family
ID=70518917
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911315244.8A Pending CN111143434A (en) | 2019-12-19 | 2019-12-19 | Intelligent data checking method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143434A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581942A (en) * | 2020-06-12 | 2020-08-25 | 上海通联金融服务有限公司 | Data file comparison method |
| CN112579376A (en) * | 2020-12-16 | 2021-03-30 | 中国建设银行股份有限公司 | Double-run-stage data verification method, system, equipment and storage medium |
| CN113591446A (en) * | 2021-06-29 | 2021-11-02 | 四川新网银行股份有限公司 | Configuration reconciliation file generation device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101458705A (en) * | 2008-12-29 | 2009-06-17 | 阿里巴巴集团控股有限公司 | Data collating method between different utility systems, apparatus and system |
| CN103577571A (en) * | 2013-10-31 | 2014-02-12 | 北京奇虎科技有限公司 | Data processing method and device |
| CN106021119A (en) * | 2016-08-03 | 2016-10-12 | 网易（杭州）网络有限公司 | Configuration file calibration method and device |
| CN108376363A (en) * | 2018-02-13 | 2018-08-07 | 平安科技（深圳）有限公司 | A kind of reconciliation difference processing method and system |
| CN109190092A (en) * | 2018-08-15 | 2019-01-11 | 深圳平安综合金融服务有限公司上海分公司 | The consistency checking method of separate sources file |
| CN109656542A (en) * | 2018-12-10 | 2019-04-19 | 深圳市亿道数码技术有限公司 | Smbios information checking method and system |
| CN109684350A (en) * | 2018-12-15 | 2019-04-26 | 平安证券股份有限公司 | Registration of securities verification of data method, apparatus, computer equipment and storage medium |
| CN110197424A (en) * | 2019-05-31 | 2019-09-03 | 上海银行股份有限公司 | Reconciliation plateform system based on Redis |
-
2019
- 2019-12-19 CN CN201911315244.8A patent/CN111143434A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101458705A (en) * | 2008-12-29 | 2009-06-17 | 阿里巴巴集团控股有限公司 | Data collating method between different utility systems, apparatus and system |
| CN103577571A (en) * | 2013-10-31 | 2014-02-12 | 北京奇虎科技有限公司 | Data processing method and device |
| CN106021119A (en) * | 2016-08-03 | 2016-10-12 | 网易（杭州）网络有限公司 | Configuration file calibration method and device |
| CN108376363A (en) * | 2018-02-13 | 2018-08-07 | 平安科技（深圳）有限公司 | A kind of reconciliation difference processing method and system |
| CN109190092A (en) * | 2018-08-15 | 2019-01-11 | 深圳平安综合金融服务有限公司上海分公司 | The consistency checking method of separate sources file |
| CN109656542A (en) * | 2018-12-10 | 2019-04-19 | 深圳市亿道数码技术有限公司 | Smbios information checking method and system |
| CN109684350A (en) * | 2018-12-15 | 2019-04-26 | 平安证券股份有限公司 | Registration of securities verification of data method, apparatus, computer equipment and storage medium |
| CN110197424A (en) * | 2019-05-31 | 2019-09-03 | 上海银行股份有限公司 | Reconciliation plateform system based on Redis |
Non-Patent Citations (1)
| Title |
|---|
| 吕瑞;马春英;: "列车运行控制数据校核及管理系统", 铁道通信信号, no. 02, pages 17 - 19 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581942A (en) * | 2020-06-12 | 2020-08-25 | 上海通联金融服务有限公司 | Data file comparison method |
| CN111581942B (en) * | 2020-06-12 | 2023-06-27 | 上海通联金融服务有限公司 | Data file comparison method |
| CN112579376A (en) * | 2020-12-16 | 2021-03-30 | 中国建设银行股份有限公司 | Double-run-stage data verification method, system, equipment and storage medium |
| CN113591446A (en) * | 2021-06-29 | 2021-11-02 | 四川新网银行股份有限公司 | Configuration reconciliation file generation device |
| CN113591446B (en) * | 2021-06-29 | 2023-11-03 | 四川新网银行股份有限公司 | Configurational account checking file generating device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108491475B (en) | Data rapid batch import method, electronic device and computer readable storage medium | |
| CN111143434A (en) | Intelligent data checking method, device, equipment and storage medium | |
| CN111723079A (en) | Data migration method and device, computer equipment and storage medium | |
| CN111125598A (en) | Intelligent data query method, device, equipment and storage medium | |
| CN109241014B (en) | Data processing method and device and server | |
| CN103077108A (en) | Method and system for warning browser crash | |
| CN112769802B (en) | Access verification method and device based on server, electronic equipment and storage medium | |
| CN110958292A (en) | File uploading method, electronic device, computer equipment and storage medium | |
| CN110659210A (en) | Information acquisition method and device, electronic equipment and storage medium | |
| CN111367982A (en) | Method and device for importing TRIRIGA basic data, computer equipment and storage medium | |
| KR101737578B1 (en) | Method and device for automatically tuning for sql sentences generated automatically | |
| CN110716804A (en) | Method and device for automatically deleting useless resources, storage medium and electronic equipment | |
| US7548979B2 (en) | Method, system, and computer-readable medium for managing a host session on a remote computer | |
| CN114201615A (en) | Scientific research data change review method based on data snapshot and server | |
| CN109684207B (en) | Method and device for packaging operation sequence, electronic equipment and storage medium | |
| CN116775575A (en) | File merging method and device, electronic equipment and storage medium | |
| CN114338850B (en) | Message checking method, device, terminal equipment and computer readable storage medium | |
| CN114511314A (en) | Payment account management method and device, computer equipment and storage medium | |
| CN110221952B (en) | Service data processing method and device and service data processing system | |
| CN114371866A (en) | Version reconfiguration test method, device and equipment of service system | |
| CN112818204A (en) | Service processing method, device, equipment and storage medium | |
| KR101737575B1 (en) | Method and device for verifying data based on sql sentences generated automatically | |
| CN111694727A (en) | Network card firmware upgrading and downgrading test method, system, terminal and storage medium | |
| CN114253642A (en) | System information processing method, device, electronic equipment and medium | |
| CN113760696A (en) | Program problem positioning method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |