CN111026763A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111026763A CN111026763A CN201911280752.7A CN201911280752A CN111026763A CN 111026763 A CN111026763 A CN 111026763A CN 201911280752 A CN201911280752 A CN 201911280752A CN 111026763 A CN111026763 A CN 111026763A
- Authority
- CN
- China
- Prior art keywords
- data
- preset
- format
- supervised
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data processing method, a device, equipment and a storage medium, wherein the data processing method comprises the following steps: acquiring data to be supervised based on a preset acquisition mode; determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule; and if the data to be supervised does not conform to the preset data format, converting the data to be supervised into a data file with the preset data format. The technical scheme of the embodiment of the invention can overcome the defects of scattered supervision and no formation of a unified supervision system, and achieves the effect of uniformly supervising the data to be supervised in the same data format.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data processing method, a data processing device, a data processing terminal and a storage medium.
Background
For the current local financial supervision work, as supervision authorities of various regions just establish, various industries belong to different management departments, the management departments develop independently, and the differences of system styles, index systems and the like are large. Meanwhile, each business state has one or more business systems, and the data formats of the business states are various, so that the supervision is scattered, and a uniform supervision system cannot be formed.
Disclosure of Invention
The invention provides a data processing method, a data processing device, a terminal and a storage medium, which achieve the effect of uniformly monitoring each data to be monitored in the same data format.
In a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:
acquiring data to be supervised based on a preset acquisition mode;
determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule;
and if the data to be supervised does not conform to the preset data format, converting the data to be supervised into a data file with the preset data format.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, where the apparatus includes:
the data acquisition module to be supervised is used for acquiring data to be supervised based on a preset acquisition mode;
the preset data format determining module is used for determining whether the data to be monitored conforms to a preset data format or not according to a preset data format check rule;
and the preset data format conversion module is used for converting the data to be supervised into a data file with a preset data format if the data to be supervised does not conform to the preset data format.
In a third aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data processing method according to any one of the embodiments of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data processing method according to any embodiment of the present invention.
The embodiment of the invention collects the data to be supervised based on a preset collection mode; determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule; if the data to be supervised does not conform to the preset data format, the data to be supervised is converted into a data file with the preset data format, the defects of supervision dispersion and no formation of a unified supervision system are overcome, and the effect of uniformly supervising the data to be supervised in the same data format is achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief description will be given below of the drawings required for the embodiments or the technical solutions in the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of a data processing method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a data processing method according to a second embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device in the fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It is to be further noted that, for the convenience of description, only a part of the structure relating to the present invention is shown in the drawings, not the whole structure.
Example one
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, where the present embodiment is applicable to a case where data in each data system needs to be uniformly monitored, and the method may be executed by a data processing apparatus, where the apparatus may be implemented in a software and/or hardware manner, and the apparatus may be configured in a computer device. As shown in fig. 1, the method of this embodiment specifically includes:
and S110, acquiring data to be supervised based on a preset acquisition mode.
Preferably, the data to be supervised may be data related to each service developed by the data department to be supervised, and may be stored in the data system to be supervised, and the data system to be supervised preferably belongs to the data door to be supervised corresponding to the data system to be supervised. The present embodiment may preferably be applied to a data supervision system, which preferably belongs to a data supervision department. The data supervision department utilizes the data supervision system to acquire data to be supervised in the system to be supervised of corresponding data through a data interface between the data supervision system and the system to be supervised of the data, and processes and supervises and manages the acquired data to be supervised. Illustratively, the data administration may be a local financial administration and, correspondingly, the data administration system may be a local financial administration system. The data to-be-supervised department can be a small loan, financing guarantee, local asset management company, business insurance, equity trading market, pawn and financing lease and the like, and correspondingly, the data to-be-supervised system can be a small loan to-be-supervised system, a financing guarantee to-be-supervised system, a local asset management company to-be-supervised system, a business insurance management to-be-supervised system, a equity trading market to-be-supervised system, a pawn to-be-supervised system and a financing lease to-be-supervised system.
The data structure elements of the data to be supervised may include subject fields, tables, data items (columns) and data instances (rows). The subject domain can be used for dividing the broad category of the supervision attention field and can comprise public information, accounting and accounting information, customer information, company information, repayment information, transaction flow information, statistics general subjects, fund service, contract information and the like. The tables may be relational database tables and one or more tables may be included in a subject domain. The data item (column) may be one or more data items (columns) under the table, i.e. the basic unit of data. Preferably, the data content of each data item may be determined by the specific acquisition requirements of the data administration system. A data instance (row) may include data defined by all data items (columns) of a row, which is the basic unit of data storage and delivery. The data instance follows the relational database second paradigm, i.e., the data instance is not repeated in the same table.
Each data item (column) may preferably correspond to a respective format, which may indicate the type and length of the data item (column). The format of the data items may include at least one of Cn, c. Where Cn may represent a fixed length of n bytes, the corresponding character length is determined by the character encoding set. N may represent a variable length of at most n bytes, the corresponding character length being determined by the character encoding set. I may represent an integer computable form. N may represent an integer computable form of at most n bits. Dw.d may represent a decimal computable form, w represents the maximum number of character digits of the entire data item including character digits before and after the decimal point, and d represents the maximum number of character digits after the decimal point. F may represent a floating point number of arbitrary length.
Preferably, the preset acquisition mode may include a full-volume acquisition mode, an incremental acquisition module and a variable-volume acquisition mode. Correspondingly, based on the preset collection mode, collecting the data to be supervised may include: based on a full-quantity acquisition mode, acquiring all data to be supervised; or, based on the incremental acquisition mode, acquiring data to be monitored, which is increased compared with the last acquisition; or acquiring the data to be monitored, which changes from the last acquisition state, based on the variation acquisition mode.
The full-volume acquisition mode may refer to acquiring all data generated before an acquisition time point and in an acquisition time period, and acquiring the data even if there is data which has not changed from the last acquisition. For example, if the data monitoring system is a local financial monitoring system, the data collected using the full-volume collection mode is all the data that occurred and was present before the collection deadline point. The full-increment acquisition mode may refer to acquiring data increased from a previous acquisition, and does not include updating, deleting, and the like of the data acquired from the previous acquisition. Illustratively, incremental pipelining pertains to incremental data, while organization name changes pertain to updates of data. The variation collection mode may preferably refer to collecting data changed from the previous collection, which may include new, updated, and deleted data. The deletion data may be a new record in which the status data item of the original record is updated to be invalid or to have equivalent meaning, and is distinguished from the original record by, for example, the collection date.
The data monitoring department preferably can determine or adjust the data acquisition mode according to actual conditions. Generally, when the data supervision system collects data from the data to-be-supervised system for the first time, it is preferable that all the data to be supervised be collected from the data to-be-supervised system at one time based on the full-volume collection mode, and the state of the finished service at the collection deadline time point before the collection time point is not traced. When the data monitoring system collects data from the system to be monitored, the data monitoring system preferably selects a specific collection mode from a full collection mode, an incremental collection mode and a variable collection mode to collect the data to be monitored according to actual conditions.
The data supervision department may preferably determine or adjust the data collection frequency according to the actual situation, taking the data supervision system as a local financial supervision system as an example, the factors to be considered when determining the data collection frequency may include: the service scale and data volume of legal organization in the jurisdiction range, the capability of governing and reporting organization data, the capacity of a collection medium or network bandwidth and the like. If the data acquisition mode is full acquisition, the acquisition frequency can be in the unit of a month; if the data collection mode is variable quantity collection, the collection frequency can be in the unit of day or week. It should be noted that the branch acquisition mode is consistent with the mode required by the regulatory department to which the legal institution belongs.
Taking the data monitoring system as a local financial monitoring system as an example, the data to be monitored is collected and transmitted preferably based on the financial private network transmission, and the local financial monitoring system can update elements such as the transmission file name in the data collection system according to the interface description. It can be understood that when the centralized collection or the data volume is huge and is not convenient to transmit through the financial private network, the data to be supervised department can report the data to be supervised to the data supervision department by using the mobile storage medium.
And S120, determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule.
The preset data format can be a unified data format preset by a data to-be-supervised department according to actual supervision requirements. The preset data format may preferably include a preset file format, a preset file naming format, a preset character format, a preset null value format, and a preset desensitization format. The preset data format check rule may preferably correspond to a preset data format. It is understood that, for a data item having a specific format, the specific data format can be directly adopted without additionally setting other corresponding data formats. Such as a telephone number, which itself is specifically formatted to include 11 digits, does not need to be converted to another format after the telephone number is collected.
Preferably, each data table of the data to be supervised corresponds to a data file to be supervised, and the preset file format of the data table can be a UTF-8 coded (BOM-free) plain text file with the extension of.
The preset file naming format may preferably be a mechanism code-table name corresponding character string-yyyymmdd.txt, wherein the separator "-" may preferably be an english dash (ASCII code 0x2D), and the "YYYYMMDD" may be the same as the data item of "date of collection" and is the deadline date of this/present-period collection.
Log, the data check file may contain file name, file size (bytes), file creation completion time (YYYY-MM-DD HH: MM: SS), whether the file is normally generated (Y or N), and the number of file records (number of lines). The data verification file can preferably be used for verifying the correctness of the collected data to be supervised.
The preset character format may preferably include a preset separator format and a preset special character format, wherein the preset separator format may include a preset line separator format, a preset data item separator format and a preset data item sub-item separator format. The preset line separator may be a UNIX style line break (ASCII code 0x0A) with no data item separators added at the end of each line; the preset data item separator may be ^ A (SOH, ASCII code 0x 01); the preset data item inner sub item separator can be English semicolon "; "(ASCII code 0x3B), which may be denoted as" 2 "if the home service subclasses" investment 2 "and" financial services 13 "are included in the" home service "data item; 13". The preset special character format can comprise that the beginning and the end of the data item are prohibited to use double quotation marks (ASCII code 0x22) folders, but the content of the data item can have the double quotation marks and does not need to be subjected to escape; the method can also comprise various control characters and non-printable characters, wherein the ASCII codes of 0x 00-0 x1F and 0x7F are not allowed to appear in the data item; if the code occurs, the code is replaced by a blank (ASCII code 0x 20).
The preset null value format may preferably be such that when a null value occurs, a default value is filled in. Specifically, the method comprises the following steps: if the data to be supervised is of a character type, setting the default value to be null; if the data to be supervised is the date type, setting the default value to 99991231; if the data to be supervised is of a year and month type, setting the default value to 999912; if the data to be monitored is of a time type, setting a default value to 000000; if the data to be supervised is of the timestamp type, setting the default value to 99991231000000000000; if the data to be supervised is of a digital type, the default value is set to 0.
Aiming at the personal identification document number, the preset desensitization format can be preferably that if the identification document number has English letters, the identity document number can be uniformly converted into capital English letters; the format of the morphed identity document number may include 38 characters, which are the first 6 bytes (6 english characters/digits, or 2 chinese characters, or 1 chinese character +3 english characters/digits) of the identity document number plus MD5 (identity document number full text UTF-8 encoding) (32 characters, english output in lower case).
Aiming at the unified numbering of the customers containing the personal identity document number, the preset desensitization format preferably can be that if English letters exist in the identity document number, the English letters are uniformly converted into capital English letters; and the preset desensitization format of the personal identity document number is utilized to deform the identity document number part.
For the client name and all the information of the name of the opposite party, the preset desensitization format can preferably perform desensitization treatment in a limited exposure mode on names within three Chinese characters (including three Chinese characters), and only the last character is reserved. For names except three Chinese characters, a preset number of Chinese characters are taken for desensitization treatment, and names with the rest lengths are not deformed. The family names of the local financial institution staff, the business handling personnel, the natural stakeholders and the unit responsible persons are not deformed.
Preferably, after the data to be supervised is acquired, whether the file format, the file naming format, the character format, the null value format and the desensitization format of the data to be supervised are the same as the corresponding preset file format, the preset file naming format, the preset character format, the preset null value format and the preset desensitization format or not can be respectively determined according to a preset data format check rule, and if at least one of the file format of the data to be supervised is different from the preset file format, the file naming format is different from the preset file naming format, the character format is different from the preset character format, the null value format is different from the preset null value format and the desensitization format is different from the preset desensitization format, the data to be supervised is determined not to be in accordance with the preset data format.
And S130, if the data to be supervised do not conform to the preset data format, converting the data to be supervised into a data file with the preset data format.
Preferably, if the file format of the data to be supervised does not conform to the preset file format and the file format of the data to be supervised is determined to be correct, converting the data to be supervised into a data file with the preset file format; and/or if the file naming format of the data to be supervised does not conform to the preset file naming format, converting the data to be supervised into a data file with the preset file naming format; and/or if the character format in the data to be supervised does not conform to the preset character format, converting the data to be supervised into a data file with the preset character format; and/or if the null value format in the data to be monitored does not conform to the preset null value format, converting the data to be monitored into a data file in the preset null value format; and/or if the desensitization format in the data to be monitored does not conform to the preset desensitization format, converting the data to be monitored into a data file with the preset desensitization format.
It should be noted that, if it is determined that the file format of the data to be supervised does not conform to the preset file format and it is determined that the file format of the data to be supervised has an error, the data supervision system generates error feedback information and sends the error feedback information to the corresponding data system to be supervised, so that the corresponding data system to be supervised processes the data to be supervised having the error. Illustratively, the data item corresponding to the data to be supervised is an identity card number, and the preset file format is a preset desensitization format. After the data supervision system sends a request for acquiring the identity card number, the received data to be supervised is pure English characters which are not subjected to desensitization processing, and the file format of the data to be supervised does not conform to the preset desensitization format and is also wrong according to the received pure English characters which are not subjected to desensitization processing. At the moment, the data supervision system generates feedback information with wrong ID number format and sends the feedback information to the corresponding data to-be-supervised system.
Preferably, if the desensitization format in the data to be supervised does not conform to the preset desensitization format, converting the data to be supervised into a data file in the preset desensitization format, including:
if the desensitization format of the personal identification card number does not conform to the preset desensitization format, converting the lower-case English letters into upper-case English letters if the personal identification card number comprises the lower-case English letters;
and performing preset desensitization treatment on the personal identification number by using at least one of English characters, numbers or Chinese characters and MD5 codes.
It can be understood that, if the data to be supervised conforms to the preset data format, it indicates that the data to be supervised has been converted into the data conforming to the preset data format by the data supervising department according to the preset data format before the data supervising department collects the data. After the data to be supervised are collected by the data supervision department, the corresponding data can be directly supervised and managed.
In the data processing method provided by this embodiment, data to be supervised is acquired based on a preset acquisition mode; determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule; if the data to be supervised does not conform to the preset data format, the data to be supervised is converted into a data file with the preset data format, the defects of scattered supervision and no formation of a unified supervision system are overcome, and the effect of uniformly supervising each data to be supervised in the same data format is achieved.
Example two
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention. On the basis of the foregoing embodiments, the present embodiment may select a preset acquisition mode to acquire data to be supervised, including: and acquiring data to be supervised which accords with preset constraint conditions based on a preset acquisition mode.
And before determining whether the data to be supervised accords with a preset data format according to a preset data format check rule, the method comprises the following steps: and determining a preset data format check rule matched with the preset data format according to the preset data format.
As shown in fig. 2, the method of this embodiment specifically includes:
and S210, acquiring data to be supervised meeting preset constraint conditions based on a preset acquisition mode.
The preset constraint condition may be a data item constraint condition preset by a data supervision department according to actual needs of data supervision. It will be appreciated that for data items for which constraints exist, it is preferable to collect the corresponding data item content as required by the constraints, and for data items for which constraints do not exist, the data item content corresponding to the data item is collected directly.
For example, the preset constraint condition may include at least one of a preset data item description, a preset data item remark and a preset data item association condition. Wherein, the preset data item description can represent the content that the corresponding data item needs to collect. The preset data item notes may represent additional requirements for the respective data items. Taking a data supervision system as an example of a local financial supervision system, for data items related to privacy of a client, corresponding preset data item remarks of the data items can comprise privacy, temporary non-taking, privacy and local financial institution deformation, and the like, wherein the remarks are privacy, the temporary non-taking data items are data items which are not collected by the local financial supervision system, and the data items are reserved in corresponding local financial institutions and are used as retrievable data items. The remark is that the data item of 'privacy, local financial institution deformation' is the data item of the local financial institution deformed according to the preset data format. The deformation rule is the same as the preset desensitization format, and the description is omitted here.
The preset data item association condition is a unique constraint reference condition of a data supervision department, contains a plurality of data association conditions and can represent the reference of a joint primary key. The preset data item association condition may be in the form of "association data item: [ table name ] - [ data item name ] "is used for representing a reference data item of the data item value. For example, "associated data item: the internal organization number indicates that the data item should be consistent with the value of the data item of the internal organization number in the organization information table. The data supervision department can adjust the preset data item association conditions according to the actual use purpose of the data.
S220, determining a preset data format verification rule matched with the preset data format according to the preset data format.
Preferably, the preset data format verification rule may determine whether the file format, the file naming format, the character format, the null value format and the desensitization format of the data to be verified are correct according to a preset file format, a preset file naming format, a preset character format, a preset null value format and a preset desensitization format.
And S230, determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule.
S240, if the data to be supervised do not conform to the preset data format, converting the data to be supervised into a data file with the preset data format.
In the data processing method provided by the embodiment, data to be supervised meeting preset constraint conditions is acquired based on a preset acquisition mode; determining a preset data format check rule matched with a preset data format according to the preset data format; determining whether the data to be supervised conforms to a preset data format or not according to a preset data format check rule; if the file format of the data to be supervised does not conform to the preset file format, converting the data to be supervised into a data file with the preset file format; and/or if the file naming format of the data to be supervised does not conform to the preset file naming format, converting the data to be supervised into a data file with the preset file naming format; and/or if the character format in the data to be supervised does not conform to the preset character format, converting the data to be supervised into a data file with the preset character format; and/or if the null value format in the data to be supervised does not conform to the preset null value format, converting the data to be supervised into a data file with the preset null value format; and/or if the desensitization format in the data to be supervised does not conform to the preset desensitization format, converting the data to be supervised into a data file with the preset desensitization format, overcoming the defects of scattered supervision and no formation of a unified supervision system, achieving the effect of uniformly supervising each data to be supervised in the same data format, and only acquiring the data to be supervised according to supervision requirements so as to reduce the workload in the data acquisition and supervision process.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention. As shown in fig. 3, the apparatus of the present embodiment includes:
a to-be-supervised data acquisition module 310, configured to acquire data to be supervised based on a preset acquisition mode;
the preset data format determining module 320 is configured to determine whether the data to be monitored conforms to a preset data format according to a preset data format check rule;
the preset data format converting module 330 is configured to convert the data to be supervised into a data file in a preset data format if the data to be supervised does not conform to the preset data format.
According to the data processing device provided by the embodiment, the data to be supervised is acquired based on the preset acquisition mode by using the data to be supervised acquisition module; determining whether the data to be supervised accords with a preset data format or not by using a preset data format determining module according to a preset data format check rule; if the data to be supervised does not conform to the preset data format, the preset data format conversion module is used for converting the data to be supervised into the data file with the preset data format, so that the defects of scattered supervision and no formation of a unified supervision system are overcome, and the effect of uniformly supervising each data to be supervised in the same data format is achieved.
On the basis of the above technical solutions, optionally, the to-be-supervised data acquisition module 310 may be specifically configured to:
based on a full collection mode, collecting all data to be supervised;
or, based on the incremental acquisition mode, acquiring data to be monitored, which is increased compared with the last acquisition;
or acquiring the data to be monitored, which changes from the last acquisition state, based on the variation acquisition mode.
On the basis of the above technical solutions, optionally, the to-be-supervised data acquisition module 310 may specifically include:
and the conditional constraint acquisition unit is used for acquiring the data to be supervised which accords with the preset constraint condition based on the preset acquisition mode.
On the basis of the above technical solutions, optionally, the preset constraint condition includes at least one of a preset data item description, a preset data item remark, and a preset data item association condition.
On the basis of the above technical solutions, optionally, the data processing apparatus may further include a preset data check rule determining module, configured to determine, according to the preset data format, a preset data format check rule that matches the preset data format before determining whether the data to be supervised conforms to the preset data format according to the preset data format check rule.
On the basis of the above technical solutions, optionally, the preset data format conversion module 330 may specifically include:
the preset file format conversion unit is used for converting the data to be supervised into a data file with a preset file format if the file format of the data to be supervised does not conform to the preset file format;
and/or the preset file naming format conversion unit is used for converting the data to be supervised into a data file with a preset file naming format if the file naming format of the data to be supervised does not conform to the preset file naming format;
and/or the preset character format conversion unit is used for converting the data to be supervised into a data file with a preset character format if the character format in the data to be supervised does not conform to the preset character format;
and/or the preset null value format conversion unit is used for converting the data to be supervised into a data file with a preset null value format if the null value format in the data to be supervised does not conform to the preset null value format;
and/or the preset desensitization format conversion unit is used for converting the data to be monitored into a data file with a preset desensitization format if the desensitization format in the data to be monitored does not conform to the preset desensitization format.
On the basis of the above technical solutions, optionally, the preset desensitization format conversion unit may be specifically configured to convert lowercase english letters into uppercase english letters if the desensitization format of the personal identification number does not conform to the preset desensitization format, if the personal identification number includes lowercase english letters;
and performing preset desensitization treatment on the personal identification number by using at least one of English characters, numbers or Chinese characters and MD5 codes.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention. FIG. 4 illustrates a block diagram of an exemplary computer device 412 suitable for use in implementing embodiments of the present invention. The computer device 412 shown in FIG. 4 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the present invention.
As shown in FIG. 4, computer device 412 is in the form of a general purpose computing device. Components of computer device 412 may include, but are not limited to: one or more processors 416, a memory 428, and a bus 418 that couples the various system components (including the memory 428 and the processors 416).
A program/utility 440 having a set (at least one) of program modules 442 may be stored, for instance, in memory 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination may include an implementation of a network environment. The program modules 442 generally perform the functions and/or methodologies of the described embodiments of the invention.
The computer device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc., where the display 424 may be configurable or not as desired), one or more devices that enable a user to interact with the computer device 412, and/or any device (e.g., network card, modem, etc.) that enables the computer device 412 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 422. Also, computer device 412 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) through network adapter 420. As shown, network adapter 420 communicates with the other modules of computer device 412 over bus 418. It should be appreciated that although not shown in FIG. 4, other hardware and/or software modules may be used in conjunction with the computer device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage, among others.
The processor 416 executes various functional applications and data processing, such as implementing data processing methods provided by embodiments of the present invention, by executing programs stored in the memory 428.
EXAMPLE five
An embodiment five of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data processing method provided in an embodiment of the present invention, and the method includes:
acquiring data to be supervised based on a preset acquisition mode;
determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule;
and if the data to be supervised does not conform to the preset data format, converting the data to be supervised into a data file with the preset data format.
Of course, the computer-readable storage medium provided by the embodiments of the present invention, the computer program stored thereon, is not limited to execute the method operations described above, and may also execute the relevant operations in the data processing method based on the computer device provided by any embodiment of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions without departing from the scope of the invention. Therefore, although the present invention has been described in more detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data processing method, comprising:
acquiring data to be supervised based on a preset acquisition mode;
determining whether the data to be supervised accords with a preset data format or not according to a preset data format check rule;
and if the data to be supervised does not conform to the preset data format, converting the data to be supervised into a data file with the preset data format.
2. The method of claim 1, wherein collecting data to be supervised based on a preset collection mode comprises:
based on a full collection mode, collecting all data to be supervised;
or, based on the incremental acquisition mode, acquiring data to be monitored, which is increased compared with the last acquisition;
or acquiring the data to be monitored, which changes from the last acquisition state, based on the variation acquisition mode.
3. The method of claim 1, wherein collecting data to be supervised based on a preset collection mode comprises:
and acquiring data to be supervised which accords with preset constraint conditions based on a preset acquisition mode.
4. The method of claim 3, wherein the preset constraint condition comprises at least one of a preset data item description, a preset data item remark and a preset data item association condition.
5. The method according to claim 1, before determining whether the data to be supervised conforms to a preset data format according to a preset data format check rule, comprising:
and determining a preset data format check rule matched with the preset data format according to the preset data format.
6. The method according to claim 1, wherein if the data to be supervised does not conform to a preset data format, converting the data to be supervised into a data file in the preset data format, including:
if the file format of the data to be supervised does not conform to the preset file format, converting the data to be supervised into a data file with the preset file format;
and/or if the file naming format of the data to be supervised does not conform to the preset file naming format, converting the data to be supervised into a data file with the preset file naming format;
and/or if the character format in the data to be supervised does not conform to the preset character format, converting the data to be supervised into a data file with the preset character format;
and/or if the null value format in the data to be supervised does not conform to the preset null value format, converting the data to be supervised into a data file with the preset null value format;
and/or if the desensitization format in the data to be monitored does not conform to the preset desensitization format, converting the data to be monitored into a data file with the preset desensitization format.
7. The method according to claim 6, wherein if the desensitization format in the data to be supervised does not conform to the preset desensitization format, converting the data to be supervised into a data file of the preset desensitization format includes:
if the desensitization format of the personal identification number does not conform to the preset desensitization format, converting the lower-case English letters into upper-case English letters if the personal identification number comprises the lower-case English letters;
and performing preset desensitization treatment on the personal identification number by using at least one of English characters, numbers or Chinese characters and MD5 codes.
8. A data processing apparatus, comprising:
the data acquisition module to be supervised is used for acquiring data to be supervised based on a preset acquisition mode;
the preset data format determining module is used for determining whether the data to be supervised accords with a preset data format according to a preset data format check rule;
and the preset data format conversion module is used for converting the data to be supervised into a data file with a preset data format if the data to be supervised does not conform to the preset data format.
9. A computer device, comprising:
one or more processing devices;
a memory for storing one or more programs;
when executed by the one or more processing devices, cause the one or more processing devices to implement the data processing method of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911280752.7A CN111026763A (en) | 2019-12-13 | 2019-12-13 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911280752.7A CN111026763A (en) | 2019-12-13 | 2019-12-13 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111026763A true CN111026763A (en) | 2020-04-17 |
Family
ID=70209042
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911280752.7A Pending CN111026763A (en) | 2019-12-13 | 2019-12-13 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026763A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112819620A (en) * | 2021-01-13 | 2021-05-18 | 中国建设银行股份有限公司 | Method and device for collecting rating data of financial institution |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101635904A (en) * | 2008-07-22 | 2010-01-27 | 中国移动通信集团吉林有限公司 | System and method for processing data |
| CN106503274A (en) * | 2016-12-22 | 2017-03-15 | 北京览群智数据科技有限责任公司 | A kind of Data Integration and searching method and server |
| CN110069755A (en) * | 2019-03-21 | 2019-07-30 | 平安普惠企业管理有限公司 | Method, apparatus, computer equipment and the storage medium that data are sent |
| CN110175165A (en) * | 2019-04-16 | 2019-08-27 | 平安普惠企业管理有限公司 | Transaction account checking method, device, computer equipment and storage medium |
| CN110245470A (en) * | 2019-04-25 | 2019-09-17 | 深圳壹账通智能科技有限公司 | Intelligent data desensitization method, device, computer equipment and storage medium |
| CN110457256A (en) * | 2019-08-01 | 2019-11-15 | 大众问问(北京)信息科技有限公司 | Date storage method, device, computer equipment and storage medium |
| CN110472434A (en) * | 2019-07-12 | 2019-11-19 | 北京字节跳动网络技术有限公司 | Data desensitization method, system, medium and electronic equipment |
-
2019
- 2019-12-13 CN CN201911280752.7A patent/CN111026763A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101635904A (en) * | 2008-07-22 | 2010-01-27 | 中国移动通信集团吉林有限公司 | System and method for processing data |
| CN106503274A (en) * | 2016-12-22 | 2017-03-15 | 北京览群智数据科技有限责任公司 | A kind of Data Integration and searching method and server |
| CN110069755A (en) * | 2019-03-21 | 2019-07-30 | 平安普惠企业管理有限公司 | Method, apparatus, computer equipment and the storage medium that data are sent |
| CN110175165A (en) * | 2019-04-16 | 2019-08-27 | 平安普惠企业管理有限公司 | Transaction account checking method, device, computer equipment and storage medium |
| CN110245470A (en) * | 2019-04-25 | 2019-09-17 | 深圳壹账通智能科技有限公司 | Intelligent data desensitization method, device, computer equipment and storage medium |
| CN110472434A (en) * | 2019-07-12 | 2019-11-19 | 北京字节跳动网络技术有限公司 | Data desensitization method, system, medium and electronic equipment |
| CN110457256A (en) * | 2019-08-01 | 2019-11-15 | 大众问问(北京)信息科技有限公司 | Date storage method, device, computer equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112819620A (en) * | 2021-01-13 | 2021-05-18 | 中国建设银行股份有限公司 | Method and device for collecting rating data of financial institution |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20180276216A1 (en) | Comparing Data Stores Using Hash Sums on Disparate Parallel Systems | |
| US7092956B2 (en) | Deduplication system | |
| CN109034988B (en) | Accounting entry generation method and device | |
| CN109241068B (en) | Method and device for comparing foreground and background data and terminal equipment | |
| CN111709527A (en) | Operation and maintenance knowledge map library establishing method, device, equipment and storage medium | |
| US11042563B1 (en) | Database interface system | |
| CN111062799A (en) | Method and device for managing family client, electronic equipment and storage medium | |
| CN113254457B (en) | Account checking method, account checking system and computer readable storage medium | |
| CN113902574A (en) | Protocol data processing method, device, computer equipment and storage medium | |
| CN111026763A (en) | Data processing method, device, equipment and storage medium | |
| US11422972B2 (en) | Relational database conversion and purge | |
| CN112579632A (en) | Data verification method, device, equipment and medium | |
| KR100796906B1 (en) | Method for Quality Control of DataBase | |
| CN111782649A (en) | Data acquisition format updating method and device, computer equipment and storage medium | |
| CN117036062A (en) | Accounting resource calculation method and device | |
| US11106692B1 (en) | Data record resolution and correlation system | |
| KR100792322B1 (en) | Framework for Quality Control of DataBase | |
| CN114518989A (en) | Abnormal data monitoring method and system | |
| KR100796905B1 (en) | System for Quality Control of DataBase | |
| US10311022B2 (en) | CDBMS (cloud database management system) distributed logical unit repository | |
| CN115017875B (en) | Enterprise information processing method, device, system, equipment and medium | |
| CN115982158A (en) | Supervision data processing method, device, equipment and medium based on data mart | |
| US11831490B1 (en) | Systems, methods, and media for performing information technology service management correlation for infrastructure environment functions | |
| CN117114843B (en) | Bank data quality control method | |
| US20240126780A1 (en) | Data processing during data replication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220922 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right |