CN114328413A - Data processing method and device, storage medium and electronic equipment - Google Patents
Data processing method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN114328413A CN114328413A CN202111660499.5A CN202111660499A CN114328413A CN 114328413 A CN114328413 A CN 114328413A CN 202111660499 A CN202111660499 A CN 202111660499A CN 114328413 A CN114328413 A CN 114328413A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- data file
- verification
- blacklist
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000012795 verification Methods 0.000 claims abstract description 151
- 238000000034 method Methods 0.000 claims abstract description 47
- 238000012546 transfer Methods 0.000 claims abstract description 25
- 238000012545 processing Methods 0.000 claims description 20
- 239000006185 dispersion Substances 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 13
- 238000012423 maintenance Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013075 data extraction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method and device, a storage medium and electronic equipment, wherein the method comprises the following steps: capturing a data file uploaded to a transfer platform by a data source, wherein the data file comprises data of credit blacklist passengers; pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking; and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification. The data files which do not meet the requirements can be screened out by carrying out pre-verification and warehousing verification on the data files, and finally, the data of the credit blacklist passengers in the data files which pass the pre-verification and warehousing verification are led into a blacklist database, so that the centralized management on the data of the credit blacklist passengers is realized, and the problem of data dispersion of the credit blacklist passengers is solved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, a storage medium, and an electronic device.
Background
In order to strengthen the credit culture construction of the civil aviation industry, maintain the order of civil aviation activities and promote the healthy development of the civil aviation industry, the collection, use, removal and other items of the credit information of the civil aviation industry need to be standardized, the records of the general information loss behavior information, the serious information loss behavior information and the like of the organization and the individual engaged in the civil aviation activities in the interior or the exterior need to be established, and the organization or the individual recorded in the credit record due to the general information loss behavior is strictly managed according to the conditions.
At present, when a credit loss behavior list of an individual or an organization is established, different units and departments respectively collect individual or combined credit data and respectively establish credit loss blacklists, and the credit loss blacklists respectively become systems, so that the credit loss blacklist data of the units and the departments are in a scattered state and are difficult to manage.
Disclosure of Invention
In view of this, the present invention provides a data processing method and apparatus, a storage medium, and an electronic device, which are applied to the present invention, and can perform centralized management on data of credit blacklist passengers, thereby solving the problem of data dispersion of credit blacklist passengers.
In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:
a first aspect of the present application discloses a data processing method, including:
capturing a data file uploaded to a transfer platform by a data source, wherein the data file comprises data of credit blacklist passengers;
pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking;
and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification.
A second aspect of the present application discloses a data processing apparatus comprising:
the system comprises a capturing unit, a transferring unit and a processing unit, wherein the capturing unit is used for capturing a data file uploaded to a transfer platform by a data source, and the data file comprises data of credit blacklist passengers;
the pre-checking unit is used for pre-checking the data file and determining the file type of the data file when the data file passes the pre-checking;
and the warehousing unit is used for performing warehousing verification corresponding to the file type on the data file and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification.
A third aspect of the present application discloses a storage medium, which includes stored instructions, wherein when the instructions are executed, a device on which the storage medium is located is controlled to execute the data processing method described above.
A fourth aspect of the present application discloses an electronic device comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by the one or more processors to perform the data processing method as described above.
Compared with the prior art, the invention has the following advantages:
the invention provides a data processing method and device, a storage medium and electronic equipment, wherein the method comprises the following steps: capturing a data file uploaded to a transfer platform by a data source, wherein the data file comprises data of credit blacklist passengers; pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking; and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification. By carrying out pre-verification and warehousing verification on the data files, the data files which do not meet the requirements can be screened out, and finally the data of the credit blacklist passengers in the data files which pass the pre-verification and warehousing verification are imported into a blacklist database, so that the centralized management of the data of the credit blacklist passengers is realized, the problem of data dispersion of the credit blacklist passengers is solved, the difficulty of later-stage data maintenance can be effectively reduced, and the maintenance cost is saved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flowchart of another method of a data processing method according to an embodiment of the present invention;
FIG. 3 is a flow chart of another method of a data processing method according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating another method of a data processing method according to an embodiment of the present invention;
FIG. 5 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 6 is a flowchart of a data processing method according to yet another embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The background art shows that the existing credit loss blacklist data established by each department and unit is in a dispersed state and is difficult to manage. In addition, because the credit loss blacklist data established by each department and unit form a system, the fragmentation and the islanding of the credit loss blacklist are serious, and the problems that the contents of the storage format, the data field and the like of the credit loss blacklist data of each department and unit have large difference, the management is difficult and the maintenance cost is high exist.
The invention can be used in a civil aviation credit data unified platform consisting of a plurality of general or special computing device environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multi-processor apparatus, distributed computing environments that include any of the above devices or equipment, and the like. Furthermore, the civil aviation credit data unified platform provided by the invention comprises a data source, a transit platform, a derivative system and a database; the data source is each department providing data files; the transfer platform is used for temporarily storing the data files, and further, the transfer platform is a file transfer station between the data source and the database; the derivative system is used for reading the data file from the transfer platform, and carrying out operations such as verification and warehousing on the data file; the data base is used for storing the data of the blacklist passengers, and the civil aviation credit data unified platform provided by the invention can be closely associated with a Chinese aviation credit PSS passenger service system and can be adapted to support more service scenes.
The execution subject of the invention is a derivative system in a unified platform of civil aviation credit data, referring to fig. 1, which is a method flowchart of a data processing method provided by the embodiment of the invention, and the specific description is as follows:
s101, capturing a data file uploaded to a transfer platform from a data source, wherein the data file comprises data of credit blacklist passengers.
Preferably, the data source transmits the data File based on a Secure File Transfer Protocol (SFTP), and the SFTP uses an encryption/decryption technology, so that the data source is more secure when uploading data.
The data file contains the data of the credit blacklist passenger, such as case ID, data source department, passenger name, certificate number, etc.
When capturing a data file in a transit platform, a derivative system acquires the data file from the transit platform through a script based on SFTP (file transfer protocol), and further the acquired data file is a decrypted file; the whole process uses a symmetric encryption algorithm for encryption transmission and storage, and more reliable data security guarantee is provided.
Preferably, the derivative system may capture the data file in the relay platform at regular time when capturing the data file in the relay platform, specifically, capture the data file in the relay platform at 0 point every day.
S102, pre-checking the data file, and determining whether the data file passes the pre-checking; if the data file passes the pre-verification, executing S103; if the data file does not pass the pre-verification, S106 is executed.
The process of pre-verifying the data file according to the present invention can refer to fig. 2, and the following details are described:
s201, judging whether the file name of the data file meets a preset file name naming standard or not; if the file name of the data file meets the file name specification, executing S202; if the file name of the data file does not satisfy the file name specification, S205 is performed.
Preferably, the file name naming specification is as follows: YYYYMMDD + two digit. xml; further, YYYYMMDD in the file naming convention may be any english capital or small.
Comparing the file name of the data file with a file name naming standard so as to judge whether the file name of the data file meets the file name standard or not; further, the specific content of the file name naming specification can be set according to actual requirements.
S202, determining whether the file format of the data file is a standard format; if the file format of the data file is the standard format, executing S203; if the file format of the data file is not the standard format, step S205 is executed.
It should be noted that there are various file formats, specifically, formats such as txt, word, and xml, where the standard format in the present invention is a specific file format set by a maintenance person, and for example, the standard format in the present invention is an xml format.
When determining whether the file format of the data file is the standard format, the format identifier of the data file can be used as a judgment basis, and when the format identifier of the data file is the same as the format identifier of the standard format, the file format of the data file is determined to be the standard format; and when the format identifier of the data file is different from the format identifier of the standard format, determining that the file format of the data file is not the standard format.
S203, checking each file field in the data file, and determining whether the file field which fails to be checked exists; if the file field which is not verified does not exist, executing S204; if the data in the file field is not verified, S205 is executed.
The data file comprises a plurality of file fields, each file field in the data file is verified, specifically, when the file fields in the data file are verified, whether the data file comprises each preset file field or not can be verified, and data in each file field can also be verified.
The data storage requirements of different file fields are different, and when the data of each file field is verified, whether the data in the file field meets the data storage requirements of the file field is verified.
Furthermore, when the data file contains each preset file field and the data in each file field meets the corresponding data storage requirement, the data file can be determined to pass the verification.
Illustratively, the preset file fields include, but are not limited to, a certificate number field, a name field, a case ID field, and an operation type field.
Illustratively, the data storage requirements of the certificate number field, the name field, the case ID field, and the operation type field in the data file are respectively explained, and the specific explanation is as follows:
certificate number field: cannot be empty; no space can be left; can only consist of numbers, letters and brackets; a length of no greater than 50;
name field: cannot be empty; spaces, brackets and special characters cannot exist; a length of no more than 200; specifically, the special characters include, but are not limited to,%, # and the like;
case ID field: cannot be empty; a length of no greater than 100; a 'pure number' or 'committee trigram + number' form;
operation type field (INFOACTIONTYPE tag): 0 or 1 or 2.
And S204, determining that the data file passes the pre-verification.
It should be noted that, when it is determined that the data file passes the verification, a verification-passing feedback file may be generated, where the verification-passing feedback file is an empty file including a file head and a file tail line and no information line.
After the data file passes the verification, the generated verification feedback file and the data file may be backed up to a pre-configured directory, and specifically, a timestamp needs to be added when the data file is backed up.
S205, determining that the data file does not pass the pre-verification.
In the method provided by the embodiment of the invention, the data files can be preliminarily screened by checking the data files, so that the system does not need to process the data files which are not screened, the workload of the system is reduced, the formats of the data files processed subsequently by the system are ensured to be uniform, and the working efficiency of the system can be improved.
S103, determining the file type of the data file.
Further, the data file is the data file in the directory described in S204, and the data file is the file that passes the verification.
Referring to fig. 3, a flowchart of a method for determining a file type of a data file according to another embodiment of the present invention is specifically described as follows:
s301, acquiring the file name of the data file.
S302, judging whether the file identification in the file name is a full identification; if the file identifier in the file name is the full identifier, executing S303; if the file identifier in the file name is not the full identifier, S304 is executed.
S303, determining the file type of the data file to be a full data file.
S304, determining the file identifier in the file name as an increment identifier, and determining the file type of the data file as an increment data file.
In the method provided by the embodiment of the present invention, the file name of the data file includes a file identifier, specifically, the full identifier may be represented by "full", and the incremental identifier may be represented by "date". Preferably, the full data file is a file formed by summarizing all data of the data source together; the incremental data file is a file in which partial data of the data source are gathered together.
The file type of the data file can be accurately determined through the file identification, and different file types execute different warehousing verification operations, so that the data file can be classified and processed, the file processing efficiency of the system is improved, the system can process different types of files, and the operability of the system is stronger.
S104, performing warehousing verification corresponding to the file type on the data file, and determining whether the data file passes the warehousing verification; if the data file passes the warehousing verification, executing S105; if the data file does not pass the warehousing check, S106 is executed.
The data file subjected to the warehousing verification is the data file passing the verification in the foregoing, and may also be regarded as a file obtained in the preset directory in step S204.
Preferably, in the method provided in the embodiment of the present invention, the warehousing checks corresponding to different file types are different, and the specific description is as follows:
when the file type of the data file is a full data file, the warehousing verification of the data file is specifically as follows:
determining whether the file name of the data file meets a preset full file naming rule or not;
when the file name of the data file meets the full file naming rule, judging whether the file format of the data file is a preset format or not;
when the file name of the data file meets the full file naming rule, determining that the data file does not pass the warehousing verification;
if the file format of the data file is a preset format, performing field verification on the data file, and determining whether the data file passes the field verification;
if the file format of the data file is not the preset format, determining that the data file does not pass the warehousing verification;
if the data file passes the field verification, determining that the data file passes the warehousing verification; and if the data file does not pass the field verification, determining that the data file does not pass the warehousing verification.
It should be noted that, when the file name of the data file contains the full identifier, it is determined that the file name of the data file meets the full file naming rule; otherwise, determining that the file name of the data file does not meet the full file naming rule; the preset format can be an xml format, and when the field of the data file is verified, whether the data file contains each preset field or not is determined, and whether the data in each field of the data file meets the storage requirement of the field or not is determined; and when the data file contains all preset fields and the data in each field in the data file meets the storage requirement of the field, determining that the data characters pass the field verification, otherwise, determining that the data characters do not pass the field verification. It should be noted that, the description of determining the file format of the data file and the description of performing the field check on the data file may refer to the related description in fig. 2, and will not be described herein again.
When the file type of the data file is an incremental data file, the warehousing verification of the data file is specifically as follows:
determining whether the file name of the data file meets a preset incremental file naming rule or not;
when the file name of the data file meets the incremental file naming rule, judging whether the file format of the data file is a preset format or not;
when the file name of the data file meets the full file naming rule, determining that the data file does not pass the warehousing verification;
if the file format of the data file is a preset format, performing field verification on the data file, and determining whether the data file passes the field verification;
if the file format of the data file is not the preset format, determining that the data file does not pass the warehousing verification;
if the data file passes the field verification, determining that the data file passes the warehousing verification; and if the data file does not pass the field verification, determining that the data file does not pass the warehousing verification.
In the method provided by the embodiment of the invention, the process of warehousing verification of the file type of the full data file and the process of warehousing verification of the file type of the incremental data file
When the file name of the data file contains the increment identification, determining that the data file meets the increment file naming rule, otherwise, determining that the data file does not meet the increment file naming rule; the process of judging the file format of the data file and performing field verification on the data file can refer to the file type as the warehousing verification process of the full data file, and is not repeated here.
According to the invention, the data files can be screened again by performing warehousing verification on the data files, so that the data files are ensured to be files which can be processed by the system, the files which cannot be processed by the system are effectively screened out, the working stability of the system is ensured, and the working efficiency of the system is improved.
And S105, importing the data of the credit blacklist passengers in the data file into a preset blacklist database.
In the method provided by the embodiment of the invention, after the data of the information blacklist passengers in the data file is imported into the preset blacklist database, the warehousing feedback file can be generated, and the warehousing feedback file only comprises a file head and tail row and a no-information row empty file.
Referring to fig. 4, a flow of a method for importing data of a credit blacklist traveler in a data file into a preset blacklist database according to another embodiment of the present invention is specifically described as follows:
s401, extracting temporary table data meeting a preset temporary table structure from the data file, and storing the temporary table data into a preset temporary library.
Temporary table data is extracted from the data file based on a preset temporary table structure, and preferably, the temporary table data can be represented in a table form, and a temporary library can be arranged in the edb database.
It should be noted that the temporary table structure corresponds to a temporary table t _ csm _ tmp _ sxr _ info, and the temporary table is one of tables in the database storage table.
The temporary table structure is specifically shown in table 1:
TABLE 1
S402, formal table data meeting a preset formal table structure are extracted from the data file, and the formal table data are stored in a preset formal library.
Formal table data is extracted from the data file based on a preset formal table structure, preferably, the formal table data can be represented in a table form, and it should be noted that a formal library can be arranged in edb database.
It should be noted that the formal table structure corresponds to a formal table t _ csm _ profile _ sxr _ info, and the formal table is one of the tables in the database storage table.
The formal table structure is specifically shown in table 2:
TABLE 2
And S403, extracting credit blacklist passenger data meeting a preset blacklist table structure from the temporary table data and the formal table data, and storing the credit blacklist passenger data in a blacklist database.
The blacklist database can be constructed by using a redis technology, and further, the blacklist database adopts a master-standby mode and comprises a main database and a standby database, when the credit blacklist passenger data is stored in the blacklist database, the credit blacklist passenger data can be stored in the main database and the standby database, and the main database and the standby database can be switched at any time.
The data corresponding to the blacklist table structure are all data that need to be stored in the blacklist database, and the blacklist table structure is shown in table 3, which is specifically described as follows:
TABLE 3
In the method provided by the embodiment of the invention, the temporary table data and the formal table data can be extracted from the data file by using the temporary table structure and the formal table structure, and the data is extracted from the temporary table data and the formal table data according to the blacklist table structure, so that the structures of the data stored in the blacklist database are unified, the problems of data dispersion and non-unified data format are effectively solved, the uniform format transmission storage and centralized collection of the data are realized, the difficulty of later maintenance and management is greatly reduced, and the labor and machine costs are saved.
And S106, generating a verification failure file of the data file, and storing the verification failure file to a transfer platform.
When the verification failure file of the data file is generated, the verification failure file is generated according to the reason of the verification error, so that the verification failure file comprises the specific reason of the verification error of the data file, the verification failure file comprises the processing state, the warehousing record and the like of the data file, the verification failure file is stored to the transfer platform, the data source can download the verification failure file from the transfer platform, and the data file is modified.
In the method provided by the embodiment of the invention, a data file uploaded to a transfer platform from a data source is captured, wherein the data file comprises data of credit blacklist passengers; pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking; and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification. By carrying out pre-verification and warehousing verification on the data files, the data files which do not meet the requirements can be screened out, and finally the data of the credit blacklist passengers in the data files which pass the pre-verification and warehousing verification are imported into a blacklist database, so that the centralized management of the data of the credit blacklist passengers is realized, the problem of data dispersion of the credit blacklist passengers is solved, the difficulty of later-stage data maintenance can be effectively reduced, and the maintenance cost is saved.
In the method provided in the embodiment of the present invention, when it is determined that the file type of the data file is a full data file, a process of performing warehousing verification corresponding to the file type on the data file, and when the data file passes the warehousing verification, importing the data of the credit blacklist traveler in the data file into a preset blacklist database may be referred to as a full data processing flow, and referring to fig. 5, a flowchart of a method of the full data processing flow provided in the embodiment of the present invention is specifically described as follows:
1. acquiring a data source file from a specified SFTP; the data source file is a data file which passes the pre-verification.
2. Carrying out file name verification on a data source file; it should be noted that, when checking the file name, it is checked whether the file name of the data source file contains the name identifier of the full file, specifically as follows: full; further file name naming specifications are: YYYYMMDD + two digit. xml; when the file name is verified, step 3 is executed, and when the file name is not verified, step 7 is executed.
3. Carrying out file format verification on a data source file; wherein, the file format check can also be called xml format check; and 4, when the data source file passes the file format verification, executing step 4, and when the data source file does not pass the file format verification, executing step 7.
4. And acquiring the data source file, and deleting the record of the data source file name in the temporary library.
5. And analyzing the data source file and entering a temporary library.
It should be noted that when the data source file is loaded into the temporary repository, field verification needs to be performed on the data source file, and when the field verification on the data source file fails, step 7 is executed, and further, the processing state and the warehousing record of the data file need to be recorded.
After the data source file passes through the field verification, extracting data from the data source file according to the data extraction rule of each preset field of the temporary library, and storing the extracted data into the temporary library, wherein the data is intercepted according to the maximum length of the field, which is exemplarily shown as follows: the case ID field is less than or equal to 200, the certificate number field is less than or equal to 100, the name field is less than or equal to 250, the gender field is less than or equal to 10, the execution department field is less than or equal to 100, and the execution basis unit field is less than or equal to 100.
6. And newly adding the blacklist information into a formal library.
It should be noted that, when the data source file is imported into the formal library, data needs to be extracted according to each preset field of the formal library, and each preset field of the formal library includes, but is not limited to, the following fields:
event ID field: 'pure number' or 'committee trigram + number';
name field: chinese or English, which is currently Chinese;
a gender field: male or female or empty;
age field: a number or null;
certificate number field: encrypted storage, 15 bits need to be converted into 18 bits;
the affiliated commission field: ID pure numbers or ID commission three-character codes + numbers.
7. And generating a verification feedback file.
When the verification feedback file is generated, if the verification feedback file is generated without passing the verification, the verification feedback file includes data such as a specific reason why the verification fails, a processing state of the data file, and a warehousing record.
If the verification feedback file is generated after the verification feedback file is put into the formal library, the verification feedback file is a null file which only comprises a head row and a tail row of the file and has no information row.
Further, after the verification feedback file is generated, the data source file and the feedback verification file may be placed in a designated directory, for example, the feedback verification file is sent to the relay platform.
In the method provided in the embodiment of the present invention, when it is determined that the file type of the data file is an incremental data file, a process of performing warehousing verification corresponding to the file type on the data file, and when the data file passes the warehousing verification, importing the data of the credit blacklist traveler in the data file into a preset blacklist database may be referred to as an incremental data processing flow, and referring to fig. 6, a flowchart of a method of the incremental data processing flow provided in the embodiment of the present invention is specifically described as follows:
1. acquiring a data source file from a specified SFTP; the data source file is a data file which passes the pre-verification.
2. Carrying out file name verification on a data source file; it should be noted that, when performing file name verification, it is verified whether the file name of the data source file contains a naming identifier of the incremental file, specifically: daily; further file name naming specifications are: YYYYMMDD + two digit. xml; when the file name is verified, step 3 is executed, and when the file name is not verified, step 7 is executed.
3. Checking the file format: wherein, the file format check can also be called xml format check; and 4, when the data source file passes the file format verification, executing step 4, and when the data source file does not pass the file format verification, executing step 7.
4. And acquiring a data source file name, deleting the record in the temporary library, which is the same as the data source file name, and cleaning the data in the temporary library, which is the same as the data source file name at this time.
5. And analyzing the data source file and entering a temporary library.
It should be noted that when the data source file is loaded into the temporary repository, field verification needs to be performed on the data source file, and when the field verification on the data source file fails, step 7 is executed, and further, the processing state and the warehousing record of the data file need to be recorded.
After the data source file passes through the field verification, extracting data from the data source file according to the data extraction rule of each preset field of the temporary library, and storing the extracted data into the temporary library, wherein the data is intercepted according to the maximum length of the field, which is exemplarily shown as follows: the case ID field is less than or equal to 200, the certificate number field is less than or equal to 100, the name field is less than or equal to 250, the gender field is less than or equal to 10, the execution department field is less than or equal to 100, and the execution basis unit field is less than or equal to 100.
6. And performing addition, deletion and modification operations according to the information operation type blacklist passenger information, and entering a formal library.
It should be noted that, when the data source file is imported into the formal library, data needs to be extracted according to each preset field of the formal library, and each preset field of the formal library includes, but is not limited to, the following fields:
event ID: 'pure number' or 'committee trigram + number'.
Name: chinese and English, which are both Chinese at present.
Sex: male or female or empty.
Age: a number or null.
The certificate number: for encrypted storage (requiring detailed writing of encryption), 15 bits need to be converted into 18 bits.
The affiliated minister: ID pure numbers or ID commission three-character codes + numbers.
7. Generating a verification feedback file; it should be noted that the verification feedback file herein includes a warehousing feedback file and a non-unique certificate number data feedback file. The data feedback file of the non-unique certificate number comprises blacklist passenger information which has the same certificate number but different names in the data file, wherein the data are arranged according to the sequence of the certificate numbers.
When the verification feedback file is generated, if the verification feedback file is generated without passing the verification, the warehousing feedback file of the verification feedback file includes specific reasons for failing the verification, the processing state of the data file, the warehousing record, and other data.
And if the verification feedback file is generated after entering the formal library, the warehousing feedback file of the verification feedback file is an empty file only containing the head and tail lines of the file and no information.
Further, after the verification feedback file is generated, the data source file and the feedback file are placed into an appointed directory, specifically, a transit platform.
Corresponding to the method shown in fig. 1, an embodiment of the present invention provides a data processing apparatus, which is configured on a unified platform for civil aviation credit data, and is used to support the implementation of the method shown in fig. 1 in actual life, and referring to fig. 7, a schematic structural diagram of the data processing apparatus provided in the embodiment of the present invention is specifically described as follows:
the capturing unit 701 is used for capturing a data file uploaded to a transit platform by a data source, wherein the data file comprises data of a credit blacklist passenger;
a pre-verification unit 702, configured to pre-verify the data file, and determine a file type of the data file when the data file passes the pre-verification;
the warehousing unit 703 is configured to perform warehousing verification on the data file, where the warehousing verification corresponds to the file type, and when the data file passes the warehousing verification, import the data of the credit blacklist traveler in the data file into a preset blacklist database.
In the device provided by the embodiment of the invention, a data file uploaded to a transfer platform from a data source is captured, wherein the data file comprises data of a credit blacklist passenger; pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking; and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification. By carrying out pre-verification and warehousing verification on the data files, the data files which do not meet the requirements can be screened out, and finally the data of the credit blacklist passengers in the data files which pass the pre-verification and warehousing verification are imported into a blacklist database, so that the centralized management of the data of the credit blacklist passengers is realized, the problem of data dispersion of the credit blacklist passengers is solved, the difficulty of later-stage data maintenance can be effectively reduced, and the maintenance cost is saved.
In an embodiment of the present application, based on the foregoing scheme, the pre-verification unit 702 may be configured to:
the first judging subunit is used for judging whether the file name of the data file meets a preset file name naming specification or not;
the first determining subunit is used for determining whether the file format of the data file is a standard format or not when the file name of the data file meets a file name naming specification;
and the checking subunit is used for checking each file field in the data file when the file format of the data file is a standard format, and determining that the data file passes the check when each file field passes the check.
In an embodiment of the present application, based on the foregoing scheme, the pre-verification unit 702 may be configured to:
the acquisition subunit is used for acquiring the file name of the data file;
the second judgment subunit is used for judging whether the file identifier in the file name is a full identifier or not;
the second determining subunit is configured to determine that the file type of the data file is a full data file if the file identifier in the file name is a full identifier;
and the third determining subunit is configured to determine that the file identifier in the file name is an incremental identifier and determine that the file type of the data file is an incremental data file, if the file identifier in the file name is not a full identifier.
In an embodiment of the present application, based on the foregoing scheme, the warehousing unit 703 may be configured to:
the first extraction subunit is used for extracting temporary table data meeting a preset temporary table structure from the data file and storing the temporary table data into a preset temporary library;
the second extraction subunit is used for extracting formal table data meeting a preset formal table structure from the data file and storing the formal table data into a preset formal library;
and the third extraction subunit is used for extracting credit blacklist passenger data meeting a preset blacklist table structure from the temporary table data and the formal table data, and storing the credit blacklist passenger data in a blacklist database.
In an embodiment of the present application, based on the foregoing solution, the apparatus may be further configured to:
and the generating unit is used for generating a verification failure file of the data file when the data file does not pass the pre-verification, and storing the verification failure file to the transfer platform.
The embodiment of the present invention further provides a storage medium, where the storage medium includes a stored instruction, where when the instruction runs, the apparatus where the storage medium is located is controlled to perform the following operations:
capturing a data file uploaded to a transfer platform by a data source, wherein the data file comprises data of credit blacklist passengers;
pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking;
and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification.
An electronic device is provided in an embodiment of the present invention, and the structural diagram of the electronic device is shown in fig. 8, which specifically includes a memory 801 and one or more instructions 802, where the one or more instructions 802 are stored in the memory 801 and configured to be executed by the one or more processors 803 to perform the following operations:
capturing a data file uploaded to a transfer platform by a data source, wherein the data file comprises data of credit blacklist passengers;
pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking;
and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification.
The specific implementation procedures and derivatives thereof of the above embodiments are within the scope of the present invention.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
While several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
The foregoing description is only exemplary of the preferred embodiments disclosed herein and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the disclosure. For example, the above features and (but not limited to) technical features having similar functions disclosed in the present disclosure are mutually replaced to form the technical solution.
Fig. 1 provides a data processing method according to one or more embodiments disclosed in the present application, including:
capturing a data file uploaded to a transfer platform by a data source, wherein the data file comprises data of credit blacklist passengers;
pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking;
performing warehousing verification corresponding to the file type on the data file, and when the data file passes the warehousing verification, importing the data of the credit blacklist passengers in the data file into a preset blacklist database;
and when the data file does not pass the pre-verification, generating a verification failure file of the data file, and storing the verification failure file to the transfer platform.
Fig. 2 provides a flowchart of a method for pre-committing data files according to one or more embodiments disclosed in the present application, including:
judging whether the file name of the data file meets a preset file name naming standard or not;
when the file name of the data file meets the file name naming specification, determining whether the file format of the data file is a standard format;
when the file format of the data file is a standard format, checking each file field in the data file, and when each file field passes the check, determining that the data file passes the check.
FIG. 3 provides a flow diagram of a method of determining a file type of a data file, in accordance with one or more embodiments disclosed herein, comprising:
acquiring the file name of the data file;
judging whether the file identifier in the file name is a full identifier or not;
if the file identifier in the file name is a full identifier, determining that the file type of the data file is a full data file;
and if the file identifier in the file name is not the full identifier, determining that the file identifier in the file name is an incremental identifier, and determining that the file type of the data file is an incremental data file.
Fig. 4 provides a flowchart of a method for importing a data file into a blacklist database according to one or more embodiments disclosed in the present application, including:
extracting temporary table data meeting a preset temporary table structure from the data file, and storing the temporary table data into a preset temporary library;
formal table data meeting a preset formal table structure are extracted from the data file, and the formal table data are stored in a preset formal library;
and credit blacklist passenger data meeting a preset blacklist table structure are extracted from the temporary table data and the formal table data, and the credit blacklist passenger data are stored in a blacklist database.
Fig. 4 provides a data processing apparatus according to one or more embodiments disclosed herein, including:
the system comprises a capturing unit, a transferring unit and a processing unit, wherein the capturing unit is used for capturing a data file uploaded to a transfer platform by a data source, and the data file comprises data of credit blacklist passengers;
the pre-checking unit is used for pre-checking the data file and determining the file type of the data file when the data file passes the pre-checking;
and the warehousing unit is used for performing warehousing verification corresponding to the file type on the data file and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data processing method, comprising:
capturing a data file uploaded to a transfer platform by a data source, wherein the data file comprises data of credit blacklist passengers;
pre-checking the data file, and determining the file type of the data file when the data file passes the pre-checking;
and performing warehousing verification corresponding to the file type on the data file, and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification.
2. The method of claim 1, wherein pre-verifying the data file comprises:
judging whether the file name of the data file meets a preset file name naming standard or not;
when the file name of the data file meets the file name naming specification, determining whether the file format of the data file is a standard format;
when the file format of the data file is a standard format, checking each file field in the data file, and when each file field passes the check, determining that the data file passes the check.
3. The method of claim 1, wherein determining the file type of the data file comprises:
acquiring the file name of the data file;
judging whether the file identifier in the file name is a full identifier or not;
if the file identifier in the file name is a full identifier, determining that the file type of the data file is a full data file;
and if the file identifier in the file name is not the full identifier, determining that the file identifier in the file name is an incremental identifier, and determining that the file type of the data file is an incremental data file.
4. The method of claim 1, wherein importing the data of the credit blacklisted passenger in the data file into a preset blacklist database comprises:
extracting temporary table data meeting a preset temporary table structure from the data file, and storing the temporary table data into a preset temporary library;
formal table data meeting a preset formal table structure are extracted from the data file, and the formal table data are stored in a preset formal library;
and credit blacklist passenger data meeting a preset blacklist table structure are extracted from the temporary table data and the formal table data, and the credit blacklist passenger data are stored in a blacklist database.
5. The method of claim 1, further comprising:
and when the data file does not pass the pre-verification, generating a verification failure file of the data file, and storing the verification failure file to the transfer platform.
6. A data processing apparatus, comprising:
the system comprises a capturing unit, a transferring unit and a processing unit, wherein the capturing unit is used for capturing a data file uploaded to a transfer platform by a data source, and the data file comprises data of credit blacklist passengers;
the pre-checking unit is used for pre-checking the data file and determining the file type of the data file when the data file passes the pre-checking;
and the warehousing unit is used for performing warehousing verification corresponding to the file type on the data file and importing the data of the credit blacklist passengers in the data file into a preset blacklist database when the data file passes the warehousing verification.
7. The apparatus of claim 6, wherein the pre-verification unit comprises:
the first judging subunit is used for judging whether the file name of the data file meets a preset file name naming specification or not;
the first determining subunit is used for determining whether the file format of the data file is a standard format or not when the file name of the data file meets a file name naming specification;
and the checking subunit is used for checking each file field in the data file when the file format of the data file is a standard format, and determining that the data file passes the check when each file field passes the check.
8. The apparatus of claim 6, wherein the pre-verification unit comprises:
the acquisition subunit is used for acquiring the file name of the data file;
the second judgment subunit is used for judging whether the file identifier in the file name is a full identifier or not;
the second determining subunit is configured to determine that the file type of the data file is a full data file if the file identifier in the file name is a full identifier;
and the third determining subunit is configured to determine that the file identifier in the file name is an incremental identifier and determine that the file type of the data file is an incremental data file, if the file identifier in the file name is not a full identifier.
9. A storage medium comprising stored instructions, wherein the instructions, when executed, control a device on which the storage medium resides to perform a data processing method according to any one of claims 1 to 5.
10. An electronic device comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by the one or more processors to perform the data processing method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111660499.5A CN114328413A (en) | 2021-12-30 | 2021-12-30 | Data processing method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111660499.5A CN114328413A (en) | 2021-12-30 | 2021-12-30 | Data processing method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114328413A true CN114328413A (en) | 2022-04-12 |
Family
ID=81019605
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111660499.5A Pending CN114328413A (en) | 2021-12-30 | 2021-12-30 | Data processing method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328413A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106528732A (en) * | 2016-11-03 | 2017-03-22 | 球宝互动（北京）网络科技有限公司 | Blacklist system and client for sports events |
| CN107798068A (en) * | 2017-09-26 | 2018-03-13 | 浙江极赢信息技术有限公司 | A kind of processing method, system and the relevant apparatus of user data of breaking one's promise |
| CN107896157A (en) * | 2017-08-31 | 2018-04-10 | 上海壹账通金融科技有限公司 | Blacklist data exchange method and application server |
| CN109165335A (en) * | 2018-06-26 | 2019-01-08 | 杭州排列科技有限公司 | Internet finance blacklist system and its application method based on big data |
| CN110941593A (en) * | 2019-12-03 | 2020-03-31 | 浪潮卓数大数据产业发展有限公司 | File warehousing system and method |
| US20200143071A1 (en) * | 2017-07-31 | 2020-05-07 | Ping An Technology (Shenzhen) Co., Ltd. | Data sharing method, device and computer readable storage medium |
| CN112148711A (en) * | 2020-09-21 | 2020-12-29 | 建信金融科技有限责任公司 | Processing method and device for batch processing tasks |
| CN112380167A (en) * | 2020-11-17 | 2021-02-19 | 深圳市和讯华谷信息技术有限公司 | Batch data verification method and device, computer equipment and storage medium |
| CN112468532A (en) * | 2020-10-12 | 2021-03-09 | 苏宁金融科技(南京)有限公司 | Credit investigation data sending method, device, system, equipment and computer storage medium |
| CN112463729A (en) * | 2020-11-27 | 2021-03-09 | 中国工商银行股份有限公司 | Data file storage method and device, electronic equipment and medium |
-
2021
- 2021-12-30 CN CN202111660499.5A patent/CN114328413A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106528732A (en) * | 2016-11-03 | 2017-03-22 | 球宝互动（北京）网络科技有限公司 | Blacklist system and client for sports events |
| US20200143071A1 (en) * | 2017-07-31 | 2020-05-07 | Ping An Technology (Shenzhen) Co., Ltd. | Data sharing method, device and computer readable storage medium |
| CN107896157A (en) * | 2017-08-31 | 2018-04-10 | 上海壹账通金融科技有限公司 | Blacklist data exchange method and application server |
| CN107798068A (en) * | 2017-09-26 | 2018-03-13 | 浙江极赢信息技术有限公司 | A kind of processing method, system and the relevant apparatus of user data of breaking one's promise |
| CN109165335A (en) * | 2018-06-26 | 2019-01-08 | 杭州排列科技有限公司 | Internet finance blacklist system and its application method based on big data |
| CN110941593A (en) * | 2019-12-03 | 2020-03-31 | 浪潮卓数大数据产业发展有限公司 | File warehousing system and method |
| CN112148711A (en) * | 2020-09-21 | 2020-12-29 | 建信金融科技有限责任公司 | Processing method and device for batch processing tasks |
| CN112468532A (en) * | 2020-10-12 | 2021-03-09 | 苏宁金融科技(南京)有限公司 | Credit investigation data sending method, device, system, equipment and computer storage medium |
| CN112380167A (en) * | 2020-11-17 | 2021-02-19 | 深圳市和讯华谷信息技术有限公司 | Batch data verification method and device, computer equipment and storage medium |
| CN112463729A (en) * | 2020-11-27 | 2021-03-09 | 中国工商银行股份有限公司 | Data file storage method and device, electronic equipment and medium |
Non-Patent Citations (1)
| Title |
|---|
| 樊小玲等: "军队卫生信息资源开发利用研究与实践", 30 June 2012, 人民军医出版社, pages: 212 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1945530B (en) | Arranging system and method for module having dependence | |
| CN107679057B (en) | Data interconnection method, device, server and storage medium | |
| US20140207741A1 (en) | Data retention component and framework | |
| CN108959385B (en) | Database deployment method, device, computer equipment and storage medium | |
| US10552293B2 (en) | Logging as a service | |
| CN111683066B (en) | Heterogeneous system integration method, heterogeneous system integration device, computer equipment and storage medium | |
| CN102576344A (en) | Method and system to recognize and inventory applications | |
| CN111125042A (en) | Method and device for determining risk operation event | |
| CN108804241B (en) | Cross-platform task scheduling method, system, computer equipment and storage medium | |
| CN109189749A (en) | File synchronisation method and terminal device | |
| CN103377406A (en) | System and method for managing test files | |
| CN112446022A (en) | Data authority control method and device, electronic equipment and storage medium | |
| CN108958969B (en) | Database disaster recovery method, device and disaster recovery and backup systems | |
| CN111708794A (en) | Data comparison method and device based on big data platform and computer equipment | |
| CN109005167B (en) | Authentication data processing method and device, server and storage medium | |
| US20140201709A1 (en) | JavaScript™ Deployment Build Tool for software code that uses an object literal to define meta data and system code. | |
| CN102801728B (en) | The management method of automatic login of client side and system | |
| US20150046393A1 (en) | Method and device for executing an enterprise process | |
| CN108108478B (en) | Data format conversion method and system and electronic equipment | |
| CN102257498B (en) | Comment generation method of configuration files and configuration file generation device | |
| US20180336171A1 (en) | System and method for constructing extensible event log with javascript object notation (json) encoded payload data | |
| CN114328413A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN107025214A (en) | Data processing method and device | |
| CN106855888A (en) | Daily record monitoring system based on Logstash distributed systems | |
| CN113609531B (en) | Information interaction method, device, equipment, medium and product based on block chain |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |