CN114936915A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114936915A CN114936915A CN202210190295.8A CN202210190295A CN114936915A CN 114936915 A CN114936915 A CN 114936915A CN 202210190295 A CN202210190295 A CN 202210190295A CN 114936915 A CN114936915 A CN 114936915A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- category information
- data dictionary
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 18
- 238000004458 analytical method Methods 0.000 claims abstract description 20
- 238000000034 method Methods 0.000 claims description 35
- 238000012545 processing Methods 0.000 claims description 26
- 238000004590 computer program Methods 0.000 claims description 14
- 230000004044 response Effects 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims 1
- 238000007405 data analysis Methods 0.000 abstract description 3
- 238000007726 management method Methods 0.000 description 22
- 230000010354 integration Effects 0.000 description 15
- 230000008859 change Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- BASFCYQUMIYNBI-UHFFFAOYSA-N platinum Chemical compound [Pt] BASFCYQUMIYNBI-UHFFFAOYSA-N 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 239000010931 gold Substances 0.000 description 1
- 229910052737 gold Inorganic materials 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 229910052697 platinum Inorganic materials 0.000 description 1
- 238000004801 process automation Methods 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/194—Calculation of difference between files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data processing method, an apparatus, an electronic device and a storage medium, and relates to the field of computers, in particular to the field of data analysis and intelligent search. The specific implementation scheme is as follows: acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; a target file is generated for analysis by a downstream system based on the target category data.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium in the fields of data analysis and intelligent search.
Background
At present, in the business processing process of a bank, a unionpay provides a data dictionary for a server of the bank, and a bank data analyzer needs to manually analyze the data dictionary to a data warehouse of the bank for analysis in a downstream system.
Disclosure of Invention
The present disclosure provides a method, an apparatus, an electronic device, and a storage medium for data processing.
According to an aspect of the present disclosure, a data processing method is provided. The method comprises the following steps: acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; a target file is generated for analysis by a downstream system based on the target category data.
According to another aspect of the present disclosure, there is also provided another data processing apparatus. The device includes: the system comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring a data dictionary, and the data dictionary is used for representing information of a bank card to be analyzed; the second acquisition unit is used for acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; a generating unit for generating a target file analyzed by a downstream system based on the target category data.
According to another aspect of the present disclosure, an electronic device is also provided. The electronic device may include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing methods of the embodiments of the present disclosure.
According to another aspect of the present disclosure, there is also provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the data processing method of the embodiments of the present disclosure.
According to another aspect of the present disclosure, there is also provided a computer program product, which may comprise a computer program, which when executed by a processor, implements the data processing method of the embodiments of the present disclosure.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 is a flow chart of a method of data processing according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of an automated parsing device for a distributed change bank card core system data dictionary according to instability characteristics in an embodiment of the disclosure;
FIG. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 4 is a block diagram of an electronic device of a data processing method according to an embodiment of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a flow chart of a data processing method according to an embodiment of the present disclosure. As shown in fig. 1, the method may include the steps of:
step S102, a data dictionary is obtained, wherein the data dictionary is used for representing information of the bank card to be analyzed.
In the technical scheme provided in the foregoing step S102 of the present disclosure, the server of the bank may obtain the data dictionary from the union-banking cloud, where the data dictionary is used to represent information of the bank card to be analyzed, may be a core data dictionary file of the bank card, and may include file name information, update frequency information, increment planning information, primary key information, field information, and the like, and this is not limited specifically here.
Optionally, the union-banking cloud may synchronize the bank card core data dictionary file to a server (banking machine) of the bank at regular time, so as to achieve the purpose that the server of the bank obtains the data dictionary.
Alternatively, the bank card may be a credit payment tool, which may refer to a credit payment tool approved to be issued to society by financial institutions (e.g., commercial banks, etc.) and having all or part of functions of consuming credit, transferring accounts, accessing cash, etc., and the bank card in this embodiment may include: credit cards (credit cards) and digital bank cards, wherein the credit cards can be divided into card issuing banks for giving a certain credit line to a card holder, the card holder can consume in the credit line first and then pay, and the credit cards and the card holder pay reserve money according to bank requirements first, and when the reserve money is not sufficient for payment, the credit cards can be quasi-credit cards overdrawn in the credit line specified by the card issuing banks; the digital bank card may be a flash payment card, etc., and is not particularly limited herein.
Alternatively, the bank card of this embodiment may further include a debit card that can transfer and withdraw money at a network or an automated teller machine, which can be classified by function into a transfer card having functions of transferring money, accessing cash and consuming money, a dedicated card used in a specific area for a dedicated purpose, and a prepaid wallet type value card; the card can be classified into a common card, a gold card and a platinum card according to grades, and can be classified into a national card and an international card according to the range of use.
The preferred application scenario of this embodiment is for the relevant data dictionary file of the credit card core system; although the business system of the debit card is independent from the business system of the credit card (debit card), the debit card and the credit card both have respective core data dictionary files, wherein the access business of the debit card is simple, and the business and analysis scenes of the credit card (debit card) are complex, so the bank card of the embodiment can not only comprise the credit card (debit card) and the digital bank card, but also comprise the debit card.
For example, the server may obtain the data dictionary to be analyzed from a credit card (debit card), a digital bank card, a debit card, or other credit payment instrument.
And step S104, acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card.
In the technical solution provided in the above step S104 of the present disclosure, the union pay cloud analyzes the information corresponding to the acquired data dictionary to acquire the target category information of the bank card corresponding to the downstream system, where the downstream system may be a system used in an application scenario of the bank card, and may be a system implemented by using a first relational database management system (mysql), a system implemented by using a second relational database management system (Oracle), a data integration system, and the like; the target category information may be used for creating category information and updating category information, and the like, which may be generated by a mysql generating program configured for a first relational database management system (mysql); creation category information and update category information that may also be generated for an Oracle generator configured for a second relational database management system (Oracle); the configuration file may be generated by a generating program configured for the data integration system, and the system generating program are not particularly limited, but are merely examples.
Optionally, a data dictionary is obtained, information corresponding to a downstream system in the data dictionary is analyzed, and target category information of the bank card is obtained.
For example, a data dictionary is obtained, and if the downstream system is a system implemented by using a first relational database management system (mysql), the creation category information and the update category information are generated based on a first relational database management system generation program.
Step S106, generating a target file analyzed by a downstream system based on the target category data.
In this embodiment, information corresponding to a downstream system in the data dictionary is analyzed to obtain target category information of the bank card, and a target file analyzed by the downstream system is generated based on the target category data, where the target file may be a table-building structured query language (sql) of different databases, a table structure change sql, a configuration file required by the data integration system, and the like.
Optionally, a data dictionary is obtained, information corresponding to a downstream system in the data dictionary is analyzed, target category information of the bank card is obtained, based on the target category information, task distribution processing is performed on the information corresponding to the data dictionary, and configuration files required by different target systems of the downstream system are generated, for example, table building sql of different databases, table structure changing sql, configuration files required by a data integration system, and the like.
For example, a data dictionary is obtained, if the downstream system is a first relational database management system (mysql), the creation category information and the update category information are generated based on a first relational database management system generation program, and a table creation sql statement and a table update sql statement are generated based on the creation category information and the update category information.
For example, a data dictionary is obtained, if the downstream system is a second relational database management system (Oracle), the creation category information and the update category information are generated based on a second relational database management system generation program, and the sql statement and the table update sql statement are created based on the creation category information and the update category information generation table.
For example, a data dictionary is obtained, and if the downstream system is a data integration system, a configuration file and the like are generated based on the data integration system.
Acquiring a data dictionary through the steps S102 to S106, wherein the data dictionary is used for representing information of the bank card to be analyzed; acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; target files analyzed by a downstream system are generated based on the target category data. That is to say, in the present disclosure, after analyzing the corresponding information in the data dictionary, the target category information is obtained, and based on the target category information, the target file is generated, so that the downstream system can directly analyze the target file, and thus, the data dictionary of the bank card can be automatically analyzed, the efficiency of processing the data of the bank card is further improved, and the technical problem of low efficiency of processing the data of the bank card is solved.
The above-described method of this embodiment is described in further detail below.
As an optional implementation manner, in step S204, the obtaining of the object class information of the bank card from the data dictionary includes: and reading the target category information from the data dictionary based on the target tool kit.
In this embodiment, the table file in the data dictionary is parsed based on the target toolkit, so as to obtain the target category information, where the target toolkit may be an extension tool (xlrd) for reading a table in a computer programming language (Python), and may be used to read the table file.
Optionally, the target toolkit based on Python parses the data dictionary table file, and generates target category information required by different downstream target systems based on a parsing result of the table file.
Optionally, the target tool is called to obtain the position of the corresponding field in the table according to the predetermined parsing logic, such as the position of the file name, the updating frequency, the primary key, the field separation rule, the field name, the field type, the field length, the field remark, and other information in the table, so as to obtain the target category information.
As an optional implementation, reading the target category information from the data dictionary based on the target toolkit includes: reading a plurality of category information from the data dictionary to a memory based on the target toolkit; and acquiring the target category information in the plurality of category information from the memory by using the controller corresponding to the target category information.
In this embodiment, a target toolkit is called to analyze a table file in a data dictionary and transmit the table file to a memory, a controller corresponding to target category information is determined, the controller obtains the target category information from the memory according to pre-developed configuration file programs required by different downstream systems and places the target category information into a designated position, so as to achieve the purpose of obtaining the target category information in a plurality of category information from the memory, wherein the controller corresponding to the target category information may be a downstream controller, and may be a create-class task controller and an update-class task controller.
Optionally, the target toolkit is called to analyze a table file in the data dictionary and transmit the table file to a memory, the task distribution module starts a downstream controller, and may start a creation-like task controller, the creation-like task controller obtains target category information from the memory according to pre-developed configuration file programs required by different downstream systems and places the target category information into a designated location, for example, the task distribution module starts the downstream controller, the controller controls the task to start, and when the downstream system is a first relational database management system, the controller obtains the target category information from the memory according to a first relational database management system configuration generation program.
Optionally, the update class task controller may be started, and the update class task controller obtains the target class information from the memory according to the configuration file programs required by different pre-developed downstream systems, and places the target class information to the designated target.
Alternatively, the create class task controller and the update class task controller may be started simultaneously, wherein. The creation-class task controller generates a creation-class task each time, and the update-class task can generate a specific update file through a configuration file program required by a downstream system when the last version dictionary file is checked to exist and the contrast exists.
As an alternative implementation, in step S206, the target category information includes update-class task data, and generating a target file analyzed by a downstream system based on the target category data includes: and updating the original file to be analyzed of the downstream system based on the updating task data to obtain the target file.
In this embodiment, when the update-type task data is obtained, the update-type task data may be compared with an original file, and when there is a difference in the comparison, the original file is updated to obtain a target file, where the original file may be a data dictionary file of a previous version smaller than a current version; the update class task data may be table data of a newly acquired data dictionary.
Optionally, the data dictionary may have time sequence numbers on file names, the data dictionary is ordered according to the data dictionary names to obtain a data dictionary file of a version before the current version, the downstream service system automatically loads the original file after detecting that the update-type latest configuration file is generated, compares the update-type latest configuration file with the original file, and updates the original file when there is a difference in comparison, so as to obtain the target file.
As an optional implementation, the method further comprises: comparing the target category information with the original category information of the original data dictionary to obtain a comparison result, wherein the original data dictionary is used for representing the original information of the bank card to be analyzed; and in response to the comparison result representing that the target category information is different from the original category information, generating update-type task data based on the comparison result.
In the embodiment, the target category information is compared with the original category information of the original data dictionary to obtain a comparison result, and if the target category information is different from the original category information of the original data dictionary, the updated task data is generated in response to the comparison result with the difference.
Optionally, the original information of the bank card to be analyzed is compared with the target category information to obtain a comparison result, and if the target category information is different from the original information, the update-type task data is generated.
Optionally, the embodiment may further set a change comparison record generating program, and when there is a difference between the target category information and the original information, the change comparison record generating program is triggered to automatically generate a change record file, where the change record may include an update date, an update content, and the like, and it is to be noted that the change record content may be set according to an actual requirement, and this is not specifically described here.
As an optional implementation, the method further comprises: issuing an update message to a downstream system, wherein the update message is used for indicating that an original file is updated into a target file; and receiving a request sent by the downstream system based on the update message, responding to the request, and sending the target file to the downstream system.
In the embodiment, the update task acquires the data of the update task from the memory according to pre-developed configuration file programs required by different downstream systems to generate corresponding target files, and places the target files into the designated directory, when an original file is updated to be the target file, the update task manager issues an update message to the downstream systems, receives a request sent by the downstream systems based on the update message, responds to the request, and issues the target files to the downstream systems.
Optionally, the update task acquires the update task data from the memory according to pre-developed configuration file programs required by different downstream systems to generate corresponding files, and the corresponding files are placed at the designated positions, and after the downstream service system detects that the target files are updated, the corresponding files are automatically loaded, so that the target files are delivered to the downstream systems.
As an optional implementation, the target category information includes creation-class task data, and generating a target file analyzed by a downstream system based on the target category data includes: a target file is created based on the create-class task data for analysis by the downstream system.
In this embodiment, the target category information includes creation class task data, which may be table data, and a target file analyzed by a downstream system is created based on the creation class task data, where the target file may create an sql statement for a table generated by a first relational database management system (mysql) configuration generator or a table generated by a second relational database management system (Oracle) configuration generator.
It should be noted that the actual table building statements and field types of mysql and Oracle are different, different creating/updating statements need to be generated, and a general mysql configuration generation program and an Oracle configuration generation program cannot be started at the same time, and can be configured and started according to the requirements of a downstream system.
Optionally, the task distribution module may start a downstream creation-class task controller, and the creation-class task acquires data of the creation-class task from the memory according to pre-developed configuration file programs (e.g., mysql configuration generation program, Oracle configuration generation program) required by different downstream systems to generate a target file analyzed by the downstream systems.
As an alternative implementation, in step S202, acquiring the data dictionary includes: and reading the data dictionary with the timestamp closest to the current time from the first storage position according to the target time period.
In this embodiment, the data dictionary with the timestamp closest to the current time is read from the first storage location according to a target time period, where the first storage location may be a designated directory, and the target time period may be set according to an actual requirement, and may be 1 or 2, and is not specifically limited herein.
Optionally, the latest data dictionary in the first storage location (specified directory) is read, and the data dictionary with the timestamp closest to the current time is read from the first storage location according to the target time period, for example, when it is determined whether there is a file of the latest day, the detection may be performed according to a T-1 manner, and if a task is triggered at 28 th 12 th 2021, a file with a data dictionary suffix of 27 th 12 th 2021 is detected, where the numbers are merely for example and are not limited specifically.
As an optional implementation, the method further comprises: and storing the target file to a second storage position, wherein the target file is acquired from the second storage position by a downstream system.
In the embodiment, a data dictionary is obtained, target class information of the bank card is obtained from the data dictionary, a target file analyzed by a downstream system is generated based on the target class information, the target file is stored to a second storage position, and the downstream system obtains the target file from the second storage position.
Optionally, after reading the latest dictionary, the parsing device may parse the dictionary file into the memory using the target toolkit, call the target toolkit to obtain target category information according to a predetermined parsing logic, for example, information such as a file name, an update frequency, a primary key, a field separation rule, a field name, a field type, a field length, a field remark, and the like, then store the target file (for example, in the form of a global shared variable, a data structure may be a set of built structures) in the memory in the second storage location, and the downstream system may directly obtain the target file from the second storage location.
In the embodiment, a data dictionary is obtained, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; target files analyzed by a downstream system are generated based on the target category data. That is to say, in the present disclosure, after analyzing the corresponding information in the data dictionary, the target category information is obtained, and based on the target category information, the target file is generated, so that the downstream system can directly analyze the target file, and thus, the data dictionary of the bank card can be automatically analyzed, the efficiency of processing the data of the bank card is further improved, and the technical problem of low efficiency of processing the data of the bank card is solved.
The above technical solutions of the embodiments of the present disclosure are further exemplified by the following preferred embodiments.
At present, in bank card business of banks, most banks can directly purchase a bank card core business system of the Unionpay, the card core system is hosted in the Unionpay cloud, a bank data analysis department cannot directly access the bank card core business system, a general Unionpay synchronizes data to a server of the bank every day through a data exchange system, and the bank analyzes the data to a data warehouse of the bank according to the data files.
In the related technology processing process, the Unionpay provides a data dictionary, the data dictionary relates to hundreds of tables, each table has hundreds of fields and dozens of fields, bank data analysts need to copy the fields one by one and modify the fields into form building statements, the tables change statements and code configuration, the Unionpay updates the fields regularly, but after each time the data dictionary is updated, the banks need to compare the specific fields manually and modify the table structure, configuration and codes.
Therefore, in order to solve the above problems, in the embodiment of the present disclosure, based on a device for automatically parsing a data dictionary table (Excel) by using an xlrd toolkit of a Python read table, a relevant configuration file for building the table Sql and developing codes is automatically generated; under the condition that a historical version exists, the historical version is compared with a previous version data dictionary to generate an Sql updating table and change records, so that the development efficiency and the accuracy are greatly improved, the change of the sensing data can be automatically compared, the relevant change records are generated, and the automatic processing of the whole process of the core system data of the subsequent union pay card can be realized.
Optionally, the data integration system needs to perform different structured analyses according to conditions such as different field lengths, delimiter types, and field similarities in the excel, the data integration system code is an abstract general code, does not need to modify the code, can adapt to different tasks by modifying the configuration, and can convert the analysis logic into the configuration required by the data integration system
Fig. 2 is a schematic diagram of an automated analysis device for a core system data dictionary of a bank card according to distribution variation of instability characteristics in an embodiment of the present disclosure, as shown in fig. 2, a union of bank may synchronize a core data dictionary file of the bank card to a bank machine at regular time, and the automated analysis device for a core system data dictionary of a bank card (hereinafter, referred to as an analysis device) and the core data dictionary file of the bank card are deployed in the same server.
Firstly, a task is timed every day to automatically trigger an analysis device.
Alternatively, the task may be timed to automatically trigger the parsing means by a component such as a timer.
And secondly, reading the latest dictionary file of the specified directory by the analysis device.
The analyzer responds to the trigger signal to judge whether the latest day file exists, and can detect according to a T-1 mode, for example, a task is triggered at 12 months and 28 months in 2021, and a file with a card core suffix of 12 months and 27 days in 2021 can be detected.
Optionally, the parsing device responds to the trigger signal, reads the latest dictionary file from the card core dictionary, and obtains file name information, update frequency information, increment rule information, primary key information, field information, and the like.
And thirdly, analyzing the content of the dictionary file into a memory by using xlrd.
And after the latest dictionary is read, a character analysis module in the analysis device analyzes the dictionary data based on xlrd, and distributes the table analysis result to the memory.
Optionally, an xlrd method is called to obtain the position of the corresponding field in the table according to a predetermined analytic logic, such as information of a file name, an update frequency, a primary key, a field separation rule, a field name, a field type, a field length, a field remark and the like, then the information is stored in a memory by using a global shared variable, a data structure is a set of paired structures (map structure), and other program codes directly read the shared variable to obtain the required information.
And fourthly, the task distribution module of the analysis device starts a downstream controller, and the downstream controller mainly comprises a creation task and an update task.
Optionally, the task distribution module starts a downstream controller, and the controller configures creation-class task control and update-class task control, where the creation-class task and the update-class task are started simultaneously, a full creation-class task is generated each time, and the update-class task generates a specific update file only when the last version dictionary file exists and a contrast difference is checked.
Optionally, the task distribution module configures a creation class task controller and an update class task controller.
And fifthly, the creating task acquires table data from the memory according to pre-developed configuration file programs required by different downstream systems to generate corresponding files and puts the files to a specified target.
Optionally, according to a downstream application scenario, the task controller selects a required configuration file program, where the configuration creation class task control may include a mysql configuration generation program of the first relational database management system, an Oracle configuration generation program of the second relational database management system, and a data integration program configuration generation program.
Optionally, the generation table may be configured to create the sql statement based on the mysql of the first relational database management system; the sql statement may be created based on an Oracle configuration generator table of the second relational database management system; the configuration file may be generated based on a data integration program configuration generator.
Optionally, the actual table building statements and field types of mysql and Oracle are different, different creation/update statements need to be generated, and an adapter can be added to extend into a data warehouse tool (Hive), a distributed database (Doris) and other databases.
And sixthly, the update task acquires table data from the memory according to pre-developed configuration file programs required by different downstream systems to generate corresponding files and places the files into an appointed directory.
Optionally, according to a downstream application scenario, the task controller selects a required configuration file program, and the configuration update class task controller may include a mysql configuration generation program of the first relational database management system, an Oracle configuration generation program of the second relational database management system, a data integration program configuration generation program, and a change comparison record generation program.
Optionally, the table creation sql statement may be generated based on a mysql configuration generator of the first relational database management system; the sql statement may be created based on an Oracle configuration generator table of the second relational database management system; the generating program can be configured based on the data integration program to generate a configuration file; the change log file may be generated based on a change comparison log generating program.
And seventhly, automatically loading the corresponding file for changing after the downstream service system detects that the update type latest configuration file is generated.
Optionally, the data dictionary of the previous version is changed, the data dictionary has time sequence numbers on file names, and the data dictionary files of the previous version smaller than the current version can be obtained by sorting according to the data dictionary names.
Optionally, after the update file is generated, the downstream system performs sorting according to the data dictionary name to obtain a data dictionary file of a previous version smaller than the current version, and changes the data dictionary file of the previous version.
The method and the device for automatically analyzing the Excel file of the core system data dictionary of the bank card based on the Python xlrd toolkit can analyze corresponding information in the data dictionary to generate configuration files required by different downstream target systems, such as table building sql of different databases, table structure changing sql and configuration files required by a data integration system, and can achieve the technical effects of improving the development efficiency and the processing efficiency of data by solving the technical problems that the dictionary is changed to the full process automation of data integration task adaptation through solving the original manual copying and comparing and copying field mode, and the process cannot be automated.
The embodiment of the disclosure also provides a data processing device for executing the data processing method of the embodiment shown in fig. 1.
Fig. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure. As shown in fig. 3, the data processing apparatus 30 may include: a first acquisition unit 31, a second acquisition unit 32 and a generation unit 33.
The first obtaining unit 31 is configured to obtain a data dictionary, where the data dictionary is used to represent information of a bank card to be analyzed.
And a second obtaining unit 32, configured to obtain target category information of the bank card from the data dictionary, where the target category information corresponds to a downstream system, and the downstream system is a system used in an application scenario of the bank card.
A generating unit 33 for generating an object file analyzed by a downstream system based on the object class data
Optionally, the second obtaining unit 32 includes: the first reading module is used for reading the target category information from the data dictionary based on the target toolkit.
Optionally, the first reading module comprises: the first acquisition submodule is used for reading the plurality of types of information from the data dictionary to the memory based on the target toolkit; and acquiring the target category information in the plurality of category information from the memory by using the controller corresponding to the target category information.
Optionally, the generating unit 33 includes: and the first updating module is used for updating the original file to be analyzed by the downstream system based on the updating task data to obtain the target file.
Optionally, the first updating module includes: the first comparison submodule is used for comparing the target category information with the original category information of the original data dictionary to obtain a comparison result, wherein the original data dictionary is used for representing the original information of the bank card to be analyzed; and in response to the comparison result representing that the target category information is different from the original category information, generating update-type task data based on the comparison result.
Optionally, the generating unit 33 includes: the second updating module is used for issuing an updating message to the downstream system, wherein the updating message is used for indicating that the original file is updated into a target file; and receiving a request sent by the downstream system based on the update message, responding to the request, and sending the target file to the downstream system.
Optionally, the generating unit 33 includes: a first creation module to create a target file for analysis by a downstream system based on the create-class task data.
Optionally, the first obtaining unit 31 includes: and the first reading unit is used for reading the data dictionary with the timestamp closest to the current time from the first storage position according to the target time period.
Optionally, the apparatus further comprises: and the storage unit is used for storing the target file to the second storage position, wherein the target file is acquired from the second storage position by a downstream system.
In the device of the disclosed embodiment, a data dictionary is acquired through a first acquisition unit, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target category information of the bank card from the data dictionary through a second acquisition unit, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; through the generation unit, the target file analyzed by the downstream system is generated based on the target category data, so that the efficiency of processing the data of the bank card is improved, and the technical problem of low efficiency of processing the data of the bank card is solved.
In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, and do not violate the good customs of the public order.
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
Embodiments of the present disclosure provide an electronic device, which may include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform the data processing method of the embodiments of the present disclosure.
Optionally, the electronic device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
According to an embodiment of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the data processing method of the embodiment of the present disclosure.
Alternatively, in the present embodiment, the above-mentioned nonvolatile storage medium may be configured to store a computer program for executing the steps of:
s1, acquiring a data dictionary, wherein the data dictionary is used for representing information of the bank card to be analyzed;
s2, acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
s3, a target file to be analyzed by the downstream system is generated based on the target category data.
Alternatively, in the present embodiment, the non-transitory computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
According to an embodiment of the present disclosure, the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, realizes the steps of:
s1, acquiring a data dictionary, wherein the data dictionary is used for representing information of the bank card to be analyzed;
s2, acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
s3, a target file to be analyzed by the downstream system is generated based on the target category data.
Fig. 4 is a block diagram of an electronic device of a data processing method according to an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 4, the apparatus 400 includes a computing unit 401 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)402 or a computer program loaded from a storage unit 408 into a Random Access Memory (RAM) 403. In the RAM403, various programs and data required for the operation of the device 400 can also be stored. The calculation unit 401, the ROM402, and the RAM403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
A number of components in device 400 are connected to I/O interface 405, including: an input unit 406 such as a keyboard, a mouse, or the like; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408 such as a magnetic disk, optical disk, or the like; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (13)
1. A method of data processing, comprising:
acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed;
acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
generating a target file for analysis by the downstream system based on the target category data.
2. The method of claim 1, wherein obtaining the object class information of the bank card from the data dictionary comprises:
and reading the target category information from the data dictionary based on a target tool kit.
3. The method of claim 2, wherein reading the target category information from the data dictionary based on a target toolkit comprises:
reading a plurality of types of information from the data dictionary to a memory based on the target toolkit;
and acquiring the target category information in the plurality of category information from the memory by using a controller corresponding to the target category information.
4. The method of claim 1, wherein the target category information comprises update-class task data, generating a target file for analysis by the downstream system based on the target category data comprises:
and updating the original file to be analyzed of the downstream system based on the updated task data to obtain the target file.
5. The method of claim 4, further comprising:
comparing the target category information with original category information of an original data dictionary to obtain a comparison result, wherein the original data dictionary is used for representing original information of the bank card to be analyzed;
in response to the comparison result representing a difference between the target category information and the original category information, generating the update-class task data based on the comparison result.
6. The method of claim 4, further comprising:
issuing an update message to the downstream system, wherein the update message is used for indicating that the original file is updated to the target file;
and receiving a request sent by the downstream system based on the update message, responding to the request, and sending the target file to the downstream system.
7. The method of claim 1, wherein the target category information comprises create-class task data, generating a target file for analysis by the downstream system based on the target category data comprises:
creating the target file analyzed by the downstream system based on the create class task data.
8. The method of any of claims 1 to 7, obtaining the data dictionary comprising:
and reading the data dictionary with the timestamp closest to the current time from the first storage position according to the target time period.
9. The method of any of claims 1 to 7, further comprising:
storing the target file to a second storage location, wherein the target file is retrieved from the second storage location by the downstream system.
10. A data processing apparatus comprising:
the system comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring a data dictionary, and the data dictionary is used for representing information of a bank card to be analyzed;
the second acquisition unit is used for acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
a generating unit for generating a target file analyzed by the downstream system based on the target category data.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.
12. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-9.
13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210190295.8A CN114936915A (en) | 2022-02-28 | 2022-02-28 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210190295.8A CN114936915A (en) | 2022-02-28 | 2022-02-28 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114936915A true CN114936915A (en) | 2022-08-23 |
Family
ID=82861820
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210190295.8A Pending CN114936915A (en) | 2022-02-28 | 2022-02-28 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114936915A (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070061245A1 (en) * | 2005-09-14 | 2007-03-15 | Jorey Ramer | Location based presentation of mobile content |
| EP2341480A1 (en) * | 1999-10-29 | 2011-07-06 | Computer Sciences Corporation | Business transaction processing systems and methods |
| CN102932419A (en) * | 2012-09-25 | 2013-02-13 | 浙江图讯科技有限公司 | Data storage system for industrial and mining enterprise oriented safety production cloud service platform |
| CN109543690A (en) * | 2018-11-27 | 2019-03-29 | 北京百度网讯科技有限公司 | Method and apparatus for extracting information |
| CN110704462A (en) * | 2019-09-06 | 2020-01-17 | 中国平安财产保险股份有限公司 | Data change notification method, electronic device, computer device, and storage medium |
| US20200098053A1 (en) * | 2018-09-26 | 2020-03-26 | Intuit Inc. | Method and system for user data driven financial transaction description dictionary construction |
| CN112069773A (en) * | 2020-07-23 | 2020-12-11 | 北京三快在线科技有限公司 | Data processing system, method, apparatus, electronic device, and computer-readable medium |
| CN112862604A (en) * | 2021-04-25 | 2021-05-28 | 腾讯科技（深圳）有限公司 | Card issuing organization information processing method, device, equipment and storage medium |
| CN112925914A (en) * | 2021-03-31 | 2021-06-08 | 携程旅游网络技术（上海）有限公司 | Data security classification method, system, device and storage medium |
| CN113269551A (en) * | 2021-05-25 | 2021-08-17 | 北京金山云网络技术有限公司 | Medical payment method, device and system and electronic equipment |
| CN113902415A (en) * | 2021-10-26 | 2022-01-07 | 中国工商银行股份有限公司 | Financial data checking method and device, computer equipment and storage medium |
-
2022
- 2022-02-28 CN CN202210190295.8A patent/CN114936915A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2341480A1 (en) * | 1999-10-29 | 2011-07-06 | Computer Sciences Corporation | Business transaction processing systems and methods |
| US20070061245A1 (en) * | 2005-09-14 | 2007-03-15 | Jorey Ramer | Location based presentation of mobile content |
| CN102932419A (en) * | 2012-09-25 | 2013-02-13 | 浙江图讯科技有限公司 | Data storage system for industrial and mining enterprise oriented safety production cloud service platform |
| US20200098053A1 (en) * | 2018-09-26 | 2020-03-26 | Intuit Inc. | Method and system for user data driven financial transaction description dictionary construction |
| CN109543690A (en) * | 2018-11-27 | 2019-03-29 | 北京百度网讯科技有限公司 | Method and apparatus for extracting information |
| CN110704462A (en) * | 2019-09-06 | 2020-01-17 | 中国平安财产保险股份有限公司 | Data change notification method, electronic device, computer device, and storage medium |
| CN112069773A (en) * | 2020-07-23 | 2020-12-11 | 北京三快在线科技有限公司 | Data processing system, method, apparatus, electronic device, and computer-readable medium |
| CN112925914A (en) * | 2021-03-31 | 2021-06-08 | 携程旅游网络技术（上海）有限公司 | Data security classification method, system, device and storage medium |
| CN112862604A (en) * | 2021-04-25 | 2021-05-28 | 腾讯科技（深圳）有限公司 | Card issuing organization information processing method, device, equipment and storage medium |
| CN113269551A (en) * | 2021-05-25 | 2021-08-17 | 北京金山云网络技术有限公司 | Medical payment method, device and system and electronic equipment |
| CN113902415A (en) * | 2021-10-26 | 2022-01-07 | 中国工商银行股份有限公司 | Financial data checking method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109034988B (en) | Accounting entry generation method and device | |
| CN111382279A (en) | Order examination method and device | |
| CN113205402A (en) | Account checking method and device, electronic equipment and computer readable medium | |
| CN113485781A (en) | Report generation method and device, electronic equipment and computer readable medium | |
| CN111427971A (en) | Business modeling method, device, system and medium for computer system | |
| CN115640300A (en) | Big data management method, system, electronic equipment and storage medium | |
| CN110895761A (en) | Method and device for processing after-sale service application information | |
| CN114936915A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN115525721A (en) | Data synchronization method, device, equipment and storage medium | |
| CN115391343A (en) | Bill data processing method and device, electronic equipment and storage medium | |
| CN112380321A (en) | Primary and secondary database distribution method based on bill knowledge graph and related equipment | |
| CN114661918A (en) | Knowledge graph construction method and device, storage medium and electronic equipment | |
| CN114860872A (en) | Data processing method, device, equipment and storage medium | |
| CN114443802A (en) | Interface document processing method and device, electronic equipment and storage medium | |
| CN114139798A (en) | Enterprise risk prediction method and device and electronic equipment | |
| CN115082179A (en) | Data processing method, device, equipment and storage medium | |
| CN113592470A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN113742322A (en) | Data quality detection method and device | |
| CN113220573A (en) | Test method and device for micro-service architecture and electronic equipment | |
| CN113961518B (en) | Log visual display method and device, electronic equipment and storage medium | |
| CN113778501B (en) | Code task processing method and device | |
| CN114565402A (en) | Information recommendation method and device and electronic equipment | |
| CN115879434A (en) | Template generation method, device, equipment and storage medium for information disclosure report | |
| CN115526403A (en) | Financial data prediction method, system, equipment, storage medium and product | |
| CN116151981A (en) | Transaction processing request processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |