CN117389978A - Automatic importing method and device for interface file, computer equipment and storage medium - Google Patents
Automatic importing method and device for interface file, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117389978A CN117389978A CN202311550323.3A CN202311550323A CN117389978A CN 117389978 A CN117389978 A CN 117389978A CN 202311550323 A CN202311550323 A CN 202311550323A CN 117389978 A CN117389978 A CN 117389978A
- Authority
- CN
- China
- Prior art keywords
- file
- configuration
- target interface
- interface file
- inspection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000007689 inspection Methods 0.000 claims abstract description 47
- 238000012545 processing Methods 0.000 claims abstract description 39
- 238000004590 computer program Methods 0.000 claims description 23
- 238000013515 script Methods 0.000 claims description 15
- 238000001514 detection method Methods 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 12
- 238000013524 data verification Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000008676 import Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 4
- 238000012795 verification Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000001172 regenerating effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
- G06F9/45508—Runtime interpretation or emulation, e g. emulator loops, bytecode interpretation
- G06F9/45512—Command shells
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application relates to an automatic importing method and device of interface files, computer equipment and a storage medium, and relates to the technical field of computers. Can be used in the field of financial science and technology or other related fields. The method comprises the following steps: responding to task scheduling, and performing file inspection on at least one target interface file to obtain an inspection result of the target interface file; under the condition that the checking result represents that the checking of each target interface file is successful, each target interface file is moved to a processing directory; and determining the configuration file of the target interface file in the processing directory, and loading the configuration file into a formal table. By adopting the method, the data importing efficiency can be improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to an automatic interface file importing method and apparatus, a computer device, and a storage medium.
Background
As various types of data are increasingly increased, the scene of data resource transmission sharing is more and more complex, and the situation of data change is more and more. There is an increasing need for a data import method tool that can flexibly configure support for dynamic changes and can be deployed on a hot basis.
In the conventional technology, a data provider and a data receiver define interface data rules, the data provider and the data receiver code and develop according to the established rules respectively, and if the data rules change, the data provider and the data receiver code and support redeployment respectively. However, the data provider and the data receiver have more workload of manually developing codes, and are easy to make mistakes when the rules of the data fields are complex; the data rule changes, the data provider and the data receiver need to carry out coding adaptation again and deployment support, and the response link time is long.
As can be seen, the conventional method for manually developing, coding and loading the received data has lower efficiency.
Disclosure of Invention
In view of the foregoing, it is desirable to provide an automatic import method, apparatus, computer device, and storage medium for an interface file, which can improve data loading efficiency.
In a first aspect, the present application provides a method for automatically importing an interface file, where the method includes:
responding to task scheduling, and performing file inspection on at least one target interface file to obtain an inspection result of the target interface file;
under the condition that the checking result represents that the checking of each target interface file is successful, each target interface file is moved to a processing directory;
and determining the configuration file of the target interface file in the processing directory, and loading the configuration file into a formal table.
In one embodiment, the method further comprises:
generating a corresponding external document and a file loading configuration script based on the internal document, wherein the file loading configuration script comprises file configuration and file field configuration of each target interface file;
storing the file configuration to a newly added file configuration table, and storing the file field configuration to the newly added file field configuration table.
In one embodiment, the responding to the task scheduling performs file inspection on at least one target interface file to obtain an inspection result of the target interface file, and the method includes:
and responding to the task scheduling defined by the real-time file loading task, and performing file inspection on the target interface file in real time to obtain an inspection result of the target interface file.
In one embodiment, the responding to the task scheduling performs file inspection on at least one target interface file to obtain an inspection result of the target interface file, and the method includes:
and responding to task scheduling defined by batch file loading tasks, and performing file inspection on a preset number of target interface files to obtain inspection results of the target interface files.
In one embodiment, the file inspection of at least one target interface file to obtain an inspection result of the target interface file includes:
reading the corresponding file configuration from the file configuration table according to the batch number of the target interface file;
acquiring the file name and the file directory of the target interface file from the file configuration;
and checking whether the target interface file and the corresponding data verification file exist under the file directory according to the file name to obtain a checking result of the target interface file.
In one embodiment, the determining the configuration file of the target interface file in the processing directory includes:
judging whether a configuration file corresponding to the target interface file exists or not according to the file number of the target interface file in the processing directory, and obtaining a judging result;
acquiring the file field configuration corresponding to the target interface file from the file field configuration table under the condition that the judging result indicates that the configuration file does not exist, and generating the configuration file according to the file field configuration corresponding to the target interface file;
or under the condition that the judging result represents that the configuration file exists, detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detecting result.
In one embodiment, the detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detection result includes:
and under the condition that a configuration update mark exists in the file configuration corresponding to the target interface file, obtaining a detection result representing that the configuration file needs to be updated, and updating the configuration file according to the file field configuration.
In one embodiment, the method further comprises:
and synchronously moving each target interface file to a backup directory while moving each target interface file to a processing directory.
In a second aspect, the present application provides an automatic importing apparatus for an interface file, where the apparatus includes:
the file checking module is used for responding to task scheduling, and performing file checking on at least one target interface file to obtain a checking result of the target interface file;
the file moving module is used for moving each target interface file to a processing catalog under the condition that the checking result represents that each target interface file is successfully checked;
and the file loading module is used for determining the configuration file of the target interface file in the processing directory and loading the configuration file into a formal table.
In one embodiment, the automatic importing device of the interface file further comprises a file conversion module, which is used for generating a corresponding external file and a file loading configuration script based on the internal file, wherein the file loading configuration script comprises file configuration and file field configuration of each target interface file; storing the file configuration to a newly added file configuration table, and storing the file field configuration to the newly added file field configuration table.
In one embodiment, the file checking module is further configured to perform file checking on the target interface file in real time in response to a task schedule defined by a real-time file loading task, so as to obtain a checking result of the target interface file.
In one embodiment, the file checking module is further configured to perform file checking on a preset number of target interface files in response to task scheduling defined by a batch file loading task, so as to obtain a checking result of the target interface files.
In one embodiment, the file checking module is further configured to read a corresponding file configuration from the file configuration table according to the batch number of the target interface file; acquiring the file name and the file directory of the target interface file from the file configuration; and checking whether the target interface file and the corresponding data verification file exist under the file directory according to the file name to obtain a checking result of the target interface file.
In one embodiment, the file loading module is further configured to determine whether a configuration file corresponding to the target interface file exists according to a file number of the target interface file in the processing directory, so as to obtain a determination result; acquiring the file field configuration corresponding to the target interface file from the file field configuration table under the condition that the judging result indicates that the configuration file does not exist, and generating the configuration file according to the file field configuration corresponding to the target interface file; or under the condition that the judging result represents that the configuration file exists, detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detecting result.
In one embodiment, the file loading module is further configured to obtain a detection result indicating that the configuration file needs to be updated when a configuration update flag exists in a file configuration corresponding to the target interface file, and update the configuration file according to the file field configuration.
In one embodiment, the automatic import device of interface files further includes a backup module for synchronously moving each of the target interface files to a backup directory while moving each of the target interface files to a processing directory.
In a third aspect, the present application further provides a computer device, where the computer device includes a memory and a processor, where the memory stores a computer program, and where the processor implements the steps of the method embodiments described above when the processor executes the computer program.
In a fourth aspect, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
The method, the device, the computer equipment, the storage medium and the computer program product for automatically importing the interface files are used for responding to task scheduling and performing file inspection on at least one target interface file to obtain an inspection result of the target interface file; under the condition that the checking result represents that the checking of each target interface file is successful, each target interface file is moved to a processing directory; and determining the configuration file of the target interface file in the processing directory, and loading the configuration file into a formal table. According to the interface file automatic importing method, device, computer equipment, storage medium and computer program product, the target interface file which is checked to pass through is moved to the processing catalog, the configuration file of the target interface file in the processing catalog is automatically determined, the corresponding target interface file is loaded into the formal table according to the configuration file, the interface file is automatically imported, the traditional manual development coding data importing method is replaced, and the data importing efficiency is greatly improved.
Drawings
FIG. 1 is a flow chart of an automatic import method of interface files in one embodiment;
FIG. 2 is a flow chart of an automatic import method of interface files in one embodiment;
FIG. 3 is a flow chart of step 102 in one embodiment;
FIG. 4 is a flow chart of step 102 in one embodiment;
FIG. 5 is a flow chart of step 102 in one embodiment;
FIG. 6 is a flow chart of step 106 in one embodiment;
FIG. 7 is a flow chart of step 606 in one embodiment;
FIG. 8 is a flow chart of an automatic import method of interface files in one embodiment;
FIG. 9 is a flow diagram of the generation of a file loading configuration script in one embodiment;
FIG. 10 is a flow diagram of file inspection in one embodiment;
FIG. 11 is a flow diagram of file loading in one embodiment;
FIG. 12 is a block diagram of an interface file automatic import device in one embodiment;
fig. 13 is an internal structural view of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
In one embodiment, as shown in fig. 1, an automatic importing method of an interface file is provided, and this embodiment is illustrated by applying the method to a server, it may be understood that the method may also be applied to a terminal, and may also be applied to a system including the terminal and the server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the steps of:
and 102, responding to task scheduling, and performing file inspection on at least one target interface file to obtain an inspection result of the target interface file.
The automatic importing of the interface data/files refers to automatically importing the data into a data table by flexibly configuring a mode without manual coding according to definition rules of the interface data files. The target interface file is file data to be loaded/imported by the data receiver, and the file data is file data which is sent to the data receiver by the data provider through the target interface and interface data rules. The task scheduling can realize the definition of two types of tasks, namely a real-time file loading task and a batch file loading task, and can be executed in a task scheduler.
In particular, in response to a task definition specified in the task schedule, a file check may be performed on at least one target interface file under the task definition. And the file checking is to check whether the target interface file under the designated directory and the corresponding data checking file exist or not according to the file configuration by reading the file configuration of the target interface file, so as to obtain a checking result.
And 104, moving each target interface file to the processing directory under the condition that the checking result represents that each target interface file is successfully checked.
And if the target interface file under the specified directory and the corresponding data verification file exist, obtaining a check result representing the verification success of each target interface file. At this time, each target interface file is moved to the processing directory and then loaded. The process directory may also be referred to as a load directory. Or if the target interface file or/and the corresponding data verification file under the appointed directory does not exist, obtaining a check result representing that verification of each target interface file fails. At this time, the processing is not continued.
And 106, determining a configuration file of the target interface file in the processing directory, and loading the configuration file into the formal table.
The configuration file of the target interface file may be a ctl configuration file corresponding to sqllrd. Sqllrd is a data loading tool, and ctll configuration file is a control file used to import and export data. It is a language similar to SQL (structured query language ) that can be used to describe the structure, location, and how data is imported or exported.
Specifically, after determining the configuration file of the target interface file in the processing directory, the sqllrd command may be used to load the target interface file into the temporary table, and then load the target interface file from the temporary table into the formal table according to the configuration file. The temporary table is in one-to-one relation with the target interface file and is used for temporarily storing the target interface file, and the temporary table is added with operation type fields (1-new, 2-modified and 3-deleted) relative to the formal table by default. The formal table is a table to which the target interface file is to be imported, and the target interface file may be imported into the formal table in increments or in full. The target interface files are imported into the same formal table, so that the temporary table data can be loaded into the formal table through a public module according to the operation type and the file main key field in the temporary table and the corresponding new/modified/deleted strategy through hot updating. The common module means that the mode of each interface file process is common. The hot deployment/hot update refers to upgrading software when the application is running, but does not need to restart the application, and has the characteristics of light weight, high speed, high efficiency and the like.
According to the interface file automatic importing method, the target interface file which is checked to pass through is moved to the processing catalog, the configuration file of the target interface file in the processing catalog is automatically determined, the corresponding target interface file is loaded into the formal table according to the configuration file, the interface file automatic importing is completed, the traditional manual development coding data importing method is replaced, and the data importing efficiency is greatly improved.
In one embodiment, as shown in fig. 2, the automatic importing method of the interface file further includes:
step 202, generating a corresponding external document and a file loading configuration script based on the internal document, wherein the file loading configuration script comprises file configuration and file field configuration of each target interface file.
The interface between the data provider and the data receiver is used for transmitting the target interface file. The interface has corresponding interface documents, which can be divided into an internal document and an external document. The interface document may include data definitions of interface names, URL (Uniform Resource Locator, uniform resource location) paths, call modes, parameter formats, response formats, and the like. The internal document may include a full amount of interface information including a file name, a lot number, a file receiving path, a file loading target table name, field information, a file record optional operation type, a file number, and the like. The external document is used when provided to the external application, and only contains information that the upstream external application needs to see, such as file name, file receiving path, field information, and the like.
Specifically, a Java tool may be written, external documents generated from internal documents, and file loaded DML (Data Manipulation Language, data operating language) configuration scripts, also referred to as file loaded configuration scripts. The file loading configuration script generates corresponding file loading configuration information according to the internal interface definition, and specifically may include a file configuration file_load_config (including a file number, a file name, a batch number, a file receiving path, a file record operable type, a temporary table name, a formal table name, and an sqllrd configuration update flag), and a file field configuration file_detail_load_config (including a file number and specific field information).
Step 204, storing the file configuration to the newly added file configuration table, and storing the file field configuration to the newly added file field configuration table.
Specifically, both the file configuration and the file field configuration require the addition of a corresponding configuration table for storage. The file configuration corresponding to each target interface file can be stored in the newly added file configuration table, and the file field configuration corresponding to each target interface file can be stored in the newly added file field configuration table.
In the embodiment of the application, the interface document generates the file configuration information through tool conversion, so that the interface definition and the file configuration information are consistent and managed, manual coding is reduced, and the importing efficiency is improved.
In one embodiment, as shown in fig. 3, in step 102, in response to task scheduling, performing file inspection on at least one target interface file to obtain an inspection result of the target interface file may include:
and step 302, responding to task scheduling defined by a real-time file loading task, and performing file inspection on the target interface file in real time to obtain an inspection result of the target interface file.
The task scheduling may be performed in advance in the task scheduler, and the real-time file loading task definition may be specified, so that the distributed task scheduling executor (XXL JOB executor) performs file inspection on the received target interface file in real time based on the real-time file loading task definition, to obtain an inspection result of the target interface file.
In the embodiment of the application, the importing mode of the target interface file is changed through task scheduling, so that the flexibility of data importing is improved.
In one embodiment, as shown in fig. 4, in step 102, in response to task scheduling, performing file inspection on at least one target interface file to obtain an inspection result of the target interface file may include:
step 402, in response to task scheduling defined by batch file loading tasks, file inspection is performed on a preset number of target interface files, and inspection results of the target interface files are obtained.
The task scheduling may be performed in advance in the task scheduler, and a batch file loading task definition may be specified, so that a distributed task scheduling executor (XXL JOB executor) performs file inspection on a preset number of target interface files that have been received based on the batch file loading task definition, to obtain an inspection result of the target interface files. For example, the batch file loading task is defined as performing batch import on 20 files, and then performing file inspection on every 20 target interface files to obtain inspection results of the target interface files.
In the embodiment of the application, the importing mode of the target interface file is changed through task scheduling, so that the flexibility of data importing is improved.
In one embodiment, as shown in fig. 5, in step 102, performing file inspection on at least one target interface file to obtain an inspection result of the target interface file may include:
step 502, according to the batch number of the target interface file, reading the corresponding file configuration from the file configuration table.
Specifically, according to the batch number of the target interface file, the file configuration file_load_config corresponding to the batch number may be read from the file configuration table.
Step 504, obtain the file name and file directory of the target interface file from the file configuration.
Specifically, the file name and the file directory of the target interface file may be acquired from the file configuration file_load_config.
Step 506, checking whether the target interface file and the corresponding data verification file exist in the file directory according to the file name, and obtaining the checking result of the target interface file.
Specifically, in the file directory corresponding to the target interface file, it may be checked whether there are a file whose file name corresponds to the target interface file and a data verification file. And if the file directory contains the file with the same file name as the target interface file and the data verification file, obtaining a check result representing that the target interface file is successfully verified. Or if the file directory does not contain the file with the same file name as the target interface file and the data verification file, obtaining a check result representing the failure of verification of the target interface file.
In the embodiment of the disclosure, the target interface file is checked based on the file configuration, so that the accuracy of importing the interface file is improved.
In one embodiment, as shown in fig. 6, determining the configuration file of the target interface file in the process catalog in step 106 may include:
step 602, judging whether a configuration file corresponding to the target interface file exists according to the file number of the target interface file in the processing directory, and obtaining a judging result.
Specifically, according to the file number of the target interface file in the processing directory, searching the configuration file corresponding to the file number, and judging whether the configuration file corresponding to the file number exists or not to obtain a corresponding judging result.
Step 604, under the condition that the judging result indicates that the configuration file does not exist, acquiring the file field configuration corresponding to the target interface file from the file field configuration table, and generating the configuration file according to the file field configuration corresponding to the target interface file.
Specifically, if the configuration file corresponding to the file number cannot be found, a judging result indicating that the configuration file of the target interface file does not exist is obtained. At this time, the file field configuration corresponding to the target interface file can be obtained from the file field configuration table, and a new configuration file is directly generated according to the file field configuration.
Step 606, or in the case that the judging result represents that the configuration file exists, detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detecting result.
Specifically, if the configuration file corresponding to the file number is found, a judging result representing the existence of the configuration file of the target interface file is obtained. At this time, the file configuration corresponding to the target interface file may be obtained from the file configuration table, whether the configuration file needs to be updated is detected according to the file configuration, and the configuration file is determined according to the detection result.
In the embodiment of the disclosure, the related configuration files needed during the importing and loading of the interface data are automatically generated by the component according to the file configuration information, so that manual coding is reduced, and the file importing efficiency is improved.
In one embodiment, as shown in fig. 7, in step 606, detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detection result may include:
step 702, obtaining a detection result that characterizes the configuration file to be updated under the condition that a configuration update mark exists in the file configuration corresponding to the target interface file, and updating the configuration file according to the file field configuration.
Specifically, the file configuration corresponding to the target interface file can be obtained from the file configuration table, and whether the configuration file needs to be updated is detected according to the sqllrd configuration update flag in the file configuration. If the data rule of the interface changes, for example, the length of the data field changes, the data field increases, etc., then the file configuration generated by the interface document of the interface will have the sqllrd configuration update flag, and the detection result that characterizes the configuration file needs to be updated is obtained at this time. And regenerating a configuration file according to the detection result, realizing the hot update of file loading configuration, and generating a synchronous update field state after completion.
Or if the data rule of the interface is not changed, the file configuration generated by the interface document of the interface will not have the sqllrd configuration update mark, and the detection result that the characterization configuration file does not need to be updated is obtained. The target interface file can be directly imported into the temporary table according to the original configuration file, and then imported into the formal table.
In the embodiment of the disclosure, the related configuration files needed during the importing and loading of the interface data are automatically generated by the component according to the file configuration information, so that manual coding is reduced, and the file importing efficiency is improved.
In one embodiment, the method for automatically importing interface files further comprises the following steps:
and synchronously moving each target interface file to the backup directory while moving each target interface file to the processing directory.
Specifically, when processing according to the existing standard file, the target receiving file is moved from the receiving directory to the loading directory and then loaded, and after loading is completed, the target receiving file is synchronously required to be moved to the backup directory. The backup catalog file is cleaned and reused for the existing batch operation, and the interface scheme is not involved.
To facilitate a further understanding of this application, a most complete embodiment is provided herein. According to each data definition in the data interface document, the method supports flexibly configurable data import, and meanwhile, if the data interface changes, the data definition is updated in time to conduct new data import in a mode of supporting hot deployment. The application provides an automatic importing method of interface files, which realizes a configurable general file loading component based on sqllrd, wherein the general file loading component can be integrated into a distributed batch scheduling framework (XXL JOB executor), and the overall flow is shown in FIG. 8. 1. Task scheduler: and the task scheduling realizes the definition of two types of tasks, namely a real-time file loading task and a batch file loading task. 2. Interface definition and file configuration information: the interface document is divided into an internal document and an external document. A Java tool is written, an external document and a DML configuration script for file loading are generated from an internal document, and a specific flow is shown in fig. 9. 3. General file loading component: 1) And (3) checking whether the data file and the data verification file exist under the designated directory according to the configuration information after the file configuration information is read, wherein the specific flow is shown in fig. 10. 2) File loading class, file movement: when processing according to the existing standard file, the file is moved from the receiving directory to the loading directory and then loaded, and after loading is completed, the file is synchronously required to be moved to the backup directory. The backup catalog file is cleaned and reused for the existing batch operation, and the interface scheme is not involved. Sqllrd configuration check: judging whether a ctl configuration file corresponding to the sqllrd exists or not according to the file number, if not, directly generating, if so, judging whether updating is needed according to an sqllrd configuration update mark in the file configuration file_load_config, and if so, regenerating to realize the hot updating of the file loading configuration. The field state is synchronously updated after the generation is completed. The Sqllrd file is loaded to a temporary table: the sqllrd command is used to load data into the temporary table, which adds the operation type field (1-add, 2-modify, 3-delete) to the formal table by default. The temporary table is loaded to the formal table: and through the public module, according to the file configuration information, the operation type and the file primary key field in the temporary table can be updated thermally according to the corresponding new adding/modifying/deleting strategy, and the temporary table data is loaded into the formal table. A specific flow of file loading is shown in fig. 11.
According to the method and the device, the interface document is converted through the tool to generate the document configuration information program, so that the document and the program are managed consistently, and manual coding is reduced. And according to the file configuration information, the components automatically generate relevant configuration files needed when the interface data is imported and loaded, so that manual coding is reduced. When the file is imported from the temporary table to the formal table, the operations of adding, deleting and changing the stock data according to the main key are supported technically, the specific operation range can be limited by each interface document, and the method can realize flexible configuration and support the automatic import of the hot deployment interface.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides an automatic interface file importing device for implementing the above-mentioned automatic interface file importing method. The implementation scheme of the device for solving the problem is similar to that described in the above method, so the specific limitation in the embodiment of the device for automatically importing an interface file provided below may be referred to the limitation of the method for automatically importing an interface file in the above description, which is not repeated here.
In one embodiment, as shown in fig. 12, there is provided an automatic import apparatus 1200 of an interface file, including:
a file checking module 1202, configured to perform file checking on at least one target interface file in response to task scheduling, to obtain a checking result of the target interface file;
the file moving module 1204 is configured to move each target interface file to the processing directory if the checking result indicates that the checking of each target interface file is successful;
the file loading module 1206 is configured to determine a configuration file of the target interface file in the processing directory, and load the configuration file into the formal table.
In one embodiment, the automatic importing device of the interface file further comprises a file conversion module, wherein the file conversion module is used for generating a corresponding external file and a file loading configuration script based on the internal file, and the file loading configuration script comprises file configuration and file field configuration of each target interface file; storing the file configuration to a newly added file configuration table, and storing the file field configuration to the newly added file field configuration table.
In one embodiment, the file checking module is further configured to respond to a task schedule defined by a real-time file loading task, and perform file checking on the target interface file in real time to obtain a checking result of the target interface file.
In one embodiment, the file checking module is further configured to perform file checking on a preset number of target interface files in response to task scheduling defined by a batch file loading task, so as to obtain a checking result of the target interface files.
In one embodiment, the file checking module is further configured to read a corresponding file configuration from the file configuration table according to the batch number of the target interface file; acquiring a file name and a file directory of a target interface file from file configuration; and checking whether the target interface file and the corresponding data verification file exist in the file directory according to the file name to obtain a checking result of the target interface file.
In one embodiment, the file loading module is further configured to determine whether a configuration file corresponding to the target interface file exists according to a file number of the target interface file in the processing directory, so as to obtain a determination result; under the condition that the judging result indicates that the configuration file does not exist, acquiring file field configuration corresponding to the target interface file from a file field configuration table, and generating a configuration file according to the file field configuration corresponding to the target interface file; or under the condition that the judging result represents that the configuration file exists, detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detecting result.
In one embodiment, the file loading module is further configured to obtain a detection result indicating that the configuration file needs to be updated when a configuration update flag exists in the file configuration corresponding to the target interface file, and update the configuration file according to the file field configuration.
In one embodiment, the automatic import device of interface files further comprises a backup module for synchronously moving each target interface file to the backup directory while moving each target interface file to the processing directory.
The above-mentioned modules of the interface file automatic import device may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 13. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing interface data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method for automatically importing an interface file.
It will be appreciated by those skilled in the art that the structure shown in fig. 13 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the computer device to which the present application applies, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, the present application further provides a computer device, where the computer device includes a memory and a processor, where the memory stores a computer program, and where the processor implements the steps in the method embodiments described above when the processor executes the computer program.
In one embodiment, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
In one embodiment, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
The method and the device for automatically importing the interface file can be used in the financial field and can also be used in any field except the financial field, and the application field of the method and the device for automatically importing the interface file is not limited.
It should be noted that, user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (12)
1. An automatic importing method of interface files, which is characterized by comprising the following steps:
responding to task scheduling, and performing file inspection on at least one target interface file to obtain an inspection result of the target interface file;
under the condition that the checking result represents that the checking of each target interface file is successful, each target interface file is moved to a processing directory;
and determining the configuration file of the target interface file in the processing directory, and loading the configuration file into a formal table.
2. The method according to claim 1, wherein the method further comprises:
generating a corresponding external document and a file loading configuration script based on the internal document, wherein the file loading configuration script comprises file configuration and file field configuration of each target interface file;
storing the file configuration to a newly added file configuration table, and storing the file field configuration to the newly added file field configuration table.
3. The method according to claim 1, wherein the performing file inspection on at least one target interface file in response to task scheduling, to obtain an inspection result of the target interface file, includes:
and responding to the task scheduling defined by the real-time file loading task, and performing file inspection on the target interface file in real time to obtain an inspection result of the target interface file.
4. The method according to claim 1, wherein the performing file inspection on at least one target interface file in response to task scheduling, to obtain an inspection result of the target interface file, includes:
and responding to task scheduling defined by batch file loading tasks, and performing file inspection on a preset number of target interface files to obtain inspection results of the target interface files.
5. The method according to claim 2, wherein the performing file inspection on at least one target interface file to obtain an inspection result of the target interface file includes:
reading the corresponding file configuration from the file configuration table according to the batch number of the target interface file;
acquiring the file name and the file directory of the target interface file from the file configuration;
and checking whether the target interface file and the corresponding data verification file exist under the file directory according to the file name to obtain a checking result of the target interface file.
6. The method of claim 2, wherein said determining a configuration file for said target interface file in said process directory comprises:
judging whether a configuration file corresponding to the target interface file exists or not according to the file number of the target interface file in the processing directory, and obtaining a judging result;
acquiring the file field configuration corresponding to the target interface file from the file field configuration table under the condition that the judging result indicates that the configuration file does not exist, and generating the configuration file according to the file field configuration corresponding to the target interface file;
or under the condition that the judging result represents that the configuration file exists, detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detecting result.
7. The method of claim 6, wherein the detecting whether the configuration file needs to be updated according to the file configuration corresponding to the target interface file, and determining the configuration file according to the detection result, includes:
and under the condition that a configuration update mark exists in the file configuration corresponding to the target interface file, obtaining a detection result representing that the configuration file needs to be updated, and updating the configuration file according to the file field configuration.
8. The method according to claim 1, wherein the method further comprises:
and synchronously moving each target interface file to a backup directory while moving each target interface file to a processing directory.
9. An automatic importing device of interface files, characterized in that the device comprises:
the file checking module is used for responding to task scheduling, and performing file checking on at least one target interface file to obtain a checking result of the target interface file;
the file moving module is used for moving each target interface file to a processing catalog under the condition that the checking result represents that each target interface file is successfully checked;
and the file loading module is used for determining the configuration file of the target interface file in the processing directory and loading the configuration file into a formal table.
10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 8 when the computer program is executed.
11. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 8.
12. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311550323.3A CN117389978A (en) | 2023-11-20 | 2023-11-20 | Automatic importing method and device for interface file, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311550323.3A CN117389978A (en) | 2023-11-20 | 2023-11-20 | Automatic importing method and device for interface file, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117389978A true CN117389978A (en) | 2024-01-12 |
Family
ID=89464981
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311550323.3A Pending CN117389978A (en) | 2023-11-20 | 2023-11-20 | Automatic importing method and device for interface file, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117389978A (en) |
-
2023
- 2023-11-20 CN CN202311550323.3A patent/CN117389978A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11163731B1 (en) | Autobuild log anomaly detection methods and systems | |
| US7877732B2 (en) | Efficient stress testing of a service oriented architecture based application | |
| CN111026568B (en) | Data and task relation construction method and device, computer equipment and storage medium | |
| US9542173B2 (en) | Dependency handling for software extensions | |
| US20070234328A1 (en) | File handling for test environments | |
| CN114637511A (en) | Code testing system, method, device, electronic equipment and readable storage medium | |
| US11099837B2 (en) | Providing build avoidance without requiring local source code | |
| CN117112060A (en) | Component library construction method and device, electronic equipment and storage medium | |
| US20230259518A1 (en) | Scalable metadata-driven data ingestion pipeline | |
| CN113535141A (en) | Database operation code generation method and device | |
| CN116561003A (en) | Test data generation method, device, computer equipment and storage medium | |
| CN111400243B (en) | Development management system based on pipeline service and file storage method and device | |
| CN117389978A (en) | Automatic importing method and device for interface file, computer equipment and storage medium | |
| CN115757172A (en) | Test execution method and device, storage medium and computer equipment | |
| CN114756293A (en) | Service processing method, device, computer equipment and storage medium | |
| US11741255B2 (en) | System and method of block chain based protection for customized data integration processes | |
| CN114238085A (en) | Interface testing method and device, computer equipment and storage medium | |
| CN113504904A (en) | User-defined function implementation method and device, computer equipment and storage medium | |
| CN114391136A (en) | Enhanced virtual machine image management system | |
| US11762760B1 (en) | Scalable test workflow service | |
| CN113741956B (en) | Method and device for transplanting version management function, version management method and device | |
| US11934800B2 (en) | Generating metadata to facilitate code generation | |
| US20240143487A1 (en) | Secure testing of attachment functionality of objects | |
| US20220207438A1 (en) | Automatic creation and execution of a test harness for workflows | |
| CN117493199A (en) | Code verification method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |