CN112463729B

CN112463729B - Data file warehousing method and device, electronic equipment and medium

Info

Publication number: CN112463729B
Application number: CN202011366538.6A
Authority: CN
Inventors: 要凯升; 胥克良; 王秦川; 朱雪东
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2020-11-27
Filing date: 2020-11-27
Publication date: 2024-02-06
Anticipated expiration: 2040-11-27
Also published as: CN112463729A

Abstract

The present disclosure provides a method for warehousing data files, comprising: acquiring a call instruction of a transmission task, wherein the call instruction is used for indicating an upstream application to download a data file to a server; responding to the call instruction, and reading task information from the configuration file, wherein the task information comprises verification task information and warehousing task information; obtaining a data file actually received by a server; and warehousing the actually received data file based on the verification task information and the warehousing task information. The disclosure also provides a data file warehousing device, an electronic device and a computer readable storage medium. The methods and apparatus provided by the present disclosure may be applied, for example, in the financial or other fields.

Description

Data file warehousing method and device, electronic equipment and medium

Technical Field

The disclosure relates to the field of data processing, and in particular relates to a method, a device, electronic equipment and a medium for warehousing data files.

Background

Along with the continuous expansion of banking business scope, applications responsible for different business fields in a row are also increased, and due to the fact that the different business fields are crossed, the situation that data files are transmitted to each other inevitably occurs among a plurality of applications, and the complexity of checking and warehousing the data files issued by upstream applications by downstream applications is increased.

The related art also provides some solutions for checking the warehousing of data files transmitted between applications. For example by providing a dedicated application as a receiving party, responsible for the transmission and distribution of data files. For a received data file issued by an upstream application, a special analysis program is utilized to process the data file, data such as the file transmission upstream application, a file transmission mode, a file transmission scene, a specific field and the like are analyzed, and then the data file issued by the upstream application is subjected to file integrity verification and warehousing according to the analyzed data. When receiving data files issued by a plurality of different applications, the data files of the different applications need to be distinguished so as to carry out different processing according to the different applications.

However, in the solution provided in the related art, for each transmission task of the data file, a corresponding program code needs to be newly added to analyze the data file, so as to increase the number of data files issued, a plurality of corresponding program codes are newly added to analyze a plurality of data files, which results in redundancy of the program codes, reduces the analysis speed of the data files to a certain extent, and affects the verification and warehousing efficiency of the data files.

Disclosure of Invention

In view of this, in order to at least partially overcome the above-mentioned technical problems in the related art, an object of reducing program redundancy and improving transmission efficiency of a plurality of data files is achieved. The disclosure provides a data file warehousing method, a data file warehousing device, electronic equipment and a data file warehousing medium.

To achieve the above object, an aspect of the present disclosure provides a method for warehousing a data file, including: and obtaining a call instruction of the transmission task, wherein the call instruction is used for instructing an upstream application to download the data file to the server. And responding to the call instruction, and reading task information from the configuration file, wherein the task information comprises verification task information and warehousing task information. And obtaining the data file actually received by the server. And warehousing the actually received data file based on the verification task information and the warehousing task information.

According to an embodiment of the present disclosure, the reading task information from the configuration file in response to the call-up instruction includes: and calling the public method in response to the call instruction. And reading the task information from the configuration file by the public method.

According to an embodiment of the present disclosure, the warehousing the actually received data file based on the verification task information and the warehousing task information includes: and carrying out file verification on the actually received data file based on the verification task information so as to obtain a verification result. And warehousing the actually received data file based on the warehousing task information under the condition that the verification result shows that the verification is passed.

According to an embodiment of the present disclosure, the performing file verification on the actually received data file based on the verification task information includes: and calling the public method, and carrying out file integrity verification on the actually received data file based on the verification task information. And performing data validity verification on the actually received data file based on the verification task information under the condition that the file integrity verification passes.

According to an embodiment of the present disclosure, the verifying the file integrity of the actually received data file based on the verification task information includes: and reading the check task information to obtain the storage address of the data file on the server. And acquiring a data file stored in the storage address, wherein the data file comprises target attribute parameters corresponding to the transmission task. And obtaining the actual attribute parameters for the actually received data file. And performing file integrity verification on the actually received data file based on the target attribute parameter and the actual attribute parameter.

According to an embodiment of the present disclosure, the file integrity check includes at least one of: and (5) checking the file name. And checking the number of files. Checking the file size.

According to an embodiment of the present disclosure, the verifying the data validity of the actually received data file based on the verification task information includes: and reading the check task information to obtain a target check field. And acquiring target data based on the target check field. And analyzing the actually received data file based on the target check field to obtain the actually received data. And performing data validity check on the actually received data based on the target data.

According to an embodiment of the present disclosure, warehousing the actually received data file based on the warehousing task information includes: and reading the warehousing task information to obtain a data file warehousing mode corresponding to the transmission task. And calling the public method, and warehousing the actually received data file according to the data file warehousing mode.

According to an embodiment of the present disclosure, the above method further includes: and responding to the call instruction, and detecting whether task information matched with the transmission task exists in the configuration file. And if the transmission task does not exist, writing the task information matched with the transmission task into the configuration file so as to update the configuration file.

According to an embodiment of the present disclosure, the above method further includes: and responding to the modification request of the configuration file, and acquiring modification information corresponding to the transmission task. And updating task information matched with the transmission task based on the modification information.

According to an embodiment of the present disclosure, the above-mentioned verification task information includes at least one of: the upstream application downloads the file size of the data file to the server. The upstream application downloads the file name of the data file to the server. The upstream application downloads file transfer session of the data file to the server. The upstream application downloads the check field of the data file to the server.

To achieve the above object, another aspect of the present disclosure provides a data file warehousing apparatus, including: and the call instruction acquisition module is used for acquiring a call instruction of the transmission task, wherein the call instruction is used for indicating an upstream application to download the data file to the server. And the task information reading module is used for responding to the call instruction and reading task information from the configuration file, wherein the task information comprises verification task information and warehousing task information. And the data file obtaining module is used for obtaining the data file actually received by the server. And the data file warehousing module is used for warehousing the actually received data file based on the verification task information and the warehousing task information.

According to an embodiment of the present disclosure, the task information reading module includes: and the public device calling sub-module is used for calling the public device in response to the calling instruction. And the task information reading sub-module is used for reading the task information from the configuration file through the public device.

According to an embodiment of the present disclosure, the data file warehousing module includes: and the verification result obtaining sub-module is used for carrying out file verification on the actually received data file based on the verification task information so as to obtain a verification result. And the data file warehousing sub-module is used for warehousing the actually received data file based on the warehousing task information under the condition that the verification result shows that the verification is passed.

According to an embodiment of the present disclosure, the verification result obtaining submodule includes: and the file integrity checking unit is used for calling the public device and checking the file integrity of the actually received data file based on the checking task information. And a data validity checking unit, configured to perform data validity checking on the actually received data file based on the checking task information when the file integrity check passes.

According to an embodiment of the present disclosure, the file integrity checking unit includes: and the storage address acquisition subunit is used for reading the verification task information to acquire the storage address of the data file on the server. And the data file acquisition subunit is used for acquiring the data file stored in the storage address, wherein the data file comprises target attribute parameters corresponding to the transmission task. And the attribute parameter acquisition subunit is used for acquiring the actual attribute parameters for the actually received data files. And an integrity checking subunit, configured to perform file integrity checking on the actually received data file based on the target attribute parameter and the actual attribute parameter.

According to an embodiment of the present disclosure, the above-described data validity checking unit includes: and the check field reading subunit is used for reading the check task information so as to obtain a target check field. And the target data acquisition subunit is used for acquiring target data based on the target check field. And the data file analysis subunit is used for analyzing the actually received data file based on the target check field so as to obtain the actually received data. And a validity checking subunit, configured to perform data validity checking on the actually received data based on the target data.

According to an embodiment of the present disclosure, the data file warehousing submodule includes: and the warehousing mode obtaining unit is used for reading the warehousing task information to obtain a data file warehousing mode corresponding to the transmission task. And a data file warehousing unit for calling the public device and warehousing the actually received data file according to the data file warehousing mode.

According to an embodiment of the present disclosure, the above apparatus further includes: and the task information detection module is used for responding to the call instruction and detecting whether task information matched with the transmission task exists in the configuration file. And a configuration file updating module, configured to write, in the absence, task information that matches the transmission task into the configuration file, so as to update the configuration file.

According to an embodiment of the present disclosure, the above apparatus further includes: and the modification information acquisition module is used for responding to the modification request of the configuration file and acquiring modification information corresponding to the transmission task. And a task information updating module for updating task information matched with the transmission task based on the modification information.

To achieve the above object, another aspect of the present disclosure provides an electronic device, including: and one or more processors, a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for warehousing data files as described above.

To achieve the above object, another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are configured to implement a warehousing method for data files as described above.

To achieve the above object, another aspect of the present disclosure provides a computer program comprising computer executable instructions for implementing a method of warehousing data files as described above when executed.

Compared with the related art, the method for warehousing the data files has the advantages that the part of the data files to be checked in the data warehouse is configured in the configuration files, the configuration in the configuration files is read to operate the files, batch warehousing of the data files can be achieved, the information of the data files to be checked in the data warehouse is configured in the configuration files, the configuration in the configuration files is read in response to the call-up instruction, the files are operated, and the technical problems that a large number of files are transmitted among multiple applications, program redundancy is caused, maintenance is not facilitated, and the test cost is too high are solved.

Drawings

For a more complete understanding of the present disclosure and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

FIG. 1 schematically illustrates a system architecture of a method and apparatus for warehousing data files suitable for use in embodiments of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a method of warehousing data files according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a method of warehousing data files according to another embodiment of the present disclosure;

FIG. 4 schematically illustrates a block diagram of a warehousing apparatus for data files according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a schematic diagram of a computer-readable storage medium product suitable for implementing the data file warehousing method described above, according to an embodiment of the present disclosure; and

fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement the above described warehousing method of data files according to an embodiment of the present disclosure.

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are generally represented by like reference numerals throughout the figures for illustrative purposes.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Some of the block diagrams and/or flowchart illustrations are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data file storage device, such that the instructions, when executed by the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks. The techniques of this disclosure may be implemented in hardware and/or software (including firmware, microcode, etc.). Additionally, the techniques of this disclosure may take the form of a computer program product on a computer-readable storage medium having instructions stored thereon, the computer program product being for use by or in connection with an instruction execution system.

It should be noted that the method and the device for warehousing the data file provided by the present disclosure may be used in the financial field, and may also be used in any field other than the financial field. Therefore, the application fields of the data file warehousing method and device provided by the disclosure are not limited.

Along with the continuous expansion of banking business scope, a plurality of applications responsible for different business fields exist in a line, and due to business intersection, the situation that data file transmission needs to be performed between the plurality of applications sometimes occurs. Although special applications are in the line and are responsible for file transmission and distribution, when a receiver receives data files of a plurality of applications, different application files need to be distinguished and different processes are performed.

Under the existing method, a special program is used for processing a single transmission file. For the transmission files of different applications, a set of special program can analyze data such as the file transmission upstream application, the file transmission mode, the file transmission scene, specific fields and the like, and based on the data, the data files transmitted by the upstream application are subjected to file integrity verification and put in storage.

With the increase of downloaded files, for each file transmission task, a new block of code needs to be added to perform verification of file transmission times, transmission types and the like and reading of data, which has the following problems: more program redundancy occurs. And is not beneficial to the maintenance of the whole method. Because a task corresponds to a file reading program, different test cases need to be implemented, and more test cost is consumed.

Fig. 1 schematically illustrates a system architecture 100 suitable for use with the warehousing methods and apparatus of data files according to embodiments of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios.

As shown in fig. 1, a system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, data, etc. acquired or generated according to the user request) to the terminal device.

It should be noted that, the method for warehousing the data file provided in the present disclosure may be generally performed by the server 105. Accordingly, the warehousing means for the data files provided by the present disclosure may be generally provided in the server 105. The warehousing method of the data file provided by the present disclosure may also be performed by other servers other than the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the warehousing means for the data files provided by the present disclosure may also be provided in other servers than the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Fig. 2 schematically illustrates a flowchart of a method of warehousing data files according to an embodiment of the present disclosure. As shown in fig. 2, the method 200 may include operations S210 to S240.

In operation S210, a call instruction of a transmission task is obtained.

In operation S220, task information is read from the configuration file in response to the call instruction.

In operation S230, a data file actually received by the server is obtained.

In operation S240, the actually received data file is binned based on the verification task information and binning task information.

According to an embodiment of the present disclosure, the call-up instruction is used to instruct the upstream application to download the data file to the server. The configuration file is configured with different fields to distinguish between different verification fields, including, but not limited to, a storage location field of the data file, a file name field of the data file, a number of files field of the data file, a file size field of the data file, a file transfer field of the data file, a location of each field in the file, and a database table in which the data is specifically stored.

According to the embodiment of the disclosure, the task information is used for representing the part to be checked corresponding to the transmission task, and may include check task information and warehouse-in task information, where the check task information is used for representing check information for checking the data file, so as to check whether the data file actually received by the server is consistent with the data file corresponding to the transmission task. The warehousing task information is used for representing a warehousing entry form of the data file so as to indicate the warehousing form of the checked data file to be stored in the database.

According to an embodiment of the present disclosure, the foregoing verification task information may include at least one of: the upstream application downloads the file size of the data file to the server; the upstream application downloads the file name of the data file to the server; the upstream application downloads file transmission times of the data files to the server; the upstream application downloads the check field of the data file to the server.

In the related art, for each file transmission task, a new code needs to be added to perform verification of file transmission times, file transmission types and the like and reading of data, so that more program redundancy occurs, and as the file transmission tasks are increased, the program redundancy is increased, and the execution efficiency of the file transmission tasks is seriously affected.

According to the embodiment of the disclosure, batch storage of the data files can be realized, information to be checked in the data file storage is configured in the configuration file, and the configuration in the configuration file is read to operate the file in response to the call instruction, so that the technical problems of program redundancy, inconvenience in maintenance, excessive test cost and the like caused by large quantity of file transmission among multiple applications are solved.

As an alternative embodiment, the foregoing reading task information from the configuration file in response to the foregoing call instruction may include: calling a public method in response to the calling instruction; and reading the task information from the configuration file by the public method.

According to the embodiment of the disclosure, in the whole data file transmission process, different processing methods are needed, including but not limited to reading of configuration files, checking of data files and warehousing of data files, and these processing methods need to be executed for each file transmission task, and the disclosure refers to the processing methods for characterizing that different file transmission tasks need to be executed, and the disclosure uniformly manages the processing methods, so that when executing different file transmission tasks, it is not necessary to re-write the part of programs, thereby achieving the effect of multiplexing.

According to the embodiment of the disclosure, since the common method represents a fixed method which is required to be executed in the transmission task, once the writing is completed, the common method can be reused by different transmission tasks, when the common method is responded to a new transmission task, different processing logics can be directly called to execute, the re-writing is not needed, after the common method is tested once, the processing logics do not need to be tested for a plurality of times in future, only a specific data entry method is needed to be concerned, and unnecessary resource loss can be reduced.

As an alternative embodiment, the aforementioned operation S240 (warehousing the actually received data file based on the verification task information and the warehousing task information) may include: based on the verification task information, performing file verification on the actually received data file to obtain a verification result; and warehousing the actually received data file based on the warehousing task information under the condition that the verification result shows that the verification is passed.

According to the embodiment of the disclosure, in the case that the foregoing verification result indicates that the verification fails, the data file actually received will not perform the data file warehousing operation. Optionally, under the condition that the verification fails, verification error information can be generated to represent that the actually received data file is inconsistent with the data file corresponding to the transmission task, so that related personnel can intervene in the transmission task in time, and the problem is solved as soon as possible.

As an optional embodiment, the performing file verification on the actually received data file based on the verification task information may include: invoking the public method, and carrying out file integrity verification on the actually received data file based on the verification task information; and carrying out data validity verification on the actually received data file based on the verification task information under the condition that the file integrity verification passes.

According to embodiments of the present disclosure, performing file verification on an actually received data file includes file integrity verification and data validity verification. The file integrity check is used to check whether the data file is complete, and refers to a file form check of the data file. The data validity check is used to check whether the data file meets the specification, and refers to the file content check of the data file.

In the specific implementation, the file integrity check can be performed first, and whether the data validity check is performed or not is determined according to the check result of the file integrity check. And under the condition that the file integrity check is passed, carrying out data validity check, namely checking the file form without errors, and then carrying out file content check. And under the condition that the file integrity check is not passed, the data validity check is not carried out, namely the file form check is wrong, and the file content check is not carried out. The verification mode of the first file form verification and the later data content verification can improve the verification efficiency of the data file and ensure the transmission safety of the data file.

As an optional embodiment, the verifying the file integrity of the actually received data file based on the verification task information may include: reading the verification task information to obtain a storage address of the data file on the server; acquiring a data file stored on the storage address, wherein the data file comprises target attribute parameters corresponding to the transmission task; obtaining actual attribute parameters aiming at the actual received data file; and performing file integrity verification on the actually received data file based on the target attribute parameter and the actual attribute parameter.

As an alternative embodiment, the foregoing file integrity check may include at least one of: checking the file name; checking the number of files; checking the file size.

In specific implementation, the storage position of the data file can be obtained, the corresponding data file is obtained from the server according to the storage position, and the corresponding relationship among the file size, the file transmission time, the data file and the verification file is verified according to the transmission number of the data file recorded in the file, so that the file transmission safety can be ensured.

As an optional embodiment, the verifying the validity of the data of the actually received data file based on the verification task information may include: reading the verification task information to obtain a target verification field; acquiring target data based on the target check field; analyzing the actually received data file based on the target check field to obtain actually received data; and performing data validity check on the actually received data based on the target data.

According to the embodiment of the disclosure, the data can be read according to the fields recorded in the configuration file and the corresponding positions of the fields in the file, and the validity of the data is verified.

As an optional embodiment, the warehousing the actually received data file based on the warehousing task information may include: reading the warehousing task information to obtain a data file warehousing mode corresponding to the transmission task; and calling the public method, and warehousing the actually received data file according to the data file warehousing mode.

According to the embodiment of the disclosure, a corresponding data entry method can be called according to an increment or a full-quantity entry mode recorded in the configuration file, and data in the data file is entered into a table.

As an alternative embodiment, the foregoing method may further include: responding to the call instruction, and detecting whether task information matched with the transmission task exists in the configuration file; and if the transmission task does not exist, writing the task information matched with the transmission task into the configuration file so as to update the configuration file.

According to the embodiment of the disclosure, each time a transmission task is newly added, task information corresponding to the newly added transmission task may be added to the configuration file, and the task information may include verification task information and warehousing task information, where the verification task information is used to characterize newly added verification information for verifying the data file, so as to verify whether the data file actually received by the server is consistent with the data file corresponding to the newly added transmission task. The warehousing task information is used for representing a newly added warehousing entry form of the data file so as to indicate the newly added warehousing form of the data file passing verification to be stored in the database.

As an alternative embodiment, the foregoing method may further include: responding to a modification request of the configuration file, and acquiring modification information corresponding to the transmission task; and updating task information matched with the transmission task based on the modification information.

According to the embodiment of the disclosure, each time one transmission task is modified, task information corresponding to the modified transmission task may be added to the configuration file, and the task information may include verification task information and warehousing task information, where the verification task information is used to characterize modification verification information for verifying the data file, so as to verify whether the data file actually received by the server is consistent with the data file corresponding to the modified transmission task. The warehousing task information is used for representing a warehousing entry form of modification of the data file so as to indicate the warehousing form of modification of the data file passing verification to be stored in the database.

By the embodiment of the disclosure, all task information of the transmission task can be maintained in one configuration file, if the task information needs to be modified, the modification can be performed in the configuration file, if the data field needs to be newly added, added or deleted, the modification is performed in the configuration file, and the corresponding field is added or deleted in the corresponding entity, so that excessive program variation is not involved, the program invasiveness is small, and the whole maintenance is facilitated.

Take the receiving user whitelist as an example. After the upstream application transmits the data file to the server, firstly, the file is read according to the file position recorded in the configuration file, the content of the check file, namely the chk file, is analyzed, the name of the data file transmitted at the time is compared with the names of the configuration file and the files recorded in the chk file, and the transmitted file is confirmed to be correct. In addition, the number of files and the size of each file can be checked according to records in the chk file. After the data file passes the verification, the data file is analyzed according to the positions of the fields recorded in the configuration file in the file, the required data is obtained from the data file, after the data compliance is verified, a database table in which the data is stored in detail is obtained from the configuration file, and the data is put in storage.

Fig. 3 schematically illustrates a flow chart of a method of warehousing data files according to another embodiment of the present disclosure.

As shown in FIG. 3, the method 300 provides a general method for implementing batch file warehousing. The batch files to be warehoused include N transmission tasks, transmission task 1, transmission task M, transmission task N. M, N are positive integers greater than 2, and M < N. Transmission task 1 corresponds to data entry method 1, a.i. Data entry method M corresponding to transmission task M once again, task N corresponds to data entry method N.

The method mainly comprises a configuration file and a public method, wherein the operation S310, the file integrity and the data verification are carried out in the public method, the operation S320 is carried out, the configuration file is read to obtain configuration, the file occasions of file transmission tasks and the like are determined, the data file transmitted by an upstream application is subjected to file verification, and then a corresponding database entry method is called to enter the file contents into a table. The storage position of the data file on the server in the configuration file, the corresponding database table, the file transmission mode, the data processing method corresponding to the data file, the data field and the file transmission field information are recorded in the incremental or full-amount table entry mode. When a new file transfer task is called, a public method is required to be called first, and specific file transfer parameters of the file transfer task are read from a configuration file. The storage position of the file is firstly acquired, the corresponding data file is acquired from the server, and the corresponding relation of the file size, the file transmission time, the data file and the verification file is checked according to the number of the file transmission time, the file size and the file name recorded in the file, so that the safety of file transmission is ensured. And then reading data according to the fields recorded in the configuration file and the corresponding positions of the fields in the file, and calling a corresponding data entry method to enter the data into a table according to the increment or the total entry mode recorded in the configuration file after verifying the validity of the data. And after the data corresponding to the transmission task 1 passes the verification, the data is put into a database by a data entry method 1, after the data corresponding to the transmission task M passes the verification, the data is put into a database by a data entry method M, and after the data corresponding to the transmission task N passes the verification, the data is put into a database by a data entry method N.

Fig. 4 schematically illustrates a block diagram of a warehousing apparatus for data files according to an embodiment of the present disclosure. As shown in fig. 4, the apparatus 400 may include a call instruction acquisition module 410, a task information reading module 420, a data file acquisition module 430, and a data file warehousing module 440.

The call instruction obtaining module 410 is configured to obtain a call instruction of the transmission task, where the call instruction is used to instruct the upstream application to download the data file to the server. Optionally, the call instruction obtaining module 410 may be used to perform the operation S210 described in fig. 2, which is not described herein.

The task information reading module 420 is configured to read task information from the configuration file in response to the call instruction, where the task information includes verification task information and warehouse entry task information. Optionally, the task information reading module 420 may be used to perform the operation S220 described in fig. 2, which is not described herein.

The data file obtaining module 430 is configured to obtain the data file actually received by the server. Optionally, the data file obtaining module 430 may be used to perform operation S230 described in fig. 2, which is not described herein.

And a data file warehousing module 440, configured to warehouse the actually received data file based on the verification task information and the warehousing task information. Optionally, the data file warehousing module 440 may be used to perform operation S240 described in fig. 2, for example, which is not described herein.

As an alternative embodiment, the task information reading module includes: and the public device calling sub-module is used for calling the public device in response to the calling instruction. And the task information reading sub-module is used for reading the task information from the configuration file through the public device.

As an alternative embodiment, the foregoing data file warehousing module may include: and the verification result obtaining sub-module is used for carrying out file verification on the actually received data file based on the verification task information so as to obtain a verification result. And the data file warehousing sub-module is used for warehousing the actually received data file based on the warehousing task information under the condition that the verification result shows that the verification is passed.

As an alternative embodiment, the foregoing verification result obtaining sub-module may include: and the file integrity checking unit is used for calling the public device and checking the file integrity of the actually received data file based on the checking task information. And a data validity checking unit, configured to perform data validity checking on the actually received data file based on the checking task information when the file integrity check passes.

As an alternative embodiment, the foregoing file integrity checking unit may include: and the storage address acquisition subunit is used for reading the check task information to acquire the storage address of the data file on the server. And the data file acquisition subunit is used for acquiring the data file stored on the storage address, wherein the data file comprises target attribute parameters corresponding to the transmission task. And the attribute parameter acquisition subunit is used for acquiring the actual attribute parameters for the actually received data file. And an integrity checking subunit, configured to perform file integrity checking on the actually received data file based on the target attribute parameter and the actual attribute parameter.

As an alternative embodiment, the foregoing file integrity check may include at least one of: and (5) checking the file name. And checking the number of files. Checking the file size.

As an alternative embodiment, the foregoing data validity checking unit may include: and the check field reading subunit is used for reading the check task information so as to obtain a target check field. And the target data acquisition subunit is used for acquiring target data based on the target check field. And the data file analysis subunit is used for analyzing the actually received data file based on the target check field so as to obtain the actually received data. And a validity checking subunit, configured to perform data validity checking on the actually received data based on the target data.

As an alternative embodiment, the foregoing data file warehousing sub-module may include: and the warehousing mode obtaining unit is used for reading the warehousing task information to obtain a data file warehousing mode corresponding to the transmission task. And a data file warehousing unit for calling the public device and warehousing the actually received data file according to the data file warehousing mode.

As an alternative embodiment, the foregoing apparatus may further include: and the task information detection module is used for responding to the call instruction and detecting whether task information matched with the transmission task exists in the configuration file. And a configuration file updating module, configured to write, in the absence, task information that matches the transmission task into the configuration file, so as to update the configuration file.

As an alternative embodiment, the foregoing apparatus may further include: and the modification information acquisition module is used for responding to the modification request of the configuration file and acquiring modification information corresponding to the transmission task. And a task information updating module for updating task information matched with the transmission task based on the modification information.

As an alternative embodiment, the foregoing verification task information may include at least one of: the upstream application downloads the file size of the data file to the server. The upstream application downloads the file name of the data file to the server. The upstream application downloads the file transfer session of the data file to the server. The upstream application downloads the check field of the data file to the server.

It should be noted that, the implementation manner, the technical problem to be solved, the function to be realized, and the technical effect to be achieved of each module in the embodiment of the apparatus portion are the same as or similar to the implementation manner, the technical problem to be solved, the function to be realized, and the technical effect to be achieved of each corresponding step in the embodiment of the method portion, respectively, and are not described herein again.

Any number of modules, sub-modules, units, sub-units, or at least some of the functionality of any number of the sub-units according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented as split into multiple modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as hardware circuitry, such as field programmable gate arrays (FNGA), programmable logic arrays (NLA), systems on chip, systems on substrate, systems on package, application Specific Integrated Circuits (ASIC), or in hardware or firmware in any other reasonable manner of integrating or packaging circuitry, or in any one of or a suitable combination of any of three implementations of software, hardware, and firmware. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be at least partially implemented as computer program modules, which when executed, may perform the corresponding functions.

For example, the call instruction obtaining module, the task information reading module, the data file obtaining module, the data file warehousing module, the public device calling sub-module, the task information reading sub-module, the verification result obtaining sub-module, the data file warehousing sub-module, the file integrity verification unit, the data validity verification unit, the storage address obtaining sub-unit, the data file obtaining sub-unit, the attribute parameter obtaining sub-unit, the integrity verification sub-unit, the verification field reading sub-unit, the target data obtaining sub-unit, the data file analyzing sub-unit, the validity verification sub-unit, the warehousing mode obtaining unit, the data file warehousing unit, the task information detecting module, the configuration file updating module, the modification information obtaining module and the task information updating module may be combined into one module to be implemented, or any one of the modules may be split into a plurality of modules. Alternatively, at least some of the functionality of one or more of the modules may be combined with, and implemented in, at least some of the functionality of other modules. According to embodiments of the present disclosure, at least one of the call instruction obtaining module, the task information reading module, the data file obtaining module, the data file warehousing module, the public device calling sub-module, the task information reading sub-module, the verification result obtaining sub-module, the data file warehousing sub-module, the file integrity checking unit, the data validity checking unit, the storage address obtaining sub-unit, the data file obtaining sub-unit, the attribute parameter obtaining sub-unit, the integrity checking sub-unit, the verification field reading sub-unit, the target data obtaining sub-unit, the data file parsing sub-unit, the validity checking sub-unit, the warehousing manner obtaining unit, the data file warehousing unit, the task information detecting module, the configuration file updating module, the modification information obtaining module, and the task information updating module may be implemented at least partially as hardware circuits, such as a field programmable gate array (FNGA), a programmable logic array (NLA), a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or any other reasonable manner or firmware that may be implemented by integrating or packaging the circuits, or any suitable combination of any of the three of the hardware, the configuration file updating module, the modification information obtaining module, and the configuration information updating module, and the task information updating module may be implemented at least partially as a suitable combination of any one of these. Or, at least one of the call instruction obtaining module, the task information reading module, the data file obtaining module, the data file warehousing module, the public device calling sub-module, the task information reading sub-module, the verification result obtaining sub-module, the data file warehousing sub-module, the file integrity verification unit, the data validity verification unit, the storage address obtaining sub-unit, the data file obtaining sub-unit, the attribute parameter obtaining sub-unit, the integrity verification sub-unit, the verification field reading sub-unit, the target data obtaining sub-unit, the data file analyzing sub-unit, the validity verification sub-unit, the warehousing mode obtaining unit, the data file warehousing unit, the task information detecting module, the configuration file updating module, the modification information obtaining module and the task information updating module may be at least partially implemented as a computer program module, which may perform corresponding functions when the computer program module is operated.

Fig. 5 schematically illustrates a schematic diagram of a computer-readable storage medium product suitable for implementing the above-described warehousing method of data files according to an embodiment of the present disclosure.

In some possible implementations, the aspects of the present invention may also be implemented in the form of a program product including program code for causing an apparatus to perform the aforementioned operations (or steps) in a warehousing method of data files according to various exemplary embodiments of the present invention described in the "exemplary method" section of this specification when the program product is run on the apparatus, e.g., the electronic apparatus may perform operations S210 through S240 as illustrated in fig. 2.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (ENROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the preceding.

As shown in fig. 5, a program product 500 for warehousing of data files according to an embodiment of the present invention is described, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a device such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device.

The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, or device. Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C", language or similar programming languages. The program code may execute entirely on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the context of remote computing devices, the remote computing device may connect to the user computing device through any kind of network, including a local area network (LAA) or wide area network (WAA), or may connect to an external computing device (e.g., through the internet using an internet service provider).

Fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement the above described warehousing method of data files according to an embodiment of the present disclosure. The electronic device shown in fig. 6 is merely an example and should not be construed to limit the functionality and scope of use of embodiments of the present disclosure in any way.

As shown in fig. 6, an electronic device 600 according to an embodiment of the present disclosure includes a processor 601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. Processor 601 may include, for example, a general-purpose microprocessor (e.g., CNU), an instruction set processor and/or an associated chipset and/or a special-purpose microprocessor (e.g., application Specific Integrated Circuit (ASIC)), or the like. Processor 601 may also include on-board memory for caching purposes. The processor 601 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.

In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or the RAM 603. Note that the program may be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also perform operations S210 to S240 shown in fig. 2 according to an embodiment of the present disclosure by executing programs stored in the one or more memories.

The electronic device 600 may also include an input/output (I/O) interface 605, the input/output (I/O) interface 605 also being connected to the bus 604, in accordance with an embodiment of the present disclosure. The system 600 may also include one or more of the following components connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as an LAA card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed on the drive 610, and a computer program read out therefrom is installed into the storage section 608 as needed.

The method flow according to embodiments of the present disclosure may be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program loaded on a computer-readable storage medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. The systems, devices, apparatus, modules, units, etc. described above, according to embodiments of the present disclosure, may be implemented by computer program modules.

The present disclosure also provides a computer-readable storage medium, which may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The above-described computer-readable storage medium carries one or more programs, which when executed, implement a file warehousing method according to an embodiment of the present disclosure, including operations S210 to S240 shown in fig. 2.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: portable computer diskette, hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (ENROM or flash memory), portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing. In this disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include ROM 602 and/or RAM 603 and/or one or more memories other than ROM 602 and RAM 603 described above.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments and/or claims of the present disclosure may be provided in a variety of combinations and/or combinations even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or in the claims may be combined in various combinations and/or combinations without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various substitutions and modifications may be made by those skilled in the art without departing from the scope of the disclosure, and such substitutions and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of warehousing data files, comprising:

obtaining a call instruction of a transmission task, wherein the call instruction is used for indicating an upstream application to download a data file to a server;

Responding to the call instruction, and reading task information from a configuration file, wherein the task information comprises verification task information and warehousing task information;

obtaining a data file actually received by the server;

based on the verification task information and the warehousing task information, warehousing the actually received data file;

wherein, the responding to the call instruction, the reading task information from the configuration file comprises:

calling a public method in response to the call instruction;

and reading task information from the configuration file through the public method.

2. The method of claim 1, wherein the warehousing the actual received data file based on the verification task information and the warehousing task information comprises:

based on the verification task information, performing file verification on the actually received data file to obtain a verification result;

and under the condition that the verification result shows that the verification is passed, warehousing the actually received data file based on the warehousing task information.

3. The method of claim 2, wherein the performing file verification on the actually received data file based on the verification task information comprises:

Invoking the public method, and performing file integrity verification on the actually received data file based on the verification task information;

and under the condition that the file integrity check is passed, carrying out data validity check on the actually received data file based on the check task information.

4. A method according to claim 3, wherein said verifying file integrity of said actually received data file based on said verification task information comprises:

reading the verification task information to obtain a storage address of the data file on the server;

acquiring a data file stored on the storage address, wherein the data file comprises target attribute parameters corresponding to the transmission task;

obtaining actual attribute parameters aiming at the actually received data file;

and carrying out file integrity verification on the actually received data file based on the target attribute parameter and the actual attribute parameter.

5. A method according to claim 3, wherein the file integrity check comprises at least one of:

checking the file name;

checking the number of files;

Checking the file size.

6. A method according to claim 3, wherein said verifying the data validity of the actually received data file based on the verification task information comprises:

reading the verification task information to obtain a target verification field;

acquiring target data based on the target check field;

analyzing the actually received data file based on the target check field to obtain actually received data;

and carrying out data validity verification on the actually received data based on the target data.

7. The method of claim 2, wherein the binning the actual received data file based on the binning task information comprises:

reading the warehousing task information to obtain a data file warehousing mode corresponding to the transmission task;

and calling the public method, and warehousing the actually received data file according to the data file warehousing mode.

8. The method of claim 1, wherein the method further comprises:

responding to the call instruction, and detecting whether task information matched with the transmission task exists in the configuration file;

And if the task information matched with the transmission task does not exist, writing the task information matched with the transmission task into the configuration file to update the configuration file.

9. The method of claim 8, wherein the method further comprises:

responding to the modification request of the configuration file, and acquiring modification information corresponding to the transmission task;

and updating task information matched with the transmission task based on the modification information.

10. The method of claim 1, wherein the verification task information comprises at least one of:

the upstream application downloads the file size of the data file to the server;

the upstream application downloads the file name of the data file to the server;

the upstream application downloads file transmission times of the data files to the server;

the upstream application downloads the check field of the data file to the server.

11. A warehousing apparatus for data files, comprising:

the system comprises a call instruction acquisition module, a call instruction processing module and a data file processing module, wherein the call instruction acquisition module is used for acquiring a call instruction of a transmission task, and the call instruction is used for indicating an upstream application to download a data file to a server;

the task information reading module is used for responding to the call instruction and reading task information from the configuration file, wherein the task information comprises verification task information and warehousing task information;

The data file obtaining module is used for obtaining the data file actually received by the server;

the data file warehousing module is used for warehousing the actually received data file based on the verification task information and the warehousing task information;

wherein the device is adapted to implement the method according to claims 1-10.

12. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1 to 10.

13. A computer readable storage medium storing computer executable instructions which when executed are adapted to implement the method of any one of claims 1 to 10.