CN112445866A

CN112445866A - Data processing method and device, computer readable medium and electronic equipment

Info

Publication number: CN112445866A
Application number: CN201910745927.0A
Authority: CN
Inventors: 穆启健
Original assignee: Beijing Jingdong Zhenshi Information Technology Co Ltd
Current assignee: Beijing Jingdong Zhenshi Information Technology Co Ltd
Priority date: 2019-08-13
Filing date: 2019-08-13
Publication date: 2021-03-05

Abstract

The embodiment of the invention provides a data processing method, a data processing device, a computer readable medium and electronic equipment, and relates to the technical field of data processing. The data processing method comprises the following steps: acquiring a form file containing financial data; reading the table file through an asynchronous task, analyzing the table file into data records, and obtaining a message queue; reading the data record from the message queue, and checking the data record according to the file type of the data record; and determining the data records passing the verification as importable data for the financial system to import into the database. According to the technical scheme of the embodiment of the invention, the form file of the financial data is analyzed into the data record, so that the financial data is uniformly processed, manual account checking is converted into automatic check, and the complexity of operation is reduced.

Description

Data processing method and device, computer readable medium and electronic equipment

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data processing method, a data processing apparatus, a computer readable medium, and an electronic device.

Background

With the rapid growth of economy, both enterprises and individuals face complex financial data, and the complex financial data imply very much information, so the processing of the financial data is particularly important.

In the transferring and docking processes of financial data, various documents and calculation models exist, and not only various calculations need to be carried out, but also operations such as reconciliation, posting and the like need to be carried out after results are calculated. At present, most of data are manually extracted, calculated, checked account and settled, and the like, so that great pressure is applied to operators. In addition, although some data can obtain the calculation result of the system by means of importing the text into the system, the operations such as reconciliation and posting still need to be performed manually, which causes a problem of high manual input cost.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present invention and therefore may include information that does not constitute prior art known to a person of ordinary skill in the art.

Disclosure of Invention

The inventor finds that in the related art, the financial data are mostly calculated, checked and settled, posted and the like in a manual mode, and the operation is complicated. And the mode of generating the script by the text import system to determine the calculation result is easy to cause long text analysis import time and low efficiency due to large data volume.

In view of the foregoing, it is an object of the embodiments of the present invention to provide a data processing method, a data processing apparatus, a computer readable medium and an electronic device, which overcome the above problems at least to some extent.

Additional features and advantages of the invention will be set forth in the detailed description which follows, or may be learned by practice of the invention.

According to a first aspect of the embodiments of the present invention, there is provided a data processing method, including: acquiring a form file containing financial data; reading the table file through an asynchronous task, analyzing the table file into data records, and obtaining a message queue; reading the data record from the message queue, and checking the data record according to the file type of the data record; and determining the data records passing the verification as importable data for the financial system to import into the database.

In an exemplary embodiment of the invention, the obtaining a form file containing financial data comprises: and storing the table file through a distributed file storage system.

In an exemplary embodiment of the present invention, after parsing the table file into data records, the method further includes: and determining the identification information of each data record so as to determine the verification result of each data record.

In an exemplary embodiment of the present invention, the determining the identification information of each data record includes: and determining the identification information of each data record through the first identification of the asynchronous task and the second identification of the financial data contained in the table file.

In an exemplary embodiment of the present invention, after verifying the data record according to the file type of the data record, the method further includes: determining the verification progress of the form file according to the identification information of the data record; and if the verification progress is verification completion, determining a verification result of the form file.

In an exemplary embodiment of the present invention, after determining the verification result of each of the data records, the method further includes: and verifying the data records which fail to be verified again according to the verification result of the data records so as to determine the abnormal data in the table file.

In an exemplary embodiment of the invention, the importing by the financial system into a database comprises: and inquiring the importable data according to the data authority of the financial system to obtain a target table for the financial system to import into a database.

According to a second aspect of the embodiments of the present invention, there is provided a data processing apparatus including: the data acquisition unit is used for acquiring a table file containing financial data; the data analysis unit is used for reading the table file through an asynchronous task, analyzing the table file into data records and acquiring a message queue; the data checking unit is used for reading the data records from the message queue and checking the data records according to the file types of the data records; and the data storage unit is used for storing the data records passing the verification to obtain importable data for the financial system to import into the database.

According to a third aspect of embodiments of the present invention, there is provided a computer-readable medium, on which a computer program is stored, which when executed by a processor, implements the data processing method as described in the first aspect of the embodiments above.

According to a fourth aspect of embodiments of the present invention, there is provided an electronic apparatus, including: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out a data processing method as described in the first aspect of the embodiments above.

The technical scheme provided by the embodiment of the invention has the following beneficial effects:

in the technical solutions provided by some embodiments of the present invention, on one hand, the financial data is normalized in the form of a table file, so that the financial data is clearer and is convenient for processing, tracking and auditing. On the other hand, the table file is processed into the data records, so that the table file is uniformly processed, the complex processing process caused by various files is avoided, and the complexity is reduced. And the data records are processed through asynchronous tasks, the acquired form files are decoupled from subsequent processing, the pressure on database operation can be reduced, and the stability of the system is improved. On the other hand, the data records are stored in the message queue, and the data records are taken out from the message queue for verification, so that asynchronous processing of the data is further realized, normal use of the database under the condition of large data volume is ensured, and the throughput, concurrency and availability of the system are improved. And moreover, the data records are verified according to different data record types, and operations such as calculation and account checking are automatically completed, so that the automation degree is improved, the operation is simplified, and the processing efficiency is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:

FIG. 1 schematically shows a system architecture diagram for implementing a data processing method of an embodiment of the present invention;

FIG. 2 schematically shows a flow diagram of a data processing method according to an embodiment of the invention;

FIG. 3 schematically shows a flow chart of a data processing method according to another embodiment of the invention;

FIG. 4 schematically shows a flow chart of a data processing method according to a further embodiment of the invention;

FIG. 5 schematically shows a flow chart of a data processing method according to another embodiment of the invention;

FIG. 6 schematically shows a flow chart of a data processing method according to a further embodiment of the invention;

FIG. 7 schematically shows a flow chart of a data processing method according to another embodiment of the invention;

FIG. 8 schematically shows a flow chart of a data processing method according to a further embodiment of the invention;

FIG. 9 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present invention;

FIG. 10 illustrates a schematic structural diagram of a computer system suitable for use with the electronic device to implement an embodiment of the invention.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations or operations have not been shown or described in detail to avoid obscuring aspects of the invention.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

First, a system architecture for implementing a data processing method is provided in an exemplary embodiment of the present disclosure. Referring to fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send request instructions or the like. The

terminal devices

101, 102, 103 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (for example only) providing support for shopping-like websites browsed by users using the

terminal devices

101, 102, 103. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.

It should be noted that the data processing method provided by the embodiment of the present disclosure is generally executed by the server 105, and accordingly, the data processing apparatus is generally disposed in the terminal device 101.

Based on the system architecture, the exemplary embodiment of the present disclosure provides a data processing method. As shown in fig. 2, the data processing method may include the steps of:

s210, acquiring a table file containing financial data;

s220, reading the table file through an asynchronous task, analyzing the table file into data records, and obtaining a message queue;

s230, reading the data records from the message queue, and checking the data records according to the file types of the data records;

and S240, storing the data records passing the verification to obtain importable data for the financial system to import into the database.

According to the data processing method of the exemplary embodiment, on one hand, the financial data is normalized in a form of a table file, so that the financial data is clearer and is convenient to process and track audit. On the other hand, the table file is processed into the data records, so that the table file is uniformly processed, the complex processing process caused by various files is avoided, and the complexity is reduced. And the data records are processed through asynchronous tasks, the acquired form files are decoupled from subsequent processing, the pressure on database operation can be reduced, and the stability of the system is improved. On the other hand, the data records are stored in the message queue, and the data records are taken out from the message queue for verification, so that asynchronous processing of the data is further realized, normal use of the database under the condition of large data volume is ensured, and the availability of the system is improved. And moreover, the data records are verified according to the file types of different data records, and operations such as calculation and account checking are automatically completed, so that the automation degree is improved, the operation is simplified, and the processing efficiency is improved.

Hereinafter, each step of the present exemplary embodiment is explained in detail.

Referring to fig. 2, in step S210, a form file containing financial data is acquired.

In the exemplary embodiment, different excel templates can be set for different service scenarios to normalize financial data. Specifically, the financial data may be classified according to business scenarios, such as product income, daily expenses, logistics data, and the like; the financial data can also be classified according to a data calculation model, such as general statistics, special calculation rules, and the like; this embodiment is not particularly limited to this. The financial data is filled into the template to obtain a form file, and an operator can upload the form file through the interface so as to obtain a plurality of form files collected offline, wherein the form files can be of different file types.

The table files uploaded through the interface can be stored through a distributed file storage system. The distributed file storage system means that physical storage resources are not necessarily directly connected with local nodes, but are connected with nodes through a computing network, a plurality of nodes jointly form a file system network, and each node can be distributed in different computers to carry out communication and data transmission among the nodes through the network. In the case of a very large amount of data, the distributed file system can effectively solve the problem of limitation of storage resources, so that a large amount of financial data, i.e., table files, can be saved. However, in alternative embodiments, the obtained table file may also be stored by other storage systems, such as cloud storage.

In step S220, the table file is read by the asynchronous task, and the table file is parsed into data records, so as to obtain a message queue.

The execution of synchronous tasks is performed in the order of the codes, each program must wait for the completion of the execution of the previous program before execution, and asynchronous tasks allow subsequent operations without blocking the currently executed program. In other words, asynchronous tasks execute subsequent programs without waiting for the statements to return results when the code is executed. Therefore, an asynchronous task can be created for each table file, and the table file is read by the asynchronous task to be analyzed. For example, when an operator uploads a plurality of form files and an asynchronous task reads the form files, the currently read form files can be analyzed without waiting for the completion of the analysis of the last uploaded form file.

Different asynchronous tasks may process different files and therefore the asynchronous task may be identified. When the asynchronous tasks are generated, an identification number can be determined for each asynchronous task code, the tasks for processing different files can be distinguished by identifying the tasks through the identification numbers, and the processing conditions of the different files are tracked.

A data record may refer to a complete set of data in the source data, i.e., a row of data in a table file. Therefore, the data record may include information such as a row number and an attribute value in the table file, however, the data record may also include other information, such as identification information of the asynchronous task, a file type to which the table belongs, and the like, which is not limited in this embodiment. Parsing the tabular file into data records enables different types of tabular data to be converted into a unified data form. Illustratively, the table file can be parsed into a strip of data records through an EazyExcel framework. Parsing the file into data records is more convenient for data transfer, i.e., data transmission and transfer. The parsed data records may be received and transmitted in the form of message queues. In other words, the data record parsed and output by the asynchronous task can be used as a producer of the message queue, and data is input at one end of the message queue.

Before the data records are analyzed, the data are stored in different table files, and after the data records are analyzed, all the data in all the table files can be stored in a message queue in a data record form, so that a plurality of table files can be mixed to be processed uniformly. The data records uniformly stored in the message queue can be distinguished through the identification information, so that the execution result of each table file can be conveniently determined, the verification result of the data records can be conveniently tracked, and the data stream can be traced. Thus, the identification information can be used to identify the table file to which the data record belongs and the data content in the corresponding table file. The identification information of the data record may be determined by a first identification of the asynchronous task and a second identification contained in the table file. Since each form file is processed by an asynchronous task, the first identification of the asynchronous task may identify the corresponding form file. And the second identifier may refer to an identifier number, such as a row number, a column number, etc., of each piece of data in the table file. A piece of financial data represented by the data record may be uniquely determined by the first identifier and the second identifier.

In addition, when the form file is read, the template to which the form file belongs can be checked, whether the form file is standard or not is judged, if the form file cannot be matched with the corresponding template, the form file can be determined to be not in accordance with the requirement, and therefore a message is sent to the node uploading the file, the uploaded data is reminded of being wrong, and the node uploads the file again.

In step S230, the data record is read from the message queue, and the data record is checked according to the file type of the data record.

In this example embodiment, after the data producer enters a data record into the message queue, the data record may be checked by creating a check task as a consumer of the message queue, and taking the data record out of the message queue. Also, to facilitate verification, the data records may be converted into entity data. The verification task can also be an asynchronous task, so that data is asynchronously processed from a file analysis layer and a file verification layer, smooth operation of the database is guaranteed under the condition of very large data volume, and usability and stability are further improved.

The verifying the data record may include performing anti-replay verification, service logic verification, data validity verification, and the like on the data record; checking other rules for the data record may also be included, such as non-null checking, deadline checking, etc., and the present example embodiment is not limited thereto. After each data record is verified, the processing progress of the table file to which the data record belongs can be determined according to the identification information of the data record, as shown in fig. 3, which specifically includes the following steps:

s301, determining the verification progress of the form file according to the identification information of the data record;

s302, if the verification progress is verification completion, determining a verification result of the form file.

Judging the table file to which the data record belongs according to the first identifier of the data record, counting the data records belonging to the same table file, if the counting result is equal to the total row number of the table file, determining that the verification progress is verified, and if the counting result is smaller than the total row number of the table file, determining that the verification progress is verified. If the verification progress of the table file is that verification is completed, all data contained in the table file are verified, and the verification result of the table file can be summarized according to the verification result of each data record in the table file, for example, m records pass verification and n records fail verification. Moreover, the check result may be embodied in a certain row in the table file, for example, the mth row data fails the check, etc.

And, according to the result of the verification, the data records can be stored separately, the data records which pass the verification are stored in the first data table, and the data records which do not pass the verification are stored in the second data table. The data records in the second data table which fail to pass the verification can be verified again for multiple times, for example, 3 times, 5 times, 6 times and the like, and the data records which still fail to pass the multiple times of verification can be determined as abnormal data, so that the abnormal data in the table file is output in the verification result.

With continued reference to FIG. 2, in step S240, the verified data records are determined to be importable data for the financial system to import into the database.

The verified data record may be referred to as the importable data. After each data record is verified, the verified data records can be stored in a data table, and the data table can be used as an intermediate table. And when the approvers approve, the imported data records are output from the intermediate table and are displayed to the approvers, and the data can be persisted to the corresponding databases of the financial systems after approval. And modifying the data records which do not pass the verification to regenerate the table file so as to parse and verify the part of data again.

For example, an operator of the financial system may upload a form file a that needs to be subjected to operations such as settlement, reconciliation, entry and the like to the distributed file system for storage, and further, when the asynchronous task b reads the form file a, the form file a is analyzed into a plurality of data records and stored in a message queue to be checked, and the message queue may further include message records corresponding to other form files. The message record that passes the verification may be determined to be importable data stored in the intermediate table. According to the asynchronous task b for processing the table file a, inquiring the message record which is analyzed by the b from the introducible data, outputting the inquired data to the financial system in a table form, displaying the inquired data to an operator, and directly importing the table into a database of the financial system if the examination and approval is passed.

In an optional embodiment, the importable data can be queried according to the data authority corresponding to the financial system, a target table is obtained, and the target table is output to the financial system for the financial system to import into the database. The data authority corresponding to the financial system can represent a table file corresponding to the financial system. The uploading interface can allow a plurality of systems to access when the form file is uploaded, so the form file stored in the distributed file system can be uploaded by the plurality of systems, and the data authority of each system is the form file uploaded by the system, namely, each financial system only has the data authority on the data in the form file uploaded by the financial system. Therefore, the data in the data authority is inquired from the importable data to generate a target table, and the target table is provided to the corresponding financial system for the user of the system to examine or view. In addition, the data permissions can be distinguished more finely, so that files uploaded in different batches of the same system have different data permissions, and related personnel can only examine and approve the files in a certain batch. For example, the table file may be identified by a batch number to determine the data rights. However, the data rights may also be divided in other ways, such as by logging into an account of the upload interface, etc.

Fig. 4 shows a process flow of the present exemplary embodiment. As shown in fig. 4, the external data file can be stored in a file storage system, and the file can be read from the file storage system and parsed into data records by a plurality of asynchronous tasks. The data records obtained by analysis can be uniformly put into a message queue, and the data records can also be classified and put into different message queues according to the actual verification requirement. Because the processing of the message queue and the processing of the file analysis are asynchronous, the embodiment can greatly reduce the pressure of the database and improve the processing speed. In addition, the speed and the quantity of data circulation can be controlled in the processing process, and the resource utilization rate of the server can be effectively improved. When data increase or processing needs increase, the horizontal expansion is convenient, and the capacity expansion can be linearly carried out as required, so that the performance is improved.

In an exemplary embodiment, as shown in fig. 5, the message production process may specifically include the following steps:

s501, analyzing the file by easy excel;

s502, determining the batch number, the line number and the file type of each data record;

s503, sending the data records to a message queue;

s504, resetting the analysis task when the data record analysis fails.

After the table file is uploaded to the file storage system, the status of the table file that is successfully uploaded may be a predetermined value, such as 0, 1, etc. The status of the form file may be a field identifying different processes of the form file, for example, the value of the status field of the successfully uploaded form may be set to 0. Therefore, when a file is analyzed by the easy excel framework, a table with a value of status of 0 can be searched, and if it exists, the file can be analyzed. The content contained in the file is analyzed into a piece of data record, and the batch number, the row number and the file type of each data record are determined, so that the data circulation can be traced back, and the data verification can be determined. And then sending the data record to a message queue to finish the production of the message. After the parsing is completed, the value of the status field of the file may be updated, for example, the status of the file that is successfully parsed is set to 1, and the status of the file that fails to be parsed is reset to 0, so as to trigger a subsequent verification process for the file that is successfully parsed, and to perform file re-acquisition or file re-parsing on the file that fails to be parsed.

In an exemplary embodiment, the consumption process of data is shown in fig. 6, and may specifically include the following steps:

s601, analyzing the data records into entity data;

s602, checking entity data;

s602, marking the checking result.

The data records may be converted into entity data or the entity data may be converted into data records using a message converter. The checking of the entity data may include multiple checks such as duplicate prevention, white list, non-null, type, etc., and may also include service check, that is, service logic of the data. After the check, the check result of each entity data is marked to determine whether the check passes, for example, status of data record passing the check is marked as 111, and status of data record failing the check is marked as 20. Furthermore, the result of verification may be marked to identify the reason of data record that fails verification, for example, status 30 indicates non-empty verification failure, status 40 indicates failure to prevent re-verification, and the like.

In an exemplary embodiment, the processing progress of the table file in step S210 may be counted in different processing procedures, as shown in fig. 7, specifically including the following steps:

s701, recording the number of successful lines and the number of failed lines of data processing;

s702, if the sum of the success line number and the failure line number is equal to the total line number of the table file, determining that the task execution is finished;

s703, generating an execution record when the task execution is finished;

and S704, generating a success file and a failure file for the user to download.

The data processing may include a process of parsing the table file into data records, or a process of verifying the data records in the message queue. In other words, the data processing may include a data production process and may also include a data consumption process. From the process of parsing the table file into data records, the number of lines of data records successfully parsed and the number of lines of failed parsing may be counted, and if the number of successful lines and the number of failed lines are equal to the total number of lines of the table file, it is determined that the execution of the parsing task with the table file is finished. Similarly, when it is determined that the execution of the verification task is finished, the execution of the data processing procedure for the table file is finished, and at this time, an execution record may be generated. The execution record may include the number of lines of generation execution and the number of lines of failure, and may also include the failure reason for the failure data. Further, a success file can be generated for the successfully executed data, and a failure file can be generated for the failed data for the user to download. For successful files, the user can directly import the files into the database for storage after approval, and for failed files, the files can be modified so as to be reprocessed.

As shown in fig. 8, the data processing procedure in the exemplary embodiment may include steps S801 to S809.

In step S801, the user may download a pre-provided template, and the data to be processed may be divided into multiple types, such as a general template and a service template, through different templates. After the user acquires the template, the data can be filled according to the template, so that the file to be processed is obtained. In step S802, a file to be processed may be acquired through the import portal. The file to be processed can be imported into the file storage system through the import port, and then the analysis task can read the file to be processed. In step S803, the parsing task may parse the excel file, and send the parsed data record to the message queue.

In step S804, the verification task reads the data record from the message queue, converts the data record into entity data, and performs verification. Specifically, the read data records may be subjected to a re-check, and then whether the data is white list data may be checked through a white list data list. Moreover, various service checks can be performed on the white list data, for example, whether the check date meets the requirements, whether the check data meets the preset file type, and the like. The entity data passing the verification is recorded in the intermediate table at step S805, and the result of the verification may be recorded. Thus, in step S806, the error data queried from the intermediate table is repeatedly checked, if the repeated check passes, the data is updated to be correct data in the intermediate table, and then the repeated check is performed on other error data that still does not pass the check for many times until no error data exists in the intermediate table or the number of times of the repeated check on the error data reaches the threshold.

In the data analysis and verification process, the number of the processed data records can be counted through the number scanning task, and the processing progress of the file to be processed is counted. Further, in step S807, if all the data in the file to be processed is processed, the file to be processed is processed to be ended, and a log file may be generated. The recording file can record correct data, error data and error reasons of the error data in the file to be processed. In step S808, the user checks the record file, and if the check is passed, the data required by the user is sent out according to the record file, so that the user can obtain and store the data in the database.

Further, the present exemplary embodiment also provides a data processing apparatus, which can be used to execute the above-mentioned data processing method of the present invention.

As shown in fig. 9, the data processing apparatus 900 may include a data acquisition unit 910, a data parsing unit 920, a data verification unit 930, and a data storage unit 940. Specifically, the data acquiring unit 910 is configured to acquire a table file containing financial data; a data parsing unit 920, configured to read the table file through an asynchronous task, parse the table file into data records, and obtain a message queue; a data checking unit 930, configured to read the data record from the message queue, and check the data record according to a file type of the data record; and the data storage unit is used for storing the data records passing the verification to obtain importable data for the financial system to import into the database.

In an exemplary embodiment, the data obtaining unit 910 may be specifically configured to store the table file through a distributed file storage system.

In an exemplary embodiment, the apparatus further includes an identification unit, which may be configured to determine identification information of each of the data records to determine a verification result of each of the data records.

In an exemplary embodiment, the identification unit may be specifically configured to determine the identification information of each data record by the first identification of the asynchronous task and the second identification of the financial data contained in the form file.

In an exemplary embodiment, the apparatus further includes a progress determination unit and a result output unit. The progress determining unit may be configured to determine a verification progress of the form file according to the identification information of the data record; the result output unit may be configured to determine a verification result of the form file if the verification progress is verification completion.

In an exemplary embodiment, the apparatus further includes an exception retry unit, which may be configured to verify the data record that fails the verification again according to a verification result of the data record to determine the abnormal data in the table file.

In an exemplary embodiment, the data storage unit 940 further includes an authority control unit, and the authority control unit may be configured to query the importable data according to the data authority of the financial system, and obtain a target table for the financial system to import into the database.

For details which are not disclosed in the embodiments of the apparatus of the present invention, reference is made to the above-described embodiments of the data processing method of the present invention for the respective functional modules of the data processing apparatus of the exemplary embodiment of the present invention correspond to the steps of the above-described exemplary embodiment of the data processing method.

Referring now to FIG. 10, shown is a block diagram of a computer system 1000 suitable for use with the electronic device implementing an embodiment of the present invention. The computer system 1000 of the electronic device shown in fig. 10 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.

As shown in fig. 10, the computer system 1000 includes a Central Processing Unit (CPU)1001 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)1002 or a program loaded from a storage section 1008 into a Random Access Memory (RAM) 1003. In the RAM 1003, various programs and data necessary for system operation are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.

The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 1008 including a hard disk and the like; and a communication section 1009 including a network interface card such as a LAN card, a modem, or the like. The communication section 1009 performs communication processing via a network such as the internet. The driver 1010 is also connected to the I/O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1010 as necessary, so that a computer program read out therefrom is mounted into the storage section 1008 as necessary.

In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the invention include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication part 1009 and/or installed from the removable medium 1011. The computer program executes the above-described functions defined in the system of the present application when executed by the Central Processing Unit (CPU) 1001.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present invention may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the data processing method as described in the above embodiments.

For example, the electronic device may implement the following as shown in fig. 2: step S210, acquiring a table file containing financial data; step S220, reading the table file through an asynchronous task, analyzing the table file into data records, and obtaining a message queue; step S230, reading the data record from the message queue, and verifying the data record according to the file type of the data record; and step S240, determining the data record passing the verification as importable data for the financial system to import into the database.

As another example, the electronic device may implement the steps shown in fig. 3.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the invention. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, comprising:

acquiring a form file containing financial data;

reading the table file through an asynchronous task, analyzing the table file into data records, and obtaining a message queue;

reading the data record from the message queue, and checking the data record according to the file type of the data record;

and determining the data records passing the verification as importable data for the financial system to import into the database.

2. The method of claim 1, wherein obtaining a form file containing financial data comprises:

and storing the table file through a distributed file storage system.

3. The method of claim 1, wherein after parsing the tabular file into data records, further comprising:

and determining the identification information of each data record so as to determine the verification result of each data record.

4. The method of claim 3, wherein said determining identification information for each of said data records comprises:

and determining the identification information of each data record through the first identification of the asynchronous task and the second identification of the financial data contained in the table file.

5. The method of claim 3, wherein after verifying the data record according to the file type of the data record, further comprising:

determining the verification progress of the form file according to the identification information of the data record;

and if the verification progress is verification completion, determining a verification result of the form file.

6. The method of claim 3, wherein after determining the verification result for each of the data records, further comprising:

and verifying the data records which fail to be verified again according to the verification result of the data records so as to determine the abnormal data in the table file.

7. The method of claim 1, wherein importing, by the financial system, the database comprises:

and inquiring the importable data according to the data authority of the financial system to obtain a target table for the financial system to import into a database.

8. A data processing apparatus, comprising:

the data acquisition unit is used for acquiring a table file containing financial data;

the data analysis unit is used for reading the table file through an asynchronous task, analyzing the table file into data records and acquiring a message queue;

the data checking unit is used for reading the data records from the message queue and checking the data records according to the file types of the data records;

and the data storage unit is used for storing the data records passing the verification to obtain importable data for the financial system to import into the database.

9. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.

10. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out a data processing method as claimed in any one of claims 1 to 7.