CN112433994A - File processing method - Google Patents
File processing method Download PDFInfo
- Publication number
- CN112433994A CN112433994A CN202011330870.7A CN202011330870A CN112433994A CN 112433994 A CN112433994 A CN 112433994A CN 202011330870 A CN202011330870 A CN 202011330870A CN 112433994 A CN112433994 A CN 112433994A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- processing
- computer
- small blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1737—Details of further file system functions for reducing power consumption or coping with limited storage space, e.g. in mobile devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/186—Templates
Abstract
The invention discloses a file processing method, which relates to the technical field of computers, and is characterized in that a file processing request sent by a client is received, a corresponding file is obtained according to the file processing request, the file is analyzed into corresponding file data according to a file template, so that the data which is persistent in the file is analyzed into data which can be directly processed by a program, the file data is written into a database and is divided into a plurality of small blocks of data based on a set segmentation rule, one block of data is selected from the plurality of small blocks of data for data processing until the plurality of small blocks of data are processed, the success rate and the efficiency of file data analysis are improved, the existing resources are not occupied, and the execution efficiency of a bank system is improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a file processing method.
Background
Currently, most of the core systems of banks, such as transaction systems, adopt a real-time processing mode, for example: the method comprises the following steps of (1) under the scenes of salary generation, batch account opening and the like, wherein the execution process comprises file analysis and service processing, and the following defects exist in the real-time processing:
firstly, files are processed in real time, the processing time of response to counter personnel is long, and the counter needs to wait for a long time, so that the efficiency is influenced; secondly, for the real-time processing of large files, the processing time is multiplied along with the size of the files; thirdly, the real-time processing of the files occupies the resources of the existing system and affects the normal use of other services.
In summary, today when hardware resources are used to the utmost, the existing execution process cannot meet the increasing business requirements, and finally becomes a bottleneck of occupying system resources, so how to improve the efficiency of file processing without occupying existing resources is a problem that needs to be solved urgently at present.
Disclosure of Invention
In order to solve the defects of the prior art, an embodiment of the present invention provides a file processing method, including the following steps:
s1, receiving a file processing request sent by a client and acquiring a corresponding file according to the file processing request;
s2, analyzing the file into corresponding file data according to the file template;
s3, writing the file data into a database and dividing the file data into a plurality of small blocks of data based on the set segmentation rule;
s4, selecting a piece of data from the small pieces of data to perform data processing;
s5 repeats step S4 until the small blocks are processed.
Preferably, the method further comprises:
and updating the state of the file in real time according to the processing progress of the file data.
Preferably, after step S5, the method further comprises:
and generating a processing result file and a processing detail file of the file.
The file processing method provided by the embodiment of the invention has the following beneficial effects:
(1) the data are put in storage and then processed, so that the success rate of file data analysis can be greatly improved;
(2) the large data is divided into a plurality of small data blocks, and a small data block is read each time to provide business processing, so that occupation of system resources caused by loading a large amount of data into a system memory is avoided.
Detailed Description
The present invention will be described in detail with reference to the following embodiments.
The file processing method provided by the embodiment of the invention comprises the following steps:
s101, receiving a file processing request sent by a client and acquiring a corresponding file according to the file processing request.
S102, analyzing the file into corresponding file data according to the file template so as to analyze the data which is persistent by the file into data which can be directly processed by a program.
S103, writing the file data into a database and dividing the file data into a plurality of small blocks of data based on the set segmentation rule.
As a specific embodiment of the present invention, processing 100 ten thousand pieces of data, segmenting (for example, representing data) according to the primary key of the data, wherein the size of each piece of data is 10000, then dividing 100W of data into one hundred pieces, each piece of data having a size of 1 ten thousand, and starting each piece as follows: 1-10000, 10001-20000, 20001-30000 ….
And S104, selecting one block of data from the plurality of small block data for data processing.
S105 repeats step S104 until the plurality of small blocks are processed.
Optionally, the method further comprises:
and updating the state of the file in real time according to the processing progress of the file data.
As a specific embodiment, a request for processing a file sent by a client will register a record for processing the current file in a database, and the state is to be processed; when the asynchronous processing file is started, the modification state is in the process, and the updating state is the final state after the asynchronous processing is finished. The purpose of this is to prevent the concurrent repeated execution of the document and to track the cycle of the entire document processing.
Optionally, after step S105, the method further comprises:
and generating a processing result file and a processing detail file of the file.
According to the file processing method provided by the embodiment of the invention, the file processing request sent by the client is received, the corresponding file is obtained according to the file processing request, the file is analyzed into the corresponding file data according to the file template, the data which is persistent in the file is analyzed into the data which can be directly processed by the program, the file data is written into the database and is divided into a plurality of small blocks of data based on the set segmentation rule, one block of data is selected from the plurality of small blocks of data for data processing until the plurality of small blocks of data are processed, the success rate and the efficiency of file data analysis are improved, the existing resources are not occupied, and the execution efficiency of a bank system is improved.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It will be appreciated that the relevant features of the method and apparatus described above are referred to one another.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (5)
1. A file processing method, comprising:
s1, receiving a file processing request sent by a client and acquiring a corresponding file according to the file processing request;
s2, analyzing the file into corresponding file data according to the file template;
s3, writing the file data into a database and dividing the file data into a plurality of small blocks of data based on the set segmentation rule;
s4, selecting a piece of data from the small pieces of data to perform data processing;
s5 repeats step S4 until the small blocks are processed.
2. The document processing method according to claim 1, characterized in that the method further comprises:
and updating the state of the file in real time according to the processing progress of the file data.
3. The document processing method according to claim 1, wherein after step S5, the method further comprises:
and generating a processing result file and a processing detail file of the file.
4. A computer program product, characterized in that the computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the method of claims 1-2.
5. A non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the method of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011330870.7A CN112433994A (en) | 2020-11-24 | 2020-11-24 | File processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011330870.7A CN112433994A (en) | 2020-11-24 | 2020-11-24 | File processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112433994A true CN112433994A (en) | 2021-03-02 |
Family
ID=74692939
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011330870.7A Pending CN112433994A (en) | 2020-11-24 | 2020-11-24 | File processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112433994A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105007322A (en) * | 2015-07-21 | 2015-10-28 | 陈丹 | File transmission method and system |
| US20160124857A1 (en) * | 2014-11-05 | 2016-05-05 | International Business Machines Corporation | Access accelerator for active hbase database regions |
| CN107797854A (en) * | 2017-06-26 | 2018-03-13 | 平安科技（深圳）有限公司 | Transaction file processing method, device, storage medium and computer equipment |
| CN109492034A (en) * | 2018-09-17 | 2019-03-19 | 中国建设银行股份有限公司 | Batch processing system, method, apparatus and storage medium |
| CN109525632A (en) * | 2018-09-30 | 2019-03-26 | 珠海派诺科技股份有限公司 | Gateway data uploads database connection digital control method, device, equipment and medium |
| CN110825598A (en) * | 2019-09-23 | 2020-02-21 | 武汉智美互联科技有限公司 | Log real-time processing method and system |
| CN110995830A (en) * | 2019-11-29 | 2020-04-10 | 武汉卓讯互动信息科技有限公司 | Network resource processing method and device |
-
2020
- 2020-11-24 CN CN202011330870.7A patent/CN112433994A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160124857A1 (en) * | 2014-11-05 | 2016-05-05 | International Business Machines Corporation | Access accelerator for active hbase database regions |
| CN105007322A (en) * | 2015-07-21 | 2015-10-28 | 陈丹 | File transmission method and system |
| CN107797854A (en) * | 2017-06-26 | 2018-03-13 | 平安科技（深圳）有限公司 | Transaction file processing method, device, storage medium and computer equipment |
| CN109492034A (en) * | 2018-09-17 | 2019-03-19 | 中国建设银行股份有限公司 | Batch processing system, method, apparatus and storage medium |
| CN109525632A (en) * | 2018-09-30 | 2019-03-26 | 珠海派诺科技股份有限公司 | Gateway data uploads database connection digital control method, device, equipment and medium |
| CN110825598A (en) * | 2019-09-23 | 2020-02-21 | 武汉智美互联科技有限公司 | Log real-time processing method and system |
| CN110995830A (en) * | 2019-11-29 | 2020-04-10 | 武汉卓讯互动信息科技有限公司 | Network resource processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101860449B (en) | Data query method, device and system | |
| CN109034993A (en) | Account checking method, equipment, system and computer readable storage medium | |
| CN107608773B (en) | Task concurrent processing method and device and computing equipment | |
| CN107870928A (en) | File reading and device | |
| CN114116665B (en) | Method for writing transaction log in parallel in database to promote processing efficiency | |
| CN110413631A (en) | A kind of data query method and device | |
| CN113918532A (en) | Portrait label aggregation method, electronic device and storage medium | |
| CN106855858A (en) | Database operation method and device | |
| CN112750027A (en) | Batch business processing method and device, computer equipment and storage medium | |
| CN112433994A (en) | File processing method | |
| CN112015819A (en) | Data updating method, device, equipment and medium for distributed graph database | |
| CN111125168A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114138424B (en) | Virtual machine memory snapshot generation method and device and electronic equipment | |
| CN113934727B (en) | Adaptive acquisition and processing system and method for multi-source heterogeneous financial data | |
| CN111737090B (en) | Log simulation method and device, computer equipment and storage medium | |
| CN112380348B (en) | Metadata processing method, apparatus, electronic device and computer readable storage medium | |
| CN111459474B (en) | Templated data processing method and device | |
| DE102021129862A1 (en) | Method and device for identifying hardware performance count events for detecting and classifying malware or workload using artificial intelligence | |
| CN113468105A (en) | Data structure of data snapshot, related data processing method, device and system | |
| CN110750569A (en) | Data extraction method, device, equipment and storage medium | |
| CN115599838B (en) | Data processing method, device, equipment and storage medium based on artificial intelligence | |
| CN113205421A (en) | Accounting method and device for financial products | |
| CN109542900B (en) | Data processing method and device | |
| CN111209292A (en) | Bin data management method, device and system | |
| CN106940698A (en) | A kind of dimension data processing method and processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |