CN111680011A - File processing method and device based on host system - Google Patents
File processing method and device based on host system Download PDFInfo
- Publication number
- CN111680011A CN111680011A CN202010494390.8A CN202010494390A CN111680011A CN 111680011 A CN111680011 A CN 111680011A CN 202010494390 A CN202010494390 A CN 202010494390A CN 111680011 A CN111680011 A CN 111680011A
- Authority
- CN
- China
- Prior art keywords
- file
- processing
- files
- splitting
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 23
- 238000004590 computer program Methods 0.000 claims description 16
- 238000010586 diagram Methods 0.000 description 12
- 230000008859 change Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000003203 everyday effect Effects 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 2
- 230000002354 daily effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a file processing method and a device based on a host system, wherein the method comprises the following steps: splitting a file to be processed into a processing file on the same day and a temporary file; configuring splitting parameters according to the total data amount in the file to be processed; carrying out average splitting processing on the files processed on the same day according to the splitting parameters to generate a plurality of first operating files with the same data volume, and processing the plurality of first operating files by utilizing a plurality of first preset programs; and according to the splitting parameters, extracting data with preset data volume from the temporary file on the next day to carry out average splitting processing, generating a plurality of second operation files with the same data volume, and processing the plurality of second operation files by utilizing a plurality of second preset programs. The invention can reduce the workload of workers and improve the working efficiency.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a file processing method and device based on a host system.
Background
The host system is a core platform for bearing business data processing in a large-scale key system, taking a QSAM file for storing sequential data on a Z-series server of a bank system as an example, when business data change is needed, many limitations are often caused, for example, the change data volume which can be born by a background every day is limited, the host system does not allow the QSAM file to be processed at one time, and the QSAM file needs to be processed for multiple days after being split.
In the prior art, generally, operation and maintenance personnel manually calculate and configure file splitting parameters according to actual data volume, the operation flow is complicated and complex, and errors are easy to occur. Moreover, because the operation names which are operated every day are different, the operation and maintenance personnel can operate the file splitting process at present by submitting change applications every day, so that the change procedures are more, the workload of the workers is increased, and the operation efficiency is lower.
Disclosure of Invention
The embodiment of the invention provides a file processing method based on a host system, which is used for reducing the workload of workers and improving the working efficiency and comprises the following steps:
splitting a file to be processed into a processing file on the same day and a temporary file;
configuring splitting parameters according to the total data amount in the file to be processed;
carrying out average splitting processing on the files processed on the same day according to the splitting parameters to generate a plurality of first operating files with the same data volume, and processing the plurality of first operating files by utilizing a plurality of first preset programs;
and according to the splitting parameters, extracting data with preset data volume from the temporary file on the next day to carry out average splitting processing, generating a plurality of second operation files with the same data volume, and processing the plurality of second operation files by utilizing a plurality of second preset programs.
Optionally, the method further includes:
and judging whether the temporary file has contents or not, and if not, giving an alarm to the user.
Optionally, the splitting parameter includes: parallelism and the data amount processed by a single file, wherein the parallelism is used for representing the corresponding processing relation between the file and the program.
Optionally, the plurality of first preset programs and the plurality of first operating files are in a one-to-one correspondence relationship;
the second preset programs and the second running files are in one-to-one correspondence.
The embodiment of the invention also provides a file processing device based on the host system, which is used for reducing the workload of workers and improving the working efficiency, and the device comprises:
the file splitting module is used for splitting the file to be processed into a processing file on the same day and a temporary file;
the parameter configuration module is used for configuring splitting parameters according to the total data amount in the file to be processed;
the first file processing module is used for carrying out average splitting processing on the files processed on the same day according to the splitting parameters, generating a plurality of first operating files with the same data volume and processing the plurality of first operating files by utilizing a plurality of first preset programs;
and the second file processing module is used for extracting data with preset data volume from the temporary file for average splitting processing on the next day according to the splitting parameters, generating a plurality of second running files with the same data volume, and processing the plurality of second running files by utilizing a plurality of second preset programs.
Optionally, the apparatus further comprises:
and the warning module is used for judging whether the temporary file has content or not, and if not, warning the user.
Optionally, the splitting parameter includes: parallelism and the data amount processed by a single file, wherein the parallelism is used for representing the corresponding processing relation between the file and the program.
Optionally, the plurality of first preset programs and the plurality of first operating files are in a one-to-one correspondence relationship;
the second preset programs and the second running files are in one-to-one correspondence.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method when executing the computer program.
An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program for executing the above method is stored.
In the embodiment of the invention, the file to be processed is divided into the file to be processed on the same day and the temporary file, the dividing parameter is configured according to the total amount of data in the file to be processed, the file to be processed on the same day is averagely divided according to the dividing parameter, a plurality of first operating files with the same data volume are generated, the plurality of first operating files are processed by a plurality of first preset programs, then the data with the preset data volume is extracted from the temporary file on the next day according to the dividing parameter for averagely dividing, a plurality of second operating files with the same data volume are generated, the plurality of second operating files are processed by a plurality of second preset programs, manual intervention is not needed, the operation flow is simple, errors are not easy to occur, change applications are not needed during daily operation, the workload of workers is reduced, and the operation efficiency is higher.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
FIG. 1 is a flowchart of a method for processing files based on a host system according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for processing files based on a host system according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a structure of a file processing apparatus based on a host system according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another structure of a file processing apparatus based on a host system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are used in an open-ended fashion, i.e., to mean including, but not limited to. Reference to the description of the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the embodiments is for illustrative purposes to illustrate the implementation of the present application, and the sequence of steps is not limited and can be adjusted as needed.
Fig. 1 is a flowchart of a file processing method based on a host system according to an embodiment of the present invention, and as shown in fig. 1, the method includes:
And 102, configuring splitting parameters according to the total data amount in the file to be processed.
In this embodiment, the splitting parameter includes: parallelism, which is used to represent the corresponding processing relationship of a file and a program, and the amount of data processed by a single file.
For the parallelism, specifically, for example, the plurality of first preset programs and the plurality of first run files are in a one-to-one correspondence relationship; the second preset programs and the second running files are in one-to-one correspondence.
In the specific implementation, in the file to be processed, data except the file processed on the current day is recorded in the temporary file.
And 104, according to the splitting parameters, extracting data with preset data volume from the temporary file for average splitting processing on the next day, generating a plurality of second operation files with the same data volume, and processing the plurality of second operation files by using a plurality of second preset programs.
In this embodiment, the file names of the first running file and the second running file may be set to be the same, so as to ensure that the operations for processing the files every day are the same, which not only can improve the job efficiency, but also can avoid errors occurring in the job process.
The invention is illustrated below in a specific embodiment:
and splitting the file to be processed into a processing file on the current day and a temporary file.
And configuring splitting parameters according to the total data amount in the file to be processed.
The first day: and submitting a splitting job, splitting the current-day processing file into a plurality of files with equal data volumes according to splitting parameters, wherein the text names are respectively F1, F2, F3, F4, … … and the like. Fixed sets of jobs are run, e.g., runs J1, J2, J3, processing files F1, F2, F3, respectively. (the sum of the data volumes of F1, F2, F3 files satisfies the date-of-day change data volume limit).
The second day: the next set of jobs, e.g., J4, J5, J6, is run, processing files F4, F5, F6, respectively. (the sum of the data volumes of F4, F5, F6 files satisfies the date-of-day change data volume limit).
The next set of jobs, e.g., J7, J8, J9, is run, processing files F7, F8, F9, respectively. (the sum of the data volumes of F7, F8, F9 files satisfies the date-of-day change data volume limit).
And repeating the steps until all the split files are processed.
As can be seen from fig. 1, a file to be processed is split into a processing file of the current day and a temporary file, splitting parameters are configured according to the total amount of data in the file to be processed, the processing file of the current day is subjected to average splitting processing according to the splitting parameters, a plurality of first operating files with the same data volume are generated, the plurality of first operating files are processed by using a plurality of first preset programs, then data with preset data volume is extracted from the temporary file for average splitting processing according to the splitting parameters, a plurality of second operating files with the same data volume are generated, and the plurality of second operating files are processed by using a plurality of second preset programs.
Fig. 2 is a flowchart of a file processing method based on a host system according to an embodiment of the present invention, in order to remind a user after a job is finished and help a worker to recognize that a file has been processed for a subsequent job, as shown in fig. 2, the method further includes: step 201, judging whether the temporary file has content, and if not, giving an alarm to the user.
Based on the same inventive concept, the embodiment of the present invention further provides a file processing apparatus based on a host system, as described in the following embodiments. Because the principle of solving the problem of the file processing device based on the host system is similar to that of the file processing method based on the host system, the implementation of the file processing device based on the host system can refer to the implementation of the file processing method based on the host system, and repeated details are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 3 is a schematic structural diagram of a file processing apparatus based on a host system according to an embodiment of the present invention, as shown in fig. 3, the apparatus includes:
the file splitting module 301 is configured to split the file to be processed into a processing file on the same day and a temporary file.
A parameter configuration module 302, configured to configure a splitting parameter according to a total amount of data in the file to be processed.
The first file processing module 303 is configured to perform average splitting processing on the files processed on the same day according to the splitting parameters, generate a plurality of first operation files with the same data volume, and process the plurality of first operation files by using a plurality of first preset programs.
The second file processing module 304 is configured to, according to the splitting parameter, extract data with a preset data size from the temporary file on the next day to perform average splitting processing, generate a plurality of second operation files with the same data size, and process the plurality of second operation files by using a plurality of second preset programs.
Fig. 4 is a schematic structural diagram of a file processing apparatus based on a host system according to an embodiment of the present invention, and as shown in fig. 4, the apparatus further includes:
and the warning module 401 is configured to determine whether there is any content in the temporary file, and if not, warn the user.
In an embodiment of the present invention, the splitting parameter includes: parallelism and the data amount processed by a single file, wherein the parallelism is used for representing the corresponding processing relation between the file and the program.
In the embodiment of the invention, a plurality of first preset programs and a plurality of first running files are in one-to-one correspondence;
the second preset programs and the second running files are in one-to-one correspondence.
The embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above method when executing the computer program.
An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program for executing the above method is stored.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A file processing method based on a host system is characterized by comprising the following steps:
splitting a file to be processed into a processing file on the same day and a temporary file;
configuring splitting parameters according to the total data amount in the file to be processed;
carrying out average splitting processing on the files processed on the same day according to the splitting parameters to generate a plurality of first operating files with the same data volume, and processing the plurality of first operating files by utilizing a plurality of first preset programs;
and according to the splitting parameters, extracting data with preset data volume from the temporary file on the next day to carry out average splitting processing, generating a plurality of second operation files with the same data volume, and processing the plurality of second operation files by utilizing a plurality of second preset programs.
2. The method of claim 1, further comprising:
and judging whether the temporary file has contents or not, and if not, giving an alarm to the user.
3. The method of claim 1, wherein the splitting parameters comprise: parallelism and the data amount processed by a single file, wherein the parallelism is used for representing the corresponding processing relation between the file and the program.
4. The method of claim 1, wherein the plurality of first preset programs and the plurality of first operating files are in a one-to-one correspondence relationship;
the second preset programs and the second running files are in one-to-one correspondence.
5. A host system-based file processing apparatus, comprising:
the file splitting module is used for splitting the file to be processed into a processing file on the same day and a temporary file;
the parameter configuration module is used for configuring splitting parameters according to the total data amount in the file to be processed;
the first file processing module is used for carrying out average splitting processing on the files processed on the same day according to the splitting parameters, generating a plurality of first operating files with the same data volume and processing the plurality of first operating files by utilizing a plurality of first preset programs;
and the second file processing module is used for extracting data with preset data volume from the temporary file for average splitting processing on the next day according to the splitting parameters, generating a plurality of second running files with the same data volume, and processing the plurality of second running files by utilizing a plurality of second preset programs.
6. The apparatus of claim 5, further comprising:
and the warning module is used for judging whether the temporary file has content or not, and if not, warning the user.
7. The apparatus of claim 5, wherein the split parameters comprise: parallelism and the data amount processed by a single file, wherein the parallelism is used for representing the corresponding processing relation between the file and the program.
8. The apparatus of claim 5, wherein the plurality of first predetermined programs are in a one-to-one correspondence with the plurality of first run files;
the second preset programs and the second running files are in one-to-one correspondence.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010494390.8A CN111680011A (en) | 2020-06-03 | 2020-06-03 | File processing method and device based on host system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010494390.8A CN111680011A (en) | 2020-06-03 | 2020-06-03 | File processing method and device based on host system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111680011A true CN111680011A (en) | 2020-09-18 |
Family
ID=72434616
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010494390.8A Pending CN111680011A (en) | 2020-06-03 | 2020-06-03 | File processing method and device based on host system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111680011A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0553989A (en) * | 1991-08-29 | 1993-03-05 | Chugoku Nippon Denki Software Kk | Daily calculation processing system |
| CN101510203A (en) * | 2009-02-25 | 2009-08-19 | 南京联创科技股份有限公司 | Big data quantity high performance processing implementing method based on parallel process of split mechanism |
| CN106126487A (en) * | 2016-06-30 | 2016-11-16 | 天脉聚源(北京)传媒科技有限公司 | A kind of journal file method for splitting and device |
| US20180253478A1 (en) * | 2017-03-01 | 2018-09-06 | Next Pathway Inc. | Method and system for parallelization of ingestion of large data sets |
-
2020
- 2020-06-03 CN CN202010494390.8A patent/CN111680011A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0553989A (en) * | 1991-08-29 | 1993-03-05 | Chugoku Nippon Denki Software Kk | Daily calculation processing system |
| CN101510203A (en) * | 2009-02-25 | 2009-08-19 | 南京联创科技股份有限公司 | Big data quantity high performance processing implementing method based on parallel process of split mechanism |
| CN106126487A (en) * | 2016-06-30 | 2016-11-16 | 天脉聚源(北京)传媒科技有限公司 | A kind of journal file method for splitting and device |
| US20180253478A1 (en) * | 2017-03-01 | 2018-09-06 | Next Pathway Inc. | Method and system for parallelization of ingestion of large data sets |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107784026B (en) | ETL data processing method and device | |
| Diethelm | An efficient parallel algorithm for the numerical solution of fractional differential equations | |
| CN105022670A (en) | Heterogeneous distributed task processing system and processing method in cloud computing platform | |
| CN104317928A (en) | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database | |
| CN104537015A (en) | Log analysis computer implementation method, computer and system | |
| CN111767320B (en) | Data blood relationship determination method and device | |
| CN111124839A (en) | Distributed log data monitoring method and device | |
| CN110134646B (en) | Knowledge platform service data storage and integration method and system | |
| CN110555185A (en) | Page customization method and system based on PC client | |
| CN111680011A (en) | File processing method and device based on host system | |
| CN112037003A (en) | File account checking processing method and device | |
| CN106843822B (en) | Execution code generation method and equipment | |
| CN111708756A (en) | Method, device and equipment for automatically processing data warehouse and storage medium | |
| CN111274667A (en) | Cross-scale material calculation software integrated calculation system and method | |
| CN111104382A (en) | File default naming method and device and electronic equipment | |
| CN106547883B (en) | Method and system for processing User Defined Function (UDF) running condition | |
| CN114417828A (en) | Entity relationship extraction method and system for server alarm log description text | |
| CN114253914A (en) | Distributed data acquisition system and method | |
| CN113127359A (en) | Method and device for obtaining test data | |
| CN112306675A (en) | Data processing method, related device and computer readable storage medium | |
| CN112000414B (en) | Configurable display method and device for parameter information | |
| CN113434321B (en) | Method, device and equipment for processing deployment errors in operation and maintenance and storage medium | |
| CN117573199B (en) | Model difference comparison analysis method, device, equipment and medium | |
| CN106997365A (en) | A kind of data processing method and device across data source | |
| CN110932903B (en) | Method, equipment and storage medium for monitoring current online equipment in real time |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200918 |