CN116226141A - Lightweight multi-data source batch processing method - Google Patents
Lightweight multi-data source batch processing method Download PDFInfo
- Publication number
- CN116226141A CN116226141A CN202211531579.5A CN202211531579A CN116226141A CN 116226141 A CN116226141 A CN 116226141A CN 202211531579 A CN202211531579 A CN 202211531579A CN 116226141 A CN116226141 A CN 116226141A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- writing
- name
- batch processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
- G06F16/2386—Bulk updating operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a lightweight multi-data source batch processing method, which comprises the following steps: s001, acquiring data stored in a designated database, calling a matching object in the data according to requirements, and generating a plurality of time sets; s002, sorting is performed based on the generated plurality of time sets, the time sets of the first data are named first name, and the time sets of the last data are named lastname; s003, inserting the analyzed data batch into a target database, and sequentially cycling batch access writing from the beginning of the first name to the end of the lastname in a sequencing order. The lightweight multi-data source batch processing method provided by the invention realizes multi-channel batch data updating and writing, avoids the increase of the background load of the server where the platform is located due to integrated writing, can realize multichannel and timing data writing, and avoids platform blocking and collapse caused by overload of the server.
Description
Technical Field
The invention relates to the technical field of multi-data batch processing systems, in particular to a lightweight multi-data source batch processing method.
Background
The company is doing customized marketing, and integrates the last data stored in the platform by the user to perform data analysis. However, all the data are analyzed by users in recent years, and the data can exist in various storage media, so that the cost for fetching the data is high, and the batch processing is performed in the service system, so that the normal operation of the service system can be influenced.
Disclosure of Invention
The invention aims to provide a lightweight multi-data source batch processing method which is used for batch processing of multi-data updating and avoids the problems of stuck system operation overload or fault code frequency occurrence caused by the updating of the multi-data.
In order to achieve the above object, the present invention provides the following technical solutions: a method of lightweight multi-data source batch processing, comprising the steps of:
s001, acquiring data stored in a designated database, calling a matching object in the data according to requirements, and generating a plurality of time sets;
s002, sorting is performed based on the generated plurality of time sets, the time sets of the first data are named first name, and the time sets of the last data are named lastname;
s003, inserting the analyzed data batch into a target database, and sequentially cycling batch access writing from the beginning of the first name to the end of the lastname in a sequencing order.
Preferably, the specified database is cloud storage;
the time set is reserved data stored in the cloud storage during platform operation within one year.
Preferably, after the execution in S002 is completed, a first notification pop-up window is generated to notify the user of the data analysis status, and the first notification pop-up window includes at least one instruction button, where the instruction button is used to execute the next data processing on the completely analyzed data.
Preferably, the target database is a preset writing path of the analysis data.
Preferably, the time set data after finishing sorting in the step 2 generates an item matching instruction and transmits the item matching instruction to the step 3, and after executing the time set data access and writing in, the step 3 reads the number of chunk and then compares the number of chunk with the item matching instruction to judge whether the data is finished or not, if yes, a second popup window is generated for prompting; and if not, listing the names of the unwritten time set data.
A lightweight multi-data source batch processing system, a method for operating the lightweight multi-data source batch processing described in the above scheme, comprising:
the plug-in writing module is used for matching the completed Job code into the platform, executing and storing an operation record Job task through the JobLanucher, executing Step in the Job task according to a specified logic sequence, and storing the result of each Step of Step;
the database processing module analyzes target database data through codes, analyzes the target database data according to QueryProvider in the codes, and sorts the target database data based on the completed analysis data, wherein the first name is first name and the last name is lastname;
and the database writing module is used for writing the data obtained through analysis through the binding database.
Preferably, the database writing module includes a paging synchronization unit, sets a plurality of tasks based on the created job task to create a plurality of paging intervals, and writes a plurality of data in a sequence in a sorting cycle within the paging intervals.
Preferably, the database processing module further includes a plurality of execution subcodes under the mysql page query provider code field, each including:
method dataSource: binding a database;
method queryProvider: binding query information;
method rowMapper: converting the query result into an object;
method pageSize: the size of each page query is set.
In the technical scheme, the lightweight multi-data source batch processing method provided by the invention has the following beneficial effects: the method realizes the data updating and writing in batches in multiple channels, avoids the increase of the background burden of the server where the platform is located due to the integrated writing, can realize the data writing in multiple channels and at regular time, and avoids the problems of platform blocking and collapse caused by the overload of the server. And the dependence of big data departments is reduced, so that the business problem can be solved more sensitively.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.
FIG. 1 is a schematic diagram of a schematic structure provided in an embodiment of the present invention;
FIG. 2 is a schematic flow chart according to an embodiment of the present invention;
FIG. 3 is a diagram of reading database codes according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a data integration processing code according to an embodiment of the present invention;
FIG. 5 is a diagram of a written database code according to an embodiment of the present invention;
FIG. 6 is a code schematic diagram of a job binding step provided by an embodiment of the present invention;
FIG. 7 is a schematic diagram of an integrated dispatch center code according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a module structure according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
As shown in fig. 1, a method for lightweight multi-data source batch processing includes the following steps:
s001, acquiring data stored in a designated database, calling a matching object in the data according to requirements, and generating a plurality of time sets;
s002, executing sorting based on the generated multiple time sets, arranging in descending order of time sequence, naming the time set of the first data with first name, and naming the time set of the last data with lastname;
s003, inserting the analyzed data batch into a target database, and sequentially cycling batch access writing from the beginning of the first name to the end of the lastname in a sequencing order.
Specifically, in the above embodiment, the database is designated as cloud storage;
the time set is reserved data stored in cloud storage during platform operation process within one year.
Further, after the execution in S002 is completed, a first notification popup is generated to notify the user of the data analysis status, and the first notification popup at least includes an instruction button, where the instruction button is used to execute the next data processing on the completely analyzed data.
Furthermore, the target database is a preset writing path of the analysis data.
Further, the time set data after finishing sorting in the step 2 generates a project matching instruction and transmits the project matching instruction to the step 3, after executing time set data access and writing in the step 3, the time set data is compared with the project matching instruction after reading the number of the chunk to judge whether the data is finished or not, if yes, a second popup window is generated for prompting; and if not, listing the names of the unwritten time set data.
According to the technical scheme, the data updating and writing of the multichannel batch are realized, the load of the background of the server where the platform is located is avoided from being increased by the set writing, the multichannel and timing data writing can be realized, and the problems of platform blocking and collapse caused by overload of the server are avoided. And the dependence of big data departments is reduced, so that the business problem can be solved more sensitively.
Example 2
As shown in fig. 8, a lightweight multi-data source batch processing system, for running the lightweight multi-data source batch processing method described in the above embodiment 1, includes:
the plug-in writing module is used for matching the completed Job code into the platform, executing and storing an operation record Job task through the JobLanucher, executing Step in the Job task according to a specified logic sequence, and storing the result of each Step of Step;
the database processing module analyzes target database data through codes, analyzes the target database data according to QueryProvider in the codes, and sorts the target database data based on the completed analysis data, wherein the first name is first name and the last name is lastname;
and the database writing module is used for writing the data obtained through analysis through the binding database.
Specifically, the database writing module includes a paging synchronization unit, sets a plurality of tasks based on the created job task to create a plurality of paging intervals, and writes a plurality of data in sequence in a sorting cycle within the paging intervals.
In the above embodiment, the database processing module further includes a plurality of execution subcodes under the mysql page query provider code field, including:
method dataSource: binding a database;
method queryProvider: binding query information;
method rowMapper: converting the query result into an object;
method pageSize: the size of each page query is set.
Example 3
Referring to fig. 2, it can be known that the foregoing problems are solved by the present invention, and the method for processing light-weight multiple data sources is shown in the flow chart, and the meaning of each english in the chart disclosed in fig. 2 is as follows:
the basic principle of fig. 2 is that Job executes Job and holds operation records, then Job executes Step in a specified logical order and holds the result of each Step of Step.
The flow of fig. 2 is further explained with specific reference to the specific code of fig. 3-7:
1. implanting read database code:
QueryProvider sets what fields, what tables, and what conditions to use.
MySqlPagingQueryProvider：
Method dataSource: binding a database;
method queryProvider: binding query information;
method rowMapper: converting the query result into an object;
method pageSize: the size of each page query is set.
2. Service processing code:
based on the data read by the persongitemserver, the first name, the last name, lastname, are sorted to generate a temporary storage library.
3. Writing database code:
inserting data in batches;
method sql: inserted sql;
method dataSource: and binding the database.
4. The job binds step code:
method step1: step consists of reader, processor and writer and the step1 method is bound.
Method importUserJob: job is composed of multiple jobs to bind step to Job
Method Chunk: the number of reader reads per time is performed.
5. Integrating the dispatch center code:
and processing codes by the logic service, executing jobs by the jobLauncher, and finally completing the writing of data.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
The embodiment of the application also provides a specific implementation manner of the electronic device capable of implementing all the steps in the method in the embodiment, and the electronic device specifically comprises the following contents:
a processor (processor), a memory (memory), a communication interface (Communications Interface), and a bus;
the processor, the memory and the communication interface complete communication with each other through the bus;
the processor is configured to invoke the computer program in the memory, and when the processor executes the computer program, the processor implements all the steps in the method in the above embodiment.
The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, the computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements all the steps of the methods in the above embodiments.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a hardware+program class embodiment, the description is relatively simple, as it is substantially similar to the method embodiment, as relevant see the partial description of the method embodiment. Although the present description provides method operational steps as described in the examples or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps recited in the embodiments is merely one way of performing the order of steps and does not represent a unique order of execution. When implemented in an actual device or end product, the instructions may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even in a distributed data processing environment) as illustrated by the embodiments or by the figures. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, it is not excluded that additional identical or equivalent elements may be present in a process, method, article, or apparatus that comprises a described element. For convenience of description, the above devices are described as being functionally divided into various modules, respectively. Of course, when implementing the embodiments of the present disclosure, the functions of each module may be implemented in the same or multiple pieces of software and/or hardware, or a module that implements the same function may be implemented by multiple sub-modules or a combination of sub-units, or the like. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form. The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description embodiments may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments. In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the embodiments of the present specification.
In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction. The foregoing is merely an example of an embodiment of the present disclosure and is not intended to limit the embodiment of the present disclosure. Various modifications and variations of the illustrative embodiments will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, or the like, which is within the spirit and principles of the embodiments of the present specification, should be included in the scope of the claims of the embodiments of the present specification.
Claims (10)
1. A method for lightweight multi-data source batch processing, comprising the steps of:
s001, acquiring data stored in a designated database, calling a matching object in the data according to requirements, and generating a plurality of time sets;
s002, sorting is performed based on the generated plurality of time sets, the time sets of the first data are named first name, and the time sets of the last data are named lastname;
s003, inserting the analyzed data batch into a target database, and sequentially cycling batch access writing from the beginning of the first name to the end of the lastname in a sequencing order.
2. The method of claim 1, wherein the specified database is cloud storage;
the time set is reserved data stored in the cloud storage during platform operation within one year.
3. The method of claim 1, wherein after the execution in S002 is completed, a first notification window is generated to notify the user of the data parsing status, and the first notification window includes at least one command button for performing the next data processing on the completely parsed data.
4. The method of claim 1, wherein the target database is a predetermined write path of the parsed data.
5. The method of claim 1, wherein the time aggregate data after sorting in step 2 generates a project matching instruction and transmits the project matching instruction to step 3, and after the time aggregate data is accessed and written in step 3, the time aggregate data is read and compared with the project matching instruction to determine whether the data is completed, if yes, a second popup window is generated for prompting; and if not, listing the names of the unwritten time set data.
6. A lightweight multi-data source batch processing system, characterized by a method for running a lightweight multi-data source batch process as claimed in any one of claims 1-5, comprising:
the plug-in writing module is used for matching the completed Job code into the platform, executing and storing an operation record Job task through the JobLanucher, executing Step in the Job task according to a specified logic sequence, and storing the result of each Step of Step;
the database processing module analyzes target database data through codes, analyzes the target database data according to QueryProvider in the codes, and sorts the target database data based on the completed analysis data, wherein the first name is first name and the last name is lastname;
and the database writing module is used for writing the data obtained through analysis through the binding database.
7. The system of claim 6, wherein the database write module includes a page synchronization unit to set a plurality of tasks based on the created job tasks to create a plurality of paging intervals, and to write the plurality of data sequentially in a sort cycle within the paging intervals.
8. The lightweight multi-data source batch processing system of claim 6, wherein the database processing module further comprises a plurality of execution subcodes under the mysqlpnag query provider code field, respectively, comprising:
method dataSource: binding a database;
method queryProvider: binding query information;
method rowMapper: converting the query result into an object;
method pageSize: the size of each page query is set.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the steps of the method of lightweight multi-data source batch processing as claimed in any one of claims 1 to 5.
10. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of lightweight multi-data source batch processing as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211531579.5A CN116226141A (en) | 2022-12-01 | 2022-12-01 | Lightweight multi-data source batch processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211531579.5A CN116226141A (en) | 2022-12-01 | 2022-12-01 | Lightweight multi-data source batch processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116226141A true CN116226141A (en) | 2023-06-06 |
Family
ID=86570375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211531579.5A Pending CN116226141A (en) | 2022-12-01 | 2022-12-01 | Lightweight multi-data source batch processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116226141A (en) |
-
2022
- 2022-12-01 CN CN202211531579.5A patent/CN116226141A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110275861A (en) | Date storage method and device, storage medium, electronic device | |
| CN103077192B (en) | A kind of data processing method and system thereof | |
| EP2609501A2 (en) | Dynamic calculation of sample profile reports | |
| CN109684616A (en) | Dynamic statement formula assembles the method and system made a report on | |
| CN112463149B (en) | Software-defined satellite-oriented reusable code library construction method and device | |
| CN110471754A (en) | Method for exhibiting data, device, equipment and storage medium in job scheduling | |
| CN102902763A (en) | Method and device for relating and retrieving information processing data and processing information tasks | |
| CN111966760B (en) | Test data generation method and device based on Hive data warehouse | |
| US20110264703A1 (en) | Importing Tree Structure | |
| CN113961523A (en) | Business file splitting and summarizing method, device and equipment | |
| CN116226141A (en) | Lightweight multi-data source batch processing method | |
| US8229946B1 (en) | Business rules application parallel processing system | |
| CN112685360B (en) | Method and device for persistence of memory data, storage medium and computer equipment | |
| CN105426541B (en) | A kind of storage method and device of general data | |
| CN110764777B (en) | ELF file generation method, ELF file, equipment and storage medium | |
| CN112632266B (en) | Data writing method and device, computer equipment and readable storage medium | |
| CN109697216B (en) | Clearing transaction information processing method, device and system | |
| CN111008036A (en) | Structural data version management system and method based on GIT | |
| CN104166739A (en) | Index file processing method and device for analysis database | |
| CN110069595A (en) | Corpus label determines method, apparatus, electronic equipment and storage medium | |
| CN115061982B (en) | Case-customization-based relational graph construction method, system, terminal and medium | |
| CN103294755A (en) | Flexible assembling analysis method | |
| CN109992293A (en) | The assemble method and device of android system complement version information | |
| CN111767048B (en) | Data computing processing method, device and system | |
| JP2009134662A (en) | Performance test data construction tool |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |