CN115905244A - Method for batch processing data based on memory - Google Patents
Method for batch processing data based on memory Download PDFInfo
- Publication number
- CN115905244A CN115905244A CN202211721048.2A CN202211721048A CN115905244A CN 115905244 A CN115905244 A CN 115905244A CN 202211721048 A CN202211721048 A CN 202211721048A CN 115905244 A CN115905244 A CN 115905244A
- Authority
- CN
- China
- Prior art keywords
- data
- memory
- deleted
- database
- class
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
S1, assembling data needing to be stored at a front end, wherein the data which exists in the data but is no longer needed needs to be removed, and initiating a data storage request to a memory; s2, presetting a data set; s3, the back end receives the request parameters and inquires all existing related data in the database according to the service; s4, setting the deletion identification of the query result data to be deleted in batch, and adding the data to the update set; s5, analyzing and modifying the data in the memory; and S6, after the data are processed in the memory, performing persistence operation on the data according to the update class data set and the addition class data set, and respectively performing batch update operation and batch addition operation on the database.
Description
Technical Field
The invention relates to the field of software development, in particular to a method for batch processing of data based on a memory.
Background
Generally, for software development, a java backend interface has single definition responsibility, either processing newly added or modified data or deleting data, and a front end which completes the same function needs to judge and respectively call corresponding interfaces, the number of calls is large, the logic is complex, and a method for batch processing based on memory and including requests for newly added, modified and deleted data is provided at the current backend, and a plurality of operations are aggregated into one interface for simplifying front-end development.
This engineering problem can be abstracted as: data generated by a certain function may need to be persisted to a database, data may need to be updated, and data may also need to be deleted.
Disclosure of Invention
The invention aims to solve the defects in the prior art and provides a batch data processing method based on an internal memory;
in order to achieve the purpose, the invention adopts the following technical scheme: a method for processing data in batches based on a memory, which comprises the following steps,
s1, assembling data needing to be stored at a front end, eliminating the data which exist in the data but are no longer needed, and initiating a data storage request to a memory;
the memory can be a hard disk, a database and the like;
s2, presetting a data set;
defining a data set in advance for service data needing to be stored;
the data set comprises two types, namely an update class set and an addition class set;
s3, the back end receives the request parameters and inquires all existing related data in the database according to the service;
s4, setting the deletion identification of the query result data to be deleted in batch, and adding the data to the update set;
s5, analyzing and modifying the data in the memory;
analyzing data cycle traversal data, and judging whether the primary key id of the data is null or not;
and S6, after the data are processed in the memory, performing persistence operation on the data according to the update class data set and the addition class data set, and respectively performing batch update operation and batch addition operation on the database.
Preferably, step S4 further comprises the sub-steps of,
s4-1, querying data which exist in the mysql database and are not deleted;
s4-2, marking the inquired data as deleted in the memory;
s4-3, the data marked as deleted are placed in an update class set, and the data are copied and placed in a Map set at the same time; .
Preferably, step S5 further comprises the sub-steps of,
s5-1, if the primary key id of the data is null, executing a newly added data logic;
repeating step S5 for a subset of data;
s5-2, if the primary key id of the data is not null, executing data updating logic;
s5-3, if the data required to be stored in the request does not contain the deleted partial data, the partial data will not be reset from deleted to undeleted.
Compared with the prior art, the invention has the beneficial effects that: according to the method for processing data, the workload of front-end and back-end development is greatly reduced, the service processing logic is simplified, the development efficiency and the service processing efficiency are improved, the data are deleted based on memory processing and the database is stored in batches, the database access times are greatly reduced, and the system performance is improved.
Drawings
FIG. 1 is a flow chart of a method for batch processing data based on memory according to the present invention.
Detailed Description
In order to further understand the objects, structures, features and functions of the present invention, the following embodiments are described in detail.
Referring to fig. 1, a method for batch processing data based on memory according to an embodiment of the present invention includes the following steps:
s1, assembling data to be stored at a front end, eliminating the data which exists in the data but is no longer needed, and initiating a data storage request to a memory;
the memory can be a hard disk, a database and the like;
s2, presetting a data set;
defining a data set in advance for service data needing to be stored;
the data set comprises two types, namely an update class set and an addition class set;
s3, the back end receives the request parameters and inquires all existing related data in the database according to the service;
s4, setting the deletion identification of the query result data to be deleted in batch, and adding the data to the update set;
s4-1, querying data which exist in the mysql database and are not deleted;
s4-2, marking the inquired data as deleted in the memory;
s4-3, the data marked as deleted are placed in an update class set, and the data are copied and placed in a Map set at the same time;
the Map set is a set of key-value types, wherein key is the main key id of data, and value is the data, so that subsequent data operation is facilitated;
the database does not need to be operated when the data is marked in the memory, because the data is not in the final state at this time, the access times of the database can be reduced, compared with the method for modifying the data state of the database by accessing the database, the efficiency is greatly improved, meanwhile, the occupation of database resources is reduced, and the performance of the database is improved;
s5, analyzing and modifying the data in the memory;
analyzing data cycle traversal data, and judging whether the primary key id of the data is null or not;
s5-1, if the primary key id of the data is null, the data is newly added, the newly added data logic is executed, the state identification is set to be undeleted, and the data is stored in an added class set;
repeating step S5 for a subset of the data;
s5-2, if the primary key id of the data is not null, the data is stored in the database, data updating logic is executed, the data is obtained from the Map set of the deleted data according to the primary key id, the data content is replaced by the data needing to be stored in the request, and the state identifier is reset from the deleted state to be undeleted;
s5-3, if the data required to be stored in the request does not contain deleted partial data, the partial data cannot be reset from deleted to undeleted, are still stored in the update class set, and are finally updated to the database together to become true deleted data;
and S6, after the data are processed in the memory, performing persistence operation on the data according to the update class data set and the addition class data set, and respectively performing batch update operation and batch addition operation on the database.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (3)
1. A method for batch processing data based on a memory is characterized in that: comprises the following steps of (a) carrying out,
s1, assembling data to be stored at a front end, eliminating the data which exists in the data but is no longer needed, and initiating a data storage request to a memory;
the memory can be a hard disk, a database and the like;
s2, presetting a data set;
defining a data set in advance for service data to be stored;
the data sets comprise two types, one is an updating class set, and the other is an adding class set;
s3, the back end receives the request parameters and inquires all existing related data in the database according to the service;
s4, setting the deletion identification of the query result data to be deleted in batch, and adding the data to the update set;
s5, analyzing and modifying the data in the memory;
analyzing data cycle traversal data, and judging whether the primary key id of the data is null or not;
and S6, after the data are processed in the memory, performing persistence operation on the data according to the update class data set and the addition class data set, and respectively performing batch update operation and batch addition operation on the database.
2. The method of claim 1, wherein the method comprises: step S4 further comprises the sub-steps of,
s4-1, querying data which exist in the mysql database and are not deleted;
s4-2, marking the inquired data as deleted in the memory;
and S4-3, putting the data marked as deleted into the update class set, copying the data of the class and storing the data into the Map set.
3. The method of claim 1, wherein the method comprises: step S5 also comprises the sub-steps of,
s5-1, if the primary key id of the data is null, executing a newly added data logic;
repeating step S5 for a subset of data;
s5-2, if the primary key id of the data is not null, executing data updating logic;
s5-3, if the data needing to be stored in the request does not contain the deleted partial data, the partial data will not be reset from deleted to undeleted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211721048.2A CN115905244A (en) | 2022-12-30 | 2022-12-30 | Method for batch processing data based on memory |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211721048.2A CN115905244A (en) | 2022-12-30 | 2022-12-30 | Method for batch processing data based on memory |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115905244A true CN115905244A (en) | 2023-04-04 |
Family
ID=86482418
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211721048.2A Pending CN115905244A (en) | 2022-12-30 | 2022-12-30 | Method for batch processing data based on memory |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115905244A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116107572A (en) * | 2023-04-07 | 2023-05-12 | 苏州万店掌网络科技有限公司 | Batch operation persistent object method, device, computer equipment and medium |
-
2022
- 2022-12-30 CN CN202211721048.2A patent/CN115905244A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116107572A (en) * | 2023-04-07 | 2023-05-12 | 苏州万店掌网络科技有限公司 | Batch operation persistent object method, device, computer equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6567809B2 (en) | Disabling and reloading enterprise java beans using database trigger programs | |
| US6304867B1 (en) | System and method for enhanced performance of a relational database management system through the use of application-specific memory-resident data | |
| CN109144994A (en) | Index updating method, system and relevant apparatus | |
| CN107038222B (en) | Database cache implementation method and system | |
| US7949685B2 (en) | Modeling and implementing complex data access operations based on lower level traditional operations | |
| US7020659B2 (en) | System and method for managing bi-directional relationships between objects | |
| US20070250517A1 (en) | Method and Apparatus for Autonomically Maintaining Latent Auxiliary Database Structures for Use in Executing Database Queries | |
| CN110188114B (en) | Data operation optimization method, device, system, equipment and storage medium | |
| CN109299101B (en) | Data retrieval method, device, server and storage medium | |
| CN112015741A (en) | Method and device for storing massive data in different databases and tables | |
| WO2021169496A1 (en) | Data reading method, apparatus, and device, and storage medium | |
| CN115905244A (en) | Method for batch processing data based on memory | |
| CN114817974A (en) | Dynamic data desensitization method and system, and data security processing method and system | |
| CN115794874A (en) | Method for accelerating GPU operator execution in heterogeneous database system and application | |
| CN115269631A (en) | Data query method, data query system, device and storage medium | |
| CN111352936A (en) | Method and storage medium for ES index reconstruction | |
| AU7879598A (en) | Language manager interface | |
| CN113051271A (en) | Cold and hot data separation method, device and equipment | |
| CN117421302A (en) | Data processing method and related equipment | |
| US20070088684A1 (en) | Partial updating in a database proxy driver | |
| US9009731B2 (en) | Conversion of lightweight object to a heavyweight object | |
| CN115098503A (en) | Null value data processing method and device, computer equipment and storage medium | |
| CN114925142A (en) | Multi-type database compatibility method, device, equipment and medium of ORM framework | |
| CN110222105B (en) | Data summarization processing method and device | |
| CN112487111A (en) | Data table association method and device based on KV database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |