CN111382209B - Data transfer and operation method of distributed memory database - Google Patents
Data transfer and operation method of distributed memory database Download PDFInfo
- Publication number
- CN111382209B CN111382209B CN202010256466.3A CN202010256466A CN111382209B CN 111382209 B CN111382209 B CN 111382209B CN 202010256466 A CN202010256466 A CN 202010256466A CN 111382209 B CN111382209 B CN 111382209B
- Authority
- CN
- China
- Prior art keywords
- order
- data
- memory database
- file system
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000012546 transfer Methods 0.000 title claims abstract description 14
- 238000013500 data storage Methods 0.000 claims abstract description 14
- 238000013508 migration Methods 0.000 claims abstract description 9
- 230000005012 migration Effects 0.000 claims abstract description 9
- 238000006243 chemical reaction Methods 0.000 claims abstract description 6
- 210000001503 joint Anatomy 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 13
- 238000007726 management method Methods 0.000 abstract description 3
- 238000003032 molecular docking Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G06Q50/40—
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a data transfer and operation method of a distributed memory database, which comprises the following steps: creating an order in a distributed memory database, writing fields of the order into an order index table, and writing order data to be processed into an instance list; migrating the fields of the processed order form the example positive list to the example year and month list, and simultaneously migrating the data of the processed order form to the distributed file system in real time; transferring the order to a data storage position identifier in a distributed file system and updating the data storage position identifier to an order index table; the data migration, docking and conversion configuration is agreed between the distributed memory database and the distributed file system through a public configuration table. By the technical scheme, the consumption of distributed memory database resources is reduced, the efficient access to the data of the archived orders is realized, and the processing efficiency, performance and data life cycle management of the orders are improved.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a data transfer and operation method of a distributed memory database.
Background
Most of the traditional order data in the telecommunication industry are stored in a relational database (such as oracle) or distributed relational data (such as distributed mysql), and the application scenario mainly brings the following problems:
problem one: the data storage resources and the host expand, which can lead to the continuous increase of the storage medium of the database, the obvious decrease of the database performance and the increase of the data block fragments over time, and increase the cost of the host resources;
and a second problem: when the order processing performance is limited and the order quantity is very large, the performance of order creation and circulation can become a bottleneck because of a database, and the order can not be processed immediately and efficiently;
problem three: the order processing capacity and efficiency cannot be improved, the service scene and the order transaction amount are limited, and the service expansion is affected.
With the rapid development of mobile internet, internet of things and 5G services, the service scene and the explosive growth of the service volume have higher and higher performance requirements on the order system, so that the order system has high concurrency continuous processing capability besides ensuring efficient circulation, and needs to store customer order data details for a long time.
Disclosure of Invention
Aiming at least one of the problems, the invention provides a data transfer and operation method of a distributed memory database, wherein the data of order creation and order circulation are stored in the distributed memory database, the performance of order creation and order processing is improved through the distributed memory database, and the orders after circulation processing are migrated to a distributed file system in real time so as to reduce the consumption of resources of the distributed memory database; meanwhile, an order index table is introduced to serially connect the data of the distributed memory database and the distributed file system, so that efficient access to the data of the archived order is realized, and the processing efficiency, performance and data life cycle management of the order are improved.
In order to achieve the above objective, the present invention provides a method for data transfer and operation of a distributed memory database, including: creating an order in a distributed memory database, writing fields of the order into an order index table, and writing order data to be processed into an instance list; migrating the fields of the processed order from the instance positive table to the instance year and month table, and simultaneously migrating the data of the processed order to a distributed file system in real time; transferring the order to a data storage position identification in the distributed file system and updating the data storage position identification to the order index table; and the distributed memory database and the distributed file system agree on data migration, butting and conversion configuration through a public configuration table.
In the above technical solution, preferably, the data transfer and operation method of the distributed memory database further includes: and when the example positive table and/or the example annual and lunar table in the distributed memory database are operated, updating the fields of the operated example positive table and/or the example annual and lunar table to the order index table.
In the above technical solution, preferably, when operating on order data in the distributed file system, the data storage location identifier corresponding to the order in the order index table is used to search the order data location in the distributed file system, and operate on the order data.
In the above technical solution, preferably, the distributed memory database and the distributed file system both use 1-master-2-slave memory databases.
In the above technical solution, preferably, a field is provided in the order index table, so as to distinguish that a certain order data is stored in the distributed memory database or the distributed file system.
In the above technical solution, preferably, the order index table is created according to order service data classification in the distributed memory database, the same class of data table in the order index table is stored in the distributed file system according to a column cluster, all Value values in the distributed file system are stored in a Json array manner and include corresponding field names, and the butt joint of the distributed memory database and the distributed file system is realized through the common configuration table.
In the above technical solution, preferably, after live migration of the processed order data to the distributed file system, the corresponding order data in the distributed memory database is deleted.
In the above technical solution, preferably, the instance positive table and the instance year and month table both use relational databases.
Compared with the prior art, the invention has the beneficial effects that: the data of order creation and order circulation are stored in a distributed memory database, the performance of order creation and order processing is improved through the distributed memory database, and the orders after circulation processing are migrated to a distributed file system in real time so as to reduce the consumption of resources of the distributed memory database; meanwhile, an order index table is introduced to serially connect the data of the distributed memory database and the distributed file system, so that efficient access to the data of the archived order is realized, and the processing efficiency, performance and data life cycle management of the order are improved.
Drawings
FIG. 1 is a schematic diagram of an order submission data flow for a method of data transfer and operation of a distributed memory database according to one embodiment of the present invention;
fig. 2 is a schematic diagram illustrating an order data operation of a data transfer and operation method of a distributed memory database according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention is described in further detail below with reference to the attached drawing figures:
as shown in fig. 1 and fig. 2, the method for data transfer and operation of a distributed memory database according to the present invention includes: creating an order in a Distributed Memory Database (DMDB), writing fields of the order into an order index table, and writing order data to be processed into an instance list; migrating the fields of the processed order from the instance schedule to the instance chronology, and simultaneously migrating the data of the processed order to a distributed file system (Hbase) in real time; transferring the order to a data storage position identifier in a distributed file system and updating the data storage position identifier to an order index table; the data migration, docking and conversion configuration is agreed between the distributed memory database and the distributed file system through a public configuration table.
In this embodiment, the distributed memory database stores mainly data of unprocessed completed orders, index table data, and basic configuration data of orders; the multi-node is adopted to disperse the order instance data, so that IO of an order hotspot table is reduced, and meanwhile, because the data is stored and processed in a memory database, the creation performance of the order and the concurrent processing capacity of order process circulation are greatly improved. The distributed file system mainly stores data of order filing annual month table, and each annual month table is set for a life cycle of 3 years according to group specifications, so that the storage pressure (mainly the pressure of memory consumption) of a distributed memory database is reduced, and meanwhile, the hardware cost is also reduced.
Specifically, the unprocessed order data and the processed order data are respectively stored in the distributed memory database through an example positive table and an example annual month table, the data formats of all the example positive tables and the example annual month tables are the storage formats of the relational database, meanwhile, an order index table is recorded, order core fields and data storage identifiers are recorded in the order index table, and the core fields also comprise Key value fields of the distributed file system so as to quickly filter and search the storage positions of the order data. The processed order data is stored in the distributed file system through an Hbase annual month table, the data is stored in a classified mode through Hbase column clusters, the data is stored in a K/V mode, and the data of the single table is stored in a json format.
In the above embodiment, after live migration of the processed order data to the distributed file system, the corresponding order data in the distributed memory database is preferably deleted.
In the foregoing embodiment, preferably, the data unloading and operation method of the distributed memory database further includes: and when the example positive table and/or the example annual month table in the distributed memory database are operated, updating the fields of the operated example positive table and/or the example annual month table to the order index table.
Preferably, the order index table is provided with a db_flag field for distinguishing that certain order data is stored in a distributed memory database or a distributed file system. After confirming the data storage position, if the DMDB is in the DMDB, the DMDB operation data is accessed, if the DMDB operation data is in Hbase, the DMDB operation data is operated through an API interface of the Hbase.
In the above embodiment, it is preferable that, when the order data in the distributed file system is operated, the order data location in the distributed file system is searched by the data storage location identifier of the corresponding order in the order index table, and the order data is operated.
In the above embodiment, preferably, the distributed memory database and the distributed file system both use 1 main memory database and 2 standby memory databases, so as to ensure high availability of the distributed memory file system.
In the above embodiment, preferably, the order index table is created according to order service data classification in the distributed memory database, the same type of data table in the order index table is stored in the distributed file system according to a column cluster, all Value values in the distributed file system are stored in a Json array manner and include corresponding field names, and the butt joint of the distributed memory database and the distributed file system is realized through the common configuration table.
In the above embodiment, the butt-joint conversion configuration of the DMDB data table and the Hbase data table is agreed by setting the common configuration table, and the butt-joint conversion configuration includes table names of two sets of databases, a column cluster of Hbase, a primary key, a migration identifier, and the like. The table is mainly used for data migration, and condition filtering and rule matching of a public data access method. The details are indicated as follows:
the docking design of the distributed memory database and the distributed file system is shown in the following table:
according to the data transfer and operation method of the distributed memory database provided by the embodiment, the order data model builds a table according to three dimensions, the DMDB builds a positive table, a chronology table and a Hbase builds a chronology table. And the order program writes and migrates data according to three dimensions and synchronously updates the index table. The public interface method of the chronology is provided, and the efficient and seamless butt joint of the DMDB and Hbase data is realized.
The processing capacity of order creation and order circulation is greatly improved through the distributed memory database, and the hot spot table IO is controlled in a reasonable range. After the data is migrated to Hbase in real time, the distributed memory database does not increase storage pressure because of processing the completed order data, and the demand for host resources is reduced. The data of the distributed memory database and the distributed file system can be efficiently and seamlessly operated through the order index table, the integrity and the high efficiency of subsequent business are ensured, the capacity expansion requirement of host hardware resources caused by business growth is reasonably controlled, and the host resource cost is reduced.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. The data transfer and operation method of the distributed memory database is characterized by comprising the following steps of:
creating an order in a distributed memory database, writing fields of the order into an order index table, and writing order data to be processed into an instance list;
migrating the fields of the processed order from the instance positive table to the instance year and month table, and simultaneously migrating the data of the processed order to a distributed file system in real time;
transferring the order to a data storage position identification in the distributed file system and updating the data storage position identification to the order index table;
the distributed memory database and the distributed file system agree on data migration, butting and conversion configuration through a public configuration table;
the order index table is created according to order service data classification in the distributed memory database, the same class of data table in the order index table is stored in the distributed file system according to column clusters, all Value values in the distributed file system are stored in a Json array mode and contain corresponding field names, and the butt joint of the distributed memory database and the distributed file system is realized through the public configuration table; and a field is arranged in the order index table and used for distinguishing that certain order data is stored in the distributed memory database or the distributed file system.
2. The method for data transfer and manipulation of a distributed memory database of claim 1, further comprising:
and when the example positive table and/or the example annual and lunar table in the distributed memory database are operated, updating the fields of the operated example positive table and/or the example annual and lunar table to the order index table.
3. The method for data transfer and manipulation of a distributed memory database of claim 2 wherein, when manipulating order data in said distributed file system, the order data location in said distributed file system is looked up by the data storage location identifier of the corresponding order in said order index table and said order data is manipulated.
4. The method of claim 1, wherein the distributed memory database and the distributed file system each use 1-master-2-slave memory databases.
5. The method of claim 1, wherein the corresponding order data in the distributed memory database is deleted after live migration of the processed order data to the distributed file system.
6. The method of claim 1, wherein the instance positive table and the instance chronology table each employ a relational database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010256466.3A CN111382209B (en) | 2020-04-02 | 2020-04-02 | Data transfer and operation method of distributed memory database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010256466.3A CN111382209B (en) | 2020-04-02 | 2020-04-02 | Data transfer and operation method of distributed memory database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111382209A CN111382209A (en) | 2020-07-07 |
| CN111382209B true CN111382209B (en) | 2023-07-25 |
Family
ID=71218943
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010256466.3A Active CN111382209B (en) | 2020-04-02 | 2020-04-02 | Data transfer and operation method of distributed memory database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111382209B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013049241A1 (en) * | 2011-09-30 | 2013-04-04 | Oracle International Corporation | High throughput global order promising system |
| CN107123031A (en) * | 2017-05-02 | 2017-09-01 | 深圳乐信软件技术有限公司 | A kind of order processing method and device |
| CN110275900A (en) * | 2019-06-25 | 2019-09-24 | 浪潮软件股份有限公司 | A method of order and early warning are monitored based on Redis caching technology |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9251214B2 (en) * | 2010-04-08 | 2016-02-02 | Microsoft Technology Licensing, Llc | In-memory database system |
| CN103631907B (en) * | 2013-11-26 | 2016-09-07 | 中国科学院信息工程研究所 | A kind of method and system that relational data is migrated to HBase |
| US20150262127A1 (en) * | 2014-03-17 | 2015-09-17 | Carsten Ziegler | Generation and optimzation of in-memory database business rule logic |
| CN106709784A (en) * | 2016-12-28 | 2017-05-24 | 广州虎鱼网络科技有限公司 | Cross-border e-commerce platform having high concurrency extensible distributed type architecture |
| US11782900B2 (en) * | 2018-04-10 | 2023-10-10 | Walmart Apollo, Llc | High throughput order fullfillment database system |
| US11036517B2 (en) * | 2018-04-27 | 2021-06-15 | Sap Se | Database management system performing column operations using a set of SIMD processor instructions selected based on performance |
| CN109949129A (en) * | 2019-03-12 | 2019-06-28 | 北京思特奇信息技术股份有限公司 | A kind of order processing system and method for high concurrent |
-
2020
- 2020-04-02 CN CN202010256466.3A patent/CN111382209B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013049241A1 (en) * | 2011-09-30 | 2013-04-04 | Oracle International Corporation | High throughput global order promising system |
| CN107123031A (en) * | 2017-05-02 | 2017-09-01 | 深圳乐信软件技术有限公司 | A kind of order processing method and device |
| CN110275900A (en) * | 2019-06-25 | 2019-09-24 | 浪潮软件股份有限公司 | A method of order and early warning are monitored based on Redis caching technology |
Non-Patent Citations (3)
| Title |
|---|
| Wenming Guo ; Zhiqiang Hu.Memory Database Index Optimization.《2010 International Conference on Computational Intelligence and Software Engineering》.2010,全文. * |
| 基于综合交易平台的程序化交易系统的设计与实现;徐建生;《CNKI》;全文 * |
| 郝晓培 ; 单杏花 ; 梅巧玲.基于用户行为的订单实时缓存技术的研究.第十一届中国智能交通年会.2016,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111382209A (en) | 2020-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107818115B (en) | Method and device for processing data table | |
| CN103294710B (en) | A kind of data access method and device | |
| CN101751406B (en) | Method and device for realizing column storage based relational database | |
| WO2020192064A1 (en) | Incremental data consistency implementation method and device | |
| WO2021238701A1 (en) | Data migration method and device | |
| CN101382953A (en) | Interface system for accessing file system in user space and file reading and writing method | |
| CN102799679B (en) | Hadoop-based massive spatial data indexing updating system and method | |
| CN103873559A (en) | Database all-in-one machine capable of realizing high-speed storage | |
| CN109521959A (en) | One kind being based on SSD-SMR disk mixing key assignments memory system data method for organizing | |
| CN101916290B (en) | Managing method of internal memory database and device | |
| CN102955792A (en) | Method for implementing transaction processing for real-time full-text search engine | |
| CN103500089A (en) | Small file storage system suitable for Mapreduce calculation model | |
| CN104462185A (en) | Digital library cloud storage system based on mixed structure | |
| CN110309233A (en) | Method, apparatus, server and the storage medium of data storage | |
| CN107301214A (en) | Data migration method, device and terminal device in HIVE | |
| CN102622434A (en) | Data storage method, data searching method and device | |
| CN100504800C (en) | Method for snapshot of magnetic disc | |
| CN103473258A (en) | Cloud storage file system | |
| CN103365987A (en) | Clustered database system and data processing method based on shared-disk framework | |
| CN116521956A (en) | Graph database query method and device, electronic equipment and storage medium | |
| CN103744952A (en) | Method and system for implementing multi-table property of NoSQL database | |
| CN109189724B (en) | Method and device for improving audio and video data storage efficiency of video monitoring system | |
| CN111382209B (en) | Data transfer and operation method of distributed memory database | |
| CN109918393A (en) | The data platform and its data query and multilist conjunctive query method of Internet of Things | |
| CN102262675A (en) | Method for querying database and smart card |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |