CN113420021A - Data storage method, device, equipment and medium - Google Patents
Data storage method, device, equipment and medium Download PDFInfo
- Publication number
- CN113420021A CN113420021A CN202110794255.XA CN202110794255A CN113420021A CN 113420021 A CN113420021 A CN 113420021A CN 202110794255 A CN202110794255 A CN 202110794255A CN 113420021 A CN113420021 A CN 113420021A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- memory type
- relational database
- failed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000004140 cleaning Methods 0.000 claims description 43
- 230000001360 synchronised effect Effects 0.000 claims description 40
- 230000002159 abnormal effect Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 5
- 238000007726 management method Methods 0.000 description 11
- 230000004044 response Effects 0.000 description 5
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The embodiment of the invention discloses a data storage method, a data storage device, data storage equipment and a data storage medium. The data storage method comprises the following steps: responding to data writing operation of a user, and storing hot data written by the user into a memory type database; according to a first synchronization frequency, synchronizing thermal data in the memory type database to a relational database as temperature data; and synchronizing the temperature data in the relational database to the distributed database as cold data according to the second synchronization frequency. According to the technical scheme of the embodiment of the invention, different databases are selected for data storage according to data characteristics, so that the data access efficiency is improved.
Description
Technical Field
Embodiments of the present invention relate to data storage technologies, and in particular, to a data storage method, apparatus, device, and medium.
Background
With the rapid development of internet technology, the global data volume is increased explosively, how to quickly query data in a large amount of data and ensure the writing efficiency of the data is very important for improving the use experience of internet users.
In order to improve data access performance, data storage is often performed in the form of sub-tables or sub-banks in the prior art. The table division means that a plurality of tables are divided under one database, and the table structure is kept unchanged, so that the union set of the data sets after the splitting is equal to the data set before the splitting. The database partitioning means that data in the same database is partitioned into a plurality of databases according to specific conditions and dimensions so as to achieve the effect of dispersing the load of a single database.
Although the data access performance is improved to a certain extent by the method, the splitting rule of the method is specific to the data service, a universal splitting rule is difficult to abstract, and the difficulty of later maintenance is increased. After the data volume reaches a certain degree, the cold data query speed is very slow and the database is easy to crash by the method, so that the normal data access of internet users is influenced.
Disclosure of Invention
Embodiments of the present invention provide a data storage method, apparatus, device, and medium, which improve data access efficiency in a large data volume case by storing data with different data characteristics in different databases.
In a first aspect, an embodiment of the present invention provides a data storage method, where the method includes:
responding to data writing operation of a user, and storing hot data written by the user into a memory type database;
according to a first synchronization frequency, synchronizing thermal data in the memory type database to a relational database as temperature data;
and synchronizing the temperature data in the relational database to the distributed database as cold data according to the second synchronization frequency.
In a second aspect, an embodiment of the present invention further provides a data storage apparatus, where the apparatus includes:
the hot data writing module is used for responding to data writing operation of a user and storing hot data written by the user into the memory type database;
the temperature data synchronization module is used for synchronizing the thermal data in the memory type database to the relational database as temperature data according to a first synchronization frequency;
and the cold data synchronization module is used for synchronizing the temperature data in the relational database to the distributed database as cold data according to a second synchronization frequency.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method provided by any embodiment of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data storage method provided in any embodiment of the present invention.
According to the technical scheme of the embodiment of the invention, in response to data writing operation of a user, hot data written by the user is stored in the memory type database, then the hot data in the memory type database is synchronized to the relational database as temperature data according to the first synchronization frequency, finally the temperature data in the relational database is synchronized to the distributed database as cold data according to the second synchronization frequency, and the data can be stored in the appropriate database according to data characteristics.
Drawings
FIG. 1 is a flow chart of a data storage method according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a data storage method according to a second embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data storage method in an embodiment of the present invention, where the technical solution of this embodiment is suitable for a case where different types of databases are used to store different types of data, and the method may be executed by a data storage device, and the device may be implemented by software and/or hardware, and may be integrated in various general-purpose computer devices.
The memory type database, the relational database and the distributed database mentioned in this embodiment may be respectively deployed in different servers, and the distributed storage management platform performs data management and status monitoring on each of the databases that are deployed in a distributed manner. The distributed storage management platform may be understood as a client for managing and monitoring each database, and may be deployed in a server deployed in any database, or may be deployed in a separate server.
Based on the distributed storage management platform, the data storage method in this embodiment specifically includes the following steps:
The hot data is data with a storage time smaller than a set time threshold, and because the storage time is short, the probability that the hot data is queried and located is larger than that of data stored earlier, and for example, the hot data is data with a storage time smaller than 1 hour. The memory type database is a database for storing thermal data, the memory type database directly operates the data in the memory, compared with a magnetic disk, the data read-write speed of the memory is higher by several orders of magnitude, the data access performance can be greatly improved by storing the data in the memory compared with accessing the data from the magnetic disk, illustratively, the memory type database is a database based on a NoSQL engine, and the memory type database can be a Redis database, a MongoDB database or a Solar database.
In this embodiment, in order to improve the reading efficiency of the thermal data, so that the server can quickly locate and read the thermal data in the mass data, when the data writing operation is obtained, the thermal data written by the user can be stored in the memory type database, wherein the thermal data can be quickly written and read by using the memory type database to store the thermal data. Illustratively, in response to a user entry information submission operation, entry information filled in by the user is stored directly in the memory-type database as hot data. The memory type database can be deployed on one server in a distributed system, and the distributed system comprises a plurality of servers.
And step 120, synchronizing the thermal data in the memory type database as temperature data to the relational database according to the first synchronization frequency.
The temperature data is data which is longer in storage time than the hot data and shorter in storage time than the cold data, the probability that the temperature data is inquired and located is smaller than that of the hot data and larger than that of the cold data, and the temperature data is data which is stored for 1 hour or more and less than one week. The relational database is a database for storing temperature data, the relational database is a database that organizes data using a relational model and stores data in rows and columns, and is exemplarily a SQL engine-based database, and commonly used relational databases include Oracle, DB2, MySQL, Microsoft SQL Server, Microsoft Access, and the like.
In this embodiment, to implement the hierarchical data storage, the hot data in the memory type database may be synchronized to the relational database as the temperature data at regular time according to a preset first synchronization frequency. Illustratively, a timing synchronization task is set, hot data in the memory type database is synchronized to the relational database every 1 minute, the data in the memory type database is prevented from being deleted due to factors such as power failure or timing clearing, meanwhile, data with long storage time can be transferred to the relational database, data layered storage is achieved, and data reading performance is guaranteed.
Optionally, when the hot data written by the user is stored in the memory type database, the hot data in the memory type database is synchronized to the relational database as the temperature data in real time, so that the data in the memory type database is prevented from being deleted due to factors such as power failure or timing clearing.
And step 130, synchronizing the temperature data in the relational database as cold data to the distributed database according to the second synchronization frequency.
Where cold data is data that has a longer storage time than warm data, the probability that the cold data will be queried and located is minimal, and for example, cold data is data that has a storage time greater than or equal to one week. The distributed database is used for storing cold data, the distributed database is a large database which is formed by connecting a plurality of computers which are deployed with local databases and located at different places through a network, the large database is integrated and globally and logically distributed, the relational database is exemplarily based on spark or MR engines, and commonly used distributed databases comprise Greenplus or Hadoop and the like.
In this embodiment, in order to implement hierarchical data storage, the medium-temperature data in the relational database may be synchronized to the distributed database as the cold data at regular time according to a preset second synchronization frequency. Illustratively, a timing synchronization task is set, temperature data in the relational database are synchronized to the distributed database every 24 hours, data layered storage is achieved, and data reading performance is guaranteed.
Optionally, when the hot data in the memory type database is synchronized to the relational database as the temperature data, the temperature data in the relational database is synchronized to the distributed database as the cold data in real time, so that the data in the relational database is prevented from being deleted due to factors such as timing clearing.
In this embodiment, the memory type database, the relational database, and the distributed database may be respectively deployed in different servers, and the distributed storage management platform performs data management and status monitoring on the databases deployed in a distributed manner, and the deployment manner may reduce the degree of coupling between the databases, and the databases may have clear functions, and may quickly locate a problem when an abnormal state occurs, thereby improving the problem solving efficiency.
According to the technical scheme of the embodiment of the invention, in response to data writing operation of a user, hot data written by the user is stored in the memory type database, then the hot data in the memory type database is synchronized to the relational database as temperature data according to the first synchronization frequency, finally the temperature data in the relational database is synchronized to the distributed database as cold data according to the second synchronization frequency, and the data can be stored in the appropriate database according to data characteristics.
Example two
Fig. 2 is a flowchart of a data storage method according to a second embodiment of the present invention, which is further detailed based on the above embodiments and provides a specific step of synchronizing thermal data in the memory type database as temperature data to the relational database according to the first synchronization frequency, and a specific step of synchronizing temperature data in the relational database as cold data to the distributed database according to the second synchronization frequency. A data storage method provided by the second embodiment of the present invention is described below with reference to fig. 2, which includes the following steps:
And step 220, synchronizing the thermal data in the memory type database as temperature data to the relational database by adopting an increment synchronization mode according to the first synchronization frequency.
In this embodiment, a specific manner of synchronizing thermal data in the memory type database as temperature data to the relational database is provided, and the thermal data in the memory type database as temperature data is synchronized to the relational database by an incremental synchronization manner according to a preset first synchronization frequency. For example, every 1 minute, whether newly-added thermal data which is not synchronized exists in the memory type database is judged, if yes, the newly-added thermal data is synchronized to the relational database as temperature data, if not, synchronization operation is not performed, and the incremental synchronization mode can be used for synchronizing only the changed data, so that the data synchronization efficiency is improved.
And step 230, synchronizing the temperature data in the relational database to the distributed database as cold data by adopting an incremental synchronization mode according to the second synchronization frequency.
In this embodiment, a specific manner of synchronizing the temperature data in the relational database to the distributed database as the cold data is provided, and the temperature data in the relational database is synchronized to the distributed database as the cold data by an incremental synchronization manner according to a preset second synchronization frequency. Illustratively, every 24 hours, whether newly added temperature data which are not synchronized exist in the relational database is judged, if yes, the newly added temperature data are taken as cold data and synchronized to the distributed database, and if not, synchronization operation is not carried out, so that the data synchronization efficiency is improved.
Optionally, the technical solution of this embodiment further includes:
and cleaning the memory type database, the relational database or the distributed database according to preset database cleaning rules.
In this optional embodiment, too large amount of data stored in a single database may affect data query and write efficiency, and in order to ensure data access speed, the distributed storage management platform may clean the memory type database, the relational database, or the distributed database according to a preset database cleaning rule, so as to ensure data access efficiency.
Illustratively, a timing cleaning task of the memory type database can be set, and data in the memory type database is cleaned once every 30 minutes, specifically, all the data can be cleaned, and partial data can be cleaned according to the storage duration of the data; setting a timed cleaning task of the relational database, cleaning the data in the relational database once every week, wherein a specific cleaning mode can be either complete cleaning or partial data cleaning aiming at the actual storage time of the data in the relational database; similarly, for the distributed data bureau, a timed cleaning task can be set, for example, cleaning or archiving the data in the distributed database once every 6 months.
Optionally, the cleaning the memory type database according to a preset database cleaning rule includes:
determining the failed thermal data in the memory type database according to preset thermal data failure time, and judging whether the failed thermal data are synchronized to the relational database;
if yes, deleting the invalid hot data in the memory type database;
if not, after synchronizing the failed hot data to the relational database, deleting the failed hot data in the memory type database.
In this optional embodiment, a method for cleaning the memory type database according to the preset database cleaning rule is provided, where first, according to the preset thermal data failure time, the failed thermal data is determined in the memory type database, and then, whether the failed thermal data is synchronized to the relational database is determined, if yes, the failed thermal data in the memory type database is directly deleted, otherwise, the failed thermal data is synchronized to the relational database first, and then, the failed thermal data in the memory type database is deleted.
Illustratively, the preset database cleaning rule includes that the valid time of the hot data is 30 minutes, when the memory type database is cleaned, the hot data with the storage time exceeding 30 minutes in the memory type database is determined as the failed hot data, the failed hot data needs to be deleted, before deletion, in order to ensure data security, it needs to be judged whether the failed hot data is synchronized to the relational database, if yes, the data can be directly deleted, otherwise, data synchronization is performed first, and then deletion operation of the failed hot data is performed.
Optionally, the cleaning the relational database according to a preset database cleaning rule includes:
determining the temperature data which is failed in a relational database according to preset temperature data failure time, and judging whether the temperature data which is failed is synchronized to a distributed database;
if yes, deleting the temperature data which are invalid in the relational database;
if not, the failed temperature data are synchronized to the distributed database, and then the failed temperature data in the relational database are deleted.
In this optional embodiment, a manner of cleaning the relational database according to a preset database cleaning rule is provided, where first, according to a preset temperature data failure time, failed temperature data is determined in the relational database, and then, it is determined whether the failed temperature data is synchronized to the distributed database, if so, the failed temperature data in the relational database is directly deleted, otherwise, the failed temperature data is synchronized to the distributed database, and then, the failed temperature data in the relational database is deleted.
Illustratively, the preset database cleaning rule includes that the valid time of temperature data is one week, when the relational database is cleaned, the temperature data with the storage time exceeding one week in the relational database is determined as failed temperature data, the failed temperature data needs to be deleted, before deletion, in order to ensure data security, it is necessary to judge whether the failed temperature data is synchronized to the distributed database, if yes, the failed temperature data can be directly deleted, otherwise, data synchronization is performed first, and then deletion operation of the failed temperature data is performed.
Optionally, the method is characterized in that the distributed database is cleaned according to a preset database cleaning rule, and the method includes:
and deleting and/or archiving the cold data which is failed in the distributed database according to the preset cold data failure time.
In this optional embodiment, a manner of cleaning the distributed database according to a preset database cleaning rule is provided, and specifically, according to cold data failure time preset in the database cleaning rule, failed cold data is determined in the distributed database, and then the failed cold data in the distributed database is deleted and/or archived.
Illustratively, cold data invalidation time included in the database cleaning rule is 1 year, when distributed database cleaning is performed, the cold data with storage time exceeding 1 year is used as invalidated cold data, and then the invalidated cold data is deleted or archived for storage.
Optionally, the technical solution of this embodiment further includes:
acquiring the state of each database in a memory type database, a relational database and a distributed database, and sending database alarm prompt information to a user when the state of at least one database is abnormal;
and acquiring the running states of the servers in which the memory type database, the relational database and the distributed database are located, and sending server alarm prompt information to a user when the running state of the server is abnormal.
In this optional embodiment, in order to ensure data security, the states of the memory type database, the relational database, and the distributed database need to be monitored in real time, and the states of the servers in which the databases are deployed need to be monitored in real time, specifically, the distributed storage management platform needs to acquire the states of the databases in the memory type database, the relational database, and the distributed database in real time, and when the state of at least one database is abnormal, send a database alarm prompt message to a user, where the abnormal state may be termination of abnormal database service, and the database alarm prompt message may be sent to the user by a short message, a telephone, or a mail, so that the user can find a problem in time, and service stability is improved.
Furthermore, the operation states of the servers in which the memory type database, the relational database and the distributed database are located are also required, and when the operation states of the servers are abnormal, server alarm prompt information is sent to a user, wherein the server operation states may include an operation state, a memory state, an input and output state and the like of a Central Processing Unit (CPU) of the server, and when one or more of the states are abnormal, the server alarm prompt information needs to be sent to the user in a short message, telephone or mail manner, so that the user can find the server problem in time.
Optionally, the technical solution of this embodiment may also manage users and roles, specifically, an administrator may perform user management according to roles of service users, including adding users, querying users, modifying user information, deleting users, and the like, and may also perform authority management and control for users with different roles, for example, a user related to service a may only access a part of data related to service a, but may not access all data in the database, so as to improve data security.
Optionally, the distributed storage management platform may further perform query and analysis on metadata, so as to facilitate management of data. The metadata is data describing data, mainly information describing data attributes, and is used for supporting functions such as indicating storage locations, history data, resource searching, file recording and the like. Metadata can be understood as an electronic catalog, and in order to achieve the purpose of cataloguing, the contents or features of data must be described and collected, so as to achieve the purpose of assisting data retrieval.
Optionally, the distributed storage management platform may further perform visual display on the data query operation of the user, for example, the user may use a visual editor to perform custom query, real-time preview, offline export of cold data, and the like on the basis of the hot data or the warm data, so that the use experience of the user may be improved.
The technical scheme of the embodiment of the invention is that in response to the data writing operation of a user, hot data written by the user is stored in a memory type database, then the hot data in the memory type database is synchronized to a relational database as temperature data by adopting an increment synchronization mode according to a first synchronization frequency, finally the temperature data in the relational database is synchronized to a distributed database as cold data by adopting an increment synchronization mode according to a second synchronization frequency, and the memory type database, the relational database or the distributed database can be cleaned according to a preset database cleaning rule, on one hand, different databases are selected for data storage according to data characteristics to improve the data reading efficiency, on the other hand, the memory type database, the relational database or the distributed database is cleaned regularly according to the database cleaning rule, the data access performance is guaranteed.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention, where the data storage device includes: a hot data write module 310, a warm data synchronization module 320, and a cold data synchronization module 330.
A hot data writing module 310, configured to respond to a data writing operation of a user, and store hot data written by the user in a memory type database;
a temperature data synchronization module 320, configured to synchronize, according to a first synchronization frequency, thermal data in the memory type database to a relational database as temperature data;
and the cold data synchronization module 330 is configured to synchronize the temperature data in the relational database to the distributed database as cold data according to a second synchronization frequency.
According to the technical scheme of the embodiment of the invention, in response to data writing operation of a user, hot data written by the user is stored in the memory type database, then the hot data in the memory type database is synchronized to the relational database as temperature data according to the first synchronization frequency, finally the temperature data in the relational database is synchronized to the distributed database as cold data according to the second synchronization frequency, and the data can be stored in the appropriate database according to data characteristics.
Optionally, the data storage device further includes:
and the database cleaning module is used for cleaning the memory type database, the relational database or the distributed database according to preset database cleaning rules.
Optionally, the database cleaning module includes:
the first synchronous judgment unit is used for determining the failed thermal data in the memory type database according to the preset thermal data failure time and judging whether the failed thermal data are synchronized to the relational database;
the hot data deleting unit is used for deleting the failed hot data in the memory type database when the failed hot data are synchronized to the relational database;
and the thermal data synchronization unit is used for synchronizing the failed thermal data to the relational database and then deleting the failed thermal data in the memory type database when the failed thermal data is not synchronized to the relational database.
Optionally, the database cleaning module further includes:
the second synchronous judgment unit is used for determining the failed temperature data in the relational database according to the preset temperature data failure time and judging whether the failed temperature data are synchronous to the distributed database;
the temperature data deleting unit is used for deleting the failed temperature data in the relational database when the failed temperature data are synchronized to the distributed database;
and the temperature data synchronization unit is used for synchronizing the failed temperature data to the distributed database and then deleting the failed temperature data in the relational database when the failed temperature data are not synchronized to the distributed database.
Optionally, the database cleaning module further includes:
and the cold data deleting unit is used for deleting and/or filing the cold data which is failed in the distributed database according to preset cold data failure time.
Optionally, the temperature data synchronization module 320 is specifically configured to:
according to the first synchronization frequency, synchronizing the thermal data in the memory type database to a relational database as temperature data by adopting an incremental synchronization mode;
the cold data synchronization module 330 is specifically configured to:
and synchronizing the temperature data in the relational database to the distributed database as cold data by adopting an incremental synchronization mode according to the second synchronization frequency.
Optionally, the data storage device further includes:
the database alarm module is used for acquiring the state of each database in the memory type database, the relational database and the distributed database, and sending database alarm prompt information to a user when the state of at least one database is abnormal;
and the server alarm module is used for acquiring the running states of the servers where the memory type database, the relational database and the distributed database are located, and sending server alarm prompt information to a user when the running states of the servers are abnormal.
The data storage device provided by the embodiment of the invention can execute the data storage method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a schematic structural diagram of an electronic apparatus according to a fourth embodiment of the present invention, as shown in fig. 4, the electronic apparatus includes a processor 40, a memory 41, an input device 42, and an output device 43; the number of processors 40 in the device may be one or more, and one processor 40 is taken as an example in fig. 4; the processor 40, the memory 41, the input means 42 and the output means 43 in the device may be connected by a bus or other means, as exemplified by the bus connection in fig. 4.
The memory 41, which is a computer-readable storage medium, may be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data storage method in the embodiment of the present invention (for example, the hot data writing module 310, the warm data synchronization module 320, and the cold data synchronization module 330 in the data storage device). The processor 40 executes various functional applications and data processing of the device by executing software programs, instructions and modules stored in the memory 41, namely, the data storage method is realized, and comprises the following steps:
responding to data writing operation of a user, and storing hot data written by the user into a memory type database;
according to a first synchronization frequency, synchronizing thermal data in the memory type database to a relational database as temperature data;
and synchronizing the temperature data in the relational database to the distributed database as cold data according to the second synchronization frequency.
The memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 41 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 41 may further include memory located remotely from processor 40, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
EXAMPLE five
An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, where the computer program is used for executing a data storage method when executed by a computer processor, and the method includes:
responding to data writing operation of a user, and storing hot data written by the user into a memory type database;
according to a first synchronization frequency, synchronizing thermal data in the memory type database to a relational database as temperature data;
and synchronizing the temperature data in the relational database to the distributed database as cold data according to the second synchronization frequency.
Of course, the storage medium provided by the embodiment of the present invention and containing the computer-executable instructions is not limited to the method operations described above, and may also perform related operations in the data storage method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, an application server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the above embodiment of the data storage device, the included units and modules are merely divided according to the functional logic, but are not limited to the above division as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method of storing data, comprising:
responding to data writing operation of a user, and storing hot data written by the user into a memory type database;
according to a first synchronization frequency, synchronizing thermal data in the memory type database to a relational database as temperature data;
and synchronizing the temperature data in the relational database to the distributed database as cold data according to the second synchronization frequency.
2. The method of claim 1, further comprising:
and cleaning the memory type database, the relational database or the distributed database according to preset database cleaning rules.
3. The method according to claim 2, wherein the cleaning the in-memory database according to a preset database cleaning rule comprises:
determining the failed thermal data in the memory type database according to preset thermal data failure time, and judging whether the failed thermal data are synchronized to the relational database;
if yes, deleting the failed hot data in the memory type database;
if not, synchronizing the failed hot data to a relational database, and deleting the failed hot data in the memory type database.
4. The method according to claim 2, wherein the cleaning the relational database according to preset database cleaning rules comprises:
determining the failed temperature data in the relational database according to preset temperature data failure time, and judging whether the failed temperature data are synchronized to the distributed database;
if yes, deleting the temperature data which are failed in the relational database;
if not, synchronizing the failed temperature data to a distributed database, and deleting the failed temperature data in the relational database.
5. The method of claim 2, wherein the cleaning the distributed database according to a preset database cleaning rule comprises:
and deleting and/or archiving the cold data which is failed in the distributed database according to the preset cold data failure time.
6. The method of claim 1, wherein synchronizing thermal data in the in-memory database as temperature data to a relational database according to a first synchronization frequency comprises:
according to the first synchronization frequency, synchronizing the thermal data in the memory type database to a relational database as temperature data by adopting an incremental synchronization mode;
synchronizing temperature data in the relational database as cold data to a distributed database according to a second synchronization frequency, comprising:
and synchronizing the temperature data in the relational database to the distributed database as cold data by adopting an incremental synchronization mode according to the second synchronization frequency.
7. The method of claim 1, further comprising:
acquiring the state of each database in the memory type database, the relational database and the distributed database, and sending database alarm prompt information to a user when the state of at least one database is abnormal;
and acquiring the running states of the servers in which the memory type database, the relational database and the distributed database are located, and sending server alarm prompt information to a user when the running state of the server is abnormal.
8. A data storage device, comprising:
the hot data writing module is used for responding to data writing operation of a user and storing hot data written by the user into the memory type database;
the temperature data synchronization module is used for synchronizing the thermal data in the memory type database to the relational database as temperature data according to a first synchronization frequency;
and the cold data synchronization module is used for synchronizing the temperature data in the relational database to the distributed database as cold data according to a second synchronization frequency.
9. An electronic device, characterized in that the device comprises:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data storage method as claimed in any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data storage method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110794255.XA CN113420021A (en) | 2021-07-14 | 2021-07-14 | Data storage method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110794255.XA CN113420021A (en) | 2021-07-14 | 2021-07-14 | Data storage method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113420021A true CN113420021A (en) | 2021-09-21 |
Family
ID=77720970
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110794255.XA Pending CN113420021A (en) | 2021-07-14 | 2021-07-14 | Data storage method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113420021A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114328466A (en) * | 2022-01-06 | 2022-04-12 | 北京微吼时代科技有限公司 | Data cold and hot storage method and device and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106934001A (en) * | 2017-03-03 | 2017-07-07 | 广州天源迪科信息技术有限公司 | Distributed quick inventory inquiry system and method |
| CN109857737A (en) * | 2019-01-03 | 2019-06-07 | 平安科技（深圳）有限公司 | A kind of cold and hot date storage method and device, electronic equipment |
| CN110083649A (en) * | 2019-04-24 | 2019-08-02 | 北京电子工程总体研究所 | It is a kind of based on cold, warm, dsc data data management system and data managing method |
| CN110134723A (en) * | 2019-05-22 | 2019-08-16 | 网易（杭州）网络有限公司 | A kind of method and database of storing data |
| CN111460236A (en) * | 2020-04-26 | 2020-07-28 | 天津七一二通信广播股份有限公司 | Big data acquisition administers quick retrieval system based on data lake |
| CN111538463A (en) * | 2020-04-30 | 2020-08-14 | 电子科技大学 | Method for realizing memory database persistence |
-
2021
- 2021-07-14 CN CN202110794255.XA patent/CN113420021A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106934001A (en) * | 2017-03-03 | 2017-07-07 | 广州天源迪科信息技术有限公司 | Distributed quick inventory inquiry system and method |
| CN109857737A (en) * | 2019-01-03 | 2019-06-07 | 平安科技（深圳）有限公司 | A kind of cold and hot date storage method and device, electronic equipment |
| CN110083649A (en) * | 2019-04-24 | 2019-08-02 | 北京电子工程总体研究所 | It is a kind of based on cold, warm, dsc data data management system and data managing method |
| CN110134723A (en) * | 2019-05-22 | 2019-08-16 | 网易（杭州）网络有限公司 | A kind of method and database of storing data |
| CN111460236A (en) * | 2020-04-26 | 2020-07-28 | 天津七一二通信广播股份有限公司 | Big data acquisition administers quick retrieval system based on data lake |
| CN111538463A (en) * | 2020-04-30 | 2020-08-14 | 电子科技大学 | Method for realizing memory database persistence |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114328466A (en) * | 2022-01-06 | 2022-04-12 | 北京微吼时代科技有限公司 | Data cold and hot storage method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022126974A1 (en) | Kafka-based incremental data synchronization method and apparatus, device, and medium | |
| EP3477914B1 (en) | Data recovery method and device, and cloud storage system | |
| KR20180021679A (en) | Backup and restore from a distributed database using consistent database snapshots | |
| CN103761165B (en) | Log backup method and device | |
| US9152683B2 (en) | Database-transparent near online archiving and retrieval of data | |
| US10108634B1 (en) | Identification and removal of duplicate event records from a security information and event management database | |
| JP6633642B2 (en) | Method and device for processing data blocks in a distributed database | |
| CN103793479A (en) | Log management method and log management system | |
| CN111651519B (en) | Data synchronization method, data synchronization device, electronic equipment and storage medium | |
| CN102426609A (en) | Index generation method and index generation device based on MapReduce programming architecture | |
| CN103812939A (en) | Big data storage system | |
| CN103595797B (en) | Caching method for distributed storage system | |
| CN102722584B (en) | Data storage system and method | |
| WO2012083754A1 (en) | Method and device for processing dirty data | |
| CN109299115A (en) | A kind of date storage method, device, server and storage medium | |
| US20210303537A1 (en) | Log record identification using aggregated log indexes | |
| CN111522880A (en) | Method for improving data read-write performance based on mysql database cluster | |
| CN111046036A (en) | Data synchronization method, device, system and storage medium | |
| US10133820B2 (en) | Techniques for performing intelligent content indexing | |
| CN113420021A (en) | Data storage method, device, equipment and medium | |
| Murugesan et al. | Audit log management in MongoDB | |
| CN113239012A (en) | Database migration method and device, electronic equipment and storage medium | |
| CN107004036B (en) | Method and system for searching logs containing a large number of entries | |
| CN110245037B (en) | Hive user operation behavior restoration method based on logs | |
| CN107894942B (en) | Method and device for monitoring data table access amount |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |