CN115774702A - Distributed database implementation method and device, electronic equipment and storage medium - Google Patents

Abstract

The disclosure provides a distributed database implementation method and device, electronic equipment and a storage medium, relates to the field of artificial intelligence such as cloud computing, cloud storage and a cloud database, and can be applied to an intelligent cloud scene. The method comprises the following steps: responding to the fact that the write operation aiming at the distributed database needs to be executed, writing data to be written into a local distributed file system, and simultaneously hanging the local distributed file system and an object storage system on the distributed database; in response to determining that a main merge operation needs to be performed on any storage unit in the distributed database, merging files belonging to the storage unit in the local distributed file system and the object storage system, and storing a new file obtained by merging into the object storage system, wherein the storage unit that needs to perform the main merge operation includes: a storage unit that meets an entry cold storage condition. By applying the scheme disclosed by the disclosure, the storage cost can be reduced under the condition of ensuring the read-write performance, and the like.

Description

Distributed database implementation method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of artificial intelligence technologies, and in particular, to a distributed database implementation method and apparatus, an electronic device, and a storage medium in the fields of cloud computing, cloud storage, and cloud databases.

Background

A distributed database (HBase, hadoop database) is a distributed and column-oriented open source database, and with the development of big data technology, the distributed database is widely used in the industry, and the deployment of the distributed database is usually complex, and the storage cost is high.

Disclosure of Invention

The disclosure provides a distributed database implementation method, a distributed database implementation device, electronic equipment and a storage medium.

A distributed database implementation method comprises the following steps:

in response to the fact that the write operation aiming at the distributed database needs to be executed, writing data to be written into a local distributed file system, wherein the local distributed file system and an object storage system are simultaneously hung on the distributed database;

in response to determining that a main merge operation needs to be performed on any storage unit in the distributed database, merging files belonging to the storage unit in the local distributed file system and the object storage system, and storing a new file obtained by merging into the object storage system, wherein the storage unit that needs to perform the main merge operation includes: a storage unit that meets an entry cold storage condition.

A distributed database implementation apparatus, comprising: the system comprises a first execution module and a second execution module;

the first execution module is used for responding to the fact that the writing operation aiming at the distributed database needs to be executed, writing data to be written into a local distributed file system, and the distributed database is simultaneously mounted with the local distributed file system and an object storage system;

the second execution module is configured to, in response to determining that a main merge operation needs to be executed for any storage unit in the distributed database, merge files belonging to the storage unit in the local distributed file system and the object storage system, and store a new file obtained by the merge into the object storage system, where a storage unit that needs to execute the main merge operation includes: and (4) storage units meeting the condition of entering cold storage.

An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described above.

A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method as described above.

A computer program product comprising computer programs/instructions which, when executed by a processor, implement a method as described above.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a flow chart of an embodiment of the HBase implementation method of the present disclosure;

FIG. 2 is a schematic diagram of the relationship among the RegionServer, store, local HDFS, object storage system, etc. according to the present disclosure;

FIG. 3 is a schematic diagram of a process for executing a MajorCompaction operation and updating a corresponding meta cache according to the present disclosure;

fig. 4 is a schematic structural diagram of a first embodiment 400 of an HBase implementing apparatus according to the present disclosure;

fig. 5 is a schematic structural diagram of a second embodiment 500 of the HBase implementing apparatus according to the present disclosure;

FIG. 6 illustrates a schematic block diagram of an electronic device 600 that may be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In addition, it should be understood that the term "and/or" herein is only one kind of association relationship describing an associated object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

Fig. 1 is a flowchart of an embodiment of the HBase implementation method according to the present disclosure. As shown in fig. 1, the following detailed implementation is included.

In step 101, in response to determining that a write operation for the HBase needs to be performed, data to be written is written into a local distributed file system (HDFS), and the HBase is simultaneously mounted with the local HDFS and an object storage (ObjectStorage) system.

In step 102, in response to determining that a main merge (MajorCompaction) operation needs to be performed on any storage unit (Store) in the HBase, merging the local HDFS and the files (hfiles) belonging to the Store in the object storage system, and storing the new HFile obtained by merging into the object storage system, where the Store that needs to perform the main merge operation includes: store meeting the entry cold storage condition.

By adopting the scheme of the method embodiment, the relative hot data can be stored in the local HDFS, the read-write performance is ensured by utilizing the characteristic of high read-write speed of the local HDFS, and the cold data can be stored in the object storage system, so that the storage cost is reduced, the respective advantages of the local HDFS and the object storage system are taken into consideration, and the performance of HBase is improved.

The object storage, which may also be referred to as object-based storage, is a general term used to describe a method for resolving and processing discrete units, where the discrete units may be referred to as objects, the object storage may support various types of data storage, such as text, multimedia, binary, and the like, and is suitable for various scenarios, such as data distribution, data backup, big data analysis, and audio/video file processing, and the object storage generally has a hierarchical storage function, that is, cold data may be gradually sunk into a more cost-effective storage medium according to an access frequency of the data, so as to reduce storage cost.

The local HDFS is an original bottom storage mode of the HBase, and in the scheme disclosed by the invention, an object storage system is further added, namely the local HDFS and the object storage system are mounted on the HBase simultaneously.

Preferably, the following operations for HBase are performed in response to determining that: the operation of the file tree structure is affected, which can be done in a double write manner.

Operations that affect the file tree structure may include creating tables, deleting tables, region (region) splitting, and region migration, among others.

By adopting a double-write mode, namely double-write operation, the consistency of the local HDFS and the file tree structure in the object storage system can be ensured, and because the proportion of operations influencing the file tree structure in the total operands is very small, the time delay caused by the double-write operation can be almost ignored.

When a write operation for the HBase needs to be executed, the data to be written can be written into the local HDFS. The write-ahead log (WAL) and the write cache (memstore) may be written in the original path, as may be conventionally done, and then flushed (flush) into the local HDFS, etc.

When determining that the MajorCompaction operation needs to be executed on any Store in the HBase, the local HDFS and the HFile belonging to the Store in the object storage system may be merged, and the new HFile obtained by merging may be stored in the object storage system.

For HBase, the best reading efficiency is usually achieved when each Store contains only one HFile, so HBase improves the reading speed by combining the existing hfiles to reduce the disk seek time for each reading data, and this combination process is called compact.

The compact operation of HBase can be divided into two types, i.e., a Minor compact operation and a MajorCompaction operation. The determination of when to perform a compact operation and the specific type of compact operation performed may be based on a predetermined policy.

When the MinorCompaction operation needs to be executed, only the HFiles on the local HDFS can be merged in a conventional manner, such as automatically checking small files and merging the small files into large files, thereby reducing fragmented files and the like.

When the MajorCompation operation needs to be executed, the HFiles belonging to the same Store in the local HDFS and the object storage system can be merged, and the new HFile obtained by merging can be stored in the object storage system. That is, the MajorCompaction operation will take out all hfiles in the same Store in the local HDFS and the object storage system, merge them to obtain a new HFile, and then write the new HFile into the object storage system. Accordingly, the old hfiles before merging will not be retained.

The compact operation can be completed by a special compact thread, and the reading of data and the like are not influenced, namely, the user can still acquire the data from the old HFile.

Preferably, each Store in the HBase may be separately monitored in real time, and in response to determining that any of the monitored stores meets the entry into cold storage condition, the Store may be determined as a Store for which the Major compact operation needs to be performed.

In practical applications, a monitoring thread may be started for monitoring the state of each Store in real time, and determining whether each Store meets the condition of entering cold storage in real time, and if any Store meets the condition of entering cold storage, a MajorCompaction operation for the Store may be triggered, so as to send the HFile of the Store into the object storage system.

The monitoring thread may be created when started by a region server (RegionServer). The specific condition which meets the cold storage entering condition may be determined according to actual needs, and may be predefined, for example, for any Store, if it is determined that the difference between the last reading time of the Store and the current time is greater than a predetermined threshold, it may be determined that the Store meets the cold storage entering condition, and for example, if the local HDFS space is about to be exhausted, that is, the remaining space is less than a predetermined threshold, it may also be considered that the Store meets the cold storage entering condition, and a specific value of the threshold may also be determined according to actual needs.

Through the processing, the separation of cold and hot data can be realized, and the storage cost is further reduced.

Preferably, for any Store, when MajorCompaction operation is performed on the local HDFS and the HFile belonging to the Store in the object storage system, the file belonging to the Store in the local HDFS may be determined according to a file tree structure in the local HDFS, the file belonging to the Store in the object storage system may be determined according to a file tree structure in the object storage system, the file tree structures in the local HDFS and the object storage system are consistent, and accordingly, the determined hfiles belonging to the Store may be merged.

Through the processing, the accuracy of the acquired HFile can be ensured, and the accuracy of the operation result of the Major compact operation is further improved.

In addition, preferably, for any Store, when performing MajorCompaction operation on the local HDFS and the HFile belonging to the Store in the object storage system, the HFile belonging to the Store in the local HDFS may be first obtained, and the obtained HFile may be merged for the first time, where the HFile belonging to the Store in the object storage system may be obtained while the first merging is performed, and then, the first merging result and the obtained HFile belonging to the Store in the object storage system may be merged for the second time, and the second merging result may be stored in the object storage system as a new HFile.

Generally, the latency of the object storage system is large, so that the HFile in the local HDFS may be merged first, that is, the HFile is merged for the first time, and then the HFile is merged for the second time with the HFile in the object storage system, where the HFile may be obtained from the object storage system by using the time of the first merging, so as to save the time of waiting for reading the file of the object storage system, thereby improving the merging efficiency.

Preferably, in response to determining that a read operation for HBase needs to be performed, the view objects may be traversed sequentially in the following order: the cache comprises a write cache and a read cache of HBase, and for the traversed viewed object, in response to determining that a read result corresponding to the read operation exists, the read result can be obtained from the viewed object and returned.

That is, when it is determined that a read operation for the HBase needs to be performed, the cache may be checked first to determine whether a hit occurs, where the cache may include a write cache and a read cache of the HBase, and if the write cache may be a memstore, the read cache may be a block cache (blockcache) and a bucket cache (buckettcache), and if the hit occurs, a read result may be obtained from the cache and returned.

If the cache fails to hit, whether the HFile in the local HDFS hits can be checked, and if the HFile hits, the reading result can be obtained and returned.

If the HFile in the local HDFS fails to hit, whether the object storage system hits or not can be further checked, and if the object storage system hits, a reading result can be obtained from the object storage system and returned.

Can look over each object of looking over in proper order by high to low according to reading efficiency, preferentially use the object of looking over that reading efficiency is higher to obtain the reading result to promoted the acquisition efficiency of reading result, and then promoted read operation's response speed etc..

If the checking objects are not hit after being checked in sequence, information such as 'not-acquired reading result' or 'reading failure' can be returned to prompt the user.

Preferably, when the object is viewed as the object storage system, whether a reading result exists in the object storage system can be determined by querying a metadata cache (metadata) in the local HDFS, and in response to determining that the reading result exists, the reading result can be obtained from the object storage system and returned, where each Store may correspond to one metadata respectively and is used for storing directory information of hfiles belonging to the Store in the object storage system.

The directory information may include an index block (indexblock) and a bloom block (DataBlock), the HFile is composed of a plurality of blocks, such as indexblock, bloomblock, and data block (DataBlock), and the actual data is stored in the DataBlock.

Because the latency of the object storage system is large, if directory information is stored in the object storage system, HFile in the object storage system may need to be frequently read, and performance will be greatly reduced, in the solution of the present disclosure, for each Store, a meta cache may be respectively created in the local HDFS for storing metadata (metadata) such as indexblock and bloom block belonging to the HFile of the Store in the object storage system, in consideration of that the data amount of the meta is usually small, so that creating the meta cache in the local HDFS does not occupy too much extra space, and in addition, if the space of the block cache is relatively abundant, the meta cache of HFile in the object storage system may also be considered to reside.

Preferably, the meta corresponding to any Store may be regenerated in response to determining that the MajorCompaction operation is performed for that Store, and/or the meta corresponding to each Store may be regenerated in response to determining that the RegionServer is restarted.

In addition, in order to prevent the transactions from running to half of a crash, which causes the HFile in the object storage system to be inconsistent with the meta, the meta of each Store can be regenerated according to the directory information acquired from the object storage system each time the registry server is restarted, and the operation can be asynchronous, so as to accelerate the speed of the registry server, and the like.

Preferably, when it is determined that a read operation for the HBase needs to be performed, in response to determining that no meta exists in the local HDFS, the meta may also be generated first, and then the generated meta cache is queried to determine whether a read result exists in the object storage system.

The required directory information can be acquired from the object storage system, then the corresponding metacache can be generated according to the acquired directory information and stored in the local HDFS, and whether the required reading result exists in the object storage system can be determined by inquiring the generated metacache, so that the success rate of returning the reading result to a user is improved, and the like.

The above-mentioned situation usually only occurs when the RegionServer just restarts to complete or the Major compact operation just ends, and accordingly, the required meta cache can be generated by using the corresponding thread.

With the above description, fig. 2 is a schematic diagram of the relationship among the RegionServer, the Store, the local HDFS, the object storage system, and the like according to the present disclosure.

As shown in fig. 2, each RegionServer may correspond to one or more stores, and for simplifying the drawing, only one Store is shown in fig. 2, each Store may include a memstore and one or more storage files (Store files), each Store file corresponds to one HFile in the local HDFS, that is, it is understood that the Store files and the hfiles are substantially the same, and only called Store files in the Store and hfiles in the local HDFS, and each Store file may be used as an object in the object storage system, and further, a meta may be additionally added in the local HDFS.

Each regionServer can respectively correspond to a blockcache and a buckatcache, which are not shown in the figure for simplifying the figure.

FIG. 3 is a schematic diagram of the process of performing the MajorCompaction operation and updating the corresponding metacache according to the present disclosure. As shown in fig. 3, 1 indicates that the first merge is performed, 2 indicates that the second merge is performed, 3 indicates that the second merge result is written into the object storage system, 4 indicates that the original meta cache is deleted, and 5 indicates that a new meta cache is regenerated. It can be seen that for each Store, there is only one HFile in the object storage system for the most part of the time, due to the MajorCompaction operation.

It should be noted that for simplicity of description, the aforementioned method embodiments are presented as a series of combinations of acts, but those skilled in the art will appreciate that the present disclosure is not limited by the order of acts, as some steps may, in accordance with the present disclosure, occur in other orders and/or concurrently. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that acts and modules are not required for the present disclosure.

In a word, by adopting the scheme of the embodiment of the method, the storage cost can be reduced under the condition of ensuring the read-write performance of the HBase as much as possible, so that the performance and the cost performance of the HBase are improved, and in addition, the file tree structure of the HBase is adopted for cold storage, so that an object storage system can be conveniently converted into an offline data warehouse, a data source is provided for big data analysis, and the like.

The above is a description of embodiments of the method, and the embodiments of the apparatus are further described below.

Fig. 4 is a schematic structural diagram of a first embodiment 400 of the HBase implementation apparatus according to the present disclosure. As shown in fig. 4, includes: a first execution module 401 and a second execution module 402.

The first execution module 401 is configured to, in response to determining that a write operation for the HBase needs to be executed, write data to be written into the local HDFS, where the HBase is simultaneously mounted with the local HDFS and the object storage system.

A second executing module 402, configured to, in response to determining that a MajorCompaction operation needs to be performed on any Store in the HBase, merge the local HDFS and the HFile belonging to the Store in the object storage system, and Store a new HFile obtained by the merging into the object storage system, where the Store that needs to perform the majorcoction operation includes: store meeting the entry cold storage condition.

By adopting the scheme of the embodiment of the device, relative hot data can be stored in the local HDFS, the read-write performance is ensured by utilizing the characteristic of high read-write speed of the local HDFS, and cold data can be stored in the object storage system, so that the storage cost is reduced, the respective advantages of the local HDFS and the object storage system are taken into consideration, and the performance of HBase is improved.

When a write operation for the HBase needs to be performed, the first execution module 401 may write data to be written into the local HDFS, for example, the write may be completed in a conventional manner.

When determining that the majorcoction operation needs to be executed on any Store in the HBase, the second execution module 402 may merge the local HDFS and the HFile belonging to the Store in the object storage system, and may Store the new HFile obtained through merging into the object storage system.

The compact operation of HBase can be divided into two types, i.e., minorcompact operation and Majorcompact operation. The determination of when to perform a compact operation and the specific type of compact operation performed may be based on a predetermined policy.

When the MinorCompaction operation needs to be executed, the second execution module 402 may merge HFiles on the local HDFS only in a conventional manner, for example, automatically check small files and merge the small files into large files, so as to reduce fragmented files and the like.

When the MajorCompaction operation needs to be executed, the second execution module 402 may merge the local HDFS and the hfiles belonging to the same Store in the object storage system, and may Store the new HFile obtained by the merging into the object storage system. That is, the MajorCompaction operation will take out all hfiles in the same Store in the local HDFS and the object storage system, merge them to obtain a new HFile, and then write the new HFile into the object storage system. Accordingly, the old hfiles before merging will not be retained.

Preferably, the second execution module 402 may monitor each Store in the HBase separately in real time, and in response to determining that any of the monitored stores meets the entry into cold storage condition, may determine that the Store is a Store for which the MajorCompaction operation needs to be performed.

In practical application, a monitoring thread can be started for monitoring the state of each Store in real time, and determining whether each Store meets the condition of entering cold storage in real time, if any Store meets the condition of entering cold storage, a MajorCompaction operation aiming at the Store can be triggered.

Preferably, for any Store, when performing majorcoction operation on the local HDFS and the HFile belonging to the Store in the object storage system, the second execution module 402 may determine, according to the file tree structure in the local HDFS, the file belonging to the Store in the local HDFS, and determine, according to the file tree structure in the object storage system, the file belonging to the Store in the object storage system, where the file tree structures in the local HDFS and the object storage system are consistent, and accordingly, may merge the determined HFile belonging to the Store.

Preferably, the second execution module 402, in response to determining that the following operations for HBase need to be performed: the operation of the file tree structure is affected, which can be done in a double write manner.

In addition, preferably, for any Store, when performing majorcoction operation on the local HDFS and the HFile belonging to the Store in the object storage system, the second execution module 402 may first obtain the HFile belonging to the Store in the local HDFS, and may perform first merging on the obtained HFile, where the HFile belonging to the Store in the object storage system may be obtained while performing the first merging, and then may perform second merging on the first merging result and the obtained HFile belonging to the Store in the object storage system, and may Store the second merging result as a new HFile in the object storage system.

Fig. 5 is a schematic structural diagram of a second embodiment 500 of the HBase implementing apparatus according to the present disclosure. As shown in fig. 5, includes: a first execution module 401, a second execution module 402, and a third execution module 403.

The first execution module 401 and the second execution module 402 are the same as those in the embodiment shown in fig. 4.

A third executing module 403, configured to, in response to determining that a read operation for the HBase needs to be executed, sequentially traverse the view objects in the following order: caching, HFile in a local HDFS, and an object storage system; the cache comprises a write cache and a read cache of HBase, and for the traversed view object, in response to determining that a read result corresponding to the read operation exists, the read result can be obtained from the view object and returned.

That is, when it is determined that a read operation for the HBase needs to be performed, the cache may be checked first to determine whether a hit occurs, where the cache may include a write cache and a read cache of the HBase, and if a hit occurs, a read result may be obtained from the cache and returned.

If the cache fails to hit, whether the HFile in the local HDFS hits can be checked, and if the HFile hits, the reading result can be obtained and returned.

If the HFile in the local HDFS fails to hit, whether the object storage system hits or not can be further checked, and if the object storage system hits, a reading result can be obtained from the object storage system and returned.

Preferably, when the object is the object storage system, the third executing module 403 may determine whether a read result exists in the object storage system by querying a meta in the local HDFS, and in response to the determination result being present, may obtain the read result from the object storage system and return the read result, where each Store may correspond to one meta, and is used to Store directory information of the HFile belonging to the Store in the object storage system. The directory information may include indexblock and blockomblock.

Preferably, the third execution module 403 may regenerate the meta corresponding to any Store in response to determining that the Major composition operation is executed for the Store, and/or may regenerate the meta corresponding to each Store in response to determining that the regionServer is restarted.

Preferably, the third execution module 403 determines that a read operation for the HBase needs to be executed, and in response to determining that no meta exists in the local HDFS, may also generate meta first, and then determine whether a read result exists in the object storage system by querying the generated meta.

The specific work flow of the device embodiments shown in fig. 4 and fig. 5 may refer to the related description in the foregoing method embodiments, and is not repeated.

In a word, by adopting the scheme of the embodiment of the device disclosed by the disclosure, the storage cost can be reduced under the condition of ensuring the read-write performance of the HBase as much as possible, so that the performance and the cost performance of the HBase are improved, and in addition, the file tree structure of the HBase is adopted for cold storage, so that an object storage system can be conveniently converted into an offline data warehouse, a data source is provided for big data analysis, and the like.

The scheme disclosed by the invention can be applied to the field of artificial intelligence, in particular to the fields of cloud computing, cloud storage, cloud databases and the like. Artificial intelligence is a subject for studying a computer to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning and the like) of a human, and has a hardware technology and a software technology, the artificial intelligence hardware technology generally comprises technologies such as a sensor, a special artificial intelligence chip, cloud computing, distributed storage, big data processing and the like, and the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, machine learning/deep learning, a big data processing technology, a knowledge graph technology and the like.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 6 illustrates a schematic block diagram of an electronic device 600 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, servers, blade servers, mainframes, and other appropriate computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 6, the device 600 comprises a computing unit 601, which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The computing unit 601 performs the various methods and processes described above, such as the methods described in this disclosure. For example, in some embodiments, the methods described in this disclosure may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of a computer program may be loaded onto and/or installed onto device 600 via ROM602 and/or communications unit 609. When the computer program is loaded into RAM603 and executed by the computing unit 601, one or more steps of the methods described in the present disclosure may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured by any other suitable means (e.g., by means of firmware) to perform the methods described in the present disclosure.

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions/acts specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims (21)

1. A distributed database implementation method comprises the following steps:

in response to the fact that the write operation aiming at the distributed database needs to be executed, writing data to be written into a local distributed file system, wherein the local distributed file system and an object storage system are simultaneously hung on the distributed database;

in response to determining that a main merge operation needs to be performed on any storage unit in the distributed database, merging files belonging to the storage unit in the local distributed file system and the object storage system, and storing a new file obtained by merging into the object storage system, wherein the storage unit that needs to perform the main merge operation includes: a storage unit that meets an entry cold storage condition.

2. The method of claim 1, wherein,

the merging the files belonging to the storage unit in the local distributed file system and the object storage system includes:

determining files belonging to the storage unit in the local distributed file system according to the file tree structure in the local distributed file system, and determining files belonging to the storage unit in the object storage system according to the file tree structure in the object storage system, wherein the file tree structures in the local distributed file system and the object storage system are consistent;

and merging the determined files belonging to the storage unit.

3. The method of claim 2, further comprising:

in response to determining that the following operations need to be performed with respect to the distributed database: and (4) influencing the operation of the file tree structure, and finishing the operation by adopting a double-write mode.

4. The method of claim 1, wherein,

the determining that a primary merge operation needs to be performed for any storage unit in the distributed database comprises:

respectively monitoring each storage unit in the distributed database in real time;

in response to determining that any of the monitored storage units meets the incoming cold storage condition, determining the storage unit as a storage unit that needs to perform the primary merge operation.

5. The method of claim 1, wherein,

the merging the files belonging to the storage unit in the local distributed file system and the object storage system, and storing the new file obtained by merging into the object storage system comprises:

acquiring files belonging to the storage unit in the local distributed file system, and performing first combination on the acquired files, wherein the files belonging to the storage unit in the object storage system are acquired while the first combination is performed;

and performing second merging on the first merging result and the acquired files belonging to the storage unit in the object storage system, and storing the second merging result as the new file into the object storage system.

6. The method of any of claims 1-5, further comprising:

in response to determining that a read operation for the distributed database needs to be performed, sequentially traversing the view objects in the following order: caching, files in the local distributed file system, the object storage system; the cache comprises a write cache and a read cache of the distributed database;

and for the traversed viewing object, in response to determining that a reading result corresponding to the reading operation exists, obtaining the reading result from the viewing object and returning.

7. The method of claim 6, wherein,

when the view object is the object storage system, the obtaining the read result from the view object and returning, in response to determining that the read result corresponding to the read operation exists for the traversed view object, includes:

determining whether the reading result exists in the object storage system by inquiring a metadata cache in the local distributed file system, responding to the determination result that the reading result exists, and acquiring the reading result from the object storage system and returning the reading result;

and each storage unit corresponds to one metadata cache respectively and is used for storing directory information of files belonging to the storage unit in the object storage system.

8. The method of claim 7, further comprising:

in response to determining that the primary merge operation is completed for any storage unit, regenerating the metadata cache corresponding to the storage unit;

and/or, in response to the restart of the server in the determined area, regenerating the metadata cache corresponding to each storage unit.

9. The method of claim 7, further comprising:

and generating the metadata cache in response to determining that the metadata cache does not exist in the local distributed file system, and determining whether the read result exists in the object storage system by querying the generated metadata cache.

10. A distributed database implementation apparatus, comprising: a first execution module and a second execution module;

the first execution module is used for responding to the fact that the write operation aiming at the distributed database needs to be executed, writing data to be written into a local distributed file system, and the distributed database is simultaneously provided with the local distributed file system and an object storage system in a hanging mode;

the second execution module is configured to, in response to determining that a main merge operation needs to be executed for any storage unit in the distributed database, merge files belonging to the storage unit in the local distributed file system and the object storage system, and store a new file obtained by the merge into the object storage system, where a storage unit that needs to execute the main merge operation includes: a storage unit that meets an entry cold storage condition.

11. The apparatus of claim 10, wherein,

the second execution module determines the files belonging to the storage unit in the local distributed file system according to the file tree structure in the local distributed file system, determines the files belonging to the storage unit in the object storage system according to the file tree structure in the object storage system, and merges the determined files belonging to the storage unit when the file tree structure in the local distributed file system is consistent with the file tree structure in the object storage system.

12. The apparatus of claim 11, wherein,

the second execution module is further to, in response to determining that the following operations are required to be performed with respect to the distributed database: and (4) influencing the operation of the file tree structure, and finishing the operation in a double-write mode.

13. The apparatus of claim 10, wherein,

and the second execution module is used for monitoring each storage unit in the distributed database in real time respectively, responding to the condition that any monitored storage unit meets the cold storage entering condition, and determining the storage unit as the storage unit needing to execute the main merging operation.

14. The apparatus of claim 10, wherein,

and the second execution module acquires the files belonging to the storage unit in the local distributed file system and performs first combination on the acquired files, wherein the files belonging to the storage unit in the object storage system are acquired while the first combination is performed, the first combination result and the acquired files belonging to the storage unit in the object storage system are subjected to second combination, and the second combination result is stored in the object storage system as the new file.

15. The apparatus of any of claims 10-14, further comprising:

a third executing module, configured to, in response to determining that a read operation for the distributed database needs to be executed, sequentially traverse the view objects in the following order: the cache comprises a write cache and a read cache of the distributed database, and for the traversed viewed object, in response to determining that a read result corresponding to the read operation exists, the read result is obtained from the viewed object and returned.

16. The apparatus of claim 15, wherein,

when the viewed object is the object storage system, the third execution module determines whether the read result exists in the object storage system by querying a metadata cache in the local distributed file system, and in response to the determination result existing, obtains the read result from the object storage system and returns the read result, and each storage unit corresponds to one metadata cache respectively and is used for storing directory information of files belonging to the storage unit in the object storage system.

17. The apparatus of claim 16, wherein,

the third executing module is further configured to, in response to determining that the main merge operation is completed for any storage unit, regenerate the metadata cache corresponding to the storage unit, and/or, in response to determining that the regional server is restarted, regenerate the metadata cache corresponding to each storage unit.

18. The apparatus of claim 16, wherein,

the third execution module is further configured to generate the metadata cache in response to determining that the metadata cache does not exist in the local distributed file system, and determine whether the read result exists in the object storage system by querying the generated metadata cache.

19. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-9.

21. A computer program product comprising a computer program/instructions which, when executed by a processor, implement the method of any one of claims 1-9.

Images