CN116302140B - Method and device for starting computing terminal based on storage and calculation separation cloud primary number bin - Google Patents
Method and device for starting computing terminal based on storage and calculation separation cloud primary number bin Download PDFInfo
- Publication number
- CN116302140B CN116302140B CN202310527772.XA CN202310527772A CN116302140B CN 116302140 B CN116302140 B CN 116302140B CN 202310527772 A CN202310527772 A CN 202310527772A CN 116302140 B CN116302140 B CN 116302140B
- Authority
- CN
- China
- Prior art keywords
- metadata
- engine
- storage
- data
- information set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/4401—Bootstrapping
- G06F9/4403—Processor initialisation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a method and a device for starting a computing terminal based on a storage and calculation separation cloud primary number bin. One embodiment of the method comprises the following steps: in response to receiving a computing terminal starting instruction, initializing a built-in persistent storage engine to obtain an engine storage metadata information set, wherein the engine storage metadata information set is a metadata information set which is stored by a data computing terminal set and aims at persistent data when the data computing terminal set performs data persistence; acquiring required metadata catalog information for a starting instruction of the computing terminal; according to the metadata information set stored by the engine and the required metadata catalog information, loading the metadata information set corresponding to the required metadata catalog information; and executing the starting operation of the computing terminal according to the metadata information set. The implementation mode is related to cloud computing, and the starting of the data computing terminal can be realized quickly and efficiently.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a method and a device for starting a computing terminal based on a storage and calculation separation cloud primary number bin.
Background
At present, related technologies of cloud primary number bins based on deposit separation are widely applied to daily life. For the start-up of a computing node (i.e. a data technology terminal) the following is generally adopted: the computing node needs to scan the underlying mass metadata files to enable the computing node to be started.
However, the inventors have found that when the computing node is started in the above manner, there are often the following technical problems:
for scanning of massive metadata files, if the IOPS (Input/Output Operations Per Second) of the corresponding disk is not high enough, the starting time of the subsequent computing nodes is long, and the starting efficiency is low.
The above information disclosed in this background section is only for enhancement of understanding of the background of the inventive concept and, therefore, may contain information that does not form the prior art that is already known to those of ordinary skill in the art in this country.
Disclosure of Invention
The disclosure is in part intended to introduce concepts in a simplified form that are further described below in the detailed description. The disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose a method and an apparatus for starting a computing terminal based on a storage and separation cloud primary number bin, to solve the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a method for starting a computing terminal based on a storage computing separation cloud primary number bin, including: in response to receiving a computing terminal starting instruction, initializing a built-in persistence storage engine to obtain an engine storage metadata information set, wherein the engine storage metadata information set is a metadata information set which is stored by a data computing terminal set and aims at persistence data when data persistence is executed; acquiring required metadata catalog information aiming at the starting instruction of the computing terminal; according to the engine storage metadata information set and the required metadata catalog information, loading the metadata information set corresponding to the required metadata catalog information; and executing the starting operation of the computing terminal according to the metadata information set.
Optionally, the storing, according to the engine, a metadata information set and the required metadata directory information, and loading the metadata information set corresponding to the required metadata directory information, includes: determining differential metadata catalog information between the engine storage metadata catalog information set and the desired metadata catalog information; and in response to determining that the difference metadata catalog information is not empty, reading a difference metadata information set corresponding to the difference metadata catalog information from the data storage terminal.
Optionally, after the reading, from the data storage terminal, the differential metadata information set corresponding to the differential metadata directory information in response to determining that the differential metadata directory information is not empty, the method further includes: in response to determining that the differential metadata directory information is empty, the engine storage metadata information set is determined to be the metadata information set.
Optionally, the engine storage metadata information set is obtained by: and in response to receiving an engine storage metadata information reading request, acquiring the engine storage metadata information set from an engine storage metadata file, wherein the engine storage metadata file is a file which is deployed in advance in a data storage terminal, each persistent storage engine in a persistent storage engine set corresponds to the engine storage metadata file with the same data sharing, and a data computing end in the data computing end set corresponds to the persistent storage engine in the persistent storage engine set.
Optionally, a built-in target persistent storage engine exists at the data computing expansion end of the data computing end set, and the target persistent storage engine and the persistent storage engine correspondingly have the same data sharing and store metadata files.
Optionally, the engine storage metadata information in the engine storage metadata file is added by: in response to receiving the data persistence instruction, determining a data computing end which performs metadata writing in and is concentrated by the data computing end as a target data computing end; in response to determining that the target data computing end works normally, indicating a built-in persistent storage engine of the target data computing end to store metadata information corresponding to the persistent data, and obtaining engine storage metadata information; and instructing the target data computing end to store the engine storage metadata information into the corresponding file position in the engine storage metadata file.
Optionally, the method further comprises: in response to determining that the target data computing end is abnormal in operation, determining a second target data computing end in the set of data computing ends as a data computing end for performing metadata writing so as to add metadata information stored by an engine.
In a second aspect, some embodiments of the present disclosure provide a computing terminal starting device based on a storage computing separation cloud primary number bin, including: an initialization unit configured to initialize a built-in persistent storage engine in response to receiving a computing terminal start instruction, to obtain an engine storage metadata information set, wherein the engine storage metadata information set is a metadata information set for persistent data stored when a data computing terminal set performs data persistence; an acquisition unit configured to acquire required metadata directory information for the above-described computing terminal start instruction; a loading unit configured to store a metadata information set and the required metadata directory information according to the engine, and load a metadata information set corresponding to the required metadata directory information; and the execution unit is configured to execute the starting operation of the computing terminal according to the metadata information set.
Alternatively, the loading unit may be configured to: determining differential metadata catalog information between the engine storage metadata catalog information set and the desired metadata catalog information; and in response to determining that the difference metadata catalog information is not empty, reading a difference metadata information set corresponding to the difference metadata catalog information from the data storage terminal.
Alternatively, the loading unit may be configured to: in response to determining that the differential metadata directory information is empty, the engine storage metadata information set is determined to be the metadata information set.
Optionally, the engine storage metadata information set is obtained by: and in response to receiving an engine storage metadata information reading request, acquiring the engine storage metadata information set from an engine storage metadata file, wherein the engine storage metadata file is a file which is deployed in advance in a data storage terminal, each persistent storage engine in a persistent storage engine set corresponds to the engine storage metadata file with the same data sharing, and a data computing end in the data computing end set corresponds to the persistent storage engine in the persistent storage engine set.
Optionally, a built-in target persistent storage engine exists at the data computing expansion end of the data computing end set, and the target persistent storage engine and the persistent storage engine correspondingly have the same data sharing and store metadata files.
Optionally, the engine storage metadata information in the engine storage metadata file is added by: in response to receiving the data persistence instruction, determining a data computing end which performs metadata writing in and is concentrated by the data computing end as a target data computing end; in response to determining that the target data computing end works normally, indicating a built-in persistent storage engine of the target data computing end to store metadata information corresponding to the persistent data, and obtaining engine storage metadata information; and instructing the target data computing end to store the engine storage metadata information into the corresponding file position in the engine storage metadata file.
Optionally, the method further comprises: in response to determining that the target data computing end is abnormal in operation, determining a second target data computing end in the set of data computing ends as a data computing end for performing metadata writing so as to add metadata information stored by an engine.
In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program when executed by a processor implements a method as described in any of the implementations of the first aspect.
In a fifth aspect, some embodiments of the present disclosure provide a computer program product comprising a computer program which, when executed by a processor, implements the method described in any of the implementations of the first aspect above.
The above embodiments of the present disclosure have the following advantageous effects: according to the method for starting the computing terminal based on the storage and calculation separation cloud primary number bin, which is disclosed by the embodiment of the invention, the starting of the data computing terminal can be realized rapidly and efficiently. In particular, the reason for the related data computing terminals not being efficient and fast to start up is that: for scanning of massive metadata files, if the IOPS of the corresponding disk is not high enough, the starting time of the subsequent computing nodes is long, and the starting efficiency is low. Based on this, the computing terminal startup method of some embodiments of the present disclosure first initializes a built-in persistent storage engine to acquire an engine storage metadata information set in response to receiving a computing terminal startup instruction. Wherein the engine storage metadata information set is a metadata information set for persistent data stored when the data computing terminal set performs data persistence. Here, by acquiring a partial metadata information set (i.e., an engine storing metadata information set) required for the data computing terminal to be started by using a built-in persistent storage engine of the data computing terminal to be started, I/O interaction of the corresponding disk can be avoided, and interaction time can be reduced, so that extremely high IOPS are not required for the corresponding disk. Next, the required metadata directory information for the computing terminal startup instruction is acquired. The metadata catalog information required by the starting of the data computing terminal to be started is acquired, and the loading of the subsequent metadata information is facilitated. Furthermore, according to the engine storing the metadata information set and the required metadata directory information, the metadata information set corresponding to the required metadata directory can be loaded efficiently and quickly. And finally, executing the starting operation of the computing terminal according to the metadata information set so as to realize the efficient starting of the data computing terminal to be started. In summary, the built-in persistent storage engine is used for primarily acquiring partial metadata information set required by starting the computing terminal, so that the I/O (input/output) interaction of a subsequent disk is reduced, the starting time is shortened, and the starting efficiency is improved.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a schematic diagram of one application scenario of a computing terminal startup method based on a memory-based split cloud primary bin according to some embodiments of the present disclosure;
FIG. 2 is a flow chart of some embodiments of a computing terminal startup method based on a memory separation cloud primary bin according to the present disclosure;
FIG. 3 is a schematic diagram of a persistent storage engine set to engine storage metadata file correspondence in some embodiments of a computing terminal startup method based on a split cloud primary bin according to the present disclosure;
FIG. 4 is a schematic diagram of a correspondence between a data computing extension and an engine storage metadata file in some embodiments of a computing terminal startup method based on a storage separation cloud primary number bin according to the present disclosure;
FIG. 5 is a schematic diagram of engine storage metadata information storage in some embodiments of a computing terminal startup method based on a split cloud primary bin according to the present disclosure;
FIG. 6 is a flowchart of further embodiments of a computing terminal startup method based on a memory separation cloud primary bin according to the present disclosure;
FIG. 7 is a schematic structural diagram of some embodiments of a computing terminal activation device based on a storage separation cloud primary bin according to the present disclosure;
fig. 8 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. Embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
Operations such as collection, storage, use, etc. of information (e.g., metadata information) referred to in the present disclosure involve the relevant organization or individual being up to the end of obligations including developing information security impact assessment, fulfilling informed obligations to information subjects, soliciting authorized consent from information subjects in advance, etc., before performing the corresponding operations.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 is a schematic diagram of an application scenario of a computing terminal startup method based on a storage separation cloud primary bin according to some embodiments of the present disclosure.
In the application scenario of fig. 1, first, in response to receiving a computing terminal start-up instruction, the electronic device 101 may initialize the built-in persistent storage engine 103 to obtain the engine storage metadata information set 104. Wherein the engine storage metadata information set 104 is a metadata information set for persistent data stored when the data computing side set performs data persistence. The electronic device 101 may then obtain the required metadata catalog information 105 for the computing terminal startup instructions described above. Next, the electronic device 101 may store the metadata information set 104 and the required metadata directory information 105 according to the engine, and load the metadata information set 106 corresponding to the required metadata directory information 105. Finally, the electronic device 101 may perform a computing terminal startup operation according to the metadata information set 106 described above.
The electronic device 101 may be hardware or software. When the electronic device is hardware, the electronic device may be implemented as a distributed cluster formed by a plurality of servers or terminal devices, or may be implemented as a single server or a single terminal device. When the electronic device is embodied as software, it may be installed in the above-listed hardware device. It may be implemented as a plurality of software or software modules, for example, for providing distributed services, or as a single software or software module. The present invention is not particularly limited herein.
It should be understood that the number of electronic devices in fig. 1 is merely illustrative. There may be any number of electronic devices as desired for an implementation.
With continued reference to fig. 2, a flow 200 of some embodiments of a computing terminal startup method based on a memory-based split cloud primary bin according to the present disclosure is shown. The method for starting the computing terminal comprises the following steps:
in step 201, in response to receiving a computing terminal start instruction, a built-in persistent storage engine is initialized to obtain an engine storage metadata information set.
In some embodiments, in response to receiving a computing terminal start-up instruction, the built-in persistent storage engine is initialized, and an execution body of the above-described computing terminal start-up method (e.g., the electronic device 101 shown in fig. 1) may initialize the built-in persistent storage engine to obtain an engine storage metadata information set. The computing terminal start instruction may be an instruction indicating that the execution subject (i.e., the data computing terminal to be started) is started. The persistent storage engine may be a RocksDB engine. The execution body may be a data computing end to be started in the data computing end set. The set of data computing terminals may be a cluster of computing nodes. Here, the calculation of the resource and the storage of the resource are performed by different physical machines (i.e., corresponding to the data calculation terminal and the data storage terminal), respectively. The data computing terminal and the data storage terminal can realize uniform cloud primary biochemistry so as to realize efficient management, greatly improve the resource utilization rate, reduce the operation cost, further reduce the cost and increase the efficiency, realize the real-time digital bin technology, and can respond PB (PetaByte) level data in a second level or even a millisecond level through aggregation analysis. The engine storage metadata information set is a metadata information set for persistent data stored when the data computing terminal set performs data persistence. The engine storage metadata information may be corresponding metadata information corresponding to the persistent data. The engine storage metadata information may be description information of data. For example, the engine storage metadata information may be a "/column. Bin" file, a "/count. Txt" file, or a "/columns. Txt" file.
It should be noted that, each data computing terminal in the data computing terminal set has a built-in persistent storage engine deployed in advance.
In some optional implementations of some embodiments, the engine storage metadata information set is obtained by:
in response to receiving an engine storage metadata information read request, the set of engine storage metadata information is obtained from an engine storage metadata file. The engine storage metadata file is a file which is deployed in advance in the data storage terminal. Each of the set of persistent storage engines corresponds to the engine storage metadata file for which the same data sharing exists (i.e., each of the set of persistent storage engines shares an engine storage metadata file). And the data computing terminals in the data computing terminal set and the persistent storage engines in the persistent storage engine set have a one-to-one correspondence. The engine storage metadata file may be a metadata file stored in a persistent storage engine set in a distributed file system in the data storage terminal.
As an example, referring to fig. 3, the data computing terminal set 301 includes: a data computing terminal 302, a data computing terminal 303, a data computing terminal 304, and a data computing terminal 305. The data computing terminal 302 has a built-in persistent storage engine 306. The data computing terminal 303 has a built-in persistent storage engine 307. The data computing terminal 304 has a built-in persistent storage engine 308. The data computing terminal 305 has a built-in persistent storage engine 309. The persistent storage engine 306, the persistent storage engine 307, the persistent storage engine 308, and the persistent storage engine 309 have an engine storage metadata file 311 in the data storage terminal 310 in common.
Optionally, there is a built-in target persistent storage engine for the data compute extensions of the data compute set. The target persistent storage engine and the persistent storage engine correspond to the engine storage metadata file in which the same data sharing exists (i.e., the target persistent storage engine and each persistent storage engine in the set of persistent storage engines share an engine storage metadata file).
As an example, referring to fig. 4, the data computing terminal set 301 includes: a data computing terminal 302, a data computing terminal 303, a data computing terminal 304, and a data computing terminal 305. The data computing terminal 302 has a built-in persistent storage engine 306. The data computing terminal 303 has a built-in persistent storage engine 307. The data computing terminal 304 has a built-in persistent storage engine 308. The data computing terminal 305 has a built-in persistent storage engine 309. The persistent storage engine 306, the persistent storage engine 307, the persistent storage engine 308, and the persistent storage engine 309 have an engine storage metadata file 311 in the data storage terminal 310 in common. The extended data computing terminals corresponding to the data computing terminal set 301 are data computing terminals 401. The data computing terminal 401 has a built-in persistent storage engine 402. Then, the persistent storage engine 306, the persistent storage engine 307, the persistent storage engine 308, the persistent storage engine 309, and the persistent storage engine 402 have the engine storage metadata file 311 in the data storage terminal 310 in common.
Optionally, the engine storage metadata information in the engine storage metadata file is added by:
first, in response to receiving a data persistence instruction, a data computing terminal that performs metadata writing in the set of data computing terminals is determined as a target data computing terminal.
The data persistence instruction may be an instruction that instructs the data computing end to store persistence data to the data storage end. The target data computing end may be a master data computing end in a data computing end set (i.e., a master node in a computing node cluster). In practice, the main data calculation end may be preset.
And secondly, in response to determining that the target data computing end works normally, indicating a built-in persistent storage engine of the target data computing end to store metadata information corresponding to the persistent data, and obtaining engine storage metadata information.
And thirdly, instructing the target data computing end to store the engine storage metadata information into the corresponding file position in the engine storage metadata file.
Optionally, after the "third step", the steps may further include:
in response to determining that the target data computing end is abnormal in operation, determining a second target data computing end in the set of data computing ends as a data computing end for performing metadata writing so as to add metadata information stored by an engine. In practice, the abnormal operation of the target data computing end may be a downtime problem of the target data computing end. In practice, the second target data computing terminal may be any one of the data computing terminals in the data computing terminal set excluding the target data computing terminal set.
As an example, referring to fig. 5, the data computing terminal set 301 includes: a data computing terminal 302, a data computing terminal 303, a data computing terminal 304, and a data computing terminal 305. The data computing terminal 302 has a built-in persistent storage engine 306. The data computing terminal 303 has a built-in persistent storage engine 307. The data computing terminal 304 has a built-in persistent storage engine 308. The data computing terminal 305 has a built-in persistent storage engine 309. The persistent storage engine 306, the persistent storage engine 307, the persistent storage engine 308, and the persistent storage engine 309 have an engine storage metadata file 311 in the data storage terminal 310 in common. The data computing terminals 302 described above may be master data computing terminals (i.e., master nodes) in the data computing terminal set 301. When the data computing terminal set 301 receives the data persistence instruction, the data computing terminal 302 is instructed to store the persistence data 501 in the value data storage terminal 310, and the engine storage metadata information 502 corresponding to the persistence data 501 is stored in the engine storage metadata file 311 in the data storage terminal 310. When the data computing terminal 302 is abnormal in operation, the data computing terminal set 301 will take the data computing terminal 302 off line and take the data computing terminal 303 as the master data computing terminal. The data computing terminal set 301 may instruct the data computing terminal 303 to store the persistent data 501 in the value data storage terminal 310 and store the engine storage metadata information 502 corresponding to the persistent data 501 in the engine storage metadata file 311 in the data storage terminal 310.
Step 202, obtaining the required metadata catalog information for the computing terminal startup instruction.
In some embodiments, the executing entity may obtain the required metadata directory information for the computing terminal startup instruction in a wired manner or a wireless manner. Wherein the required metadata directory information may be a directory of metadata information required for executing the subject startup.
As an example, the above-described execution subject may traverse from a distributed file system in the data storage terminal to generate a metadata directory list as the required metadata directory information.
Step 203, according to the engine storing metadata information set and the required metadata catalog information, loading the metadata information set corresponding to the required metadata catalog information.
In some embodiments, the execution body may store a metadata information set and the required metadata directory information according to the engine, and load a metadata information set corresponding to the required metadata directory information.
As an example, first, the above-described execution subject may generate a metadata information storage directory corresponding to an engine storage metadata information set. Then, a data difference between the metadata information storage directory and the above-described required metadata directory is determined. And finally, acquiring metadata information aiming at the data difference so as to realize loading of the metadata information set corresponding to the required metadata catalogue.
And step 204, executing the starting operation of the computing terminal according to the metadata information set.
In some embodiments, the executing body may execute the computing terminal start operation according to the metadata information set.
As an example, the execution body may execute the computing terminal start operation according to the metadata information set according to the computing terminal start method. The specific starting method is not described herein.
The above embodiments of the present disclosure have the following advantageous effects: according to the method for starting the computing terminal based on the storage and calculation separation cloud primary number bin, which is disclosed by the embodiment of the invention, the starting of the data computing terminal can be realized rapidly and efficiently. In particular, the reason for the related data computing terminals not being efficient and fast to start up is that: for scanning of massive metadata files, if the IOPS of the corresponding disk is not high enough, the starting time of the subsequent computing nodes is long, and the starting efficiency is low. Based on this, the computing terminal startup method of some embodiments of the present disclosure first initializes a built-in persistent storage engine to acquire an engine storage metadata information set in response to receiving a computing terminal startup instruction. Wherein the engine storage metadata information set is a metadata information set for persistent data stored when the data computing terminal set performs data persistence. Here, by acquiring a partial metadata information set (i.e., an engine storing metadata information set) required for the data computing terminal to be started by using a built-in persistent storage engine of the data computing terminal to be started, I/O interaction of the corresponding disk can be avoided, and interaction time can be reduced, so that extremely high IOPS are not required for the corresponding disk. Next, the required metadata directory information for the computing terminal startup instruction is acquired. The metadata catalog information required by the starting of the data computing terminal to be started is acquired, and the loading of the subsequent metadata information is facilitated. Furthermore, according to the engine storing the metadata information set and the required metadata directory information, the metadata information set corresponding to the required metadata directory can be loaded efficiently and quickly. And finally, executing the starting operation of the computing terminal according to the metadata information set so as to realize the efficient starting of the data computing terminal to be started. In summary, the built-in persistent storage engine is used for primarily acquiring partial metadata information set required by starting the computing terminal, so that the I/O interaction of a subsequent disk is reduced, the starting time is shortened, and the starting efficiency is improved.
With further reference to fig. 6, a flow 600 of further embodiments of a computing terminal startup method based on a memory-based split cloud primary bin according to the present disclosure is shown. The method for starting the computing terminal comprises the following steps:
in step 601, in response to receiving a computing terminal start instruction, a built-in persistent storage engine is initialized to obtain an engine storage metadata information set.
Step 602, obtaining the required metadata directory information for the above-mentioned computing terminal start instruction.
Step 603, determining difference metadata catalog information between the engine storage metadata information set and the required metadata catalog information.
In some embodiments, the executing body (e.g., the electronic device 101 shown in FIG. 1) may determine differential metadata catalog information between the set of engine storage metadata information and the desired metadata catalog information.
In step 604, in response to determining that the differential metadata directory information is not null, a differential metadata information set corresponding to the differential metadata directory information is read from the data storage terminal.
In some embodiments, in response to determining that the differential metadata directory information is not empty, the execution body may read a differential metadata information set corresponding to the differential metadata directory information from the data storage terminal.
In some alternative implementations of some embodiments, after step 604, the steps further include:
in response to determining that the differential metadata directory information is empty, the execution body may determine the engine storage metadata information set as the metadata information set.
Step 605, executing the operation of starting the computing terminal according to the metadata information set.
In some embodiments, the specific implementation of steps 601-602, 605 and the technical effects thereof may refer to steps 201-202, 204 in the corresponding embodiment of fig. 2, which are not described herein.
As can be seen from fig. 6, compared with the description of some embodiments corresponding to fig. 2, in some embodiments corresponding to fig. 6, the process 600 of the computing terminal starting method based on the storage separation cloud primary number bin obtains the complete metadata information set by determining the differential metadata directory information between the engine storage metadata information set and the above-mentioned required metadata directory to scan the differential metadata information set corresponding to the differential metadata directory information from the data storage terminal, so as to ensure the normal starting of the subsequent data computing terminal. Based on the method, the I/O interaction of the corresponding disk can be reduced by preliminarily acquiring the metadata information set stored by the engine and acquiring the difference metadata information set, so that the starting time is shortened, and the starting efficiency is improved.
With further reference to fig. 7, as an implementation of the method shown in the foregoing figures, the present disclosure provides some embodiments of a computing terminal starting device based on a storage separation cloud primary number bin, where the device embodiments correspond to those method embodiments shown in fig. 2, and the computing terminal starting device may be specifically applied to various electronic devices.
As shown in fig. 7, a computing terminal starting apparatus 700 includes: an initialization unit 701, an acquisition unit 702, a loading unit 703, and an execution unit 704. The initialization unit 701 is configured to initialize a built-in persistent storage engine in response to receiving a computing terminal start instruction, so as to obtain an engine storage metadata information set, wherein the engine storage metadata information set is a metadata information set for persistent data stored when a data computing terminal set performs data persistence; an obtaining unit 702 configured to obtain required metadata directory information for the above-described computing terminal start instruction; a loading unit 703 configured to load a metadata information set corresponding to the required metadata directory information according to the engine storage metadata information set and the required metadata directory information; and an execution unit 704 configured to execute a computing terminal start operation according to the metadata information set.
In some alternative implementations of some embodiments, the loading unit 703 may be further configured to: determining differential metadata catalog information between the engine storage metadata catalog information set and the desired metadata catalog information; and in response to determining that the difference metadata catalog information is not empty, reading a difference metadata information set corresponding to the difference metadata catalog information from the data storage terminal.
In some alternative implementations of some embodiments, the loading unit 703 may be further configured to: in response to determining that the differential metadata directory information is empty, the engine storage metadata information set is determined to be the metadata information set.
In some optional implementations of some embodiments, in response to receiving an engine storage metadata information reading request, the engine storage metadata information set is obtained from an engine storage metadata file, where the engine storage metadata file is a file pre-deployed in a data storage terminal, each persistent storage engine in a persistent storage engine set corresponds to the engine storage metadata file having the same data sharing, and a data computing end in the data computing end set has a correspondence with a persistent storage engine in the persistent storage engine set.
In some optional implementations of some embodiments, there is a built-in target persistent storage engine for the data computing extension of the set of data computing ends, where the target persistent storage engine and the persistent storage engine correspond to the engine storage metadata file for which there is the same data sharing.
In some optional implementations of some embodiments, the engine storage metadata information in the engine storage metadata file is added by: in response to receiving the data persistence instruction, determining a data computing end which performs metadata writing in and is concentrated by the data computing end as a target data computing end; in response to determining that the target data computing end works normally, indicating a built-in persistent storage engine of the target data computing end to store metadata information corresponding to the persistent data, and obtaining engine storage metadata information; and instructing the target data computing end to store the engine storage metadata information into the corresponding file position in the engine storage metadata file.
It will be appreciated that the elements described in the computing terminal activation device 700 correspond to the various steps in the method described with reference to fig. 2. Thus, the operations, features and advantages described above with respect to the method are equally applicable to the computing terminal activation device 700 and the units contained therein, and are not described here again.
Referring now to fig. 8, a schematic diagram of an electronic device 800 (e.g., electronic device 101 of fig. 1) suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device shown in fig. 8 is merely an example, and should not impose any limitations on the functionality and scope of use of embodiments of the present disclosure.
As shown in fig. 8, the electronic device 800 may include a processing means (e.g., a central processor, a graphics processor, etc.) 801 that may perform various appropriate actions and processes according to programs stored in a read-only memory 802 or programs loaded from a storage means 808 into a random access memory 803. In the random access memory 803, various programs and data necessary for the operation of the electronic device 800 are also stored. The processing device 801, the read-only memory 802, and the random access memory 803 are connected to each other through a bus 804. An input/output interface 805 is also connected to the bus 804.
In general, the following devices may be connected to the input/output interface 805: input devices 806 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 807 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, etc.; storage 808 including, for example, magnetic tape, hard disk, etc.; communication means 809. The communication means 809 may allow the electronic device 800 to communicate wirelessly or by wire with other devices to exchange data. While fig. 8 shows an electronic device 800 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 8 may represent one device or a plurality of devices as needed.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via communication device 809, or from storage device 808, or from read only memory 802. The above-described functions defined in the methods of some embodiments of the present disclosure are performed when the computer program is executed by the processing device 801.
It should be noted that, in some embodiments of the present disclosure, the computer readable medium may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: in response to receiving a computing terminal starting instruction, initializing a built-in persistence storage engine to obtain an engine storage metadata information set, wherein the engine storage metadata information set is a metadata information set which is stored by a data computing terminal set and aims at persistence data when data persistence is executed; acquiring required metadata catalog information aiming at the starting instruction of the computing terminal; according to the engine storage metadata information set and the required metadata catalog information, loading the metadata information set corresponding to the required metadata catalog information; and executing the starting operation of the computing terminal according to the metadata information set.
Computer program code for carrying out operations for some embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes an initialization unit, an acquisition unit, a loading unit, and an execution unit. The names of these units do not constitute a limitation on the unit itself in some cases, and for example, the acquisition unit may also be described as "a unit that acquires the required metadata directory information for the above-described computing terminal startup instruction".
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
Some embodiments of the present disclosure also provide a computer program product comprising a computer program which, when executed by a processor, implements any one of the above-described computing terminal startup methods based on a storage separation cloud primary number bin.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually substituted with (but not limited to) the features having similar functions disclosed in the embodiments of the present disclosure.
Claims (9)
1. A computing terminal starting method based on a storage and calculation separation cloud primary number bin comprises the following steps:
in response to receiving a computing terminal starting instruction, initializing a built-in persistence storage engine to obtain an engine storage metadata information set, wherein the engine storage metadata information set is a metadata information set which is stored by a data computing terminal set and aims at persistence data when data persistence is executed;
acquiring required metadata catalog information for a starting instruction of the computing terminal;
determining differential metadata catalog information between the engine storage metadata information set and the required metadata catalog information;
Reading a difference metadata information set corresponding to the difference metadata directory information from a data storage terminal in response to determining that the difference metadata directory information is not empty;
and executing the starting operation of the computing terminal according to the metadata information set.
2. The method of claim 1, wherein after the reading the differential metadata information set corresponding to the differential metadata directory information from the data storage terminal in response to determining that the differential metadata directory information is not empty, the method further comprises:
in response to determining that the differential metadata catalog information is empty, the engine storage metadata information set is determined to be the metadata information set.
3. The method of claim 1, wherein the engine storage metadata information set is obtained by:
and responding to the received engine storage metadata information reading request, acquiring the engine storage metadata information set from an engine storage metadata file, wherein the engine storage metadata file is a file which is deployed in advance in a data storage terminal, each persistent storage engine in a persistent storage engine set corresponds to the engine storage metadata file with the same data sharing, and a data computing end in a data computing end set has a corresponding relation with the persistent storage engines in the persistent storage engine set.
4. The method of claim 3, wherein a built-in target persistent storage engine exists for the data compute extension of the set of data compute ends, the target persistent storage engine corresponding to the engine storage metadata file for which the same data sharing exists.
5. A method according to claim 3, wherein the engine storage metadata information in the engine storage metadata file is added by:
in response to receiving the data persistence instruction, determining a data computing end which performs metadata writing in and is concentrated by the data computing end as a target data computing end;
in response to determining that the target data computing end works normally, indicating a built-in persistent storage engine of the target data computing end to store metadata information corresponding to the persistent data, and obtaining engine storage metadata information;
and indicating the target data computing end to store the engine storage metadata information into the corresponding file position in the engine storage metadata file.
6. The method of claim 5, wherein the method further comprises:
and in response to determining that the target data computing end works abnormally, determining a second target data computing end in the data computing end set as the data computing end for executing metadata writing so as to add the engine storage metadata information.
7. A computing terminal starting device based on a storage separation cloud primary number bin, comprising:
an initialization unit configured to initialize a built-in persistent storage engine in response to receiving a computing terminal start instruction to acquire an engine storage metadata information set, wherein the engine storage metadata information set is a metadata information set for persistent data stored when a data computing terminal set performs data persistence;
an acquisition unit configured to acquire required metadata directory information for the computing terminal start instruction;
a loading unit configured to determine differential metadata catalog information between the engine storage metadata information set and the required metadata catalog information; reading a difference metadata information set corresponding to the difference metadata directory information from a data storage terminal in response to determining that the difference metadata directory information is not empty;
and the execution unit is configured to execute the starting operation of the computing terminal according to the metadata information set.
8. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.
9. A computer readable medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310527772.XA CN116302140B (en) | 2023-05-11 | 2023-05-11 | Method and device for starting computing terminal based on storage and calculation separation cloud primary number bin |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310527772.XA CN116302140B (en) | 2023-05-11 | 2023-05-11 | Method and device for starting computing terminal based on storage and calculation separation cloud primary number bin |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116302140A CN116302140A (en) | 2023-06-23 |
| CN116302140B true CN116302140B (en) | 2023-09-22 |
Family
ID=86796216
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310527772.XA Active CN116302140B (en) | 2023-05-11 | 2023-05-11 | Method and device for starting computing terminal based on storage and calculation separation cloud primary number bin |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116302140B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377436A (en) * | 2019-07-12 | 2019-10-25 | 清华大学 | Data storage access method, equipment and the device of persistence memory |
| CN114528255A (en) * | 2022-01-13 | 2022-05-24 | 深圳爱捷云科技有限公司 | Metadata management method, electronic device and computer program product |
| CN115407933A (en) * | 2022-06-30 | 2022-11-29 | 中国科学院计算技术研究所 | File metadata storage method and system based on nonvolatile memory |
| CN115599747A (en) * | 2022-04-22 | 2023-01-13 | 北京志凌海纳科技有限公司（Cn） | Metadata synchronization method, system and equipment of distributed storage system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9104619B2 (en) * | 2010-07-23 | 2015-08-11 | Brocade Communications Systems, Inc. | Persisting data across warm boots |
| US10936713B2 (en) * | 2015-12-17 | 2021-03-02 | The Charles Stark Draper Laboratory, Inc. | Techniques for metadata processing |
-
2023
- 2023-05-11 CN CN202310527772.XA patent/CN116302140B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377436A (en) * | 2019-07-12 | 2019-10-25 | 清华大学 | Data storage access method, equipment and the device of persistence memory |
| CN114528255A (en) * | 2022-01-13 | 2022-05-24 | 深圳爱捷云科技有限公司 | Metadata management method, electronic device and computer program product |
| CN115599747A (en) * | 2022-04-22 | 2023-01-13 | 北京志凌海纳科技有限公司（Cn） | Metadata synchronization method, system and equipment of distributed storage system |
| CN115407933A (en) * | 2022-06-30 | 2022-11-29 | 中国科学院计算技术研究所 | File metadata storage method and system based on nonvolatile memory |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116302140A (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220253458A1 (en) | Method and device for synchronizing node data | |
| CN111432001B (en) | Method, apparatus, electronic device and computer readable medium for jumping scenes | |
| CN111625422B (en) | Thread monitoring method, thread monitoring device, electronic equipment and computer readable storage medium | |
| CN114328097A (en) | File monitoring method and device, electronic equipment and storage medium | |
| CN111596992B (en) | Navigation bar display method and device and electronic equipment | |
| CN111309366B (en) | Method, device, medium and electronic equipment for managing registration core | |
| CN110888773B (en) | Method, device, medium and electronic equipment for acquiring thread identification | |
| CN110990038B (en) | Method, apparatus, electronic device and medium for applying local update | |
| CN116302140B (en) | Method and device for starting computing terminal based on storage and calculation separation cloud primary number bin | |
| CN116302271A (en) | Page display method and device and electronic equipment | |
| CN111756833B (en) | Node processing method, node processing device, electronic equipment and computer readable medium | |
| CN111309367B (en) | Method, device, medium and electronic equipment for managing service discovery | |
| CN111404824B (en) | Method, apparatus, electronic device, and computer-readable medium for forwarding request | |
| CN117369953B (en) | Mirror synchronization method, device, equipment and storage medium | |
| CN116361254B (en) | Image storage method, apparatus, electronic device, and computer-readable medium | |
| CN111626787B (en) | Resource issuing method, device, medium and equipment | |
| CN113157365B (en) | Program running method, program running device, electronic equipment and computer readable medium | |
| CN113765694B (en) | Log information display method, device, electronic equipment and computer readable medium | |
| CN117369952B (en) | Cluster processing method, device, equipment and storage medium | |
| CN116319322B (en) | Power equipment node communication connection method, device, equipment and computer medium | |
| CN110855767B (en) | Method, device, equipment and storage medium for responding operation request | |
| CN116820354B (en) | Data storage method, data storage device and data storage system | |
| CN111314457B (en) | Method and device for setting virtual private cloud | |
| CN116414528A (en) | Transaction processing method, system, equipment and storage medium | |
| CN117579474A (en) | Data collection method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |